The only time I have ever noticed that I lost skill points is at pre-school visits.
Usually when you visit and do the little dance game with the nanny you get one skill point for visiting pre-school. But yesterday, I visited and lost a skill point after I got back
I think it might have been because my tama child fell over twice in the dance... - if they only fall once you get a point, if they fall twice you lose a point... this is only my guess - not a proved fact