Same with mine.Well, I'm only on Gen4 so far, so I haven't had that many robbers... I suppose it depends how long you've been running a V4
I did notice that I get the Robber more if I am not playing with my V4 much - like just checking it's stats and feeding/snacking.
If I am playing games with it, sending it to school and connecting / using items, etc. I don't seem to get the Robber so much
Maybe the Robber knows when you aren't playing with your tama enough and steals some points from you?!