I don't really mind, Its not really a big problem in my opinion, I dont really mind earning back the points or anything, he dosent take that mutch so I don't really mind!
He's never taken more than 1000 GP from me. I don't think he's really that big of a deal. He adds some flavour and chance to the game, really, just like those awful poos and snakes. (I happen to think the snakes and poos are much more annoying than the robber...)