My husband is disabled so I know what a nightmare it can be! It seems nowadays that virtually all disabled parking spaces are taken. Went shopping today, no disabled spaces - there was a guy parked in the disabled space next to us (we parked in a normal space but there was luckily space to the side). The guy was sat in his car (presumably waiting for his wife to finish shopping) with no badge displayed.
I think the best thing to do (which is what I'm going to start doing), is telling them that they've forgotten to display their badge!
Unfortunately, many car parks will not fine people if they're parked in a space without a badge.
I've thought about keeping some notes in the car to shove on people's windscreens, just saying how selfish they are to be parked there and that we've had to struggle.
So yes, it definitely bothers me! It's not until you're disabled that you realise what a nightmare it is. Also, there are 16,000 blue badges still in use where the holder has died!!!