I'm just back from a long walk ,it's tipping it down .so not a pretty sight with waterproof gear on .usually meet some people at the shop getting out of the cars they don't hesitate to say are you all there , mad ,silly so and so out a morning like that .i often wonder if this is chance to for them to say something of an insult and this is an opening for them to do so .People would not say that under other circumstances .
So people don't wait on the bad weather for a chance to criticise others didn't think so but now the great and good of answerbank have spoke I'm sure .
The majority of people who comment like this are just being friendly in my opinion. Like saying "You must be mad!" to hillwalkers, joggers, short-wearers in winter, etc.
I think you are reading a bit much into the chit chat of strangers.
I'm sure that they don't mean to offend you weecalf. If it bothers you so much then tell them. They will probably be mortified that you have taken offence.