I don't take mine out if I'm just bobbing out for an hour or two. Like a lot of older people I grew up without a phone in the house, didn't know anyone who had a phone and so far, touch wood, have never needed to call anyone urgently.
I can't think of anyone who would need to contact me immediately, either.
If I were to have an accident while I'm out for that short time I know there would be lots of people nearby who have a phone and would call for help (after videoing the scene first, I expect).
I always take my phone when I go out in the car. Far more likely to need it for breakdown or accident but I keep it in the glovebox, turned off.