Agree with Evian, it's all in the context.
However if you're asking the question, it suggests to me that you are incomfortable with how this has been expressed to you in the workplace, and that your boss is not exactly your mate, mucker or bosom-buddy.
So first off, only you can judge your boss and workplace, but if you have been hurt or upset by his/her remark it needs to be brought to their attention. This can be done by you, or by your representative either from HR or from your union.
My instincts and experience are that HR are of limited support to employees in need, but only you can judge the situation you have at your workplace.