I've been married for eight years, we work together too. Every now and then, a rumour goes about that there is something going on between us. No s@@t Sherlock - we have the same surname, for a start! I don't even react. Colleagues that know we are married find it amusing, and wind up the situation. 'What, him? Not possibly so, he like blokes' etc. I'm a private person, too. But everyone likes gossip! Don't go to your manager, just looks like you are telling teacher and can't defend yourself. Use a bit of humour, they'll soon get bored and move onto someone or something else.