Boo is right, it stops them being absorbent. My mum does it all the time, to make them soft, but everytime I go to stay with her I take my own towel. Hers just move the water around your body!
Debenhams used to display a notice not to use fabric conditioner on towels as it could also lead to 'fibre loss'.
Basically, conditioner puts a plastic coating on the fibres, making them slightly slippery and less absorbent. (Hair conditioner does the same!). This coating also slows the evaporation of the water from the fibre, making it harder to dry. (And, as you've probably noticed, this applies to your hair, too!).
You could say that about some washing powders & other things that make contact with your skin on a daily basis. I think it's all down to personal choice really.