Hi, this is not a legal reply, but having been in a similar situation I understand your worries. How old is your son? My eldest was 11 when his dad left so was capable of looking after himself and his younger sister when they were supposedly in the care of their father. On one very rare occassion that my ex actually came into my house [on bringing the children home] he was very obviously stoned, I totally lost it with him, I couldn't believe that he had driven the car in the state that he was in....and he only had a couple of miles to drive. I told him in no uncertain terms that if I saw him in this state again I would report him to the police, I also made my son promise that if his dad was ever in this state again [whilst looking after my kids] that he would tell me and I could get them home safetly.
If your ex insists on taking your son to his house.... you have a complete dilemna...He may be 'straight' when he picks him up....but who knows what happens whist your son is with him? Unless your child is of an age where he can phone you and tell you he is worried DON'T let him go! You are not denying access and if your ex can not grow up and act like a responsible adult he does not deserve to look after his son.
Hopefully someone will answer your question on a legal level, but I completely understand your concern..... So for now....say no
Lisa x