Yes, but they need to give you at least 24 hours notice before they do. if they turn up on the doorstep, you do not have to invite them in, although they can then say they will be back in 24 hours.
Obviously, if you want to keep relations cordial, then you should let them in
I understand that I should let them in and they should give 24 hours notice but what if they want to come at a time where it is not possible for me to be there are they allowed to go anyway and let themselves in?
Actually, legally they can't demand to be let in even with 24 hours notice and even if it says they can in your AST. Having said that, most tenants are reasonable and allow the landlord to have access at a reasonable time if needed. So, you can deny access but you may end up with a s21 notice (notice to quit) if you come across as unreasonable.