Sounds like he's OK while you're around, but perhaps less happy in other areas without your presence. If this is what it is, maybe you could just spend a little time with him in places you wouldn't normally linger - hallways, landings, spare room etc.
When I got my cat, the rescue recommended keeping him in one room for the first week, but it sounds like your cat has had the run of the place - once he gets the idea that the house is a safe place he should be fine.