When my cousin's cat had kittens, her other cat (the kitten's uncle (neutered)) was no problem at all - in fact he was a bit scared of them and spent a lot of time out of the house, in general avoided them until they were bigger! If your male cat has been neutered and is generally a good boy then i think there's no reason to worry that there'll be a problem. He probably won't have much to do with them at all until they are old enough to put up a fight, and when they are very small their mum will keep a protective eye on them.