He might find have been finding it difficult fitting in because he's not a (real-life) family member, as many other cast members are:
Buster Brown (Danny O'Carroll ) is played by Brendan O'Carroll's son.
Maria Brown (Fiona O'Carroll) is played by his daughter.
Winnie McGoogan (Eilish O'Carroll) is played his sister.
Cathy Brown (Jennifer Gibney) is played by his wife.
Bono Brown (Jamie O’Carroll) is played by his grandson.
Betty Brown (Amanda Woods) is played by his daughter-in-law.
Trevor Brown (Martin Delany) is played by his son-in-law.
Dermot Brown (Paddy Houlihan) isn't a family member but he's Danny O'Carroll's best friend.
With the natural closeness of that family grouping, Rory Cowan might well have felt like an outsider for much of his time with the cast.