Smurfchops,
Hi - Firstly why did you plonk this in 'Food And Drink'?
Secondly, it is the fence on the gardens left ( the house looking out at the front's right ) that is your responsibilty. There should be some mention on the Land registry document to say this and if you have more that is yours. The way that you describe the house at the back of you gives me reason to believe that it is their fence so hence, their responsibilty.
My advice, if you do not know them, is to simply call at the house, be courteous and just ask them if that is their fence and please could they fix it. Nine times out of ten you should find them only happy to help, as most do not want any hassle or falling out
Hope it works out.