As has been said here many times before, there's no hard and fast rule.
However, if you think for a moment about how you'd erect a fence at the front of your property, next to the footpath, you'll probably be able to see that most people erect fences with the posts on 'their' side. (i.e. facing inwards towards their own house). So, if the fence between your garden and that of your neighbour was erected in a similar fashion, it's your neighbour who owns it.
However owning a fence isn't the same as having responsibility for repairing it. Unless (unusually) there's a covenant in the property deeds of one of your houses, requiring the owner to keep the fence in a good state of repair, neither of you is responsible for repairing it.
If, as seems likely, it's your who neighbour actually owns the fence, he's perfectly entitled (subject to the provisions of any covenant, as above) to leave it exactly as it is or to simply remove it altogether. (He could then say to you, "OK, if you want a fence there, you'll have to provide one").