Yes, you can put them back in their garden. Yes, you are responsible for getting rid of them if you ask them and they don�t want them.
You are also entitled to remove any branch overhanging your garden if it is encroaching on your garden, which again you can return to them.
In the interests of a happy neighbourhood I strongly recommend that you don�t cut off a branch or lob conkers over the wall without talking to your neighbours first.
And be careful that the tree is not protected or you are in a conservation area.
Best just to clear them away yourself.