You can be charged by the agent even though the agent is working for the landlord but this must be agreed by you in advance - i.e. you should have been given terms and conditions to sign which would have stipulated the fees. Check the documents you signed, if no fee is mentioned then you shouldn't have to pay. The downside of doing this is that the agent will possibly try to get the landlord to evict you and get new tenants in who they can charge. A bit risky from the agent's point of view though as the landlord may drop the agent.
If you're not happy with charges that the agent is trying to make you pay then you can contact the landlord direct and put your case. The agent is legally obliged to provide an address for the landlord. I'm sure the landlord would rather keep the existing tenants (you) rather than lose money by having a void period until new tenants are found.