VAT is chargeable, where applicable, on the value of the goods or services supplied. By handling it in the way you are, you are treating the VAT paid by the drivers as a 'tax deductible expense', in the same way that you might be permitted reimburse the expense cost of a sandwich incurred during a long trip where a meal break was necessary. VAT doesn't work like that.
Contrary to what Buenchico says, the Revenue does lose out. Firstly the drivers cannot be paid for the VAT as an expense, which they then presumably do not declare for SA income tax and NI purposes - the whole sum paid by you to the driver is part of their income.
Secondly, if you are including this cost in an invoice to pass onto a customer and your business is VAT-registered, as you are, this cost must have VAT levied on it.
Traceability is but one small part of it.
Stating the blindingly obvious, there has to be a reason for wanting to attempt to do it this convoluted way in the first place - to save someone money. If someone saves, someone else (aka HMRC) loses.
The VAT arm of HMRC are on a helpline at 0845 010 9000; local tax offices do not deal with VAT.