Firstly, get rid of him - if he just sits there chatting, and particularly on his mobile phone, he is not worth having. His total concentration should be on you, your driving, and what's going on on the roads.
I had one driving instructor who was absolutely obsessed with his car - he'd yell at me if there was the slightest chance I might brush the hedge etc and would reel off a list of what each part of the car would cost to get fixed. I got rid of his as soon as I could.
You wouldn't get penalised for taking a test in your own car - many people do. It's important you feel happy and confident in what you're driving. If you're only failing on driving too slow, and this is caused by you being nervous in your instructor's car, then have a go in your own.
If there are several other minors that contribute to a fail, or it's doing the wrong speed at the wrong time (such as coming up to junctions etc) then I'd probably find a different instructor to have a few lessons with before you have another test. They may be more aware of small things that you may be doing wrong, and can hopefully help you put them right. Learning to drive can be very frustrating - good luck!