I used to be in the same position as an Occupational Therapist. I dealt with people with disabilities, both temporary and permanent and with people who had had hip and knee replacements. They would ask me about driving and other activities. I did have to give an opinion on people's activities outside the hospital (living independently at home, handling hot kettles going up and down stairs and so on - risky activities) The line was that we could state what we had seen the patient do on assessment and if that had been in our opinion unsafe then we could suggest an alternative. People are independent creatures with the right to make their own decisions however so we could not and did not say that a person was safe or unsafe, just state what we had seen.
In your case, my advice would be around whether the activity would be safe for their injury/disability/limitation so eg whether it would make the injured limb worse, NOT whether they would be safe in terms of actually controlling the vehicle and making decisions. After all, they might be mad manic drivers and a cast isn't going to fix that!
So If I was you, the kind of thing that i would be saying is "driving will not make your injury worse" or "driving will make your injury worse, don't do it" If you are saying that driving will not make their injury worse then remind them to check with their insurance company and also that any pain meds that they take may affect their ability to drive/operate heavy machinery.
Yes IMO if you say that someone is "safe to drive" then that might be held against you if it could be proved that you said it.
The other thing that I would say (as ex clinician and manager) is that this is worth taking to your line management and professional advice line structure if they are different to ensure that you are all singing from the same hymn sheet