If I give someone my bank details (ac number and sort code) to transfer money into my account, can they commit a fraud, or am I being a bit distrusting?
There's no need to worry. Giving someone these details is a fairly standard way of avoiding charges for transferring money.
If you've got a cheque book, take a look at it. Every cheque bears the account number and sort code for your bank account but you wouldn't worry about writing a cheque for this reason, would you?