We have been to our surgery a couple of times during this pandemic and I can honestly say its almost the safest place other than our home. Assuming you can drive there of course and that getting there doesnt involve either a taxi or public transport, then I think you will be fine.
They take our temperature before allowing us in, they insist on us wearing a clinical mask issued at the door and its virtually empty inside with max 2/3 in the waiting areas which normally hold 20 or so each.
The invitation shows that the surgery is not overwhelmed with Covid related issues, which is very good news, but as others have said, delay if its still making you anxious.