If I were to go swimming 3-4 times a week would this type of exercise be enough to keep me fit (taking into account diet, etc) or would I need to incorporate other exercise as well?
It is not so much about how many times I go swimming but more about type of exercise. I'm in my late 40's.
Swimming is very good exercise especially for the heart and lungs which is the basis of everything. You might want to do something else as well just for a change of scene, cycling maybe.