If worried about the calories in the small amount of oil you consume I suspect you may be monitoring your intake a little too closely. My advice is to choose something healthy that you enjoy, and if it really is higher in calories than you'd like, consume less of it.
Weight control should be about instinctively healthy eating anyway, rather than monitoring and denying yourself stuff.