Extra drag and weight is extra drag and weight. If the flying model is correct you should be lighter, faster and quicker without drop tanks. Aerial combat survival is all about gaining an edge. Don't give your opponent and edge and get all of the small factors on your side that you can. No prizes for second place.
I enter into combat Air to Air or Air to Ground without the drop tanks. I, also, tell my flight when to drop theirs too. Clean is fast, speed is life. Don't ever forget it or compromise. Punch them off near the expected fight (usually prior to the IP - square box on the map). It is a good combat habit to get into regardless of the era or the aircraft.
Besides, think of all the jobs you are providing, requiring new tanks for every mission............