Why bother keeping the bottom of your board fair when you have those things dragging through the water? I know that it probably doesn't make that much difference but those little things trigger my mild OCD about drag, etc.
And, yes, I'm the type of guy who got box fillers to seal up my empty track slots on the bottom of my board.