I mean, if we're not keeping time while playing fills we've got a problem. Also, as soon as we're playing anything, including a patterned beat, we are filling up a sonic space in the music.
Another problematic element is the idea that somehow playing some sort of pattern or groove, especially a hypnotic, repeating one, is boring, and the "fills" are where we get to shine . I must admit when I was younger I too, suffered from this limited thinking. No wonder when I listen back to old recordings I hear that impatience and lack of maturity in the grooves I was attempting to play. If all I was thinking about while I was keeping time was about how I was going to dazzle everyone when I played a fill, I wasn't truly in the present, where most great music lives.
Yes, I admit there are times in the music where it makes sense to stay on one area of the drum set and play something solid that doesn't change constantly. ( Philly Joe on "Milestones', anyone? ) Then there are times when the music requests the excitement that moving around the drums and cymbals creates.
Some music requires no fills, some music requires so much moving around the instrument that one can't tell where the "time" stops and the "fills" start. In short, figure out what the music needs and judge yourself accordingly.
As a great example of someone to appears not to be worried and/or making value judgments about the dreaded time and fills, I give you the wonderful Paul DeLong. Yes, sometimes he spends more time on the toms and other locations in the music he's focused on the hi-hat, but MOST IMPORTANTLY, he's always playing and representing the flow of the music. Beautiful. See you soon.