I personally find this to be another "it depends" case. I shoot 90% in AV and I'm pretty darn serious about photography. I might spend time in the same environment for a bit, but a simple move of a foot can completely change the lighting requirements, so simply being in the same "scene" doesn't always track. Following kids can be hard enough without messing with the shutter for every shot. If things are slower paced or I see that things are heavily backlit or whatever, I'll compensate, but for the most part I've had good results. When shooting autocross, I use TV, because again, there usually isn't time to mess with the aperture when panning.
When I shoot using my medium format cameras, I always use manual, because I'm in a much more even-paced mindset, and of course using an external meter dictates that anyway.