I will sometimes use Shutter priority when there is a specific need to either deliberately freeze motion or deliberately blur motion. Since most of my photography doesn't involve deliberately freezing or blurring motion, I only occasionally use this mode.
I argue that if you are using manual, but you trust your camera's meter reading, then you're basically using either Aperture or Shutter priority... you're just slower at it. I say this because you'll set one element of exposure (such as Aperture) first, then adjust the complimentary setting (such as Shutter) until the in-camera meter indicates a correct exposure. But had you used a semi-auto mode then the camera would have selected the EXACT same exposure you just picked... but it would have done it faster.
If I'm shooting a situation with rapidly changing lighting conditions then I'm definitely in a semi-auto mode because you'd never keep up with the light changes in manual mode.
However I do use Manual mode quite a bit too...
One key factor that puts me in Manual mode is if I realize that for whatever reason the camera's light meter isn't going to be effective. For example (a subject near & dear to the OP's heart) if I shoot food photography then I'm using flash and lots of light modifiers. The camera's meter isn't going to know about all of these because they aren't "lit" when I'm metering the shot. So I'm manually dialing in the exposure that I know will work best when the shutter is actually open and the flashes are firing.
I actually own an incident meter with flash-metering and flash-contribution capabilities for more advanced metering and exposure than a built-in meter can handle. If I'm using that, then the camera is also on Manual.
If I'm using ND filters, the camera is on Manual (again, this is a situation where you probably cannot trust the camera's built-in meter.)
I never use full "auto" mode. If I hand my camera to someone else (or if my spouse wants to use the camera) then I generally switch it to Program mode ... knowing that it'll take "safe" exposures.