The Canon S series has been my compact of choice for the past few years. First the s90, then s95, and now s100. It fits in the front pocket of my blue jeans so I can carry it anywhere, and I shoot in RAW so the low light, indoor, evening photos I take when I'm carrying it at various events and parties clean up great in Lightroom.
The challenges I have with mirrorless and/or larger compacts are that 1) I'll still end up spending money on lenses (specifically with a mirrorless body of course), and 2) they don't fit in my pocket so I'd still have some sort of bag that I would need to carry around. As I look through the thousands of pictures I have from my S series compacts, the majority of those pictures were ONLY obtained because the S series was in my pocket in situations where I would be have been highly unlikely to have been carrying a camera bag. Of course you could argue that you always have a camera phone with you anyway, but IQ isn't remotely close to what you can achieve after some mild post-processing of a RAW image from an S series.
Here's an example of what an S100 can do in low light. A camera phone picture wouldn't look nearly this good, and had I had a larger P&S like a G series or even a mirrorless, I would have been much less likely to have been carrying it at this concert.
So....if I need a camera to carry in a pocket, I grab the S100. If I want or need more capabilities and want to carry a bag, I grab the 5D3. I personally have a hard time justifying anything in between, as I'm not sure what scenarios I'd actually use it for.