I usually try to carry "my camera" everywhere I go.
That's was the benefit of switching to the Fuji X system, I pretty much bring my XT2 everywhere except like shopping or eating at a restaurant (I just use my phone in those situations). Sometimes I bring my camera along and I don't take any pictures. That's just the way it is, but hey...at least I brought it!
Some say the Fuji 18-55 is too big but when you had a full frame DSLR and the full frame glass along with it, the XT2 + 18-55 is a pretty darn compact and lightweight system. If I put the 35 f/2 on, it's super lightweight and even more compact. I've considered getting one of those pancake lenses but everyone says they aren't very good. Then again plenty of people managed to get incredible photos with thoses pancake lenses. It doesn't always have to be about the best IQ but having the right camera with you at the right time can make the shot and if a pancake lens gets you there then it's worth it regardless of the reviews.