If I'm doing a shot of one or two people, a candid portrait, I usually ask. Since I shoot digital, I can click away until they get bored and go back to whatever they were doing. If it's a group shot, I don't ask. If someone indicates they don't want their picture taken, I don't. I was shooting two chess players one evening and one noticed me and waved me away. I smiled, waved back, and went on.
Actually, asking is too strong a word. Frequently, it involves holding the camera up a bit and smiling. If I get a smile back I take pictures. At a parade one day a baby was behind the bars on a window with her parents and I held my camera up, the parents smiled and beamed at the baby, and I took the picture. It isn't tough.
I'm not under the illusion that the one picture I missed would be great art.