I take photos in Oaxaca, Mexico. One winter visitor uses a long lens and hides in bushes and I find that creepy and rather offensive. If I'm taking a photo of one or two people I ask permission, often by holding the camera up, smiling, and getting a nod. I find with digital if I take some photos people go back to doing whatever they were doing an forget about me. Sometimes I live the photos of eyes on the camera, especially with kids.
I'm not in the U.S. People here are usually polite and permit photos. If they don't want photos taken I smile and thank them and move on. Later, on another day, the same person might not have a problem. We get a lot of seriously rude tourists who foul the nest.
If it's a group or just a street scene I just shoot.