Well, you are in a sense being intrusive. If I'm taking a photo of one or two people, I generally get approval. It can be as simple as holding my camera up and getting a smile or schmoozing a bit. With digital, I can snap away until the person gets bored posing and goes back to whatever they were doing before. If it's a basic street scene, I snap away.
I have seen people being incredibly intrusive, such as snapping a blind beggar from less than 24" and ignoring the blind beggars questions about what was going on. I had the feeling that had I said anything I would have gotten a lecture on rights so I waited until he joined friends at an outdoor cafe for beer and I went over and started taking photos of him drinking beer from less than 24". Guess what? He got seriously irritated.
I'm not very social but I am pretty good at chatting with strangers. For a little girl dressed as an angel for a school pageant, "Oh, your daughter really is an angel. May I take some photos?" For an elderly woman begging, "The history of Mexico is in your face. May I take a portrait?"
For people who are frequent fixture on the street, I'll have a print made and give it to them later. The word has spread and people don't mind my photos.
I did take a photo of two street vendors, husband and wife, sitting back to back and catching a nap. I gave them copies of the prints. The husband laughed and was delighted and the wife wasn't.
When my son was leaving home I told him that 95% of the people in the world are nice. After time in Japan, Korea, Indonesia, and Turkey, he agreed.