Several issues have been discussed, so I will respond in my opinion.
1. Ethically, taking pictures of anyone in the public view that isn't harassment in OK to me. If a little old lady is sitting in her underwear outside, and I think it's a particularly interesting shot, I'd take the picture.
2. LEGALLY taking a picture and PUBLISHING the picture, in the USA, are two different animals. I can take ANY picture, even on PRIVATE property, since the act of trespassing and the act of taking a picture are two different events. No one ever ever ever ever EVER has the right to take your camera/film etc... Publishing that picture is a different story, and matters for different reasons, but the rule of thumb is you cannot use the picture you took for COMMERCIAL reasons. That doesn't mean you can't sell the picture, or even prints of it, as ART, but you cannot use the picture of a girl in front of a fountain as part of a perfume ad!
3. How to handle this situation? Well personally I only have two experiences where I had some trouble.
The first, I was in the mall, taking some pictures of random people like this one:
I took these pictures in the mall... without many people noticing. Is it wrong for me to take these pictures, especially of a little girl? I personally don't think so, although I got some funny looks from some of the participants. At one point, one of the guys I just took a picture of walked up to me, and I was preparing to explain to him my rights as a photographer... and he asked for the time. I could tell he wanted to ask to see the picture, and he was smiling, but he didn't. I should've offered...
Second time when I was taking pictures of the Tampa Skyline:
For this series of shots a friend of mine and I went behind Tampa General Hospital, technically private property. I even was behind a construction fence on the edge of the seawall. Eventually a police boat drove near us, put a light on us for a second but never said a word. Soon though, they were docking right behind us, at which point I packed up and left.... I am positive they were going to ask us what we were doing, but they seemed content with letting me leave.