I got confronted so many times from paranoid strangers. I had drivers blown the horn at me. Some stick a middle finger at me. When I was photographing the events, one guest was stern and told me, "Don't take my pictures!" It was a monthly event, and he kept telling the same thing again every month. I got to the point that I got very annoying and walked away from him. Then, he followed me around. Another time at different occasions, a lady took pictures at me with her iPhone, because she said I take pictures of her.
Most people don't have a problem, but very few do. When you do get confronted, there is not much you can do, and you will not win. Either you talk to them out of it and give a reason why, or walk away and take pictures at different location or different time. When I get permission to photograph people, I make them feel comfortable and feel welcome by smiling and interacting with them. Most of them are very friendly and curios. They will talk to you and ask questions about your camera and ask you "What is the reason for shooting?" Just because you have Asperger, or you are antisocial, it doesn't really matter. If you can not handle those kinds of people, you will lose your privilege as a photographer in public places.
I have photograph large crowds with heavy polices and sheriffs present. However, I have never gotten confronted or arrested by them.