Well, mine is not exactly your case, but has something to do.
A couple of weeks ago I decided to go out and photograph some of the houses and streets around. I must say that I live in a little town -I believe this has something to do with what it happened. And what happened was that not only once but twice, in just a few minutes lapse, these persons came to me asking me about why I was taking those pictures, etc One even suspected that I was from the inssurance company. Me! Ha ha! After the second time, I was felt so frustrated and uncomfortable that I decided to put the camera in the bag and forget about the "session".
I found this very weird. It was like a feeling of constant suspicion in the air. I am currently living in the US, but I'm not american; I mean, maybe this kind of reaction is normal here, but for me was a real surprise and really strange.
Well, after all, I guess that I was lucky. At least I was only asked about what I was doing, not asked to leave! -for me, this is even much more strange than my case!!