Lets look at this a bit differently.
.
Yes, thank you.
But, to bring it closer to how you would act in real life.
What if:
1) if you had the opportunity to take an image where the subject was all right but close onlookers would almost certainly think it was wrong?
Respect the crowd. Trust me, Ive dealt with this before locally. Sometimes its just better to pick your fights.
2) if you had the opportunity to take an image where the subject would never know that a photo had been taken?
That really depends on the intent.
Unlike the NSA and other acronymal gov. agencies that get off on spying on little kids and down women's shirts in an "effort to fight terrorism", I generally try to respect people not wanting to be in an image.
The biggest example of this is building/architectural photography (to whit certain techniques are used to prevent such as long time exposures) or try to frame it to respect those folks. Even if they "don't know".
Sorry I have a tendency to work from an older set of Middle Age ethics that traditionally respects one's space.
LL