Here's the thing, as far as I know. It is perfectly legal for you to take and post any pictures of any person so long as they are in a public place. They have no expectation of privacy and legally do not have any grounds to ask you to take them down. If you were
selling the pictures, then, absolutely he could sue you (and win), but aside from this, you are legally on solid ground.
The definition of "public place" is a bit more broad than people think. In a truly public place, owned by the government, like a park or the sidewalk, there is no question. In a semi-public place, like a mall or a business, then the business sets the rules, but for people photography, it is still considered public so long as the owners of the establishment allow photography. Private locations include things like people's homes (not visible from public property, as in front yard doesn't count), or private venues, bathrooms, or anywhere a person has a reasonable expectation of privacy.
That being said, it might make this situation a lot calmer if you just took them down and apologized. I, on the other hand, don't shy away from confrontation
Note: I am not a lawyer, nor have I ever (really) been in this situation. This is based on the understanding of photograhy laws I have gleaned from a good bit of reading so I'd have ammunition against the gung-ho security guards of the world.
EDIT: Oh, shizzz, I missed that the phtos were his. Yeah, take those down ASAP.