I saw the same thing happen to someone else once. I had a close friend 20+ years ago who was a professional photographer. We were in a store parking lot in my old Dodge cargo van and he was sitting in the back. A pickup driven by some 20-something hottie parked right next to us. She was wearing a tight, skimpy top. Jim took a light reading out of the windshield, estimated the focal depth, and got his camera all ready. Then he opened the rear side door of the van, clicked the shot, and slammed the door shut again.
And the little cutie driving the pickup just went nuts. Oh was she hopping mad. She jumped-out and began banging on the side of my van, screaming in some sort of frightened rage. She wanted the film out of the camera. Jim wouldn't give it to her and she kept screaming at him. Fortunately my other friend came out of the store and we drove away, leaving one fiery-tempered gal standing there. We sure thought that it was funny then.
I guess that I enjoy having my picture taken when I know what is going on, and even enjoy composing shots and taking photos myself. I prefer to take photos of old industrial buildings and machinery, and landscapes when the sun is low in the sky. Just yesterday I photographed a large abandoned coal mine and an eerie out-of-business ski resort that has sat abandoned for the last 7 years. They had ski-in/ski-out condos for $69K if you didn't mind if the place was out-of-business. But my one bad experience with photographing unwilling models has perhaps soured me on anonymous photography.
Sorry to hear that you didn't enjoy having your picture taken without permission. I am sure that the guy taking the photos never even gave it much thought. I'm almost certain that he wasn't expecting to make anyone uncomfortable.
Hope that you feel better soon,
"We stay here, we die here. We've got to keep moving". Trucker Mark