Man I have heard the same thing with different subjects. In school several women in our seminar course announced that we males had no idea what it was like, sexual harassment, and I had to hold my tongue cause I was not ready yet to talk about CSA, and certainly not in a hostile environment like that. It was like they were saying the same thing that dude said to you...that not every kid has it as good as we did. If they only knew that at home I got beaten and verbally trashed, and that made it easy for the predators to find me. I know it's not exactly what you are talking about, but it's all about the assumptions people make about others. We survivors know a lot about how hard life is for some kids, cause we lived it in the shadows. They are the ones who don't get it. Sorry you had to endure that.
Lao Tzu said: "When I let go of what I am, I become what I might be."