I don't really have any advice for dealing with the situation. But you said this:
Who Am I to deny him the right to see his grandchildren.
And i just think, what right does he have to see his grandchildren? Just because someone has your genes it doesn't give them any right to anything. You've seen him 3 times in 22 years. It's hardly like he's made an effort. I don't know the backstory of it all but is he even someone you want your children to meet? (I grew up without any grandparents - i believe most of them are still alive, but i've just never met them. It never bothered me.) Sounds like meeting him is just going to ruin your time away, so I'd go with the "go f*** yourself option". But like you said it's easier said than done. (and I don't think you sounded childish at all, I think you sounded perfectly reasonable)