Maybe transparency would work. If he brings the mother or brother up first, let him know you do not want him to and if it upsets you enough when he does so let him know you're mood will [fill in the blank] until he learns to STOP IT/CUT IT OUT or else phone call over, visit over, dinner together etc over and let him also know that eventually you'll drift apart.
That's probably passive agressive behaviour, I don't know. BUT, it could be seen as standing your ground on something VERY important to you (and sounds worthwhile to me). If he uses the term "hunky dory" use it back at him and ask how it feels?
I'm trying to say to you maybe it's time to teach the parent, in this case you're the teacher. I think if he does care he'll come around. Sounds like he will.
Good work! (I rarely say good luck, lol! I think luck is a lame four letter word).