CdnDW. I'm just throwing this out for discussion.
Do you do anything together for fun? Just the two of you. (not with the kids) Any interests you share? If not then if you both had something that you were both interested in it migh make it easier for him to connect with you. It may not be at the level that you would like but it could be a step in the right direction.
Another thing that I wanted to say earlier was about the therapy. And this can just as easily apply with your relationship. I think you (we) need to set goals. What do we want it all to look like in the end. How could we get there. What incremental goals and steps can be set as targets to achieve.
It really sucks to have to structure a relationship in this way but progress will not happen just by itself. Of course both sides have to buy in. (Can't stress that enough)