I'd have to say you're doing ok, and that you recognise that patience, in industrial amounts, will probably be needed.
However, I feel like he always keeps me at arm's length because he doesn't think I can understand and he expects me to judge him. This makes me afraid to say anything.
Our abuse can be a very personal thing for us, especially in the earlier stages of recovery when we can often think that we're all alone, no matter what our logic tells us.
Using a site like this however confirms that we aren't alone, and there are other men out there feeling like we do. I can remember the day I met another group of male survivors and it was so emotional and enpowering, I gained strength from their experiences.
Perhaps your partner still has feelings of isolation, however much he 'knows' the facts?
Feeling like this can often lead to the survivor thinking that nobody else will ever fully understand what they actually feel, perhaps non survivors wont ever fully comprehend what we feel, although I know for a fact many partners come damn close to it.
Maybe it's trust that he's struggling with?
And I don't mean that he doesn't trust you in particular, he quite possibly doesn't really trust anyone - especially himself - in regard to abuse and healing issues.
Sex abuse often has more to do with the abuse of power and trust than the actual sex, sex was the weapon of choice to the abuser.
Once trust has been shattered so badly it takes a hell of a lot of rebuilding.
What you wrote about how you are dealing with his abuse seems to indicate that you are making great efforts to support and understand him, but we can only work at our pace unfortunately, so patience really is a virtue.