I think (hope) "punching bag" was used in the emotional sense and not physical. But that can hurt as much just the same. My CSA scars are not due to any physical harm.
"But regarding boundaries, I am having trouble setting them."
I would think that would be the case. You for setting them and him for accepting them. I'm sure the situation you have was developed into a normal situation (normal as in established) and trying to change things will seem as a threat or confrontation. While I sympothise with your situation I unfortunately don't have anything constructive to say.