As Lost Cowboy says, not so silly! It's great to see you and your boyfriend have a caring and loving relationship, but I imagine that only you and he can answer the question you pose at the end of your post.
It isn't unusual for an abused boy to feel sad or mourn when the abuse stops. He may have appreciated the attention for all sorts of reasons, or the abuser may have succeeded in isolating him from other sources of love and validation. Many boys cope with what is happening by convincing themselves that what is happening is okay, and the abuser of course encourages this.
My guess (and that's all it is) is that as a boy your boyfriend came to connect his sexuality with feelings of confusion, danger, loss and instablity. Perhaps he ends relationships because he feels that if they become sexual they are doomed - I really don't know.
I guess the real point here is how important a sexual dimension to your relationship is to you. It's a pretty big thing to just give up!
You might want to suggest that the two of you see someone about this, but tread carefully. This sounds like it is a huge issue for him, and myself, well, I doubt that he is handling the problem as well as he thinks. A lot of the guys here would tell you that they thought they were "handling it", only to have everything eventually fall apart in a really big way.
Nobody living can ever stop me
As I go walking my freedom highway.
Nobody living can make me turn back:
This land was made for you and me. (Woody Guthrie)