First off, I just wanted to acknowledge your courage for posting here and asking what must be a tough question to ask. Looking back and opening up to the doubt and confusion is never easy for any of us. Regardless of whether or not you think you've done something courageous or exemplary, know that you indeed have done something positive and healthy by sharing your doubts.
With regards to your question, the line that defines what abuse IS and what it ISN'T is, for good reasons, not a clear one. The specific actions are somewhat less important than the subjective experience of boundary violation. Contradictory as it may sound, sexual abuse has nothing to do with sex itself. The violation is one of power and control over one's own body, one's boundaries. It was not appropriate for your brother to bring sex into your life in the way he did. The exploration of our our sexuality and the sharing of our bodies should always be a voluntary act. Anyone who has had that chance taken away from them, no matter what the circumstances, has been violated in some way.
While you may not have felt the trauma that many others here have experienced, something very precious was taken away from you. That may be a place to start you investigations about your own sexuality from.
Best of luck in your journey. Please know that this community is always here to stand by you and support you in whatever way you need.
"I am not a mechanism, an assembly of various sections.
And it is not because the mechanism is working wrongly, that I am ill.
I am ill because of wounds to the soul, to the deep emotional self...."
Healing D.H. Lawrence