The whole process really made me question my definition of CSA.
Can I call it rape when he 'only' manipulated me into giving him oral sex at 5 or is it 'just' sexual abuse?
In many countries even the legal system considers it rape. My T told me, not long ago, that my reaction to being forced to suck my foster-father's dick would these days be called "rape trauma syndrome". Yes, oral rape is rape.
Did my father sexually abuse us even though he didn't touch us?
It is very possible, yes. He exposed you to an environment that was not appropriate for your age. He failed to protect you. In my mind, that is also abuse.
Could my brothers have had a sexual relationship on an equal level even though they where 6 years apart?
No. There is a reason why just about every country in the world have age-of-consent laws. A child isn't mentally and emotionally capable of "informed consent", and consent that isn't informed is not consent. Besides, is anything EVER equal between children who are 6 years apart?
How far does this pattern of CSA go?
It stops with you. That is all that matters.
Am I just making a big deal about nothing?
No. If it wasn't a big deal, you'd have been able to shrug it off. The fact that you can't, and the fact that you needed to come here and discuss it with other male survivors, means it IS a big deal. You have the right to feel they way you do!