It's so hard to know - you're in such a tough dilemma - precisely because your oldest brother's behavior contaminated your relationship with your middle brother.
To some degree, it is normal for brothers to see each other naked, including erections. To some degree, it is normal for brothers who are each old enough to masturbate to discuss that behavior, pornography, etc. To some degree, it is normal for boys of close age to "experiment" with one another and since your brother had already been doing that with one of his own friends, then it wouldn't be out of bounds for him to recommend to you the same with one of yours. The second darkest part of the story is when he tried to penetrate you - but even then, you say that when you complained, he stopped. A key characteristic of abuse is that it is unavoidable - that one participant drives it regardless of what the other wants. It might have been the dark side of experimentation, gone too far, but then stopped. Maybe.
But really, the darkest part of all of this is the fact that you straight-up identify him as your abuser. That emotion, that recognition, is ultimately all that counts. At least some of what the two of you had might not have been abusive - but enough damage had already been done that it all falls within the same boundaries. Like - if one brother was fond of abusive tickling (an under-recognized form of physical abuse - relentlessly tickling the victim until they can't breathe and lose control of bladder function) and the other just plain old tickled you like a good brother.... they'd probably both feel equally bad.
Your own judgment is ultimately what matters - if it felt like abuse, if it was unwanted at the time and/or had unwanted effects afterwards, that's that. However, you also describe feeling real affection and conflict towards him, and you miss the closeness you had for many years after all of these behaviors were over.
Honestly.... given the fact that you both see him as your "hero" and miss the non-sexual, adult-aged interactions you used to have, while at the same time from his perspective he might not have perceived any of this as anything beyond normal brotherly showing-off / experimenting / boasting, *and* the fact that you say he kind of teases you and puts you down often.... I honestly think you have to prepare yourself for being very vulnerable around him. Like, if you try to address the matter at all he might completely minimize it or be put in shock, having no idea what you are talking about.
The best case scenario: Middle brother is in a "serious" mindset. You say something honest but undetailed: "I've been trying really hard to work through the memories of what happened when we were all kids. Mostly with older brother but to a lesser extent with you too. Some of it really damaged me, and the rest, I'm not sure what to think of. I'd like it if we could talk about it." And he talks about it seriously, comprehending your needs.... maybe saying he feels the same from older brother. You can kind of uneasily start to rebuild the close, adult-aged friendship that you used to have.
Worst case scenario: You bring it up to your brother. He angrily denies any guilt, calls you a sissy for still caring about shit from decades ago, and makes it the basis of some life-comparison putdowns that you've described. Now... since you say he really seems to miss you, I think A is more likely than B.
But.... be ready.
And if you feel uncomfortable, BAIL OUT. And if that means excluding him from your life, then EXCLUDE HIM. You're being very gracious by even attempting friendly interactions with someone whom you define as an abuser, you've bent over backwards to make one excuse after another for him - "he was a confused kid who didn't understand what he was doing." I said above that some of what he was doing fits the description of "normal," but I wasn't there - it didn't happen to me - and I don't feel what you felt. If it feels wrong to be around him, then.... don't.
I hope that made some sense. It really is a tough situation and you've got to plan it out thoughtfully in advance - you've worked so hard and don't deserve any setbacks from a family member who might not understand now, for the sake of the *memory* of a family member with whom you had a different sort of relationship when you were at a different place in your life, under a different emotional state.
"Don't think it hasn't been a little slice of heaven just because it hasn't!" --Bugs Bunny