.... I feel the man was buying my silence. By doing such nice things for my brother, certainly if I ever tried to tell, my brother wouldn't believe me because the man was so nice to him.
You are likely right about this. Its classic pedophile behavior to cover their tracks this way. Good for your brother for have put you first and believed you. And good for you for taking this step. Its scary as hell, but the relief you've gotten is worth it.
"But now old friends are acting strange,
they shake their heads, they say I've changed.
Something's lost but something's gained in living every day
....it's life's illusions I recall, I really don't know life at all. "Joni Mitchell