A very small percentage of "victims", go on to abuse others. However, a very large percentage of "abusers" were victimized themselves. Please understand the difference.
I can't tell you who to tell. That is a personal choice. However, if there are children who are in danger now, I would hope you give it some serious consideration. I also hope you understand that the effects of sex abuse are very powerful. The shame and guilt can last a lifetime. I would encourage anyone who has experienced abuse, to talk to a therapist. There are many issues that need addressing. If you share this information with others in your family, and other victims are identified, it could begin the healing process. The sooner, the better.
As far as your brothers fear of retaliation goes, it may seem irrational to you (and it probably is, due to his advanced age), but it probably is very "real" to your brother. When someone is abused, it is normal for the victim to give the abuser almost "supernatural" power over the victim. It sounds very strange to think of a grown man is afraid of an 80+ year old man, but I can easily see how he could feel that way. The abuse does strange things to a mans mind.
I wish you and your family all the best. If you want to discuss this further, you are welcome to send me an e-mail at firstname.lastname@example.org