I think this is entirely a judgment call. It looks to me like you have to weigh things out here: what do you want and need from your brother by way of a relationship, as opposed to whether you think you can get this without a lot of new pain and trauma. But I wouldn't think in terms of the past (what could have been); we can't do anything about that. The question is what can be done now...for the future.
Nobody living can ever stop me
As I go walking my freedom highway.
Nobody living can make me turn back:
This land was made for you and me. (Woody Guthrie)