allalone- I am a survivor of sexual assault and I do care. I am going to agree with what farmer boy said about the ongoing affair. It might be easier if it had been just random acts, but add the element of emotion... well, its hard. My spouse also had an emotional affair. I was devastated.
From what I read, I see that your H is looking for someone to blame for his misconduct, his acting out. Don't accept that. It isn't your fault that others were brought into your marriage, into your family. Don't accept his anger, or his loneliness, or his shame, or his guilt, or his grief. He made it happen. It belongs to him and he needs it. He needs to heal from whatever is hurting him.
Accept your loneliness, and your anger, and your grief. The two are different. Focus on you. ...and btw... from where I stand as an observer of what you have shared... I also would want to know the story behind any actions my significant other played out with another in an affair. You had and do have the right to investigate. It just makes you stronger, more focused, and more aware. You do what is necessary to deal and to heal.
I am so sorry you have been given this to deal with and I felt your pain as you wrote what was taking place. Good luck to you. (((((allalone))))).
For now we see through a glass, darkly.