I wish I had some magic answer for you. This is such a awkward position. I don't feel qualified to offer advice here. But I really feel for you both.
I think he is blaming you for talking to the other husband because he is stuck in a victim mind set. It is so easy to focus on how others have hurt us rather than how we have hurt others. It took strength for you to not react and yes that would not have gone well.
While I can understand his confusion ... being torn between what he thinks he wants and what he actually wants. For your relationship to continue he really has to decide to stop this affair. I think there is a difference between acting out with random people purely on a sexual level and having a 'relationship' with another person. To me that is a greater level of betrayal.
You have drawn a line in the sand ... the rest is up to him.
More than meets the eye!