I can so much relate with you. I'm embarrassed to even count how many girl friends I have, they are highly over-numbering my male friends
I've never feeling at home in some huge group of men. Especially if someone is loud and aggressive there.
And certainly I'm somehow nervous if I have to meet and spend some time with some men that I don't know. I've been forcing my self a little bit there and I've started to play tennis with guys that I don't know, so far it was great.
But there were some failures too. I went on two occasions on ski weekends with couple of guys (all alfa males) with whom I don't have nothing in common and I survived drinking and eating in huge quantities and making terrible mess in house but I would not repeat it
At my work women are coming in my room for some chat all time and it is always some laugh and noise there and some male colleagues are jealous but I really don't have any intentions, I'm just more relaxed in company of women