Your post is very interesting and revealing to my family background as well. I am now raising two four year olds. I am separated from their mother, but we still do a lot together, spending time in her home and going to dinner twice a week.
What's interesting is I'm having memories of growing up with my parents having difficulty on issues like this. My father worked in entertainment. There was no room for lateness. My mother stayed home and took us to activities but had a tendency to be late. I'm beginning to realize how this created major trust issues between my parents, who eventually divorced.
Now that I am parenting, I feel the strong feelings when we don't get somewhere together on time easily. During those moments, I'm reminded of how my mother was probably trying to stay in control, while my father remained distant and self-involved. Since I started working on my sexual history and that of my family, it has emerged that my mother had one sexual abuse incident with a brother of hers growing up. I think this led to some deeper anxiety and need to control situations with men perhaps. (This anxiety was given over to me in more ways than one.)
My kid's mom and I both have damaging sexual histories and when the anxieties come up, it's hard to know where to turn. I know this might sound strange, but early on in our relationship, before we had kids, I came up with a distinction between
"leaving" and "going". "Leaving" is getting stuff together and "going" is actually physically leaving to go somewhere. This distinction has helped me feel less anxious and given me something to communicate to others around me. It's also a kind of fun way to discuss what's going on with everyone involved.
One more thing. When you have had an unacknowledged sexual history that is damaging in the family, you tend to blame yourself. I think this perception that you yourself are the problem is what exacerbates small tensions in those moments when you are relied on for something, such as when you have to be somewhere on time. It's like that becomes the moment when you want to scream, "I am not the only problem here!".
You might suggest to him that if that is what is going on, it is a good moment to try deep breathing. It helps you get back to the present.
Hope you can find more stable ground together.
Lose the drama; life is a poem.