Also, he is much more comfortable with public displays of affection, for example holding hands, or sitting or standing close.
Is there something maybe I'm not admitting to? Something I'm not giving in to, or even letting go of?
I wouldn't think of it as something you are doing or not doing. Everybody has their own level of comfort about their sexuality and if and when they display it. Even straight people. But your comfort zone can grow of shrink.
If you are afraid of being perceived as gay, well, I guess that's a rational fear, to some degree. There are those who would like to harm gay people simply for their sexuality in a way that there are not those who would want to harm straight people for theirs.
I would suggest you take it in increments (where have I heard that before?). But really, you can expand your comfort zone if you want to. If you don't want to "push the envelope" then that's your choice too, but it's not like there is something you are supposed or not supposed to do beyond what you want to do, and what you feel is right for you. I feel like Ann friggin Landers here.