I don’t know if you’re familiar with the term, but what you’re describing is similar to the experience that many agender folk describe.
Suffice to say, I experience gender very differently to you. I’ve “felt” my gender since before I hit puberty. Before I had the words to understand it, before I knew what femininity or masculinity even were, before I experienced my sexuality…
Nor did I. For me, it came around the same time I started to understand gender and sex. The more I understood it, the more I knew it was wrong.
For me, it was initially tied in the physical. I knew my body should have been different. I wished it was different. I dreamed, prayed, hoped and fantasized that it would be different. It was an awareness that I was “like them” with girls and “not like them” with boys. I knew it was wrong when I was grouped with boys.
That’s what it felt like. Not an understanding of others peoples experiences, but an understanding of how my own sense of self was at odds with both my body, and the assumptions that my body created in people.
For someone who doesn’t feel gender, then of course you aren’t going to understand the experience of folk who do, anymore than I can understand what it’s like to not feel it. All I can is that analogies about colour aren’t particularly apt here, because it doesn’t work like that. My gender doesn’t exist because of shared consensus (although it is shaped by that consensus). My gender doesn’t exist because I was able to understand other peoples experiences. My gender is just something I’ve always felt, and that I’ve tried to make sense of over the years. I describe it now in clear, defined terms, but when I was younger, it didn’t work like that. I knew my body was wrong, but the social stuff, the gender stuff? Finding the words for that would take decades. But even as I said, I was finding the words to describe an experience that was always there.