Let me start by saying that gender and sexuality are not the same thing. They are somewhat related, they effect each other, but they are not the same. Trans people and non-binaries and everyone else can have any sexuality. But, people have an easier time understanding sexuality than gender. Probably just the way the world is going- or maybe it's because they can understand "I find X hot" easier than "I know myself to be X".
I continually see binary-gendered people try to define everyone else as between the gender binary. This even happens right after a non-binary expresses that this is not true, usually accompanied by a condescending "Silly non-binary, you don't know what you're talking about" or a lovely amount of derailing. This is trying to base our gender on your identities, even if we insist that they are not based on them. This is annoying at best. So let's see if using sexuality will make this finally get through.
I'm going to say that sexuality is a 1-dimensional line with straight on one side and gay on the other. Someone insists that they aren't, so I just add points between those two and call it "bisexual" and say that most people are more gay or more straight with only a few really bisexuals (and some song and dance about how "we're all a little bisexual so I know what you're going through"), and call it a day.
But what about asexuals, someone else argues. Well, that's the same as bisexual, right? It doesn't insult anyone's sexuality to say that not being attracted to anyone and being attracted to both sexes are the same thing, right? So you still fit on our line. Yes, you in the back, you say that it's more complicated than that for you? Maybe your sexual and romantic attraction don't match, maybe you're attracted exclusively to masculine women and feminine men, maybe you're multi-gendered and your sexuality changes with your gender, maybe you're sapiosexual or pansexual or anything else. Maybe your attraction is extremely fluid or complicated the point where you can't fit it in any one label and want to just say "I like who I like how I like them". It's even possible to be attracted to intersexed bodies and/or genderqueer minds. Where do those fit on the line? I can still cram them into the middle of that line if I really want to, but doing so completely dismisses the complexity and reality of their attraction. It says "My understanding of sexuality is more important than who you know you're attracted to. I am allowed to deny your sexuality to make it fit into my view of the world." To be true to their sexuality, I need to upgrade my 1D model to something a bit more complex.
Now apply that to gender. Insisting that gender is a strict binary where everyone else is either in between or on the fence, even though many people insist this isn't true, is the same as insisting that everyone is really gay or straight, and maybe there are a few bisexuals thrown in for flavor. It's denying the reality and complexity of quite a few people's gender just to try and fit everyone into a binary system.
A ton of these theories seem to be at least partially based on the idea that the rest of us are so uncommon, as if that matters. We're dealing with minorities here. Intersexed- 1% (.1% obvious at birth) Homosexual ~ 3-4% Bisexual, Polyamorous, Otherkin- I still haven't found any statistics (I can't wait for it to be something like 50% of the populace is bisexual and heterosexual is actually a minority... A person can dream) Transgendered- .5% (at best) Asexual- 1% (or as many as homosexual, depending) These are hardly powerhouses. So what if there aren't "that many" non-binaries? There aren't "that many" of any of us!
Also: Prove it. We don't have any information on non-binaries either way. I haven't seen a serious study done on how common it is. Most studies on trans people are based on transsexuals who have or are in the process of transitioning. We don't even know how many non-op binary-gendered trans people there are out there! And it seems more common for non-binaries to not transition (or transition completely, which inflates those statistics) than binary-gendered trans people. More non-binaries are unable to get the body they want with current medical technology (or aren't allowed to due to our current system), and find ways to not need to. So the system of getting statistics on trans people (how many have had SRS?) is even less effective when trying to determine how many non-binaries there are. In other words: We Don't Know. You can't say "omg, non-binaries are so rare so STFU" without proof.
Either way: Stop trying to fit us into your binarist gender structure. Tell me something, binary trans people, how did it feel when cissexist people tried to fit you into their gender structure? You know that "Transsexual males all really men who fetishize being women or men who want to trick straight men into sleeping with them"; "Transsexual females are all really women who have internalized male privilege or lesbians who want to be accepted by society" that people still believe thanks to "professionals" spouting it. Does that feel awesome? Do you feel it to be an accurate representation of your identity? Do you feel honored to have cisprivileged people try to cram your identity into their world view- that they took the time and effort because trans people are so bloody difficult to understand, so freakish and abnormal that it's such a trial just for a cis person to manage that? Or does it feel like they're denying who you know you are just because you don't fit their system and they're too [whatever] to fix it?
Yeah. I really hope you get the picture It might reflect some people- but for the most part it doesn't reflect trans women or men particularly well. Neither does insisting that all non-binaries can be defined by a binary structure. They can't.
Stop. Doing. It.