Stanford University Science and Genetics - "When the SRY gene isn't working, the resulting condition is called Swyer syndrome, or XY gonadal dysgenesis. Individuals with this condition are genetically male (XY), but look female."
Like you say, these are extremely rare cases enough to even baffle genetic scientists how people with different genetic makeup can develop.
There will be developmental disorders waiting to happen that mankind has never seen before. There's billions of potential variations. Most will have no 'symptoms' so are completely missed.
For example, one I was reading up on last year called Kleefstra Syndrome, caused by the disabling of a gene just like SRY/XY one during development. Super rare, but happens.
I'm sure there will be many more developmental sexual disorders found. But their existence isn't evidence of a sexual spectrum. They are an anomaly of the status quo. What happens when things go wrong.