We know that for whatever reason there's less neaderthal on the X chromosome, but that doesn't have anything to do with sterility of offspring at a high frequency.

It most likely was just that any mutation that did make it over, was outcompeted. Which comes back to the prevailing theory that "modern" humans main advantage was reproducing like bunnies, and that advantage was carried on our X chromosome.

That would mean the neanderthal DNA that was passed down and still around, came over from the crossing of male neanderthals with female humans to male sons. Which (going off memory) we do have evidence to support.

Those "hybrids" would have children with "human" X's even if they were daughters, but be introducing neanderthal DNA back into a "double human X" mother, ensuring her daughter still had the reproductive advantages over a neanderthal mother still, but retaining neanderthal genes adaptive to the northern climate.

Shake and bake a couple generations, you get white people.

But at no point does it mean any "hybrid" was sterile, just that a thousand years later they didn't have direct living descendants, an incredibly common thing especially back them.

Edit:

This already happened with cro-magnons:

https://en.wikipedia.org/wiki/Cro-Magnon

Because we only had morphology to go off of, they were labelled a separate species.

However that was largely due to their harder lifestyle than genetics and with DNA testing they're now recognized as a "sub species" which is why later waves of "modern" humans reabsorbed them into the gene pool so quickly, likely along with some neanderthal DNA as icing.

All I'm saying. Is that eventually the other "archaics" will get the same recognition as "siblings" on the family tree and not "cousins".

It's just human variation is far wider than popular opinion or present examples.