I could be wrong, but I think that up until the recent manufactured outrage, gendered bathrooms were merely a social convention and not actually any sort of enforced rule anyway. Like, the sign on the door doesn't have the force of law to make it trespassing or something to enter if you don't meet the condition; the only consequence to barging into the wrong one would be that other occupants might yell at you to get out. Alternatively, they might have even not objected, if they could reasonably see that it was a genuine toilet emergency and you were acting out of desperation, not creepiness.
It was never a big deal until the fascists made it one in order to blood libel trans people as an excuse to push authoritarianism.