As a trans woman, I'm going into the bathroom to poop or pee and wash my hands. That's it.
I'm not saying it has never happened, but I've not heard of a single incident of someone getting assaulted in a ladies room by a "man in a dress," but plenty of trans women are assaulted inside and out of restrooms by cis men due to hate and fear.
Until a trans woman (or man for that matter) personally affects you or someone you know, please stop buying into the fear bait.