I am nervous walking in the dark by myself. I simultaneously am relatively tall and will be perceived as male by anyone, so I also try to be wary of how I might make anyone else nervous.

The actual experience that most women have of smaller aggressions even in safer contexts probably also plays a role. I'm probably nervous walking alone at night because of a combination of being physically quite weak in spite of my looks, having experienced bullying throughout my childhood so "unpleasant random encounters" is a concept engraved into my brain, and because I'm affected by reading/hearing about any kind of assault happening to people walking alone.

Replace the bullying with "random men being assholes/threatening/worse" and most women have all 3 of those factors as well.