Probably for the best if downvotes remain less easy to access, at the very least. There's a myth that people who are suicidal will "find a way even if you take away some of the easier methods", which is explicitly false. If you take away the easy option, you are directly reducing the harm that easy option might have caused. https://gizmodo.com/why-have-people-stopped-committing-suicide-with-gas-5959303
If the admins take away the quick and easy option for seeing who downvoted your passionate comment, the mods are directly reducing the number of people who go on rants about downvotes and targeted vitriol.
It has nothing to do with privacy; this is a public forum that by it's very nature, requires that all activity be easily available to all the sites you federate with. There is not privacy in that.
This is about the type of community that forms around the software. Do we want to encourage, and make easily available, the list of people who disagree with you? Or do we want to to put minor barriers around that to help keep the number of people who do that low?
ftfy. That comma is an important part.