Yes, and this would be fairly easy to make them at least pseudonymous without even needing to modify activitypub itself.
That said, I still don't support anything which lowers the friction of vote stalking like exposing votes in even more places. Technically people can look up my address from my license plate number if they really care to, but that doesn't mean I want to list it in bold letters on my windshield.