I actually really wish we could flag communities or posts as "excluded from the main feed."
There have been several posts I've made or that my bot has made (from community specific RSS feeds) that get down votes seemingly from people completely outside of the community.
For instance, I had a post about a heavy Standard Notes discount ... pretty relevant to Standard Notes users; down voted like crazy in the early days of lemmy.world.
Similarly, the other day Bungie made several post about Destiny around the same time. The bot faithfully posted all of them, but several got down voted, almost definitely because someone who didn't care about Destiny down voted the "spam" since several posts were about Destiny around the same time in the "main feed" (and they were probably sorting by "New").
My retort/unpopular opinion: There's no recommendation algorithm, if it's not a community you're subscribed to, and you don't care about it ... what the heck are you doing engaging with it? Move on to the next post or block the community.
Alternatively, we should be able to block interactions from people who haven't subscribed to the community without limiting federation or making it moderator only.
When you down vote a post of a community you're not a part of you're actively hurting its surfacing in the feeds of people in that community that use feed algorithms other than "New" or "Controversial".