I've addressed this in another comment. At first, it's quite likely that we'd see an increase in behavior. But the way to correct this would be by reporting "serial downvoters" and brigaders to moderators, which could then be empowered to enforce "don't downvote just because you disagree" guidelines.

Hackernews, for all its faults, does this very well. Their moderation team is quite small, yet it rarely falls into screaming matches between users. Their guidelines are clear and let people understand what is/is not acceptable. Mods are rarely seen threatening to ban someone, but often calling out bad behavior and simply asking people to stop doing whatever they are doing before it escalates further.