A reputation system is really important. Open federation may not be viable long term without one.

That's an interesting idea. For each instance give users the ability to mark as spam comments/posts, then make it so each instance keeps track of what the ratio of spam vs not-spam is coming from peer instances and block any that exceed a certain ratio. It could easily be made automatic with manual intervention for edge cases.

One issue I could see is that it could be used as a way of blacklisting smaller instances from larger instances by using bot accounts on the larger instances to mark the smaller instance's legitimate traffic as spam. It would likely be necessary to implement a limit on how young/active an account can be to mark comments/posts as spam, as well as rate-limiting for situations where a given smaller community that is a subset of the larger one decides to dogpile on a smaller instance in an attempt to block them from the entire community.