and banning whole servers results in some of your users becoming unable to reach all of theirs, incurring a penalty on both sides.

Which is good, so the incentive is to federate more than not.

To give an example of a technology that works the same way but is mature, there's email. You can send an email between hotmail and gmail seamlessly, but to set up an account on one of those two you need to provide something traceable to meatspace. That's because a server with open signups would instantly be flooded with spam accounts, and would get on the giant blacklists of domains that have existed for a long time.

A reminder that users can just join another instance, so if they're legitimate users and can prove it it's more of an inconvenience than anything.