heavily reliant on the mods

As any solution to this sort of problem should be, IMO.

If the mods suck, then go make another community. If enough of the mods are good, they can be a huge part of the solution. I'm envisioning this:

1. users report other users
2. mods ban users based on reports, if the reports have merit
3. admins block instances based on reports from mods, if the reports are consistent

Transparency keeps the mods honest, or at least allows users in the community to name and shame bad mods.

Some automated tools to help mods out are always welcome.