I would like if you just purge those communities or let users block them. The same way we are technically federated with lemmygrad but if you go to the "All" tab you won't see many posts from them.

As you've said a community with 10 members would do much less damage than a community with 7k members, we are an instance with less than 20 active users. We aren't doing too much damage and the "damage" we are doing could be mitigated pretty easy without defederation.