A lot of it is self-selecting IMO. We're big enough to have a good sized community, but we aren't so big as to become a big target. That will change as Lemmy gets bigger, but it least for now, it's small enough that the trolls probably don't get enough attention to bother sticking around.

If you want trash, you can find it on Lemmy, just check the list of most defederated instances. You don't see it because... well, they're defederated, which itself is one of the moderation tools available in the Fediverse.