I feel like that is more or less to be expected. A ton of people found Lemmy during the reddit protests. Now that the protests are gone and Lemmy has had its growing pains some users are leaving, going back to reddit or other places. If we keep using it and making content users will grow organically.

Lemmy is having an identity crisis of sorts. It was built to be decentralized yet we (users) seem to want to centralize everything and we all go to a few of the largest instances.