It crashes decently often but still worth it

They do, but since lemmy is open source they can store it before hashing it, just use basic security practices and use a password manager

See how fast it is on a server with a lot of people and messages, it gets slower and slower over time

It is kinda slow, the clients it has are bad and also this https://news.ycombinator.com/item?id=32780665#:~:text=based%20on...-,If%20you're%20privacy%20focused%2C%20do%20not%20use%20matrix%E2%80%A6,metadata%20it%20leaks%20is%20astonishing.&text=The%20metadata%20that%20Matrix%20%E2%80%9Cleaks,email%2C%20or%20OMEMO%20encrypted%20XMPP.

At this point, I'd wait for MTL