People keep finding significant vulnerabilities in its cryptography and the Matrix team tries to deflect or create strawmans for why it isnt actually a vuln. Soatok found a vulnerability in 2024 by just browsing the source code for tiny bit of time, and again just two weeks ago after looking for a couple hours. In both cases, Matrix then responded to his vuln report with hostility, saying it wasnt actually a vulnerability. He is sitting on another vulnerability.

Having a cleartext mode is a security downgrade and no secure messenger should support cleartext. It only barely got functional forward secrecy recently. VoIP in most Matrix clients (and servers) still use Jitsi backend which isn't E2EE, even with the release of the newer (secure) Element call protocol. Matrix leaks tons of metadata, such as usernames, room names, emoji reactions, generate URL embedded previews. Rooms arent encrypted by default. It is also a UX nightmare and often times you cant decrypt your messages.

Matrix is not secure. You'd be better off with XMPP and OMEMO which has its own problems and isn't secure either. Sill better than Matrix.

It's better than Signal since you don't have to disclose any personal info, but people have pointed out some issues with federation in it. Again, it's one of those things that may or may not matter based on your use case.