Signal is a centralized app, run by a company. If they are offered enough money or legal threat they will sell out or close.
I am sure people will make an argument that its FOSS and people will just fork it if it goes bad, but a new fork will have 0 users and Signal will still have all of your old contacts. Why not make a switch now? Before it is even more popular and you have more reasons to stay? Why fork it if there are already decentralized apps that use same encryption, like XMPP apps?
Sure you can find flaws in every app, including XMPP implementations, but if we will have to write code for a new Signal fork, why not just fix whatever is that bugs you in XMPP clients?
If you want to use Matrix, that is fine as well, we can always bridge the two open protocols. But you cant bridge Signal if their company doesn't allow it.
That is THE ISSUE with email. I can secure my server all i want but when you use Gmail and they hand over the keys to whomever they want i get screwed.
As for XMPP security, you have to do e2e a layer above. Use XMPP or any other protocol and encrypt the messages you send. The catch is that you need to always encrypt everything so that your Happy Birthday to your Grandma is just as unintelligible as your secret bank pin yoh send me to get you bail money. At that point the meta data is useless as we don't really know who gets important messages and who doesn't.