Signal is a centralized app, run by a company. If they are offered enough money or legal threat they will sell out or close.
I am sure people will make an argument that its FOSS and people will just fork it if it goes bad, but a new fork will have 0 users and Signal will still have all of your old contacts. Why not make a switch now? Before it is even more popular and you have more reasons to stay? Why fork it if there are already decentralized apps that use same encryption, like XMPP apps?
Sure you can find flaws in every app, including XMPP implementations, but if we will have to write code for a new Signal fork, why not just fix whatever is that bugs you in XMPP clients?
If you want to use Matrix, that is fine as well, we can always bridge the two open protocols. But you cant bridge Signal if their company doesn't allow it.
I don't really get the complaint here. There is no perfect solution with regards to privacy.
Either you have some centralization to help facilitate its ease-of-use for customers, or it becomes more difficult to setup and use. Much like the rest of life, there are trade-offs.