It's because they're all shit at politics. Any leader with an ounce of charisma and courage would at least be able to get us back into the single market. They all argued vociferously against it when the vote was happening. Then after a 52-48 vote 9 years ago they still can't muster up the courage to change something.
It's like the self-imposed fiscal rules: they could get rid of them overnight, but they'd need to make the case for it, and have a plan for how to use that new freedom. But they'd rather throw up their hands and complain about all the obstacles in their way.