Consent-based power structures generally rarely last for long. Power has a tendency to accumulate, as there will always be people who hoard it. And historically, that power is often given willingly, obtained through deception or through sheer strength. Smaller communities have no real ability to resist such larger hierarchies that will form when someone or some group decides to no longer play by anarchist rules.

I personally think anarchism correctly identifies the accumulation of power as an inherent threat to society. But I don't think power in general is the problem, but rather the accumulation in a single entity is. A measure of power is necessary to protect a community.

It's why the whole "separation of powers" idea has worked fairly well in the past, though perhaps the powers should be separated even further. Separated domains, each with clearly defined limits on their powers.