Isn't work in prisons usually a privilege? Is this something that is actually used? What happens when someone just doesn't do what is asked of them?

Oh go away, they did not vote for "slavery". They voted to allow making convicted felons work as part of their sentence. It's slavery in the same sense that working to buy food and pay rent is "slavery". Metaphorically yes, but calling it slavery devalues the experiences of all the people who were kidnapped from their homes, brought here in chains on ships, and sold in a market.

People likely voted for not repealing the provision allowing involuntary servitude as a punishment for crime and not for keeping slavery.

Whoever thought combining those two things in one vote was a good idea is an idiot