So what happens if you don't pay? If it's a literal tax, that means it's a crime to evade it right?

IMO getting the government to enforce a church tithe is an overreach. The church should deal with that themselves and the most they should be allowed to do under a secular legal system is ban you from the church.

Or if the government doesn't care whether you actually pay it or not, my entire premise for having a problem with this doesn't apply in which case I stand corrected.