Because the earth is finite and you having that land means it's denied to everyone else. You shouldn't have to pay rent to a landlord just because his great great grandfather was a Marquis, but you should compensate everyone else for excluding them from something they have as much right to as you do. The idea of the land value tax is not to punish you, but to prevent you from profiting from the work of others that increases the value of your land without you having to do anything. Henry George explained all of this better than I can.