I think I understand the claim: the energy cost of keeping heat outside of a box should be proportional to the surface area, not how much stuff is in the box.

This is true; but only once the contents of the box are already cold. I think what it neglects is that the stuff you are putting in is not already 0 K (or your fridge temp), it is usually much warmer. So the fridge must work rather hard to pump all the heat you add back out. (Incidentally, the fridge has an even harder job if the volume of the fridge container is bigger, since there are more places for the heat to hide/cluster.)

We see the opposite with old fashioned fridges (an insulated box that you put ice into, and removed the water when it melted) or modern coolers. By making an insulated box, you make the interior become the average temperature of the stuff inside. To make the stuff inside cold, you must add something much colder to bring down that average, like a pack of ice. It's pretty hard to get 0K stuff on earth, so many things to bump into, hence very hard to use refrigeration to get things down to 0K.

You might also be tempted by a selective insulator, that keeps hot stuff out but lets fast moving particles inside escape (so that the contents become cold). This is a classic thought experiment! Maxwell's demon. It turns out that any such intelligent barrier will itself need energy.