There are plenty of people (EDIT: some of whom are in this very thread) who like to site that there are more vacant houses in the country than there are homeless people, as if to imply we already have all the housing we need.

I feel like you're taking very much the wrong implication from that statement then. Again, I can't seem to see any meaningful number of these people actively advocating against building more housing. That doesn't seem to be a position that anyone seriously takes. What is being said is that we clearly have capacity that is not being properly utilized. And we're both clearly in agreement as to why that's the case.

I think it's important to remember that when people are pushing back on a position generally held in bad faith (e.g. "The only solution to our housing crisis is to build more housing", a framing that is basically designed to protect the wealthy and ultimately maintain the status quo), they're going to frame their own arguments against the position they're pushing back on. They're not laying out an election platform. They're not going to take the time to establish the specific nuances of their position for every possible context and audience. If you were to ask that same person "Do you think we should never build any more housing, ever" the percentage that are going to say "Yes" is going to be a rounding error. You have to read people's arguments in the context in which they are given.