Economic "growth" seems to be doing less and less for most people. Its financial benefits mainly accrue at the very top of society; most people just get squeezed. Less housing, depressed wages, ever more crowded and less available services, the list of consequences of constant growth goes on.
The issue has a toxic element of anti-immigrant racism, but many are turning against the idea because they think the net negatives outweigh the positives. Switzerland's upcoming referendum is this in a microcosm. The right-wing anti-immigrant Swiss People's Party got 100,000 signatures to trigger their referendum, but support for the measure is also coming from outside their base. Polling has the result at near 50:50. If it passes, it will force a Western government to do something no one has ever had to do before - run a country where you cannot have endless economic growth.
Switzerland to vote on plan to cap population at 10mn: Country has 9.1mn permanent residents and experts fear the move will limit companies’ access to foreign talent
Almost no modern country wants MORE people, it is the median age that is problematic. And increasing overall population is the most primitive way to increase a working age population.