Yeah, I'm aware of the Haber-Bosch process.

I'd honestly have to do the math, but I suspect we'd be able to get rid of synthetic fertilizers if we actually wanted to. Afterall, what do you think happens to the nitrogen after we eat it? We pee and poop it out, for the most part. Yes, there are losses to the air when you till the soil, but a proper farm that focuses on soil health has ways to deal with that problem.

Right now we use the system we have because it's cheap and easy to do so on an individual level. Growers want to simplify their workflow; they don't want to actually manage the health of the land they work. It's too much effort.

Plus, there's a bunch of government policy that encourages bad farming practices and discourages good ones. Corn subsidies, banning the use of treated sewage for fertilizer, blatant blind-eye enforcement of labor laws, price-dropping policy instead of price-stabilizing policy, etc.

It's not that we would starve, not in a properly structured system, anyway. It's that food would become more expensive and some of us would transition to careers in agriculture. The pay would become seductive when the farms become desperate for labor. A farm that actually takes care of the land and the animals is absolutely more labor-intensive, and that's why very few modern farms do it.

Edit: I should also say that the plants and animals we have today are not the same as the ones we had when the Haber process was invented. We wouldn't be going back to the yields of the early 1900s. Even if we did everything exactly the same as they did back then, we'd still get better returns and have a more robust food delivery system. Hell, they didn't even have refrigeration back then.