Animals eat food, too. If you eat meat, you're actually creating more demand for crops than you would if you ate the crops directly. Furthermore, migrant workers also work on animal farms, in slaughterhouses, etc. I hear it's not always great.
I guess what I'm saying is, I'm fairly sure that going vegan helps both animals and the children of migrant workers.
One caveat is that I'm assuming you'd eat the same classes of crops that an animal would, namely things like corn and grains. But honestly that sounds about right for most people, vegans included. Many vegans eat a lot of processed shit too lol. (me included)
Edit: I should add that the most commonly suggested vegan diet that I've heard from other vegans is to have rice and lentils as your staple foods. I'm fairly sure those aren't typically harvested by hand, but I could be wrong.
Personally, I find a lot of Peter Singer's arguments to be pretty questionable. As for some of the ones you've mentioned:
For one, killing humans, no matter how humanely the means, is seen by most to be an act of cruelty. I do not want to be killed in my sleep, so why is it okay to assume that animals would be okay with it? While he is a utilitarian and doesn't believe in rights, killing a sentient being seems to me to have much greater negative utility than the positive utility of the enjoyment of eating a chicken.
Also, farming animals for slaughter will always be destructive towards habitats and native species. Even if broiler chickens were kept alive for their natural lifespan of 3-7 years instead of 8 weeks to alleviate any kind of ethical issue with farming them, there is still an opportunity and environmental cost to farming chickens. We could use that land for to cultivate native species and wildlife, or for growing more nutritious and varied crops for people to eat, yet instead we continue to raze the amazon rainforest to make more land for raising farm animals and growing feed. De-densification of farms would only make the demand for farmland even greater than it already is.
Finally, the de-densification of farms would mean a significant increase in the costs of mear production. We'd be pricing lower income groups out of eating meat, while allowing middle- and upper-class folks to carry on consuming animal products as usual. We should not place the burdens of societal progress on the lower class.