It is really tricky to genuinely discuss this topic. Many omnivores use this as a straw man argument to discredit vegans for not being fully consequential. On top of that, reasons for being vegan and where people draw the line also vary hugely.

Anyways, I would argue that eating plants and also fungi is very different to eating animal products. First of all, if you are vegan for ethical reasons (as I am) then usually the argument is that one can infer from one's own feelings onto other animals. Sure, this isn't always that easy and we will never know how other animals really feel. This includes fellow humans btw. But it is certainly very definitive that many animals feel pain, discomfort and many other emotions not unlike we feel them.

Plants and fungi on the other hand have completely different body plans. Plants are modular organisms and you simply cannot relate cutting your arm off with cutting a branch. We may deepen our understanding on plants and maybe we will find some form of conscience one day. But this is still far off and for now we can only speculate. Fungi are very different as well and we usually just eat their fruiting bodies anyways.

Secondly, as someone else pointed out, for ecological reasons and for the sheer quantity that is necessary to sustain humans, going vegan is always the better choice. Animals live on plants, too, and just use a lot of the plants' energy on their own metabolism.