I can go into detail on this, actually. It centers around personal insecurity, and a lack of empathy. Let me explain:

Keep in mind I used to eat meat. I've been a vegetarian for well over 10 years now.

So often they joke or sometimes project with, "how do you know someone's a vegetarian/vegan? They'll tell you."

Now I get this can just be a joke, but oftentimes it comes off more passive-aggressive, reeking of insecurity. So entertain some possible reasons why vegetarians / vegans "tell you":

Whose experience has been that bad with vegetarians/vegans, truly? Apart from what's perpetuated and inflated in media and pop-culture. I'm literally surrounded by 97% of meat-eaters. How do I know they're meat-eaters? I can guess because... 97% are meat-eaters. And that's fine. But my sample of meat-eaters mocking what I choose to eat is far larger than the reverse. 3% means you aren't bumping into too many of us. Speaking for myself, most of the moral arguments are fanned by those wondering why I chose to be one and cornered me into revealing I am a vegetarian (usually because the office or my group of friends is getting lunch or something, or I'm offered something I cannot have). Every single time I kindly explain my reasons and make an effort to note, "I don't care what anyone else does, I just want to do my thing and lead by my own example." Yet understand: it was THEY who asked ME. Food and eating with people is just an event that happens frequently. As a result it's bound to happen that someone orders you something you can't eat or attempts to and you explain to them why you cannot. They ask what you brought for lunch and the moment tofu crosses your mouth, cue the questions. Apparently they resent this and store this as ammunition later, and then remember it being you getting on a soapbox telling them what an awful person they are. (That's probably their inner-conscience talking).

So — duh — it's because you asked.

I assure you, speaking as a former meat-eater who was a meat-eater longer than a vegetarian, we receive A LOT more shit than we dish out. Most of us just want to do our thing and not even tell you. I don't particularly like the attention when it's pointed out — as is the case with most of my peers. I don't mind questions, I just want mutual respect for my choices. Thus considering the sheer ratio of encounters between omnivores/carnivores vs. Plant-based diets, combined with the fact that most vegetarians/vegans have at one point been on the other side while the reverse isn't true, I'd say people me know better the perspectives of both positions.

The problem is that diet becomes deeply personal and political for people. When they see someone else do something they don't feel they can do themselves, then they either (a) try to elevate themselves above that person (case-in-point with @straypet below), or (b) try to bring the other person down. This has everything to do with ego and self-esteem.