The difference here being that the swastika was used as the official symbol of a fascist government that conquered a substantial portion of Europe and perpetrated the holocaust. Outside of contexts where its religious significance overshadows this, it's pretty universally associated with Nazis, and even in the places where its other uses are overshadowed it's presented differently than the black, tilted swastika in the middle of a white circle on a red field. There are swastikas on the sign of a local Indian restaurant down the street from me, but nobody confuses them with the very clear nazi version. Pretty unambiguous.

Meanwhile Pepe, a relatively recent creation, was coopted by a handful of fascists who don't use it as their primary symbol and haven't done anything nearly as drastic or impactful with it, and has since become a widely used emoji character with a huge number of variations that's used by all sorts of people. Have fascists made use of it? Yeah. But most of the people using these emojis in their discord posts have nothing to do with alt-right fascists and aren't even especially likely to be right-wingers in any sense at all.

It simply doesn't have the same connotation that the symbols it's being compared to here do. It's easy for people who aren't versed in spaces where it's used regularly and innocuously to assert that it is, but that doesn't make it any more true.

What it does do is make those people seem incredibly out of touch to people who are accustomed to seeing cringespin and sleepypeepo and grabbyhands in their discord chats posted by literal queer leftists.