It's strange that this concern for context only ever goes in one direction. Symbolism, like words, develop meaning through their usage.

If I were to say that I ejaculated during intercourse with your wife last night, would you take that to be an insult or would you be dying on that same context hill that the verb to ejaculate used to refer to suddenly making a statement and that intercourse used to refer to having a discussion with someone?

Probably not.

Would you say that the swastika isn't a Nazi symbol because it originated in Indo-European religious and cultural symbology?

Maybe. I can't speak for you.

The origin of something doesn't determine its usage.