First off, when someone judges or shuns you for your interests, that says more about them than it does about you. It took me a long time to really internalize that, and it wasn't easy. Also, it's a good indicator for people you want to keep around.
I've taken the approach of "thank you for letting me know we're not compatible, so bye Felicia." (I say this to myself, not out loud)
Read up on Ugol's Law. There's a >50% chance that many other people share some of your interests.