At this point it's something of a running gag for source readers since it gets the harem allegations thrown at it every time the series comes up.

At a glance it looks like every other harem show. But MC has absolutely ZERO interest in his party. There aren't any "ups I just stumbled and happen to enter the girls bath" shenanigans and MC has a single target of his affection who isn't in his party. He never tries anything with the girls in his party and as a LN reader we get the inner thoughts of the protagonist as well and know that he never even thinks about trying anything.

There are a few things that doesn't help though. While it doesn't exactly break the source canon, I believe that the anime adaptation was deliberately ambiguous about MCs thoughts regarding the party in order to make it more compelling to the horny otaku audience. That just might be my prejudice speaking but a anime only watcher never gets that definitive "not a harem" that a LN reader gets. I think that this decision hurts the source. Then there is one girl that likes to make lewd suggestions to MC that can be easily clipped out of context to further the harem narrative. The girl is a reincarnate herself and was in her past life an old granny. Her personality is basically a slightly more sane version of Dan Dan Dan's Turbo Granny. She makes inappropriate jokes and comments all the time. The gap-moe is funny in context but out of context it can be used by someone who wants a gotcha moment.