Definitely enable; e.g.: lets say based on my sorting algorithm and frequency of frontpage visits, I usually see posts with ~20 upvotes (sounds like a weird oversimplification but that is kind of the case for me foe some reason)

For those 20 upvotes there might be 500 more people that believe that a post is very low quality, inappropriate for the community's topic, etc., but I still see it, since, well its score was ~20.

Without upvotes it was very frequent that I saw a post that was not (or was very-very slightly) related to the community's theme, and there is absolutely no other way for the community to filter it out. It forces me (everyone) to scroll across much irrelevant content, until maybe I give up and unsubscribe from some communities altogether, instead of helping to curate them into something people enjoy visiting.

The only other solution to this problem is to have moderators check every post that is uploaded, and, besides putting more strain to moderators, one moderator's decision might even be debatable sometimes (e.g. a post might technically fit in a community, but the people who visit it might not find it enjoyable, or will just prefer to see it in a more appropriate community).

Downvotes provide a simple solution that involves many opinions in this whole process.

The only case that not having downvotes might make sense, is "Top", in which I feel I only see posts with an insane number of upvotes (that I btw might not enjoy seeing) and nothing else. I also feel that this sorting algorithm also promotes the visibility of more generic content that a larger variety of people will enjoy, and will just upvote without considering the community it was posted in etc. But in those cases, the posts with many upvotes, can only get more upvotes (promoting the phenomenon), whereas with the ability to downvote, the final score will be more balanced, or at least leave the choice to the user (maybe indirectly via the choice sorting algorithm, or their client's settings), for if they want to see controversial posts.