You can run tdarr which is automated transcoding.

https://home.tdarr.io/

I think you're probably better off using something that's build for media servers. One really nice feature is adding more processing nodes to make things go faster, like a gaming PC while not playing anything. I don't think Handbrake can do that by itself.

I just finished setting up transcoding for my media library, and the options I found were Tdarr, FileFlows and Unmanic. They all use ffmpeg and/or Handbrake under the hood, so it kinda comes down to preference. I went with FileFlows because it seemed the most intuitive to me, and it can also process other media like photos, music, audiobooks and ebooks.

I was going to suggest there should be a way to run handbrake locally while pointing its under-the-hood functions to handbrake-cli (also, yes, there's handbrake-cli) hosted on your server, but I found this instead:
https://medium.com/@joshuaavalon/encode-video-with-handbrake-on-server-17b6127f6ac7
godsspeed, OP

ffmpeg

Tdarr is the way to go

Container should have negligible overhead, the main problem you're going to see is where the in/out files are located and shared. Another plus there for docker, can map that volume to anywhere.

If you need a GUI, then handbrake with a web version sounds great, when I did it years ago they didn't have it.

Since then, I've learned ffmpeg and use that regularly and can schedule jobs well now. Definitely more advanced, but more built for server applications.

Not really any performance impact from using it in docker.

But you could also use ffmpeg as that's what handbrake uses under the hood IIRC.

What are you using it for