Restart an OOM killed docker automatically (lemmy.world)

submitted 1 month ago by RustyNova@lemmy.world to c/selfhosted@lemmy.world

32 comments fedilink hide all child comments

I got an home server that is running docker for all my self hosted apps. But sometimes I accidentally trigger Earlyoom by remotely starting expensive docker builds, which kill docker.

I don't have access to my server outside of my home network, so I can't manually restart docker in those situations.

What would be the best way to restart it automatically? I don't mind doing a full system restart if needed

you are viewing a single comment's thread
view the rest of the comments

[-] lemmyng@lemmy.ca 17 points 1 month ago

Use -m and limit the build job's memory so it doesn't kill the docker daemon.

[+] RustyNova@lemmy.world -18 points 1 month ago

Fair enough. But I don't want a bandaid fix solution. Even more that I do all my docker through portainer and the option isn't there.

It could also be useful if a container got a memory leak and is unbounded

[-] Badabinski@kbin.earth 16 points 1 month ago

The other person may have responded with a fair amount of hostility, but they're absolutely correct. I run Kubernetes clusters hosting millions of containers across hundreds of thousands of VMs at my job, and OOMKills are just a fact of life. Apps will leak memory, and you're powerless to fix it unless you're willing to debug the app and fix the leak. It's better for the container to run out of memory and trigger a cgroup-scoped OOM kill. A system-wide OOM kill will murder the things you love, shit in your hat, and lick your face like David Tennant licked Krysten Ritter.

[-] RustyNova@lemmy.world 1 points 1 month ago

Oh that's not a problem to let a container get killed. It's perfectly fine. What I want is just not crippling my whole server because one container did a funny.

If it keeps docker and the portainer VM I'll be 100% ok, because I can just restart it. I don't want to have remote access to my server outside of my home for security reasons, so this is just the bare minimum

[-] Findmysec@infosec.pub 5 points 1 month ago

Those remote access fears can be solved with a wireguard VPN

[-] null@slrpnk.net 0 points 1 month ago

I don't want to have remote access to my server outside of my home for security reasons, so this is just the bare minimum

What are your security concerns?

load more comments (15 replies)

this post was submitted on 12 Jul 2024

25 points (83.8% liked)

Selfhosted

38633 readers

101 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago

MODERATORS

HybridSarcasm@lemmy.world

HybridSarcasm@lemmy.hybridsarcasm.xyz