I get what the author is talking about with losing trust in your services when they are unavailable. I have found that a Raspberry Pi 3/4 cluster running k3s utilizing NAS storage has very good uptime and honestly I think my internet service goes down more than my homelab services.
Thanks! I find most of the issues occur during upgrades to services, but that is to be expected.
My internet service is usually more of an issue than most services I run. Though some things take longer to get tweaked and running well and that can cause issues.
Upgrades to services - that’s why I run dev namespaces and copy over my production shares to dev and utilize zfs snapshotting.
I haven’t set up testing yet and only just started with prometheus monitoring but so far things run pretty well.
Oh, dev namespaces are a good idea. Do you have a dev domain then too?
I toyed with dev domain but ended up using namespace.tld and postfixing -dev to my namespace so it works out to service.tld and service-dev.tld.
Ah okay that makes sense, you’re using the internal cluster domain to route to services
I have automated traefik to route the traffic, it sets the dns and ingress route. I’m also doing as you suggested for service to service connections.
That makes sense!
Have you played with anything like Istio to secure in-cluster communications? I think Hashicorp Consul can do something similar to encrypt service to service communications.
I looked into it but I felt at the time it was too complex, maybe I’ll look at it again. Currently I am using wireguard for all cluster node-to-node traffic. It seemed like a reasonable tradeoff at the time, but it is at the network layer instead of application, so I really should revisit that at some point.
Yeah it very adds some extra complexity and it’s more important for if you are hosting in public clouds anyways IMO.