Kubernetes

[EDIT (solved)]: Turns out, cilium did not remove its network links, and somehow kept updating to my current CIDIR leading to a duplicate, removing the links worked.

I keep on getting issues with CNI and networking.. I just want my cluster to work.. anyways

Apr 28 17:14:30 raspberrypi k3s[2373903]: time="2025-04-28T17:14:30+12:00" level=error msg="flannel exited: failed to register flannel network: failed to configure interface flannel.1: failed to set interface flannel.1 to UP state: address already in use"

How do i see what is using flannel Here is my server arguments:

ExecStart=/usr/local/bin/k3s \
    server \
        --kubelet-arg=allowed-unsafe-sysctls=net.core.rmem_max,net.core.wmem_max,net.ipv4.ip_forward \
        --flannel-backend vxlan \
        --disable=traefik \
        --write-kubeconfig-mode 644

So I am using the default flannel backend, I tried repeatedly uninstalling then re-installing k3s, I deleted the current flannel interface with ip link, there is no other k3s instance is running, so why am I getting this issue?

external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:625] MmapAligned() failed - unable to allocate with tag (hint, size, alignment) - is something limiting address placement? 0x177840000000 1073741824 1073741824 @ 0x555b5fccc4 0x555b5f90e0 0x555b5f89a0 0x555b5d81d0 0x555b5f6694 0x555b5f6468 0x555b5cd988 0x555b4e3c84 0x555b4e09a0 0x7fb3918614
external/com_github_google_tcmalloc/tcmalloc/arena.cc:58] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size); is something preventing mmap from succeeding (sandbox, VSS limitations)? 131072 632 @ 0x555b5fd034 0x555b5d8260 0x555b5f6694 0x555b5f6468 0x555b5cd988 0x555b4e3c84 0x555b4e09a0 0x7fb3918614
spiderunderurbed@raspberrypi:~/k8s $ 

Does anyone know how to fix the memory issue with cilium? or could link me to the docs or any issues about this. I just followed the instructions to install cilium, most stuff is up, I think tje daemon set is down? more specifically this pod cilium-envoy-chzf8 is in a crashloop of this memory issue, I have 3gbs+ of RAM avalible, but I dont think cilium would take up the rest of my memory nor does it look like size is a issue, here is a example:

               total        used        free      shared  buff/cache   available
Mem:           7.6Gi       4.0Gi       374Mi       197Mi       3.7Gi       3.7Gi
Swap:          511Mi       188Mi       323Mi
spiderunderurbed@raspberrypi:~/k8s $ 

Cilium status: https://pastebin.com/yRRbcT6v

[EDIT] Soo.. kinda fixed? It was my backend, turns out, it forwards /nextcloud onto the nextcloud service, which does not know what to do with it unless I set something like site-url to include that path. So I made a middleware to strip the prefix, but now it cannot access any of its files because it will use the wrong path. I will look for siteurl settings but I dont think all of my services have one, so any advice would be appreciated for a general solution

So currently my raspberrypi is connected to my internet under the ip, 192.168.68.77, (I configured traefik to work with that host and alternative hosts if need be). According to traefik logs I think that it does not work because it is missing access to the api server, although i could be wrong, i installed traefik via helm, and I have a config file for it, and disabled the default traefik given by k3s. here is the traefik config and logs: config: https://pastebin.com/XYH2LKF9 logs: https://pastebin.com/sbjPZCXv pods and svcs (all): https://pastebin.com/4y8h5YUK

The ingress is exposed properly, I know because of the curl behavior, so traffic going to 192.168.68.77 is going to traefik, and trying any of the services I have exposed like /traefik or /nextcloud does not work (404 error):

***
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: nextcloud-route
  namespace: default
spec:
  entryPoints:
    - web
  routes:
    - match: (Host(`192.168.68.77`) && PathPrefix(`/nextcloud`)) || (Host(`192.168.1.22`) && PathPrefix(`/nextcloud`))
      kind: Rule
      middlewares:
        - name: general-middleware
      services:
        - name: nextcloud-service
          port: 80

Example of my routes

It was my backend, turns out, it forwards /nextcloud onto the nextcloud service, which does not know what to do with it unless I set something like site-url to include that path. So I made a middleware to strip the prefix, but now it cannot access any of its files because it will use the wrong path.

By this I mean, I have a powerdns server running in my cluster, I would like Kubernetes to add/update dns entries in my dns server to reflect all services or any domains that would be used within the cluster, this is to fix a current issue I am having, and for general control and centralization purposes.

  --image=nicolaka/netshoot \
  --restart=Never \
  -- /bin/bash
If you don't see a command prompt, try pressing enter.
network-tools:~# cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.43.0.10
options ndots:5
network-tools:~# 

DNS does not work in my k8s cluster. I dont know how to debug this, this is all my logs are in Coredns and kubedns:

[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server

This probably isnt enough, but what more can I do to debug this? I dont think its anything to do with my CNI, I am using calico, 1.1.1.1 as a nameserver or any nameserver works, but the issue is that internal to external dns mappings do not work, dns cannot resolve outside. Maybe not inside either according to this:

spiderunderurbed@raspberrypi:~/k8s $ kubectl run -it --rm network-tools-2   --image=nicolaka/netshoot   --restart=Never   -- /bin/bash
If you don't see a command prompt, try pressing enter.
network-tools-2:~# ping traefik.com
ping: traefik.com: Try again
network-tools-2:~# 

the services for kubedns and coredns does not work, but the logs as I sent above, dont show me much.

My cluster has been showing my raspberrypi node as "Ready" but according to the description of the node, the last log was "NodeNotReady" all debug guides say look for any pressure, like disk, pid, or so on, but there is no pressure, no absence of network. Here is the logs of my pi and pi status: https://pastebin.com/UULz6Hcy My pods are stuck in unknown (except jellyfin which is awaiting another node to come on): https://pastebin.com/vw2masAC A description of one of my pods if that helps: https://pastebin.com/s5W03s0E

also i already tried re-installing k3s

Someone added a PR to an app of mine adding instructions for k8s setup. I do like the idea of providing these instructions, but I don't have any experience with k8s whatsoever. The commits look fine to me, but in case anyone is experienced, I'd appreciate if you can take a look. I don't want to inadvertently add something malicious. Here's a link to the PR: https://github.com/SinTan1729/chhoto-url/pull/48, thanks.

When combined with today’s other vulnerabilities, CVE-2025-1974 means that anything on the Pod network has a good chance of taking over your Kubernetes cluster, with no credentials or administrative access required.

Authors: Daniel Vega-Myhre (Google), Abdullah Gharaibeh (Google), Kevin Hannon (Red Hat)

In this article, we introduce JobSet, an open source API for representing distributed jobs. The goal of JobSet is to provide a unified API for distributed ML training and HPC workloads on Kubernetes.

[...]

[T]he Job API fixed many gaps for running batch workloads, including Indexed completion mode, higher scalability, Pod failure policies and Pod backoff policy to mention a few of the most recent enhancements. However, running ML training and HPC workloads using the upstream Job API requires extra orchestration to fill the following gaps:

Multi-template Pods : Most HPC or ML training jobs include more than one type of Pods. The different Pods are part of the same workload, but they need to run a different container, request different resources or have different failure policies. A common example is the driver-worker pattern.

Job groups : Large scale training workloads span multiple network topologies, running across multiple racks for example. Such workloads are network latency sensitive, and aim to localize communication and minimize traffic crossing the higher-latency network links. To facilitate this, the workload needs to be split into groups of Pods each assigned to a network topology.

Inter-Pod communication : Create and manage the resources (e.g. headless Services) necessary to establish communication between the Pods of a job.

Startup sequencing : Some jobs require a specific start sequence of pods; sometimes the driver is expected to start first (like Ray or Spark), in other cases the workers are expected to be ready before starting the driver (like MPI).

JobSet aims to address those gaps using the Job API as a building block to build a richer API for large-scale distributed HPC and ML use cases.

cross-posted from: https://lemmy.ml/post/20234044

Do you know about using Kubernetes Debug containers? They're really useful for troubleshooting well-built, locked-down images that are running in your cluster. I was thinking it would be nice if k9s had this feature, and lo and behold, it has a plugin! I just had to add that snippet to my ${HOME}/.config/k9s/plugins.yaml, run k9s, find the pod, press enter to get into the pod's containers, select a container, and press Shift-D. The debug-container plugin uses the nicolaka/netshoot image, which has a bunch of useful tools on it. Easy debugging in k9s!