Selfhosted

I had the same problem: Debian host + official Jellyfin Docker image, all set up according to the official guide, but it would fail to transcode anything.

There was no relevant information about what was wrong in the logs so what I did was:

• Copy the ffmpeg command from the logs.
• docker exec -it into the Jellyfin container.
• run the same ffmpeg command manually so I could see the error directly.

Long story short, because the Nvidia toolkit uses the driver/libraries from the host, the error was that I was missing the library libnvidia-encode1 on the host. After installing that, everything works as it should.

I have an intel igpu. It was hella painful to pass through the guy into a normal container and I never figured it out. I just ended up running the container with the —privileged flag. QuickSync hwaccel works fine now, I assume it would be the same for NVENC, since the flag basically just passes everything to the container.

Here's mine, no judging for how I set the ip

name: jellyfin
services:
    jellyfin:
        container_name: jellyfin.live
        networks:
            mynet:
                ipv4_address: 192.168.5.3
        ports:
            - 8096:8096
        volumes:
            - jellyfin-config:/config
            - jellyfin-cache:/cache
            - type: bind
              source: /mnt
              target: /Media Disk 1
            - type: bind
              source: /mnt
              target: /Media Disk 2
            - type: bind
              source: /mnt
              target: /Media Disk 3
        environment:
            - HEALTHCHECK_URL=192.168.5.3
        restart: unless-stopped
        deploy:
            resources:
                reservations:
                    devices:
                        - driver: nvidia
                          count: all
                          capabilities:
                              - gpu
        image: jellyfin/jellyfin:latest
networks:
    mynet:
        external: true
        name: mynet
volumes:
    jellyfin-config:
        external: true
        name: jellyfin-config
    jellyfin-cache:
        external: true
        name: jellyfin-cache

The section on the GPU looks like yours. I had to do something to get it working, but it's been forever.

The biggest concern here would be 1) have you installed the Nvidia container toolkit, and 2) how are you passing the GPU into the Jellyfin docker container.

I’ve got an Ansible-playbook that takes care of the Nvidia stuff. I’ve also got a compose file I can share. Will edit this post when I can provide a link.

This works for me, rtx 4060

   jellyfin:
     image: jellyfin/jellyfin:latest
     container_name: jellyfin
     user: 108:114
     network_mode: 'host'
     environment:
       - JELLYFIN_CACHE_DIR=/var/cache/jellyfin
       - JELLYFIN_CONFIG_DIR=/etc/jellyfin
       - JELLYFIN_DATA_DIR=/var/lib/jellyfin
       - JELLYFIN_LOG_DIR=/var/log/jellyfin
       - JELLYFIN_PublishedServerUrl=URL_REDACTED
       - NVIDIA_DRIVER_CAPABILITIES=all
       - NVIDIA_VISIBLE_DEVICES=all
     volumes:
       - /etc/jellyfin:/etc/jellyfin
       - /mnt/driveF/jellyfin/cache:/var/cache/jellyfin
       - /mnt/driveF/jellyfin/data:/var/lib/jellyfin
       - /mnt/driveF/jellyfin/log:/var/log/jellyfin
       - /mnt/Movies:/movies
       - /mnt/TV:/tv
       - /mnt/Music:/music
     runtime: nvidia
     deploy:
       resources:
         reservations:
           devices:
             - driver: nvidia
               count: all
               capabilities: [gpu]
     restart: 'unless-stopped'
     extra_hosts:
       - "host.docker.internal:host-gateway"

Here are some relevant stuff, also have nvidia drivers and vids libs installed.

Running in a podman quadlet on fedora

After=nvidia-cdi-generator

[Container]

Image=docker.io/jellyfin/jellyfin

PodmanArgs=--privileged --gpus=all

Environment=NVIDIA_VISIBLE_DEVICES=all

AddDevice=/dev/dri/card0:/dev/dri/card0