Jellyfin’s k3s Deployment had no GPU passthrough, so every transcode ran in software on a cluster with no GPU access. Moving it to a dedicated LXC with VAAPI access to the host’s APU (BMAX Mini PC* with its AMD Radeon Vega iGPU) is simple in principle: stand up the LXC, run Jellyfin there via Ansible, and point the existing Traefik IngressRoute at the new location instead of a cluster pod.
That last step, pointing a Kubernetes Service at an external IP without changing anything downstream, should have been the most boring part of the migration. It produced the more interesting of the two bugs covered here.
View the complete homelab infrastructure source on GitHub
Routing a Service to an IP Outside the Cluster
Traefik’s IngressRoute for Jellyfin references a Service by name: services: [{name: jellyfin, port: 8096}]. To avoid touching that IngressRoute (and its Authelia middleware) at all, the plan was to keep the Service object but back it with the LXC’s IP instead of a pod selector. Kubernetes supports exactly this: omit the Service’s selector field and create an object that lists the backend addresses by hand.
The modern way to do that is an EndpointSlice:
apiVersion: v1
kind: Service
metadata:
name: jellyfin
namespace: apps
spec:
ports:
- port: 8096
targetPort: 8096
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: jellyfin
namespace: apps
labels:
kubernetes.io/service-name: jellyfin
addressType: IPv4
ports:
- name: ""
port: 8096
endpoints:
- addresses:
- 10.0.20.254
Every kubectl check said this was correct: the Service existed, the EndpointSlice existed with the right label linking it to the Service, and the endpoint address was listed. By every signal kubectl could give, the job was done.
Every request to media.woitzik.dev returned 404.
Reading the Actual Error Instead of Re-Checking kubectl
When kubectl says everything is fine, the instinct is to re-check kubectl: verify the label matches exactly, look for typos, look for a missing field. None of that was the problem. The answer had been in Traefik’s own logs the whole time:
subset not found for apps/jellyfin
The Traefik version running here is v2.11 (Helm chart 27.x), and its Kubernetes CRD provider resolves a Service’s backend addresses through the legacy v1/Endpoints API, the object with subsets. It never looks at EndpointSlice, even though that is the API Kubernetes itself prefers for new code, and kubectl has no way of telling you which of the two a given consumer reads. Kubernetes keeps serving both APIs for compatibility gaps like this one (v1/Endpoints has been deprecated since Kubernetes 1.33, but it still works). Traefik only moved its Kubernetes providers to EndpointSlices in v3.1.
The fix is a swap to the older object:
apiVersion: v1
kind: Endpoints
metadata:
name: jellyfin
namespace: apps
subsets:
- addresses:
- ip: 10.0.20.254
ports:
- port: 8096
curl against the public hostname confirmed it immediately, and Traefik’s logs went quiet for that route. Same concept, different API, and only one of the two is what this consumer reads.
The general lesson: “kubectl shows it’s configured correctly” tells you the API server accepted and stored the object. It says nothing about whether the consumer reading that object supports the API you used. For anything where a controller or ingress provider reads Kubernetes objects indirectly (CRD-based routing is the clearest example), check that controller’s logs and documented compatibility before you assume a kubectl-clean object is a working one.
The Second Gotcha: A PVC Shared by Reference Across Unrelated Files
Removing Jellyfin’s old in-cluster Deployment and its PVC surfaced a second problem. This one had real data-loss potential, and it was caught only because a Kubernetes safety mechanism bought time.
The media PVC was defined in jellyfin.yml. From that file alone, it looked like it belonged to Jellyfin and nothing else. It didn’t. Four Deployments in a different file, usenet.yml (Sonarr, Radarr, Bazarr, and SABnzbd), referenced the same PVC by name:
# usenet.yml - nothing in this file defines "media", it just claims it
volumes:
- name: media
persistentVolumeClaim:
claimName: media
Removing the PVC’s definition from jellyfin.yml put it into Terminating. It did not disappear, because of kubernetes.io/pvc-protection, a built-in finalizer that blocks PVC deletion while any pod still uses it. The four usenet pods kept running, the PVC stayed Bound from their point of view, and nothing broke in that moment. That is a temporary window, not a safe state. Any of those pods that restarted (a node drain, an OOM kill, a routine rollout) would fail to start, because a new pod cannot mount a PVC that is being deleted. Once the last running pod was gone, the finalizer would clear and the PVC would be deleted, taking the media library’s volume with it, subject to its reclaim policy.
# What would have caught this before deleting anything:
grep -rn "claimName: media" kubernetes/
# → jellyfin.yml (the file being edited)
# → usenet.yml (NOT obvious from looking at jellyfin.yml alone)
The fix: recreate the same NFS-backed volume under new names, owned by usenet.yml (the file whose Deployments still needed it), and repoint all four claimName references to the new names. The old PVC finished terminating cleanly once nothing referenced it.
The general lesson: before deleting a PersistentVolumeClaim, or any named object other resources reference indirectly (Secrets, ConfigMaps, Services), grep the entire manifest tree for that name, not just the file that appears to define it. claimName, secretName, and similar cross-references are invisible from the defining file. Protective finalizers are useful, but they can create a false sense of safety: “nothing broke yet” does not mean “this was safe.”
Both Gotchas, Side by Side
| EndpointSlice/Endpoints | Shared PVC | |
|---|---|---|
What kubectl showed | Completely correct | Completely correct (until deletion) |
| What actually mattered | Whether the consuming controller (Traefik) supports the object’s API | Whether other files reference the same object by name |
| Where the real signal was | The controller’s own logs, not the API server | A repo-wide grep, not the file being edited |
| Safety net that bought time | None, direct outage | pvc-protection finalizer, temporary, not a fix |
Both failures have the same shape. kubectl (or any direct API inspection) confirms an object exists and is well-formed, but the real question (does this consumer read it, does this name have other dependents) lives somewhere kubectl doesn’t look. In both cases the fix came from a different source of truth: the consuming controller’s logs in one, a full-tree text search in the other.
Check the EndpointSlice/Endpoints behavior against your own ingress controller version before relying on either object. Provider support for newer Kubernetes APIs varies and changes between releases, as Traefik’s switch in v3.1 shows. The same pattern applies on AKS when a Service fronts something outside the cluster (an on-prem system, a VM): back it with the object your ingress controller actually reads.