cgroup v2 is a production migration, not a node-image upgrade
Moving Kubernetes nodes from cgroup v1 to v2 changes the resource-accounting contract for runtimes, applications, and observability. Treat it like a production migration.
There is a very tempting cgroup v2 migration plan.
Pick the new node image. Roll a node pool. Watch the nodes join. Drain the old pool. Check that Pods are Running. Declare victory.
That plan proves that Kubernetes can schedule containers on the new machines. It does not prove that the containers understand their limits, that the metrics still describe the same reality, or that the tools you will need during the first memory incident still work.
That is the important distinction.
cgroup v2 is not a Kubernetes YAML migration. It is a change to the Linux resource-accounting contract underneath Kubernetes. A lot of software has quietly built assumptions around that contract: JVMs sizing their heap, Node.js deciding how much memory it can use, agents scraping cgroup files, security tools finding process groups, dashboards interpreting container metrics, and on-call runbooks pointing at paths that used to exist.
If those pieces disagree, the cluster can look healthy right up until it is under pressure. Then you are debugging a production incident with an application that believes it has node memory, an agent that reports a different limit, and an alert that lost its denominator.
That is not an image refresh. That is a migration.
the filesystem is part of the API
The visible difference is deceptively small. cgroup v1 exposes separate controller hierarchies and familiar files such as memory.limit_in_bytes and cpu.shares. cgroup v2 uses a unified hierarchy and different names and semantics, including memory.max and cpu.weight.
The deeper difference is that the controllers are designed as one system. That gives Kubernetes a better foundation for resource management, but it also means an old filesystem reader is not merely looking in the wrong directory. It may be applying the wrong model to the data it finds.
Kubernetes has been clear about the direction of travel. cgroup v1 support moved into maintenance mode in v1.31; with v1.35, kubelet defaults to refusing cgroup v1 nodes. There is a temporary failCgroupV1: false escape hatch, but Kubernetes has scheduled cgroup v1 removal for v1.38.
An escape hatch is useful for a controlled rollback. It is a terrible migration strategy.
The v2-only features are another reason not to pretend the switch is cosmetic. Memory QoS uses cgroup v2 controls such as memory.high, memory.min, and memory.low. Kubernetes’ container-aware OOM behavior and its memory-pressure signals change too. In-place resizing of memory-backed emptyDir volumes also requires cgroup v2 nodes.
In other words: the platform will keep gaining behaviors whose correctness depends on this layer. Delaying compatibility work just accumulates interest.
the first victims are usually the least visible
The obvious workload test is a service answering an HTTP health check. That is necessary, and spectacularly insufficient.
Start with the process that guesses its own memory budget.
An old JVM running on an Amazon Linux 2023 EKS node can fail to detect the container limit under cgroup v2. Amazon documents the concrete outcome for JDK 8 releases before 8u372: the JVM can size its heap from node memory instead of the Pod limit, then hit an OOM and restart. Nothing in the Deployment manifest necessarily changed. The interface below it did.
Node.js has the same category of risk: container-awareness depends on a sufficiently recent runtime. Kubernetes maintains a compatibility table for Java, Node.js, automaxprocs, and cAdvisor because these are not academic edge cases. They are the boring dependencies that turn a clean node rollout into a strange capacity incident.
Then come the tools.
Some agents read /sys/fs/cgroup directly because it was the simplest way to obtain a limit, a usage value, or a process grouping. Some Docker-era dashboards expect the v1 metrics layout. Some security products correlate processes and containers using the old hierarchy. Some home-grown exporter exists because it solved a useful problem five years ago and nobody has needed to touch it since.
Those are exactly the components I would expect to fail late: after the upgrade, during load, while everyone assumes the problem must be in the application.
This is why “our Pods started” is not a meaningful acceptance criterion. A migration is not finished until the service, the runtime, and the evidence used to diagnose the service all agree about its resources.
driver alignment is still production work
There is one configuration detail that deserves less drama and more discipline: the cgroup driver.
When systemd manages the host, Kubernetes recommends the systemd cgroup driver for both kubelet and the container runtime. A kubelet/runtime mismatch can create two competing views of the same node. Under resource pressure, that becomes an instability problem rather than a configuration nicety.
Do not check this only in the new image’s release notes. Inspect the running kubelet and runtime configuration on a canary node. Make the intended driver explicit. Verify it after joining the node. Then test the thing that exposes disagreement: memory pressure, eviction, and OOM behavior.
The same goes for CPU. cgroup v1’s cpu.shares and v2’s cpu.weight are not interchangeable labels. OCI runtime updates have changed the conversion to preserve more useful priority behavior at small requests. If a policy tool or capacity dashboard assumes exact values, its arithmetic needs a fresh test, not a hopeful rename.
a preflight I would actually run
I would treat the rollout as a small compatibility programme with a deliberately unglamorous checklist:
- Inventory direct readers. Search application images, DaemonSets, sidecars, startup scripts, and operational repositories for
/sys/fs/cgroup, v1 filenames, and parsers that assume controller-specific mount paths. - Make a runtime matrix. Record the JDK, Node.js, OCI runtime, container runtime, cAdvisor, and agent versions running in production. Compare each to the upstream cgroup v2 compatibility guidance. Do not let “latest in the base image” stand in for evidence.
- Canary real workloads. Put representative Java, Node.js, batch, I/O-heavy, and multi-process workloads on the new pool. Verify the limit observed inside the process, not only the limit declared in the Pod spec.
- Test pressure on purpose. Exercise memory pressure, a controlled OOM, CPU contention, eviction, and restart handling. Check application logs, kubelet events, dashboards, alerts, and security telemetry against the same event.
- Diff the observability contract. Compare the metrics, labels, cgroup paths, and alert thresholds from equivalent v1 and v2 workloads. A graph that still renders can still be lying.
- Practice the rollback. Keep the temporary v1 path time-boxed and documented, but prove that it works before the incident where you might need it.
I would add one operational test that teams often skip: hand the canary incident to somebody who did not build the migration. Give them an OOM or a throttled service and ask them to explain what happened using normal dashboards and runbooks. If they cannot, the migration is not ready. You have merely moved the mystery closer to production.
stop calling the node image the change
Platform upgrades are often described by the artifact being replaced: an AMI, an OS, a kernel, a container runtime, a node image.
That is convenient for a ticket. It is a poor model for risk.
The real change here is that workloads and tooling observe Linux resource controls through a different interface, with different semantics and new capabilities. Kubernetes is forcing the issue because cgroup v1 is on its way out, but the deadline is not the interesting part. The interesting part is whether your production system can still tell the truth about itself after the move.
My rule is simple: an infrastructure migration is complete only when the workload and the tools used to explain its failures agree on reality.
For cgroup v2, nodes joining the cluster is the beginning of that proof, not the end.
references
- Kubernetes: The Shift to cgroup v2 in Kubernetes
- Kubernetes: About cgroup v2
- Kubernetes: Container runtimes and cgroup drivers
- Docker: Runtime metrics
- Amazon EKS: AL2 and AL2-accelerated AMI transition
- Kubernetes: Resize CPU and memory resources assigned to containers
To test my projects, I use Railway. If you want $20 USD to get started, use this link.