kubernetes resources are a control loop now

For a long time, Kubernetes had a comforting contract.

You wrote CPU and memory requests into a deployment. You chose a replica count. The scheduler placed the Pods. Then, apart from a rollout, a node failure, or an operator doing something exciting, that shape stayed put.

An enormous amount of platform machinery grew around that contract. Capacity plans treat requests as reservations. Cost dashboards treat replicas as a stable unit of spend. Alerts treat zero replicas as an outage. PodDisruptionBudgets are designed around drains and failures. Runbooks assume that changing resources means shipping a new spec and waiting for replacement Pods.

Kubernetes 1.37 is where that model stops being a safe default.

Not because one feature landed. Because several did, from different corners of the project, and they all make the same thing possible: a workload can change shape at runtime. It can be parked at zero, given more memory where it already runs, or protected from memory pressure according to a policy that belongs to the node rather than the Pod.

That is useful. It is also a control loop.

And the important question is no longer “can Kubernetes resize this?” It is: who owns the loop, what is it allowed to do, and what happens when it is wrong?

the deploy-time contract is gone

In-place Pod resize became stable in Kubernetes 1.35. CPU and memory can change through the resize subresource without automatically recreating the Pod. In 1.37, scheduler preemption for a resize is available behind an Alpha gate: when a high-priority Pod has a valid resize stuck in Deferred, the scheduler can evict lower-priority Pods on that same node to make room.

The HorizontalPodAutoscaler can now scale a workload to zero by default, provided it uses an object or external metric. Memory QoS is Beta and enabled by default on supported nodes, although its new safe default deliberately does not turn on memory throttling for existing workloads.

Each change is defensible in isolation. Together, they remove the old separation between “the desired spec” and “the runtime.” A VPA, HPA, scheduler, kubelet, external-metrics adapter, and node configuration can now collectively change availability and resource allocation while nobody merges a deployment PR.

This is not Kubernetes becoming magical. It is Kubernetes becoming more honest about what it already is: a collection of controllers acting on feedback.

The failure modes did not disappear. They moved into policy.

zero replicas is now a state machine

The first trap is familiar because it looks harmless: minReplicas: 0.

For queue consumers, this is genuinely good. If no work exists, paying for idle workers is silly. The HPA can observe a queue length that exists outside the Pods, scale workers from zero when the queue grows, and return to zero when it drains.

But CPU and memory metrics cannot wake a workload that has no workload running. There are no Pods left to measure. The HPA therefore requires at least one object or external metric for scale-to-zero; resource metrics alone are rejected. That is not an implementation detail. It is the design test.

If the metric disappears, the autoscaler cannot invent capacity. If an HTTP service goes to zero without a buffering layer, Kubernetes Services do not hold requests until a Pod becomes ready. The request just discovers that the clever cost optimization was actually a cold-start outage.

The practical distinction is between work that can wait and work that cannot:

• A durable queue can absorb the wait while workers start.
• A synchronous request path needs an explicit buffer, a proxy that understands the cold start, or a minimum replica above zero.

There is a second operational change that is easy to miss. A replica count of zero is no longer enough information. In 1.37, an HPA records ScaledToZero=True when it owns that state. Zero replicas without that condition still means a manual pause. Your dashboards and alerts need to learn the difference, otherwise every successful downscale becomes an incident page—or every actual accidental zero gets ignored because the team got tired of the noise.

During an upgrade, control-plane skew makes this sharper. The API server may accept minReplicas: 0, while a controller manager without the feature enabled interprets zero as a manual pause and leaves the workload stranded there. “We upgraded the cluster” is not an answer; the API server and controller manager need the same semantic contract.

resize preemption is a new kind of eviction

In-place resize has always sounded like a tidy rightsizing story: VPA changes a request, the kubelet changes a cgroup, no restart, everybody goes home.

That was never the whole story. A Pod cannot grow if its node has no headroom.

Before 1.37, a requested increase could sit in Deferred indefinitely. The scheduler did not treat a running Pod as something that needed scheduling. The usual escape routes were manual eviction, a node-level autoscaling event, or simply maintaining enough idle capacity that the resize never mattered.

Scheduler-driven resize preemption changes the answer. The scheduler watches deferred resizes and can evict eligible lower-priority Pods on the running Pod’s own node. It is deliberately local: this is not a cluster-wide re-placement exercise. If clearing that node still cannot satisfy the request, the resize remains deferred.

That makes priorities much more consequential. A PriorityClass no longer only influences who gets placed during a shortage; it can decide who stays alive when an already-running neighbor wants to grow. PodDisruptionBudgets and graceful termination policy are involved too, but they are not a substitute for deciding whether the eviction hierarchy makes sense.

Teams should stop treating PriorityClasses as a small YAML taxonomy owned by application teams. They are now part of a runtime resource policy. Somebody needs to own the questions behind them:

• Which workloads may grow in place?
• Which workloads may be evicted to make that possible?
• Is a PDB protecting a user-visible service, or accidentally protecting low-priority batch work from the policy that was meant to displace it?
• Which nodes should opt out because another controller owns their capacity decisions?

Kubernetes provides a node-level opt-out for resize preemption. That is useful, but it is also an admission that this behavior changes the blast radius of a resize. Do not enable it merely because it is there. Make it part of the same policy review as priorities, disruption budgets, and cluster autoscaling.

One more awkward detail: “in place” does not mean every process experiences a memory change safely. The Kubernetes project explicitly calls out Java and Python runtimes as not supporting memory resizing without a restart. That is the kind of caveat that turns a generic VPA rollout into a language-runtime inventory.

memory qos is a node policy, not a pod feature

Memory QoS is the easiest feature to misunderstand because its default is intentionally quiet.

In 1.37 the feature is Beta and on by default, but memoryThrottlingFactor defaults to null. So an upgrade does not suddenly write memory.high for workloads that never ran with memory throttling before. That is a good migration decision.

It is also how a major operational model change slips past teams: nothing happens until someone decides to configure it, and most teams never decide.

When you do decide, the decision is not per workload. memoryReservationPolicy: TieredReservation applies to every Pod on a node. Guaranteed Pods receive memory.min; Burstable Pods receive memory.low. There is no per-Pod opt-in. A node mixing “this must be hard-reserved” with “this should remain reclaimable” has to pick one policy for both.

Hard reservation also covers memory charged to the cgroup, including page cache. A Pod reading large files can reserve memory the kernel would otherwise reclaim for its neighbors. So the useful unit of analysis is no longer only a container’s heap or RSS. It is the working set of everything sharing that node.

This is why Memory QoS should be a node-pool decision, documented alongside instance type, cgroup version, and workload placement rules. A generic node pool with a generic memory policy is not neutral. It is simply an undocumented trade-off.

the pre-upgrade checklist that matters

Most Kubernetes upgrade checklists are API removals plus a green control plane. Add a control-loop section before either of those gives you false confidence.

1. Find every HPA that could reach zero. Classify it as queue-driven, request-driven, or something else. For every candidate, prove that the object or external metric is available while no Pods exist.
2. Test the metric path directly. Do not accept a pretty Grafana graph as evidence. Ask the External Metrics API for the exact series the HPA will use, then test scale from zero and back under realistic cold-start time.
3. Rewrite zero-replica alerts. Page on a zero state without ScaledToZero=True, an unavailable scale-up metric, or a failed readiness objective—not simply on a replica count.
4. Assign ownership of priorities and PDBs. Decide which controllers can request growth, which workloads are valid resize-preemption victims, and which nodes must opt out.
5. Make a runtime inventory for resizing. Include Java and Python workloads, swap, and nodes using the static CPU or Memory Manager, all of which limit the clean in-place story.
6. Choose Memory QoS per node pool. Confirm cgroup v2 prerequisites, then document whether the pool is using throttling, tiered reservation, neither, or a deliberately isolated mix of workloads.
7. Plan the skew and rollback. Do not create scale-to-zero HPAs until both the API server and controller manager support the feature. Before downgrading, move affected HPAs back to at least one replica and wake anything currently at zero.

That list is not bureaucratic overhead. It is the missing specification for a system that is now allowed to change production capacity by itself.

defaults are safe, which is why this needs attention

There is a contrarian read of all this: Kubernetes did the cautious thing.

Scale-to-zero requires a metric that survives zero. Memory QoS avoids silently introducing throttling. Resize preemption is Alpha and gated. The project is not forcing anybody into chaos.

Good. But safe defaults can still alter the operational model. They put the capability in the cluster, then wait for a well-meaning team to enable a setting, add a VPA mode, or copy an HPA example into a chart. The first incident will not look like “we adopted a new scheduling policy.” It will look like a deployment at zero replicas, a batch job that vanished, or a node that became oddly unreclaimable.

The control loop exists whether or not you have an opinion about it. The teams that get value from it will be the ones that write down who may shrink a workload, who may evict another one, which metrics represent demand, and what condition actually pages a human.

That is platform engineering now. Not setting resource numbers once, but governing the systems that change them later.

references

• Kubernetes v1.37: Scale Workloads to Zero with HorizontalPodAutoscaler
• Kubernetes v1.37: Scheduler Preemption for In-Place Pod Resize
• Kubernetes v1.37: Memory QoS Graduates to Beta
• Kubernetes 1.35: In-Place Pod Resize Graduates to Stable
• Kubernetes release support window
• Memory QoS mixed-workload limitation

To test my projects, I use Railway. If you want $20 USD to get started, use this link.