GPU allocation is becoming a platform contract
I think nvidia.com/gpu: 1 is one of the most expensive innocent-looking lines of YAML in a modern platform.
It says almost nothing about the decision being made. Does this job need an entire H100, or would a hardware-isolated slice do? Can it share? Does it need to sit next to three other GPUs on the same fabric? Is it a production inference service, a weekend experiment, or a training run that can wait until Tuesday? Who pays when it idles for six hours?
Kubernetes has been happy to accept the integer. The business has still had to live with all the questions behind it.
Dynamic Resource Allocation (DRA) changes the vocabulary. In Kubernetes 1.37, support for using extended resource names with DRA became stable, providing a practical migration route from the familiar device-plugin request to DRA-backed allocation. But the interesting part is not a newer API. It is that accelerators can finally be described as deliberate platform choices instead of magic numbers in a container spec.
That makes GPU allocation a platform contract.
The old interface was deliberately too small
The classic device-plugin model did a useful job: advertise a GPU resource on a node, ask for a count, and let the scheduler place the Pod. For a cluster with a few homogeneous cards and one team, it is an entirely reasonable place to start.
It becomes awkward as soon as the fleet is real. An integer cannot convey GPU memory, architecture, MIG geometry, a connectivity requirement, whether a device is shareable, or what should happen when the desired hardware is busy. We usually compensate with labels, node pools, convention, tribal knowledge, and increasingly inventive Helm values.
That is not an API design critique. It is a warning about where policy has been hiding.
DRA lets a workload reference a ResourceClaim, while a platform publishes device classes and the driver selects devices by attributes. NVIDIA’s DRA driver, for example, can select on properties such as model, architecture, memory, and PCI/topology information. The claim becomes a real allocation object rather than an implied side effect of a resource limit. NVIDIA’s comparison of DRA and the device plugin shows why that distinction matters.
The practical payoff is not that developers need to learn more YAML. It is that developers can ask for a product the platform understands.
Give developers names that mean something
I would not expose every SKU as a developer-facing contract. h100-sxm5-80gb-nvlink-island-7 is an inventory record, not a helpful interface.
Instead, define a small set of classes that communicate workload intent. The exact names are yours, but they should make a reviewer understand the request without decoding node labels:
gpu.inference.shared: latency-tolerant inference that may use an approved sharing mode.gpu.inference.dedicated: a whole device for a production service with a predictable performance requirement.gpu.training.partitioned: a hardware-isolated MIG slice with a stated memory floor.gpu.training.fabric: a multi-GPU training allocation whose topology is part of the promise.gpu.research.best-effort: capacity that is deliberately cheap to preempt, delay, or refuse.
Those labels are not implementation details. They are commitments. Behind gpu.training.fabric, the platform might require a compatible ComputeDomain, suitable interconnect, and an all-or-nothing admission rule. Behind gpu.inference.shared, it might set a strict concurrency ceiling and make no tail-latency promise.
If the class cannot be described in one sentence to the workload owner, it is probably too close to the hardware.
Sharing is a performance policy, not a utilization trick
The phrase “GPU sharing” hides several very different behaviours.
MIG creates hardware-enforced partitions on supported NVIDIA GPUs: each slice has its own compute engines, memory bandwidth, and cache allocation. That is a sensible basis for multi-tenant batch work when the profile fits. Time-slicing is different: workloads take turns. MPS is different again: processes run concurrently with configured limits. DRA can also account for consumable capacity across claims. The NVIDIA DRA allocation guide lays out those models clearly.
They should never be interchangeable just because they improve a utilization dashboard.
My default policy would be blunt:
- Production latency and untrusted workloads get isolation. Whole devices or hardware partitions. No accidental neighbours.
- Batch inference gets approved sharing only after measurement. Set a concurrency and memory policy; observe queueing and tail latency; stop pretending the device is dedicated.
- Interactive research gets a separate, explicitly lossy lane. It is fine for a notebook to share. It is not fine to surprise a production customer with that decision.
This is also where enthusiasm needs a brake. NVIDIA documents several DRA sharing capabilities as alpha and disabled by default, including consumable shares, dynamic MIG, MPS support, and time-slicing settings. DRA makes the contract expressible; it does not turn immature sharing features into a safe default. Read the GPU Operator’s current maturity table before turning a feature gate into a capacity plan.
A queue is where fairness becomes real
Once two teams want the same GPU, allocation is no longer only a scheduler problem. It is admission control.
Kueue brings queues, quotas, resource flavors, borrowing, priority, and topology-aware admission into the conversation. Its DRA integration explicitly deals with ResourceClaim templates, extended-resource migration, quota accounting, and shared or partitionable devices. Kueue’s DRA documentation is worth reading even if you choose another queuing layer, because it asks the right operational questions.
Who has a guaranteed quota? Who may borrow idle capacity? Which jobs can be preempted, and which must be admitted all at once? How long can a research team hold an allocation before a queued production workload takes precedence?
There is no universal fair answer. The unhealthy answer is leaving those choices implicit in who knows the cluster administrator, who submits first, or who has learned which node selector happens to work.
A modest policy is enough to begin:
- reserve a small, named production quota;
- let batch queues borrow unused capacity but make the recall rule explicit;
- require a runtime estimate for large training jobs, even if it is approximate;
- use priority sparingly and publish who can use it;
- reject a distributed job that cannot obtain its minimum viable topology instead of starting half of it and wasting the rest.
The last item sounds severe. It is kinder than keeping eight scarce GPUs occupied by a job that cannot make progress.
Topology is part of the request
For a single-GPU job, the scheduler can often make a perfectly boring choice. For distributed training, a GPU count without topology can be a lie of a different kind.
Four devices in the same NVLink domain and four devices spread across nodes may both satisfy 4, yet they are not the same training environment. The CPU, NIC, storage path, and fabric can all turn an apparently adequate allocation into a slow and costly run.
This is where device classes earn their keep. Make the topology assumption an explicit part of the class, document the supported job shapes, and test the failure mode. The NVIDIA DRA driver includes ComputeDomain support for coordinated multi-node NVLink workloads, while full GPU and existing MIG allocation are listed as generally available. That is useful capability, not a substitute for deciding which training workloads deserve it. NVIDIA’s operator documentation also makes clear that capability maturity varies.
Cost and observability must meet the claim
The platform should be able to answer two questions for every allocation:
- What was promised?
- What actually happened?
The first comes from the claim, queue, class, owner, and workload metadata. The second needs telemetry: allocation duration, GPU utilization, memory use, SM occupancy where relevant, queue wait time, retries, and a useful end state. A three-hour claim that drove almost no work is not necessarily waste, but it is a conversation the owning team should be able to have with data.
Do not confuse node health with allocation health. NVIDIA notes that its DRA driver’s NVML device-health check is alpha and disabled by default; its basic gRPC probe only reports kubelet-plugin availability, while DCGM supplies telemetry. That is exactly the sort of boundary an operating model needs to state plainly. A green Pod is not proof that a costly allocation was healthy or productive.
Cost attribution should use the same identifiers as the policy: class, queue, namespace, team, model or service, and claim lifetime. If finance needs a spreadsheet mapping node names back to teams every month, the allocation contract is incomplete.
Fallback needs to be intentional too
When a workload requests a class that is unavailable, the safe default is usually to wait or fail with a clear reason.
Silent downgrade is tempting. A training job requested a fabric-aware allocation; perhaps it can run on any four cards. A latency-sensitive service requested dedicated hardware; perhaps it can time-slice for a while. Sometimes that is a reasonable decision. It should be the workload owner’s decision, represented by an explicit fallback class or an application-level degradation mode, not a scheduler surprise.
Write the choices down:
- wait for the requested class;
- fall back to a named lower tier after a declared timeout;
- run a smaller, valid job shape;
- fail fast and notify the owner.
An honest Pending condition with a queue position is often more useful than a cheap allocation that makes an expensive workload fail slowly.
DRA is the grammar, not the policy
This is the slightly unglamorous conclusion. Enabling DRA is an engineering task. Defining a GPU allocation contract is a platform-design task.
The former has versions, operators, feature gates, and migration guides. The latter has trade-offs: fairness versus utilization, isolation versus density, fast experiments versus predictable production, and local autonomy versus a shared budget. No API can choose those for you.
Kubernetes 1.37 makes the bridge from extended resources more credible. NVIDIA’s operator is actively positioning DRA alongside its existing device-plugin path. Managed Kubernetes documentation is starting to surface DRA-specific operational caveats. Those are all good reasons to experiment now. They are not a reason to move every workload before the policy exists.
What I would do on Monday
- Inventory GPU workloads. List who runs them, their runtime and memory profile, whether they are latency-sensitive, and what happens when they wait.
- Define three to five device classes. Use names based on workload intent; write the isolation, topology, availability, and cost expectations beside each one.
- Choose the queue rules. Set baseline quotas, borrowing, priority, and preemption behaviour before the first contention incident chooses them for you.
- Pick one migration path. Move a low-risk workload from an extended-resource request through DRA’s compatibility bridge and validate its operational tooling.
- Measure before buying. Track queue time, allocation time, utilization, memory pressure, and failed or abandoned claims. A low GPU-utilization graph is not an excuse to buy more hardware; it is a prompt to ask a better question.
The goal is not to make GPUs feel scarce and bureaucratic. They already are scarce and expensive. The goal is to make the choices visible, reviewable, and predictable enough that the platform can support both serious production workloads and the experiments that create the next ones.
References
- Kubernetes v1.37: DRA updates
- Kubernetes v1.37 release announcement
- About Dynamic Resource Allocation in GKE
- NVIDIA GPU Operator: DRA driver for NVIDIA GPUs
- NVIDIA DRA driver: GPU allocation
- Kueue: Dynamic Resource Allocation
To test my projects, I use Railway. If you want $20 USD to get started, use this link.