GPU AI Workloads on Kubernetes: Scheduling, Isolation, and Cost

GPU scheduling in Kubernetes is the process of matching a Pod’s accelerator request to an accessible GPU on a node that satisfies its placement constraints. Kubernetes exposes GPUs through vendor device plugins such as nvidia.com/gpu or through Dynamic Resource Allocation (DRA); isolation, sharing, topology, and cost controls remain platform responsibilities.

Adding nvidia.com/gpu: 1 to a Pod is not a GPU platform. Drivers, container runtime integration, device plugins, labels, taints, memory behaviour, sharing mode, telemetry, and budgets must work as one control plane.

How Kubernetes discovers GPUs

Kubernetes does not discover NVIDIA hardware by itself. A compatible driver and container runtime run on the node; a vendor device-plugin DaemonSet registers devices with kubelet; kubelet publishes capacity and allocatable; the scheduler places the Pod; the runtime exposes the assigned device.

kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu,ALLOC:.status.allocatable.nvidia\.com/gpu
kubectl -n kube-system get pods | grep -E 'nvidia|gpu'
kubectl exec inference-worker -- nvidia-smi -L

A correct GPU Pod manifest

apiVersion: v1
kind: Pod
metadata:
  name: inference-worker
  labels:
    app: inference
    model: llama-70b
    cost-center: ai-platform
spec:
  restartPolicy: Never
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: accelerator
                operator: In
                values: ["nvidia-h100"]
  containers:
    - name: server
      image: registry.example.com/inference@sha256:REPLACE
      resources:
        requests:
          cpu: "8"
          memory: 64Gi
        limits:
          cpu: "16"
          memory: 96Gi
          nvidia.com/gpu: 1

Extended GPU resources belong in limits; if requests are also present they must be equal. Fractional nvidia.com/gpu values are not the sharing mechanism. Use MIG, time-slicing, vGPU, or a DRA-driver policy. Tolerations open a tainted node; affinity chooses the intended GPU class.

How to fix 0/1 nodes are available: 1 Insufficient nvidia.com/gpu

The message means the scheduler found no free allocatable NVIDIA GPU satisfying all Pod constraints. The physical card may exist while the plugin fails to publish it, another Pod owns it, affinity or taints exclude the node, or the workload asks for the wrong resource name.

kubectl describe pod <pod> | sed -n '/Events:/,$p'
kubectl describe node <node> | sed -n '/Allocated resources:/,/Events:/p'
kubectl get node <node> -o jsonpath='{.status.allocatable.nvidia\.com/gpu}'
kubectl get pods -A -o wide

Affinity, taints, and topology

Use managed accelerator labels for GPU generation and memory class, taint GPU nodes to exclude generic workloads, then combine workload toleration with required affinity. Distributed inference and training also care about NVLink, NVSwitch, PCIe, NUMA, NIC locality, and RDMA—not just GPU count.

GPU sharing: MIG or time-slicing?

MIG provides stronger compute and memory partitioning on supported hardware. Time-slicing can improve occupancy but one virtual request is not a full-GPU performance guarantee. Choose from model VRAM, p99, tenant boundary, burst behaviour, and fragmentation measurements.

DRA for GPU allocation

DRA requests devices through DeviceClass and ResourceClaim rather than only numeric extended resources. It first appeared in Kubernetes 1.30; major APIs advanced in 1.34, and the core feature is documented stable since 1.35. Verify the installed DRA driver and any optional device features separately.

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: inference-gpu
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: gpu.nvidia.com
            count: 1
---
apiVersion: v1
kind: Pod
metadata:
  name: dra-inference
spec:
  resourceClaims:
    - name: gpu
      resourceClaimTemplateName: inference-gpu
  containers:
    - name: model
      image: registry.example.com/model@sha256:REPLACE
      resources:
        claims:
          - name: gpu

The DeviceClass name is driver-specific. Do not copy the placeholder without inspecting the cluster’s actual classes. Extended resources remain valid; migrate when feature selection, claim lifecycle, or sharing needs justify DRA.

Separate inference, batch, training, and notebooks

One queue cannot optimize every objective. Prevent online traffic from starving training and training from consuming every GPU. Preemption without checkpoint awareness can destroy hours of compute.

Isolation and device health

Device visibility is not tenant isolation. Apply Pod Security, seccomp/AppArmor, read-only filesystems, restricted hostPath/privileged access, least-privilege service accounts, NetworkPolicy, secret boundaries, and image provenance. Treat GPU drivers and privileged device plugins as supply-chain-critical components.

XID, ECC, reset, and thermal events need a node-health runbook. Remove unhealthy devices from allocatable, drain safely, and prevent restart loops onto the same failing node. Rescheduling uncheckpointed training may multiply cost.

GPU cost mathematics

allocated_gpu_hours = sum(gpu_count * allocation_seconds / 3600)
useful_gpu_hours = allocated_gpu_hours * average_SM_active_ratio
idle_cost = allocated_gpu_hours * hourly_rate * (1 - useful_ratio)
cost_per_1k_requests = total_inference_cost / successful_requests * 1000

Illustration: four GPUs allocated for 24 hours at a 2.50-dollar hourly GPU equivalent cost 240 dollars. At 35 percent useful utilization, the underutilized share is about 156 dollars. Replace every assumption with your provider, commitment, node, CPU/RAM, and measured utilization data.

Prometheus and DCGM exporter

NVIDIA DCGM exporter exposes selected DCGM fields at /metrics and can map GPU samples to Kubernetes pod, namespace, and container labels. Configuring a field does not guarantee hardware support; verify the exporter endpoint, DCGM field semantics, units, versions, and entity level.

Do not terminate a model server merely because utilization is low; idle periods can be normal. Correlate GPU signals with queue depth, request rate, latency, readiness, and allocation.

Autoscaling and cold starts

GPU utilization alone is a late signal. Scale online inference from queue depth, in-flight work, tokens per second, p95 latency, and model readiness. Node boot, driver readiness, image pull, and model load form the cold-start budget. Use stabilization windows and minimum warm replicas to avoid oscillation.

Distributed topology and gang scheduling

Eight GPUs are not interchangeable when tensor parallelism depends on NVLink/NVSwitch, PCIe, NUMA, and network locality. Admit all required replicas together; otherwise early Pods reserve expensive GPUs while waiting for peers. Kueue or a compatible batch scheduler can provide quota and gang semantics, but validate topology behaviour for your versions.

Production checklist

1. Pin driver, toolkit, plugin, Kubernetes, and image versions.
2. Validate capacity, allocatable, and health.
3. Use managed labels, taints, tolerations, and affinity.
4. Separate online, batch, training, and dev queues.
5. Benchmark MIG or time-slicing with VRAM, p99, and isolation.
6. Verify DRA classes, claims, drivers, and feature state.
7. Test DCGM Pod/MIG/DRA mapping and metric units.
8. Limit GPU, CPU, RAM, storage, network, and tenant quota.
9. Enforce Pod Security, RBAC, NetworkPolicy, secrets, and provenance.
10. Allocate cost by job, model, tenant, and cost centre.
11. Runbook XID, ECC, thermal, checkpoint, and preemption failures.

Conclusion

Reliable Kubernetes GPU operations combine discovery, placement, sharing, topology, queue admission, telemetry, isolation, and cost accounting. A GPU limit in YAML is only the first input to that system.

Continue with event-driven LLM workers, multi-agent failure isolation, and the MCP server guide.