Making a GPU Visible to Kubernetes: GPU Operator, DCGM, and Two Wrong Values

A GPU node had joined an RKE2 cluster correctly, yet GPU pods sat Pending and the platform on top reported zero GPU slots. The GPU and its driver were fine. They simply weren’t published to Kubernetes.

Setup: an RKE2 cluster managed by Rancher, one bare-metal arm64 GPU node with an NVIDIA GB10 (driver 580.142, installed on the host), and three control-plane VMs.

Two gaps, one symptom

1. No device plugin. The node’s allocatable resources had no nvidia.com/gpu:

kubectl get node gx10-726d -o jsonpath='{.status.allocatable}'
{"cpu":"20","ephemeral-storage":"933678855051","memory":"127535912Ki","pods":"110"}

There was no device-plugin DaemonSet anywhere, no nvidia.com API resources and no GPU Operator Helm release. In Prometheus, kube_node_status_allocatable{resource="nvidia_com_gpu"} returned zero series. The platform’s own gate — available slots computed from schedulable GPUs — therefore read 0.

2. DCGM running, but never scraped. A manually installed dcgm-exporter chart was happily reading the GPU, but it had been installed with serviceMonitor.enabled = false. The live Prometheus config had 19 scrape jobs and none of them was DCGM. A ServiceMonitor someone had written “wasn’t detected” for a simple reason: it had never been applied (NotFound).

Decision: copy what worked on the old cluster

Rather than reconstruct a configuration, I read back the one the previous cluster already ran. That cluster showed unavailable in Rancher, but Rancher’s proxy still served its cached objects, so the real settings were recoverable.

That relabelling matters more than it looks: the node-name labels the platform expects come from the operator’s ServiceMonitor, not from the exporter itself.

The install, in order

Order matters. Remove the manual exporter before the operator arrives, so the operator’s exporter is the only DCGM source.

1. Add the chart repo in Rancher (Apps → Repositories → nvidia-ngc, https://helm.ngc.nvidia.com/nvidia).
2. Remove the manual dcgm-exporter App. It was installed as a Rancher App, so delete it as one. A raw helm uninstall just gets reconciled back.
3. Install the operator (gpu-operator v26.3.2, namespace gpu-operator) with the values file below.
4. Verify the ClusterPolicy, node labels, allocatable GPUs, Prometheus targets and series.
5. Run a GPU pod and call nvidia-smi.

helm upgrade --install gpu-operator nvidia-ngc/gpu-operator \
  --version v26.3.2 -n gpu-operator --create-namespace \
  -f values-gpu-operator.yaml --wait --timeout 20m

Verify your Helm values by rendering them

Helm silently ignores keys it doesn’t recognise, so a typo looks exactly like success. I rendered the values through the real chart and asserted each key (20 of 20 passed):

helm template gpu-operator ./gpu-operator-v26.3.2 -n gpu-operator -f values-gpu-operator.yaml

It paid off immediately: a draft line validator.enabled: true turned out not to be a real key in this chart version and was being ignored. It’s gone now.

Two values I got wrong

Both survived the render check and failed on the live cluster. Rendering proves a key exists, not that the setting is sane.

1. Time-slicing with replicas: 1 is invalid. The device plugin refuses to start: error starting plugins: … number of replicas must be >= 2. It crash-looped, so no GPU was ever advertised, and the ClusterPolicy showed errors for the device plugin, the DCGM exporter and GPU feature discovery. The fix was to drop the sharing config entirely. The plugin’s default is exactly one device per physical GPU, which is what I wanted.

2. dcgmExporter.hostNetwork: true was missing, even though the old cluster’s configuration had it. Without it Prometheus scraped the pod IP and timed out — not because of the exporter, but because of a pod-network fault on that node (covered in When Calico Reads a Different Object Than the One You Edited). Restoring it brought the scrape up.

Why one device per GPU, not time-sliced

The platform computes capacity from two sources: total GPUs from DCGM (physical devices) and allocated GPUs from Kubernetes pod requests (pods). With time-slicing above 1, that arithmetic breaks: after the first instance it would report zero free slots. Sharing would need at least two slices and a change to the platform’s capacity calculation. That’s why the install uses no sharing at all.

Result

ClusterPolicy : ready | Ready=True
node          : nvidia.com/gpu.product=NVIDIA-GB10  .count=1  driver=580.142
allocatable   : nvidia.com/gpu = 1
Prometheus    : DCGM target health=up, DCGM_FI_DEV_GPU_UTIL/TEMP/POWER_USAGE present
platform app  : HEALTHY, 1 GPU discovered
GPU pod       : nvidia-smi → NVIDIA GB10, CUDA 13.0

What’s still not right

• VRAM reads 0. The GB10 exports no framebuffer metrics, and nvidia-smi itself reports Memory-Usage: Not Supported on this unified-memory device. The “24 GB” the UI shows is a hardcoded fallback in the app, not a measurement.
• Cinder CSI is broken on this node. Its node plugin had 249 restarts: it tries to read instance metadata from 169.254.169.254, which doesn’t exist on bare metal. Any pod needing a Cinder volume can’t attach there.
• Ingress controller restarts. The ingress controller on the same node had 364 restarts (exit 143). I haven’t investigated.
• Version drift. Chart v26.3.2 was proven on Kubernetes 1.34.6; this cluster runs 1.35.8. The chart metadata allows it and it worked here, but I wouldn’t call it broadly tested. The fallback is the next chart release.

Lessons

1. If one component “exists but nothing happens”, check whether anything is consuming it (a ServiceMonitor, a device plugin) before suspecting the component.
2. Copying a known-good configuration beats reconstructing one, even if its source cluster is half-dead.
3. A render check proves your keys exist. Only the live cluster proves your values are valid.