Onboarding a Team onto a Shared GPU Kubernetes Cluster: Tiers, Taints and Local NVMe Storage

When a new team gets access to a shared GPU cluster, the first week usually disappears into the same questions: how do I connect, which GPU is which, why is my pod Pending, and where do my files go?

This is the onboarding guide I wrote to answer those up front. It covers connecting to the cluster, setting a default namespace, the three GPU tiers, the rules every GPU workload must follow, and three templates you can copy. In the examples the team’s namespace is ml-team; use your own.

1. The cluster at a glance

The cluster hides the physical hardware behind three profiles:

Profile	Hardware	Arch	GPU memory	Node selector	`NVIDIA_VISIBLE_DEVICES`	StorageClass
GPU 1	NVIDIA GB10	`arm64`	Unified, high-speed	`gpu.tier/gpu-1: "true"`	`"all"` (or `"0"`)	`gpu-local`
GPU 2	NVIDIA RTX PRO 6000	`amd64`	96 GB	`gpu.tier/gpu-2: "true"`	`"0"`	`rtx-local`
GPU 3	NVIDIA RTX A4000	`amd64`	16 GB	`gpu.tier/gpu-3: "true"`	`"1"`	`rtx-local`

Pick the tier by workload: GPU 2 for large models and high-VRAM training, GPU 3 for lightweight inference and vision, GPU 1 when you need the ARM64 node.

2. Step 1: connect

Save your kubeconfig and point kubectl at it:

export KUBECONFIG=~/<kubeconfig>.yaml

Set your namespace as the default, so you don’t type -n ml-team on every command:

kubectl config set-context --current --namespace=ml-team

Verify:

kubectl get namespace ml-team        # you can see your namespace
kubectl get pods,pvc                 # what already runs in it
kubectl get nodes -L gpu-tier        # nodes and their tier labels

3. Step 2: the four deployment rules

GPU pods in the namespace must follow these rules or they won’t schedule or won’t see a GPU:

1. Tolerate the GPU taint. GPU nodes are tainted so ordinary CPU workloads stay off them. Your pod must tolerate it:

tolerations:
  - key: "nvidia.com/gpu"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

2. Use the NVIDIA runtime class.

runtimeClassName: nvidia

3. Match the StorageClass to the tier. GPU 1 uses gpu-local; GPU 2 and GPU 3 use rtx-local.

4. Keep replicas on the same host. If you scale to more than one replica, add podAffinity so they stay on one physical machine:

affinity:
  podAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app
              operator: In
              values: ["<your-app-name>"]
        topologyKey: "kubernetes.io/hostname"

4. Production templates

Each template is a PersistentVolumeClaim for dedicated local NVMe storage plus a Deployment pinned to a tier.

Template 1: high-memory workload (GPU 2, RTX PRO 6000, 96 GB)

For large language models, deep-learning training and other high-VRAM jobs. Save as rtx6000-workload.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rtx6000-pvc
  namespace: ml-team
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: rtx-local
  resources:
    requests:
      storage: 50Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rtx6000-app
  namespace: ml-team
  labels:
    app: rtx6000-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rtx6000-app
  template:
    metadata:
      labels:
        app: rtx6000-app
    spec:
      runtimeClassName: nvidia
      nodeSelector:
        gpu.tier/gpu-2: "true"
        kubernetes.io/arch: "amd64"
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values: ["rtx6000-app"]
              topologyKey: "kubernetes.io/hostname"
      containers:
        - name: worker
          image: ubuntu:24.04
          command: ["/bin/bash", "-c"]
          args:
            - |
              echo "RTX 6000 workload initialised at $(date)" > /workspace/init.log
              sleep infinity
          env:
            - name: NVIDIA_VISIBLE_DEVICES
              value: "0"                    # the dedicated 96 GB card
            - name: NVIDIA_DRIVER_CAPABILITIES
              value: "compute,utility"
          volumeMounts:
            - name: storage
              mountPath: /workspace
      volumes:
        - name: storage
          persistentVolumeClaim:
            claimName: rtx6000-pvc

Template 2: standard workload (GPU 3, RTX A4000, 16 GB)

For lightweight inference, computer vision and medium models. Save as rtxa4000-workload.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: a4000-pvc
  namespace: ml-team
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: rtx-local
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: a4000-app
  namespace: ml-team
  labels:
    app: a4000-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: a4000-app
  template:
    metadata:
      labels:
        app: a4000-app
    spec:
      runtimeClassName: nvidia
      nodeSelector:
        gpu.tier/gpu-3: "true"
        kubernetes.io/arch: "amd64"
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values: ["a4000-app"]
              topologyKey: "kubernetes.io/hostname"
      containers:
        - name: worker
          image: ubuntu:24.04
          command: ["sleep", "infinity"]
          env:
            - name: NVIDIA_VISIBLE_DEVICES
              value: "1"                    # the dedicated 16 GB card
            - name: NVIDIA_DRIVER_CAPABILITIES
              value: "compute,utility"
          volumeMounts:
            - name: storage
              mountPath: /workspace
      volumes:
        - name: storage
          persistentVolumeClaim:
            claimName: a4000-pvc

Template 3: ARM64 workload (GPU 1, NVIDIA GB10)

For the ARM64 node. Use a multi-arch image. Save as gb10-workload.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gb10-pvc
  namespace: ml-team
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: gpu-local
  resources:
    requests:
      storage: 50Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gb10-app
  namespace: ml-team
  labels:
    app: gb10-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gb10-app
  template:
    metadata:
      labels:
        app: gb10-app
    spec:
      runtimeClassName: nvidia
      nodeSelector:
        gpu.tier/gpu-1: "true"
        kubernetes.io/arch: "arm64"
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values: ["gb10-app"]
              topologyKey: "kubernetes.io/hostname"
      containers:
        - name: worker
          image: nvcr.io/nvidia/pytorch:25.01-py3     # multi-arch image
          command: ["sleep", "infinity"]
          env:
            - name: NVIDIA_VISIBLE_DEVICES
              value: "all"
            - name: NVIDIA_DRIVER_CAPABILITIES
              value: "compute,utility"
          volumeMounts:
            - name: storage
              mountPath: /workspace
      volumes:
        - name: storage
          persistentVolumeClaim:
            claimName: gb10-pvc

I rebuilt the YAML indentation from a flattened copy, so run kubectl apply --dry-run=server -f <file> before your first real apply.

5. Step 3: deploy, verify, operate

Deploy:

kubectl apply -f rtx6000-workload.yaml

Check scheduling and storage:

kubectl get pods -o wide      # which physical node did it land on?
kubectl get pvc               # should be Bound

Expected: the pod is Running and the PVC is Bound, with the volume created under the node’s local path (for the RTX StorageClass, /var/lib/rtx-local/).

Confirm the GPU is visible inside the pod:

kubectl exec -it deployment/rtx6000-app -- nvidia-smi

For GPU 2 you should see the RTX PRO 6000 with roughly 97,887 MiB of memory.

Open a shell:

kubectl exec -it deployment/rtx6000-app -- /bin/bash

Scale (and check co-location):

kubectl scale deployment/rtx6000-app --replicas=2
kubectl get pods -o wide      # both replicas on the same node, thanks to podAffinity

Clean up when the job is done:

kubectl delete -f rtx6000-workload.yaml

Deleting the PVC releases and prunes the local NVMe directory on the host. Treat “delete the PVC” as “delete the data” — copy results out first.

Things to know before you rely on this

These templates make GPU placement a convention, not an enforcement, and it’s worth being clear about the difference:

Pods don’t request a GPU resource. They choose a card with NVIDIA_VISIBLE_DEVICES and a node with a label. The scheduler therefore doesn’t know the card is “taken”, so two teams’ pods can land on the same GPU and share it with no memory limit.
Local storage ties data to one node. A ReadWriteOnce local volume lives on the host’s NVMe. If that node is down or drained, your data is unavailable until it’s back, and the PVC doesn’t follow your pod elsewhere.
The tiers are labels, so the cluster trusts you to use them. Nothing stops a pod from selecting the wrong tier except review.

If you need real isolation (a hard memory cap per team, or guaranteed exclusive use), look at DRA-style allocation instead; I wrote up how that works on this same mix of hardware in NVIDIA DRA on a Mixed Cluster, and how the GPUs became schedulable in the first place in Making a GPU Visible to Kubernetes.

For getting a first workload running on day one, though, the conventions above are simple, and they work.