The symptom is always the same: Rancher’s UI marks a downstream cluster unavailable with “Cluster agent is not connected.” The apps and storage on that cluster are fine — Kubernetes itself is unaffected. Only the connection back to Rancher is broken.

It happened to me three times, each after something restarted:

  • 2026-07-05: Rancher upgrade (v2.13.3 → v2.14.1)
  • 2026-08-10: the hypervisor cluster rebooted, restarting the downstream nodes
  • 2026-09-08: a node ran out of memory and rebooted

The first time I deleted and re-imported the cluster, which changes the management ID and causes downtime. The fix below avoids that entirely.

What’s actually wrong

cattle-cluster-agent is in CrashLoopBackOff. Its Deployment holds stale configuration, and there are up to three separate reasons:

  1. Stale CATTLE_CA_CHECKSUM. Rancher rotated its CA certificate, and the agent now rejects the new one: cacerts checksum (<new>) does not match given --ca-checksum (<old>)
  2. Stale credential secret. The token in cattle-credentials-* is no longer active on Rancher: Failed to connect to proxy. Response status: 400 ... cluster not found
  3. The agent pod is stuck on an unhealthy node. After a VM reboot a node’s kubelet isn’t fully ready (in my case, local etcd still initialising), so the replacement agent pod sits in ContainerCreating.

Patching the checksum alone is not enough. On the third occurrence that was the trap: the credential token had to be fixed too.

The fix

1. Get the active CA checksum from Rancher. It’s the SHA-256 of the CA value including a trailing newline:

curl -sk https://<rancher-host>/v3/settings/cacerts | jq -r .value | sha256sum

(That one-liner is my rendering of “SHA-256 of value plus newline”. Check it against the checksum in Rancher’s own registration command for the cluster before relying on it.)

2. Patch the Deployment with the active checksum:

kubectl -n cattle-system patch deployment cattle-cluster-agent --type=strategic -p \
  '{"spec":{"template":{"spec":{"containers":[{"name":"cluster-register","env":[{"name":"CATTLE_CA_CHECKSUM","value":"<new-checksum>"}]}]}}}}'

3. Make the credential secret active (important). Compare the token inside cattle-credentials-<name> with the active one from GET /v3/clusters/<id>/clusterregistrationtokens. If they differ, patch data.token with the active token, base64-encoded:

kubectl -n cattle-system patch secret cattle-credentials-<name> --type=strategic \
  -p '{"data":{"token":"<base64-of-active-token>"}}'

Once the secret is correct, Rancher’s controller reconciles itself: it creates a fresh secret, rolls out the Deployment and the cluster goes Connected on its own.

4. If a node is unhealthy, cordon it so the agent starts somewhere healthy, then uncordon when it recovers:

kubectl cordon <slow-node>                                   # agent reschedules elsewhere
kubectl delete pod -n cattle-system -l app=cattle-cluster-agent
kubectl uncordon <slow-node>                                 # once the node is Ready

Verify

kubectl -n cattle-system get pods | grep cluster-agent        # Running, 2/2 ready
curl -sk -u "$RANCHER_ACCESS_KEY:$RANCHER_SECRET_KEY" \
  https://<rancher-host>/v3/clusters/<cluster-id> | jq .state # "active"

You want state: active, Connected=True and Ready=True.

Notes for next time

  • It recurs whenever Rancher rotates its CA or the downstream nodes restart, so keep this as a runbook rather than a one-off.
  • Delete-and-re-import works, but it changes the cluster’s management ID and causes downtime. The patch approach has neither cost.
  • The pattern across all three incidents: the agent is configured with a snapshot of the CA checksum and a registration token. Anything that changes either one on the Rancher side leaves the agent holding stale values until you patch them.