When Calico Reads a Different Object Than the One You Edited

The GPU node joined the cluster correctly. The GPU Operator installed cleanly, the device plugin advertised the card, DCGM metrics reached Prometheus, and nvidia-smi ran inside a pod.

Pod-to-pod networking across that node was still dead.

This is the story of tracking it down — hop by hop, then protocol by protocol — and the one object-level mistake that cost me three attempts after the root cause was already found.

Symptom

A pod on the GPU node could not reach any pod outside it. DNS failed. Traffic in the other direction failed too. Meanwhile:

• kubectl get nodes — all Ready
• TCP to the API server (6443), join port (9345), kubelet (10250) — fine
• GPU workloads — reporting healthy metrics via the same network path

So: TCP worked, the node was healthy, and only the overlay was broken. That combination keeps pointing you at the CNI, and it is right — but the reason was not in the CNI’s configuration.

Step 1 — Which direction is broken?

VXLAN is UDP. If UDP is being dropped, the overlay dies while everything else survives. Two captures, same path:

1) GPU node egress:  VXLAN DOES leave correctly
     IP 10.10.10.62.53305 > 10.10.60.127.4789: VXLAN, vni 4096
     IP 10.42.182.1 > 10.42.168.210: ICMP echo request

2) Control-plane capture, filtered to the GPU node:  VXLAN NEVER ARRIVES
     zero `udp dst port 4789` from 10.10.10.62
     but TCP flows fine: 10.10.10.62 -> 10.10.60.209:6443 / :9345 / :10250

A controlled test made it undeniable — a listener on the control-plane node, a sender on the GPU node:

TCP 14790  -> DELIVERED
UDP 14789  -> NOT DELIVERED

Cause A: UDP is blocked in that direction. VXLAN is UDP-only, so this alone breaks GPU → cluster.

Step 2 — And the other direction?

Inbound VXLAN was delivered, but it arrived NAT-rewritten, and Calico discards it:

GPU node:  ipset cali40all-vxlan-net = {10.10.60.209, 10.10.60.127, 10.10.60.232}
           iptables -L cali-INPUT -n -v
             0    0 ACCEPT  "Allow IPv4 VXLAN packets from allowed hosts"
           579 63690 DROP   "Drop IPv4 VXLAN packets from non-allowed hosts"

CP node:   ipset = {10.10.60.127, 10.10.60.232, 10.10.10.62}
           ACCEPT 65M packets / DROP 0

Inbound VXLAN reached the GPU node with source 10.10.10.244 — a router — instead of the real 10.10.60.x. Calico filters VXLAN by real node address, so 579 packets were silently dropped.

Cause B: the packets arrive with the wrong source address. Both causes live outside the cluster, which is why this started as a network ticket.

Step 3 — Why the obvious fix was impossible

The obvious fix: move the GPU node onto the cluster’s own network. That dead-ends, and the dead end is the most informative finding in the whole investigation.

openstack network show vm60 / ext-net   -> provider:network_type = None   (TENANT networks)
neutron ML2: tenant_network_types = vxlan        <-- tenant networks are VXLAN
             tunnel_types = vxlan, local_ip = 10.10.10.14
             bridge_mappings = physnet1:br-ex, physnet2:br-vm60
compute01:   enp94s0f0np0 = 10.10.70.14/24 (10G, VLAN70); br-vm60 DOWN

vm60 (10.10.60.0/24) is a VXLAN tenant network encapsulated inside OpenStack’s OVS. Its gateway 10.10.60.1 is a Neutron router tenant port that exists only in namespaces — no switch port can ever be a member of that L2. VLAN tagging on the physical switch cannot reach it. The cluster’s network does not exist on the wire.

Step 4 — The decisive experiment

One path, two protocols, two different NAT treatments:

UDP  CP -> GPU :  10.10.10.244.10051 > 10.10.10.62.14777    UDP     <-- SNAT'd (router external IP)
IPIP CP -> GPU :  10.10.60.209      > 10.10.10.62          proto 4  <-- REAL source, NO SNAT

Same route, same router. Only UDP gets rewritten.

So the blocker was never simply “UDP is blocked”. Opening UDP fixed reachability but not acceptance — the packets had to arrive with the real node address, and protocol 4 preserves it.

Evidence before changing anything, using throwaway tunnels over the real path:

GPU -> CP : 20/20 packets, 0% loss
CP -> GPU : 20/20 packets, 0% loss
3 rounds  => 120/120, ~1 ms latency
1300-byte payload through the tunnel: OK
tunnel MTU 1480 (underlay 1500 - 20)

Step 5 — The fix, and the three attempts it took

The change is conceptually one field: switch the Calico IP pool encapsulation from VXLAN to IPIP.

Attempt 1 — patched Installation.spec.calicoNetwork.ipPools[0].encapsulation: IPIP. The operator rejected it:

"Invalid Installation provided"   reason=InvalidConfigurationError
error: "with BIRD cluster routing mode, IPIP encapsulation requires that BGP is enabled"

Attempt 2 — added clusterRoutingMode: Felix so Felix owns the IPIP routes. The Installation reported Degraded=False — accepted. The data plane did not change.

Attempt 3 — patched the IPPool CR directly (ipipMode: Always, vxlanMode: Never). This worked.

Why attempts 1 and 2 failed

The rule, from tigera-operator v1.42.3 pkg/controller/installation/validation.go:

felixOwnsIPIPRoutes := clusterRoutingMode(install) in {Felix, FelixIPIPOnly}
if !felixOwnsIPIPRoutes && (BGP == nil || BGP == Disabled) {
    error("IPIP requires BGP")
}

With clusterRoutingMode unset, this operator resolves the default to BIRD — the FelixIPIPOnly default only arrives in Calico v3.33. On calico-node v3.32.1 / operator v1.42.3, an IPIP pool without BGP is invalid.

Attempt 2 made the config legal but changed nothing, because RKE2-managed Calico has no Calico v3 API server — v3.projectcalico.org APIService is absent. The operator only updates existing IP pools through that API:

controller_ippool: "Unable to modify IP pools while Calico API server is unavailable"

So the Installation looked correct and Felix never changed. Worse: a rejected config is not partially applied — the operator stops reconciling entirely, so the IPPool stayed on VXLAN while the Installation claimed IPIP.

Installation ipPools[0].encapsulation = IPIP     <- what I edited
ippool CR:   ipipMode = Never   vxlanMode = Always   <- what Felix actually obeys

Verification on the config alone would have declared victory. The effective-mode check is what caught it.

Do not use a merge patch on ipPools. It is a list, so a merge patch replaces the whole array and silently drops cidr, blockSize, natOutgoing, nodeSelector and allowedUses. A JSON patch targets one field, and the dry-run must be read back to prove the other fields survived.

Verified result

After the IPPool patch:

• tunl0 UP on all 4 nodes
• Routes became 10.42.x/26 via <peer> dev tunl0 mtu 1480 (previously via vxlan.calico mtu 1450)
• Pod MTU rose 1450 → 1480
• Pod → web / redis / ClusterIP / prometheus all reachable from a pod on the GPU node
• DNS resolves (kubernetes.default -> 10.43.0.1)
• A control-plane pod reaches a pod on the GPU node
• DCGM scrape still up, all production pods Running

Rollback is the same JSON patch back to VXLAN.

The lesson that cost three attempts

A controller reads one object; the data plane reads another.

The tigera-operator reconciles the Installation CR. Felix reads the IPPool CR. On RKE2, the operator has no Calico API server, so it silently cannot update IP pools — the Installation looks right while the data plane is untouched.

Verify the object the data plane consumes, not the object you edited. kubectl get ippools.crd.projectcalico.org -o yaml is the source of truth for encapsulation mode; the Installation is only the intent.

And the wider one: when TCP works and only the overlay fails, the problem is rarely in the overlay. Measure at the wire, characterise what actually arrives, and question every assumption the topology implies — including “there is a network there”.