My platform provisions GPU instances as Kubernetes pods: each gets a Jupyter ingress and an SSH NodePort Service. I wanted users to reach them with a plain ssh command against a public name, through Pangolin, a self-hosted tunnelling reverse proxy.

This is the story of getting there. It took three attempts, and the working answer was a small patch to the controller that mirrors Kubernetes objects into Pangolin.

Where this stands: the controller patch works and assigns ports. End-to-end public SSH also depends on gateway-side setup (published ports, Traefik entry points, a cloud firewall rule). When I last checked, port 40001 still timed out because that firewall rule wasn’t open yet. Don’t read this as a finished, fully tested pipeline.

The intended path

laptop: ssh -p <PORT> root@ssh-<instance>.example.com
   │  DNS wildcard *.example.com → gateway
   ▼
Pangolin gateway  (port <PORT> = this resource)
   ▼  TCP forward through the tunnel
Kubernetes Service  ssh-<instance>  (NodePort → pod :22)
   ▼
pod sshd :22  →  authorized_keys check

Two properties matter. DNS is a wildcard to the gateway, and SSH is routed by port, not by hostname: unlike HTTP, there’s no Host header to inspect, so each SSH resource needs its own public port. Port 22 on the gateway is the admin SSH, not a resource.

Attempt 1: Pangolin’s ssh resource mode

The natural choice. A controller (pangolin-autodns) already watches HTTP Ingresses and mirrors them into Pangolin, so I extended it to watch Services named ssh-* on port 22 and create an SSH-mode resource targeting the NodePort. RBAC needed services [get list watch patch update]. It created the resources and the logs looked right.

Connecting didn’t work. In ssh mode the client signs its key through Pangolin to get a certificate, and the auth daemon port (22123) was unreachable from outside.

Why: reading the Pangolin source at that time, the SSH-key signing path is gated behind a licence check (isLicensedOrSubscribed), so this mode appeared to be paid/cloud-only and not available in the self-hosted OSS build. I’m going from my reading of the code, so verify against the version you run.

Attempt 2: raw tcp mode — but proxyPort stays null

TCP mode is plain passthrough through the Newt tunnel and works self-hosted. Newt started the proxies fine:

Started tcp proxy to 10.10.30.160:30842

But every resource had proxyPort: null, so Traefik never exposed a public port and nothing could connect.

Why: a feature flag. On the server, proxyPort is only accepted when raw resources are allowed:

// createResource.ts / updateResource.ts
if (!config.getRawConfig().flags?.allow_raw_resources) {
    if (data.proxyPort !== undefined) return false;   // "Cannot update proxyPort"
}
// traefikConfigProvider.ts also checks traefik.allow_raw_resources

Fix, on the Pangolin server (not the cluster):

# Pangolin server config.yml
flags:
  allow_raw_resources: true
traefik:
  allow_raw_resources: true

Restart the server and recreate the resources.

Attempt 3: nobody assigns the port — so the controller must

Even with raw resources enabled, a freshly provisioned instance still got proxyPort: null, for two independent reasons:

  1. The controller’s _create_resource() sent only {name, mode, subdomain, domainId} — no port.
  2. Pangolin doesn’t pick one either; createResource.ts reads proxyPort straight from the request body.

So someone had to assign a port by hand in the dashboard for every instance. The fix is to let the controller choose one from a configured range:

SSH_PROXY_PORT_RANGE = os.environ.get("SSH_PROXY_PORT_RANGE", "40000-40100")
_ssh_port_min, _ssh_port_max = (int(x) for x in SSH_PROXY_PORT_RANGE.split("-", 1))

def _used_proxy_ports(cli):
    """Ports already taken by any resource in the org."""
    return {int(r["proxyPort"]) for r in _list_resources(cli) if r.get("proxyPort")}

def _set_proxy_port(cli, rid, port):
    """Assign a public proxyPort to a TCP resource (POST, not PATCH)."""
    r = cli.request("POST", f"/resource/{rid}", json={"proxyPort": port})
    r.raise_for_status()

def _pick_free_ssh_port(cli):
    """Lowest free port in the range."""
    used = _used_proxy_ports(cli)
    for p in range(_ssh_port_min, _ssh_port_max + 1):
        if p not in used:
            return p
    raise kopf.PermanentError(f"no free proxy port in {SSH_PROXY_PORT_RANGE}")

It’s used in two places in _ensure_ssh_mapping. For an existing resource, backfill the port if it’s missing. For a new one, assign it right after creating the target, and clean up if anything fails:

existing = _find_resource_by_fqdn(cli, fqdn)
if existing:
    rid = existing["resourceId"]
    if not existing.get("proxyPort"):
        port = _pick_free_ssh_port(cli)
        _set_proxy_port(cli, rid, port)
    return {"resourceId": rid, "fqdn": fqdn}

# new resource
rid = res["resourceId"]
try:
    _create_target(cli, rid, port=node_port)
    port = _pick_free_ssh_port(cli)
    _set_proxy_port(cli, rid, port)
except Exception:
    try: _delete_resource(cli, rid)
    except Exception: pass
    raise

Two latent bugs found on the way

TCP resources often have fullDomain: null. They carry only a subdomain. The existing lookup compared fullDomain only, so it missed them, retried the create, got 409 Conflict, and kopf raised a PermanentError forever. Add a subdomain fallback:

def _find_resource_by_fqdn(cli, fqdn):
    for res in _list_resources(cli):
        if res.get("fullDomain") == fqdn:
            return res
        if res.get("subdomain") and fqdn.startswith(res["subdomain"] + "."):
            return res
    return None

Two related fixes from the earlier version: skip the “disable SSO” call for TCP resources (it returns 400 for that mode), and make the orphan sweep count ssh-* Services as live. Otherwise it deletes the TCP resources as orphans.

The SSH target uses the ingress node’s IP rather than the node running the pod. NodePorts answer on every node, so it works, but targeting the pod’s own node would be more precise. I noted it and left it alone.

Deploying and verifying

kubectl -n pangolin-autodns create configmap pangolin-autodns-code \
  --from-file=controller.py=<patched-file> --dry-run=client -o yaml | kubectl apply -f -
kubectl -n pangolin-autodns set env deploy/pangolin-autodns SSH_PROXY_PORT_RANGE=40000-40100
kubectl -n pangolin-autodns rollout restart deploy/pangolin-autodns

# force re-processing of an existing Service
kubectl -n <ns> annotate svc ssh-<name> pangolin.example.com/last-handled-
kubectl -n <ns> annotate svc ssh-<name> pangolin.example.com/resource-id-

kubectl -n pangolin-autodns logs deploy/pangolin-autodns --tail=50 -f | grep proxyPort
# ssh-<name>.example.com: created ssh resource=<id> proxyPort=40001 -> <node-ip>:<nodeport>

What the patch doesn’t do

It only assigns the port. For it to work publicly, three gateway layers must already exist (one-time, not per instance):

  1. Docker Compose: the Gerbil container publishes the port range.
  2. Traefik static config: one tcp-<port> entry point per port. There’s no range syntax, so a 101-port range means 101 entry points. A smaller range (say 40001–40010) keeps the gateway manageable.
  3. Cloud firewall / NSG: inbound TCP allowed on that range. This was the missing piece when I last tested.

Housekeeping

  • The live ConfigMap is the source of truth for the controller code, not the git repo (mine lagged by about 85 lines). Patch the cluster, verify, then commit.
  • Back up before patching: controller.py.<timestamp>.bak.
  • A placeholder public key in a test pod’s authorized_keys (with no matching private key) wasted time. Verify the key you test with actually exists.

Takeaways

  1. A “mode” in a self-hosted tool can be quietly gated by a licence check. Read the code path before building on it.
  2. When a field stays null, find out who is supposed to set it. Here neither side was.
  3. Port-based protocols and hostname-based reverse proxies don’t mix without a port allocator.