An Autonomous IT Team with a Hard Approval Gate: Two Agents, Signed Commands and a Free Stack

Giving an AI agent shell access to production is a short sentence with a long list of ways to go wrong. The version I’m interested in isn’t “let the agent fix things.” It’s: let agents do the tedious 90% — watching, triaging, diagnosing, drafting the fix — and make a human approve the exact commands before anything runs.

This is the blueprint I wrote for that: an operations team of two agents, split into read-only subagents, with a hard approval gate in the middle, built entirely from free and open-source parts.

What this is, and isn’t. It’s a design with a skeleton implementation, not a deployed system. The code below is an architecture sketch that I haven’t run end to end. Library APIs, especially for LangGraph’s checkpointers, change between versions, so check the current documentation before copying anything. Free tiers and licences also change; verify them too.

The architecture

Infrastructure & monitoring
        │ metrics, alerts
        ▼
┌─ Agent 1: L1 Operations ─────────────────┐
│  Health Check   (read-only)              │
│  Triage         (dedupe, severity P1–P4) │
│  Comms          (ticket + email)         │
└──────────────┬───────────────────────────┘
               │ ticket + handoff
               ▼
┌─ Agent 2: L2/L3 Engineering ─────────────┐
│  RCA        (logs, config diff)          │
│  Runbook    (fix, blast radius, rollback)│
│  Execution  (locked write access)        │
│  Reporting  (post-mortem, audit log)     │
└───────┬───────────────────────▲──────────┘
        │ request approval      │ approved + scoped token
        ▼                       │
   ┌─ Human approval gate ──────┴─┐
   │ approve · modify · reject    │
   └──────────────────────────────┘

Agent 1 watches and raises tickets. Agent 2 investigates and proposes. The human decides. Only one subagent in the whole system can write to infrastructure, and it only wakes up with an approval in hand.

Why subagents: least privilege, enforced

Splitting each agent into specialised subagents isn’t about neatness. It’s how you make least privilege real.

Monitoring and diagnosis never hold write credentials. That means a prompt-injected log line, a hallucinated plan, or a buggy tool in those subagents has a blast radius of “wrote a wrong ticket.”

The workflow

1. Detect and triage (Agent 1). The Health Check subagent polls CPU, memory, disk, container state and error logs, or reacts to an alert webhook. The Triage subagent correlates recent alerts so one incident doesn’t become ten tickets, and assigns a severity from P1 to P4.
2. Ticket and hand off (Agent 1 → Agent 2). The Comms subagent opens a ticket with raw metrics, timestamps and affected resource IDs, emails the human, and triggers Agent 2 through a shared state queue.
3. Investigate (Agent 2, read-only). The RCA subagent digs deeper: recent deployments (git diff), dependency graphs, configuration drift, stack traces. The Runbook subagent then writes the proposed fix: the exact CLI/API commands, a blast-radius assessment and a rollback plan.
4. Approval gate (human). The orchestrator saves the workflow state and pauses. The human gets the RCA summary and the exact script, and chooses: approve (unlocks a temporary, scoped token for the Execution subagent), modify (loops back to the Runbook subagent with comments), or reject (aborts, logs, escalates to manual handling).
5. Execute and verify (Agent 2 + Agent 1). The Execution subagent runs exactly the approved commands, streaming output; on error it runs the pre-approved rollback. Then Agent 1’s Health Check re-runs baseline diagnostics, and the Reporting subagent closes the ticket with the full RCA, logs and verification.

One shared state object

Every agent reads and writes a single structured record, persisted in SQLite or PostgreSQL. The approval request is just a snapshot of it:

{
  "incident_id": "INC-2026-0842",
  "status": "AWAITING_HUMAN_APPROVAL",
  "agent_1_output": {
    "severity": "P2",
    "affected_service": "payment-gateway-prod-02",
    "symptom": "Memory utilization sustained above 92% for 10 minutes",
    "ticket_id": "OPS-104"
  },
  "agent_2_rca": {
    "root_cause": "Memory leak in worker container v2.4.1 introduced in last deployment",
    "proposed_action": "Rolling restart and rollback image to v2.4.0",
    "commands_to_run": [
      "kubectl rollout undo deployment/payment-gateway -n prod",
      "kubectl rollout status deployment/payment-gateway -n prod"
    ],
    "command_sha256": "<sha256 of commands_to_run>",
    "rollback_plan": "Re-apply manifest v2.4.1 and shift traffic to secondary cluster",
    "estimated_downtime": "0 seconds (rolling update)"
  },
  "human_review": { "approved": null, "comments": null, "approval_timestamp": null }
}

(The names and values here are illustrative.) Note command_sha256 — it matters in the guardrails below.

Three guardrails that carry the weight

A prompt that says “don’t change anything without approval” is a request, not a control. These make it a control:

1. Hard credential separation. Bind the approval directly to an SSH/IAM wrapper, so the Execution subagent cannot authenticate to production until the approval flag is true. The agent isn’t trusted to wait; it’s unable to proceed.
2. Command hashing (tamper protection). When Agent 2 sends the approval request, compute a SHA-256 of commands_to_run. Before executing, recompute and compare. If a single flag changed between what you approved and what’s about to run — by hallucination, a bug, or an attacker — the hashes differ and nothing executes.
3. Approval timeouts and stale-state checks. If a request sits pending for more than 30 minutes, make Agent 1 re-verify the infrastructure before Agent 2 executes. The fix you approved was written for the world as it was, not as it is now.

Free orchestration options

You don’t need a paid platform. The four I looked at:

Licences are as listed in my notes; verify them before depending on any of them.

A free end-to-end stack

The monitoring and Telegram layers are the ones I’ve actually built pieces of: see Alerts That Reach You, and for keeping the model credentials off the agents entirely, AI Gateway.

The skeleton, in LangGraph

This shows the shape: Agent 1, then Agent 2’s read-only plan, then a hard stop at the human gate, then execution. State is saved to a local SQLite file while it waits.

from typing import TypedDict, List, Optional
from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.sqlite import SqliteSaver
from langgraph.types import interrupt

class IncidentState(TypedDict):
    incident_id: str
    symptom: str
    ticket_id: Optional[str]
    rca_summary: Optional[str]
    commands_to_run: List[str]
    human_approved: Optional[bool]
    human_comments: Optional[str]
    execution_logs: Optional[str]

def agent_1_l1_supervisor(state: IncidentState):
    # Health Check -> Triage -> Comms (ticket + email)
    return {"ticket_id": "INC-101"}

def agent_2_rca_and_runbook(state: IncidentState):
    # RCA -> Runbook (read-only)
    return {
        "rca_summary": "Nginx worker connections exhausted due to a stuck upstream.",
        "commands_to_run": ["systemctl restart nginx", "systemctl status nginx"],
    }

def human_approval_gate(state: IncidentState):
    # Pauses and saves state until a human responds
    decision = interrupt({
        "question": "Approve execution of these commands?",
        "ticket_id": state["ticket_id"],
        "rca": state["rca_summary"],
        "commands": state["commands_to_run"],
    })
    return {"human_approved": decision.get("approved"),
            "human_comments": decision.get("comments", "")}

def agent_2_execute_and_report(state: IncidentState):
    if not state.get("human_approved"):
        return {"execution_logs": "Aborted by reviewer."}
    # Execution subagent runs ONLY the approved commands, then Agent 1 re-checks health
    return {"execution_logs": "Commands executed. Service healthy."}

builder = StateGraph(IncidentState)
builder.add_node("agent_1_l1", agent_1_l1_supervisor)
builder.add_node("agent_2_plan", agent_2_rca_and_runbook)
builder.add_node("human_gate", human_approval_gate)
builder.add_node("agent_2_execute", agent_2_execute_and_report)
builder.add_edge(START, "agent_1_l1")
builder.add_edge("agent_1_l1", "agent_2_plan")
builder.add_edge("agent_2_plan", "human_gate")
builder.add_edge("human_gate", "agent_2_execute")
builder.add_edge("agent_2_execute", END)

graph = builder.compile(checkpointer=SqliteSaver.from_conn_string("incidents.sqlite"))

Two things this skeleton deliberately leaves out, because they’re the hard parts: the hash comparison between approval and execution, and the credential wrapper that makes the Execution step unable to act without approval. Without those two, the interrupt is a polite suggestion.

What I’d want proven before trusting it

• Test the gate adversarially. Feed the Runbook subagent a ticket that tries to talk it into widening its commands, and confirm the hash check refuses.
• Test the pause. Kill the orchestrator while it waits for approval and confirm it resumes from the saved state, not from the top.
• Start with Agent 1 only. Triage and ticketing are low risk and prove the signal quality. Add Agent 2’s read-only RCA next. Give anything write access last, and only for a narrow set of reversible actions.
• Measure the humans. An approval gate that gets rubber-stamped at 3 a.m. isn’t a control. Track how often approvals are modified or rejected.