Alerts That Reach You: Prometheus, Alertmanager, an AI Agent and Telegram

A dashboard nobody opens doesn’t monitor anything. For my homelab I wanted alerts to arrive where I already am — Telegram — and to arrive with some context, not just a rule name.

The design: Prometheus evaluates rules, Alertmanager routes firing alerts to a webhook, and an AI agent receives that webhook, looks at the metrics around it, and messages me as a first-line support engineer.

A note on what this is. My homelab-monitoring repository holds the write-up and configuration for this stack. It’s a design I documented, not a long-running production deployment, so treat the configs below as a starting point to adapt and test, not something I’m vouching for under load. (The agent is the one from Packing an Agent into a Container.)

Architecture

┌──────────────────────── Docker host (Ubuntu 24.04) ─────────────────────────┐
│  Node Exporter :9100 ──▶ Prometheus :9090 ──▶ Grafana :3000                  │
│                              │                                               │
│                              ▼ firing alerts                                 │
│                        Alertmanager :9093                                    │
│                              │ webhook                                       │
│                              ▼                                               │
│                         AI agent (container)  ◀── can query Prometheus       │
└──────────────────────────────┼───────────────────────────────────────────────┘
                               ▼
                           Telegram

Everything shares one Docker network (monitoring-net), so the services reach each other by container name.

The repository layout

homelab-monitoring/
├── docker-compose.yml
├── .env.example
├── prometheus/
│   ├── prometheus.yml        # scrape config
│   └── alerts.yml            # alert rules
├── grafana/provisioning/datasources/prometheus.yml
├── alertmanager/alertmanager.yml   # webhook to the agent
└── .github/workflows/publish.yml   # mirror images to GHCR

Configuration comes from a .env file, kept out of git:

GRAFANA_PASSWORD=<choose-one>
AGENT_WEBHOOK_URL=http://hermes:PORT/webhook

Then docker compose up -d, and Grafana is on :3000, Prometheus on :9090, Alertmanager on :9093.

The alerts

Three rules cover the basics of a single host:

After editing prometheus/alerts.yml, a docker compose restart prometheus picks up the change.

The webhook receiver

The only Alertmanager-specific part is the receiver. Replace hermes:PORT with your agent’s container name and port, and check its documentation for the exact path:

# alertmanager/alertmanager.yml
receivers:
  - name: hermes
    webhook_configs:
      - url: "http://hermes:PORT/webhook"
        send_resolved: true

send_resolved: true matters more than it looks. Without it you get told when something breaks and never when it recovers, which leaves you wondering whether to go and check.

What the agent does with it

Running on the same host and network, the agent:

• receives each alert from Alertmanager over the webhook,
• talks to you on Telegram as a personal level-1 support engineer, and
• can query Prometheus directly, so a “high CPU” alert can come with what the process table and the last hour of load actually looked like.

That last point is the reason to put an agent here instead of a plain Telegram webhook. A bare rule name tells you that something is wrong; the agent can start answering what. (For the simpler, scripted version of this idea, see Automating Real-Time Security Alerts with Fail2ban and Telegram.)

Publishing the images to your own registry

I wanted the whole stack portable: a new host should be one docker compose pull away, without depending on Docker Hub being reachable. A matrix job mirrors each upstream image into GHCR on every push to main:

name: Publish Images to GHCR
on:
  push:
    branches: [main]

env:
  REGISTRY: ghcr.io
  OWNER: ${{ github.repository_owner }}

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    strategy:
      matrix:
        include:
          - { source: "prom/prometheus:latest",    target: prometheus }
          - { source: "prom/node-exporter:latest", target: node-exporter }
          - { source: "grafana/grafana:latest",    target: grafana }
          - { source: "prom/alertmanager:latest",  target: alertmanager }
    steps:
      - uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Pull, retag, and push ${{ matrix.target }}
        run: |
          docker pull ${{ matrix.source }}
          docker tag  ${{ matrix.source }} ${{ env.REGISTRY }}/${{ env.OWNER }}/${{ matrix.target }}:latest
          docker push ${{ env.REGISTRY }}/${{ env.OWNER }}/${{ matrix.target }}:latest

GITHUB_TOKEN is provided by Actions, so no extra secret is needed. For a first run, log in to GHCR once with a personal access token that has write:packages, push the images by hand, and let the workflow take over.

Moving to a new host is then:

echo $GITHUB_TOKEN | docker login ghcr.io -u <you> --password-stdin
git clone https://github.com/<you>/homelab-monitoring.git && cd homelab-monitoring
cp .env.example .env && nano .env
docker compose pull && docker compose up -d

Security notes

• Never commit .env; keep it in .gitignore.
• Change the Grafana password before first use.
• Prometheus and Alertmanager have no authentication by default. Don’t expose :9090 or :9093 publicly without a reverse proxy and basic auth, and keep Grafana behind a firewall or VPN.
• Remember the agent has a Telegram channel and can reach Prometheus. Treat its webhook as an authenticated internal endpoint, not an open one.

What I’d tighten

The mirror job is convenient but it re-tags :latest to :latest: every run can silently change the Prometheus or Grafana version you run on the next compose pull. For anything you care about, mirror a pinned version (for example prom/prometheus:v<x.y.z>) and keep that tag in your compose file. Portability is the goal, and reproducibility is what makes it safe.

It’s also worth putting a rule on the monitoring itself. A stack that watches everything but has no alert for “Prometheus stopped scraping” or “the agent stopped answering” fails silently — which is exactly what monitoring exists to prevent. I would add a dead-man’s-switch alert that fires constantly and a Telegram message when it stops.