A GitHub Actions Pipeline for a Prometheus and Alertmanager Stack — and the Secret I Baked into an Image

I wanted monitoring that deploys itself: change an alert rule, push to main, and a few minutes later the new rule is live on the monitoring VM. No SSH, no copying files.

The result is a small stack — Prometheus, Alertmanager and a node exporter — built and deployed by GitHub Actions, with alerts going out by email. This post walks through how it fits together, and then through the part I’d change, because one design choice put a credential somewhere it shouldn’t be.

The shape of it

push to main
   │
   ▼
job 1: build-and-push   (GitHub-hosted runner: ubuntu-latest)
   • build Prometheus image  (config baked in)  → ghcr.io/<repo>-prometheus
   • render alertmanager.yml from a template, build image → ghcr.io/<repo>-alertmanager
   │
   ▼
job 2: deploy           (self-hosted runner, on the monitoring VM)
   • write docker-compose.yml, log in to GHCR
   • docker compose pull && up -d && image prune

Two jobs, two kinds of runner. The build runs on a GitHub-hosted runner, so it needs nothing from my network. The deploy runs on a self-hosted runner installed on the monitoring VM itself, which means the VM pulls the new images and restarts its own containers. Nothing has to reach into the VM from outside.

The images

Each service gets a three-line Dockerfile that bakes its configuration in:

# Dockerfile.prometheus
FROM prom/prometheus:latest
COPY prometheus.yml /etc/prometheus/prometheus.yml
COPY rules.yml /etc/prometheus/rules.yml

# Dockerfile.alertmanager
FROM prom/alertmanager:latest
COPY alertmanager.yml /etc/alertmanager/alertmanager.yml

The point of baking config in is that the image is the deployment unit: whatever is in git on main is exactly what runs.

The alert rules

rules.yml has five rules:

The CPU rule is the usual idle-based expression:

- alert: HostHighCpuLoad
  expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 80
  for: 2m

Alertmanager groups by alertname, waits 10 seconds to batch related alerts, repeats an unresolved alert every hour, and sends resolved notices too. The subject line carries the status and the summary, so an email reads [PROMETHEUS ALERT] FIRING: Instance … down.

The workflow

The build job logs in to GHCR with the built-in GITHUB_TOKEN (so the job needs packages: write, and no personal access token is involved), then builds and pushes both images using docker/metadata-action for tags and labels.

The step in the middle is the interesting one:

- name: Substitute Alertmanager Configuration Secrets
  env:
    SMTP_USERNAME: ${{ secrets.SMTP_USERNAME }}
    SMTP_PASSWORD: ${{ secrets.SMTP_PASSWORD }}
    SMTP_TO_EMAIL: ${{ secrets.SMTP_TO_EMAIL }}
  run: |
    envsubst < alertmanager.yml.template > alertmanager.yml

alertmanager.yml.template holds ${SMTP_USERNAME} and friends; envsubst fills them in from repository secrets, and the result is copied into the image. The intent was that the rendered file stays out of git, so the password never appears in the repo. (That intent is not what the repo actually did — see the first item below.)

The deploy job then writes a docker-compose.yml on the runner with a heredoc, logs in to GHCR, and updates:

cd ~/prometheus-stack
echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
docker compose pull
docker compose up -d
docker image prune -f

Setting it up

1. Gmail app password. Turn on 2-step verification, then create an app password for Alertmanager.
2. Repository secrets (Settings → Secrets and variables → Actions): SMTP_USERNAME, SMTP_PASSWORD, SMTP_TO_EMAIL.
3. Self-hosted runner installed on the monitoring VM, with the default self-hosted label.
4. Push to main. The workflow builds, publishes and deploys.

To try the stack without CI, docker compose up -d brings it up locally: Prometheus on :9090, Alertmanager on :9093, node exporter on :9100.

What I’d change

It works, but a review of my own pipeline turned up things I wouldn’t repeat.

0. The “git-ignored” file wasn’t ignored. While writing this up I checked the repository. The README says alertmanager.yml is git-ignored, but the matching line in .gitignore was commented out, and the file was committed with a literal SMTP password in it rather than a placeholder. The pipeline overwrites the file at build time, so the pipeline itself worked — but the committed copy, and every commit it appears in, has to be treated as exposed. If you have this pattern: rotate the credential first, then remove the file from history, then fix the ignore rule. Check git ls-files rather than trusting the README.

1. The SMTP password is baked into an image layer. This is the big one. The README says the secrets are injected “securely during the container build”, but build-time injection and runtime injection are very different. envsubst writes the real password into alertmanager.yml, COPY puts that file into the image, and the image is pushed to a registry. Anyone with pull access to that package can read the password out of it, and it stays in the layer history forever, even if you later rotate the secret. Repository secrets keep it out of git; they don’t keep it out of the image.

The fix is to keep the image config-free of secrets and supply the file at runtime: render alertmanager.yml on the runner (the deploy job already has the secrets) and bind-mount it into the container, or use Docker secrets. Then the published image contains no credentials at all.

2. Floating tags. The Dockerfiles use :latest bases and the deploy pulls :main. A rebuild a month later can silently change Prometheus’s version. Pin the base versions and tag images with the commit SHA so a deploy is reproducible and a rollback is a one-line change.

3. The compose file lives inside the workflow. Writing it with a heredoc means the deployment definition is a string inside YAML, and the repo’s own docker-compose.yml (for local use) is a different file that can drift. Keep one compose file in the repo and have the runner check it out.

4. Old action versions. The workflow uses checkout@v3, docker/login-action@v2 and build-push-action@v4. They work, but they’re several majors behind; bump them.

5. Targets are hard-coded and unauthenticated. prometheus.yml scrapes fixed IPs, and Prometheus and Alertmanager have no authentication. Fine on a lab VLAN; put a reverse proxy with auth in front before exposing them anywhere else.

6. Add a smoke test. The pipeline never checks that the new Prometheus loaded its rules. promtool check rules in the build job would catch a broken rule before it ships, not after.

The pattern I’d keep is the split between a build job on a hosted runner and a deploy job on a self-hosted one. It keeps inbound access closed and gives each push a single, auditable path to production. The secret handling is what I’d redo — see how I handled the same problem for an agent image in Image Tags That Don’t Lie.