From e91b1298410e6324c452dfdfa757e1eb0d501cbd Mon Sep 17 00:00:00 2001 From: Semprini Date: Sun, 21 Jun 2026 17:20:06 +1200 Subject: [PATCH] Initial commit: autonomous maintenance agent for semprini stack systemd-deployed daemon that drives the semprini-core stack-support agent headless via Claude Code, monitors health via Uptime Kuma, applies safe upgrades, and escalates to the operator over Matrix. Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 11 + CLAUDE.md | 70 +++++ README.md | 141 +++++++++ config.example.json | 30 ++ docs/architecture.md | 63 ++++ install.sh | 43 +++ maintainer.py | 593 ++++++++++++++++++++++++++++++++++++ register-matrix-bot.sh | 108 +++++++ semprini-maintainer.service | 31 ++ 9 files changed, 1090 insertions(+) create mode 100644 .gitignore create mode 100644 CLAUDE.md create mode 100644 README.md create mode 100644 config.example.json create mode 100644 docs/architecture.md create mode 100755 install.sh create mode 100755 maintainer.py create mode 100755 register-matrix-bot.sh create mode 100644 semprini-maintainer.service diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..42108e1 --- /dev/null +++ b/.gitignore @@ -0,0 +1,11 @@ +# Secrets — hold the Matrix bot token + password +config.json +.bot-secrets + +# Python +__pycache__/ +*.pyc + +# Runtime state (also lives under ~/.local/state by default) +state.json +*.log diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..be55ad9 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,70 @@ +# semprini-maintainer — Claude Instructions + +## What this is + +A standalone, always-on **autonomous maintenance agent** for the semprini.me +container stack, deployed as a systemd service (`semprini-maintainer`). It keeps +the stack healthy and current and talks to the operator over Matrix +(chat.semprini.me). + +It is the *unattended driver* for the canonical **stack-support agent**, which +lives in the separate **semprini-core** repo. This project does not reimplement +that logic — it gathers state and invokes the stack-support agent headless via +Claude Code, then relays/acts on the agent's JSON result. + +## Relationship to other projects + +- **semprini-core** (`../semprini-core`) — the Enterprise Landing Zone this agent + maintains. At runtime the daemon reads, from the path in `config.json` + `repo_dir`: + - `agents/stack-support.agent.md` (injected as system prompt) + - `docs/stack-support-runbook.md` + - `core_stack/compose*.yml` + Keep the stack-support agent definition canonical **in semprini-core** — do not + fork a copy here. +- **Other projects** (e.g. `../semprini-blog`) that register Uptime Kuma monitors + are also maintained; their repos are exposed to the agent via `extra_dirs`. + +## How it works (one screen) + +- **Scope** = whatever **Uptime Kuma** tracks at check time, re-read every cycle + (never hard-coded). Kuma's heartbeat is the health signal — the daemon does not + re-probe. Kuma's root-owned `kuma.db` is read via a throwaway `keinos/sqlite3` + container mounting the `uptime-kuma-data` volume read-only. +- **Health loop** (5 min): down services confirmed over `down_confirmations` + cycles → invoke stack-support agent to remediate → report to Matrix. +- **Upgrade loop** (7 days): resolve containers behind tracked services → agent + researches + applies safe upgrades (test, pin, rollback); escalates + major/breaking ones to Matrix. +- **Matrix**: bot `@maintainer:semprini.me` DMs `@paul:semprini.me`. Escalations + are numbered questions; the reply is fed back to the agent to carry out. +- **Autonomy** (`config.json` → `autonomy`): `full` (default) | `fix-only` | + `notify`. + +## Conventions / guardrails + +- Runs as host user **`paul`** (in `docker` group), **not root** — Claude Code + refuses `--permission-mode bypassPermissions` under root. +- `maintainer.py` is **stdlib-only** — keep it dependency-free so it deploys + without a venv. +- Secrets live in `config.json` and `.bot-secrets` — **never commit** (gitignored). +- Prefer editing existing files over adding new ones. + +## Files + +| File | Purpose | +|---|---| +| `maintainer.py` | The daemon (stdlib only). | +| `config.example.json` | Template — copy to `config.json`. | +| `register-matrix-bot.sh` | Creates `@maintainer` account + DM room; writes token to config. | +| `install.sh` | Syntax-check, install unit, enable + start. | +| `semprini-maintainer.service` | systemd unit (runs as `paul`). | +| `docs/architecture.md` | Architecture decision record. | + +## Test without the service + +```bash +python3 maintainer.py --config config.json --once health +python3 maintainer.py --config config.json --once upgrade +python3 maintainer.py --config config.json --once replies +``` diff --git a/README.md b/README.md new file mode 100644 index 0000000..0a7e636 --- /dev/null +++ b/README.md @@ -0,0 +1,141 @@ +# semprini-maintainer — autonomous stack maintenance agent + +A standalone systemd service that keeps the container stack healthy and current, +and talks to the operator over Matrix (chat.semprini.me). + +It is the *unattended driver* for the canonical stack-support agent that lives +in the **semprini-core** repo (`agents/stack-support.agent.md`): the daemon +gathers state and hands the judgement calls (diagnosis, remediation, upgrade +planning) to that agent, run headless via Claude Code. + +## Relationship to semprini-core + +This is a **separate project** from the stack it maintains. It does not bundle +its own copy of the stack logic — at runtime it reads, from the semprini-core +checkout pointed to by `repo_dir` in `config.json`: + +- `agents/stack-support.agent.md` — injected as the agent's system prompt +- `docs/stack-support-runbook.md` — the remediation runbook +- `core_stack/compose*.yml` — the compose files it operates on + +So **a semprini-core checkout and the running core stack must be present on the +host** (they are what this agent maintains). The canonical stack-support agent +definition stays in semprini-core to avoid drift. Other projects that register +Uptime Kuma monitors (e.g. `semprini-blog`) are also maintained — their repos +are made available to the agent via `extra_dirs`. + +### Prerequisites + +- Docker (host user in the `docker` group) +- Python 3 (stdlib only — no venv needed) +- [Claude Code](https://claude.com/claude-code) CLI on `PATH` +- A semprini-core checkout + the core stack running (incl. Uptime Kuma) + +## What it does + +| Cadence | Action | +|---|---| +| **Every 5 min** (`health_interval_seconds`) | Read Uptime Kuma to learn which services are tracked **right now** and their up/down status. If a service is confirmed down across `down_confirmations` cycles, invoke the stack-support agent to diagnose + remediate, then report to Matrix. | +| **Every 7 days** (`upgrade_interval_seconds`) | Resolve the containers behind tracked services, find the owning compose file, and have the agent research + apply upgrades (test, pin, rollback on failure). Breaking/major upgrades are escalated to Matrix for a decision. | +| **Continuous** (`matrix_poll_seconds`) | Poll the Matrix room for the admin's replies — to answer escalations, or to take ad-hoc instructions. | +| **Daily** (`heartbeat_interval_seconds`) | Quiet "N/M services up" heartbeat when all is well. | + +### Scope is dynamic + +Scope is **whatever Uptime Kuma is tracking at the moment the check runs** — the +monitor list is re-read every cycle, nothing is hard-coded. This deliberately +covers services from *other* projects that register Kuma monitors (e.g. +`semprini-blog`), not just `semprini-core`. Other project repos are made +available to the agent via `extra_dirs`. + +Kuma's own heartbeat result is the health signal, so the daemon never +re-implements probes — it trusts the monitoring system already in the stack. + +## Architecture + +``` + ┌─────────────────────────────────────────┐ + │ maintainer.py (daemon) │ + Uptime Kuma ──┤ read monitors+status (sqlite in volume) │ + (kuma.db) │ │ + │ on down / on schedule: │ + │ claude -p --append-system-prompt │──▶ stack-support + │ (agents/stack-support.agent.md) │ agent does the + │ │ diagnosis/fix + │ parse JSON result envelope │◀── + returns JSON + │ │ + Matrix ◀─────┤ send summaries / ask questions │ + (@paul) ─────┤ poll replies → resume escalations │ + └─────────────────────────────────────────┘ +``` + +The agent returns a small JSON envelope the daemon relays: + +```json +{"status":"ok|action_taken|escalate|blocked","summary":"…","question":"…","options":["…"]} +``` + +`escalate` posts the question (numbered options) to Matrix and pauses upgrades +until the admin replies; the reply is fed back to the agent to carry out. + +## Autonomy + +Set `autonomy` in `config.json`: + +- `full` *(default)* — remediate **and** apply safe upgrades autonomously + (test + auto-rollback); escalate only major/breaking upgrades. +- `fix-only` — remediate health issues; report upgrades but never apply them. +- `notify` — never change anything; only report to Matrix. + +## Why it runs as `paul`, not root + +- Claude Code refuses `--permission-mode bypassPermissions` under root. +- `paul` is in the `docker` group, so the agent can drive Docker/Compose. +- Kuma's root-owned `kuma.db` is read via a throwaway `keinos/sqlite3` + container mounting the named volume read-only — no host file access needed. + +## Install + +```bash +cd /home/paul/Dev/semprini-maintainer + +# 1. Config +cp config.example.json config.json +# set repo_dir to the semprini-core checkout; edit paths/intervals if needed +# (defaults match this host) + +# 2. Provision the Matrix bot + private room with the admin +./register-matrix-bot.sh +# then, one-time: accept the room invite in Element as @paul:semprini.me + +# 3. Install + start the service +./install.sh +``` + +`config.json` and `.bot-secrets` hold the bot token/password and are gitignored. + +## Operate + +```bash +journalctl -u semprini-maintainer -f # logs +systemctl status semprini-maintainer +sudo systemctl restart semprini-maintainer + +# One-off cycles without the service (for testing): +python3 maintainer.py --config config.json --once health +python3 maintainer.py --config config.json --once upgrade +python3 maintainer.py --config config.json --once replies +``` + +## Files + +| File | Purpose | +|---|---| +| `maintainer.py` | The daemon. Stdlib only. | +| `config.example.json` | Template — copy to `config.json`. | +| `register-matrix-bot.sh` | Creates the `@maintainer` account + room, writes token to config. | +| `semprini-maintainer.service` | systemd unit (runs as `paul`). | +| `install.sh` | Syntax-check, install, enable, start. | + +See [`docs/architecture.md`](docs/architecture.md) for the architecture +decision record. diff --git a/config.example.json b/config.example.json new file mode 100644 index 0000000..a6ad8e7 --- /dev/null +++ b/config.example.json @@ -0,0 +1,30 @@ +{ + "repo_dir": "/home/paul/Dev/semprini-core", + "extra_dirs": ["/home/paul/Dev"], + "agent_file": "agents/stack-support.agent.md", + "runbook_file": "docs/stack-support-runbook.md", + "claude_bin": "/home/paul/.local/bin/claude", + "claude_model": null, + "claude_extra_args": [], + "agent_timeout_seconds": 1800, + + "kuma_volume": "uptime-kuma-data", + "sqlite_image": "keinos/sqlite3", + + "autonomy": "full", + "health_interval_seconds": 300, + "upgrade_interval_seconds": 604800, + "matrix_poll_seconds": 30, + "down_confirmations": 2, + "heartbeat_interval_seconds": 86400, + + "state_file": "/home/paul/.local/state/semprini-maintainer/state.json", + + "matrix": { + "homeserver": "https://matrix.semprini.me", + "user_id": "@maintainer:semprini.me", + "access_token": "FILLED_BY_register-matrix-bot.sh", + "room_id": "FILLED_BY_register-matrix-bot.sh", + "admin_user_id": "@paul:semprini.me" + } +} diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..b8a2920 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,63 @@ +# Maintenance Agent (semprini-maintainer) + +Autonomous, always-on agent that keeps the container stack healthy and current. +Implementation: this repository — see [`README.md`](../README.md). It maintains +the **semprini-core** stack (a separate repo) referenced via `repo_dir`. + +## Purpose + +Continuously maintain the running stack without an operator at the keyboard: + +1. Detect and **remediate** service outages. +2. Periodically **check for and apply container upgrades** (with test + rollback). +3. **Communicate with the operator over Matrix** (chat.semprini.me), including + asking questions when an upgrade needs a human decision. + +It is the unattended driver for the canonical stack-support agent +(`agents/stack-support.agent.md`) — it does not re-implement that logic, it +feeds the agent state and relays its results. + +## Key decisions + +- **Scope = Uptime Kuma's live monitor list.** Read every health cycle, never + hard-coded. Anything Kuma tracks is in scope — including services owned by + other projects (e.g. `semprini-blog`) that register Kuma monitors. This keeps + the maintained set and the monitored set identical by construction. +- **Health signal = Kuma heartbeat.** The daemon reads Kuma's own up/down + result rather than re-probing, reusing the stack's existing monitoring. +- **Kuma data access** via a throwaway `keinos/sqlite3` container mounting the + `uptime-kuma-data` volume read-only (the daemon user has Docker but not host + access to root-owned volume files). +- **Runner = Claude Code headless** (`claude -p`), with + `agents/stack-support.agent.md` injected as the system prompt. The agent + returns a JSON envelope (`status/summary/question/options`) the daemon acts on. +- **Runs as `paul`** (in the `docker` group), not root — Claude Code refuses + `bypassPermissions` as root. +- **Autonomy = full** by default: remediate and apply safe upgrades; escalate + only major/breaking upgrades to Matrix and resume on reply. Configurable to + `fix-only` or `notify`. +- **Matrix transport**: dedicated bot `@maintainer:semprini.me` (created via the + Synapse registration shared secret) in a private room with `@paul:semprini.me`. + +## Cadence + +| Loop | Default interval | Config key | +|---|---|---| +| Health check + remediation | 5 min | `health_interval_seconds` | +| Upgrade review | 7 days | `upgrade_interval_seconds` | +| Matrix reply polling | 30 s | `matrix_poll_seconds` | +| Quiet health heartbeat | daily | `heartbeat_interval_seconds` | + +## Escalation flow + +1. Agent returns `status=escalate` with a `question` + `options`. +2. Daemon posts a numbered question to Matrix and pauses upgrade cycles. +3. Admin replies (a number or free text). +4. Daemon feeds the decision back to the agent, which carries it out and reports. + +## Operational notes + +- The bot DM room invite must be accepted once by `@paul` in Element. +- Secrets (`config.json`, `.bot-secrets`) are gitignored. +- A single unanswered escalation blocks new upgrade cycles (health remediation + continues regardless). diff --git a/install.sh b/install.sh new file mode 100755 index 0000000..95e0482 --- /dev/null +++ b/install.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# install.sh — install & enable the semprini-maintainer systemd service. +# +# Run from this directory. Requires sudo for the systemd parts. +set -euo pipefail + +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +UNIT="semprini-maintainer.service" +DEST="/etc/systemd/system/$UNIT" + +echo "→ Pre-flight checks" +command -v docker >/dev/null || { echo "ERROR: docker not found"; exit 1; } +command -v python3 >/dev/null || { echo "ERROR: python3 not found"; exit 1; } + +CLAUDE_BIN="$(python3 -c "import json;print(json.load(open('$DIR/config.json'))['claude_bin'])" 2>/dev/null || echo /home/paul/.local/bin/claude)" +[ -x "$CLAUDE_BIN" ] || echo "WARN: claude binary not executable at $CLAUDE_BIN" + +if [ ! -f "$DIR/config.json" ]; then + echo "ERROR: config.json missing. Run:" + echo " cp $DIR/config.example.json $DIR/config.json" + echo " $DIR/register-matrix-bot.sh" + exit 1 +fi + +TOKEN="$(python3 -c "import json;print(json.load(open('$DIR/config.json'))['matrix']['access_token'])")" +case "$TOKEN" in + ""|FILLED_BY_*) echo "ERROR: Matrix not provisioned. Run register-matrix-bot.sh first."; exit 1;; +esac + +echo "→ Syntax-checking maintainer.py" +python3 -m py_compile "$DIR/maintainer.py" + +echo "→ Installing $UNIT" +sudo cp "$DIR/$UNIT" "$DEST" +sudo systemctl daemon-reload +sudo systemctl enable "$UNIT" +sudo systemctl restart "$UNIT" + +echo +echo "✓ Installed and started." +echo " Logs: journalctl -u $UNIT -f" +echo " Status: systemctl status $UNIT" +echo " Stop: sudo systemctl stop $UNIT" diff --git a/maintainer.py b/maintainer.py new file mode 100755 index 0000000..79c3225 --- /dev/null +++ b/maintainer.py @@ -0,0 +1,593 @@ +#!/usr/bin/env python3 +""" +semprini-maintainer — autonomous stack maintenance daemon. + +Runs as a systemd service. Periodically: + * (frequent) reads Uptime Kuma to learn which services are tracked and their + up/down status, and — when something is down — invokes the + canonical stack-support agent (Claude Code, headless) to + diagnose and remediate fully autonomously. + * (infrequent) resolves the containers behind the tracked services, finds the + compose file that owns each, and invokes the stack-support + agent to research + apply upgrades (test, then rollback on + failure). Ambiguous / breaking upgrades are escalated to the + admin over Matrix and resumed once they reply. + +All operator communication happens through the core stack's Matrix +(chat.semprini.me) via a dedicated bot account. + +The scope ("which services to maintain") is whatever Uptime Kuma is tracking at +the moment the health check runs — it is re-read every cycle, never hard-coded. + +Design notes +------------ +* Pure Python stdlib only (urllib, sqlite via a throwaway container, subprocess) + so the service deploys without a venv. +* The daemon is intentionally "dumb": it gathers state and hands the fuzzy work + (mapping a monitor to a container, choosing a fix, planning an upgrade) to the + LLM agent defined in agents/stack-support.agent.md. The agent returns a small + JSON envelope which the daemon relays / acts on. +""" + +import argparse +import json +import os +import re +import subprocess +import sys +import time +import traceback +import urllib.parse +import urllib.request +from datetime import datetime, timezone + +# --------------------------------------------------------------------------- # +# Config / state +# --------------------------------------------------------------------------- # + +DEFAULTS = { + "repo_dir": "/home/paul/Dev/semprini-core", + # Extra directories the agent may read/operate on (other projects that use + # the core stack, e.g. semprini-blog). Passed to claude via --add-dir. + "extra_dirs": ["/home/paul/Dev"], + "agent_file": "agents/stack-support.agent.md", + "runbook_file": "docs/stack-support-runbook.md", + "claude_bin": "/home/paul/.local/bin/claude", + "claude_model": None, # None = claude default + "claude_extra_args": [], + "agent_timeout_seconds": 1800, + "kuma_volume": "uptime-kuma-data", + "sqlite_image": "keinos/sqlite3", + # autonomy: "full" (fix + upgrade), "fix-only" (remediate, only report + # upgrades), or "notify" (never act, only report). + "autonomy": "full", + "health_interval_seconds": 300, # 5 min + "upgrade_interval_seconds": 604800, # 7 days + "matrix_poll_seconds": 30, + # A monitor must be seen DOWN this many consecutive cycles before we act, + # to ride out restarts / flapping. + "down_confirmations": 2, + # Send a quiet "all healthy" heartbeat to Matrix at most this often. + "heartbeat_interval_seconds": 86400, # daily + "state_file": "/home/paul/.local/state/semprini-maintainer/state.json", + "matrix": { + "homeserver": "https://matrix.semprini.me", + "user_id": "", + "access_token": "", + "room_id": "", + "admin_user_id": "@paul:semprini.me", + }, +} + + +def load_config(path): + cfg = json.loads(json.dumps(DEFAULTS)) # deep copy + if path and os.path.exists(path): + with open(path) as fh: + user = json.load(fh) + for k, v in user.items(): + if k == "matrix" and isinstance(v, dict): + cfg["matrix"].update(v) + else: + cfg[k] = v + return cfg + + +def load_state(cfg): + path = cfg["state_file"] + if os.path.exists(path): + try: + with open(path) as fh: + return json.load(fh) + except Exception: + log("WARN: could not parse state file, starting fresh") + return { + "last_health": 0, + "last_upgrade": 0, + "last_heartbeat": 0, + "down_streak": {}, # monitor name -> consecutive-down count + "matrix_since": None, # /sync pagination token + "pending": None, # outstanding question awaiting admin reply + } + + +def save_state(cfg, state): + path = cfg["state_file"] + os.makedirs(os.path.dirname(path), exist_ok=True) + tmp = path + ".tmp" + with open(tmp, "w") as fh: + json.dump(state, fh, indent=2) + os.replace(tmp, path) + + +def log(msg): + print(f"[{datetime.now(timezone.utc).isoformat()}] {msg}", flush=True) + + +# --------------------------------------------------------------------------- # +# Uptime Kuma — read tracked monitors + latest status +# --------------------------------------------------------------------------- # + +KUMA_QUERY = ( + "select json_group_array(json_object(" + "'id', m.id, 'name', m.name, 'type', m.type, 'url', m.url, " + "'hostname', m.hostname, 'port', m.port, " + "'status', (select h.status from heartbeat h where h.monitor_id=m.id " + "order by h.time desc limit 1), " + "'msg', (select h.msg from heartbeat h where h.monitor_id=m.id " + "order by h.time desc limit 1))) " + "from monitor m where m.active=1 order by m.id;" +) + +# Uptime Kuma heartbeat status codes. +KUMA_DOWN, KUMA_UP, KUMA_PENDING, KUMA_MAINT = 0, 1, 2, 3 + + +def read_kuma_monitors(cfg): + """Return list of active monitors with their latest status. + + Reads kuma.db out of the named Docker volume via a throwaway sqlite + container (the file is root-owned; the daemon user only has docker access). + """ + script = f"cp /src/kuma.db /tmp/k.db && sqlite3 /tmp/k.db \"{KUMA_QUERY}\"" + out = subprocess.run( + [ + "docker", "run", "--rm", + "-v", f"{cfg['kuma_volume']}:/src:ro", + cfg["sqlite_image"], "sh", "-c", script, + ], + capture_output=True, text=True, timeout=120, + ) + if out.returncode != 0: + raise RuntimeError(f"kuma read failed: {out.stderr.strip()}") + data = out.stdout.strip() or "[]" + return json.loads(data) + + +def docker_ps(): + out = subprocess.run( + ["docker", "ps", "-a", "--format", + "table {{.Names}}\t{{.Status}}\t{{.Image}}"], + capture_output=True, text=True, timeout=60, + ) + return out.stdout.strip() + + +# --------------------------------------------------------------------------- # +# Matrix client (client-server API over stdlib) +# --------------------------------------------------------------------------- # + +def _matrix_req(cfg, method, path, body=None, params=None): + base = cfg["matrix"]["homeserver"].rstrip("/") + url = f"{base}{path}" + if params: + url += "?" + urllib.parse.urlencode(params) + data = json.dumps(body).encode() if body is not None else None + req = urllib.request.Request(url, data=data, method=method) + req.add_header("Authorization", f"Bearer {cfg['matrix']['access_token']}") + req.add_header("Content-Type", "application/json") + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read().decode() + return json.loads(raw) if raw else {} + + +def _md_to_html(text): + """Very small markdown -> HTML for Matrix formatted bodies.""" + import html + esc = html.escape(text) + esc = re.sub(r"\*\*(.+?)\*\*", r"\1", esc) + esc = re.sub(r"`(.+?)`", r"\1", esc) + return esc.replace("\n", "
") + + +def matrix_send(cfg, text): + if not cfg["matrix"].get("room_id") or not cfg["matrix"].get("access_token"): + log(f"MATRIX (not configured) >> {text}") + return + txn = str(int(time.time() * 1000)) + room = urllib.parse.quote(cfg["matrix"]["room_id"]) + try: + _matrix_req( + cfg, "PUT", + f"/_matrix/client/v3/rooms/{room}/send/m.room.message/{txn}", + body={ + "msgtype": "m.text", + "body": text, + "format": "org.matrix.custom.html", + "formatted_body": _md_to_html(text), + }, + ) + except Exception as e: + log(f"ERROR sending to Matrix: {e}") + + +def matrix_poll_admin(cfg, state): + """Return new text messages from the admin in our room since last sync.""" + if not cfg["matrix"].get("access_token"): + return [] + params = {"timeout": "0"} + if state.get("matrix_since"): + params["since"] = state["matrix_since"] + try: + resp = _matrix_req(cfg, "GET", "/_matrix/client/v3/sync", params=params) + except Exception as e: + log(f"ERROR polling Matrix: {e}") + return [] + state["matrix_since"] = resp.get("next_batch", state.get("matrix_since")) + messages = [] + room_id = cfg["matrix"]["room_id"] + admin = cfg["matrix"]["admin_user_id"] + rooms = resp.get("rooms", {}).get("join", {}) + room = rooms.get(room_id) + if not room: + return messages + for ev in room.get("timeline", {}).get("events", []): + if ev.get("type") != "m.room.message": + continue + if ev.get("sender") != admin: + continue + content = ev.get("content", {}) + if content.get("msgtype") == "m.text": + messages.append(content.get("body", "").strip()) + return messages + + +# --------------------------------------------------------------------------- # +# Stack-support agent invocation (Claude Code, headless) +# --------------------------------------------------------------------------- # + +RESULT_PROTOCOL = """ +When you have finished, end your reply with EXACTLY ONE fenced json block: + +```json +{"status": "ok|action_taken|escalate|blocked", + "summary": "one or two sentence operator summary (markdown ok)", + "details": "optional longer detail", + "question": "optional — only if status is escalate: the decision you need", + "options": ["optional", "list", "of", "choices"]} +``` + +status meanings: + ok — everything healthy, nothing to do + action_taken — you fixed/changed something; describe it in summary + escalate — you need a human decision (set question + options) + blocked — hard blocker, cannot proceed safely (explain in summary) +""".strip() + + +def run_agent(cfg, task): + """Invoke the canonical stack-support agent headless and return its output.""" + agent_path = os.path.join(cfg["repo_dir"], cfg["agent_file"]) + with open(agent_path) as fh: + agent_md = fh.read() + + system = ( + "You are the semprini-core Stack Maintenance Agent, running unattended " + "as a systemd service. The following is your canonical agent definition; " + "follow it as the source of truth, together with " + f"{cfg['runbook_file']} and CLAUDE.md in this repository.\n\n" + "You operate FULLY AUTONOMOUSLY per the CLAUDE.md sysadmin role: own the " + "fix end to end. Make the smallest safe change that restores service. " + "Pin versions after successful upgrades; auto-rollback on failure. " + "Only set status=escalate when a genuine human decision is required " + "(e.g. a major/breaking upgrade with trade-offs).\n\n" + "Services in scope are exactly those Uptime Kuma tracks. Other projects " + "(e.g. semprini-blog) live alongside semprini-core and use it for " + "IAM/reverse-proxy; maintain their containers too when they are tracked.\n\n" + "--- BEGIN agents/stack-support.agent.md ---\n" + f"{agent_md}\n" + "--- END agents/stack-support.agent.md ---\n\n" + f"{RESULT_PROTOCOL}" + ) + + cmd = [ + cfg["claude_bin"], "-p", task, + "--append-system-prompt", system, + "--permission-mode", "bypassPermissions", + "--output-format", "text", + ] + if cfg.get("claude_model"): + cmd += ["--model", cfg["claude_model"]] + for d in cfg.get("extra_dirs", []): + cmd += ["--add-dir", d] + cmd += list(cfg.get("claude_extra_args", [])) + + log(f"Invoking stack-support agent ({len(task)} char task)…") + proc = subprocess.run( + cmd, cwd=cfg["repo_dir"], capture_output=True, text=True, + timeout=cfg["agent_timeout_seconds"], + ) + if proc.returncode != 0: + log(f"WARN: agent exited {proc.returncode}: {proc.stderr.strip()[:500]}") + return proc.stdout.strip() + + +def parse_agent_result(output): + """Pull the trailing JSON envelope out of the agent's reply.""" + blocks = re.findall(r"```json\s*(\{.*?\})\s*```", output, re.DOTALL) + if blocks: + try: + return json.loads(blocks[-1]) + except Exception: + pass + # Fallback: try a bare trailing {...} + m = re.search(r"(\{[^{}]*\"status\"[^{}]*\})\s*$", output, re.DOTALL) + if m: + try: + return json.loads(m.group(1)) + except Exception: + pass + return {"status": "unknown", + "summary": (output[-800:] or "no output from agent")} + + +def relay_result(cfg, state, header, result): + """Post an agent result to Matrix and register any escalation.""" + status = result.get("status", "unknown") + summary = result.get("summary", "") + icon = {"ok": "✅", "action_taken": "🔧", "escalate": "❓", + "blocked": "⛔", "unknown": "⚠️"}.get(status, "•") + msg = f"{icon} **{header}** — {status}\n{summary}" + if result.get("details"): + msg += f"\n\n{result['details']}" + + if status == "escalate" and result.get("question"): + opts = result.get("options") or [] + msg += f"\n\n**Decision needed:** {result['question']}" + if opts: + msg += "\n" + "\n".join(f" {i+1}. {o}" for i, o in enumerate(opts)) + msg += "\n\n_Reply with a number or a short answer._" + state["pending"] = { + "header": header, + "question": result["question"], + "options": opts, + "asked_at": time.time(), + } + matrix_send(cfg, msg) + + +# --------------------------------------------------------------------------- # +# Cycles +# --------------------------------------------------------------------------- # + +def status_label(code): + return {KUMA_DOWN: "DOWN", KUMA_UP: "UP", KUMA_PENDING: "PENDING", + KUMA_MAINT: "MAINTENANCE"}.get(code, f"?{code}") + + +def health_cycle(cfg, state): + log("Health cycle: reading Uptime Kuma…") + monitors = read_kuma_monitors(cfg) + streak = state.setdefault("down_streak", {}) + + down_now = [] + for m in monitors: + name = m["name"] + if m.get("status") == KUMA_DOWN: + streak[name] = streak.get(name, 0) + 1 + down_now.append(m) + else: + streak.pop(name, None) + + confirmed = [m for m in down_now + if streak.get(m["name"], 0) >= cfg["down_confirmations"]] + + log(f" {len(monitors)} tracked, {len(down_now)} down, " + f"{len(confirmed)} confirmed down") + + if not confirmed: + # quiet heartbeat + if (time.time() - state.get("last_heartbeat", 0) + >= cfg["heartbeat_interval_seconds"]): + up = sum(1 for m in monitors if m.get("status") == KUMA_UP) + matrix_send( + cfg, + f"✅ **Daily health** — {up}/{len(monitors)} tracked services up.", + ) + state["last_heartbeat"] = time.time() + return + + if cfg["autonomy"] == "notify": + lines = "\n".join( + f" • **{m['name']}** ({m.get('url') or m.get('hostname')}): " + f"{m.get('msg') or 'down'}" for m in confirmed + ) + matrix_send(cfg, f"⚠️ **Services down** (notify-only mode):\n{lines}") + return + + incident = "\n".join( + f" • {m['name']} | type={m['type']} | " + f"target={m.get('url') or (m.get('hostname') or '') + ':' + str(m.get('port') or '')} | " + f"last_msg={m.get('msg') or 'n/a'}" + for m in confirmed + ) + task = ( + "ROUTINE HEALTH CHECK — Uptime Kuma reports the following tracked " + "services DOWN (confirmed over multiple cycles):\n\n" + f"{incident}\n\n" + "Current containers:\n" + f"{docker_ps()}\n\n" + "For each down service: map the monitor target (a compose service " + "name / network alias) to its container, diagnose using the runbook, " + "and remediate autonomously. Re-verify recovery. If a service is down " + "by design or needs a human decision, set status=escalate." + ) + result = parse_agent_result(run_agent(cfg, task)) + relay_result(cfg, state, "Health remediation", result) + + +def resolve_tracked_images(cfg, monitors): + """Best-effort map of tracked services -> running image (for context).""" + ps = subprocess.run( + ["docker", "ps", "--format", "{{.Names}}\t{{.Image}}"], + capture_output=True, text=True, timeout=60, + ).stdout.strip().splitlines() + name_to_image = {} + for line in ps: + if "\t" in line: + n, img = line.split("\t", 1) + name_to_image[n] = img + rows = [] + for m in monitors: + target = (m.get("hostname") or m.get("url") or "").lower() + token = re.sub(r"^https?://", "", target).split("/")[0].split(":")[0] + matches = [f"{n} -> {img}" for n, img in name_to_image.items() + if token and token in n] + rows.append(f" • {m['name']} (target={token or 'n/a'}): " + f"{', '.join(matches) or 'no obvious container match'}") + return "\n".join(rows) + + +def upgrade_cycle(cfg, state): + if cfg["autonomy"] == "notify": + autonomy_note = ( + "Mode is NOTIFY-ONLY: do NOT change anything. Report available " + "upgrades and recommendations only; set status=escalate with the list." + ) + elif cfg["autonomy"] == "fix-only": + autonomy_note = ( + "Mode is FIX-ONLY for upgrades: research and report available " + "upgrades; do NOT apply them. Set status=escalate to ask which to apply." + ) + else: + autonomy_note = ( + "Mode is FULL: apply safe upgrades autonomously (pull, test for " + "stability, pin the new tag in the owning compose file, redeploy, " + "rollback on failure). For MAJOR/breaking upgrades with trade-offs, " + "set status=escalate with options instead of applying." + ) + + log("Upgrade cycle…") + monitors = read_kuma_monitors(cfg) + context = resolve_tracked_images(cfg, monitors) + task = ( + "PERIODIC UPGRADE REVIEW. Scope = services Uptime Kuma currently tracks. " + "Tracked services and their running images:\n\n" + f"{context}\n\n" + "Use the upgrade-services skill (agents/skills/upgrade-services/SKILL.md). " + "For each tracked service: find the compose file that owns it (search " + "core_stack/compose*.yml and other project repos under the added dirs, " + "e.g. semprini-blog/docker-compose.yml), determine the current pinned " + "tag, check Docker Hub / GitHub for newer stable releases, and assess " + "breaking changes.\n\n" + f"{autonomy_note}\n\n" + "Summarise what is up to date, what you upgraded, and anything needing " + "a decision." + ) + result = parse_agent_result(run_agent(cfg, task)) + relay_result(cfg, state, "Upgrade review", result) + + +def handle_replies(cfg, state): + msgs = matrix_poll_admin(cfg, state) + if not msgs: + return + pending = state.get("pending") + for body in msgs: + log(f"Admin reply: {body!r}") + if not pending: + # Unprompted instruction from the admin — treat as an ad-hoc task. + task = ( + "The operator sent this instruction over Matrix; act on it " + f"autonomously and report back:\n\n{body}" + ) + result = parse_agent_result(run_agent(cfg, task)) + relay_result(cfg, state, "Operator request", result) + pending = state.get("pending") + continue + + # Resolve the outstanding escalation with the admin's answer. + answer = body + opts = pending.get("options") or [] + if body.isdigit() and 1 <= int(body) <= len(opts): + answer = opts[int(body) - 1] + task = ( + f"Earlier you escalated this decision:\n" + f" Question: {pending['question']}\n" + f" Options: {opts}\n\n" + f"The operator decided: {answer}\n\n" + "Carry out that decision now, autonomously, with full testing and " + "rollback on failure. Report the outcome." + ) + state["pending"] = None + save_state(cfg, state) + result = parse_agent_result(run_agent(cfg, task)) + relay_result(cfg, state, "Decision carried out", result) + pending = state.get("pending") + + +# --------------------------------------------------------------------------- # +# Main loop +# --------------------------------------------------------------------------- # + +def main(): + ap = argparse.ArgumentParser(description="semprini stack maintenance daemon") + ap.add_argument("--config", + default="/home/paul/Dev/semprini-maintainer/config.json") + ap.add_argument("--once", choices=["health", "upgrade", "replies"], + help="run a single cycle and exit (for testing)") + args = ap.parse_args() + + cfg = load_config(args.config) + state = load_state(cfg) + + if args.once: + log(f"Running single '{args.once}' cycle…") + {"health": health_cycle, "upgrade": upgrade_cycle, + "replies": handle_replies}[args.once](cfg, state) + save_state(cfg, state) + return + + log("semprini-maintainer starting up.") + matrix_send(cfg, "🤖 **stack-maintainer online** — monitoring tracked " + "services via Uptime Kuma.") + + while True: + now = time.time() + try: + handle_replies(cfg, state) + + if now - state.get("last_health", 0) >= cfg["health_interval_seconds"]: + health_cycle(cfg, state) + state["last_health"] = now + + # Hold upgrades while an escalation is unanswered. + if (not state.get("pending") + and now - state.get("last_upgrade", 0) + >= cfg["upgrade_interval_seconds"]): + upgrade_cycle(cfg, state) + state["last_upgrade"] = now + + save_state(cfg, state) + except Exception as e: + log("ERROR in main loop:\n" + traceback.format_exc()) + try: + matrix_send(cfg, f"⛔ **maintainer error**: {e}") + except Exception: + pass + + time.sleep(cfg["matrix_poll_seconds"]) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/register-matrix-bot.sh b/register-matrix-bot.sh new file mode 100755 index 0000000..0199e2d --- /dev/null +++ b/register-matrix-bot.sh @@ -0,0 +1,108 @@ +#!/usr/bin/env bash +# register-matrix-bot.sh — provision the Matrix bot account the maintenance +# agent uses to talk to the operator, and a private room shared with the admin. +# +# Idempotent-ish: re-running re-uses the existing account (login still works if +# the password is unchanged) and creates a fresh room only if one isn't set. +# +# Requirements: run on the core-stack host, with the `user-synapse` container +# running. Uses the registration shared secret already in the synapse config. +# +# Usage: +# ./register-matrix-bot.sh # uses config.json next to this script +# CONFIG=/path/config.json ./register-matrix-bot.sh +set -euo pipefail + +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CONFIG="${CONFIG:-$DIR/config.json}" +SECRETS="$DIR/.bot-secrets" # gitignored: stores the bot password + +if [ ! -f "$CONFIG" ]; then + echo "ERROR: $CONFIG not found. Copy config.example.json to config.json first." + exit 1 +fi + +jq_get() { python3 -c "import json,sys; print(json.load(open('$CONFIG'))$1)"; } + +HOMESERVER="$(jq_get "['matrix']['homeserver']")" +BOT_USER_ID="$(jq_get "['matrix']['user_id']")" +ADMIN_USER_ID="$(jq_get "['matrix']['admin_user_id']")" +SYNAPSE_CONTAINER="${SYNAPSE_CONTAINER:-user-synapse}" + +# localpart from @maintainer:semprini.me -> maintainer +BOT_LOCAL="${BOT_USER_ID#@}"; BOT_LOCAL="${BOT_LOCAL%%:*}" + +echo "→ Homeserver : $HOMESERVER" +echo "→ Bot : $BOT_USER_ID" +echo "→ Admin : $ADMIN_USER_ID" + +# 1. Password — reuse if we created one before, else generate and persist. +if [ -f "$SECRETS" ]; then + # shellcheck disable=SC1090 + source "$SECRETS" +fi +if [ -z "${BOT_PASSWORD:-}" ]; then + BOT_PASSWORD="$(openssl rand -hex 32)" + umask 077; echo "BOT_PASSWORD=$BOT_PASSWORD" > "$SECRETS" + echo "→ Generated new bot password (saved to $SECRETS)" +fi + +# 2. Create the account (no-op if it already exists). +echo "→ Registering account in $SYNAPSE_CONTAINER…" +if docker exec "$SYNAPSE_CONTAINER" register_new_matrix_user \ + -u "$BOT_LOCAL" -p "$BOT_PASSWORD" --no-admin \ + -c /data/homeserver.yaml http://localhost:8008 2>&1 | tee /tmp/reg.out; then + : +fi +grep -qiE "already taken|User ID already" /tmp/reg.out && \ + echo " (account already exists — continuing)" + +# 3. Log in to obtain an access token. +echo "→ Logging in…" +LOGIN_JSON="$(curl -fsS -X POST "$HOMESERVER/_matrix/client/v3/login" \ + -H 'Content-Type: application/json' \ + -d "{\"type\":\"m.login.password\", + \"identifier\":{\"type\":\"m.id.user\",\"user\":\"$BOT_LOCAL\"}, + \"password\":\"$BOT_PASSWORD\", + \"initial_device_display_name\":\"semprini-maintainer\"}")" +ACCESS_TOKEN="$(python3 -c "import json,sys; print(json.loads(sys.argv[1])['access_token'])" "$LOGIN_JSON")" +[ -n "$ACCESS_TOKEN" ] || { echo "ERROR: no access token"; exit 1; } +echo " got access token" + +# 4. Ensure a private room exists with the admin invited. +ROOM_ID="$(jq_get "['matrix'].get('room_id','')" 2>/dev/null || echo "")" +case "$ROOM_ID" in + ""|FILLED_BY_*|!*) : ;; +esac +if [[ "$ROOM_ID" != \!* ]]; then + echo "→ Creating maintenance room and inviting $ADMIN_USER_ID…" + ROOM_JSON="$(curl -fsS -X POST "$HOMESERVER/_matrix/client/v3/createRoom" \ + -H "Authorization: Bearer $ACCESS_TOKEN" \ + -H 'Content-Type: application/json' \ + -d "{\"name\":\"Stack Maintenance\", + \"topic\":\"semprini-core autonomous stack maintenance\", + \"preset\":\"trusted_private_chat\", + \"is_direct\":true, + \"invite\":[\"$ADMIN_USER_ID\"]}")" + ROOM_ID="$(python3 -c "import json,sys; print(json.loads(sys.argv[1])['room_id'])" "$ROOM_JSON")" + echo " room: $ROOM_ID" +else + echo "→ Re-using existing room $ROOM_ID" +fi + +# 5. Write token + room back into config.json. +python3 - "$CONFIG" "$ACCESS_TOKEN" "$ROOM_ID" <<'PY' +import json, sys +path, token, room = sys.argv[1], sys.argv[2], sys.argv[3] +cfg = json.load(open(path)) +cfg.setdefault("matrix", {}) +cfg["matrix"]["access_token"] = token +cfg["matrix"]["room_id"] = room +json.dump(cfg, open(path, "w"), indent=2) +print(f" wrote access_token + room_id to {path}") +PY + +echo +echo "✓ Done." +echo " ACTION REQUIRED (one-time): in Element (chat.semprini.me) as" +echo " $ADMIN_USER_ID, accept the invite to the 'Stack Maintenance' room." diff --git a/semprini-maintainer.service b/semprini-maintainer.service new file mode 100644 index 0000000..2f51d77 --- /dev/null +++ b/semprini-maintainer.service @@ -0,0 +1,31 @@ +[Unit] +Description=semprini-core autonomous stack maintenance agent +Documentation=file:///home/paul/Dev/semprini-maintainer/README.md +After=docker.service network-online.target +Wants=network-online.target +Requires=docker.service + +[Service] +Type=simple +# Runs as the host user (must be in the 'docker' group). NOT root: Claude Code +# refuses --permission-mode bypassPermissions when running as root. +User=paul +Group=paul +SupplementaryGroups=docker +WorkingDirectory=/home/paul/Dev/semprini-maintainer +Environment=HOME=/home/paul +Environment=PATH=/home/paul/.local/bin:/usr/local/bin:/usr/bin:/bin +ExecStart=/usr/bin/python3 /home/paul/Dev/semprini-maintainer/maintainer.py --config /home/paul/Dev/semprini-maintainer/config.json +Restart=always +RestartSec=30 +# Give a long-running upgrade plenty of time before any stop is considered hung. +TimeoutStopSec=120 + +# Light hardening (kept permissive — the agent legitimately drives Docker). +NoNewPrivileges=false +StandardOutput=journal +StandardError=journal +SyslogIdentifier=semprini-maintainer + +[Install] +WantedBy=multi-user.target