Initial commit: autonomous maintenance agent for semprini stack

systemd-deployed daemon that drives the semprini-core stack-support agent
headless via Claude Code, monitors health via Uptime Kuma, applies safe
upgrades, and escalates to the operator over Matrix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.gitignore

@@ -0,0 +1,11 @@
+# Secrets — hold the Matrix bot token + password
+config.json
+.bot-secrets
+
+# Python
+__pycache__/
+*.pyc
+
+# Runtime state (also lives under ~/.local/state by default)
+state.json
+*.log

CLAUDE.md

@@ -0,0 +1,70 @@
+# semprini-maintainer — Claude Instructions
+
+## What this is
+
+A standalone, always-on **autonomous maintenance agent** for the semprini.me
+container stack, deployed as a systemd service (`semprini-maintainer`). It keeps
+the stack healthy and current and talks to the operator over Matrix
+(chat.semprini.me).
+
+It is the *unattended driver* for the canonical **stack-support agent**, which
+lives in the separate **semprini-core** repo. This project does not reimplement
+that logic — it gathers state and invokes the stack-support agent headless via
+Claude Code, then relays/acts on the agent's JSON result.
+
+## Relationship to other projects
+
+- **semprini-core** (`../semprini-core`) — the Enterprise Landing Zone this agent
+  maintains. At runtime the daemon reads, from the path in `config.json`
+  `repo_dir`:
+  - `agents/stack-support.agent.md` (injected as system prompt)
+  - `docs/stack-support-runbook.md`
+  - `core_stack/compose*.yml`
+  Keep the stack-support agent definition canonical **in semprini-core** — do not
+  fork a copy here.
+- **Other projects** (e.g. `../semprini-blog`) that register Uptime Kuma monitors
+  are also maintained; their repos are exposed to the agent via `extra_dirs`.
+
+## How it works (one screen)
+
+- **Scope** = whatever **Uptime Kuma** tracks at check time, re-read every cycle
+  (never hard-coded). Kuma's heartbeat is the health signal — the daemon does not
+  re-probe. Kuma's root-owned `kuma.db` is read via a throwaway `keinos/sqlite3`
+  container mounting the `uptime-kuma-data` volume read-only.
+- **Health loop** (5 min): down services confirmed over `down_confirmations`
+  cycles → invoke stack-support agent to remediate → report to Matrix.
+- **Upgrade loop** (7 days): resolve containers behind tracked services → agent
+  researches + applies safe upgrades (test, pin, rollback); escalates
+  major/breaking ones to Matrix.
+- **Matrix**: bot `@maintainer:semprini.me` DMs `@paul:semprini.me`. Escalations
+  are numbered questions; the reply is fed back to the agent to carry out.
+- **Autonomy** (`config.json` → `autonomy`): `full` (default) | `fix-only` |
+  `notify`.
+
+## Conventions / guardrails
+
+- Runs as host user **`paul`** (in `docker` group), **not root** — Claude Code
+  refuses `--permission-mode bypassPermissions` under root.
+- `maintainer.py` is **stdlib-only** — keep it dependency-free so it deploys
+  without a venv.
+- Secrets live in `config.json` and `.bot-secrets` — **never commit** (gitignored).
+- Prefer editing existing files over adding new ones.
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `maintainer.py` | The daemon (stdlib only). |
+| `config.example.json` | Template — copy to `config.json`. |
+| `register-matrix-bot.sh` | Creates `@maintainer` account + DM room; writes token to config. |
+| `install.sh` | Syntax-check, install unit, enable + start. |
+| `semprini-maintainer.service` | systemd unit (runs as `paul`). |
+| `docs/architecture.md` | Architecture decision record. |
+
+## Test without the service
+
+```bash
+python3 maintainer.py --config config.json --once health
+python3 maintainer.py --config config.json --once upgrade
+python3 maintainer.py --config config.json --once replies
+```

README.md

@@ -0,0 +1,141 @@
+# semprini-maintainer — autonomous stack maintenance agent
+
+A standalone systemd service that keeps the container stack healthy and current,
+and talks to the operator over Matrix (chat.semprini.me).
+
+It is the *unattended driver* for the canonical stack-support agent that lives
+in the **semprini-core** repo (`agents/stack-support.agent.md`): the daemon
+gathers state and hands the judgement calls (diagnosis, remediation, upgrade
+planning) to that agent, run headless via Claude Code.
+
+## Relationship to semprini-core
+
+This is a **separate project** from the stack it maintains. It does not bundle
+its own copy of the stack logic — at runtime it reads, from the semprini-core
+checkout pointed to by `repo_dir` in `config.json`:
+
+- `agents/stack-support.agent.md` — injected as the agent's system prompt
+- `docs/stack-support-runbook.md` — the remediation runbook
+- `core_stack/compose*.yml` — the compose files it operates on
+
+So **a semprini-core checkout and the running core stack must be present on the
+host** (they are what this agent maintains). The canonical stack-support agent
+definition stays in semprini-core to avoid drift. Other projects that register
+Uptime Kuma monitors (e.g. `semprini-blog`) are also maintained — their repos
+are made available to the agent via `extra_dirs`.
+
+### Prerequisites
+
+- Docker (host user in the `docker` group)
+- Python 3 (stdlib only — no venv needed)
+- [Claude Code](https://claude.com/claude-code) CLI on `PATH`
+- A semprini-core checkout + the core stack running (incl. Uptime Kuma)
+
+## What it does
+
+| Cadence | Action |
+|---|---|
+| **Every 5 min** (`health_interval_seconds`) | Read Uptime Kuma to learn which services are tracked **right now** and their up/down status. If a service is confirmed down across `down_confirmations` cycles, invoke the stack-support agent to diagnose + remediate, then report to Matrix. |
+| **Every 7 days** (`upgrade_interval_seconds`) | Resolve the containers behind tracked services, find the owning compose file, and have the agent research + apply upgrades (test, pin, rollback on failure). Breaking/major upgrades are escalated to Matrix for a decision. |
+| **Continuous** (`matrix_poll_seconds`) | Poll the Matrix room for the admin's replies — to answer escalations, or to take ad-hoc instructions. |
+| **Daily** (`heartbeat_interval_seconds`) | Quiet "N/M services up" heartbeat when all is well. |
+
+### Scope is dynamic
+
+Scope is **whatever Uptime Kuma is tracking at the moment the check runs** — the
+monitor list is re-read every cycle, nothing is hard-coded. This deliberately
+covers services from *other* projects that register Kuma monitors (e.g.
+`semprini-blog`), not just `semprini-core`. Other project repos are made
+available to the agent via `extra_dirs`.
+
+Kuma's own heartbeat result is the health signal, so the daemon never
+re-implements probes — it trusts the monitoring system already in the stack.
+
+## Architecture
+
+```
+                 ┌─────────────────────────────────────────┐
+                 │           maintainer.py (daemon)         │
+   Uptime Kuma ──┤  read monitors+status (sqlite in volume) │
+   (kuma.db)     │                                          │
+                 │  on down / on schedule:                  │
+                 │     claude -p  --append-system-prompt     │──▶ stack-support
+                 │        (agents/stack-support.agent.md)    │     agent does the
+                 │                                          │     diagnosis/fix
+                 │  parse JSON result envelope              │◀──  + returns JSON
+                 │                                          │
+   Matrix  ◀─────┤  send summaries / ask questions          │
+   (@paul)  ─────┤  poll replies → resume escalations       │
+                 └─────────────────────────────────────────┘
+```
+
+The agent returns a small JSON envelope the daemon relays:
+
+```json
+{"status":"ok|action_taken|escalate|blocked","summary":"…","question":"…","options":["…"]}
+```
+
+`escalate` posts the question (numbered options) to Matrix and pauses upgrades
+until the admin replies; the reply is fed back to the agent to carry out.
+
+## Autonomy
+
+Set `autonomy` in `config.json`:
+
+- `full` *(default)* — remediate **and** apply safe upgrades autonomously
+  (test + auto-rollback); escalate only major/breaking upgrades.
+- `fix-only` — remediate health issues; report upgrades but never apply them.
+- `notify` — never change anything; only report to Matrix.
+
+## Why it runs as `paul`, not root
+
+- Claude Code refuses `--permission-mode bypassPermissions` under root.
+- `paul` is in the `docker` group, so the agent can drive Docker/Compose.
+- Kuma's root-owned `kuma.db` is read via a throwaway `keinos/sqlite3`
+  container mounting the named volume read-only — no host file access needed.
+
+## Install
+
+```bash
+cd /home/paul/Dev/semprini-maintainer
+
+# 1. Config
+cp config.example.json config.json
+#   set repo_dir to the semprini-core checkout; edit paths/intervals if needed
+#   (defaults match this host)
+
+# 2. Provision the Matrix bot + private room with the admin
+./register-matrix-bot.sh
+#   then, one-time: accept the room invite in Element as @paul:semprini.me
+
+# 3. Install + start the service
+./install.sh
+```
+
+`config.json` and `.bot-secrets` hold the bot token/password and are gitignored.
+
+## Operate
+
+```bash
+journalctl -u semprini-maintainer -f          # logs
+systemctl status semprini-maintainer
+sudo systemctl restart semprini-maintainer
+
+# One-off cycles without the service (for testing):
+python3 maintainer.py --config config.json --once health
+python3 maintainer.py --config config.json --once upgrade
+python3 maintainer.py --config config.json --once replies
+```
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `maintainer.py` | The daemon. Stdlib only. |
+| `config.example.json` | Template — copy to `config.json`. |
+| `register-matrix-bot.sh` | Creates the `@maintainer` account + room, writes token to config. |
+| `semprini-maintainer.service` | systemd unit (runs as `paul`). |
+| `install.sh` | Syntax-check, install, enable, start. |
+
+See [`docs/architecture.md`](docs/architecture.md) for the architecture
+decision record.

config.example.json

@@ -0,0 +1,30 @@
+{
+  "repo_dir": "/home/paul/Dev/semprini-core",
+  "extra_dirs": ["/home/paul/Dev"],
+  "agent_file": "agents/stack-support.agent.md",
+  "runbook_file": "docs/stack-support-runbook.md",
+  "claude_bin": "/home/paul/.local/bin/claude",
+  "claude_model": null,
+  "claude_extra_args": [],
+  "agent_timeout_seconds": 1800,
+
+  "kuma_volume": "uptime-kuma-data",
+  "sqlite_image": "keinos/sqlite3",
+
+  "autonomy": "full",
+  "health_interval_seconds": 300,
+  "upgrade_interval_seconds": 604800,
+  "matrix_poll_seconds": 30,
+  "down_confirmations": 2,
+  "heartbeat_interval_seconds": 86400,
+
+  "state_file": "/home/paul/.local/state/semprini-maintainer/state.json",
+
+  "matrix": {
+    "homeserver": "https://matrix.semprini.me",
+    "user_id": "@maintainer:semprini.me",
+    "access_token": "FILLED_BY_register-matrix-bot.sh",
+    "room_id": "FILLED_BY_register-matrix-bot.sh",
+    "admin_user_id": "@paul:semprini.me"
+  }
+}

docs/architecture.md

@@ -0,0 +1,63 @@
+# Maintenance Agent (semprini-maintainer)
+
+Autonomous, always-on agent that keeps the container stack healthy and current.
+Implementation: this repository — see [`README.md`](../README.md). It maintains
+the **semprini-core** stack (a separate repo) referenced via `repo_dir`.
+
+## Purpose
+
+Continuously maintain the running stack without an operator at the keyboard:
+
+1. Detect and **remediate** service outages.
+2. Periodically **check for and apply container upgrades** (with test + rollback).
+3. **Communicate with the operator over Matrix** (chat.semprini.me), including
+   asking questions when an upgrade needs a human decision.
+
+It is the unattended driver for the canonical stack-support agent
+(`agents/stack-support.agent.md`) — it does not re-implement that logic, it
+feeds the agent state and relays its results.
+
+## Key decisions
+
+- **Scope = Uptime Kuma's live monitor list.** Read every health cycle, never
+  hard-coded. Anything Kuma tracks is in scope — including services owned by
+  other projects (e.g. `semprini-blog`) that register Kuma monitors. This keeps
+  the maintained set and the monitored set identical by construction.
+- **Health signal = Kuma heartbeat.** The daemon reads Kuma's own up/down
+  result rather than re-probing, reusing the stack's existing monitoring.
+- **Kuma data access** via a throwaway `keinos/sqlite3` container mounting the
+  `uptime-kuma-data` volume read-only (the daemon user has Docker but not host
+  access to root-owned volume files).
+- **Runner = Claude Code headless** (`claude -p`), with
+  `agents/stack-support.agent.md` injected as the system prompt. The agent
+  returns a JSON envelope (`status/summary/question/options`) the daemon acts on.
+- **Runs as `paul`** (in the `docker` group), not root — Claude Code refuses
+  `bypassPermissions` as root.
+- **Autonomy = full** by default: remediate and apply safe upgrades; escalate
+  only major/breaking upgrades to Matrix and resume on reply. Configurable to
+  `fix-only` or `notify`.
+- **Matrix transport**: dedicated bot `@maintainer:semprini.me` (created via the
+  Synapse registration shared secret) in a private room with `@paul:semprini.me`.
+
+## Cadence
+
+| Loop | Default interval | Config key |
+|---|---|---|
+| Health check + remediation | 5 min | `health_interval_seconds` |
+| Upgrade review | 7 days | `upgrade_interval_seconds` |
+| Matrix reply polling | 30 s | `matrix_poll_seconds` |
+| Quiet health heartbeat | daily | `heartbeat_interval_seconds` |
+
+## Escalation flow
+
+1. Agent returns `status=escalate` with a `question` + `options`.
+2. Daemon posts a numbered question to Matrix and pauses upgrade cycles.
+3. Admin replies (a number or free text).
+4. Daemon feeds the decision back to the agent, which carries it out and reports.
+
+## Operational notes
+
+- The bot DM room invite must be accepted once by `@paul` in Element.
+- Secrets (`config.json`, `.bot-secrets`) are gitignored.
+- A single unanswered escalation blocks new upgrade cycles (health remediation
+  continues regardless).

install.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# install.sh — install & enable the semprini-maintainer systemd service.
+#
+# Run from this directory. Requires sudo for the systemd parts.
+set -euo pipefail
+
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+UNIT="semprini-maintainer.service"
+DEST="/etc/systemd/system/$UNIT"
+
+echo "→ Pre-flight checks"
+command -v docker  >/dev/null || { echo "ERROR: docker not found"; exit 1; }
+command -v python3 >/dev/null || { echo "ERROR: python3 not found"; exit 1; }
+
+CLAUDE_BIN="$(python3 -c "import json;print(json.load(open('$DIR/config.json'))['claude_bin'])" 2>/dev/null || echo /home/paul/.local/bin/claude)"
+[ -x "$CLAUDE_BIN" ] || echo "WARN: claude binary not executable at $CLAUDE_BIN"
+
+if [ ! -f "$DIR/config.json" ]; then
+  echo "ERROR: config.json missing. Run:"
+  echo "    cp $DIR/config.example.json $DIR/config.json"
+  echo "    $DIR/register-matrix-bot.sh"
+  exit 1
+fi
+
+TOKEN="$(python3 -c "import json;print(json.load(open('$DIR/config.json'))['matrix']['access_token'])")"
+case "$TOKEN" in
+  ""|FILLED_BY_*) echo "ERROR: Matrix not provisioned. Run register-matrix-bot.sh first."; exit 1;;
+esac
+
+echo "→ Syntax-checking maintainer.py"
+python3 -m py_compile "$DIR/maintainer.py"
+
+echo "→ Installing $UNIT"
+sudo cp "$DIR/$UNIT" "$DEST"
+sudo systemctl daemon-reload
+sudo systemctl enable "$UNIT"
+sudo systemctl restart "$UNIT"
+
+echo
+echo "✓ Installed and started."
+echo "  Logs:    journalctl -u $UNIT -f"
+echo "  Status:  systemctl status $UNIT"
+echo "  Stop:    sudo systemctl stop $UNIT"

maintainer.py

@@ -0,0 +1,593 @@
+#!/usr/bin/env python3
+"""
+semprini-maintainer — autonomous stack maintenance daemon.
+
+Runs as a systemd service. Periodically:
+  * (frequent)   reads Uptime Kuma to learn which services are tracked and their
+                 up/down status, and — when something is down — invokes the
+                 canonical stack-support agent (Claude Code, headless) to
+                 diagnose and remediate fully autonomously.
+  * (infrequent) resolves the containers behind the tracked services, finds the
+                 compose file that owns each, and invokes the stack-support
+                 agent to research + apply upgrades (test, then rollback on
+                 failure). Ambiguous / breaking upgrades are escalated to the
+                 admin over Matrix and resumed once they reply.
+
+All operator communication happens through the core stack's Matrix
+(chat.semprini.me) via a dedicated bot account.
+
+The scope ("which services to maintain") is whatever Uptime Kuma is tracking at
+the moment the health check runs — it is re-read every cycle, never hard-coded.
+
+Design notes
+------------
+* Pure Python stdlib only (urllib, sqlite via a throwaway container, subprocess)
+  so the service deploys without a venv.
+* The daemon is intentionally "dumb": it gathers state and hands the fuzzy work
+  (mapping a monitor to a container, choosing a fix, planning an upgrade) to the
+  LLM agent defined in agents/stack-support.agent.md. The agent returns a small
+  JSON envelope which the daemon relays / acts on.
+"""
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import time
+import traceback
+import urllib.parse
+import urllib.request
+from datetime import datetime, timezone
+
+# --------------------------------------------------------------------------- #
+# Config / state
+# --------------------------------------------------------------------------- #
+
+DEFAULTS = {
+    "repo_dir": "/home/paul/Dev/semprini-core",
+    # Extra directories the agent may read/operate on (other projects that use
+    # the core stack, e.g. semprini-blog). Passed to claude via --add-dir.
+    "extra_dirs": ["/home/paul/Dev"],
+    "agent_file": "agents/stack-support.agent.md",
+    "runbook_file": "docs/stack-support-runbook.md",
+    "claude_bin": "/home/paul/.local/bin/claude",
+    "claude_model": None,                 # None = claude default
+    "claude_extra_args": [],
+    "agent_timeout_seconds": 1800,
+    "kuma_volume": "uptime-kuma-data",
+    "sqlite_image": "keinos/sqlite3",
+    # autonomy: "full" (fix + upgrade), "fix-only" (remediate, only report
+    # upgrades), or "notify" (never act, only report).
+    "autonomy": "full",
+    "health_interval_seconds": 300,        # 5 min
+    "upgrade_interval_seconds": 604800,    # 7 days
+    "matrix_poll_seconds": 30,
+    # A monitor must be seen DOWN this many consecutive cycles before we act,
+    # to ride out restarts / flapping.
+    "down_confirmations": 2,
+    # Send a quiet "all healthy" heartbeat to Matrix at most this often.
+    "heartbeat_interval_seconds": 86400,   # daily
+    "state_file": "/home/paul/.local/state/semprini-maintainer/state.json",
+    "matrix": {
+        "homeserver": "https://matrix.semprini.me",
+        "user_id": "",
+        "access_token": "",
+        "room_id": "",
+        "admin_user_id": "@paul:semprini.me",
+    },
+}
+
+
+def load_config(path):
+    cfg = json.loads(json.dumps(DEFAULTS))  # deep copy
+    if path and os.path.exists(path):
+        with open(path) as fh:
+            user = json.load(fh)
+        for k, v in user.items():
+            if k == "matrix" and isinstance(v, dict):
+                cfg["matrix"].update(v)
+            else:
+                cfg[k] = v
+    return cfg
+
+
+def load_state(cfg):
+    path = cfg["state_file"]
+    if os.path.exists(path):
+        try:
+            with open(path) as fh:
+                return json.load(fh)
+        except Exception:
+            log("WARN: could not parse state file, starting fresh")
+    return {
+        "last_health": 0,
+        "last_upgrade": 0,
+        "last_heartbeat": 0,
+        "down_streak": {},      # monitor name -> consecutive-down count
+        "matrix_since": None,   # /sync pagination token
+        "pending": None,        # outstanding question awaiting admin reply
+    }
+
+
+def save_state(cfg, state):
+    path = cfg["state_file"]
+    os.makedirs(os.path.dirname(path), exist_ok=True)
+    tmp = path + ".tmp"
+    with open(tmp, "w") as fh:
+        json.dump(state, fh, indent=2)
+    os.replace(tmp, path)
+
+
+def log(msg):
+    print(f"[{datetime.now(timezone.utc).isoformat()}] {msg}", flush=True)
+
+
+# --------------------------------------------------------------------------- #
+# Uptime Kuma — read tracked monitors + latest status
+# --------------------------------------------------------------------------- #
+
+KUMA_QUERY = (
+    "select json_group_array(json_object("
+    "'id', m.id, 'name', m.name, 'type', m.type, 'url', m.url, "
+    "'hostname', m.hostname, 'port', m.port, "
+    "'status', (select h.status from heartbeat h where h.monitor_id=m.id "
+    "order by h.time desc limit 1), "
+    "'msg', (select h.msg from heartbeat h where h.monitor_id=m.id "
+    "order by h.time desc limit 1))) "
+    "from monitor m where m.active=1 order by m.id;"
+)
+
+# Uptime Kuma heartbeat status codes.
+KUMA_DOWN, KUMA_UP, KUMA_PENDING, KUMA_MAINT = 0, 1, 2, 3
+
+
+def read_kuma_monitors(cfg):
+    """Return list of active monitors with their latest status.
+
+    Reads kuma.db out of the named Docker volume via a throwaway sqlite
+    container (the file is root-owned; the daemon user only has docker access).
+    """
+    script = f"cp /src/kuma.db /tmp/k.db && sqlite3 /tmp/k.db \"{KUMA_QUERY}\""
+    out = subprocess.run(
+        [
+            "docker", "run", "--rm",
+            "-v", f"{cfg['kuma_volume']}:/src:ro",
+            cfg["sqlite_image"], "sh", "-c", script,
+        ],
+        capture_output=True, text=True, timeout=120,
+    )
+    if out.returncode != 0:
+        raise RuntimeError(f"kuma read failed: {out.stderr.strip()}")
+    data = out.stdout.strip() or "[]"
+    return json.loads(data)
+
+
+def docker_ps():
+    out = subprocess.run(
+        ["docker", "ps", "-a", "--format",
+         "table {{.Names}}\t{{.Status}}\t{{.Image}}"],
+        capture_output=True, text=True, timeout=60,
+    )
+    return out.stdout.strip()
+
+
+# --------------------------------------------------------------------------- #
+# Matrix client (client-server API over stdlib)
+# --------------------------------------------------------------------------- #
+
+def _matrix_req(cfg, method, path, body=None, params=None):
+    base = cfg["matrix"]["homeserver"].rstrip("/")
+    url = f"{base}{path}"
+    if params:
+        url += "?" + urllib.parse.urlencode(params)
+    data = json.dumps(body).encode() if body is not None else None
+    req = urllib.request.Request(url, data=data, method=method)
+    req.add_header("Authorization", f"Bearer {cfg['matrix']['access_token']}")
+    req.add_header("Content-Type", "application/json")
+    with urllib.request.urlopen(req, timeout=60) as resp:
+        raw = resp.read().decode()
+    return json.loads(raw) if raw else {}
+
+
+def _md_to_html(text):
+    """Very small markdown -> HTML for Matrix formatted bodies."""
+    import html
+    esc = html.escape(text)
+    esc = re.sub(r"\*\*(.+?)\*\*", r"<b>\1</b>", esc)
+    esc = re.sub(r"`(.+?)`", r"<code>\1</code>", esc)
+    return esc.replace("\n", "<br>")
+
+
+def matrix_send(cfg, text):
+    if not cfg["matrix"].get("room_id") or not cfg["matrix"].get("access_token"):
+        log(f"MATRIX (not configured) >> {text}")
+        return
+    txn = str(int(time.time() * 1000))
+    room = urllib.parse.quote(cfg["matrix"]["room_id"])
+    try:
+        _matrix_req(
+            cfg, "PUT",
+            f"/_matrix/client/v3/rooms/{room}/send/m.room.message/{txn}",
+            body={
+                "msgtype": "m.text",
+                "body": text,
+                "format": "org.matrix.custom.html",
+                "formatted_body": _md_to_html(text),
+            },
+        )
+    except Exception as e:
+        log(f"ERROR sending to Matrix: {e}")
+
+
+def matrix_poll_admin(cfg, state):
+    """Return new text messages from the admin in our room since last sync."""
+    if not cfg["matrix"].get("access_token"):
+        return []
+    params = {"timeout": "0"}
+    if state.get("matrix_since"):
+        params["since"] = state["matrix_since"]
+    try:
+        resp = _matrix_req(cfg, "GET", "/_matrix/client/v3/sync", params=params)
+    except Exception as e:
+        log(f"ERROR polling Matrix: {e}")
+        return []
+    state["matrix_since"] = resp.get("next_batch", state.get("matrix_since"))
+    messages = []
+    room_id = cfg["matrix"]["room_id"]
+    admin = cfg["matrix"]["admin_user_id"]
+    rooms = resp.get("rooms", {}).get("join", {})
+    room = rooms.get(room_id)
+    if not room:
+        return messages
+    for ev in room.get("timeline", {}).get("events", []):
+        if ev.get("type") != "m.room.message":
+            continue
+        if ev.get("sender") != admin:
+            continue
+        content = ev.get("content", {})
+        if content.get("msgtype") == "m.text":
+            messages.append(content.get("body", "").strip())
+    return messages
+
+
+# --------------------------------------------------------------------------- #
+# Stack-support agent invocation (Claude Code, headless)
+# --------------------------------------------------------------------------- #
+
+RESULT_PROTOCOL = """
+When you have finished, end your reply with EXACTLY ONE fenced json block:
+
+```json
+{"status": "ok|action_taken|escalate|blocked",
+ "summary": "one or two sentence operator summary (markdown ok)",
+ "details": "optional longer detail",
+ "question": "optional — only if status is escalate: the decision you need",
+ "options": ["optional", "list", "of", "choices"]}
+```
+
+status meanings:
+  ok           — everything healthy, nothing to do
+  action_taken — you fixed/changed something; describe it in summary
+  escalate     — you need a human decision (set question + options)
+  blocked      — hard blocker, cannot proceed safely (explain in summary)
+""".strip()
+
+
+def run_agent(cfg, task):
+    """Invoke the canonical stack-support agent headless and return its output."""
+    agent_path = os.path.join(cfg["repo_dir"], cfg["agent_file"])
+    with open(agent_path) as fh:
+        agent_md = fh.read()
+
+    system = (
+        "You are the semprini-core Stack Maintenance Agent, running unattended "
+        "as a systemd service. The following is your canonical agent definition; "
+        "follow it as the source of truth, together with "
+        f"{cfg['runbook_file']} and CLAUDE.md in this repository.\n\n"
+        "You operate FULLY AUTONOMOUSLY per the CLAUDE.md sysadmin role: own the "
+        "fix end to end. Make the smallest safe change that restores service. "
+        "Pin versions after successful upgrades; auto-rollback on failure. "
+        "Only set status=escalate when a genuine human decision is required "
+        "(e.g. a major/breaking upgrade with trade-offs).\n\n"
+        "Services in scope are exactly those Uptime Kuma tracks. Other projects "
+        "(e.g. semprini-blog) live alongside semprini-core and use it for "
+        "IAM/reverse-proxy; maintain their containers too when they are tracked.\n\n"
+        "--- BEGIN agents/stack-support.agent.md ---\n"
+        f"{agent_md}\n"
+        "--- END agents/stack-support.agent.md ---\n\n"
+        f"{RESULT_PROTOCOL}"
+    )
+
+    cmd = [
+        cfg["claude_bin"], "-p", task,
+        "--append-system-prompt", system,
+        "--permission-mode", "bypassPermissions",
+        "--output-format", "text",
+    ]
+    if cfg.get("claude_model"):
+        cmd += ["--model", cfg["claude_model"]]
+    for d in cfg.get("extra_dirs", []):
+        cmd += ["--add-dir", d]
+    cmd += list(cfg.get("claude_extra_args", []))
+
+    log(f"Invoking stack-support agent ({len(task)} char task)…")
+    proc = subprocess.run(
+        cmd, cwd=cfg["repo_dir"], capture_output=True, text=True,
+        timeout=cfg["agent_timeout_seconds"],
+    )
+    if proc.returncode != 0:
+        log(f"WARN: agent exited {proc.returncode}: {proc.stderr.strip()[:500]}")
+    return proc.stdout.strip()
+
+
+def parse_agent_result(output):
+    """Pull the trailing JSON envelope out of the agent's reply."""
+    blocks = re.findall(r"```json\s*(\{.*?\})\s*```", output, re.DOTALL)
+    if blocks:
+        try:
+            return json.loads(blocks[-1])
+        except Exception:
+            pass
+    # Fallback: try a bare trailing {...}
+    m = re.search(r"(\{[^{}]*\"status\"[^{}]*\})\s*$", output, re.DOTALL)
+    if m:
+        try:
+            return json.loads(m.group(1))
+        except Exception:
+            pass
+    return {"status": "unknown",
+            "summary": (output[-800:] or "no output from agent")}
+
+
+def relay_result(cfg, state, header, result):
+    """Post an agent result to Matrix and register any escalation."""
+    status = result.get("status", "unknown")
+    summary = result.get("summary", "")
+    icon = {"ok": "✅", "action_taken": "🔧", "escalate": "❓",
+            "blocked": "⛔", "unknown": "⚠️"}.get(status, "•")
+    msg = f"{icon} **{header}** — {status}\n{summary}"
+    if result.get("details"):
+        msg += f"\n\n{result['details']}"
+
+    if status == "escalate" and result.get("question"):
+        opts = result.get("options") or []
+        msg += f"\n\n**Decision needed:** {result['question']}"
+        if opts:
+            msg += "\n" + "\n".join(f"  {i+1}. {o}" for i, o in enumerate(opts))
+            msg += "\n\n_Reply with a number or a short answer._"
+        state["pending"] = {
+            "header": header,
+            "question": result["question"],
+            "options": opts,
+            "asked_at": time.time(),
+        }
+    matrix_send(cfg, msg)
+
+
+# --------------------------------------------------------------------------- #
+# Cycles
+# --------------------------------------------------------------------------- #
+
+def status_label(code):
+    return {KUMA_DOWN: "DOWN", KUMA_UP: "UP", KUMA_PENDING: "PENDING",
+            KUMA_MAINT: "MAINTENANCE"}.get(code, f"?{code}")
+
+
+def health_cycle(cfg, state):
+    log("Health cycle: reading Uptime Kuma…")
+    monitors = read_kuma_monitors(cfg)
+    streak = state.setdefault("down_streak", {})
+
+    down_now = []
+    for m in monitors:
+        name = m["name"]
+        if m.get("status") == KUMA_DOWN:
+            streak[name] = streak.get(name, 0) + 1
+            down_now.append(m)
+        else:
+            streak.pop(name, None)
+
+    confirmed = [m for m in down_now
+                 if streak.get(m["name"], 0) >= cfg["down_confirmations"]]
+
+    log(f"  {len(monitors)} tracked, {len(down_now)} down, "
+        f"{len(confirmed)} confirmed down")
+
+    if not confirmed:
+        # quiet heartbeat
+        if (time.time() - state.get("last_heartbeat", 0)
+                >= cfg["heartbeat_interval_seconds"]):
+            up = sum(1 for m in monitors if m.get("status") == KUMA_UP)
+            matrix_send(
+                cfg,
+                f"✅ **Daily health** — {up}/{len(monitors)} tracked services up.",
+            )
+            state["last_heartbeat"] = time.time()
+        return
+
+    if cfg["autonomy"] == "notify":
+        lines = "\n".join(
+            f"  • **{m['name']}** ({m.get('url') or m.get('hostname')}): "
+            f"{m.get('msg') or 'down'}" for m in confirmed
+        )
+        matrix_send(cfg, f"⚠️ **Services down** (notify-only mode):\n{lines}")
+        return
+
+    incident = "\n".join(
+        f"  • {m['name']} | type={m['type']} | "
+        f"target={m.get('url') or (m.get('hostname') or '') + ':' + str(m.get('port') or '')} | "
+        f"last_msg={m.get('msg') or 'n/a'}"
+        for m in confirmed
+    )
+    task = (
+        "ROUTINE HEALTH CHECK — Uptime Kuma reports the following tracked "
+        "services DOWN (confirmed over multiple cycles):\n\n"
+        f"{incident}\n\n"
+        "Current containers:\n"
+        f"{docker_ps()}\n\n"
+        "For each down service: map the monitor target (a compose service "
+        "name / network alias) to its container, diagnose using the runbook, "
+        "and remediate autonomously. Re-verify recovery. If a service is down "
+        "by design or needs a human decision, set status=escalate."
+    )
+    result = parse_agent_result(run_agent(cfg, task))
+    relay_result(cfg, state, "Health remediation", result)
+
+
+def resolve_tracked_images(cfg, monitors):
+    """Best-effort map of tracked services -> running image (for context)."""
+    ps = subprocess.run(
+        ["docker", "ps", "--format", "{{.Names}}\t{{.Image}}"],
+        capture_output=True, text=True, timeout=60,
+    ).stdout.strip().splitlines()
+    name_to_image = {}
+    for line in ps:
+        if "\t" in line:
+            n, img = line.split("\t", 1)
+            name_to_image[n] = img
+    rows = []
+    for m in monitors:
+        target = (m.get("hostname") or m.get("url") or "").lower()
+        token = re.sub(r"^https?://", "", target).split("/")[0].split(":")[0]
+        matches = [f"{n} -> {img}" for n, img in name_to_image.items()
+                   if token and token in n]
+        rows.append(f"  • {m['name']} (target={token or 'n/a'}): "
+                    f"{', '.join(matches) or 'no obvious container match'}")
+    return "\n".join(rows)
+
+
+def upgrade_cycle(cfg, state):
+    if cfg["autonomy"] == "notify":
+        autonomy_note = (
+            "Mode is NOTIFY-ONLY: do NOT change anything. Report available "
+            "upgrades and recommendations only; set status=escalate with the list."
+        )
+    elif cfg["autonomy"] == "fix-only":
+        autonomy_note = (
+            "Mode is FIX-ONLY for upgrades: research and report available "
+            "upgrades; do NOT apply them. Set status=escalate to ask which to apply."
+        )
+    else:
+        autonomy_note = (
+            "Mode is FULL: apply safe upgrades autonomously (pull, test for "
+            "stability, pin the new tag in the owning compose file, redeploy, "
+            "rollback on failure). For MAJOR/breaking upgrades with trade-offs, "
+            "set status=escalate with options instead of applying."
+        )
+
+    log("Upgrade cycle…")
+    monitors = read_kuma_monitors(cfg)
+    context = resolve_tracked_images(cfg, monitors)
+    task = (
+        "PERIODIC UPGRADE REVIEW. Scope = services Uptime Kuma currently tracks. "
+        "Tracked services and their running images:\n\n"
+        f"{context}\n\n"
+        "Use the upgrade-services skill (agents/skills/upgrade-services/SKILL.md). "
+        "For each tracked service: find the compose file that owns it (search "
+        "core_stack/compose*.yml and other project repos under the added dirs, "
+        "e.g. semprini-blog/docker-compose.yml), determine the current pinned "
+        "tag, check Docker Hub / GitHub for newer stable releases, and assess "
+        "breaking changes.\n\n"
+        f"{autonomy_note}\n\n"
+        "Summarise what is up to date, what you upgraded, and anything needing "
+        "a decision."
+    )
+    result = parse_agent_result(run_agent(cfg, task))
+    relay_result(cfg, state, "Upgrade review", result)
+
+
+def handle_replies(cfg, state):
+    msgs = matrix_poll_admin(cfg, state)
+    if not msgs:
+        return
+    pending = state.get("pending")
+    for body in msgs:
+        log(f"Admin reply: {body!r}")
+        if not pending:
+            # Unprompted instruction from the admin — treat as an ad-hoc task.
+            task = (
+                "The operator sent this instruction over Matrix; act on it "
+                f"autonomously and report back:\n\n{body}"
+            )
+            result = parse_agent_result(run_agent(cfg, task))
+            relay_result(cfg, state, "Operator request", result)
+            pending = state.get("pending")
+            continue
+
+        # Resolve the outstanding escalation with the admin's answer.
+        answer = body
+        opts = pending.get("options") or []
+        if body.isdigit() and 1 <= int(body) <= len(opts):
+            answer = opts[int(body) - 1]
+        task = (
+            f"Earlier you escalated this decision:\n"
+            f"  Question: {pending['question']}\n"
+            f"  Options: {opts}\n\n"
+            f"The operator decided: {answer}\n\n"
+            "Carry out that decision now, autonomously, with full testing and "
+            "rollback on failure. Report the outcome."
+        )
+        state["pending"] = None
+        save_state(cfg, state)
+        result = parse_agent_result(run_agent(cfg, task))
+        relay_result(cfg, state, "Decision carried out", result)
+        pending = state.get("pending")
+
+
+# --------------------------------------------------------------------------- #
+# Main loop
+# --------------------------------------------------------------------------- #
+
+def main():
+    ap = argparse.ArgumentParser(description="semprini stack maintenance daemon")
+    ap.add_argument("--config",
+                    default="/home/paul/Dev/semprini-maintainer/config.json")
+    ap.add_argument("--once", choices=["health", "upgrade", "replies"],
+                    help="run a single cycle and exit (for testing)")
+    args = ap.parse_args()
+
+    cfg = load_config(args.config)
+    state = load_state(cfg)
+
+    if args.once:
+        log(f"Running single '{args.once}' cycle…")
+        {"health": health_cycle, "upgrade": upgrade_cycle,
+         "replies": handle_replies}[args.once](cfg, state)
+        save_state(cfg, state)
+        return
+
+    log("semprini-maintainer starting up.")
+    matrix_send(cfg, "🤖 **stack-maintainer online** — monitoring tracked "
+                     "services via Uptime Kuma.")
+
+    while True:
+        now = time.time()
+        try:
+            handle_replies(cfg, state)
+
+            if now - state.get("last_health", 0) >= cfg["health_interval_seconds"]:
+                health_cycle(cfg, state)
+                state["last_health"] = now
+
+            # Hold upgrades while an escalation is unanswered.
+            if (not state.get("pending")
+                    and now - state.get("last_upgrade", 0)
+                    >= cfg["upgrade_interval_seconds"]):
+                upgrade_cycle(cfg, state)
+                state["last_upgrade"] = now
+
+            save_state(cfg, state)
+        except Exception as e:
+            log("ERROR in main loop:\n" + traceback.format_exc())
+            try:
+                matrix_send(cfg, f"⛔ **maintainer error**: {e}")
+            except Exception:
+                pass
+
+        time.sleep(cfg["matrix_poll_seconds"])
+
+
+if __name__ == "__main__":
+    sys.exit(main())

register-matrix-bot.sh

@@ -0,0 +1,108 @@
+#!/usr/bin/env bash
+# register-matrix-bot.sh — provision the Matrix bot account the maintenance
+# agent uses to talk to the operator, and a private room shared with the admin.
+#
+# Idempotent-ish: re-running re-uses the existing account (login still works if
+# the password is unchanged) and creates a fresh room only if one isn't set.
+#
+# Requirements: run on the core-stack host, with the `user-synapse` container
+# running. Uses the registration shared secret already in the synapse config.
+#
+# Usage:
+#   ./register-matrix-bot.sh            # uses config.json next to this script
+#   CONFIG=/path/config.json ./register-matrix-bot.sh
+set -euo pipefail
+
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+CONFIG="${CONFIG:-$DIR/config.json}"
+SECRETS="$DIR/.bot-secrets"   # gitignored: stores the bot password
+
+if [ ! -f "$CONFIG" ]; then
+  echo "ERROR: $CONFIG not found. Copy config.example.json to config.json first."
+  exit 1
+fi
+
+jq_get() { python3 -c "import json,sys; print(json.load(open('$CONFIG'))$1)"; }
+
+HOMESERVER="$(jq_get "['matrix']['homeserver']")"
+BOT_USER_ID="$(jq_get "['matrix']['user_id']")"
+ADMIN_USER_ID="$(jq_get "['matrix']['admin_user_id']")"
+SYNAPSE_CONTAINER="${SYNAPSE_CONTAINER:-user-synapse}"
+
+# localpart from @maintainer:semprini.me -> maintainer
+BOT_LOCAL="${BOT_USER_ID#@}"; BOT_LOCAL="${BOT_LOCAL%%:*}"
+
+echo "→ Homeserver : $HOMESERVER"
+echo "→ Bot        : $BOT_USER_ID"
+echo "→ Admin      : $ADMIN_USER_ID"
+
+# 1. Password — reuse if we created one before, else generate and persist.
+if [ -f "$SECRETS" ]; then
+  # shellcheck disable=SC1090
+  source "$SECRETS"
+fi
+if [ -z "${BOT_PASSWORD:-}" ]; then
+  BOT_PASSWORD="$(openssl rand -hex 32)"
+  umask 077; echo "BOT_PASSWORD=$BOT_PASSWORD" > "$SECRETS"
+  echo "→ Generated new bot password (saved to $SECRETS)"
+fi
+
+# 2. Create the account (no-op if it already exists).
+echo "→ Registering account in $SYNAPSE_CONTAINER…"
+if docker exec "$SYNAPSE_CONTAINER" register_new_matrix_user \
+      -u "$BOT_LOCAL" -p "$BOT_PASSWORD" --no-admin \
+      -c /data/homeserver.yaml http://localhost:8008 2>&1 | tee /tmp/reg.out; then
+  :
+fi
+grep -qiE "already taken|User ID already" /tmp/reg.out && \
+  echo "  (account already exists — continuing)"
+
+# 3. Log in to obtain an access token.
+echo "→ Logging in…"
+LOGIN_JSON="$(curl -fsS -X POST "$HOMESERVER/_matrix/client/v3/login" \
+  -H 'Content-Type: application/json' \
+  -d "{\"type\":\"m.login.password\",
+       \"identifier\":{\"type\":\"m.id.user\",\"user\":\"$BOT_LOCAL\"},
+       \"password\":\"$BOT_PASSWORD\",
+       \"initial_device_display_name\":\"semprini-maintainer\"}")"
+ACCESS_TOKEN="$(python3 -c "import json,sys; print(json.loads(sys.argv[1])['access_token'])" "$LOGIN_JSON")"
+[ -n "$ACCESS_TOKEN" ] || { echo "ERROR: no access token"; exit 1; }
+echo "  got access token"
+
+# 4. Ensure a private room exists with the admin invited.
+ROOM_ID="$(jq_get "['matrix'].get('room_id','')" 2>/dev/null || echo "")"
+case "$ROOM_ID" in
+  ""|FILLED_BY_*|!*) : ;;
+esac
+if [[ "$ROOM_ID" != \!* ]]; then
+  echo "→ Creating maintenance room and inviting $ADMIN_USER_ID…"
+  ROOM_JSON="$(curl -fsS -X POST "$HOMESERVER/_matrix/client/v3/createRoom" \
+    -H "Authorization: Bearer $ACCESS_TOKEN" \
+    -H 'Content-Type: application/json' \
+    -d "{\"name\":\"Stack Maintenance\",
+         \"topic\":\"semprini-core autonomous stack maintenance\",
+         \"preset\":\"trusted_private_chat\",
+         \"is_direct\":true,
+         \"invite\":[\"$ADMIN_USER_ID\"]}")"
+  ROOM_ID="$(python3 -c "import json,sys; print(json.loads(sys.argv[1])['room_id'])" "$ROOM_JSON")"
+  echo "  room: $ROOM_ID"
+else
+  echo "→ Re-using existing room $ROOM_ID"
+fi
+
+# 5. Write token + room back into config.json.
+python3 - "$CONFIG" "$ACCESS_TOKEN" "$ROOM_ID" <<'PY'
+import json, sys
+path, token, room = sys.argv[1], sys.argv[2], sys.argv[3]
+cfg = json.load(open(path))
+cfg.setdefault("matrix", {})
+cfg["matrix"]["access_token"] = token
+cfg["matrix"]["room_id"] = room
+json.dump(cfg, open(path, "w"), indent=2)
+print(f"  wrote access_token + room_id to {path}")
+PY
+
+echo
+echo "✓ Done."
+echo "  ACTION REQUIRED (one-time): in Element (chat.semprini.me) as"
+echo "  $ADMIN_USER_ID, accept the invite to the 'Stack Maintenance' room."

semprini-maintainer.service

@@ -0,0 +1,31 @@
+[Unit]
+Description=semprini-core autonomous stack maintenance agent
+Documentation=file:///home/paul/Dev/semprini-maintainer/README.md
+After=docker.service network-online.target
+Wants=network-online.target
+Requires=docker.service
+
+[Service]
+Type=simple
+# Runs as the host user (must be in the 'docker' group). NOT root: Claude Code
+# refuses --permission-mode bypassPermissions when running as root.
+User=paul
+Group=paul
+SupplementaryGroups=docker
+WorkingDirectory=/home/paul/Dev/semprini-maintainer
+Environment=HOME=/home/paul
+Environment=PATH=/home/paul/.local/bin:/usr/local/bin:/usr/bin:/bin
+ExecStart=/usr/bin/python3 /home/paul/Dev/semprini-maintainer/maintainer.py --config /home/paul/Dev/semprini-maintainer/config.json
+Restart=always
+RestartSec=30
+# Give a long-running upgrade plenty of time before any stop is considered hung.
+TimeoutStopSec=120
+
+# Light hardening (kept permissive — the agent legitimately drives Docker).
+NoNewPrivileges=false
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=semprini-maintainer
+
+[Install]
+WantedBy=multi-user.target