feat: Implement initial setup for Financial Crime domain

- Added Makefile for orchestrating domain instantiation and management. - Created deploy README for guiding the setup process of the Financial Crime domain. - Introduced domain configuration file for mapping metadata to the deployment template. - Implemented pre-commit hook for validating metadata before commits. - Developed instantiation script to manage the lifecycle of the Financial Crime domain. - Added Salesforce CRM source with associated metadata and transformation files. - Added SAP Fraud Management source with associated metadata and transformation files. - Added Temenos Payment source with associated metadata and transformation files. - Removed obsolete payment event and parties transformation files.
2026-06-14 16:46:06 +12:00
parent f4c46700c5
commit 0aeb893c0f
25 changed files with 612 additions and 85 deletions
@@ -0,0 +1,64 @@
+# Deploy — Financial Crime domain (Phase 1)
+
+Phase 1 stands up a *running, stable* Financial Crime data domain from this repo's
+MD-DDL metadata plus the `semprini-data-domain` template, and registers it into the
+ecosystem (Keycloak, CoreDNS, Uptime-Kuma, Prometheus, OpenMetadata).
+
+Everything is driven from the metadata + [domain.config.yml](domain.config.yml) by
+[instantiate.py](instantiate.py). This repo acts as the template's *data root*: the
+domain is scaffolded into `./domains/`, and a few platform-owned pieces are bridged
+in via repo-local symlinks (all gitignored).
+
+## Prerequisites
+
+- **Docker** running, and the shared external networks present: `semprini_bus`,
+  `semprini_internal`, `semprini_proxy` (created by the core + bus stacks).
+- **semprini-core** running (Keycloak, CoreDNS, Kuma, Prometheus, step-ca) and
+  **semprini-data** management zone (OpenMetadata, Apicurio). These live at
+  `../semprini-core` and `../semprini-data`.
+- `../semprini-data/.env` populated with admin creds (`KEYCLOAK_ADMIN*`,
+  `KUMA_API_KEY`, `KUMA_URL`, OpenMetadata config). The driver symlinks this in as
+  `./.env` and appends the domain's own secrets (generated, never committed).
+- **VPN** (headscale/tailscale) to reach the `*.financial-crime.data.internal` hosts.
+- Python deps: `pyyaml` (already in `.venv`).
+
+## One-command stand-up
+
+```bash
+make instantiate     # preflight → wire → secrets → scaffold → build → up → ducklake → register → activate → verify
+make verify          # re-run health checks
+make teardown        # unregister + stop (reverses everything; core/mgmt left clean)
+```
+
+Individual steps (useful for debugging): `make wire secrets scaffold build up
+ducklake register activate`. See `python3 deploy/instantiate.py --help`-style step
+list at the top of [instantiate.py](instantiate.py).
+
+## Metadata gate
+
+```bash
+make preflight        # validate the MD-DDL domain
+make install-hooks    # install a pre-commit hook that runs preflight on every commit
+```
+
+This project is not on GitHub, so the gate is local (a git pre-commit hook). If the
+local Gitea grows Actions later, mirror the same `preflight.py financial_crime` call.
+
+## Naming note
+
+The MD-DDL metadata folder is `financial_crime` (underscore). `new-domain.sh` requires
+a lowercase + hyphen name, so the **infra** domain name is `financial-crime`. Both
+uppercase to the same env prefix `FINANCIAL_CRIME`.
+
+## Known blocker (as of 2026-06-14)
+
+`make build` currently fails on the **postgres-ducklake** image: the template's
+`postgres/Dockerfile` downloads `pg_ducklake` / `pg_duckpipe` `.deb` packages from
+GitHub releases that return **404** — `duckdb/pg_ducklake` has no releases, and the
+real project `relytcloud/pg_ducklake` / `relytcloud/pg_duckpipe` have **zero published
+releases**. The `ariadne` image builds fine.
+
+This is a template (submodule) dependency issue, not a metadata issue — the submodule
+is not edited here; it should be raised upstream against `semprini-data-domain`. Once a
+valid artifact source (or prebuilt `semprini-postgres-ducklake:16` image) is available,
+`make instantiate` completes the live stand-up unchanged.
@@ -0,0 +1,37 @@
+# Deploy config for instantiating the Financial Crime domain.
+#
+# This is the one infra-facing input MD-DDL itself does not own. It maps the
+# metadata in this repo onto the semprini-data-domain template's instantiation
+# scripts. Read by deploy/instantiate.py.
+#
+# Data-root model: this repo (random_corp) IS the data root. The template
+# submodule's new-domain.sh resolves its data root as its own parent, so it
+# scaffolds into ./domains/, reads ./.env, and resolves ../semprini-core. The
+# driver bridges the few platform-owned pieces via repo-local symlinks (see
+# below): the build-context name, the shared .env, and the registration lib/.
+#
+# Naming note: the MD-DDL metadata folder is `financial_crime` (underscore),
+# but new-domain.sh requires lowercase + hyphens (^[a-z][a-z0-9-]+$), so the
+# infra domain name is `financial-crime`. Both uppercase to the same env
+# prefix FINANCIAL_CRIME (hyphens/underscores -> underscores).
+
+metadata_dir: financial_crime        # MD-DDL domain folder (preflight target)
+domain: financial-crime              # infra domain name passed to the template scripts
+port_base: 22000                     # API=22000, Flink=22001, MinIO=22002, Analytics=22003
+
+template_dir: .semprini-data-domain  # template submodule (new-domain.sh, domain-integration.sh)
+platform_repo: ../semprini-data      # source of the shared .env + lib/ (admin creds + integ helpers)
+
+# Repo-local symlinks the driver creates to satisfy the template's path
+# assumptions while keeping random_corp as the data root (all gitignored):
+#   semprini-data-domain -> .semprini-data-domain   (compose build contexts: ../../semprini-data-domain)
+#   .env                 -> ../semprini-data/.env    (shared secrets + admin creds)
+#   lib                  -> ../semprini-data/lib      (domain-integration.sh sources lib/core-integration.sh)
+
+# Secrets ensured in the shared .env (generated if missing, never committed).
+# Key names follow the template's DOMAIN_UPPER convention.
+secrets:
+  - POSTGRES_FINANCIAL_CRIME_PASSWORD
+  - MINIO_FINANCIAL_CRIME_ACCESS_KEY
+  - MINIO_FINANCIAL_CRIME_SECRET_KEY
+  - ANALYTICS_FINANCIAL_CRIME_PASSWORD
@@ -0,0 +1,20 @@
+#!/bin/bash
+# Pre-commit metadata gate: block commits that break the MD-DDL domain.
+# Install with: make install-hooks
+set -euo pipefail
+
+REPO_ROOT="$(git rev-parse --show-toplevel)"
+METADATA_DIR="$(
+  python3 - "$REPO_ROOT/deploy/domain.config.yml" <<'PY'
+import sys, yaml
+print(yaml.safe_load(open(sys.argv[1]))["metadata_dir"])
+PY
+)"
+
+echo "pre-commit: running MD-DDL pre-flight on '$METADATA_DIR'…"
+if ! python3 "$REPO_ROOT/.github/scripts/preflight.py" "$METADATA_DIR"; then
+    echo "pre-commit: metadata pre-flight FAILED — commit blocked." >&2
+    echo "  Fix the findings above, or bypass with 'git commit --no-verify'." >&2
+    exit 1
+fi
+echo "pre-commit: metadata OK."
@@ -0,0 +1,339 @@
+#!/usr/bin/env python3
+"""Metadata-driven instantiation of the Financial Crime data domain.
+
+This driver turns the MD-DDL metadata in this repo into a running, registered
+domain by orchestrating the semprini-data-domain template scripts. It does not
+re-implement them — it wires this repo as the template's "data root", ensures
+secrets, then calls new-domain.sh / docker compose / domain-integration.sh in
+the right order.
+
+Usage:
+    python3 deploy/instantiate.py <step> [<step> ...]
+    python3 deploy/instantiate.py all          # full stand-up
+    python3 deploy/instantiate.py teardown     # reverse it
+
+Steps (also runnable individually):
+    preflight  validate metadata (blocks on findings)
+    wire       create repo-local symlinks the template expects
+    secrets    ensure domain secrets exist in the shared .env
+    scaffold   new-domain.sh -> domains/<domain>/
+    build      build the postgres-ducklake + ariadne images
+    up         bring the stack up; wait for bucket + postgres health
+    ducklake   attach the DuckLake catalog in the analytics DB
+    register   register with core (Keycloak/DNS/Kuma/Prometheus) + mgmt (OpenMetadata)
+    activate   restart ariadne + flink to pick up OIDC + metrics
+    verify     health-check the running domain
+"""
+from __future__ import annotations
+
+import os
+import secrets as secretslib
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+import yaml
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+CONFIG_PATH = REPO_ROOT / "deploy" / "domain.config.yml"
+
+
+# --------------------------------------------------------------------------- #
+# helpers
+# --------------------------------------------------------------------------- #
+def log(msg: str) -> None:
+    print(f"\033[1;36m›\033[0m {msg}", flush=True)
+
+
+def ok(msg: str) -> None:
+    print(f"  \033[1;32m✓\033[0m {msg}", flush=True)
+
+
+def die(msg: str) -> None:
+    print(f"\033[1;31m✗ {msg}\033[0m", file=sys.stderr, flush=True)
+    sys.exit(1)
+
+
+def run(cmd: list[str], cwd: Path | None = None, check: bool = True,
+        capture: bool = False) -> subprocess.CompletedProcess:
+    printable = " ".join(cmd)
+    log(f"$ {printable}")
+    return subprocess.run(
+        cmd, cwd=str(cwd or REPO_ROOT), check=check,
+        text=True, capture_output=capture,
+    )
+
+
+class Cfg:
+    def __init__(self, raw: dict):
+        self.metadata_dir: str = raw["metadata_dir"]
+        self.domain: str = raw["domain"]
+        self.port_base: int = int(raw["port_base"])
+        self.template_dir: str = raw["template_dir"]
+        self.platform_repo: str = raw["platform_repo"]
+        self.secrets: list[str] = list(raw["secrets"])
+
+    @property
+    def upper(self) -> str:
+        return self.domain.replace("-", "_").upper()
+
+    @property
+    def template(self) -> Path:
+        return REPO_ROOT / self.template_dir
+
+    @property
+    def compose(self) -> Path:
+        return REPO_ROOT / "domains" / self.domain / "compose.yml"
+
+    @property
+    def env_file(self) -> Path:
+        return REPO_ROOT / ".env"
+
+
+def load_cfg() -> Cfg:
+    if not CONFIG_PATH.exists():
+        die(f"config not found: {CONFIG_PATH}")
+    return Cfg(yaml.safe_load(CONFIG_PATH.read_text()))
+
+
+def compose_cmd(cfg: Cfg, *args: str) -> list[str]:
+    return ["docker", "compose", "-f", str(cfg.compose), *args]
+
+
+def env_value(cfg: Cfg, key: str) -> str | None:
+    for line in cfg.env_file.read_text().splitlines():
+        if line.startswith(f"{key}="):
+            return line.split("=", 1)[1]
+    return None
+
+
+# --------------------------------------------------------------------------- #
+# steps
+# --------------------------------------------------------------------------- #
+def step_preflight(cfg: Cfg) -> None:
+    log("preflight: validating MD-DDL metadata")
+    checker = REPO_ROOT / ".github" / "scripts" / "preflight.py"
+    run([sys.executable, str(checker), cfg.metadata_dir])
+    ok("metadata pre-flight passed")
+
+
+def step_wire(cfg: Cfg) -> None:
+    """Repo-local symlinks so the template treats this repo as the data root."""
+    log("wire: creating repo-local symlinks")
+    links = {
+        # compose build contexts use ../../semprini-data-domain
+        "semprini-data-domain": cfg.template_dir,
+        # shared platform secrets + admin creds
+        ".env": f"{cfg.platform_repo}/.env",
+        # domain-integration.sh sources lib/core-integration.sh
+        "lib": f"{cfg.platform_repo}/lib",
+    }
+    for name, target in links.items():
+        link = REPO_ROOT / name
+        if link.is_symlink() or link.exists():
+            ok(f"{name} -> {os.readlink(link) if link.is_symlink() else '(exists)'}")
+            continue
+        link.symlink_to(target)
+        ok(f"created {name} -> {target}")
+    # ensure the bind-mount dirs ariadne expects are present
+    for d in ("schemas", "resolvers", "flink-jobs"):
+        (REPO_ROOT / "domains" / cfg.domain / d).mkdir(parents=True, exist_ok=True)
+
+
+def step_secrets(cfg: Cfg) -> None:
+    log("secrets: ensuring domain secrets in shared .env")
+    env_file = cfg.env_file
+    if not env_file.exists():
+        die(f"{env_file} missing (run 'wire' first; check {cfg.platform_repo}/.env)")
+    existing = {
+        line.split("=", 1)[0]
+        for line in env_file.read_text().splitlines()
+        if "=" in line and not line.startswith("#")
+    }
+    to_add: list[str] = []
+    for key in cfg.secrets:
+        if key in existing:
+            ok(f"{key} present")
+            continue
+        if "ACCESS_KEY" in key:
+            val = "fincrime" + secretslib.token_hex(8)
+        else:
+            val = secretslib.token_urlsafe(24)
+        to_add.append(f"{key}={val}")
+        ok(f"generated {key}")
+    port_key = f"{cfg.upper}_PORT_BASE"
+    if port_key not in existing:
+        to_add.append(f"{port_key}={cfg.port_base}")
+        ok(f"set {port_key}={cfg.port_base}")
+    if to_add:
+        with env_file.open("a") as fh:
+            fh.write("\n# --- " + cfg.domain + " domain secrets ---\n")
+            fh.write("\n".join(to_add) + "\n")
+
+
+def step_scaffold(cfg: Cfg) -> None:
+    log(f"scaffold: new-domain.sh {cfg.domain} {cfg.port_base}")
+    if cfg.compose.exists():
+        ok(f"already scaffolded at {cfg.compose.parent}")
+        return
+    run([str(cfg.template / "new-domain.sh"), cfg.domain, str(cfg.port_base)])
+    ok("domain scaffolded")
+
+
+def step_build(cfg: Cfg) -> None:
+    log("build: postgres-ducklake + ariadne images")
+    run(compose_cmd(cfg, "build",
+                    f"postgres-{cfg.domain}", f"ariadne-{cfg.domain}"))
+    ok("images built")
+
+
+def _wait(cfg: Cfg, desc: str, check_fn, timeout: int = 180, interval: int = 5) -> None:
+    log(f"waiting for {desc} (≤{timeout}s)")
+    deadline = time.time() + timeout
+    while time.time() < deadline:
+        if check_fn():
+            ok(desc)
+            return
+        time.sleep(interval)
+    die(f"timed out waiting for {desc}")
+
+
+def step_up(cfg: Cfg) -> None:
+    log("up: starting the domain stack")
+    run(compose_cmd(cfg, "up", "-d"))
+
+    def bucket_ready() -> bool:
+        r = subprocess.run(
+            ["docker", "inspect", "-f", "{{.State.Status}}:{{.State.ExitCode}}",
+             f"minio-init-{cfg.domain}"],
+            text=True, capture_output=True,
+        )
+        return r.stdout.strip() == "exited:0"
+
+    def pg_ready() -> bool:
+        r = subprocess.run(
+            ["docker", "inspect", "-f", "{{.State.Health.Status}}",
+             f"postgres-{cfg.domain}"],
+            text=True, capture_output=True,
+        )
+        return r.stdout.strip() == "healthy"
+
+    _wait(cfg, f"{cfg.domain}-iceberg bucket (minio-init exit 0)", bucket_ready)
+    _wait(cfg, f"postgres-{cfg.domain} healthy", pg_ready)
+
+
+def step_ducklake(cfg: Cfg) -> None:
+    """Post-startup: attach the DuckLake catalog inside the analytics DB."""
+    log("ducklake: attaching catalog in analytics DB")
+    access = env_value(cfg, f"MINIO_{cfg.upper}_ACCESS_KEY")
+    secret = env_value(cfg, f"MINIO_{cfg.upper}_SECRET_KEY")
+    pg_pw = env_value(cfg, f"POSTGRES_{cfg.upper}_PASSWORD")
+    if not all([access, secret, pg_pw]):
+        die("missing MinIO/postgres secrets in .env for analytics attach")
+    sql = f"""
+SELECT duckdb.create_secret(
+    '{cfg.domain}_minio', 'S3',
+    key_id := '{access}', secret := '{secret}',
+    endpoint := 'minio-{cfg.domain}:9000', url_style := 'path', use_ssl := false
+);
+SELECT duckdb.raw_query($q$
+    ATTACH 'ducklake:postgres:host=postgres-{cfg.domain} port=5432 dbname={cfg.domain} user={cfg.domain} password={pg_pw}'
+    AS "{cfg.domain}_lake";
+$q$);
+"""
+    # analytics service runs as user 'analytics', db '<domain>'
+    run([
+        "docker", "exec", "-i", f"analytics-{cfg.domain}",
+        "psql", "-v", "ON_ERROR_STOP=1", "-U", "analytics", "-d", cfg.domain,
+    ] + ["-c", sql], check=False)
+    ok("analytics DuckLake attach attempted (idempotent; safe to re-run)")
+
+
+def step_register(cfg: Cfg) -> None:
+    log("register: core + management (reversible, one-way)")
+    run([str(cfg.template / "domain-integration.sh"), cfg.domain, "register", "all"])
+    ok("registered with core + management")
+
+
+def step_activate(cfg: Cfg) -> None:
+    log("activate: restart ariadne + flink to pick up OIDC + metrics")
+    run(compose_cmd(cfg, "restart",
+                    f"ariadne-{cfg.domain}", f"flink-jm-{cfg.domain}"))
+    ok("ariadne + flink restarted")
+
+
+def step_verify(cfg: Cfg) -> None:
+    log("verify: checking domain stability")
+    expected = [f"{p}-{cfg.domain}" for p in
+                ("postgres", "kafka", "flink-jm", "flink-tm", "minio", "analytics", "ariadne")]
+    r = subprocess.run(["docker", "ps", "--format", "{{.Names}}"],
+                       text=True, capture_output=True)
+    running = set(r.stdout.split())
+    missing = [c for c in expected if c not in running]
+    if missing:
+        die(f"services not running: {', '.join(missing)}")
+    ok(f"{len(expected)} services running: {', '.join(expected)}")
+
+    # MinIO bucket
+    rc = subprocess.run(
+        ["docker", "exec", f"minio-{cfg.domain}", "sh", "-c",
+         f"mc alias set local http://localhost:9000 $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD >/dev/null 2>&1; "
+         f"mc ls local/{cfg.domain}-iceberg >/dev/null 2>&1 && echo OK"],
+        text=True, capture_output=True,
+    )
+    ok(f"{cfg.domain}-iceberg bucket present") if "OK" in rc.stdout else \
+        print(f"  ! could not confirm bucket (non-fatal): {rc.stderr.strip()}")
+
+    print("\n  Next (VPN required) — manual spot checks:")
+    print(f"    GraphQL : http://api.{cfg.domain}.data.internal:{cfg.port_base}/health")
+    print(f"    Flink   : http://flink.{cfg.domain}.data.internal:{cfg.port_base + 1}")
+    print(f"    Keycloak: client data-{cfg.domain} + /data-{cfg.domain}/* groups")
+    print(f"    Catalog : OpenMetadata domain '{cfg.domain}'")
+
+
+def step_teardown(cfg: Cfg) -> None:
+    log("teardown: unregister + stop (reverses the stand-up)")
+    run([str(cfg.template / "domain-integration.sh"), cfg.domain, "unregister", "all"],
+        check=False)
+    run(compose_cmd(cfg, "down", "-v"), check=False)
+    ok("domain unregistered + stopped")
+
+
+STEPS = {
+    "preflight": step_preflight,
+    "wire": step_wire,
+    "secrets": step_secrets,
+    "scaffold": step_scaffold,
+    "build": step_build,
+    "up": step_up,
+    "ducklake": step_ducklake,
+    "register": step_register,
+    "activate": step_activate,
+    "verify": step_verify,
+    "teardown": step_teardown,
+}
+
+ALL = ["preflight", "wire", "secrets", "scaffold", "build", "up",
+       "ducklake", "register", "activate", "verify"]
+
+
+def main() -> None:
+    args = sys.argv[1:] or ["all"]
+    requested: list[str] = []
+    for a in args:
+        if a == "all":
+            requested.extend(ALL)
+        elif a in STEPS:
+            requested.append(a)
+        else:
+            die(f"unknown step '{a}'. valid: {', '.join(STEPS)}, all")
+    cfg = load_cfg()
+    log(f"domain={cfg.domain}  metadata={cfg.metadata_dir}  port_base={cfg.port_base}")
+    for name in requested:
+        STEPS[name](cfg)
+    log("done")
+
+
+if __name__ == "__main__":
+    main()