css-test/instruct_kerberos.md

Kerberos auth Airflow supports kerberos auth. To configure it we need standard Service account (gMSA is not supported). in this example BRANCH\zGDSAppDEV user will be used to configure arflow follow next steps:

use airflow user to perform configuration

sudo -iu airflow pwd you should be located in home directory of the airflow user

we now need to create keytab file which will be used to authenticate our service account

ktutil add_entry -password -p zGDSAppDEV@PROD.ASBGROUP.CO.NZ -k 1 -e aes256-cts-hmac-sha1-96 wkt airflow.keytab quit have your service account ready as above commands will ask to enter service account password

check that file was created and you can see content of keytab file which holds Pricipal klist -ekt airflow.keytab

initiate ticket for airflow user

sudo -u airflow kinit zAirflowDEV@PROD.ASBGROUP.CO.NZ -V -k -t /opt/airflow/airflow.keytab -c /opt/airflow/airflow-krb5-ticket.cache now we need configure airflow to use this keytab file. Open airflow configuration file vim airflow.cfg

Then change following keys in config file

security = kerberos ccache = /tmp/airflow_krb5_ccache principal = zAirflowDEV@PROD.ASBGROUP.CO.NZ keytab = /home/airflow/airflow.keytab using your cmf account restart both airflow-scheduler and airflow-webserver services sudo systemctl restart airflow-scheduler airflow-webserver

test that ticket is assigned to airflow user sudo -u airflow klist

configure systemd unit to renew kerberos ticket periodically sudo vim /etc/systemd/system/airflow-kerberos-renewal.service

[Unit] Description=Airflow Kerberos Ticket Renewal Process Requires=network-online.target After=network-online.target

[Service] Type=simple Environment="PATH=$PATH:/opt/airflow/.venv:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin" Environment="AIRFLOW_HOME=/opt/airflow" User=airflow Group=airflow ExecStart=/usr/bin/bash -c 'source /opt/airflow/.venv/bin/activate ; airflow kerberos' Restart=always RestartSec=5s PrivateTmp=true

[Install] WantedBy=multi-user.target Reload daemons sudo systemctl daemon-reload

Start service sudo systemctl enable --now airflow-kerberos-renewal

Check status sudo systemctl status airflow-kerberos-renewal

Useful information for troubleshooting in journal sudo journalctl -u airflow-kerberos-renewal -f

#!/usr/bin/env bash
# =============================================================================
# deploy.sh — Recon Ranger deployment orchestrator
# =============================================================================
#
# Usage:  sudo ./deploy.sh [--clean] [path/to/deploy.conf]
#
# Options:
#   --clean    Remove and recreate the Python venv before installing deps.
#              Useful when a previous deploy left the venv in a broken state.
#
# This script sources deploy.conf for all configuration, then runs each
# deployment step in order. It uses a roll-forward strategy: if a step fails,
# the error is logged and the remaining steps continue.
#
# See deploy.conf.example for the full list of configuration variables.
# =============================================================================

set -uo pipefail

# ---------------------------------------------------------------------------
# Resolve our own location so relative paths work regardless of cwd.
# ---------------------------------------------------------------------------
DEPLOY_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# ---------------------------------------------------------------------------
# Load shared functions
# ---------------------------------------------------------------------------
source "${DEPLOY_DIR}/lib/common.sh"

# ---------------------------------------------------------------------------
# Parse arguments
# ---------------------------------------------------------------------------
DEPLOY_CLEAN=false

while [[ $# -gt 0 ]]; do
    case "$1" in
        --clean) DEPLOY_CLEAN=true; shift ;;
        *)       break ;;
    esac
done

# ---------------------------------------------------------------------------
# Root check
# ---------------------------------------------------------------------------
if [[ $EUID -ne 0 ]]; then
    log_error "This script must be run as root (or via sudo)"
    return 1 2>/dev/null || exit 1
fi

# ---------------------------------------------------------------------------
# Load configuration
# ---------------------------------------------------------------------------
CONF_FILE="${1:-${DEPLOY_DIR}/deploy.conf}"
# Resolve to absolute path so step scripts can find it regardless of cwd.
CONF_FILE="$(readlink -f "$CONF_FILE")"

if [[ ! -f "$CONF_FILE" ]]; then
    log_error "Config file not found: $CONF_FILE"
    log_error "Copy deploy.conf.example to deploy.conf and fill in your values."
    return 1 2>/dev/null || exit 1
fi

log_info "Loading configuration from: $CONF_FILE"
source "$CONF_FILE"

# ---------------------------------------------------------------------------
# Resolve APP_ROOT (repo root + optional subdirectory)
# ---------------------------------------------------------------------------
if [[ -n "${APP_SUBDIR:-}" ]]; then
    APP_ROOT="${APP_DIR}/${APP_SUBDIR}"
    log_info "APP_SUBDIR set — app root resolved to: $APP_ROOT"
else
    APP_ROOT="${APP_DIR}"
fi

# Track whether tests passed — used to gate service restart
DEPLOY_TESTS_PASSED=true

# ---------------------------------------------------------------------------
# Run deployment steps
# ---------------------------------------------------------------------------

log_step "Application Deployment — $(date)"

run_step "0. Proxy Setup"          "${DEPLOY_DIR}/steps/00-proxy-setup.sh"
run_step "1. User Setup"           "${DEPLOY_DIR}/steps/01-user-setup.sh"

if [[ "${APP_SSL_ENABLED:-true}" == "true" ]]; then
    run_step "2. SSL Certificates" "${DEPLOY_DIR}/steps/02-ssl-certs.sh"
else
    log_info "Skipping SSL certificates (APP_SSL_ENABLED=false)"
fi

run_step "3. App Install"          "${DEPLOY_DIR}/steps/03-app-install.sh"
run_step "3a. App Environment"     "${DEPLOY_DIR}/steps/03a-app-env.sh"
run_step "3b. DB Migrations"       "${DEPLOY_DIR}/steps/03b-db-migrate.sh"
run_step "3c. Tests"               "${DEPLOY_DIR}/steps/03c-tests.sh"
run_step "4. Firewall"             "${DEPLOY_DIR}/steps/04-firewall.sh"

if [[ -n "${CIFS_MOUNTS:-}" ]]; then
    run_step "5. Network Mounts"   "${DEPLOY_DIR}/steps/05-network-mounts.sh"
else
    log_info "Skipping network mounts (CIFS_MOUNTS not set)"
fi

run_step "6. App Service"          "${DEPLOY_DIR}/steps/06-app-service.sh"

# ---------------------------------------------------------------------------
# Summary
# ---------------------------------------------------------------------------
echo ""
echo "====================================================================="
if [[ $DEPLOY_HAS_ERRORS -ne 0 ]]; then
    log_warn "Deployment completed with errors — review the log above,"
    log_warn "fix the issue(s), and re-run this script."
    return 1 2>/dev/null || exit 1
else
    log_info "Deployment completed successfully."
fi
echo "====================================================================="

#!/usr/bin/env bash
# =============================================================================
# 01-user-setup.sh — Create the application system user and group
# =============================================================================

require_vars APP_USER APP_GROUP || return 1

# ---- Group ----
if getent group "$APP_GROUP" &>/dev/null; then
    log_info "Group already exists: $APP_GROUP"
else
    log_info "Creating system group: $APP_GROUP"
    groupadd --system "$APP_GROUP" || { log_error "Failed to create group: $APP_GROUP"; return 1; }
fi

# ---- User ----
if id "$APP_USER" &>/dev/null; then
    log_info "User already exists: $APP_USER"
else
    log_info "Creating system user: $APP_USER (group: $APP_GROUP, shell: /sbin/nologin)"
    useradd \
        --system \
        --gid "$APP_GROUP" \
        --shell /sbin/nologin \
        --no-create-home \
        "$APP_USER" \
        || { log_error "Failed to create user: $APP_USER"; return 1; }
fi

#!/usr/bin/env bash
# =============================================================================
# 06-app-service.sh — Create and enable the systemd service unit
# =============================================================================

require_vars APP_SERVICE_NAME APP_ROOT APP_DIR APP_ENV_DIR APP_HOST APP_PORT \
             APP_USER APP_GROUP APP_MODULE || return 1

# APP_ENV_DIR defaults to APP_ROOT if not explicitly set in config
APP_ENV_DIR="${APP_ENV_DIR:-$APP_ROOT}"

UNIT_FILE="/etc/systemd/system/${APP_SERVICE_NAME}.service"

# ---- Build the After= line ----
AFTER_TARGETS="network.target"
if [[ -n "${CIFS_MOUNTS:-}" ]]; then
    AFTER_TARGETS="network.target remote-fs.target"
fi

# ---- Capability for privileged ports ----
CAP_LINE=""
if (( APP_PORT < 1024 )); then
    CAP_LINE="AmbientCapabilities=CAP_NET_BIND_SERVICE"
    log_info "Port ${APP_PORT} < 1024 — adding CAP_NET_BIND_SERVICE"
fi

# ---- Verify uvicorn is installed ----
if ! "${APP_ROOT}/.venv/bin/python" -c "import uvicorn" &>/dev/null; then
    log_error "uvicorn not importable in ${APP_ROOT}/.venv — was step 3 (App Install) successful?"
    return 1
fi

# ---- Build ExecStart command ----
# Use the venv python to run uvicorn as a module, avoiding shebang path issues.
EXEC_START="${APP_ROOT}/.venv/bin/python -m uvicorn ${APP_MODULE} --host ${APP_HOST} --port ${APP_PORT}"

if [[ "${APP_SSL_ENABLED:-true}" == "true" ]]; then
    EXEC_START="${EXEC_START} --ssl-keyfile \${SSL_KEYFILE} --ssl-certfile \${SSL_CERTFILE}"
fi

# ---- Write the unit file ----
log_info "Writing systemd unit file: $UNIT_FILE"
cat > "$UNIT_FILE" <<UNITEOF
[Unit]
Description=${APP_SERVICE_NAME} FastAPI Application
After=${AFTER_TARGETS}

[Service]
Type=simple
User=${APP_USER}
Group=${APP_GROUP}
WorkingDirectory=${APP_DIR}
EnvironmentFile=${APP_ENV_DIR}/.env
ExecStart=${EXEC_START}
Restart=on-failure
RestartSec=5
${CAP_LINE}

[Install]
WantedBy=multi-user.target
UNITEOF

# ---- Enable and restart ----
log_info "Reloading systemd and enabling ${APP_SERVICE_NAME}"
systemctl daemon-reload \
    || { log_error "systemctl daemon-reload failed"; return 1; }

systemctl enable "$APP_SERVICE_NAME" \
    || { log_error "Failed to enable ${APP_SERVICE_NAME}"; return 1; }

# ---- Gate restart on test results ----
if [[ "${DEPLOY_TESTS_PASSED}" != "true" ]]; then
    log_warn "Tests failed — skipping service restart to preserve the current running version"
    log_warn "Fix the failing tests and re-run the deployment to restart the service"
    return 1
fi

log_info "Restarting ${APP_SERVICE_NAME}"
systemctl restart "$APP_SERVICE_NAME" \
    || { log_error "Failed to restart ${APP_SERVICE_NAME}"; return 1; }

systemctl --no-pager status "$APP_SERVICE_NAME"

Files created steps/05a-kerberos.sh — new deploy step (style matches 01-user-setup.sh and 06-app-service.sh). scripts/mssql_probe.py — SQLAlchemy integrated-auth probe. Wire it into deploy.sh Insert between the network-mounts and app-service lines:

if ; then run_step "5a. Kerberos" "${DEPLOY_DIR}/steps/05a-kerberos.sh" else log_info "Skipping Kerberos setup (KRB_ENABLED=false)" fi Put it before 6. App Service so the ticket exists when the app starts.

New deploy.conf variables Var Purpose Example KRB_ENABLED Gates the whole step true KRB_PRINCIPAL Service principal zReconRangerDEV@PROD.ASBGROUP.CO.NZ KRB_KEYTAB_SRC Path on the deployer where the keytab has been pre-staged (e.g. from a secret store) /root/secrets/recon-ranger.keytab KRB_KEYTAB_PATH Where the step installs it (owned by APP_USER, mode 0600) /opt/recon-ranger/recon-ranger.keytab KRB_CCACHE_PATH Ticket cache on disk, shared with the app service /var/lib/recon-ranger/krb5_ccache KRB_RENEW_INTERVAL Systemd timer OnUnitActiveSec (optional, default 30min) 30min Generate the keytab once (on a trusted host) with ktutil — same pattern as the airflow doc — then copy it to $KRB_KEYTAB_SRC before running deploy.

Runtime env vars for the app (.env, loaded by the app service)

KRB5CCNAME=FILE:/var/lib/recon-ranger/krb5_ccache MSSQL_HOST=sql01.prod.example MSSQL_PORT=1433 MSSQL_DB=ReconRanger MSSQL_ODBC_DRIVER=ODBC Driver 18 for SQL Server The existing 06-app-service.sh already does EnvironmentFile=${APP_ENV_DIR}/.env, so SQLAlchemy/pyodbc will pick up KRB5CCNAME automatically — no changes to that unit needed.

Python deps for the probe + app Add to your app's pyproject.toml:

sqlalchemy pyodbc And on the RHEL host you'll need the MS ODBC driver (msodbcsql18) + unixODBC-devel before pyodbc can build/install. Those belong in your 03-app-install.sh or a new prereqs step.

How the renewal works The step installs two units: a .service (Type=oneshot, runs kinit -k -t … -c FILE:…) and a .timer that fires it every KRB_RENEW_INTERVAL. This is cleaner than airflow's Type=simple loop — systemd tracks each kinit invocation individually, so failures show up clearly in journalctl -u -kerberos-renewal.service.

Verify after deploy:

systemctl list-timers | grep kerberos sudo -u recon-ranger klist -c FILE:/var/lib/recon-ranger/krb5_ccache sudo -u recon-ranger KRB5CCNAME=FILE:/var/lib/recon-ranger/krb5_ccache
/opt/recon-ranger/.venv/bin/python /opt/recon-ranger/scripts/mssql_probe.py