362 lines
13 KiB
Markdown
362 lines
13 KiB
Markdown
Kerberos auth
|
|
Airflow supports kerberos auth. To configure it we need standard Service account (gMSA is not supported). in this example BRANCH\zGDSAppDEV user will be used
|
|
to configure arflow follow next steps:
|
|
|
|
use airflow user to perform configuration
|
|
|
|
sudo -iu airflow
|
|
pwd
|
|
you should be located in home directory of the airflow user
|
|
|
|
we now need to create keytab file which will be used to authenticate our service account
|
|
|
|
ktutil
|
|
add_entry -password -p zGDSAppDEV@PROD.ASBGROUP.CO.NZ -k 1 -e aes256-cts-hmac-sha1-96
|
|
wkt airflow.keytab
|
|
quit
|
|
have your service account ready as above commands will ask to enter service account password
|
|
|
|
check that file was created and you can see content of keytab file which holds Pricipal
|
|
klist -ekt airflow.keytab
|
|
|
|
initiate ticket for airflow user
|
|
|
|
|
|
sudo -u airflow kinit zAirflowDEV@PROD.ASBGROUP.CO.NZ -V -k -t /opt/airflow/airflow.keytab -c /opt/airflow/airflow-krb5-ticket.cache
|
|
now we need configure airflow to use this keytab file. Open airflow configuration file
|
|
vim airflow.cfg
|
|
|
|
Then change following keys in config file
|
|
|
|
|
|
security = kerberos
|
|
ccache = /tmp/airflow_krb5_ccache
|
|
principal = zAirflowDEV@PROD.ASBGROUP.CO.NZ
|
|
keytab = /home/airflow/airflow.keytab
|
|
using your cmf account restart both airflow-scheduler and airflow-webserver services
|
|
sudo systemctl restart airflow-scheduler airflow-webserver
|
|
|
|
test that ticket is assigned to airflow user
|
|
sudo -u airflow klist
|
|
|
|
configure systemd unit to renew kerberos ticket periodically sudo vim /etc/systemd/system/airflow-kerberos-renewal.service
|
|
|
|
|
|
[Unit]
|
|
Description=Airflow Kerberos Ticket Renewal Process
|
|
Requires=network-online.target
|
|
After=network-online.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
Environment="PATH=$PATH:/opt/airflow/.venv:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"
|
|
Environment="AIRFLOW_HOME=/opt/airflow"
|
|
User=airflow
|
|
Group=airflow
|
|
ExecStart=/usr/bin/bash -c 'source /opt/airflow/.venv/bin/activate ; airflow kerberos'
|
|
Restart=always
|
|
RestartSec=5s
|
|
PrivateTmp=true
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
Reload daemons sudo systemctl daemon-reload
|
|
|
|
Start service sudo systemctl enable --now airflow-kerberos-renewal
|
|
|
|
Check status sudo systemctl status airflow-kerberos-renewal
|
|
|
|
Useful information for troubleshooting in journal sudo journalctl -u airflow-kerberos-renewal -f
|
|
|
|
```bash
|
|
#!/usr/bin/env bash
|
|
# =============================================================================
|
|
# deploy.sh — Recon Ranger deployment orchestrator
|
|
# =============================================================================
|
|
#
|
|
# Usage: sudo ./deploy.sh [--clean] [path/to/deploy.conf]
|
|
#
|
|
# Options:
|
|
# --clean Remove and recreate the Python venv before installing deps.
|
|
# Useful when a previous deploy left the venv in a broken state.
|
|
#
|
|
# This script sources deploy.conf for all configuration, then runs each
|
|
# deployment step in order. It uses a roll-forward strategy: if a step fails,
|
|
# the error is logged and the remaining steps continue.
|
|
#
|
|
# See deploy.conf.example for the full list of configuration variables.
|
|
# =============================================================================
|
|
|
|
set -uo pipefail
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Resolve our own location so relative paths work regardless of cwd.
|
|
# ---------------------------------------------------------------------------
|
|
DEPLOY_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Load shared functions
|
|
# ---------------------------------------------------------------------------
|
|
source "${DEPLOY_DIR}/lib/common.sh"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Parse arguments
|
|
# ---------------------------------------------------------------------------
|
|
DEPLOY_CLEAN=false
|
|
|
|
while [[ $# -gt 0 ]]; do
|
|
case "$1" in
|
|
--clean) DEPLOY_CLEAN=true; shift ;;
|
|
*) break ;;
|
|
esac
|
|
done
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Root check
|
|
# ---------------------------------------------------------------------------
|
|
if [[ $EUID -ne 0 ]]; then
|
|
log_error "This script must be run as root (or via sudo)"
|
|
return 1 2>/dev/null || exit 1
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Load configuration
|
|
# ---------------------------------------------------------------------------
|
|
CONF_FILE="${1:-${DEPLOY_DIR}/deploy.conf}"
|
|
# Resolve to absolute path so step scripts can find it regardless of cwd.
|
|
CONF_FILE="$(readlink -f "$CONF_FILE")"
|
|
|
|
if [[ ! -f "$CONF_FILE" ]]; then
|
|
log_error "Config file not found: $CONF_FILE"
|
|
log_error "Copy deploy.conf.example to deploy.conf and fill in your values."
|
|
return 1 2>/dev/null || exit 1
|
|
fi
|
|
|
|
log_info "Loading configuration from: $CONF_FILE"
|
|
source "$CONF_FILE"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Resolve APP_ROOT (repo root + optional subdirectory)
|
|
# ---------------------------------------------------------------------------
|
|
if [[ -n "${APP_SUBDIR:-}" ]]; then
|
|
APP_ROOT="${APP_DIR}/${APP_SUBDIR}"
|
|
log_info "APP_SUBDIR set — app root resolved to: $APP_ROOT"
|
|
else
|
|
APP_ROOT="${APP_DIR}"
|
|
fi
|
|
|
|
# Track whether tests passed — used to gate service restart
|
|
DEPLOY_TESTS_PASSED=true
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Run deployment steps
|
|
# ---------------------------------------------------------------------------
|
|
|
|
log_step "Application Deployment — $(date)"
|
|
|
|
run_step "0. Proxy Setup" "${DEPLOY_DIR}/steps/00-proxy-setup.sh"
|
|
run_step "1. User Setup" "${DEPLOY_DIR}/steps/01-user-setup.sh"
|
|
|
|
if [[ "${APP_SSL_ENABLED:-true}" == "true" ]]; then
|
|
run_step "2. SSL Certificates" "${DEPLOY_DIR}/steps/02-ssl-certs.sh"
|
|
else
|
|
log_info "Skipping SSL certificates (APP_SSL_ENABLED=false)"
|
|
fi
|
|
|
|
run_step "3. App Install" "${DEPLOY_DIR}/steps/03-app-install.sh"
|
|
run_step "3a. App Environment" "${DEPLOY_DIR}/steps/03a-app-env.sh"
|
|
run_step "3b. DB Migrations" "${DEPLOY_DIR}/steps/03b-db-migrate.sh"
|
|
run_step "3c. Tests" "${DEPLOY_DIR}/steps/03c-tests.sh"
|
|
run_step "4. Firewall" "${DEPLOY_DIR}/steps/04-firewall.sh"
|
|
|
|
if [[ -n "${CIFS_MOUNTS:-}" ]]; then
|
|
run_step "5. Network Mounts" "${DEPLOY_DIR}/steps/05-network-mounts.sh"
|
|
else
|
|
log_info "Skipping network mounts (CIFS_MOUNTS not set)"
|
|
fi
|
|
|
|
run_step "6. App Service" "${DEPLOY_DIR}/steps/06-app-service.sh"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Summary
|
|
# ---------------------------------------------------------------------------
|
|
echo ""
|
|
echo "====================================================================="
|
|
if [[ $DEPLOY_HAS_ERRORS -ne 0 ]]; then
|
|
log_warn "Deployment completed with errors — review the log above,"
|
|
log_warn "fix the issue(s), and re-run this script."
|
|
return 1 2>/dev/null || exit 1
|
|
else
|
|
log_info "Deployment completed successfully."
|
|
fi
|
|
echo "====================================================================="
|
|
```
|
|
|
|
```bash
|
|
#!/usr/bin/env bash
|
|
# =============================================================================
|
|
# 01-user-setup.sh — Create the application system user and group
|
|
# =============================================================================
|
|
|
|
require_vars APP_USER APP_GROUP || return 1
|
|
|
|
# ---- Group ----
|
|
if getent group "$APP_GROUP" &>/dev/null; then
|
|
log_info "Group already exists: $APP_GROUP"
|
|
else
|
|
log_info "Creating system group: $APP_GROUP"
|
|
groupadd --system "$APP_GROUP" || { log_error "Failed to create group: $APP_GROUP"; return 1; }
|
|
fi
|
|
|
|
# ---- User ----
|
|
if id "$APP_USER" &>/dev/null; then
|
|
log_info "User already exists: $APP_USER"
|
|
else
|
|
log_info "Creating system user: $APP_USER (group: $APP_GROUP, shell: /sbin/nologin)"
|
|
useradd \
|
|
--system \
|
|
--gid "$APP_GROUP" \
|
|
--shell /sbin/nologin \
|
|
--no-create-home \
|
|
"$APP_USER" \
|
|
|| { log_error "Failed to create user: $APP_USER"; return 1; }
|
|
fi
|
|
```
|
|
|
|
```bash
|
|
#!/usr/bin/env bash
|
|
# =============================================================================
|
|
# 06-app-service.sh — Create and enable the systemd service unit
|
|
# =============================================================================
|
|
|
|
require_vars APP_SERVICE_NAME APP_ROOT APP_DIR APP_ENV_DIR APP_HOST APP_PORT \
|
|
APP_USER APP_GROUP APP_MODULE || return 1
|
|
|
|
# APP_ENV_DIR defaults to APP_ROOT if not explicitly set in config
|
|
APP_ENV_DIR="${APP_ENV_DIR:-$APP_ROOT}"
|
|
|
|
UNIT_FILE="/etc/systemd/system/${APP_SERVICE_NAME}.service"
|
|
|
|
# ---- Build the After= line ----
|
|
AFTER_TARGETS="network.target"
|
|
if [[ -n "${CIFS_MOUNTS:-}" ]]; then
|
|
AFTER_TARGETS="network.target remote-fs.target"
|
|
fi
|
|
|
|
# ---- Capability for privileged ports ----
|
|
CAP_LINE=""
|
|
if (( APP_PORT < 1024 )); then
|
|
CAP_LINE="AmbientCapabilities=CAP_NET_BIND_SERVICE"
|
|
log_info "Port ${APP_PORT} < 1024 — adding CAP_NET_BIND_SERVICE"
|
|
fi
|
|
|
|
# ---- Verify uvicorn is installed ----
|
|
if ! "${APP_ROOT}/.venv/bin/python" -c "import uvicorn" &>/dev/null; then
|
|
log_error "uvicorn not importable in ${APP_ROOT}/.venv — was step 3 (App Install) successful?"
|
|
return 1
|
|
fi
|
|
|
|
# ---- Build ExecStart command ----
|
|
# Use the venv python to run uvicorn as a module, avoiding shebang path issues.
|
|
EXEC_START="${APP_ROOT}/.venv/bin/python -m uvicorn ${APP_MODULE} --host ${APP_HOST} --port ${APP_PORT}"
|
|
|
|
if [[ "${APP_SSL_ENABLED:-true}" == "true" ]]; then
|
|
EXEC_START="${EXEC_START} --ssl-keyfile \${SSL_KEYFILE} --ssl-certfile \${SSL_CERTFILE}"
|
|
fi
|
|
|
|
# ---- Write the unit file ----
|
|
log_info "Writing systemd unit file: $UNIT_FILE"
|
|
cat > "$UNIT_FILE" <<UNITEOF
|
|
[Unit]
|
|
Description=${APP_SERVICE_NAME} FastAPI Application
|
|
After=${AFTER_TARGETS}
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=${APP_USER}
|
|
Group=${APP_GROUP}
|
|
WorkingDirectory=${APP_DIR}
|
|
EnvironmentFile=${APP_ENV_DIR}/.env
|
|
ExecStart=${EXEC_START}
|
|
Restart=on-failure
|
|
RestartSec=5
|
|
${CAP_LINE}
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
UNITEOF
|
|
|
|
# ---- Enable and restart ----
|
|
log_info "Reloading systemd and enabling ${APP_SERVICE_NAME}"
|
|
systemctl daemon-reload \
|
|
|| { log_error "systemctl daemon-reload failed"; return 1; }
|
|
|
|
systemctl enable "$APP_SERVICE_NAME" \
|
|
|| { log_error "Failed to enable ${APP_SERVICE_NAME}"; return 1; }
|
|
|
|
# ---- Gate restart on test results ----
|
|
if [[ "${DEPLOY_TESTS_PASSED}" != "true" ]]; then
|
|
log_warn "Tests failed — skipping service restart to preserve the current running version"
|
|
log_warn "Fix the failing tests and re-run the deployment to restart the service"
|
|
return 1
|
|
fi
|
|
|
|
log_info "Restarting ${APP_SERVICE_NAME}"
|
|
systemctl restart "$APP_SERVICE_NAME" \
|
|
|| { log_error "Failed to restart ${APP_SERVICE_NAME}"; return 1; }
|
|
|
|
systemctl --no-pager status "$APP_SERVICE_NAME"
|
|
```
|
|
|
|
Files created
|
|
steps/05a-kerberos.sh — new deploy step (style matches 01-user-setup.sh and 06-app-service.sh).
|
|
scripts/mssql_probe.py — SQLAlchemy integrated-auth probe.
|
|
Wire it into deploy.sh
|
|
Insert between the network-mounts and app-service lines:
|
|
|
|
|
|
if [[ "${KRB_ENABLED:-false}" == "true" ]]; then
|
|
run_step "5a. Kerberos" "${DEPLOY_DIR}/steps/05a-kerberos.sh"
|
|
else
|
|
log_info "Skipping Kerberos setup (KRB_ENABLED=false)"
|
|
fi
|
|
Put it before 6. App Service so the ticket exists when the app starts.
|
|
|
|
New deploy.conf variables
|
|
Var Purpose Example
|
|
KRB_ENABLED Gates the whole step true
|
|
KRB_PRINCIPAL Service principal zReconRangerDEV@PROD.ASBGROUP.CO.NZ
|
|
KRB_KEYTAB_SRC Path on the deployer where the keytab has been pre-staged (e.g. from a secret store) /root/secrets/recon-ranger.keytab
|
|
KRB_KEYTAB_PATH Where the step installs it (owned by APP_USER, mode 0600) /opt/recon-ranger/recon-ranger.keytab
|
|
KRB_CCACHE_PATH Ticket cache on disk, shared with the app service /var/lib/recon-ranger/krb5_ccache
|
|
KRB_RENEW_INTERVAL Systemd timer OnUnitActiveSec (optional, default 30min) 30min
|
|
Generate the keytab once (on a trusted host) with ktutil — same pattern as the airflow doc — then copy it to $KRB_KEYTAB_SRC before running deploy.
|
|
|
|
Runtime env vars for the app (.env, loaded by the app service)
|
|
|
|
KRB5CCNAME=FILE:/var/lib/recon-ranger/krb5_ccache
|
|
MSSQL_HOST=sql01.prod.example
|
|
MSSQL_PORT=1433
|
|
MSSQL_DB=ReconRanger
|
|
MSSQL_ODBC_DRIVER=ODBC Driver 18 for SQL Server
|
|
The existing 06-app-service.sh already does EnvironmentFile=${APP_ENV_DIR}/.env, so SQLAlchemy/pyodbc will pick up KRB5CCNAME automatically — no changes to that unit needed.
|
|
|
|
Python deps for the probe + app
|
|
Add to your app's pyproject.toml:
|
|
|
|
|
|
sqlalchemy
|
|
pyodbc
|
|
And on the RHEL host you'll need the MS ODBC driver (msodbcsql18) + unixODBC-devel before pyodbc can build/install. Those belong in your 03-app-install.sh or a new prereqs step.
|
|
|
|
How the renewal works
|
|
The step installs two units: a .service (Type=oneshot, runs kinit -k -t … -c FILE:…) and a .timer that fires it every KRB_RENEW_INTERVAL. This is cleaner than airflow's Type=simple loop — systemd tracks each kinit invocation individually, so failures show up clearly in journalctl -u <service>-kerberos-renewal.service.
|
|
|
|
Verify after deploy:
|
|
|
|
|
|
systemctl list-timers | grep kerberos
|
|
sudo -u recon-ranger klist -c FILE:/var/lib/recon-ranger/krb5_ccache
|
|
sudo -u recon-ranger KRB5CCNAME=FILE:/var/lib/recon-ranger/krb5_ccache \
|
|
/opt/recon-ranger/.venv/bin/python /opt/recon-ranger/scripts/mssql_probe.py
|
|
|