diff --git a/.gitattributes b/.gitattributes index 4fd0c6b..a9bddbd 100644 --- a/.gitattributes +++ b/.gitattributes @@ -5,3 +5,7 @@ *.sh text eol=lf *.yml text eol=lf *.yaml text eol=lf + +# systemd chokes on trailing CR in unit values. +*.service text eol=lf +*.timer text eol=lf diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 45bdb15..29e163a 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -83,27 +83,84 @@ Push to `main`, or run the workflow manually with a ref. The workflow refuses to --- -## 4. TLS +## 4. Domains and TLS — dynamic by default -No certificate is issued during provisioning, because certbot cannot validate a domain that does not yet point at the server. +Domains arrive continuously: one today, five tomorrow. Nothing here requires a person per domain. -Per domain, once its A record resolves here: +**HTTP already needs zero configuration.** The nginx catch-all serves *any* `Host`, and the SPA resolves its tenant from that header. Point a domain's A record at the server and it works over port 80 immediately. Only TLS needs a certificate per name — that is the whole problem this section solves. + +Two mechanisms, used together: + +### 4.1 Wildcard — tenants on our own apex + +One certificate covers every `.`. A new tenant subdomain is then live over HTTPS the moment DNS resolves, with **no certificate work at all**. + +```bash +sudo bash setup-wildcard-tls.sh \ + --apex marketplaces.example.com \ + --email ops@example.com \ + --dns cloudflare --creds /root/cloudflare.ini +``` + +Wildcards require DNS-01 validation, so certbot must write a `_acme-challenge` TXT record. With a provider plugin (`cloudflare`, `route53`) renewal is unattended. `--dns manual` works but prompts for a TXT record at **every** renewal — fine to prove the setup out, not acceptable as a steady state. + +**Hostinger has no certbot plugin.** If DNS lives there: either move DNS to a provider that has one (Cloudflare is free, minutes of work), or drive issuance from the [Phase 9](backend/PHASE-9-TENANT-REGISTRY-DOMAINS-CONTRACT.md) domain-automation API once it exists. + +### 4.2 Reconciler — tenants on their own domains + +A wildcard cannot cover a customer's own domain. `sync-domains.sh` runs on a 10-minute timer and reconciles the live set against a desired list: + +- issues certificates for domains that lack one +- skips domains whose certificate has more than 30 days left +- skips subdomains already covered by `WILDCARD_APEX` +- leaves domains alone while their DNS has not propagated yet, and retries next tick +- disables server blocks for domains removed from the source — **without deleting the certificate**, so re-adding one later is instant +- caps issuance per run, so a misconfigured source cannot burn the weekly ACME budget in a single pass + +Configure `/etc/marketplaces/domains.env`: + +```bash +DOMAINS_SOURCE=file:/etc/marketplaces/domains.txt +CERTBOT_EMAIL=ops@example.com +MAX_ISSUE_PER_RUN=10 +``` + +Then: + +```bash +sudo systemctl enable --now marketplaces-domains.timer +sudo /srv/marketplaces/bin/sync-domains.sh --dry-run # see the plan, change nothing +``` + +Adding a domain becomes: append a line to `/etc/marketplaces/domains.txt` (or add the row in the backend registry), point DNS, wait one tick. + +### 4.3 Backend-driven, once Phase 9 ships + +Point the reconciler at the registry instead of a file and the loop closes — `MarketplaceDomain` already carries exactly the statuses this needs (`planned → dns_pending → ssl_pending → active → failed`): + +```bash +DOMAINS_SOURCE=https://api.example.com/api/admin/v2/domains +DOMAINS_API_TOKEN=... +``` + +The script accepts a bare JSON array of hostnames, or objects with `domain` + `status`, in which case it acts only on `active` rows. **A fetch failure aborts the run rather than reading as "remove every domain."** + +### 4.4 One-off + +For a single domain, outside the reconciler: ```bash sudo bash add-domain.sh shop.example.com --email ops@example.com --with-www ``` -This writes a server block for that domain (same root — it exists only to give certbot a concrete `server_name`), issues the certificate, enables the HTTP→HTTPS redirect, and enables `certbot.timer` for renewal. - -Verify: +### 4.5 Verify ```bash curl -I https://shop.example.com/health sudo certbot certificates +journalctl -u marketplaces-domains.service --since "1 hour ago" ``` -**On wildcards:** `add-domain.sh` uses HTTP-01, which cannot issue wildcards. If tenants all live under one apex (`*.marketplaces.example.com`), a DNS-01 wildcard is fewer moving parts — but it requires API credentials for the DNS provider and a different certbot plugin. Not set up here; raise it when the tenant count makes per-domain issuance annoying. - --- ## 5. Rollback diff --git a/scripts/deploy/server-setup.sh b/scripts/deploy/server-setup.sh index a80eeab..b434655 100755 --- a/scripts/deploy/server-setup.sh +++ b/scripts/deploy/server-setup.sh @@ -29,7 +29,7 @@ done echo "==> packages" export DEBIAN_FRONTEND=noninteractive apt-get update -qq -apt-get install -y -qq nginx certbot python3-certbot-nginx rsync ufw +apt-get install -y -qq nginx certbot python3-certbot-nginx rsync ufw jq curl openssl echo "==> deploy user: $DEPLOY_USER" if ! id -u "$DEPLOY_USER" >/dev/null 2>&1; then @@ -133,6 +133,45 @@ nginx -t systemctl enable --now nginx systemctl reload nginx +echo "==> dynamic domain reconciler" +SRC_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +install -d -m 755 "$BASE/bin" /etc/marketplaces "/var/lib/marketplaces" +if [[ -f "$SRC_DIR/sync-domains.sh" ]]; then + install -m 755 "$SRC_DIR/sync-domains.sh" "$BASE/bin/sync-domains.sh" + + if [[ ! -f /etc/marketplaces/domains.env ]]; then + cat > /etc/marketplaces/domains.env <<'ENVFILE' +# Where the desired domain list comes from. +# file:/etc/marketplaces/domains.txt one hostname per line +# https://api.example.com/api/admin/v2/domains JSON, once the backend exists +DOMAINS_SOURCE=file:/etc/marketplaces/domains.txt + +# Required: certbot expiry notices. +CERTBOT_EMAIL= + +# Cap per run so a bad source cannot burn the weekly ACME budget in one pass. +MAX_ISSUE_PER_RUN=10 + +# Set by setup-wildcard-tls.sh. Subdomains of this apex skip per-domain issuance. +#WILDCARD_APEX= +ENVFILE + chmod 600 /etc/marketplaces/domains.env + fi + touch /etc/marketplaces/domains.txt + + if [[ -d "$SRC_DIR/systemd" ]]; then + install -m 644 "$SRC_DIR/systemd/marketplaces-domains.service" /etc/systemd/system/ + install -m 644 "$SRC_DIR/systemd/marketplaces-domains.timer" /etc/systemd/system/ + systemctl daemon-reload + # Not started yet: CERTBOT_EMAIL is still blank. Enable it after filling in + # /etc/marketplaces/domains.env, or the first run just fails on every tick. + echo " timer installed but NOT started - set CERTBOT_EMAIL first, then:" + echo " systemctl enable --now marketplaces-domains.timer" + fi +else + echo " sync-domains.sh not found next to this script - skipping" +fi + echo "==> sudoers: let the deploy user reload nginx, nothing else" cat > /etc/sudoers.d/marketplaces-deploy <. needs no certificate work at all — +# DNS record, and it is live over HTTPS immediately. +# +# setup-wildcard-tls.sh --apex marketplaces.example.com --email ops@example.com --dns cloudflare +# setup-wildcard-tls.sh --apex marketplaces.example.com --email ops@example.com --dns manual +# +# Wildcards require DNS-01 validation — HTTP-01 cannot issue them. That means +# certbot must create a _acme-challenge TXT record, which needs either a DNS +# provider plugin (automatic, renews unattended) or manual intervention every +# 60-90 days. Prefer a plugin. Use manual only to prove the idea out. +# +# Tenants on their OWN domains are not covered by a wildcard; those are handled +# per-domain by sync-domains.sh. + +set -euo pipefail + +APEX=""; EMAIL=""; DNS_PLUGIN="manual"; CREDS="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --apex) APEX="$2"; shift 2 ;; + --email) EMAIL="$2"; shift 2 ;; + --dns) DNS_PLUGIN="$2"; shift 2 ;; + --creds) CREDS="$2"; shift 2 ;; + *) echo "unknown argument: $1" >&2; exit 2 ;; + esac +done + +[[ $EUID -eq 0 ]] || { echo "must run as root" >&2; exit 1; } +[[ -n "$APEX" ]] || { echo "--apex is required" >&2; exit 2; } +[[ -n "$EMAIL" ]] || { echo "--email is required" >&2; exit 2; } + +export DEBIAN_FRONTEND=noninteractive +CERT_ARGS=(-d "$APEX" -d "*.$APEX") + +case "$DNS_PLUGIN" in + cloudflare) + apt-get install -y -qq python3-certbot-dns-cloudflare + [[ -n "$CREDS" ]] || { echo "--creds required for cloudflare (contains the API token)" >&2; exit 2; } + chmod 600 "$CREDS" + CERT_ARGS+=(--dns-cloudflare --dns-cloudflare-credentials "$CREDS" --dns-cloudflare-propagation-seconds 30) + ;; + route53) + apt-get install -y -qq python3-certbot-dns-route53 + CERT_ARGS+=(--dns-route53) # credentials come from the instance role or ~/.aws + ;; + manual) + cat >&2 <<'WARN' +WARNING: manual DNS-01. + +certbot will print a TXT record for you to create by hand, and will do so again +at every renewal (every 60-90 days). Unattended renewal will NOT work. This is +acceptable to prove the setup out; it is not acceptable as the steady state. + +Hostinger has no certbot plugin. If DNS lives there, the options are: move DNS +to a provider with a plugin (Cloudflare is free and takes minutes), or drive +issuance from the Phase 9 domain-automation API instead. + +WARN + CERT_ARGS+=(--manual --preferred-challenges dns) + ;; + *) + echo "unsupported --dns: $DNS_PLUGIN (cloudflare|route53|manual)" >&2; exit 2 ;; +esac + +echo "==> issuing wildcard for $APEX and *.$APEX via $DNS_PLUGIN" +certbot certonly "${CERT_ARGS[@]}" \ + --agree-tos --email "$EMAIL" --keep-until-expiring \ + $([[ "$DNS_PLUGIN" != "manual" ]] && echo --non-interactive) + +LIVE="/etc/letsencrypt/live/$APEX" +[[ -f "$LIVE/fullchain.pem" ]] || { echo "certificate not found at $LIVE" >&2; exit 1; } + +echo "==> nginx: TLS on the catch-all, so every subdomain is served immediately" +cat > /etc/nginx/snippets/marketplaces-wildcard-tls.conf < /etc/nginx/sites-available/marketplaces-tls.conf <> /etc/marketplaces/domains.env + +echo +echo "done. every .$APEX is now served over HTTPS with no further action." +echo "verify: curl -I https://anything.$APEX/health" diff --git a/scripts/deploy/sync-domains.sh b/scripts/deploy/sync-domains.sh new file mode 100644 index 0000000..215f68a --- /dev/null +++ b/scripts/deploy/sync-domains.sh @@ -0,0 +1,234 @@ +#!/usr/bin/env bash +# +# Reconcile the set of TLS-enabled domains on this server against a desired +# list. Idempotent and safe to run on a timer: it issues what is missing, +# leaves what is current alone, and disables what has been removed. +# +# The nginx catch-all already serves ANY Host over HTTP with no config, so a +# new domain works on port 80 the moment DNS resolves. This script exists only +# because TLS needs a certificate per name. +# +# sync-domains.sh # reconcile from $DOMAINS_SOURCE +# sync-domains.sh --dry-run # print the plan, change nothing +# +# Config: /etc/marketplaces/domains.env +# DOMAINS_SOURCE=file:/etc/marketplaces/domains.txt +# DOMAINS_SOURCE=https://api.example.com/api/admin/v2/domains (JSON array) +# CERTBOT_EMAIL=ops@example.com +# MAX_ISSUE_PER_RUN=10 +# +# Let's Encrypt caps new certificates per registered domain per week. The +# per-run issuance cap keeps a misconfigured source from burning that budget +# in one pass; the remainder is picked up on the next run. + +set -euo pipefail + +CONFIG="/etc/marketplaces/domains.env" +STATE_DIR="/var/lib/marketplaces" +DRY_RUN=0 +RENEW_WINDOW_DAYS=30 + +[[ "${1:-}" == "--dry-run" ]] && DRY_RUN=1 + +# shellcheck source=/dev/null +[[ -f "$CONFIG" ]] && source "$CONFIG" + +DOMAINS_SOURCE="${DOMAINS_SOURCE:-file:/etc/marketplaces/domains.txt}" +CERTBOT_EMAIL="${CERTBOT_EMAIL:-}" +MAX_ISSUE_PER_RUN="${MAX_ISSUE_PER_RUN:-10}" +WILDCARD_APEX="${WILDCARD_APEX:-}" + +[[ $EUID -eq 0 ]] || { echo "must run as root" >&2; exit 1; } +[[ -n "$CERTBOT_EMAIL" ]] || { echo "CERTBOT_EMAIL not set in $CONFIG" >&2; exit 1; } + +mkdir -p "$STATE_DIR" +log() { printf '%s %s\n' "$(date -Is)" "$*"; } + +# ---------------------------------------------------------------- desired set + +fetch_desired() { + case "$DOMAINS_SOURCE" in + file:*) + local path="${DOMAINS_SOURCE#file:}" + [[ -f "$path" ]] || { log "source file $path missing"; return 1; } + # one domain per line; # comments and blanks ignored + sed -e 's/#.*//' -e 's/[[:space:]]//g' "$path" | grep -v '^$' || true + ;; + http://*|https://*) + # Expected shape: ["a.example.com","b.example.com"] or + # [{"domain":"a.example.com","status":"active"}, ...] + local body + body="$(curl -fsS --max-time 20 ${DOMAINS_API_TOKEN:+-H "Authorization: Bearer $DOMAINS_API_TOKEN"} "$DOMAINS_SOURCE")" || { + log "ERROR: could not fetch $DOMAINS_SOURCE — leaving current config untouched" + return 1 + } + echo "$body" | jq -r ' + if type=="array" and (.[0]|type)=="object" + then .[] | select((.status // "active") == "active") | .domain + else .[] end' 2>/dev/null || { + log "ERROR: unparseable response from $DOMAINS_SOURCE" + return 1 + } + ;; + *) + log "ERROR: unsupported DOMAINS_SOURCE: $DOMAINS_SOURCE"; return 1 ;; + esac +} + +# A source failure must never be read as "all domains removed". Bail instead. +if ! DESIRED_RAW="$(fetch_desired)"; then + log "reconcile aborted: desired set unavailable" + exit 1 +fi + +# Reject anything that is not a plausible hostname before it reaches certbot +# or an nginx server_name. +DESIRED="$(echo "$DESIRED_RAW" | grep -Ei '^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+$' | sort -u || true)" +REJECTED="$(comm -23 <(echo "$DESIRED_RAW" | sort -u) <(echo "$DESIRED") || true)" +[[ -n "$REJECTED" ]] && log "WARNING: ignoring malformed entries: $(echo "$REJECTED" | tr '\n' ' ')" + +if [[ -z "$DESIRED" ]]; then + log "desired set is empty — nothing to do (not treating this as 'remove everything')" + exit 0 +fi + +log "desired: $(echo "$DESIRED" | wc -l) domain(s)" + +# ------------------------------------------------------------------ helpers + +covered_by_wildcard() { + # A domain one label under the wildcard apex needs no certificate of its own. + local d="$1" + [[ -n "$WILDCARD_APEX" ]] || return 1 + [[ "$d" == *".$WILDCARD_APEX" ]] || return 1 + [[ "${d%.$WILDCARD_APEX}" != *.* ]] +} + +cert_is_current() { + local d="$1" live="/etc/letsencrypt/live/$1/cert.pem" + [[ -f "$live" ]] || return 1 + openssl x509 -in "$live" -noout -checkend $((RENEW_WINDOW_DAYS * 86400)) >/dev/null 2>&1 +} + +resolves_here() { + local d="$1" + local got want + got="$(getent hosts "$d" | awk '{print $1}' | sort -u)" + [[ -n "$got" ]] || return 1 + # Compare against every address this host actually answers on. + want="$(hostname -I | tr ' ' '\n' | grep -v '^$')" + grep -qxF -f <(echo "$want") <(echo "$got") +} + +write_block() { + local d="$1" conf="/etc/nginx/sites-available/tenant-$1.conf" + cat > "$conf" </dev/null 2>&1; then + log "ERROR: nginx config invalid after adding $d — reverting that block" + rm -f "/etc/nginx/sites-enabled/tenant-$d.conf" + failed=$((failed + 1)); continue + fi + systemctl reload nginx + + if certbot --nginx -d "$d" --non-interactive --agree-tos \ + --email "$CERTBOT_EMAIL" --redirect --keep-until-expiring >>"$STATE_DIR/certbot.log" 2>&1; then + issued=$((issued + 1)) + log "issued: $d" + else + log "ERROR: certbot failed for $d (see $STATE_DIR/certbot.log) — HTTP still served, HTTPS not yet" + failed=$((failed + 1)) + fi +done <<< "$DESIRED" + +# Domains dropped from the source: stop serving them, but never delete the +# certificate — a domain re-added next week should not need a fresh issuance. +for link in /etc/nginx/sites-enabled/tenant-*.conf; do + [[ -e "$link" ]] || continue + name="$(basename "$link")"; name="${name#tenant-}"; name="${name%.conf}" + if ! grep -qxF "$name" <<< "$DESIRED"; then + if [[ $DRY_RUN -eq 1 ]]; then + log "DRY RUN would disable: $name" + else + log "disabling (removed from source): $name" + rm -f "$link" + fi + fi +done + +if [[ $DRY_RUN -eq 0 ]]; then + nginx -t && systemctl reload nginx +fi + +log "done. issued=$issued current=$skipped wildcard=$wildcarded awaiting-dns=$waiting failed=$failed" +[[ $failed -eq 0 ]] diff --git a/scripts/deploy/systemd/marketplaces-domains.service b/scripts/deploy/systemd/marketplaces-domains.service new file mode 100644 index 0000000..627c947 --- /dev/null +++ b/scripts/deploy/systemd/marketplaces-domains.service @@ -0,0 +1,12 @@ +[Unit] +Description=Reconcile tenant TLS domains +After=network-online.target nginx.service +Wants=network-online.target + +[Service] +Type=oneshot +ExecStart=/srv/marketplaces/bin/sync-domains.sh +# A failed run must not tear down what is already serving; the next run retries. +SuccessExitStatus=0 +StandardOutput=journal +StandardError=journal diff --git a/scripts/deploy/systemd/marketplaces-domains.timer b/scripts/deploy/systemd/marketplaces-domains.timer new file mode 100644 index 0000000..749b1aa --- /dev/null +++ b/scripts/deploy/systemd/marketplaces-domains.timer @@ -0,0 +1,12 @@ +[Unit] +Description=Reconcile tenant TLS domains every 10 minutes + +[Timer] +OnBootSec=2min +OnUnitActiveSec=10min +# Spread load so many servers do not all hit the ACME API at once. +RandomizedDelaySec=90s +Persistent=true + +[Install] +WantedBy=timers.target