fix: parallel CD trigger, remove env blocker, cleanup release pipeline

2026-08-06 22:01:29 +02:00
1 parent 74bf7e19db
commit 4c02ab9d55
@@ -1,23 +1,18 @@
 # CD workflow — TEST LAB DEPLOYMENT
 # Deploys to test lab host 192.168.88.40:2375.
 # Production Hetzner target TBD — switch DOCKER_HOST when provisioned.
-# See TODO.md Production Deployment section for cutover plan.
+# See DEPLOYMENT.md Production Deployment section for cutover plan.
 name: CD
 on:
-  workflow_run:
+  push:
    workflows: ["CI"]
    types:
      - completed
    branches: [main]
  workflow_dispatch:
 jobs:
  deploy:
    runs-on: [ubuntu-latest]
    if: ${{ github.event.workflow_run.conclusion == 'success' || github.event_name == 'workflow_dispatch' }}
    environment: production
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
@@ -27,13 +22,6 @@ jobs:
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Log in to Gitea Container Registry
        uses: docker/login-action@v3
        with:
          registry: git.allucanget.biz
          username: ${{ secrets.CI_REGISTRY_USER }}
          password: ${{ secrets.CI_REGISTRY_TOKEN }}
      - name: Prepare environment
        run: |
          cat > .env << 'EOF'
@@ -43,7 +43,7 @@ jobs:
        run: cd backend/chat && GOTOOLCHAIN=local go vet ./...
      - name: Run Tests
-        run: echo "Tests will be added in future CI steps (per TODO.md)"
+        run: echo "Tests will be added in future CI steps"
  docker:
    needs: test
@@ -132,20 +132,3 @@ jobs:
          labels: ${{ steps.meta-nginx.outputs.labels }}
          cache-from: type=registry,ref=git.allucanget.biz/zwitschi/linearvoid/buildcache:nginx
          cache-to: type=registry,ref=git.allucanget.biz/zwitschi/linearvoid/buildcache:nginx,mode=max
  release:
    needs: docker
    runs-on: [ubuntu-latest]
    if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Create Release
        uses: softprops/action-gh-release@v1
        with:
          generate_release_notes: true
        env:
          GITHUB_TOKEN: ${{ secrets.CI_REGISTRY_TOKEN }}
@@ -12,6 +12,10 @@
 ### Changed
 - **Prod prometheus**: Switched from empty named volume to bind mounts for config files (consistent with dev).
 - **CD pipeline**: Trigger on `push` (parallel with CI), removed `environment: production` (was causing silent abort).
 - **CI pipeline**: Removed redundant `release` job (handled by `release.yml`).
 - **Tag format**: Standardized on `*.*.*` (e.g. `1.0.9`), removed `v` prefix from existing tags.
 - **DEPLOYMENT.md**: Documented parallel CI/CD pipeline flow with workflow table.
 ### Removed
@@ -154,7 +158,7 @@ Full backend and frontend implementation complete.
 - **Frontend** (`frontend/`): Vite + TypeScript + Tailwind + Alpine.js, HLS player with hls.js, chat sidebar with room switching and private messaging, "Now Playing" overlay, and modern UI (ADR-002, ADR-003).
 - Complete `docker-compose.yml` for local development with all services, volumes, and healthchecks.
 - All ADRs (001-007) created and linked in documentation.
- `TODO.md`, `DONE.md`, and `CHANGELOG.md` updated with full traceability.
+- `CHANGELOG.md` updated with full traceability.
 - Git repository initialized with comprehensive initial commit.
 The project is a fully functional, self-hosted 24/7 linear video channel with synchronized playback, custom chat, R2 delivery, moderation, and admin-ready structure. It matches the architecture defined in the arc42 template and all ADRs.
@@ -7,9 +7,8 @@ Thank you for considering contributing to Linear Void!
 1. **Read the Architecture First**: All design decisions are documented in `docs/architecture/README.md` (arc42 template) and the ADRs in `docs/architecture/adr/` (ADR-001 to ADR-007).
 2. **Follow the Structure**: Use the folder layout defined in ADR-002 (`frontend/`, `backend/playout/`, `backend/chat/`, `infrastructure/`, etc.).
 3. **Update Documentation Before Code**: Create or update an ADR before implementing significant changes.
-4. **Update TODO.md**: Plan tasks in `.github/instructions/TODO.md`, mark as completed, and move finished sections to `DONE.md`.
+4. **Commit Messages**: Reference the relevant ADR (e.g., "feat(playout): add R2 sync per ADR-001").
-5. **Commit Messages**: Reference the relevant ADR (e.g., "feat(playout): add R2 sync per ADR-001").
+5. **Test with Docker**: Run `docker compose up --build` to test changes.
 6. **Test with Docker**: Run `docker compose up --build` to test changes.
 ## Repository
@@ -40,23 +39,22 @@ Linear Void follows **Semantic Versioning** ([SemVer](https://semver.org)): `MAJ
 - **MINOR** — new features (endpoints, admin tabs, UI capabilities) that are backward-compatible.
 - **PATCH** — bug fixes, optimizations, documentation polish.
-Git tags use a `v` prefix: `1.0.2`, `1.1.0`, `v2.0.0`.
+Git tags use no prefix: `1.0.2`, `1.1.0`, `2.0.0`.
 ### Where Versions Are Declared
-| File                           | Token          | Example                   |
+| File                    | Token         | Example                   |
-| ------------------------------ | -------------- | ------------------------- |
+| ----------------------- | ------------- | ------------------------- |
-| `frontend/package.json`        | `"version"`    | `"1.0.2"`                 |
+| `frontend/package.json` | `"version"`   | `"1.0.2"`                 |
-| `CHANGELOG.md`                 | `## [X.Y.Z]`   | `## [1.0.2] - 2026-08-05` |
+| `CHANGELOG.md`          | `## [X.Y.Z]`  | `## [1.0.2] - 2026-08-05` |
-| `README.md`                    | Version badge  | `version-1.0.2-blue`      |
+| `README.md`             | Version badge | `version-1.0.2-blue`      |
 | `.github/instructions/TODO.md` | Section header | `## Version 1.0.2`        |
 ### Tag Triggers
-| Tag Pattern         | Workflow            | What Happens                                                                                                       |
+| Tag Pattern            | Workflow            | What Happens                                                                                                       |
-| ------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------ |
+| ---------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------ |
-| `v*` (e.g. `1.0.2`) | `release.yml`       | Builds all 3 images, pushes to registry with semver tags (`1.0.2`, `1.0`, `latest`), creates Gitea Release via API |
+| `*.*.*` (e.g. `1.0.2`) | `release.yml`       | Builds all 3 images, pushes to registry with semver tags (`1.0.2`, `1.0`, `latest`), creates Gitea Release via API |
-| Push to `main`      | `ci.yml` → `cd.yml` | CI builds + pushes images with `main`/`latest` tags, CD deploys to production                                      |
+| Push to `main`         | `ci.yml` → `cd.yml` | CI builds + pushes images with `main`/`latest` tags, CD deploys to production                                      |
 ### Full Release Procedure
@@ -87,7 +85,7 @@ Add entries under `## [X.Y.Z]` in `CHANGELOG.md` following [Keep a Changelog](ht
 ```bash
 # Stage all version bumps + changelog
-git add frontend/package.json CHANGELOG.md README.md .github/instructions/TODO.md
+git add frontend/package.json CHANGELOG.md README.md
 # Commit
 git commit -m "chore: bump version to X.Y.Z"
@@ -145,6 +143,6 @@ git.allucanget.biz/zwitschi/linearvoid/playout:latest
 ### Moving Completed Tasks
-After release, move completed sections from `TODO.md` to `DONE.md` and update `CHANGELOG.md` if needed. The `DONE.md` file serves as the historical record linking tasks to releases.
+After release, update `CHANGELOG.md` if needed.
 See the ADRs for architectural decisions and `docs/architecture/README.md` for the big picture. Thank you for contributing!
@@ -1,5 +1,33 @@
 # Deployment Guide
 ## Table of Contents
 - [Deployment Guide](#deployment-guide)
  - [Table of Contents](#table-of-contents)
  - [Environments](#environments)
  - [Production Target (Hetzner)](#production-target-hetzner)
    - [Planned Infrastructure](#planned-infrastructure)
    - [Cutover Plan](#cutover-plan)
    - [Firewall / Security Group](#firewall--security-group)
    - [Pipeline Readiness](#pipeline-readiness)
  - [Target Topology](#target-topology)
  - [Prerequisites](#prerequisites)
  - [Environment File](#environment-file)
  - [Pull-and-Run Deployment](#pull-and-run-deployment)
  - [First-Time or Recovery Provisioning](#first-time-or-recovery-provisioning)
  - [Health Verification](#health-verification)
  - [Zero/Low-Downtime Update Pattern](#zerolow-downtime-update-pattern)
  - [Rollback](#rollback)
  - [Operational Tasks](#operational-tasks)
    - [View logs](#view-logs)
    - [Restart one service](#restart-one-service)
    - [Stop stack](#stop-stack)
    - [Keep volumes (recommended) vs remove volumes (destructive)](#keep-volumes-recommended-vs-remove-volumes-destructive)
  - [Nginx and Proxy Considerations](#nginx-and-proxy-considerations)
  - [Security Baseline](#security-baseline)
  - [Common Deployment Pitfalls](#common-deployment-pitfalls)
  - [Troubleshooting](#troubleshooting)
 ## Environments
 Linear Void runs in two separate environments:
@@ -9,7 +37,7 @@ Linear Void runs in two separate environments:
 | **Test Lab**   | `192.168.88.40` (internal) | `tcp://192.168.88.40:2375` | Integration testing, staging, CI/CD validation |
 | **Production** | Hetzner VPS (TBD)          | `tcp://<hetzner-ip>:2375`  | Public linearvoid.tv serving real viewers      |
-**Current state (2026-08-06):** Test lab is active and receives automated CD deploys. Production Hetzner VPS is **not yet provisioned**. The CD workflow (`cd.yml`) and `docker-compose.prod.yml` currently target the test lab. Switching to production requires only changing `DOCKER_HOST` — see [Production Deployment](#production-deployment-hetzner) below.
+**Current state (2026-08-06):** Test lab is active and receives automated CD deploys. Production Hetzner VPS is **not yet provisioned**. The CD workflow (`cd.yml`) and `docker-compose.prod.yml` currently target the test lab. Switching to production requires only changing `DOCKER_HOST` — see [Production Deployment](#production-target-hetzner) below.
 ## Production Target (Hetzner)
@@ -28,7 +56,7 @@ Linear Void runs in two separate environments:
 ### Cutover Plan
-1. Provision Hetzner VPS (see `TODO.md` #92)
+1. Provision Hetzner VPS (see [Production Deployment](#production-target-hetzner) section)
 2. Add production secrets to Gitea (`CI_PROD_*` prefix)
 3. Update `DOCKER_HOST` in CD workflow to Hetzner public IP
 4. Trigger deploy → validate health checks
@@ -49,7 +77,17 @@ Linear Void runs in two separate environments:
 ### Pipeline Readiness
-The Gitea Actions CD workflow (`cd.yml`) already supports `DOCKER_HOST` as a configurable variable. To switch targets:
+Three Gitea Actions workflows drive the full pipeline:
 | Workflow      | Trigger          | Purpose                                                                                  |
 | ------------- | ---------------- | ---------------------------------------------------------------------------------------- |
 | `ci.yml`      | Push to `main`   | Lint, build frontend, vet Go, build & push Docker images (`:latest`, `:main`)            |
 | `cd.yml`      | Push to `main`   | Deploy to test lab (`192.168.88.40:2375`) — runs **in parallel** with CI                 |
 | `release.yml` | Tag push `*.*.*` | Build & push versioned Docker images (`:X.Y.Z`, `:X.Y`, `:latest`), create Gitea Release |
 CI and CD run simultaneously on every push to `main`. The deploy pulls whatever `:latest` images are available in the registry at that moment.
 The CD workflow (`cd.yml`) uses `DOCKER_HOST` to target the deployment host. To switch targets:
 ```bash
 # Test lab (current)
@@ -59,7 +97,7 @@ DOCKER_HOST=tcp://192.168.88.40:2375
 DOCKER_HOST=tcp://<hetzner-public-ip>:2375
 ```
-No code changes needed in the pipeline. See `TODO.md` #91 and #92–#99 for full production deployment checklist.
+No code changes needed in the pipeline. See `DEPLOYMENT.md` Production Deployment section for full checklist.
 ## Target Topology
@@ -99,7 +137,7 @@ See .env.example and SETUP.md for full variable reference.
 Images are built and pushed automatically by the CI and release workflows (see [CONTRIBUTING.md](CONTRIBUTING.md#versioning-and-releases) for the release procedure).
-Every push to `main` produces `:main` and `:latest` tags. Semantic version tags (`v*`) produce `:X.Y.Z`, `:X.Y`, and `:latest` tags.
+Every push to `main` triggers CI (builds `:main` and `:latest` tags) and CD (deploys `:latest` to test lab) in parallel. Semantic version tags (`*.*.*`, e.g. `1.0.9`) trigger the release workflow which builds `:X.Y.Z`, `:X.Y`, and `:latest` tags and creates a Gitea Release.
 From repository root:
@@ -112,7 +112,7 @@ This structure directly reflects the building blocks, runtime flows, deployment
 ## Getting Started
-> **Note:** The `docker compose up` below targets local development. The test lab (192.168.88.40) receives automated CD deploys. Production Hetzner VPS is not yet provisioned — see `TODO.md`.
+> **Note:** The `docker compose up` below targets local development. The test lab (192.168.88.40) receives automated CD deploys. Production Hetzner VPS is not yet provisioned — see `DEPLOYMENT.md`.
 ```bash
 git clone https://git.allucanget.biz/zwitschi/linearvoid.git
@@ -223,11 +223,11 @@ Contributions, issues, and further evolution (LL-HLS, ABR, multi-channel, E2EE)
 Contributions are welcome! See [`CONTRIBUTING.md`](CONTRIBUTING.md) for guidelines. Quick overview:
 1. Read the [architecture docs](docs/architecture/README.md) and ADRs first.
-2. Plan tasks in `.github/instructions/TODO.md`.
+2. Code against the defined module structure (frontend, backend/playout, backend/chat, infrastructure, scripts, monitoring).
 3. Open PRs on [Gitea](https://git.allucanget.biz/zwitschi/linearvoid).
 4. CI validates builds and tests automatically.
-Since this a linear project, we also rely on a linear development and release model: Only main branch is used for releases, and all features are merged into main after review and CI validation. Releases are tagged with semantic versioning (1.0.3, 1.0.4, etc.) and automatically built/pushed to the container registry.
+Since this a linear project, we also rely on a linear development and release model: Only main branch is used for releases, and all features are merged into main after review and CI validation. Releases are tagged with semantic versioning (1.0.3, 1.0.4, etc.) and automatically built/pushed to the container registry. Version numbers are declared in `frontend/package.json`, `CHANGELOG.md`, and the version badge in this README. For patches, we use even numbers for stable versions (1.0.2, 1.0.4) and odd numbers for pre-release/unstable versions (1.0.3, 1.0.5).
 [MIT License](LICENSE)
@@ -55,7 +55,7 @@ flowchart LR
 **Test Lab (active):** Internal network `192.168.88.0/24`, host `192.168.88.40`. Receives automated CD deploys from Gitea Actions. Used for integration testing and staging validation.
-**Production (pending):** Hetzner Cloud VPS, Ashburn VA. Not yet provisioned. CD workflow and compose files are ready — only `DOCKER_HOST` needs changing. See `TODO.md` Production Deployment section.
+**Production (pending):** Hetzner Cloud VPS, Ashburn VA. Not yet provisioned. CD workflow and compose files are ready — only `DOCKER_HOST` needs changing. See Production Deployment section in [DEPLOYMENT.md](./DEPLOYMENT.md) for cutover plan.
 ## Development Prerequisites
@@ -1,6 +1,6 @@
 # docker-compose.prod.yml — TEST LAB DEPLOYMENT
 # Target: test lab host 192.168.88.40 (NOT Hetzner production).
-# Production Hetzner VPS not yet provisioned — see TODO.md Production Deployment section.
+# Production Hetzner VPS not yet provisioned — see DEPLOYMENT.md Production Deployment section.
 # Uses pre-built images from Gitea Container Registry (no local build).
 # Override DOCKER_HOST in CD workflow to switch target.
@@ -20,7 +20,14 @@
  - [Building Block View](#building-block-view)
    - [Whitebox Overall System](#whitebox-overall-system)
    - [Level 2](#level-2)
      - [Playout Service (`backend/playout/src/playlist.go`)](#playout-service-backendplayoutsrcplaylistgo)
      - [Chat Service (`backend/chat/src/main.go`)](#chat-service-backendchatsrcmaingo)
      - [Frontend SPA (`frontend/`)](#frontend-spa-frontend)
      - [Reverse Proxy (`infrastructure/nginx/nginx.conf`)](#reverse-proxy-infrastructurenginxnginxconf)
    - [Level 3](#level-3)
      - [Chat Service Modules](#chat-service-modules)
      - [Playout Service Modules](#playout-service-modules)
      - [Frontend Modules](#frontend-modules)
  - [Runtime View](#runtime-view)
    - [Playout \& Delivery Scenario](#playout--delivery-scenario)
  - [Deployment View](#deployment-view)
@@ -453,7 +460,7 @@ flowchart LR
 ### Infrastructure Level 2
-**Hetzner VPS (planned):** Will run playout (FFmpeg/MediaMTX), chat backend, Nginx. Monitoring via dashboards + UptimeRobot. Backups to R2. Currently test lab (`192.168.88.40`) serves this role. Production Hetzner instance not yet provisioned — see `TODO.md` Production Deployment section.
+**Hetzner VPS (planned):** Will run playout (FFmpeg/MediaMTX), chat backend, Nginx. Monitoring via dashboards + UptimeRobot. Backups to R2. Currently test lab (`192.168.88.40`) serves this role. Production Hetzner instance not yet provisioned — see Production Deployment section in [`DEPLOYMENT.md`](../../DEPLOYMENT.md).
 **Cloudflare Layer**:
@@ -60,8 +60,6 @@ Create a dedicated **admin web interface** in a new `frontend/admin/` directory
 - Integration with existing playout service and R2 keeps the architecture consistent.
 - Admin actions will be fully auditable, paving the way for ADR-005 (moderation and user management).
 This decision was planned in `TODO.md` before implementation, per AGENTS.md.
 ## Consequences
 **Positive**:
@@ -59,8 +59,6 @@ Extend the admin interface started in ADR-004 with additional protected sections
 - Leveraging Prometheus/Grafana for analytics avoids duplicating metrics collection.
 - Immutable audit logging supports future compliance and moderation review (key requirement for a public chat system).
 This ADR was planned in `TODO.md` before any implementation, per AGENTS.md.
 ## Consequences
 **Positive**:
@@ -92,8 +90,6 @@ This ADR was planned in `TODO.md` before any implementation, per AGENTS.md.
 7. Update authentication middleware and tests.
 8. Document API contracts and update `docker-compose.yml` to expose the admin interface securely.
 **Next Steps**: After both admin ADRs are complete, move to implementation of the Playout Service (highest priority in the core modules section of TODO.md).
 ## References
 - ADR-001: Chat features, moderation requirements, monitoring needs.
@@ -95,8 +95,6 @@ We must explicitly document:
 6. Add monitoring (Prometheus exporter) and admin audit logging for configuration changes.
 7. Write tests for rate limiting, presence, and admin configuration flows.
 **Next Steps**: After ADR-006 and ADR-007 are reviewed, mark the "Create ADRs for Data Storage, Caching, and Admin Capabilities" section as completed in TODO.md and move it to DONE.md. Then proceed with implementation of the Playout Service.
 ## References
 - ADR-001: Chat features requiring presence, rate limiting, and pub/sub.
@@ -4,15 +4,11 @@ Triage order for Linear Void incidents. Work top-to-bottom — each tier depends
 ## Triage Order
 ```
 1. Edge Reachability (Cloudflare)
 2. Origin Health (VPS/Docker)
 3. Playout Progression (HLS)
 4. Chat Connectivity (WebSocket)
 5. Data-Plane (Redis/DB)
 ```
 ---
 ## 1. Edge Reachability (Cloudflare)
@@ -28,17 +24,17 @@ curl -I https://linearvoid.tv/health
 ```
 **Common causes:**
 - DNS misconfiguration (A record pointing to wrong IP)
 - Cloudflare proxy disabled (grey cloud → origin exposed directly)
 - SSL certificate expired (check Cloudflare dashboard → SSL/TLS)
- Cloudflare outage → https://www.cloudflarestatus.com
+- Cloudflare outage → check [https://www.cloudflarestatus.com](https://www.cloudflarestatus.com)
 **Mitigation:**
 - Verify A record: `linearvoid.tv` → origin IP (proxied, orange cloud)
 - Bypass Cloudflare temporarily: `curl -H "Host: linearvoid.tv" http://<origin-ip>/health`
 ---
 ## 2. Origin Health (VPS/Docker)
 **Symptoms:** Cloudflare returns 502/504, containers down.
@@ -53,18 +49,18 @@ curl http://192.168.88.40/health
 ```
 **Common causes:**
 - Host out of disk space: `df -h`
 - Docker daemon stopped: `systemctl status docker`
 - Container restart loop: `docker ps -a --filter "status=exited"`
 - Resource exhaustion (OOM): `dmesg | grep -i oom`
 **Mitigation:**
 - Restart stack: `docker compose -f docker-compose.prod.yml up -d`
 - Free disk: `docker system prune -f`
 - Check logs per service: `docker logs linearvoid-<service> --tail 100`
 ---
 ## 3. Playout Progression (HLS)
 **Symptoms:** Viewer sees slate or buffering, playlist not advancing.
@@ -83,19 +79,19 @@ docker logs linearvoid-playout --tail 100
 ```
 **Common causes:**
 - FFmpeg crash → slate fallback active (health shows `degraded`)
 - Playlist file empty/missing: `cat data/playlist/playlist.txt`
 - Media file missing from playlist: `ls data/media/`
 - Disk full → HLS segments can't write
 **Mitigation:**
 - Verify playlist has valid entries
 - Check media files exist at paths in playlist
 - Restart playout: `docker compose restart playout`
 - Generate demo playlist if empty: see `backend/playout/start-playout.sh`
 ---
 ## 4. Chat Connectivity (WebSocket)
 **Symptoms:** Chat not loading, messages not sending, presence broken.
@@ -112,18 +108,18 @@ docker logs linearvoid-chat --tail 100
 ```
 **Common causes:**
 - Redis down → presence/rate-limiting degraded (chat still works)
 - Database down → messages not persisting
 - Port not exposed: `docker port linearvoid-chat`
 **Mitigation:**
 - Restart chat: `docker compose restart chat`
 - Check Redis: `docker exec linearvoid-redis redis-cli ping`
 - Check DB: `docker exec linearvoid-db pg_isready -U linearvoid`
 - Chat works without Redis (graceful degradation) — check logs for "Redis connection warning"
 ---
 ## 5. Data-Plane (Redis/DB)
 **Symptoms:** Chat works but no persistence/presence, admin stats empty.
@@ -139,32 +135,30 @@ docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"
 ```
 **Common causes:**
 - Redis OOM (check `maxmemory` setting)
 - PostgreSQL connection limit reached
 - Disk full on volume mount
 **Mitigation:**
 - Redis: `docker compose restart redis` (AOF persist enabled)
 - PostgreSQL: `docker compose restart db`
 - Restore from backup: see `scripts/verify-backups.sh` for procedure
 ---
 ## Escalation
-| Severity | When | Action |
+| Severity | When                                 | Action                                             |
-|---|---|---|
+| -------- | ------------------------------------ | -------------------------------------------------- |
-| Critical | Slate active >30min, complete outage | Trigger DR plan, provision fallback host |
+| Critical | Slate active >30min, complete outage | Trigger DR plan, provision fallback host           |
-| Warning | Redis down, high CPU >30min | Investigate during business hours, restart service |
+| Warning  | Redis down, high CPU >30min          | Investigate during business hours, restart service |
-| Info | Single service blip | Monitor, no action unless recurring |
+| Info     | Single service blip                  | Monitor, no action unless recurring                |
 ## Post-Incident
 1. Document timeline in incident channel
 2. Identify root cause
-3. Add prevention to TODO.md if actionable
+3. Add prevention to docs if actionable
 4. Update this runbook if new failure mode discovered
 ---
 **Recovery targets:** RPO ≤24h, RTO ≤60min. Quarterly DR drill: simulate host loss, restore from backup.