refactor: modularize chat backend, update docs for .m4s segments

docs/incident-runbook.md

@@ -161,4 +161,69 @@ docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"
 3. Add prevention to docs if actionable
 4. Update this runbook if new failure mode discovered
 
-**Recovery targets:** RPO ≤24h, RTO ≤60min. Quarterly DR drill: simulate host loss, restore from backup.
+## Quarterly Disaster Recovery Drill
+
+Goal: validate recovery targets under controlled host-loss simulation.
+
+**Recovery targets:** RPO <= 24h, RTO <= 60min.
+
+### Prerequisites
+
+- Latest backup artifacts available from `scripts/backup-db.sh` and `scripts/backup-weekly.sh`
+- Operator access to Docker host and DNS controls
+- Current deployment variables available (`.env`, compose file, image tags)
+
+### Drill Scope
+
+- Simulate complete origin loss (all app containers unavailable)
+- Rebuild stack on clean host
+- Restore DB and media/playlist mounts
+- Verify viewer playback, chat, admin auth, monitoring targets
+
+### Drill Steps
+
+1. Record drill start timestamp and on-call participants.
+2. Capture pre-drill baseline:
+   - `curl -fsS http://<origin>/health`
+   - `curl -fsS http://<origin>:3000/health`
+   - `curl -fsS http://<origin>:8081/health`
+3. Simulate origin outage:
+   - `docker compose -f docker-compose.prod.yml down`
+4. Provision clean recovery host (or clean test-lab VM) and copy deployment bundle.
+5. Restore required data paths:
+   - `data/playlist/`
+   - `data/media/`
+   - `data/chat/`
+   - `data/hls/` (optional warm cache)
+6. Restore database from latest backup dump.
+7. Start stack:
+   - `docker compose -f docker-compose.prod.yml up -d`
+8. Validate service health and runtime:
+   - `docker ps --format "table {{.Names}}\t{{.Status}}"`
+   - `curl -fsS http://<origin>/health`
+   - `curl -fsS http://<origin>/hls/playlist.m3u8 | head -n 5`
+   - `curl -fsS http://<origin>:9090/-/ready`
+9. Validate app behavior:
+   - Viewer SPA loads and HLS plays
+   - WebSocket chat connects and message round-trip works
+   - Admin login/session endpoint returns success
+10. Run backup integrity check:
+    - `scripts/verify-backups.sh`
+11. Record drill end timestamp and compute measured RTO/RPO.
+
+### Pass/Fail Criteria
+
+- Pass if all core services healthy and functional in <= 60 minutes.
+- Pass if latest recoverable data point is <= 24 hours old.
+- Fail if any critical path remains degraded (playout, chat, admin auth, monitoring).
+
+### Evidence Template
+
+Document each drill in incident tracker with:
+
+- Date/time window
+- Environment (test-lab, staging, production)
+- Backup artifact IDs and timestamps
+- Measured RTO and RPO
+- Issues found and remediation owner
+- Follow-up ticket links