linearvoid/docs/incident-runbook.md

Incident Runbook

Triage order for Linear Void incidents. Work top-to-bottom — each tier depends on the one above.

Triage Order

1. Edge Reachability (Cloudflare)
2. Origin Health (VPS/Docker)
3. Playout Progression (HLS)
4. Chat Connectivity (WebSocket)
5. Data-Plane (Redis/DB)

1. Edge Reachability (Cloudflare)

Symptoms: linearvoid.tv not resolving, SSL errors, 5xx from Cloudflare.

dig linearvoid.tv
dig www.linearvoid.tv
curl -I https://linearvoid.tv/health

Common causes: DNS misconfiguration (A record → wrong IP); Cloudflare proxy disabled (grey cloud exposes origin); SSL certificate expired (Cloudflare → SSL/TLS); Cloudflare outage (cloudflarestatus.com).

Mitigation: Verify A record points linearvoid.tv → origin IP (proxied, orange cloud). Bypass Cloudflare temporarily:

curl -H "Host: linearvoid.tv" http://<origin-ip>/health

2. Origin Health (VPS/Docker)

Symptoms: Cloudflare 502/504, containers down.

docker -H tcp://192.168.88.40:2375 ps
docker -H tcp://192.168.88.40:2375 logs linearvoid-nginx --tail 50
curl http://192.168.88.40/health

Common causes: Host disk full (df -h); Docker daemon stopped (systemctl status docker); container restart loop (docker ps -a --filter "status=exited"); OOM (dmesg | grep -i oom).

Mitigation:

docker compose -f docker-compose.prod.yml up -d   # restart stack
docker system prune -f                            # free disk
docker logs linearvoid-<service> --tail 100       # per-service logs

3. Playout Progression (HLS)

Symptoms: Viewer sees slate or buffering, playlist not advancing.

curl http://192.168.88.40:8081/health             # "mode": "live" or "slate"
curl -I http://192.168.88.40/hls/playlist.m3u8
ls -la data/hls/
docker logs linearvoid-playout --tail 100

Common causes: FFmpeg crash → slate fallback (health degraded); playlist empty/missing (cat data/playlist/playlist.txt); media file missing (ls data/media/); disk full → HLS segments can't write.

Mitigation: Validate playlist entries and media paths; restart playout (docker compose restart playout); regenerate demo playlist if empty (see backend/playout/start-playout.sh).

4. Chat Connectivity (WebSocket)

Symptoms: Chat not loading, messages not sending, presence broken.

curl http://192.168.88.40:3000/health
wscat -c ws://192.168.88.40/ws?username=test&room=main
docker logs linearvoid-chat --tail 100

Common causes: Redis down → presence/rate-limiting degraded (chat still works); DB down → messages not persisting; port not exposed (docker port linearvoid-chat).

Mitigation: Restart chat (docker compose restart chat); check Redis (docker exec linearvoid-redis redis-cli ping); check DB (docker exec linearvoid-db pg_isready -U linearvoid). Chat degrades gracefully without Redis — look for "Redis connection warning" in logs.

5. Data-Plane (Redis/DB)

Symptoms: Chat works but no persistence/presence, admin stats empty.

docker exec linearvoid-redis redis-cli ping
docker exec linearvoid-redis redis-cli info memory
docker exec linearvoid-db pg_isready -U linearvoid
docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"

Common causes: Redis OOM (check maxmemory); PostgreSQL connection limit; disk full on volume mount.

Mitigation: Restart redis (docker compose restart redis, AOF persist enabled); restart db (docker compose restart db); restore from backup (see scripts/verify-backups.sh).

Escalation

Severity	When	Action
Critical	Slate active >30min, complete outage	Trigger DR plan, provision fallback host
Warning	Redis down, high CPU >30min	Investigate business hours, restart service
Info	Single service blip	Monitor; no action unless recurring

Post-Incident

1. Document timeline in incident channel
2. Identify root cause
3. Add prevention to docs if actionable
4. Update this runbook if new failure mode discovered

Quarterly Disaster Recovery Drill

Validate recovery targets under controlled host-loss simulation: RPO ≤ 24h, RTO ≤ 60min.

Prerequisites

• Latest backups from scripts/backup-db.sh and scripts/backup-weekly.sh
• Operator access to Docker host and DNS controls
• Current deployment variables (.env, compose file, image tags)

Scope

• Simulate complete origin loss (all app containers unavailable)
• Rebuild stack on clean host
• Restore DB and media/playlist mounts
• Verify viewer playback, chat, admin auth, monitoring targets

Steps

1. Record drill start timestamp and on-call participants.
2. Capture pre-drill baseline:

   curl -fsS http://<origin>/health
   curl -fsS http://<origin>:3000/health
   curl -fsS http://<origin>:8081/health
   

3. Simulate outage: docker compose -f docker-compose.prod.yml down
4. Provision clean recovery host (or clean test-lab VM); copy deployment bundle.
5. Restore data paths: data/playlist/, data/media/, data/chat/, data/hls/ (optional warm cache).
6. Restore database from latest backup dump.
7. Start stack: docker compose -f docker-compose.prod.yml up -d
8. Validate health and runtime:

   docker ps --format "table {{.Names}}\t{{.Status}}"
   curl -fsS http://<origin>/health
   curl -fsS http://<origin>/hls/playlist.m3u8 | head -n 5
   curl -fsS http://<origin>:9090/-/ready
   

9. Validate app behavior: viewer SPA loads and HLS plays; WebSocket chat connects with message round-trip; admin login/session endpoint succeeds.
10. Run backup integrity check: scripts/verify-backups.sh
11. Record drill end timestamp; compute measured RTO/RPO.

Pass/Fail Criteria

• Pass: all core services healthy and functional in ≤ 60 min.
• Pass: latest recoverable data point ≤ 24 h old.
• Fail: any critical path remains degraded (playout, chat, admin auth, monitoring).

Evidence Template

Record in incident tracker: date/time window; environment (test-lab, staging, production); backup artifact IDs and timestamps; measured RTO and RPO; issues found and remediation owner; follow-up ticket links.