linearvoid/docs/incident-runbook.md

Incident Runbook

Triage order for Linear Void incidents. Work top-to-bottom — each tier depends on the one above.

Triage Order

1. Edge Reachability (Cloudflare)
2. Origin Health (VPS/Docker)
3. Playout Progression (HLS)
4. Chat Connectivity (WebSocket)
5. Data-Plane (Redis/DB)

1. Edge Reachability (Cloudflare)

Symptoms: linearvoid.tv not resolving, SSL errors, 5xx from Cloudflare.

# Check DNS
dig linearvoid.tv
dig www.linearvoid.tv

# Check Cloudflare status
curl -I https://linearvoid.tv/health

Common causes:

• DNS misconfiguration (A record pointing to wrong IP)
• Cloudflare proxy disabled (grey cloud → origin exposed directly)
• SSL certificate expired (check Cloudflare dashboard → SSL/TLS)
• Cloudflare outage → check https://www.cloudflarestatus.com

Mitigation:

• Verify A record: linearvoid.tv → origin IP (proxied, orange cloud)
• Bypass Cloudflare temporarily: curl -H "Host: linearvoid.tv" http://<origin-ip>/health

2. Origin Health (VPS/Docker)

Symptoms: Cloudflare returns 502/504, containers down.

# Remote Docker (test lab)
docker -H tcp://192.168.88.40:2375 ps
docker -H tcp://192.168.88.40:2375 logs linearvoid-nginx --tail 50

# Direct health check
curl http://192.168.88.40/health

Common causes:

• Host out of disk space: df -h
• Docker daemon stopped: systemctl status docker
• Container restart loop: docker ps -a --filter "status=exited"
• Resource exhaustion (OOM): dmesg | grep -i oom

Mitigation:

• Restart stack: docker compose -f docker-compose.prod.yml up -d
• Free disk: docker system prune -f
• Check logs per service: docker logs linearvoid-<service> --tail 100

3. Playout Progression (HLS)

Symptoms: Viewer sees slate or buffering, playlist not advancing.

# Check playout health
curl http://192.168.88.40:8081/health
# Response includes "mode": "live" or "slate"

# Check HLS output
curl -I http://192.168.88.40/hls/playlist.m3u8
ls -la data/hls/

# Check playout logs
docker logs linearvoid-playout --tail 100

Common causes:

• FFmpeg crash → slate fallback active (health shows degraded)
• Playlist file empty/missing: cat data/playlist/playlist.txt
• Media file missing from playlist: ls data/media/
• Disk full → HLS segments can't write

Mitigation:

• Verify playlist has valid entries
• Check media files exist at paths in playlist
• Restart playout: docker compose restart playout
• Generate demo playlist if empty: see backend/playout/start-playout.sh

4. Chat Connectivity (WebSocket)

Symptoms: Chat not loading, messages not sending, presence broken.

# Check chat health
curl http://192.168.88.40:3000/health

# Test WebSocket
wscat -c ws://192.168.88.40/ws?username=test&room=main

# Check chat logs
docker logs linearvoid-chat --tail 100

Common causes:

• Redis down → presence/rate-limiting degraded (chat still works)
• Database down → messages not persisting
• Port not exposed: docker port linearvoid-chat

Mitigation:

• Restart chat: docker compose restart chat
• Check Redis: docker exec linearvoid-redis redis-cli ping
• Check DB: docker exec linearvoid-db pg_isready -U linearvoid
• Chat works without Redis (graceful degradation) — check logs for "Redis connection warning"

5. Data-Plane (Redis/DB)

Symptoms: Chat works but no persistence/presence, admin stats empty.

# Redis
docker exec linearvoid-redis redis-cli ping
docker exec linearvoid-redis redis-cli info memory

# PostgreSQL
docker exec linearvoid-db pg_isready -U linearvoid
docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"

Common causes:

• Redis OOM (check maxmemory setting)
• PostgreSQL connection limit reached
• Disk full on volume mount

Mitigation:

• Redis: docker compose restart redis (AOF persist enabled)
• PostgreSQL: docker compose restart db
• Restore from backup: see scripts/verify-backups.sh for procedure

Escalation

Severity	When	Action
Critical	Slate active >30min, complete outage	Trigger DR plan, provision fallback host
Warning	Redis down, high CPU >30min	Investigate during business hours, restart service
Info	Single service blip	Monitor, no action unless recurring

Post-Incident

1. Document timeline in incident channel
2. Identify root cause
3. Add prevention to docs if actionable
4. Update this runbook if new failure mode discovered

Quarterly Disaster Recovery Drill

Goal: validate recovery targets under controlled host-loss simulation.

Recovery targets: RPO <= 24h, RTO <= 60min.

Prerequisites

• Latest backup artifacts available from scripts/backup-db.sh and scripts/backup-weekly.sh
• Operator access to Docker host and DNS controls
• Current deployment variables available (.env, compose file, image tags)

Drill Scope

• Simulate complete origin loss (all app containers unavailable)
• Rebuild stack on clean host
• Restore DB and media/playlist mounts
• Verify viewer playback, chat, admin auth, monitoring targets

Drill Steps

1. Record drill start timestamp and on-call participants.
2. Capture pre-drill baseline:
   • curl -fsS http://<origin>/health
   • curl -fsS http://<origin>:3000/health
   • curl -fsS http://<origin>:8081/health
3. Simulate origin outage:
   • docker compose -f docker-compose.prod.yml down
4. Provision clean recovery host (or clean test-lab VM) and copy deployment bundle.
5. Restore required data paths:
   • data/playlist/
   • data/media/
   • data/chat/
   • data/hls/ (optional warm cache)
6. Restore database from latest backup dump.
7. Start stack:
   • docker compose -f docker-compose.prod.yml up -d
8. Validate service health and runtime:
   • docker ps --format "table {{.Names}}\t{{.Status}}"
   • curl -fsS http://<origin>/health
   • curl -fsS http://<origin>/hls/playlist.m3u8 | head -n 5
   • curl -fsS http://<origin>:9090/-/ready
9. Validate app behavior:
   • Viewer SPA loads and HLS plays
   • WebSocket chat connects and message round-trip works
   • Admin login/session endpoint returns success
10. Run backup integrity check:
    • scripts/verify-backups.sh
11. Record drill end timestamp and compute measured RTO/RPO.

Pass/Fail Criteria

• Pass if all core services healthy and functional in <= 60 minutes.
• Pass if latest recoverable data point is <= 24 hours old.
• Fail if any critical path remains degraded (playout, chat, admin auth, monitoring).

Evidence Template

Document each drill in incident tracker with:

• Date/time window
• Environment (test-lab, staging, production)
• Backup artifact IDs and timestamps
• Measured RTO and RPO
• Issues found and remediation owner
• Follow-up ticket links