linearvoid/docs/incident-runbook.md

Incident Runbook

Triage order for Linear Void incidents. Work top-to-bottom — each tier depends on the one above.

Triage Order

1. Edge Reachability (Cloudflare)
2. Origin Health (VPS/Docker)
3. Playout Progression (HLS)
4. Chat Connectivity (WebSocket)
5. Data-Plane (Redis/DB)

---

1. Edge Reachability (Cloudflare)

Symptoms: linearvoid.tv not resolving, SSL errors, 5xx from Cloudflare.

# Check DNS
dig linearvoid.tv
dig www.linearvoid.tv

# Check Cloudflare status
curl -I https://linearvoid.tv/health

Common causes:

• DNS misconfiguration (A record pointing to wrong IP)
• Cloudflare proxy disabled (grey cloud → origin exposed directly)
• SSL certificate expired (check Cloudflare dashboard → SSL/TLS)
• Cloudflare outage → https://www.cloudflarestatus.com

Mitigation:

• Verify A record: linearvoid.tv → origin IP (proxied, orange cloud)
• Bypass Cloudflare temporarily: curl -H "Host: linearvoid.tv" http://<origin-ip>/health

---

2. Origin Health (VPS/Docker)

Symptoms: Cloudflare returns 502/504, containers down.

# Remote Docker (test lab)
docker -H tcp://192.168.88.40:2375 ps
docker -H tcp://192.168.88.40:2375 logs linearvoid-nginx --tail 50

# Direct health check
curl http://192.168.88.40/health

Common causes:

• Host out of disk space: df -h
• Docker daemon stopped: systemctl status docker
• Container restart loop: docker ps -a --filter "status=exited"
• Resource exhaustion (OOM): dmesg | grep -i oom

Mitigation:

• Restart stack: docker compose -f docker-compose.prod.yml up -d
• Free disk: docker system prune -f
• Check logs per service: docker logs linearvoid-<service> --tail 100

---

3. Playout Progression (HLS)

Symptoms: Viewer sees slate or buffering, playlist not advancing.

# Check playout health
curl http://192.168.88.40:8081/health
# Response includes "mode": "live" or "slate"

# Check HLS output
curl -I http://192.168.88.40/hls/playlist.m3u8
ls -la data/hls/

# Check playout logs
docker logs linearvoid-playout --tail 100

Common causes:

• FFmpeg crash → slate fallback active (health shows degraded)
• Playlist file empty/missing: cat data/playlist/playlist.txt
• Media file missing from playlist: ls data/media/
• Disk full → HLS segments can't write

Mitigation:

• Verify playlist has valid entries
• Check media files exist at paths in playlist
• Restart playout: docker compose restart playout
• Generate demo playlist if empty: see backend/playout/start-playout.sh

---

4. Chat Connectivity (WebSocket)

Symptoms: Chat not loading, messages not sending, presence broken.

# Check chat health
curl http://192.168.88.40:3000/health

# Test WebSocket
wscat -c ws://192.168.88.40/ws?username=test&room=main

# Check chat logs
docker logs linearvoid-chat --tail 100

Common causes:

• Redis down → presence/rate-limiting degraded (chat still works)
• Database down → messages not persisting
• Port not exposed: docker port linearvoid-chat

Mitigation:

• Restart chat: docker compose restart chat
• Check Redis: docker exec linearvoid-redis redis-cli ping
• Check DB: docker exec linearvoid-db pg_isready -U linearvoid
• Chat works without Redis (graceful degradation) — check logs for "Redis connection warning"

---

5. Data-Plane (Redis/DB)

Symptoms: Chat works but no persistence/presence, admin stats empty.

# Redis
docker exec linearvoid-redis redis-cli ping
docker exec linearvoid-redis redis-cli info memory

# PostgreSQL
docker exec linearvoid-db pg_isready -U linearvoid
docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"

Common causes:

• Redis OOM (check maxmemory setting)
• PostgreSQL connection limit reached
• Disk full on volume mount

Mitigation:

• Redis: docker compose restart redis (AOF persist enabled)
• PostgreSQL: docker compose restart db
• Restore from backup: see scripts/verify-backups.sh for procedure

---

Escalation

Severity	When	Action
Critical	Slate active >30min, complete outage	Trigger DR plan, provision fallback host
Warning	Redis down, high CPU >30min	Investigate during business hours, restart service
Info	Single service blip	Monitor, no action unless recurring

Post-Incident

1. Document timeline in incident channel
2. Identify root cause
3. Add prevention to TODO.md if actionable
4. Update this runbook if new failure mode discovered

---

Recovery targets: RPO ≤24h, RTO ≤60min. Quarterly DR drill: simulate host loss, restore from backup.