# Incident Runbook

Triage order for Linear Void incidents. Work top-to-bottom — each tier depends on the one above.

## Triage Order

1. Edge Reachability (Cloudflare)
2. Origin Health (VPS/Docker)
3. Playout Progression (HLS)
4. Chat Connectivity (WebSocket)
5. Data-Plane (Redis/DB)

## 1. Edge Reachability (Cloudflare)

**Symptoms:** `linearvoid.tv` not resolving, SSL errors, 5xx from Cloudflare.

```bash
# Check DNS
dig linearvoid.tv
dig www.linearvoid.tv

# Check Cloudflare status
curl -I https://linearvoid.tv/health
```

**Common causes:**

- DNS misconfiguration (A record pointing to wrong IP)
- Cloudflare proxy disabled (grey cloud → origin exposed directly)
- SSL certificate expired (check Cloudflare dashboard → SSL/TLS)
- Cloudflare outage → check [https://www.cloudflarestatus.com](https://www.cloudflarestatus.com)

**Mitigation:**

- Verify A record: `linearvoid.tv` → origin IP (proxied, orange cloud)
- Bypass Cloudflare temporarily: `curl -H "Host: linearvoid.tv" http://<origin-ip>/health`

## 2. Origin Health (VPS/Docker)

**Symptoms:** Cloudflare returns 502/504, containers down.

```bash
# Remote Docker (test lab)
docker -H tcp://192.168.88.40:2375 ps
docker -H tcp://192.168.88.40:2375 logs linearvoid-nginx --tail 50

# Direct health check
curl http://192.168.88.40/health
```

**Common causes:**

- Host out of disk space: `df -h`
- Docker daemon stopped: `systemctl status docker`
- Container restart loop: `docker ps -a --filter "status=exited"`
- Resource exhaustion (OOM): `dmesg | grep -i oom`

**Mitigation:**

- Restart stack: `docker compose -f docker-compose.prod.yml up -d`
- Free disk: `docker system prune -f`
- Check logs per service: `docker logs linearvoid-<service> --tail 100`

## 3. Playout Progression (HLS)

**Symptoms:** Viewer sees slate or buffering, playlist not advancing.

```bash
# Check playout health
curl http://192.168.88.40:8081/health
# Response includes "mode": "live" or "slate"

# Check HLS output
curl -I http://192.168.88.40/hls/playlist.m3u8
ls -la data/hls/

# Check playout logs
docker logs linearvoid-playout --tail 100
```

**Common causes:**

- FFmpeg crash → slate fallback active (health shows `degraded`)
- Playlist file empty/missing: `cat data/playlist/playlist.txt`
- Media file missing from playlist: `ls data/media/`
- Disk full → HLS segments can't write

**Mitigation:**

- Verify playlist has valid entries
- Check media files exist at paths in playlist
- Restart playout: `docker compose restart playout`
- Generate demo playlist if empty: see `backend/playout/start-playout.sh`

## 4. Chat Connectivity (WebSocket)

**Symptoms:** Chat not loading, messages not sending, presence broken.

```bash
# Check chat health
curl http://192.168.88.40:3000/health

# Test WebSocket
wscat -c ws://192.168.88.40/ws?username=test&room=main

# Check chat logs
docker logs linearvoid-chat --tail 100
```

**Common causes:**

- Redis down → presence/rate-limiting degraded (chat still works)
- Database down → messages not persisting
- Port not exposed: `docker port linearvoid-chat`

**Mitigation:**

- Restart chat: `docker compose restart chat`
- Check Redis: `docker exec linearvoid-redis redis-cli ping`
- Check DB: `docker exec linearvoid-db pg_isready -U linearvoid`
- Chat works without Redis (graceful degradation) — check logs for "Redis connection warning"

## 5. Data-Plane (Redis/DB)

**Symptoms:** Chat works but no persistence/presence, admin stats empty.

```bash
# Redis
docker exec linearvoid-redis redis-cli ping
docker exec linearvoid-redis redis-cli info memory

# PostgreSQL
docker exec linearvoid-db pg_isready -U linearvoid
docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"
```

**Common causes:**

- Redis OOM (check `maxmemory` setting)
- PostgreSQL connection limit reached
- Disk full on volume mount

**Mitigation:**

- Redis: `docker compose restart redis` (AOF persist enabled)
- PostgreSQL: `docker compose restart db`
- Restore from backup: see `scripts/verify-backups.sh` for procedure

## Escalation

| Severity | When                                 | Action                                             |
| -------- | ------------------------------------ | -------------------------------------------------- |
| Critical | Slate active >30min, complete outage | Trigger DR plan, provision fallback host           |
| Warning  | Redis down, high CPU >30min          | Investigate during business hours, restart service |
| Info     | Single service blip                  | Monitor, no action unless recurring                |

## Post-Incident

1. Document timeline in incident channel
2. Identify root cause
3. Add prevention to docs if actionable
4. Update this runbook if new failure mode discovered

## Quarterly Disaster Recovery Drill

Goal: validate recovery targets under controlled host-loss simulation.

**Recovery targets:** RPO <= 24h, RTO <= 60min.

### Prerequisites

- Latest backup artifacts available from `scripts/backup-db.sh` and `scripts/backup-weekly.sh`
- Operator access to Docker host and DNS controls
- Current deployment variables available (`.env`, compose file, image tags)

### Drill Scope

- Simulate complete origin loss (all app containers unavailable)
- Rebuild stack on clean host
- Restore DB and media/playlist mounts
- Verify viewer playback, chat, admin auth, monitoring targets

### Drill Steps

1. Record drill start timestamp and on-call participants.
2. Capture pre-drill baseline:
   - `curl -fsS http://<origin>/health`
   - `curl -fsS http://<origin>:3000/health`
   - `curl -fsS http://<origin>:8081/health`
3. Simulate origin outage:
   - `docker compose -f docker-compose.prod.yml down`
4. Provision clean recovery host (or clean test-lab VM) and copy deployment bundle.
5. Restore required data paths:
   - `data/playlist/`
   - `data/media/`
   - `data/chat/`
   - `data/hls/` (optional warm cache)
6. Restore database from latest backup dump.
7. Start stack:
   - `docker compose -f docker-compose.prod.yml up -d`
8. Validate service health and runtime:
   - `docker ps --format "table {{.Names}}\t{{.Status}}"`
   - `curl -fsS http://<origin>/health`
   - `curl -fsS http://<origin>/hls/playlist.m3u8 | head -n 5`
   - `curl -fsS http://<origin>:9090/-/ready`
9. Validate app behavior:
   - Viewer SPA loads and HLS plays
   - WebSocket chat connects and message round-trip works
   - Admin login/session endpoint returns success
10. Run backup integrity check:
    - `scripts/verify-backups.sh`
11. Record drill end timestamp and compute measured RTO/RPO.

### Pass/Fail Criteria

- Pass if all core services healthy and functional in <= 60 minutes.
- Pass if latest recoverable data point is <= 24 hours old.
- Fail if any critical path remains degraded (playout, chat, admin auth, monitoring).

### Evidence Template

Document each drill in incident tracker with:

- Date/time window
- Environment (test-lab, staging, production)
- Backup artifact IDs and timestamps
- Measured RTO and RPO
- Issues found and remediation owner
- Follow-up ticket links