# Incident Runbook

Triage order for Linear Void incidents. Work top-to-bottom — each tier depends on the one above.

## Triage Order

1. **Edge Reachability** (Cloudflare)
2. **Origin Health** (VPS/Docker)
3. **Playout Progression** (HLS)
4. **Chat Connectivity** (WebSocket)
5. **Data-Plane** (Redis/DB)

## 1. Edge Reachability (Cloudflare)

**Symptoms:** `linearvoid.tv` not resolving, SSL errors, 5xx from Cloudflare.

```bash
dig linearvoid.tv
dig www.linearvoid.tv
curl -I https://linearvoid.tv/health
```

**Common causes:** DNS misconfiguration (A record → wrong IP); Cloudflare proxy disabled (grey cloud exposes origin); SSL certificate expired (Cloudflare → SSL/TLS); Cloudflare outage ([cloudflarestatus.com](https://www.cloudflarestatus.com)).

**Mitigation:** Verify A record points `linearvoid.tv` → origin IP (proxied, orange cloud). Bypass Cloudflare temporarily:

```bash
curl -H "Host: linearvoid.tv" http://<origin-ip>/health
```

## 2. Origin Health (VPS/Docker)

**Symptoms:** Cloudflare 502/504, containers down.

```bash
docker -H tcp://192.168.88.40:2375 ps
docker -H tcp://192.168.88.40:2375 logs linearvoid-nginx --tail 50
curl http://192.168.88.40/health
```

**Common causes:** Host disk full (`df -h`); Docker daemon stopped (`systemctl status docker`); container restart loop (`docker ps -a --filter "status=exited"`); OOM (`dmesg | grep -i oom`).

**Mitigation:**

```bash
docker compose -f docker-compose.prod.yml up -d   # restart stack
docker system prune -f                            # free disk
docker logs linearvoid-<service> --tail 100       # per-service logs
```

## 3. Playout Progression (HLS)

**Symptoms:** Viewer sees slate or buffering, playlist not advancing.

```bash
curl http://192.168.88.40:8081/health             # "mode": "live" or "slate"
curl -I http://192.168.88.40/hls/playlist.m3u8
ls -la data/hls/
docker logs linearvoid-playout --tail 100
```

**Common causes:** FFmpeg crash → slate fallback (health `degraded`); playlist empty/missing (`cat data/playlist/playlist.txt`); media file missing (`ls data/media/`); disk full → HLS segments can't write.

**Mitigation:** Validate playlist entries and media paths; restart playout (`docker compose restart playout`); regenerate demo playlist if empty (see `backend/playout/start-playout.sh`).

## 4. Chat Connectivity (WebSocket)

**Symptoms:** Chat not loading, messages not sending, presence broken.

```bash
curl http://192.168.88.40:3000/health
wscat -c ws://192.168.88.40/ws?username=test&room=main
docker logs linearvoid-chat --tail 100
```

**Common causes:** Redis down → presence/rate-limiting degraded (chat still works); DB down → messages not persisting; port not exposed (`docker port linearvoid-chat`).

**Mitigation:** Restart chat (`docker compose restart chat`); check Redis (`docker exec linearvoid-redis redis-cli ping`); check DB (`docker exec linearvoid-db pg_isready -U linearvoid`). Chat degrades gracefully without Redis — look for "Redis connection warning" in logs.

## 5. Data-Plane (Redis/DB)

**Symptoms:** Chat works but no persistence/presence, admin stats empty.

```bash
docker exec linearvoid-redis redis-cli ping
docker exec linearvoid-redis redis-cli info memory
docker exec linearvoid-db pg_isready -U linearvoid
docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"
```

**Common causes:** Redis OOM (check `maxmemory`); PostgreSQL connection limit; disk full on volume mount.

**Mitigation:** Restart redis (`docker compose restart redis`, AOF persist enabled); restart db (`docker compose restart db`); restore from backup (see `scripts/verify-backups.sh`).

## Escalation

| Severity | When                                 | Action                                      |
| -------- | ------------------------------------ | ------------------------------------------- |
| Critical | Slate active >30min, complete outage | Trigger DR plan, provision fallback host    |
| Warning  | Redis down, high CPU >30min          | Investigate business hours, restart service |
| Info     | Single service blip                  | Monitor; no action unless recurring         |

## Post-Incident

1. Document timeline in incident channel
2. Identify root cause
3. Add prevention to docs if actionable
4. Update this runbook if new failure mode discovered

## Quarterly Disaster Recovery Drill

Validate recovery targets under controlled host-loss simulation: **RPO ≤ 24h, RTO ≤ 60min**.

### Prerequisites

- Latest backups from `scripts/backup-db.sh` and `scripts/backup-weekly.sh`
- Operator access to Docker host and DNS controls
- Current deployment variables (`.env`, compose file, image tags)

### Scope

- Simulate complete origin loss (all app containers unavailable)
- Rebuild stack on clean host
- Restore DB and media/playlist mounts
- Verify viewer playback, chat, admin auth, monitoring targets

### Steps

1. Record drill start timestamp and on-call participants.
2. Capture pre-drill baseline:
   ```bash
   curl -fsS http://<origin>/health
   curl -fsS http://<origin>:3000/health
   curl -fsS http://<origin>:8081/health
   ```
3. Simulate outage: `docker compose -f docker-compose.prod.yml down`
4. Provision clean recovery host (or clean test-lab VM); copy deployment bundle.
5. Restore data paths: `data/playlist/`, `data/media/`, `data/chat/`, `data/hls/` (optional warm cache).
6. Restore database from latest backup dump.
7. Start stack: `docker compose -f docker-compose.prod.yml up -d`
8. Validate health and runtime:
   ```bash
   docker ps --format "table {{.Names}}\t{{.Status}}"
   curl -fsS http://<origin>/health
   curl -fsS http://<origin>/hls/playlist.m3u8 | head -n 5
   curl -fsS http://<origin>:9090/-/ready
   ```
9. Validate app behavior: viewer SPA loads and HLS plays; WebSocket chat connects with message round-trip; admin login/session endpoint succeeds.
10. Run backup integrity check: `scripts/verify-backups.sh`
11. Record drill end timestamp; compute measured RTO/RPO.

### Pass/Fail Criteria

- **Pass:** all core services healthy and functional in ≤ 60 min.
- **Pass:** latest recoverable data point ≤ 24 h old.
- **Fail:** any critical path remains degraded (playout, chat, admin auth, monitoring).

### Evidence Template

Record in incident tracker: date/time window; environment (test-lab, staging, production); backup artifact IDs and timestamps; measured RTO and RPO; issues found and remediation owner; follow-up ticket links.