fix: parallel CD trigger, remove env blocker, cleanup release pipeline
This commit is contained in:
1 parent
74bf7e19db
commit
4c02ab9d55
13 files changed
+95
-91
No files matched your search
+17
-23
@@ -4,15 +4,11 @@ Triage order for Linear Void incidents. Work top-to-bottom — each tier depends
|
||||
|
||||
## Triage Order
|
||||
|
||||
```
|
||||
1. Edge Reachability (Cloudflare)
|
||||
2. Origin Health (VPS/Docker)
|
||||
3. Playout Progression (HLS)
|
||||
4. Chat Connectivity (WebSocket)
|
||||
5. Data-Plane (Redis/DB)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Edge Reachability (Cloudflare)
|
||||
|
||||
@@ -28,17 +24,17 @@ curl -I https://linearvoid.tv/health
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
|
||||
- DNS misconfiguration (A record pointing to wrong IP)
|
||||
- Cloudflare proxy disabled (grey cloud → origin exposed directly)
|
||||
- SSL certificate expired (check Cloudflare dashboard → SSL/TLS)
|
||||
- Cloudflare outage → https://www.cloudflarestatus.com
|
||||
- Cloudflare outage → check [https://www.cloudflarestatus.com](https://www.cloudflarestatus.com)
|
||||
|
||||
**Mitigation:**
|
||||
|
||||
- Verify A record: `linearvoid.tv` → origin IP (proxied, orange cloud)
|
||||
- Bypass Cloudflare temporarily: `curl -H "Host: linearvoid.tv" http://<origin-ip>/health`
|
||||
|
||||
---
|
||||
|
||||
## 2. Origin Health (VPS/Docker)
|
||||
|
||||
**Symptoms:** Cloudflare returns 502/504, containers down.
|
||||
@@ -53,18 +49,18 @@ curl http://192.168.88.40/health
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
|
||||
- Host out of disk space: `df -h`
|
||||
- Docker daemon stopped: `systemctl status docker`
|
||||
- Container restart loop: `docker ps -a --filter "status=exited"`
|
||||
- Resource exhaustion (OOM): `dmesg | grep -i oom`
|
||||
|
||||
**Mitigation:**
|
||||
|
||||
- Restart stack: `docker compose -f docker-compose.prod.yml up -d`
|
||||
- Free disk: `docker system prune -f`
|
||||
- Check logs per service: `docker logs linearvoid-<service> --tail 100`
|
||||
|
||||
---
|
||||
|
||||
## 3. Playout Progression (HLS)
|
||||
|
||||
**Symptoms:** Viewer sees slate or buffering, playlist not advancing.
|
||||
@@ -83,19 +79,19 @@ docker logs linearvoid-playout --tail 100
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
|
||||
- FFmpeg crash → slate fallback active (health shows `degraded`)
|
||||
- Playlist file empty/missing: `cat data/playlist/playlist.txt`
|
||||
- Media file missing from playlist: `ls data/media/`
|
||||
- Disk full → HLS segments can't write
|
||||
|
||||
**Mitigation:**
|
||||
|
||||
- Verify playlist has valid entries
|
||||
- Check media files exist at paths in playlist
|
||||
- Restart playout: `docker compose restart playout`
|
||||
- Generate demo playlist if empty: see `backend/playout/start-playout.sh`
|
||||
|
||||
---
|
||||
|
||||
## 4. Chat Connectivity (WebSocket)
|
||||
|
||||
**Symptoms:** Chat not loading, messages not sending, presence broken.
|
||||
@@ -112,18 +108,18 @@ docker logs linearvoid-chat --tail 100
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
|
||||
- Redis down → presence/rate-limiting degraded (chat still works)
|
||||
- Database down → messages not persisting
|
||||
- Port not exposed: `docker port linearvoid-chat`
|
||||
|
||||
**Mitigation:**
|
||||
|
||||
- Restart chat: `docker compose restart chat`
|
||||
- Check Redis: `docker exec linearvoid-redis redis-cli ping`
|
||||
- Check DB: `docker exec linearvoid-db pg_isready -U linearvoid`
|
||||
- Chat works without Redis (graceful degradation) — check logs for "Redis connection warning"
|
||||
|
||||
---
|
||||
|
||||
## 5. Data-Plane (Redis/DB)
|
||||
|
||||
**Symptoms:** Chat works but no persistence/presence, admin stats empty.
|
||||
@@ -139,32 +135,30 @@ docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
|
||||
- Redis OOM (check `maxmemory` setting)
|
||||
- PostgreSQL connection limit reached
|
||||
- Disk full on volume mount
|
||||
|
||||
**Mitigation:**
|
||||
|
||||
- Redis: `docker compose restart redis` (AOF persist enabled)
|
||||
- PostgreSQL: `docker compose restart db`
|
||||
- Restore from backup: see `scripts/verify-backups.sh` for procedure
|
||||
|
||||
---
|
||||
|
||||
## Escalation
|
||||
|
||||
| Severity | When | Action |
|
||||
|---|---|---|
|
||||
| Critical | Slate active >30min, complete outage | Trigger DR plan, provision fallback host |
|
||||
| Warning | Redis down, high CPU >30min | Investigate during business hours, restart service |
|
||||
| Info | Single service blip | Monitor, no action unless recurring |
|
||||
| Severity | When | Action |
|
||||
| -------- | ------------------------------------ | -------------------------------------------------- |
|
||||
| Critical | Slate active >30min, complete outage | Trigger DR plan, provision fallback host |
|
||||
| Warning | Redis down, high CPU >30min | Investigate during business hours, restart service |
|
||||
| Info | Single service blip | Monitor, no action unless recurring |
|
||||
|
||||
## Post-Incident
|
||||
|
||||
1. Document timeline in incident channel
|
||||
2. Identify root cause
|
||||
3. Add prevention to TODO.md if actionable
|
||||
3. Add prevention to docs if actionable
|
||||
4. Update this runbook if new failure mode discovered
|
||||
|
||||
---
|
||||
|
||||
**Recovery targets:** RPO ≤24h, RTO ≤60min. Quarterly DR drill: simulate host loss, restore from backup.
|
||||
Reference in new issue
Block a user