Refactor load testing scripts to support Bash and PowerShell
- Updated README.md to reflect changes in load testing commands and added usage examples for Bash scripts. - Converted PowerShell load testing scripts to Bash equivalents for better cross-platform compatibility. - Added new Bash scripts: run-loadtests.sh, summarize-artillery.sh, and summarize-k6.sh to mirror existing PowerShell functionality. - Updated admin-api.js and ws-chat.yml for consistency and improved performance metrics reporting. - Enhanced error handling and output formatting in summary scripts for better readability.
This commit is contained in:
zwitschi committed
1 parent
9f60087a4c
commit
33b6296c5e
82 files changed
+19682
-21133
No files matched your search
+57
-126
@@ -4,155 +4,101 @@ Triage order for Linear Void incidents. Work top-to-bottom — each tier depends
|
||||
|
||||
## Triage Order
|
||||
|
||||
1. Edge Reachability (Cloudflare)
|
||||
2. Origin Health (VPS/Docker)
|
||||
3. Playout Progression (HLS)
|
||||
4. Chat Connectivity (WebSocket)
|
||||
5. Data-Plane (Redis/DB)
|
||||
1. **Edge Reachability** (Cloudflare)
|
||||
2. **Origin Health** (VPS/Docker)
|
||||
3. **Playout Progression** (HLS)
|
||||
4. **Chat Connectivity** (WebSocket)
|
||||
5. **Data-Plane** (Redis/DB)
|
||||
|
||||
## 1. Edge Reachability (Cloudflare)
|
||||
|
||||
**Symptoms:** `linearvoid.tv` not resolving, SSL errors, 5xx from Cloudflare.
|
||||
|
||||
```bash
|
||||
# Check DNS
|
||||
dig linearvoid.tv
|
||||
dig www.linearvoid.tv
|
||||
|
||||
# Check Cloudflare status
|
||||
curl -I https://linearvoid.tv/health
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
**Common causes:** DNS misconfiguration (A record → wrong IP); Cloudflare proxy disabled (grey cloud exposes origin); SSL certificate expired (Cloudflare → SSL/TLS); Cloudflare outage ([cloudflarestatus.com](https://www.cloudflarestatus.com)).
|
||||
|
||||
- DNS misconfiguration (A record pointing to wrong IP)
|
||||
- Cloudflare proxy disabled (grey cloud → origin exposed directly)
|
||||
- SSL certificate expired (check Cloudflare dashboard → SSL/TLS)
|
||||
- Cloudflare outage → check [https://www.cloudflarestatus.com](https://www.cloudflarestatus.com)
|
||||
**Mitigation:** Verify A record points `linearvoid.tv` → origin IP (proxied, orange cloud). Bypass Cloudflare temporarily:
|
||||
|
||||
**Mitigation:**
|
||||
|
||||
- Verify A record: `linearvoid.tv` → origin IP (proxied, orange cloud)
|
||||
- Bypass Cloudflare temporarily: `curl -H "Host: linearvoid.tv" http://<origin-ip>/health`
|
||||
```bash
|
||||
curl -H "Host: linearvoid.tv" http://<origin-ip>/health
|
||||
```
|
||||
|
||||
## 2. Origin Health (VPS/Docker)
|
||||
|
||||
**Symptoms:** Cloudflare returns 502/504, containers down.
|
||||
**Symptoms:** Cloudflare 502/504, containers down.
|
||||
|
||||
```bash
|
||||
# Remote Docker (test lab)
|
||||
docker -H tcp://192.168.88.40:2375 ps
|
||||
docker -H tcp://192.168.88.40:2375 logs linearvoid-nginx --tail 50
|
||||
|
||||
# Direct health check
|
||||
curl http://192.168.88.40/health
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
|
||||
- Host out of disk space: `df -h`
|
||||
- Docker daemon stopped: `systemctl status docker`
|
||||
- Container restart loop: `docker ps -a --filter "status=exited"`
|
||||
- Resource exhaustion (OOM): `dmesg | grep -i oom`
|
||||
**Common causes:** Host disk full (`df -h`); Docker daemon stopped (`systemctl status docker`); container restart loop (`docker ps -a --filter "status=exited"`); OOM (`dmesg | grep -i oom`).
|
||||
|
||||
**Mitigation:**
|
||||
|
||||
- Restart stack: `docker compose -f docker-compose.prod.yml up -d`
|
||||
- Free disk: `docker system prune -f`
|
||||
- Check logs per service: `docker logs linearvoid-<service> --tail 100`
|
||||
```bash
|
||||
docker compose -f docker-compose.prod.yml up -d # restart stack
|
||||
docker system prune -f # free disk
|
||||
docker logs linearvoid-<service> --tail 100 # per-service logs
|
||||
```
|
||||
|
||||
## 3. Playout Progression (HLS)
|
||||
|
||||
**Symptoms:** Viewer sees slate or buffering, playlist not advancing.
|
||||
|
||||
```bash
|
||||
# Check playout health
|
||||
curl http://192.168.88.40:8081/health
|
||||
# Response includes "mode": "live" or "slate"
|
||||
|
||||
# Check HLS output
|
||||
curl http://192.168.88.40:8081/health # "mode": "live" or "slate"
|
||||
curl -I http://192.168.88.40/hls/playlist.m3u8
|
||||
ls -la data/hls/
|
||||
|
||||
# Check playout logs
|
||||
docker logs linearvoid-playout --tail 100
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
**Common causes:** FFmpeg crash → slate fallback (health `degraded`); playlist empty/missing (`cat data/playlist/playlist.txt`); media file missing (`ls data/media/`); disk full → HLS segments can't write.
|
||||
|
||||
- FFmpeg crash → slate fallback active (health shows `degraded`)
|
||||
- Playlist file empty/missing: `cat data/playlist/playlist.txt`
|
||||
- Media file missing from playlist: `ls data/media/`
|
||||
- Disk full → HLS segments can't write
|
||||
|
||||
**Mitigation:**
|
||||
|
||||
- Verify playlist has valid entries
|
||||
- Check media files exist at paths in playlist
|
||||
- Restart playout: `docker compose restart playout`
|
||||
- Generate demo playlist if empty: see `backend/playout/start-playout.sh`
|
||||
**Mitigation:** Validate playlist entries and media paths; restart playout (`docker compose restart playout`); regenerate demo playlist if empty (see `backend/playout/start-playout.sh`).
|
||||
|
||||
## 4. Chat Connectivity (WebSocket)
|
||||
|
||||
**Symptoms:** Chat not loading, messages not sending, presence broken.
|
||||
|
||||
```bash
|
||||
# Check chat health
|
||||
curl http://192.168.88.40:3000/health
|
||||
|
||||
# Test WebSocket
|
||||
wscat -c ws://192.168.88.40/ws?username=test&room=main
|
||||
|
||||
# Check chat logs
|
||||
docker logs linearvoid-chat --tail 100
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
**Common causes:** Redis down → presence/rate-limiting degraded (chat still works); DB down → messages not persisting; port not exposed (`docker port linearvoid-chat`).
|
||||
|
||||
- Redis down → presence/rate-limiting degraded (chat still works)
|
||||
- Database down → messages not persisting
|
||||
- Port not exposed: `docker port linearvoid-chat`
|
||||
|
||||
**Mitigation:**
|
||||
|
||||
- Restart chat: `docker compose restart chat`
|
||||
- Check Redis: `docker exec linearvoid-redis redis-cli ping`
|
||||
- Check DB: `docker exec linearvoid-db pg_isready -U linearvoid`
|
||||
- Chat works without Redis (graceful degradation) — check logs for "Redis connection warning"
|
||||
**Mitigation:** Restart chat (`docker compose restart chat`); check Redis (`docker exec linearvoid-redis redis-cli ping`); check DB (`docker exec linearvoid-db pg_isready -U linearvoid`). Chat degrades gracefully without Redis — look for "Redis connection warning" in logs.
|
||||
|
||||
## 5. Data-Plane (Redis/DB)
|
||||
|
||||
**Symptoms:** Chat works but no persistence/presence, admin stats empty.
|
||||
|
||||
```bash
|
||||
# Redis
|
||||
docker exec linearvoid-redis redis-cli ping
|
||||
docker exec linearvoid-redis redis-cli info memory
|
||||
|
||||
# PostgreSQL
|
||||
docker exec linearvoid-db pg_isready -U linearvoid
|
||||
docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
**Common causes:** Redis OOM (check `maxmemory`); PostgreSQL connection limit; disk full on volume mount.
|
||||
|
||||
- Redis OOM (check `maxmemory` setting)
|
||||
- PostgreSQL connection limit reached
|
||||
- Disk full on volume mount
|
||||
|
||||
**Mitigation:**
|
||||
|
||||
- Redis: `docker compose restart redis` (AOF persist enabled)
|
||||
- PostgreSQL: `docker compose restart db`
|
||||
- Restore from backup: see `scripts/verify-backups.sh` for procedure
|
||||
**Mitigation:** Restart redis (`docker compose restart redis`, AOF persist enabled); restart db (`docker compose restart db`); restore from backup (see `scripts/verify-backups.sh`).
|
||||
|
||||
## Escalation
|
||||
|
||||
| Severity | When | Action |
|
||||
| -------- | ------------------------------------ | -------------------------------------------------- |
|
||||
| Critical | Slate active >30min, complete outage | Trigger DR plan, provision fallback host |
|
||||
| Warning | Redis down, high CPU >30min | Investigate during business hours, restart service |
|
||||
| Info | Single service blip | Monitor, no action unless recurring |
|
||||
| Severity | When | Action |
|
||||
| -------- | ------------------------------------ | ------------------------------------------- |
|
||||
| Critical | Slate active >30min, complete outage | Trigger DR plan, provision fallback host |
|
||||
| Warning | Redis down, high CPU >30min | Investigate business hours, restart service |
|
||||
| Info | Single service blip | Monitor; no action unless recurring |
|
||||
|
||||
## Post-Incident
|
||||
|
||||
@@ -163,67 +109,52 @@ docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"
|
||||
|
||||
## Quarterly Disaster Recovery Drill
|
||||
|
||||
Goal: validate recovery targets under controlled host-loss simulation.
|
||||
|
||||
**Recovery targets:** RPO <= 24h, RTO <= 60min.
|
||||
Validate recovery targets under controlled host-loss simulation: **RPO ≤ 24h, RTO ≤ 60min**.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Latest backup artifacts available from `scripts/backup-db.sh` and `scripts/backup-weekly.sh`
|
||||
- Latest backups from `scripts/backup-db.sh` and `scripts/backup-weekly.sh`
|
||||
- Operator access to Docker host and DNS controls
|
||||
- Current deployment variables available (`.env`, compose file, image tags)
|
||||
- Current deployment variables (`.env`, compose file, image tags)
|
||||
|
||||
### Drill Scope
|
||||
### Scope
|
||||
|
||||
- Simulate complete origin loss (all app containers unavailable)
|
||||
- Rebuild stack on clean host
|
||||
- Restore DB and media/playlist mounts
|
||||
- Verify viewer playback, chat, admin auth, monitoring targets
|
||||
|
||||
### Drill Steps
|
||||
### Steps
|
||||
|
||||
1. Record drill start timestamp and on-call participants.
|
||||
2. Capture pre-drill baseline:
|
||||
- `curl -fsS http://<origin>/health`
|
||||
- `curl -fsS http://<origin>:3000/health`
|
||||
- `curl -fsS http://<origin>:8081/health`
|
||||
3. Simulate origin outage:
|
||||
- `docker compose -f docker-compose.prod.yml down`
|
||||
4. Provision clean recovery host (or clean test-lab VM) and copy deployment bundle.
|
||||
5. Restore required data paths:
|
||||
- `data/playlist/`
|
||||
- `data/media/`
|
||||
- `data/chat/`
|
||||
- `data/hls/` (optional warm cache)
|
||||
```bash
|
||||
curl -fsS http://<origin>/health
|
||||
curl -fsS http://<origin>:3000/health
|
||||
curl -fsS http://<origin>:8081/health
|
||||
```
|
||||
3. Simulate outage: `docker compose -f docker-compose.prod.yml down`
|
||||
4. Provision clean recovery host (or clean test-lab VM); copy deployment bundle.
|
||||
5. Restore data paths: `data/playlist/`, `data/media/`, `data/chat/`, `data/hls/` (optional warm cache).
|
||||
6. Restore database from latest backup dump.
|
||||
7. Start stack:
|
||||
- `docker compose -f docker-compose.prod.yml up -d`
|
||||
8. Validate service health and runtime:
|
||||
- `docker ps --format "table {{.Names}}\t{{.Status}}"`
|
||||
- `curl -fsS http://<origin>/health`
|
||||
- `curl -fsS http://<origin>/hls/playlist.m3u8 | head -n 5`
|
||||
- `curl -fsS http://<origin>:9090/-/ready`
|
||||
9. Validate app behavior:
|
||||
- Viewer SPA loads and HLS plays
|
||||
- WebSocket chat connects and message round-trip works
|
||||
- Admin login/session endpoint returns success
|
||||
10. Run backup integrity check:
|
||||
- `scripts/verify-backups.sh`
|
||||
11. Record drill end timestamp and compute measured RTO/RPO.
|
||||
7. Start stack: `docker compose -f docker-compose.prod.yml up -d`
|
||||
8. Validate health and runtime:
|
||||
```bash
|
||||
docker ps --format "table {{.Names}}\t{{.Status}}"
|
||||
curl -fsS http://<origin>/health
|
||||
curl -fsS http://<origin>/hls/playlist.m3u8 | head -n 5
|
||||
curl -fsS http://<origin>:9090/-/ready
|
||||
```
|
||||
9. Validate app behavior: viewer SPA loads and HLS plays; WebSocket chat connects with message round-trip; admin login/session endpoint succeeds.
|
||||
10. Run backup integrity check: `scripts/verify-backups.sh`
|
||||
11. Record drill end timestamp; compute measured RTO/RPO.
|
||||
|
||||
### Pass/Fail Criteria
|
||||
|
||||
- Pass if all core services healthy and functional in <= 60 minutes.
|
||||
- Pass if latest recoverable data point is <= 24 hours old.
|
||||
- Fail if any critical path remains degraded (playout, chat, admin auth, monitoring).
|
||||
- **Pass:** all core services healthy and functional in ≤ 60 min.
|
||||
- **Pass:** latest recoverable data point ≤ 24 h old.
|
||||
- **Fail:** any critical path remains degraded (playout, chat, admin auth, monitoring).
|
||||
|
||||
### Evidence Template
|
||||
|
||||
Document each drill in incident tracker with:
|
||||
|
||||
- Date/time window
|
||||
- Environment (test-lab, staging, production)
|
||||
- Backup artifact IDs and timestamps
|
||||
- Measured RTO and RPO
|
||||
- Issues found and remediation owner
|
||||
- Follow-up ticket links
|
||||
Record in incident tracker: date/time window; environment (test-lab, staging, production); backup artifact IDs and timestamps; measured RTO and RPO; issues found and remediation owner; follow-up ticket links.
|
||||
Reference in new issue
Block a user