Refactor load testing scripts to support Bash and PowerShell
- Updated README.md to reflect changes in load testing commands and added usage examples for Bash scripts. - Converted PowerShell load testing scripts to Bash equivalents for better cross-platform compatibility. - Added new Bash scripts: run-loadtests.sh, summarize-artillery.sh, and summarize-k6.sh to mirror existing PowerShell functionality. - Updated admin-api.js and ws-chat.yml for consistency and improved performance metrics reporting. - Enhanced error handling and output formatting in summary scripts for better readability.
This commit is contained in:
zwitschi committed
1 parent
9f60087a4c
commit
33b6296c5e
82 files changed
+19682
-21133
No files matched your search
+136
-136
@@ -1,136 +1,136 @@
|
||||
# BACKLOG
|
||||
|
||||
> All current work is planned in [TODO](./TODO.md) (2026-08-13). This file lists backlog items; new unplanned work is added here.
|
||||
|
||||
## Setup phases remaining
|
||||
|
||||
### Phase 5: complete manual Cloudflare and external-service setup
|
||||
|
||||
- [ ] Apply Cloudflare Workers, Tiered Cache, WAF, rate limiting, and HTTPS/HSTS settings in the dashboard
|
||||
- [ ] Cloudflare Workers: link custom domain and cut over frontend hosting
|
||||
- [ ] Enable Smart Tiered Cache; validate `Cf-Cache-Status: HIT` from upper tier
|
||||
- [ ] WAF: enable OWASP Managed Ruleset (Simulate → review 48h → Block), enable Bot Fight Mode
|
||||
- [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action
|
||||
- [ ] HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2; enforce at edge and origin
|
||||
- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
|
||||
- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword `#EXTM3U`, 5min)
|
||||
- [ ] UptimeRobot: alert on 2 consecutive failures via email + Discord/Slack webhook
|
||||
- [ ] Capture evidence in `docs/operations/cloudflare-uptimerobot-checklist.md`
|
||||
|
||||
### Phase 6: rehearse production deployment end to end
|
||||
|
||||
- [ ] Provision Hetzner CX22 (2 vCPU, 4 GB, 40 GB) in Ashburn VA, Ubuntu 24.04
|
||||
- [ ] Install Docker Engine + Compose plugin; open firewall: 80/443 public, 2375 runner-only
|
||||
- [ ] Configure Docker daemon for remote TLS access (`-H tcp://0.0.0.0:2376` with certs)
|
||||
- [ ] Create `/srv/linearvoid/data/{playlist,media,hls,chat,slate}` directories
|
||||
- [ ] Add `CI_PROD_*` secrets to Gitea (DOCKER_HOST, R2 keys, admin password, DB password, Grafana password)
|
||||
- [ ] Trigger CD workflow with `DOCKER_HOST` pointing to Hetzner IP
|
||||
- [ ] Verify all 11 containers healthy: `/health`, `/hls/playlist.m3u8`, WebSocket
|
||||
- [ ] Update Cloudflare A record `linearvoid.tv` → Hetzner IP, SSL Full (strict); test HTTPS + WSS
|
||||
- [ ] Smoke test: viewer SPA loads, HLS plays, chat connects, admin login → upload → playlist → playout
|
||||
- [ ] Set up Prometheus alerting notifications + UptimeRobot on production URL
|
||||
- [ ] Run `scripts/backup-db.sh` + `scripts/verify-backups.sh` on production host
|
||||
- [ ] Document production IP, SSH access, rollback procedure, scaling thresholds
|
||||
|
||||
### Phase 7: baseline scaling and performance
|
||||
|
||||
- [ ] Execute WebSocket load tests — ramp 10→100→500→1000 connections; measure p95 latency + errors
|
||||
- [ ] Execute admin API benchmarks — `wrk`/`k6` against `/api/admin/*`; target p95 <1s
|
||||
- [ ] Measure manifest TTFB and cache-hit distribution from multiple US regions (WebPageTest 3+ regions, <4s warm cache)
|
||||
- [ ] Tune manifest TTL to 0.5× segment duration; validate `Cf-Cache-Status` distribution
|
||||
- [ ] Track cache-hit ratio and R2 ops in production (Cloudflare GraphQL → Grafana panel, alert <90%)
|
||||
- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min
|
||||
- [ ] Plan Phase 2 service separation (playout CPU, chat latency, DB state, monitoring)
|
||||
- [ ] Plan Phase 3 multi-origin resilience (second Hetzner location, warm standby, DNS failover, DB/Redis replication)
|
||||
|
||||
### Phase 8: close out remaining test and validation gaps
|
||||
|
||||
- [ ] Admin frontend: user moderation tools tests (mute/ban/report, view moderation logs)
|
||||
- [ ] Frontend: coverage for all critical user + admin components and interactions
|
||||
- [ ] Frontend: document test cases and expected outcomes
|
||||
- [ ] Backend R2: test retrieval of segments/manifests via Cloudflare edge
|
||||
- [ ] Backend R2: validate edge cache hit/miss behavior with repeat requests (`scripts/validate-edge-cache.sh`)
|
||||
|
||||
### Playout
|
||||
|
||||
- [ ] Configure an R2 lifecycle rule on the `hls/` prefix as defense-in-depth (manual)
|
||||
- [ ] Full-loop E2E (2-file playlist, assert `Looping...` with no `CRASHED` across the boundary) — manual via `scripts/test-playout-e2e.sh`
|
||||
|
||||
## Cloudflare Configuration
|
||||
|
||||
### DNS & Workers
|
||||
|
||||
- [ ] Cloudflare Workers: link custom domain (manual)
|
||||
- [ ] Create Cloudflare Worker and cut over frontend hosting (manual)
|
||||
|
||||
### Security & TLS
|
||||
|
||||
- [ ] Enable Tiered Cache, WAF, rate limiting (manual)
|
||||
- [ ] Tiered Cache: enable Smart Tiered Cache, validate `Cf-Cache-Status: HIT` from upper tier
|
||||
- [ ] WAF: enable OWASP Managed Ruleset, set to Simulate → review 48h → Block, enable Bot Fight Mode
|
||||
- [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action
|
||||
- [ ] HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2
|
||||
- [ ] HTTPS/WSS: enforce at edge (Cloudflare) and origin (nginx), redirect HTTP → HTTPS
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
### External Monitoring
|
||||
|
||||
- [ ] Set up UptimeRobot external monitoring
|
||||
- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
|
||||
- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword: `#EXTM3U`, 5min)
|
||||
- [ ] Alert on 2 consecutive failures, notify via email + Discord/Slack webhook
|
||||
|
||||
## External Services
|
||||
|
||||
> Evidence tracker: `docs/operations/cloudflare-uptimerobot-checklist.md`
|
||||
|
||||
- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
|
||||
- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword: `#EXTM3U`, 5min)
|
||||
- [ ] Alert on 2 consecutive failures, notify via email + Discord/Slack webhook
|
||||
|
||||
## Production Deployment
|
||||
|
||||
### Host Provisioning (Hetzner — not provisioned)
|
||||
|
||||
- [ ] Provision Hetzner CX22 (2 vCPU, 4 GB, 40 GB) in Ashburn VA, Ubuntu 24.04
|
||||
- [ ] Install Docker Engine + Compose plugin, open firewall: 80/443 (public), 2375 (runner IP only)
|
||||
- [ ] Configure Docker daemon for remote TLS access (`-H tcp://0.0.0.0:2376` with certs)
|
||||
- [ ] Create `/srv/linearvoid/data/{playlist,media,hls,chat,slate}` directories
|
||||
|
||||
### CI/CD & Secrets
|
||||
|
||||
- [ ] Add `CI_PROD_*` secrets to Gitea: DOCKER_HOST, R2 keys, admin password, DB password, Grafana password
|
||||
- [ ] Trigger CD workflow with `DOCKER_HOST` pointing to Hetzner IP
|
||||
|
||||
### Smoke Test & Cutover
|
||||
|
||||
- [ ] Verify all 11 containers start healthy, check `/health`, `/hls/playlist.m3u8`, WebSocket
|
||||
- [ ] Update Cloudflare A record `linearvoid.tv` → Hetzner IP, SSL Full (strict), test HTTPS + WSS
|
||||
- [ ] Smoke test: viewer SPA loads, HLS plays, chat connects, admin login → upload → playlist → playout
|
||||
|
||||
### Production Operations
|
||||
|
||||
- [ ] Set up Prometheus alerting notifications + UptimeRobot on production URL
|
||||
- [ ] Run `scripts/backup-db.sh` + `scripts/verify-backups.sh` on production host
|
||||
- [ ] Document production IP, SSH access, rollback procedure, scaling thresholds
|
||||
|
||||
## Scaling & Performance
|
||||
|
||||
### Load Testing
|
||||
|
||||
- [ ] Execute WebSocket load tests with real infrastructure — ramp 10→100→500→1000 connections, measure p95 latency + errors
|
||||
- [ ] Execute admin API benchmarks with real infrastructure — `wrk`/`k6` against `/api/admin/*`, target p95 <1s
|
||||
|
||||
### Cache & TTFB Optimization
|
||||
|
||||
- [ ] Optimize segment immutability and manifest TTL for cache efficiency
|
||||
- [ ] Tune manifest TTL: measure `Cf-Cache-Status` distribution, adjust to 0.5× segment duration
|
||||
- [ ] Measure playback TTFB from multiple US regions — WebPageTest from 3+ regions, target <4s warm cache
|
||||
- [ ] Track cache hit ratio and R2 ops in production — Cloudflare GraphQL → Grafana panel, alert <90%
|
||||
|
||||
### Capacity & Future Planning
|
||||
|
||||
- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min
|
||||
- [ ] Plan Phase 2 service separation — document boundaries (playout CPU, chat latency, DB state, monitoring), estimate per-service resources
|
||||
- [ ] Plan Phase 3 multi-origin resilience — research second Hetzner location, design warm standby + DNS failover + DB/Redis replication
|
||||
# BACKLOG
|
||||
|
||||
> All current work is planned in [TODO](./TODO.md) (2026-08-13). This file lists backlog items; new unplanned work is added here.
|
||||
|
||||
## Setup phases remaining
|
||||
|
||||
### Phase 5: complete manual Cloudflare and external-service setup
|
||||
|
||||
- [ ] Apply Cloudflare Workers, Tiered Cache, WAF, rate limiting, and HTTPS/HSTS settings in the dashboard
|
||||
- [ ] Cloudflare Workers: link custom domain and cut over frontend hosting
|
||||
- [ ] Enable Smart Tiered Cache; validate `Cf-Cache-Status: HIT` from upper tier
|
||||
- [ ] WAF: enable OWASP Managed Ruleset (Simulate → review 48h → Block), enable Bot Fight Mode
|
||||
- [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action
|
||||
- [ ] HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2; enforce at edge and origin
|
||||
- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
|
||||
- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword `#EXTM3U`, 5min)
|
||||
- [ ] UptimeRobot: alert on 2 consecutive failures via email + Discord/Slack webhook
|
||||
- [ ] Capture evidence in `docs/operations/cloudflare-uptimerobot-checklist.md`
|
||||
|
||||
### Phase 6: rehearse production deployment end to end
|
||||
|
||||
- [ ] Provision Hetzner CX22 (2 vCPU, 4 GB, 40 GB) in Ashburn VA, Ubuntu 24.04
|
||||
- [ ] Install Docker Engine + Compose plugin; open firewall: 80/443 public, 2375 runner-only
|
||||
- [ ] Configure Docker daemon for remote TLS access (`-H tcp://0.0.0.0:2376` with certs)
|
||||
- [ ] Create `/srv/linearvoid/data/{playlist,media,hls,chat,slate}` directories
|
||||
- [ ] Add `CI_PROD_*` secrets to Gitea (DOCKER_HOST, R2 keys, admin password, DB password, Grafana password)
|
||||
- [ ] Trigger CD workflow with `DOCKER_HOST` pointing to Hetzner IP
|
||||
- [ ] Verify all 11 containers healthy: `/health`, `/hls/playlist.m3u8`, WebSocket
|
||||
- [ ] Update Cloudflare A record `linearvoid.tv` → Hetzner IP, SSL Full (strict); test HTTPS + WSS
|
||||
- [ ] Smoke test: viewer SPA loads, HLS plays, chat connects, admin login → upload → playlist → playout
|
||||
- [ ] Set up Prometheus alerting notifications + UptimeRobot on production URL
|
||||
- [ ] Run `scripts/backup-db.sh` + `scripts/verify-backups.sh` on production host
|
||||
- [ ] Document production IP, SSH access, rollback procedure, scaling thresholds
|
||||
|
||||
### Phase 7: baseline scaling and performance
|
||||
|
||||
- [ ] Execute WebSocket load tests — ramp 10→100→500→1000 connections; measure p95 latency + errors
|
||||
- [ ] Execute admin API benchmarks — `wrk`/`k6` against `/api/admin/*`; target p95 <1s
|
||||
- [ ] Measure manifest TTFB and cache-hit distribution from multiple US regions (WebPageTest 3+ regions, <4s warm cache)
|
||||
- [ ] Tune manifest TTL to 0.5× segment duration; validate `Cf-Cache-Status` distribution
|
||||
- [ ] Track cache-hit ratio and R2 ops in production (Cloudflare GraphQL → Grafana panel, alert <90%)
|
||||
- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min
|
||||
- [ ] Plan Phase 2 service separation (playout CPU, chat latency, DB state, monitoring)
|
||||
- [ ] Plan Phase 3 multi-origin resilience (second Hetzner location, warm standby, DNS failover, DB/Redis replication)
|
||||
|
||||
### Phase 8: close out remaining test and validation gaps
|
||||
|
||||
- [ ] Admin frontend: user moderation tools tests (mute/ban/report, view moderation logs)
|
||||
- [ ] Frontend: coverage for all critical user + admin components and interactions
|
||||
- [ ] Frontend: document test cases and expected outcomes
|
||||
- [ ] Backend R2: test retrieval of segments/manifests via Cloudflare edge
|
||||
- [ ] Backend R2: validate edge cache hit/miss behavior with repeat requests (`scripts/validate-edge-cache.sh`)
|
||||
|
||||
### Playout
|
||||
|
||||
- [ ] Configure an R2 lifecycle rule on the `hls/` prefix as defense-in-depth (manual)
|
||||
- [ ] Full-loop E2E (2-file playlist, assert `Looping...` with no `CRASHED` across the boundary) — manual via `scripts/test-playout-e2e.sh`
|
||||
|
||||
## Cloudflare Configuration
|
||||
|
||||
### DNS & Workers
|
||||
|
||||
- [ ] Cloudflare Workers: link custom domain (manual)
|
||||
- [ ] Create Cloudflare Worker and cut over frontend hosting (manual)
|
||||
|
||||
### Security & TLS
|
||||
|
||||
- [ ] Enable Tiered Cache, WAF, rate limiting (manual)
|
||||
- [ ] Tiered Cache: enable Smart Tiered Cache, validate `Cf-Cache-Status: HIT` from upper tier
|
||||
- [ ] WAF: enable OWASP Managed Ruleset, set to Simulate → review 48h → Block, enable Bot Fight Mode
|
||||
- [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action
|
||||
- [ ] HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2
|
||||
- [ ] HTTPS/WSS: enforce at edge (Cloudflare) and origin (nginx), redirect HTTP → HTTPS
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
### External Monitoring
|
||||
|
||||
- [ ] Set up UptimeRobot external monitoring
|
||||
- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
|
||||
- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword: `#EXTM3U`, 5min)
|
||||
- [ ] Alert on 2 consecutive failures, notify via email + Discord/Slack webhook
|
||||
|
||||
## External Services
|
||||
|
||||
> Evidence tracker: `docs/operations/cloudflare-uptimerobot-checklist.md`
|
||||
|
||||
- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
|
||||
- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword: `#EXTM3U`, 5min)
|
||||
- [ ] Alert on 2 consecutive failures, notify via email + Discord/Slack webhook
|
||||
|
||||
## Production Deployment
|
||||
|
||||
### Host Provisioning (Hetzner — not provisioned)
|
||||
|
||||
- [ ] Provision Hetzner CX22 (2 vCPU, 4 GB, 40 GB) in Ashburn VA, Ubuntu 24.04
|
||||
- [ ] Install Docker Engine + Compose plugin, open firewall: 80/443 (public), 2375 (runner IP only)
|
||||
- [ ] Configure Docker daemon for remote TLS access (`-H tcp://0.0.0.0:2376` with certs)
|
||||
- [ ] Create `/srv/linearvoid/data/{playlist,media,hls,chat,slate}` directories
|
||||
|
||||
### CI/CD & Secrets
|
||||
|
||||
- [ ] Add `CI_PROD_*` secrets to Gitea: DOCKER_HOST, R2 keys, admin password, DB password, Grafana password
|
||||
- [ ] Trigger CD workflow with `DOCKER_HOST` pointing to Hetzner IP
|
||||
|
||||
### Smoke Test & Cutover
|
||||
|
||||
- [ ] Verify all 11 containers start healthy, check `/health`, `/hls/playlist.m3u8`, WebSocket
|
||||
- [ ] Update Cloudflare A record `linearvoid.tv` → Hetzner IP, SSL Full (strict), test HTTPS + WSS
|
||||
- [ ] Smoke test: viewer SPA loads, HLS plays, chat connects, admin login → upload → playlist → playout
|
||||
|
||||
### Production Operations
|
||||
|
||||
- [ ] Set up Prometheus alerting notifications + UptimeRobot on production URL
|
||||
- [ ] Run `scripts/backup-db.sh` + `scripts/verify-backups.sh` on production host
|
||||
- [ ] Document production IP, SSH access, rollback procedure, scaling thresholds
|
||||
|
||||
## Scaling & Performance
|
||||
|
||||
### Load Testing
|
||||
|
||||
- [ ] Execute WebSocket load tests with real infrastructure — ramp 10→100→500→1000 connections, measure p95 latency + errors
|
||||
- [ ] Execute admin API benchmarks with real infrastructure — `wrk`/`k6` against `/api/admin/*`, target p95 <1s
|
||||
|
||||
### Cache & TTFB Optimization
|
||||
|
||||
- [ ] Optimize segment immutability and manifest TTL for cache efficiency
|
||||
- [ ] Tune manifest TTL: measure `Cf-Cache-Status` distribution, adjust to 0.5× segment duration
|
||||
- [ ] Measure playback TTFB from multiple US regions — WebPageTest from 3+ regions, target <4s warm cache
|
||||
- [ ] Track cache hit ratio and R2 ops in production — Cloudflare GraphQL → Grafana panel, alert <90%
|
||||
|
||||
### Capacity & Future Planning
|
||||
|
||||
- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min
|
||||
- [ ] Plan Phase 2 service separation — document boundaries (playout CPU, chat latency, DB state, monitoring), estimate per-service resources
|
||||
- [ ] Plan Phase 3 multi-origin resilience — research second Hetzner location, design warm standby + DNS failover + DB/Redis replication
|
||||
Reference in new issue
Block a user