Refactor load testing scripts to support Bash and PowerShell
CI / test (push) Successful in 2m50s
CI / docker (push) Successful in 5m12s

- Updated README.md to reflect changes in load testing commands and added usage examples for Bash scripts.
- Converted PowerShell load testing scripts to Bash equivalents for better cross-platform compatibility.
- Added new Bash scripts: run-loadtests.sh, summarize-artillery.sh, and summarize-k6.sh to mirror existing PowerShell functionality.
- Updated admin-api.js and ws-chat.yml for consistency and improved performance metrics reporting.
- Enhanced error handling and output formatting in summary scripts for better readability.
This commit is contained in:
zwitschi committed 2026-09-02 19:47:34 +02:00
1 parent 9f60087a4c
commit 33b6296c5e
82 files changed
+19682 -21133

No files matched your search

+136 -136
View File
@@ -1,136 +1,136 @@
# BACKLOG
> All current work is planned in [TODO](./TODO.md) (2026-08-13). This file lists backlog items; new unplanned work is added here.
## Setup phases remaining
### Phase 5: complete manual Cloudflare and external-service setup
- [ ] Apply Cloudflare Workers, Tiered Cache, WAF, rate limiting, and HTTPS/HSTS settings in the dashboard
- [ ] Cloudflare Workers: link custom domain and cut over frontend hosting
- [ ] Enable Smart Tiered Cache; validate `Cf-Cache-Status: HIT` from upper tier
- [ ] WAF: enable OWASP Managed Ruleset (Simulate → review 48h → Block), enable Bot Fight Mode
- [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action
- [ ] HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2; enforce at edge and origin
- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword `#EXTM3U`, 5min)
- [ ] UptimeRobot: alert on 2 consecutive failures via email + Discord/Slack webhook
- [ ] Capture evidence in `docs/operations/cloudflare-uptimerobot-checklist.md`
### Phase 6: rehearse production deployment end to end
- [ ] Provision Hetzner CX22 (2 vCPU, 4 GB, 40 GB) in Ashburn VA, Ubuntu 24.04
- [ ] Install Docker Engine + Compose plugin; open firewall: 80/443 public, 2375 runner-only
- [ ] Configure Docker daemon for remote TLS access (`-H tcp://0.0.0.0:2376` with certs)
- [ ] Create `/srv/linearvoid/data/{playlist,media,hls,chat,slate}` directories
- [ ] Add `CI_PROD_*` secrets to Gitea (DOCKER_HOST, R2 keys, admin password, DB password, Grafana password)
- [ ] Trigger CD workflow with `DOCKER_HOST` pointing to Hetzner IP
- [ ] Verify all 11 containers healthy: `/health`, `/hls/playlist.m3u8`, WebSocket
- [ ] Update Cloudflare A record `linearvoid.tv` → Hetzner IP, SSL Full (strict); test HTTPS + WSS
- [ ] Smoke test: viewer SPA loads, HLS plays, chat connects, admin login → upload → playlist → playout
- [ ] Set up Prometheus alerting notifications + UptimeRobot on production URL
- [ ] Run `scripts/backup-db.sh` + `scripts/verify-backups.sh` on production host
- [ ] Document production IP, SSH access, rollback procedure, scaling thresholds
### Phase 7: baseline scaling and performance
- [ ] Execute WebSocket load tests — ramp 10→100→500→1000 connections; measure p95 latency + errors
- [ ] Execute admin API benchmarks — `wrk`/`k6` against `/api/admin/*`; target p95 <1s
- [ ] Measure manifest TTFB and cache-hit distribution from multiple US regions (WebPageTest 3+ regions, <4s warm cache)
- [ ] Tune manifest TTL to 0.5× segment duration; validate `Cf-Cache-Status` distribution
- [ ] Track cache-hit ratio and R2 ops in production (Cloudflare GraphQL → Grafana panel, alert <90%)
- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min
- [ ] Plan Phase 2 service separation (playout CPU, chat latency, DB state, monitoring)
- [ ] Plan Phase 3 multi-origin resilience (second Hetzner location, warm standby, DNS failover, DB/Redis replication)
### Phase 8: close out remaining test and validation gaps
- [ ] Admin frontend: user moderation tools tests (mute/ban/report, view moderation logs)
- [ ] Frontend: coverage for all critical user + admin components and interactions
- [ ] Frontend: document test cases and expected outcomes
- [ ] Backend R2: test retrieval of segments/manifests via Cloudflare edge
- [ ] Backend R2: validate edge cache hit/miss behavior with repeat requests (`scripts/validate-edge-cache.sh`)
### Playout
- [ ] Configure an R2 lifecycle rule on the `hls/` prefix as defense-in-depth (manual)
- [ ] Full-loop E2E (2-file playlist, assert `Looping...` with no `CRASHED` across the boundary) — manual via `scripts/test-playout-e2e.sh`
## Cloudflare Configuration
### DNS & Workers
- [ ] Cloudflare Workers: link custom domain (manual)
- [ ] Create Cloudflare Worker and cut over frontend hosting (manual)
### Security & TLS
- [ ] Enable Tiered Cache, WAF, rate limiting (manual)
- [ ] Tiered Cache: enable Smart Tiered Cache, validate `Cf-Cache-Status: HIT` from upper tier
- [ ] WAF: enable OWASP Managed Ruleset, set to Simulate → review 48h → Block, enable Bot Fight Mode
- [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action
- [ ] HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2
- [ ] HTTPS/WSS: enforce at edge (Cloudflare) and origin (nginx), redirect HTTP → HTTPS
## Monitoring & Observability
### External Monitoring
- [ ] Set up UptimeRobot external monitoring
- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword: `#EXTM3U`, 5min)
- [ ] Alert on 2 consecutive failures, notify via email + Discord/Slack webhook
## External Services
> Evidence tracker: `docs/operations/cloudflare-uptimerobot-checklist.md`
- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword: `#EXTM3U`, 5min)
- [ ] Alert on 2 consecutive failures, notify via email + Discord/Slack webhook
## Production Deployment
### Host Provisioning (Hetzner — not provisioned)
- [ ] Provision Hetzner CX22 (2 vCPU, 4 GB, 40 GB) in Ashburn VA, Ubuntu 24.04
- [ ] Install Docker Engine + Compose plugin, open firewall: 80/443 (public), 2375 (runner IP only)
- [ ] Configure Docker daemon for remote TLS access (`-H tcp://0.0.0.0:2376` with certs)
- [ ] Create `/srv/linearvoid/data/{playlist,media,hls,chat,slate}` directories
### CI/CD & Secrets
- [ ] Add `CI_PROD_*` secrets to Gitea: DOCKER_HOST, R2 keys, admin password, DB password, Grafana password
- [ ] Trigger CD workflow with `DOCKER_HOST` pointing to Hetzner IP
### Smoke Test & Cutover
- [ ] Verify all 11 containers start healthy, check `/health`, `/hls/playlist.m3u8`, WebSocket
- [ ] Update Cloudflare A record `linearvoid.tv` → Hetzner IP, SSL Full (strict), test HTTPS + WSS
- [ ] Smoke test: viewer SPA loads, HLS plays, chat connects, admin login → upload → playlist → playout
### Production Operations
- [ ] Set up Prometheus alerting notifications + UptimeRobot on production URL
- [ ] Run `scripts/backup-db.sh` + `scripts/verify-backups.sh` on production host
- [ ] Document production IP, SSH access, rollback procedure, scaling thresholds
## Scaling & Performance
### Load Testing
- [ ] Execute WebSocket load tests with real infrastructure — ramp 10→100→500→1000 connections, measure p95 latency + errors
- [ ] Execute admin API benchmarks with real infrastructure — `wrk`/`k6` against `/api/admin/*`, target p95 <1s
### Cache & TTFB Optimization
- [ ] Optimize segment immutability and manifest TTL for cache efficiency
- [ ] Tune manifest TTL: measure `Cf-Cache-Status` distribution, adjust to 0.5× segment duration
- [ ] Measure playback TTFB from multiple US regions — WebPageTest from 3+ regions, target <4s warm cache
- [ ] Track cache hit ratio and R2 ops in production — Cloudflare GraphQL → Grafana panel, alert <90%
### Capacity & Future Planning
- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min
- [ ] Plan Phase 2 service separation — document boundaries (playout CPU, chat latency, DB state, monitoring), estimate per-service resources
- [ ] Plan Phase 3 multi-origin resilience — research second Hetzner location, design warm standby + DNS failover + DB/Redis replication
# BACKLOG
> All current work is planned in [TODO](./TODO.md) (2026-08-13). This file lists backlog items; new unplanned work is added here.
## Setup phases remaining
### Phase 5: complete manual Cloudflare and external-service setup
- [ ] Apply Cloudflare Workers, Tiered Cache, WAF, rate limiting, and HTTPS/HSTS settings in the dashboard
- [ ] Cloudflare Workers: link custom domain and cut over frontend hosting
- [ ] Enable Smart Tiered Cache; validate `Cf-Cache-Status: HIT` from upper tier
- [ ] WAF: enable OWASP Managed Ruleset (Simulate → review 48h → Block), enable Bot Fight Mode
- [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action
- [ ] HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2; enforce at edge and origin
- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword `#EXTM3U`, 5min)
- [ ] UptimeRobot: alert on 2 consecutive failures via email + Discord/Slack webhook
- [ ] Capture evidence in `docs/operations/cloudflare-uptimerobot-checklist.md`
### Phase 6: rehearse production deployment end to end
- [ ] Provision Hetzner CX22 (2 vCPU, 4 GB, 40 GB) in Ashburn VA, Ubuntu 24.04
- [ ] Install Docker Engine + Compose plugin; open firewall: 80/443 public, 2375 runner-only
- [ ] Configure Docker daemon for remote TLS access (`-H tcp://0.0.0.0:2376` with certs)
- [ ] Create `/srv/linearvoid/data/{playlist,media,hls,chat,slate}` directories
- [ ] Add `CI_PROD_*` secrets to Gitea (DOCKER_HOST, R2 keys, admin password, DB password, Grafana password)
- [ ] Trigger CD workflow with `DOCKER_HOST` pointing to Hetzner IP
- [ ] Verify all 11 containers healthy: `/health`, `/hls/playlist.m3u8`, WebSocket
- [ ] Update Cloudflare A record `linearvoid.tv` → Hetzner IP, SSL Full (strict); test HTTPS + WSS
- [ ] Smoke test: viewer SPA loads, HLS plays, chat connects, admin login → upload → playlist → playout
- [ ] Set up Prometheus alerting notifications + UptimeRobot on production URL
- [ ] Run `scripts/backup-db.sh` + `scripts/verify-backups.sh` on production host
- [ ] Document production IP, SSH access, rollback procedure, scaling thresholds
### Phase 7: baseline scaling and performance
- [ ] Execute WebSocket load tests — ramp 10→100→500→1000 connections; measure p95 latency + errors
- [ ] Execute admin API benchmarks — `wrk`/`k6` against `/api/admin/*`; target p95 <1s
- [ ] Measure manifest TTFB and cache-hit distribution from multiple US regions (WebPageTest 3+ regions, <4s warm cache)
- [ ] Tune manifest TTL to 0.5× segment duration; validate `Cf-Cache-Status` distribution
- [ ] Track cache-hit ratio and R2 ops in production (Cloudflare GraphQL → Grafana panel, alert <90%)
- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min
- [ ] Plan Phase 2 service separation (playout CPU, chat latency, DB state, monitoring)
- [ ] Plan Phase 3 multi-origin resilience (second Hetzner location, warm standby, DNS failover, DB/Redis replication)
### Phase 8: close out remaining test and validation gaps
- [ ] Admin frontend: user moderation tools tests (mute/ban/report, view moderation logs)
- [ ] Frontend: coverage for all critical user + admin components and interactions
- [ ] Frontend: document test cases and expected outcomes
- [ ] Backend R2: test retrieval of segments/manifests via Cloudflare edge
- [ ] Backend R2: validate edge cache hit/miss behavior with repeat requests (`scripts/validate-edge-cache.sh`)
### Playout
- [ ] Configure an R2 lifecycle rule on the `hls/` prefix as defense-in-depth (manual)
- [ ] Full-loop E2E (2-file playlist, assert `Looping...` with no `CRASHED` across the boundary) — manual via `scripts/test-playout-e2e.sh`
## Cloudflare Configuration
### DNS & Workers
- [ ] Cloudflare Workers: link custom domain (manual)
- [ ] Create Cloudflare Worker and cut over frontend hosting (manual)
### Security & TLS
- [ ] Enable Tiered Cache, WAF, rate limiting (manual)
- [ ] Tiered Cache: enable Smart Tiered Cache, validate `Cf-Cache-Status: HIT` from upper tier
- [ ] WAF: enable OWASP Managed Ruleset, set to Simulate → review 48h → Block, enable Bot Fight Mode
- [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action
- [ ] HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2
- [ ] HTTPS/WSS: enforce at edge (Cloudflare) and origin (nginx), redirect HTTP → HTTPS
## Monitoring & Observability
### External Monitoring
- [ ] Set up UptimeRobot external monitoring
- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword: `#EXTM3U`, 5min)
- [ ] Alert on 2 consecutive failures, notify via email + Discord/Slack webhook
## External Services
> Evidence tracker: `docs/operations/cloudflare-uptimerobot-checklist.md`
- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword: `#EXTM3U`, 5min)
- [ ] Alert on 2 consecutive failures, notify via email + Discord/Slack webhook
## Production Deployment
### Host Provisioning (Hetzner — not provisioned)
- [ ] Provision Hetzner CX22 (2 vCPU, 4 GB, 40 GB) in Ashburn VA, Ubuntu 24.04
- [ ] Install Docker Engine + Compose plugin, open firewall: 80/443 (public), 2375 (runner IP only)
- [ ] Configure Docker daemon for remote TLS access (`-H tcp://0.0.0.0:2376` with certs)
- [ ] Create `/srv/linearvoid/data/{playlist,media,hls,chat,slate}` directories
### CI/CD & Secrets
- [ ] Add `CI_PROD_*` secrets to Gitea: DOCKER_HOST, R2 keys, admin password, DB password, Grafana password
- [ ] Trigger CD workflow with `DOCKER_HOST` pointing to Hetzner IP
### Smoke Test & Cutover
- [ ] Verify all 11 containers start healthy, check `/health`, `/hls/playlist.m3u8`, WebSocket
- [ ] Update Cloudflare A record `linearvoid.tv` → Hetzner IP, SSL Full (strict), test HTTPS + WSS
- [ ] Smoke test: viewer SPA loads, HLS plays, chat connects, admin login → upload → playlist → playout
### Production Operations
- [ ] Set up Prometheus alerting notifications + UptimeRobot on production URL
- [ ] Run `scripts/backup-db.sh` + `scripts/verify-backups.sh` on production host
- [ ] Document production IP, SSH access, rollback procedure, scaling thresholds
## Scaling & Performance
### Load Testing
- [ ] Execute WebSocket load tests with real infrastructure — ramp 10→100→500→1000 connections, measure p95 latency + errors
- [ ] Execute admin API benchmarks with real infrastructure — `wrk`/`k6` against `/api/admin/*`, target p95 <1s
### Cache & TTFB Optimization
- [ ] Optimize segment immutability and manifest TTL for cache efficiency
- [ ] Tune manifest TTL: measure `Cf-Cache-Status` distribution, adjust to 0.5× segment duration
- [ ] Measure playback TTFB from multiple US regions — WebPageTest from 3+ regions, target <4s warm cache
- [ ] Track cache hit ratio and R2 ops in production — Cloudflare GraphQL → Grafana panel, alert <90%
### Capacity & Future Planning
- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min
- [ ] Plan Phase 2 service separation — document boundaries (playout CPU, chat latency, DB state, monitoring), estimate per-service resources
- [ ] Plan Phase 3 multi-origin resilience — research second Hetzner location, design warm standby + DNS failover + DB/Redis replication
+57 -57
View File
@@ -1,57 +1,57 @@
# TODO
> Completed tasks moved to [DONE](./DONE.md). Remaining tasks in [BACKLOG](./BACKLOG.md).
## Chat sidebar
- [x] remove search bar from chat sidebar (not needed for single-room)
- [x] remove "user" toggle button from chat sidebar (hide for now, add back when user interface is stable)
- [x] make login/registration form a modal and only toggle it when user is not logged in and clicks "Login" or "Register"
- [x] when user is logged in, do not show login/registration form
- [x] make sure each message is only rendered once in the chat log (no duplicates on reconnect or initial load)
### Plan
Files: `frontend/index.html`, `frontend/src/main.ts`, `frontend/src/style.css`.
1. **Remove search bar** (single-room)
- `index.html`: delete the `<div id="dm-search-row">` block.
- `main.ts`: remove `searchDM()` and the `dm-search-btn` listener; drop the `#dm-search-row` show/hide logic in `switchRoom()` (the `searchRow.hidden = !isDM` lines). Keep `currentDMTarget` only if the "New DM" flow stays.
2. **Remove "Users" toggle button**
- `index.html`: delete the `<button id="users-toggle-btn">` button.
- `main.ts`: remove `toggleUserList()` and the `users-toggle-btn` listener; drop the now-unused `usersVisible` variable.
3. **Login/registration form as a modal**
- `index.html`: wrap the `#auth-panel` form in a modal structure (`.auth-modal` + backdrop) with a close button.
- `style.css`: add `.auth-modal` / `.auth-modal-backdrop` styles.
- `main.ts`: add `openAuthModal()` / `closeAuthModal()`; change the `auth-toggle-btn` handler to open the modal (only when not authenticated) instead of toggling `#auth-panel` inline; wire a close/backdrop-click to `closeAuthModal()`.
4. **Hide form when logged in**
- `main.ts` `updateAuthUI()`: when `authenticated` → hide the "Sign in" button and call `closeAuthModal()`; show the user label + "Sign out" button. When not authenticated → show "Sign in" and hide "Sign out".
5. **No duplicate messages**
- `main.ts`: add `clearMessages()` (empties `#messages`) and call it at the top of `connectWebSocket()` so each (re)connect renders history fresh instead of appending.
- Fix the initial double-connect: make the initial `connectWebSocket("main")` wait for `refreshUserSession()` (or connect once with the resolved username) so history isn't pushed twice on load.
## Frontend Tests
### User Frontend
- [ ] Test coverage for all critical frontend components and interactions
- [ ] Add code coverage reporting and enforce minimum coverage threshold in CI/CD pipeline
- [ ] Document test cases and expected outcomes
### Admin Frontend
- [ ] Test coverage for all critical admin UI components and interactions
- [ ] Add code coverage reporting and enforce minimum coverage threshold in CI/CD pipeline
- [ ] Document test cases and expected outcomes
## Backend Integration Tests
### Cloudflare R2
- [ ] Test retrieval of segments and manifests from R2 via Cloudflare edge
- [ ] Validate cache hit/miss behavior and performance metrics
- [ ] Validate edge cache hit/miss behavior with repeat requests — script `scripts/validate-edge-cache.sh` created
# TODO
> Completed tasks moved to [DONE](./DONE.md). Remaining tasks in [BACKLOG](./BACKLOG.md).
## Chat sidebar
- [x] remove search bar from chat sidebar (not needed for single-room)
- [x] remove "user" toggle button from chat sidebar (hide for now, add back when user interface is stable)
- [x] make login/registration form a modal and only toggle it when user is not logged in and clicks "Login" or "Register"
- [x] when user is logged in, do not show login/registration form
- [x] make sure each message is only rendered once in the chat log (no duplicates on reconnect or initial load)
### Plan
Files: `frontend/index.html`, `frontend/src/main.ts`, `frontend/src/style.css`.
1. **Remove search bar** (single-room)
- `index.html`: delete the `<div id="dm-search-row">` block.
- `main.ts`: remove `searchDM()` and the `dm-search-btn` listener; drop the `#dm-search-row` show/hide logic in `switchRoom()` (the `searchRow.hidden = !isDM` lines). Keep `currentDMTarget` only if the "New DM" flow stays.
2. **Remove "Users" toggle button**
- `index.html`: delete the `<button id="users-toggle-btn">` button.
- `main.ts`: remove `toggleUserList()` and the `users-toggle-btn` listener; drop the now-unused `usersVisible` variable.
3. **Login/registration form as a modal**
- `index.html`: wrap the `#auth-panel` form in a modal structure (`.auth-modal` + backdrop) with a close button.
- `style.css`: add `.auth-modal` / `.auth-modal-backdrop` styles.
- `main.ts`: add `openAuthModal()` / `closeAuthModal()`; change the `auth-toggle-btn` handler to open the modal (only when not authenticated) instead of toggling `#auth-panel` inline; wire a close/backdrop-click to `closeAuthModal()`.
4. **Hide form when logged in**
- `main.ts` `updateAuthUI()`: when `authenticated` → hide the "Sign in" button and call `closeAuthModal()`; show the user label + "Sign out" button. When not authenticated → show "Sign in" and hide "Sign out".
5. **No duplicate messages**
- `main.ts`: add `clearMessages()` (empties `#messages`) and call it at the top of `connectWebSocket()` so each (re)connect renders history fresh instead of appending.
- Fix the initial double-connect: make the initial `connectWebSocket("main")` wait for `refreshUserSession()` (or connect once with the resolved username) so history isn't pushed twice on load.
## Frontend Tests
### User Frontend
- [ ] Test coverage for all critical frontend components and interactions
- [ ] Add code coverage reporting and enforce minimum coverage threshold in CI/CD pipeline
- [ ] Document test cases and expected outcomes
### Admin Frontend
- [ ] Test coverage for all critical admin UI components and interactions
- [ ] Add code coverage reporting and enforce minimum coverage threshold in CI/CD pipeline
- [ ] Document test cases and expected outcomes
## Backend Integration Tests
### Cloudflare R2
- [ ] Test retrieval of segments and manifests from R2 via Cloudflare edge
- [ ] Validate cache hit/miss behavior and performance metrics
- [ ] Validate edge cache hit/miss behavior with repeat requests — script `scripts/validate-edge-cache.sh` created