zwitschi
33b6296c5e
- Updated README.md to reflect changes in load testing commands and added usage examples for Bash scripts. - Converted PowerShell load testing scripts to Bash equivalents for better cross-platform compatibility. - Added new Bash scripts: run-loadtests.sh, summarize-artillery.sh, and summarize-k6.sh to mirror existing PowerShell functionality. - Updated admin-api.js and ws-chat.yml for consistency and improved performance metrics reporting. - Enhanced error handling and output formatting in summary scripts for better readability.
225 lines
11 KiB
Markdown
225 lines
11 KiB
Markdown
# Scaling and Performance Playbook
|
||
|
||
Purpose: define a repeatable method to measure performance, capture baselines, and decide scaling actions for the "Linear Void" streaming app. Scope: planning/execution for the "Scaling & Performance (planning only)" TODO items.
|
||
|
||
## 1) Manifest TTL Tuning Plan
|
||
|
||
Goal: tune the `.m3u8` TTL to ~`0.5x` segment duration while keeping playback stable.
|
||
|
||
Inputs: segment duration (`HLS_SEGMENT_DURATION`), current manifest TTL (Cloudflare cache rule), cache outcome distribution (`Cf-Cache-Status`).
|
||
|
||
Procedure:
|
||
1. Start from current manifest TTL (2s).
|
||
2. Capture 24h baseline headers from repeated `playlist.m3u8` requests.
|
||
3. Test candidate TTLs: 1s, 2s, 3s.
|
||
4. Compare rebuffer events (player logs), origin request increase/decrease, and HIT ratio changes.
|
||
5. Select the lowest TTL that keeps playback stable and origin load acceptable.
|
||
|
||
Acceptance: no material increase in rebuffer rate; manifest staleness not observable by users.
|
||
|
||
## 2) TTFB Baseline Plan
|
||
|
||
Goal: measure warm-cache TTFB from 3+ US regions with target `< 4s`.
|
||
|
||
Regions: US East (Virginia), US Central, US West (California/Oregon).
|
||
|
||
Method:
|
||
1. Use WebPageTest with repeat view enabled.
|
||
2. Run at least 5 samples per region.
|
||
3. Record median and p95 TTFB.
|
||
4. Store results in a table (date, region, median, p95, test URL).
|
||
|
||
Acceptance: warm-cache median < 4s in all regions; outliers documented with likely cause.
|
||
|
||
### TTFB Baseline Worksheet
|
||
Use one row per region; keep the raw WebPageTest result URL with the summary for comparison.
|
||
|
||
- US East (Virginia) / US Central / US West (California/Oregon): Date/time, Test URL `https://linearvoid.tv`, WebPageTest result URL, Median TTFB, p95 TTFB, Notes.
|
||
|
||
Capture notes: use repeat view for warm-cache; keep the same browser profile and test URL across regions; note any CDN cache-rule changes or deploys between samples.
|
||
|
||
## 3) WebSocket Load-Test Plan
|
||
|
||
Goal: characterize chat behavior across 10 → 100 → 500 → 1000 concurrent clients.
|
||
|
||
Tooling option A (Artillery):
|
||
```bash
|
||
npx artillery run scripts/loadtest/ws-chat.yml
|
||
```
|
||
|
||
Tooling option B (custom Go harness): open N concurrent WS clients, send a message every X seconds, measure round-trip latency and failures.
|
||
|
||
Metrics: p50/p95 message latency; error rate (connect/send/receive); dropped connections; CPU/memory for `chat`, `redis`, `nginx`.
|
||
|
||
Acceptance: p95 chat latency ≤ 1s at target concurrency tier; error rate ≤ 1% during steady state.
|
||
|
||
### WebSocket Load-Test Worksheet
|
||
Date/time, Environment, Commit SHA/image tag, Test URL, Tool used, Scenario duration, Concurrency tiers reached, p50 message latency, p95 message latency, Error rate, Dropped connections, CPU/memory notes for chat/redis/nginx, Raw results location, Conclusion, Next action.
|
||
|
||
## 4) Admin API Benchmark Plan
|
||
|
||
Goal: measure admin API p95 and identify slow endpoints.
|
||
|
||
Endpoints: `/api/admin/overview`, `/api/admin/analytics`, `/api/admin/chat-log`, `/api/admin/users`.
|
||
|
||
Tooling option A (`k6`):
|
||
```bash
|
||
k6 run scripts/loadtest/admin-api.js
|
||
```
|
||
|
||
Tooling option B (`wrk`):
|
||
```bash
|
||
wrk -t4 -c100 -d60s http://<host>/api/admin/overview
|
||
```
|
||
|
||
Acceptance: p95 < 1s for primary admin endpoints; slow endpoint list with remediation notes.
|
||
|
||
### Admin API Benchmark Worksheet
|
||
Date/time, Environment, Commit SHA/image tag, Test URL, Tool used, Scenario duration, Concurrency/virtual users, `/api/admin/overview|analytics|chat-log|users` p95, Error rate, Slowest endpoint, Cache or DB notes, Raw results location, Conclusion, Next action.
|
||
|
||
## 5) Cache Hit Ratio Tracking Plan
|
||
|
||
Goal: track Cloudflare cache hit ratio and alert when below threshold.
|
||
|
||
Implementation:
|
||
1. Pull Cloudflare GraphQL analytics on schedule.
|
||
2. Export to a Prometheus-compatible metric path.
|
||
3. Add a Grafana panel with 5m/1h/24h windows.
|
||
4. Alert when hit ratio < 90% for 15m.
|
||
|
||
Reference exporter:
|
||
- `scripts/cache-hit-ratio.js`
|
||
- `scripts/cache-hit-ratio.sh` for scheduled tasks / manual runs
|
||
- Example run: `export CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_TAG=... CLOUDFLARE_HOSTNAME=media.linearvoid.tv; node scripts/cache-hit-ratio.js`
|
||
- Intended output: Prometheus text format on stdout, suitable for cron, a Node wrapper, or a small scrape endpoint.
|
||
|
||
Metric target: `cache_hit_ratio >= 0.90`.
|
||
|
||
### Cache Hit Ratio Worksheet
|
||
Date/time, Environment, Commit SHA/image tag, Hostname, Window minutes, Cloudflare zone tag, Export command, Raw exporter output location, Hit requests, Total requests, Cache hit ratio, Threshold met, Notes.
|
||
|
||
## 6) Capacity Trigger Rules
|
||
|
||
Operational triggers:
|
||
- Cache miss ratio > 10% for 15m
|
||
- Chat p95 latency > 1s for 5m
|
||
- Host CPU > 80% for 10m
|
||
- Redis memory usage ratio (`linearvoid_chat_redis_memory_usage_ratio`) > 0.85 for 10m
|
||
|
||
Action mapping:
|
||
- Level 1: investigate + restart non-critical components
|
||
- Level 2: scale service resources (CPU/RAM)
|
||
- Level 3: activate standby planning path
|
||
|
||
### Capacity Trigger Review
|
||
Date/time, Environment, Commit SHA/image tag, Trigger hit, Threshold value, Observed value, Duration above threshold, Level selected, Immediate action taken, Services involved, Follow-up owner, Raw metrics/alert link, Notes.
|
||
|
||
## 7) Phase 2 Service Separation Plan
|
||
|
||
Objective: separate bottlenecked services while keeping ops simple.
|
||
|
||
Boundaries:
|
||
- `playout` isolated first (CPU-heavy)
|
||
- `chat` + `redis` grouped for latency
|
||
- `db` isolated for IO stability
|
||
- monitoring stack optionally separated after sustained load
|
||
|
||
Initial sizing:
|
||
- playout host: 4 vCPU / 8 GB
|
||
- chat host: 2–4 vCPU / 4–8 GB
|
||
- db host: 2 vCPU / 4 GB + fast disk
|
||
|
||
Exit criteria: repeated trigger events over 7 days; capacity saturation despite tuning.
|
||
|
||
### Phase 2 Boundary Worksheet
|
||
Date/time, Environment, Commit SHA/image tag, Trigger history summary, Repeated trigger count over 7 days, Primary bottleneck, Proposed split, `playout`/`chat`/`db` target host size, Monitoring separation needed, Data/state migration notes, Implementation owner, Review decision, Next action.
|
||
|
||
## 8) Phase 3 Warm Standby Plan
|
||
|
||
Objective: second-region resilience with DNS failover.
|
||
|
||
Layout:
|
||
- Primary: Hetzner Ashburn (US-East)
|
||
- Standby: Hetzner Hillsboro (US-West)
|
||
|
||
Design: warm standby stack deployed and health-checked; Redis replication or fallback mode documented; Postgres replication strategy selected (streaming replica or periodic snapshot restore); Cloudflare DNS failover policy defined (health-based or manual emergency switch).
|
||
|
||
### Phase 3 Warm Standby Worksheet
|
||
Date/time, Environment, Commit SHA/image tag, Primary region, Standby region, DNS failover mode, Redis strategy, Postgres strategy, Health check endpoints, Simulated cutover performed, Measured RTO, Measured RPO, Open risks, Implementation owner, Review decision, Notes.
|
||
|
||
## 9) Data Capture Template
|
||
For each benchmark run capture: Date/time, Environment (test-lab/staging/production), Commit SHA/image tag, Test scenario and duration, Raw metrics (latency, errors, resource usage), Conclusion and next action.
|
||
|
||
## 10) Execution Order
|
||
1. TTFB baseline
|
||
2. WebSocket load profile
|
||
3. Admin API benchmark
|
||
4. Cache ratio panel + alerting
|
||
5. Trigger tuning
|
||
6. Phase 2 boundary decision
|
||
7. Phase 3 standby design review
|
||
|
||
## 11) Benchmark Run Sheet
|
||
Use the same worksheet for every baseline/re-test so results stay comparable.
|
||
|
||
Preconditions: confirm target environment and commit SHA/image tag; confirm Cloudflare cache rules are active when testing TTFB or manifest TTL; confirm chat, admin, and playout services are healthy before load testing.
|
||
|
||
Run template:
|
||
|
||
| Item | Value |
|
||
| -------------------------- | ----- |
|
||
| Date/time | |
|
||
| Environment | |
|
||
| Commit SHA / image tag | |
|
||
| Test scenario | |
|
||
| Concurrency / sample count | |
|
||
| Command or tool | |
|
||
| Raw results location | |
|
||
| Conclusion | |
|
||
| Next action | |
|
||
|
||
Suggested command mapping:
|
||
- TTFB baseline: WebPageTest repeat-view run against `https://linearvoid.tv`.
|
||
- WebSocket profile: `npx artillery run scripts/loadtest/ws-chat.yml` or the Go harness referenced above.
|
||
- Admin API benchmark: `k6 run scripts/loadtest/admin-api.js` or `wrk -t4 -c100 -d60s http://<host>/api/admin/overview`.
|
||
|
||
## 12) Scaling Decision Gate
|
||
|
||
Default review gate after any benchmark batch:
|
||
1. If warm-cache TTFB meets target but cache hit ratio falls below 90%, tune cache behavior before increasing capacity.
|
||
2. If chat or admin latency exceeds target while CPU and memory remain low, prioritize query or handler optimization.
|
||
3. If CPU, memory, or Redis saturation appears first, treat the Phase 2 service boundary guidance as the next action.
|
||
4. If the same threshold trips on multiple runs across 7 days, open the Phase 2 or Phase 3 planning path instead of tuning in place.
|
||
|
||
Decision note: keep the smallest change that restores the target metric and record the reason in the run sheet.
|
||
|
||
## 13) Artifact-to-Worksheet Field Mapping
|
||
Use when filling the WebSocket and Admin API worksheets from helper-generated artifacts.
|
||
|
||
### WebSocket worksheet mapping
|
||
Artifact sources: `artifacts/<timestamp>/ws-chat-report.json`, `artifacts/<timestamp>/ws-chat-summary.txt`.
|
||
|
||
Field mapping:
|
||
- `Tool used` → `artillery` (or `scripts/loadtest/run-loadtests.sh -s ws`)
|
||
- `Raw results location` → artifact folder path for this run
|
||
- `Concurrency tiers reached` → `ws-chat.yml` phases (`warmup-10`, `sustain-100`, `sustain-500`, `sustain-1000`)
|
||
- `Error rate` → `vusers.failure_rate_percent` from `ws-chat-summary.txt`
|
||
- `Dropped connections` → derive from `vusers.created - vusers.completed` (use `vusers.failed` as failure signal)
|
||
- `p95 message latency` → first latency line under `summary.*` in `ws-chat-summary.txt` containing `p95=...`
|
||
- `p50 message latency` → first latency line under `summary.*` in `ws-chat-summary.txt` containing `median=...`
|
||
|
||
Notes: the exact `summary.*` metric names in Artillery output can vary by engine/plugins; if no latency summary field is emitted, record this explicitly and use server-side latency metrics as fallback evidence.
|
||
|
||
### Admin API worksheet mapping
|
||
Artifact sources: `artifacts/<timestamp>/admin-api-summary.json`, `artifacts/<timestamp>/admin-api-output.txt`, `artifacts/<timestamp>/admin-api-endpoint-summary.txt`.
|
||
|
||
Field mapping:
|
||
- `Tool used` → `k6` (or `scripts/loadtest/run-loadtests.sh -s admin`)
|
||
- `Raw results location` → artifact folder path for this run
|
||
- `/api/admin/overview|analytics|chat-log|users` p95 → `overview|analytics|chat_log|users` row `P95Ms` in `admin-api-endpoint-summary.txt`
|
||
- `Error rate` → `Global failure rate` in `admin-api-endpoint-summary.txt`
|
||
- `Slowest endpoint` → endpoint row with highest `P95Ms`
|
||
|
||
Notes: keep the raw `admin-api-summary.json` file for audits and re-parsing; if an endpoint row is missing, treat it as a test/data issue and re-run with full endpoint coverage.
|
||
|
||
Summary helper (bash): run `scripts/loadtest/summarize-artillery.sh <report.json>` after an Artillery WS run, and `scripts/loadtest/summarize-k6.sh <summary.json>` after a k6 admin run, to produce the artifact files used in the mappings above. |