zwitschi/linearvoid

0 Fork 0

Files

T

zwitschi 33b6296c5e

CI / test (push) Successful in 2m50s

Details

CI / docker (push) Successful in 5m12s

Details

Refactor load testing scripts to support Bash and PowerShell

- Updated README.md to reflect changes in load testing commands and added usage examples for Bash scripts.
- Converted PowerShell load testing scripts to Bash equivalents for better cross-platform compatibility.
- Added new Bash scripts: run-loadtests.sh, summarize-artillery.sh, and summarize-k6.sh to mirror existing PowerShell functionality.
- Updated admin-api.js and ws-chat.yml for consistency and improved performance metrics reporting.
- Enhanced error handling and output formatting in summary scripts for better readability.

2026-09-02 19:47:34 +02:00

11 KiB

Raw Permalink Blame History

Scaling and Performance Playbook

Purpose: define a repeatable method to measure performance, capture baselines, and decide scaling actions for the "Linear Void" streaming app. Scope: planning/execution for the "Scaling & Performance (planning only)" TODO items.

1) Manifest TTL Tuning Plan

Goal: tune the .m3u8 TTL to ~0.5x segment duration while keeping playback stable.

Inputs: segment duration (HLS_SEGMENT_DURATION), current manifest TTL (Cloudflare cache rule), cache outcome distribution (Cf-Cache-Status).

Procedure:

Start from current manifest TTL (2s).
Capture 24h baseline headers from repeated playlist.m3u8 requests.
Test candidate TTLs: 1s, 2s, 3s.
Compare rebuffer events (player logs), origin request increase/decrease, and HIT ratio changes.
Select the lowest TTL that keeps playback stable and origin load acceptable.

Acceptance: no material increase in rebuffer rate; manifest staleness not observable by users.

2) TTFB Baseline Plan

Goal: measure warm-cache TTFB from 3+ US regions with target < 4s.

Regions: US East (Virginia), US Central, US West (California/Oregon).

Method:

Use WebPageTest with repeat view enabled.
Run at least 5 samples per region.
Record median and p95 TTFB.
Store results in a table (date, region, median, p95, test URL).

Acceptance: warm-cache median < 4s in all regions; outliers documented with likely cause.

TTFB Baseline Worksheet

Use one row per region; keep the raw WebPageTest result URL with the summary for comparison.

US East (Virginia) / US Central / US West (California/Oregon): Date/time, Test URL https://linearvoid.tv, WebPageTest result URL, Median TTFB, p95 TTFB, Notes.

Capture notes: use repeat view for warm-cache; keep the same browser profile and test URL across regions; note any CDN cache-rule changes or deploys between samples.

3) WebSocket Load-Test Plan

Goal: characterize chat behavior across 10 → 100 → 500 → 1000 concurrent clients.

Tooling option A (Artillery):

npx artillery run scripts/loadtest/ws-chat.yml

Tooling option B (custom Go harness): open N concurrent WS clients, send a message every X seconds, measure round-trip latency and failures.

Metrics: p50/p95 message latency; error rate (connect/send/receive); dropped connections; CPU/memory for chat, redis, nginx.

Acceptance: p95 chat latency ≤ 1s at target concurrency tier; error rate ≤ 1% during steady state.

WebSocket Load-Test Worksheet

Date/time, Environment, Commit SHA/image tag, Test URL, Tool used, Scenario duration, Concurrency tiers reached, p50 message latency, p95 message latency, Error rate, Dropped connections, CPU/memory notes for chat/redis/nginx, Raw results location, Conclusion, Next action.

4) Admin API Benchmark Plan

Goal: measure admin API p95 and identify slow endpoints.

Endpoints: /api/admin/overview, /api/admin/analytics, /api/admin/chat-log, /api/admin/users.

Tooling option A (k6):

k6 run scripts/loadtest/admin-api.js

Tooling option B (wrk):

wrk -t4 -c100 -d60s http://<host>/api/admin/overview

Acceptance: p95 < 1s for primary admin endpoints; slow endpoint list with remediation notes.

Admin API Benchmark Worksheet

Date/time, Environment, Commit SHA/image tag, Test URL, Tool used, Scenario duration, Concurrency/virtual users, /api/admin/overview|analytics|chat-log|users p95, Error rate, Slowest endpoint, Cache or DB notes, Raw results location, Conclusion, Next action.

5) Cache Hit Ratio Tracking Plan

Goal: track Cloudflare cache hit ratio and alert when below threshold.

Implementation:

Pull Cloudflare GraphQL analytics on schedule.
Export to a Prometheus-compatible metric path.
Add a Grafana panel with 5m/1h/24h windows.
Alert when hit ratio < 90% for 15m.

Reference exporter:

scripts/cache-hit-ratio.js
scripts/cache-hit-ratio.sh for scheduled tasks / manual runs
Example run: export CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_TAG=... CLOUDFLARE_HOSTNAME=media.linearvoid.tv; node scripts/cache-hit-ratio.js
Intended output: Prometheus text format on stdout, suitable for cron, a Node wrapper, or a small scrape endpoint.

Metric target: cache_hit_ratio >= 0.90.

Cache Hit Ratio Worksheet

Date/time, Environment, Commit SHA/image tag, Hostname, Window minutes, Cloudflare zone tag, Export command, Raw exporter output location, Hit requests, Total requests, Cache hit ratio, Threshold met, Notes.

6) Capacity Trigger Rules

Operational triggers:

Cache miss ratio > 10% for 15m
Chat p95 latency > 1s for 5m
Host CPU > 80% for 10m
Redis memory usage ratio (linearvoid_chat_redis_memory_usage_ratio) > 0.85 for 10m

Action mapping:

Level 1: investigate + restart non-critical components
Level 2: scale service resources (CPU/RAM)
Level 3: activate standby planning path

Capacity Trigger Review

Date/time, Environment, Commit SHA/image tag, Trigger hit, Threshold value, Observed value, Duration above threshold, Level selected, Immediate action taken, Services involved, Follow-up owner, Raw metrics/alert link, Notes.

7) Phase 2 Service Separation Plan

Objective: separate bottlenecked services while keeping ops simple.

Boundaries:

playout isolated first (CPU-heavy)
chat + redis grouped for latency
db isolated for IO stability
monitoring stack optionally separated after sustained load

Initial sizing:

playout host: 4 vCPU / 8 GB
chat host: 2–4 vCPU / 4–8 GB
db host: 2 vCPU / 4 GB + fast disk

Exit criteria: repeated trigger events over 7 days; capacity saturation despite tuning.

Phase 2 Boundary Worksheet

Date/time, Environment, Commit SHA/image tag, Trigger history summary, Repeated trigger count over 7 days, Primary bottleneck, Proposed split, playout/chat/db target host size, Monitoring separation needed, Data/state migration notes, Implementation owner, Review decision, Next action.

8) Phase 3 Warm Standby Plan

Objective: second-region resilience with DNS failover.

Layout:

Primary: Hetzner Ashburn (US-East)
Standby: Hetzner Hillsboro (US-West)

Design: warm standby stack deployed and health-checked; Redis replication or fallback mode documented; Postgres replication strategy selected (streaming replica or periodic snapshot restore); Cloudflare DNS failover policy defined (health-based or manual emergency switch).

Phase 3 Warm Standby Worksheet

Date/time, Environment, Commit SHA/image tag, Primary region, Standby region, DNS failover mode, Redis strategy, Postgres strategy, Health check endpoints, Simulated cutover performed, Measured RTO, Measured RPO, Open risks, Implementation owner, Review decision, Notes.

9) Data Capture Template

For each benchmark run capture: Date/time, Environment (test-lab/staging/production), Commit SHA/image tag, Test scenario and duration, Raw metrics (latency, errors, resource usage), Conclusion and next action.

10) Execution Order

TTFB baseline
WebSocket load profile
Admin API benchmark
Cache ratio panel + alerting
Trigger tuning
Phase 2 boundary decision
Phase 3 standby design review

11) Benchmark Run Sheet

Use the same worksheet for every baseline/re-test so results stay comparable.

Preconditions: confirm target environment and commit SHA/image tag; confirm Cloudflare cache rules are active when testing TTFB or manifest TTL; confirm chat, admin, and playout services are healthy before load testing.

Run template:

Item	Value
Date/time
Environment
Commit SHA / image tag
Test scenario
Concurrency / sample count
Command or tool
Raw results location
Conclusion
Next action

Suggested command mapping:

TTFB baseline: WebPageTest repeat-view run against https://linearvoid.tv.
WebSocket profile: npx artillery run scripts/loadtest/ws-chat.yml or the Go harness referenced above.
Admin API benchmark: k6 run scripts/loadtest/admin-api.js or wrk -t4 -c100 -d60s http://<host>/api/admin/overview.

12) Scaling Decision Gate

Default review gate after any benchmark batch:

If warm-cache TTFB meets target but cache hit ratio falls below 90%, tune cache behavior before increasing capacity.
If chat or admin latency exceeds target while CPU and memory remain low, prioritize query or handler optimization.
If CPU, memory, or Redis saturation appears first, treat the Phase 2 service boundary guidance as the next action.
If the same threshold trips on multiple runs across 7 days, open the Phase 2 or Phase 3 planning path instead of tuning in place.

Decision note: keep the smallest change that restores the target metric and record the reason in the run sheet.

13) Artifact-to-Worksheet Field Mapping

Use when filling the WebSocket and Admin API worksheets from helper-generated artifacts.

WebSocket worksheet mapping

Artifact sources: artifacts/<timestamp>/ws-chat-report.json, artifacts/<timestamp>/ws-chat-summary.txt.

Field mapping:

Tool used → artillery (or scripts/loadtest/run-loadtests.sh -s ws)
Raw results location → artifact folder path for this run
Concurrency tiers reached → ws-chat.yml phases (warmup-10, sustain-100, sustain-500, sustain-1000)
Error rate → vusers.failure_rate_percent from ws-chat-summary.txt
Dropped connections → derive from vusers.created - vusers.completed (use vusers.failed as failure signal)
p95 message latency → first latency line under summary.* in ws-chat-summary.txt containing p95=...
p50 message latency → first latency line under summary.* in ws-chat-summary.txt containing median=...

Notes: the exact summary.* metric names in Artillery output can vary by engine/plugins; if no latency summary field is emitted, record this explicitly and use server-side latency metrics as fallback evidence.

Admin API worksheet mapping

Artifact sources: artifacts/<timestamp>/admin-api-summary.json, artifacts/<timestamp>/admin-api-output.txt, artifacts/<timestamp>/admin-api-endpoint-summary.txt.