linearvoid/docs/operations/scaling-performance-playbook.md

Scaling and Performance Playbook

Purpose: define repeatable method to measure performance, capture baseline data, and decide scaling actions.

Scope: planning and execution guide for TODO items under "Scaling & Performance (planning only)".

1) Manifest TTL Tuning Plan

Goal: tune .m3u8 TTL to about 0.5x segment duration while keeping playback stable.

Inputs:

• Segment duration (HLS_SEGMENT_DURATION)
• Current manifest TTL (Cloudflare cache rule)
• Cache outcome distribution (Cf-Cache-Status)

Procedure:

1. Start from current manifest TTL (2s).
2. Capture 24h baseline headers from repeated playlist.m3u8 requests.
3. Test candidate TTLs: 1s, 2s, 3s.
4. Compare:
   • rebuffer events from player logs
   • origin request increase/decrease
   • HIT ratio changes
5. Select lowest TTL that keeps playback stable and origin load acceptable.

Acceptance:

• No material increase in rebuffer rate.
• Manifest staleness not observable by users.

2) TTFB Baseline Plan

Goal: measure warm-cache TTFB from 3+ US regions with target < 4s.

Suggested regions:

• US East (Virginia)
• US Central
• US West (California/Oregon)

Method:

1. Use WebPageTest with repeat view enabled.
2. Run at least 5 samples per region.
3. Record median and p95 TTFB.
4. Store results in table (date, region, median, p95, test URL).

Acceptance:

• Warm-cache median < 4s in all tested regions.
• Outliers documented with likely cause.

TTFB Baseline Worksheet

Use one row per region and keep the raw WebPageTest result URL with the summary so later runs can be compared against the same evidence.

• US East (Virginia)
  • Date/time:
  • Test URL: https://linearvoid.tv
  • WebPageTest result URL:
  • Median TTFB:
  • p95 TTFB:
  • Notes:
• US Central
  • Date/time:
  • Test URL: https://linearvoid.tv
  • WebPageTest result URL:
  • Median TTFB:
  • p95 TTFB:
  • Notes:
• US West (California/Oregon)
  • Date/time:
  • Test URL: https://linearvoid.tv
  • WebPageTest result URL:
  • Median TTFB:
  • p95 TTFB:
  • Notes:

Suggested capture notes:

• Use repeat view for the warm-cache measurement.
• Keep the same browser profile and test URL across regions.
• Note any CDN cache-rule changes or deploys that occurred between samples.

3) WebSocket Load-Test Plan

Goal: characterize chat service behavior across 10 -> 100 -> 500 -> 1000 concurrent clients.

Tooling option A (Artillery):

npx artillery run scripts/loadtest/ws-chat.yml

Tooling option B (custom Go harness):

• open N concurrent WebSocket clients
• send message every X seconds
• measure round-trip latency and failures

Metrics to capture:

• p50/p95 message latency
• error rate (connect, send, receive)
• dropped connections
• CPU/memory for chat, redis, nginx

Acceptance:

• p95 chat latency <= 1s at target concurrency tier.
• error rate <= 1% during steady state.

WebSocket Load-Test Worksheet

• Date/time:
• Environment:
• Commit SHA / image tag:
• Test URL:
• Tool used:
• Scenario duration:
• Concurrency tiers reached:
• p50 message latency:
• p95 message latency:
• Error rate:
• Dropped connections:
• CPU/memory notes for chat:
• CPU/memory notes for redis:
• CPU/memory notes for nginx:
• Raw results location:
• Conclusion:
• Next action:

4) Admin API Benchmark Plan

Goal: measure admin API p95 and identify slow endpoints.

Tooling option A (k6):

k6 run scripts/loadtest/admin-api.js

Tooling option B (wrk):

wrk -t4 -c100 -d60s http://<host>/api/admin/overview

Endpoints:

• /api/admin/overview
• /api/admin/analytics
• /api/admin/chat-log
• /api/admin/users

Acceptance:

• p95 < 1s for primary admin endpoints.
• slow endpoint list with remediation notes.

Admin API Benchmark Worksheet

• Date/time:
• Environment:
• Commit SHA / image tag:
• Test URL:
• Tool used:
• Scenario duration:
• Concurrency / virtual users:
• /api/admin/overview p95:
• /api/admin/analytics p95:
• /api/admin/chat-log p95:
• /api/admin/users p95:
• Error rate:
• Slowest endpoint:
• Cache or DB notes:
• Raw results location:
• Conclusion:
• Next action:

5) Cache Hit Ratio Tracking Plan

Goal: track Cloudflare cache hit ratio and alert when below threshold.

Implementation outline:

1. Pull Cloudflare GraphQL analytics on schedule.
2. Export to Prometheus-compatible metric path.
3. Add Grafana panel with 5m/1h/24h windows.
4. Alert when hit ratio < 90% for 15m.

Reference exporter:

• scripts/cache-hit-ratio.js
• scripts/cache-hit-ratio.ps1 for Windows scheduled tasks / manual runs
• Example run: CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_TAG=... CLOUDFLARE_HOSTNAME=media.linearvoid.tv node scripts/cache-hit-ratio.js
• Windows example: pwsh -File scripts/cache-hit-ratio.ps1
• Intended output: Prometheus text format on stdout, suitable for cron, a Node wrapper, or a small scrape endpoint.

Metric target:

• cache_hit_ratio >= 0.90

Cache Hit Ratio Worksheet

• Date/time:
• Environment:
• Commit SHA / image tag:
• Hostname:
• Window minutes:
• Cloudflare zone tag:
• Export command:
• Raw exporter output location:
• Hit requests:
• Total requests:
• Cache hit ratio:
• Threshold met:
• Notes:

6) Capacity Trigger Rules

Define operational triggers:

• Cache miss ratio > 10% for 15m
• Chat p95 latency > 1s for 5m
• Host CPU > 80% for 10m
• Redis memory usage ratio (linearvoid_chat_redis_memory_usage_ratio) > 0.85 for 10m

Action mapping:

• Level 1: investigate + restart non-critical components
• Level 2: scale service resources (CPU/RAM)
• Level 3: activate standby planning path

Capacity Trigger Review

• Date/time:
• Environment:
• Commit SHA / image tag:
• Trigger hit:
• Threshold value:
• Observed value:
• Duration above threshold:
• Level selected:
• Immediate action taken:
• Services involved:
• Follow-up owner:
• Raw metrics / alert link:
• Notes:

7) Phase 2 Service Separation Plan

Objective: separate bottlenecked services while keeping simple ops.

Boundaries:

• playout isolated first (CPU-heavy)
• chat + redis grouped for latency
• db isolated for IO stability
• monitoring stack optionally separated after sustained load

Initial sizing guidance:

• playout host: 4 vCPU / 8 GB
• chat host: 2-4 vCPU / 4-8 GB
• db host: 2 vCPU / 4 GB + fast disk

Exit criteria for Phase 2:

• repeated trigger events over 7 days
• capacity saturation despite tuning

Phase 2 Boundary Worksheet

• Date/time:
• Environment:
• Commit SHA / image tag:
• Trigger history summary:
• Repeated trigger count over 7 days:
• Primary bottleneck:
• Proposed split:
• playout target host size:
• chat target host size:
• db target host size:
• Monitoring separation needed:
• Data/state migration notes:
• Implementation owner:
• Review decision:
• Next action:

8) Phase 3 Warm Standby Plan

Objective: second-region resilience with DNS failover.

Candidate layout:

• Primary: Hetzner Ashburn (US-East)
• Standby: Hetzner Hillsboro (US-West)

Design:

• Warm standby stack deployed and health-checked
• Redis replication or fallback mode documented
• Postgres replication strategy selected (streaming replica or periodic snapshot restore)
• Cloudflare DNS failover policy defined (health-based or manual emergency switch)

Failover drill requirements:

Phase 3 Warm Standby Worksheet

• Date/time:
• Environment:
• Commit SHA / image tag:
• Primary region:
• Standby region:
• DNS failover mode:
• Redis strategy:
• Postgres strategy:
• Health check endpoints:
• Simulated cutover performed:
• Measured RTO:
• Measured RPO:
• Open risks:
• Implementation owner:
• Review decision:
• Notes:

9) Data Capture Template

For each benchmark run, capture:

• Date/time
• Environment (test-lab/staging/production)
• Commit SHA / image tag
• Test scenario and duration
• Raw metrics (latency, errors, resource usage)
• Conclusion and next action

10) Execution Order

1. TTFB baseline
2. WebSocket load profile
3. Admin API benchmark
4. Cache ratio panel + alerting
5. Trigger tuning
6. Phase 2 boundary decision
7. Phase 3 standby design review

11) Benchmark Run Sheet

Use the same worksheet for every baseline or re-test so results stay comparable.

Preconditions:

• Confirm the target environment and commit SHA/image tag.
• Confirm Cloudflare cache rules are active when testing TTFB or manifest TTL.
• Confirm the chat, admin, and playout services are healthy before load testing.

Run template:

Item	Value
Date/time
Environment
Commit SHA / image tag
Test scenario
Concurrency / sample count
Command or tool
Raw results location
Conclusion
Next action

Suggested command mapping:

• TTFB baseline: WebPageTest repeat-view run against https://linearvoid.tv.
• WebSocket profile: npx artillery run scripts/loadtest/ws-chat.yml or the Go harness referenced above.
• Admin API benchmark: k6 run scripts/loadtest/admin-api.js or wrk -t4 -c100 -d60s http://<host>/api/admin/overview.

12) Scaling Decision Gate

Treat the following as the default review gate after any benchmark batch:

1. If warm-cache TTFB meets target but cache hit ratio falls below 90%, tune cache behavior before increasing capacity.
2. If chat or admin latency exceeds target while CPU and memory remain low, prioritize query or handler optimization.
3. If CPU, memory, or Redis saturation appears first, treat the service boundary guidance in Phase 2 as the next action.
4. If the same threshold trips on multiple runs across 7 days, open the Phase 2 or Phase 3 planning path instead of tuning in place.

Decision note:

• Keep the smallest change that restores the target metric and record the reason in the run sheet.

13) Artifact-to-Worksheet Field Mapping

Use this section when filling the WebSocket and Admin API worksheets from helper-generated artifacts.

WebSocket worksheet mapping

Artifact sources:

• artifacts/<timestamp>/ws-chat-report.json
• artifacts/<timestamp>/ws-chat-summary.txt

Field mapping:

• Worksheet Tool used -> artillery (or run-loadtests.ps1 -Scenario ws)
• Worksheet Raw results location -> artifact folder path for this run
• Worksheet Concurrency tiers reached -> ws-chat.yml phases (warmup-10, sustain-100, sustain-500, sustain-1000)
• Worksheet Error rate -> vusers.failure_rate_percent from ws-chat-summary.txt
• Worksheet Dropped connections -> derive from vusers.created - vusers.completed (use vusers.failed as failure signal)
• Worksheet p95 message latency -> first available latency line under summary.* in ws-chat-summary.txt that includes p95=...
• Worksheet p50 message latency -> first available latency line under summary.* in ws-chat-summary.txt that includes median=...

Notes:

• The exact summary.* metric names in Artillery output can vary by engine/plugins.
• If no latency summary field is emitted, record this explicitly and use server-side latency metrics as fallback evidence.

Admin API worksheet mapping

Artifact sources:

• artifacts/<timestamp>/admin-api-summary.json
• artifacts/<timestamp>/admin-api-output.txt
• artifacts/<timestamp>/admin-api-endpoint-summary.txt

Field mapping:

• Worksheet Tool used -> k6 (or run-loadtests.ps1 -Scenario admin)
• Worksheet Raw results location -> artifact folder path for this run
• Worksheet /api/admin/overview p95 -> overview row P95Ms in admin-api-endpoint-summary.txt
• Worksheet /api/admin/analytics p95 -> analytics row P95Ms in admin-api-endpoint-summary.txt
• Worksheet /api/admin/chat-log p95 -> chat_log row P95Ms in admin-api-endpoint-summary.txt
• Worksheet /api/admin/users p95 -> users row P95Ms in admin-api-endpoint-summary.txt
• Worksheet Error rate -> Global failure rate in admin-api-endpoint-summary.txt
• Worksheet Slowest endpoint -> endpoint row with highest P95Ms

Notes:

• Keep the raw admin-api-summary.json file for audits and re-parsing.
• If an endpoint row is missing, treat as test/data issue and re-run with full endpoint coverage.