linearvoid/docs/operations/scaling-performance-playbook.md

Scaling and Performance Playbook

Purpose: define repeatable method to measure performance, capture baseline data, and decide scaling actions.

Scope: planning and execution guide for TODO items under "Scaling & Performance (planning only)".

1) Manifest TTL Tuning Plan

Goal: tune .m3u8 TTL to about 0.5x segment duration while keeping playback stable.

Inputs:

• Segment duration (HLS_SEGMENT_DURATION)
• Current manifest TTL (Cloudflare cache rule)
• Cache outcome distribution (Cf-Cache-Status)

Procedure:

1. Start from current manifest TTL (2s).
2. Capture 24h baseline headers from repeated playlist.m3u8 requests.
3. Test candidate TTLs: 1s, 2s, 3s.
4. Compare:
   • rebuffer events from player logs
   • origin request increase/decrease
   • HIT ratio changes
5. Select lowest TTL that keeps playback stable and origin load acceptable.

Acceptance:

• No material increase in rebuffer rate.
• Manifest staleness not observable by users.

2) TTFB Baseline Plan

Goal: measure warm-cache TTFB from 3+ US regions with target < 4s.

Suggested regions:

• US East (Virginia)
• US Central
• US West (California/Oregon)

Method:

1. Use WebPageTest with repeat view enabled.
2. Run at least 5 samples per region.
3. Record median and p95 TTFB.
4. Store results in table (date, region, median, p95, test URL).

Acceptance:

• Warm-cache median < 4s in all tested regions.
• Outliers documented with likely cause.

3) WebSocket Load-Test Plan

Goal: characterize chat service behavior across 10 -> 100 -> 500 -> 1000 concurrent clients.

Tooling option A (Artillery):

npx artillery run scripts/loadtest/ws-chat.yml

Tooling option B (custom Go harness):

• open N concurrent WebSocket clients
• send message every X seconds
• measure round-trip latency and failures

Metrics to capture:

• p50/p95 message latency
• error rate (connect, send, receive)
• dropped connections
• CPU/memory for chat, redis, nginx

Acceptance:

• p95 chat latency <= 1s at target concurrency tier.
• error rate <= 1% during steady state.

4) Admin API Benchmark Plan

Goal: measure admin API p95 and identify slow endpoints.

Tooling option A (k6):

k6 run scripts/loadtest/admin-api.js

Tooling option B (wrk):

wrk -t4 -c100 -d60s http://<host>/api/admin/overview

Endpoints:

• /api/admin/overview
• /api/admin/analytics
• /api/admin/chat-log
• /api/admin/users

Acceptance:

• p95 < 1s for primary admin endpoints.
• slow endpoint list with remediation notes.

5) Cache Hit Ratio Tracking Plan

Goal: track Cloudflare cache hit ratio and alert when below threshold.

Implementation outline:

1. Pull Cloudflare GraphQL analytics on schedule.
2. Export to Prometheus-compatible metric path.
3. Add Grafana panel with 5m/1h/24h windows.
4. Alert when hit ratio < 90% for 15m.

Metric target:

• cache_hit_ratio >= 0.90

6) Capacity Trigger Rules

Define operational triggers:

• Cache miss ratio > 10% for 15m
• Chat p95 latency > 1s for 5m
• Host CPU > 80% for 10m
• Redis memory > 85% sustained

Action mapping:

• Level 1: investigate + restart non-critical components
• Level 2: scale service resources (CPU/RAM)
• Level 3: activate standby planning path

7) Phase 2 Service Separation Plan

Objective: separate bottlenecked services while keeping simple ops.

Boundaries:

• playout isolated first (CPU-heavy)
• chat + redis grouped for latency
• db isolated for IO stability
• monitoring stack optionally separated after sustained load

Initial sizing guidance:

• playout host: 4 vCPU / 8 GB
• chat host: 2-4 vCPU / 4-8 GB
• db host: 2 vCPU / 4 GB + fast disk

Exit criteria for Phase 2:

• repeated trigger events over 7 days
• capacity saturation despite tuning

8) Phase 3 Warm Standby Plan

Objective: second-region resilience with DNS failover.

Candidate layout:

• Primary: Hetzner Ashburn (US-East)
• Standby: Hetzner Hillsboro (US-West)

Design:

• Warm standby stack deployed and health-checked
• Redis replication or fallback mode documented
• Postgres replication strategy selected (streaming replica or periodic snapshot restore)
• Cloudflare DNS failover policy defined (health-based or manual emergency switch)

Failover drill requirements:

• periodic simulated cutover
• measured RTO/RPO logged in incident records

9) Data Capture Template

For each benchmark run, capture:

• Date/time
• Environment (test-lab/staging/production)
• Commit SHA / image tag
• Test scenario and duration
• Raw metrics (latency, errors, resource usage)
• Conclusion and next action

10) Execution Order

1. TTFB baseline
2. WebSocket load profile
3. Admin API benchmark
4. Cache ratio panel + alerting
5. Trigger tuning
6. Phase 2 boundary decision
7. Phase 3 standby design review