refactor: modularize chat backend, update docs for .m4s segments

docs/operations/scaling-performance-playbook.md

@@ -0,0 +1,203 @@
+# Scaling and Performance Playbook
+
+Purpose: define repeatable method to measure performance, capture baseline data, and decide scaling actions.
+
+Scope: planning and execution guide for TODO items under "Scaling & Performance (planning only)".
+
+## 1) Manifest TTL Tuning Plan
+
+Goal: tune `.m3u8` TTL to about `0.5x` segment duration while keeping playback stable.
+
+Inputs:
+
+- Segment duration (`HLS_SEGMENT_DURATION`)
+- Current manifest TTL (Cloudflare cache rule)
+- Cache outcome distribution (`Cf-Cache-Status`)
+
+Procedure:
+
+1. Start from current manifest TTL (2s).
+2. Capture 24h baseline headers from repeated `playlist.m3u8` requests.
+3. Test candidate TTLs: 1s, 2s, 3s.
+4. Compare:
+   - rebuffer events from player logs
+   - origin request increase/decrease
+   - HIT ratio changes
+5. Select lowest TTL that keeps playback stable and origin load acceptable.
+
+Acceptance:
+
+- No material increase in rebuffer rate.
+- Manifest staleness not observable by users.
+
+## 2) TTFB Baseline Plan
+
+Goal: measure warm-cache TTFB from 3+ US regions with target `< 4s`.
+
+Suggested regions:
+
+- US East (Virginia)
+- US Central
+- US West (California/Oregon)
+
+Method:
+
+1. Use WebPageTest with repeat view enabled.
+2. Run at least 5 samples per region.
+3. Record median and p95 TTFB.
+4. Store results in table (date, region, median, p95, test URL).
+
+Acceptance:
+
+- Warm-cache median < 4s in all tested regions.
+- Outliers documented with likely cause.
+
+## 3) WebSocket Load-Test Plan
+
+Goal: characterize chat service behavior across 10 -> 100 -> 500 -> 1000 concurrent clients.
+
+Tooling option A (Artillery):
+
+```bash
+npx artillery run scripts/loadtest/ws-chat.yml
+```
+
+Tooling option B (custom Go harness):
+
+- open N concurrent WebSocket clients
+- send message every X seconds
+- measure round-trip latency and failures
+
+Metrics to capture:
+
+- p50/p95 message latency
+- error rate (connect, send, receive)
+- dropped connections
+- CPU/memory for `chat`, `redis`, `nginx`
+
+Acceptance:
+
+- p95 chat latency <= 1s at target concurrency tier.
+- error rate <= 1% during steady state.
+
+## 4) Admin API Benchmark Plan
+
+Goal: measure admin API p95 and identify slow endpoints.
+
+Tooling option A (`k6`):
+
+```bash
+k6 run scripts/loadtest/admin-api.js
+```
+
+Tooling option B (`wrk`):
+
+```bash
+wrk -t4 -c100 -d60s http://<host>/api/admin/overview
+```
+
+Endpoints:
+
+- `/api/admin/overview`
+- `/api/admin/analytics`
+- `/api/admin/chat-log`
+- `/api/admin/users`
+
+Acceptance:
+
+- p95 < 1s for primary admin endpoints.
+- slow endpoint list with remediation notes.
+
+## 5) Cache Hit Ratio Tracking Plan
+
+Goal: track Cloudflare cache hit ratio and alert when below threshold.
+
+Implementation outline:
+
+1. Pull Cloudflare GraphQL analytics on schedule.
+2. Export to Prometheus-compatible metric path.
+3. Add Grafana panel with 5m/1h/24h windows.
+4. Alert when hit ratio < 90% for 15m.
+
+Metric target:
+
+- `cache_hit_ratio >= 0.90`
+
+## 6) Capacity Trigger Rules
+
+Define operational triggers:
+
+- Cache miss ratio > 10% for 15m
+- Chat p95 latency > 1s for 5m
+- Host CPU > 80% for 10m
+- Redis memory > 85% sustained
+
+Action mapping:
+
+- Level 1: investigate + restart non-critical components
+- Level 2: scale service resources (CPU/RAM)
+- Level 3: activate standby planning path
+
+## 7) Phase 2 Service Separation Plan
+
+Objective: separate bottlenecked services while keeping simple ops.
+
+Boundaries:
+
+- `playout` isolated first (CPU-heavy)
+- `chat` + `redis` grouped for latency
+- `db` isolated for IO stability
+- monitoring stack optionally separated after sustained load
+
+Initial sizing guidance:
+
+- playout host: 4 vCPU / 8 GB
+- chat host: 2-4 vCPU / 4-8 GB
+- db host: 2 vCPU / 4 GB + fast disk
+
+Exit criteria for Phase 2:
+
+- repeated trigger events over 7 days
+- capacity saturation despite tuning
+
+## 8) Phase 3 Warm Standby Plan
+
+Objective: second-region resilience with DNS failover.
+
+Candidate layout:
+
+- Primary: Hetzner Ashburn (US-East)
+- Standby: Hetzner Hillsboro (US-West)
+
+Design:
+
+- Warm standby stack deployed and health-checked
+- Redis replication or fallback mode documented
+- Postgres replication strategy selected (streaming replica or periodic snapshot restore)
+- Cloudflare DNS failover policy defined (health-based or manual emergency switch)
+
+Failover drill requirements:
+
+- periodic simulated cutover
+- measured RTO/RPO logged in incident records
+
+## 9) Data Capture Template
+
+For each benchmark run, capture:
+
+- Date/time
+- Environment (test-lab/staging/production)
+- Commit SHA / image tag
+- Test scenario and duration
+- Raw metrics (latency, errors, resource usage)
+- Conclusion and next action
+
+## 10) Execution Order
+
+1. TTFB baseline
+2. WebSocket load profile
+3. Admin API benchmark
+4. Cache ratio panel + alerting
+5. Trigger tuning
+6. Phase 2 boundary decision
+7. Phase 3 standby design review