# Scaling and Performance Playbook

Purpose: define repeatable method to measure performance, capture baseline data, and decide scaling actions.

Scope: planning and execution guide for TODO items under "Scaling & Performance (planning only)".

## 1) Manifest TTL Tuning Plan

Goal: tune `.m3u8` TTL to about `0.5x` segment duration while keeping playback stable.

Inputs:

- Segment duration (`HLS_SEGMENT_DURATION`)
- Current manifest TTL (Cloudflare cache rule)
- Cache outcome distribution (`Cf-Cache-Status`)

Procedure:

1. Start from current manifest TTL (2s).
2. Capture 24h baseline headers from repeated `playlist.m3u8` requests.
3. Test candidate TTLs: 1s, 2s, 3s.
4. Compare:
   - rebuffer events from player logs
   - origin request increase/decrease
   - HIT ratio changes
5. Select lowest TTL that keeps playback stable and origin load acceptable.

Acceptance:

- No material increase in rebuffer rate.
- Manifest staleness not observable by users.

## 2) TTFB Baseline Plan

Goal: measure warm-cache TTFB from 3+ US regions with target `< 4s`.

Suggested regions:

- US East (Virginia)
- US Central
- US West (California/Oregon)

Method:

1. Use WebPageTest with repeat view enabled.
2. Run at least 5 samples per region.
3. Record median and p95 TTFB.
4. Store results in table (date, region, median, p95, test URL).

Acceptance:

- Warm-cache median < 4s in all tested regions.
- Outliers documented with likely cause.

## 3) WebSocket Load-Test Plan

Goal: characterize chat service behavior across 10 -> 100 -> 500 -> 1000 concurrent clients.

Tooling option A (Artillery):

```bash
npx artillery run scripts/loadtest/ws-chat.yml
```

Tooling option B (custom Go harness):

- open N concurrent WebSocket clients
- send message every X seconds
- measure round-trip latency and failures

Metrics to capture:

- p50/p95 message latency
- error rate (connect, send, receive)
- dropped connections
- CPU/memory for `chat`, `redis`, `nginx`

Acceptance:

- p95 chat latency <= 1s at target concurrency tier.
- error rate <= 1% during steady state.

## 4) Admin API Benchmark Plan

Goal: measure admin API p95 and identify slow endpoints.

Tooling option A (`k6`):

```bash
k6 run scripts/loadtest/admin-api.js
```

Tooling option B (`wrk`):

```bash
wrk -t4 -c100 -d60s http://<host>/api/admin/overview
```

Endpoints:

- `/api/admin/overview`
- `/api/admin/analytics`
- `/api/admin/chat-log`
- `/api/admin/users`

Acceptance:

- p95 < 1s for primary admin endpoints.
- slow endpoint list with remediation notes.

## 5) Cache Hit Ratio Tracking Plan

Goal: track Cloudflare cache hit ratio and alert when below threshold.

Implementation outline:

1. Pull Cloudflare GraphQL analytics on schedule.
2. Export to Prometheus-compatible metric path.
3. Add Grafana panel with 5m/1h/24h windows.
4. Alert when hit ratio < 90% for 15m.

Metric target:

- `cache_hit_ratio >= 0.90`

## 6) Capacity Trigger Rules

Define operational triggers:

- Cache miss ratio > 10% for 15m
- Chat p95 latency > 1s for 5m
- Host CPU > 80% for 10m
- Redis memory > 85% sustained

Action mapping:

- Level 1: investigate + restart non-critical components
- Level 2: scale service resources (CPU/RAM)
- Level 3: activate standby planning path

## 7) Phase 2 Service Separation Plan

Objective: separate bottlenecked services while keeping simple ops.

Boundaries:

- `playout` isolated first (CPU-heavy)
- `chat` + `redis` grouped for latency
- `db` isolated for IO stability
- monitoring stack optionally separated after sustained load

Initial sizing guidance:

- playout host: 4 vCPU / 8 GB
- chat host: 2-4 vCPU / 4-8 GB
- db host: 2 vCPU / 4 GB + fast disk

Exit criteria for Phase 2:

- repeated trigger events over 7 days
- capacity saturation despite tuning

## 8) Phase 3 Warm Standby Plan

Objective: second-region resilience with DNS failover.

Candidate layout:

- Primary: Hetzner Ashburn (US-East)
- Standby: Hetzner Hillsboro (US-West)

Design:

- Warm standby stack deployed and health-checked
- Redis replication or fallback mode documented
- Postgres replication strategy selected (streaming replica or periodic snapshot restore)
- Cloudflare DNS failover policy defined (health-based or manual emergency switch)

Failover drill requirements:

- periodic simulated cutover
- measured RTO/RPO logged in incident records

## 9) Data Capture Template

For each benchmark run, capture:

- Date/time
- Environment (test-lab/staging/production)
- Commit SHA / image tag
- Test scenario and duration
- Raw metrics (latency, errors, resource usage)
- Conclusion and next action

## 10) Execution Order

1. TTFB baseline
2. WebSocket load profile
3. Admin API benchmark
4. Cache ratio panel + alerting
5. Trigger tuning
6. Phase 2 boundary decision
7. Phase 3 standby design review