# Scaling and Performance Playbook

Purpose: define a repeatable method to measure performance, capture baselines, and decide scaling actions for the "Linear Void" streaming app. Scope: planning/execution for the "Scaling & Performance (planning only)" TODO items.

## 1) Manifest TTL Tuning Plan

Goal: tune the `.m3u8` TTL to ~`0.5x` segment duration while keeping playback stable.

Inputs: segment duration (`HLS_SEGMENT_DURATION`), current manifest TTL (Cloudflare cache rule), cache outcome distribution (`Cf-Cache-Status`).

Procedure:
1. Start from current manifest TTL (2s).
2. Capture 24h baseline headers from repeated `playlist.m3u8` requests.
3. Test candidate TTLs: 1s, 2s, 3s.
4. Compare rebuffer events (player logs), origin request increase/decrease, and HIT ratio changes.
5. Select the lowest TTL that keeps playback stable and origin load acceptable.

Acceptance: no material increase in rebuffer rate; manifest staleness not observable by users.

## 2) TTFB Baseline Plan

Goal: measure warm-cache TTFB from 3+ US regions with target `< 4s`.

Regions: US East (Virginia), US Central, US West (California/Oregon).

Method:
1. Use WebPageTest with repeat view enabled.
2. Run at least 5 samples per region.
3. Record median and p95 TTFB.
4. Store results in a table (date, region, median, p95, test URL).

Acceptance: warm-cache median < 4s in all regions; outliers documented with likely cause.

### TTFB Baseline Worksheet
Use one row per region; keep the raw WebPageTest result URL with the summary for comparison.

- US East (Virginia) / US Central / US West (California/Oregon): Date/time, Test URL `https://linearvoid.tv`, WebPageTest result URL, Median TTFB, p95 TTFB, Notes.

Capture notes: use repeat view for warm-cache; keep the same browser profile and test URL across regions; note any CDN cache-rule changes or deploys between samples.

## 3) WebSocket Load-Test Plan

Goal: characterize chat behavior across 10 → 100 → 500 → 1000 concurrent clients.

Tooling option A (Artillery):
```bash
npx artillery run scripts/loadtest/ws-chat.yml
```

Tooling option B (custom Go harness): open N concurrent WS clients, send a message every X seconds, measure round-trip latency and failures.

Metrics: p50/p95 message latency; error rate (connect/send/receive); dropped connections; CPU/memory for `chat`, `redis`, `nginx`.

Acceptance: p95 chat latency ≤ 1s at target concurrency tier; error rate ≤ 1% during steady state.

### WebSocket Load-Test Worksheet
Date/time, Environment, Commit SHA/image tag, Test URL, Tool used, Scenario duration, Concurrency tiers reached, p50 message latency, p95 message latency, Error rate, Dropped connections, CPU/memory notes for chat/redis/nginx, Raw results location, Conclusion, Next action.

## 4) Admin API Benchmark Plan

Goal: measure admin API p95 and identify slow endpoints.

Endpoints: `/api/admin/overview`, `/api/admin/analytics`, `/api/admin/chat-log`, `/api/admin/users`.

Tooling option A (`k6`):
```bash
k6 run scripts/loadtest/admin-api.js
```

Tooling option B (`wrk`):
```bash
wrk -t4 -c100 -d60s http://<host>/api/admin/overview
```

Acceptance: p95 < 1s for primary admin endpoints; slow endpoint list with remediation notes.

### Admin API Benchmark Worksheet
Date/time, Environment, Commit SHA/image tag, Test URL, Tool used, Scenario duration, Concurrency/virtual users, `/api/admin/overview|analytics|chat-log|users` p95, Error rate, Slowest endpoint, Cache or DB notes, Raw results location, Conclusion, Next action.

## 5) Cache Hit Ratio Tracking Plan

Goal: track Cloudflare cache hit ratio and alert when below threshold.

Implementation:
1. Pull Cloudflare GraphQL analytics on schedule.
2. Export to a Prometheus-compatible metric path.
3. Add a Grafana panel with 5m/1h/24h windows.
4. Alert when hit ratio < 90% for 15m.

Reference exporter:
- `scripts/cache-hit-ratio.js`
- `scripts/cache-hit-ratio.sh` for scheduled tasks / manual runs
- Example run: `export CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_TAG=... CLOUDFLARE_HOSTNAME=media.linearvoid.tv; node scripts/cache-hit-ratio.js`
- Intended output: Prometheus text format on stdout, suitable for cron, a Node wrapper, or a small scrape endpoint.

Metric target: `cache_hit_ratio >= 0.90`.

### Cache Hit Ratio Worksheet
Date/time, Environment, Commit SHA/image tag, Hostname, Window minutes, Cloudflare zone tag, Export command, Raw exporter output location, Hit requests, Total requests, Cache hit ratio, Threshold met, Notes.

## 6) Capacity Trigger Rules

Operational triggers:
- Cache miss ratio > 10% for 15m
- Chat p95 latency > 1s for 5m
- Host CPU > 80% for 10m
- Redis memory usage ratio (`linearvoid_chat_redis_memory_usage_ratio`) > 0.85 for 10m

Action mapping:
- Level 1: investigate + restart non-critical components
- Level 2: scale service resources (CPU/RAM)
- Level 3: activate standby planning path

### Capacity Trigger Review
Date/time, Environment, Commit SHA/image tag, Trigger hit, Threshold value, Observed value, Duration above threshold, Level selected, Immediate action taken, Services involved, Follow-up owner, Raw metrics/alert link, Notes.

## 7) Phase 2 Service Separation Plan

Objective: separate bottlenecked services while keeping ops simple.

Boundaries:
- `playout` isolated first (CPU-heavy)
- `chat` + `redis` grouped for latency
- `db` isolated for IO stability
- monitoring stack optionally separated after sustained load

Initial sizing:
- playout host: 4 vCPU / 8 GB
- chat host: 2–4 vCPU / 4–8 GB
- db host: 2 vCPU / 4 GB + fast disk

Exit criteria: repeated trigger events over 7 days; capacity saturation despite tuning.

### Phase 2 Boundary Worksheet
Date/time, Environment, Commit SHA/image tag, Trigger history summary, Repeated trigger count over 7 days, Primary bottleneck, Proposed split, `playout`/`chat`/`db` target host size, Monitoring separation needed, Data/state migration notes, Implementation owner, Review decision, Next action.

## 8) Phase 3 Warm Standby Plan

Objective: second-region resilience with DNS failover.

Layout:
- Primary: Hetzner Ashburn (US-East)
- Standby: Hetzner Hillsboro (US-West)

Design: warm standby stack deployed and health-checked; Redis replication or fallback mode documented; Postgres replication strategy selected (streaming replica or periodic snapshot restore); Cloudflare DNS failover policy defined (health-based or manual emergency switch).

### Phase 3 Warm Standby Worksheet
Date/time, Environment, Commit SHA/image tag, Primary region, Standby region, DNS failover mode, Redis strategy, Postgres strategy, Health check endpoints, Simulated cutover performed, Measured RTO, Measured RPO, Open risks, Implementation owner, Review decision, Notes.

## 9) Data Capture Template
For each benchmark run capture: Date/time, Environment (test-lab/staging/production), Commit SHA/image tag, Test scenario and duration, Raw metrics (latency, errors, resource usage), Conclusion and next action.

## 10) Execution Order
1. TTFB baseline
2. WebSocket load profile
3. Admin API benchmark
4. Cache ratio panel + alerting
5. Trigger tuning
6. Phase 2 boundary decision
7. Phase 3 standby design review

## 11) Benchmark Run Sheet
Use the same worksheet for every baseline/re-test so results stay comparable.

Preconditions: confirm target environment and commit SHA/image tag; confirm Cloudflare cache rules are active when testing TTFB or manifest TTL; confirm chat, admin, and playout services are healthy before load testing.

Run template:

| Item                       | Value |
| -------------------------- | ----- |
| Date/time                  |       |
| Environment                |       |
| Commit SHA / image tag     |       |
| Test scenario              |       |
| Concurrency / sample count |       |
| Command or tool            |       |
| Raw results location       |       |
| Conclusion                 |       |
| Next action                |       |

Suggested command mapping:
- TTFB baseline: WebPageTest repeat-view run against `https://linearvoid.tv`.
- WebSocket profile: `npx artillery run scripts/loadtest/ws-chat.yml` or the Go harness referenced above.
- Admin API benchmark: `k6 run scripts/loadtest/admin-api.js` or `wrk -t4 -c100 -d60s http://<host>/api/admin/overview`.

## 12) Scaling Decision Gate

Default review gate after any benchmark batch:
1. If warm-cache TTFB meets target but cache hit ratio falls below 90%, tune cache behavior before increasing capacity.
2. If chat or admin latency exceeds target while CPU and memory remain low, prioritize query or handler optimization.
3. If CPU, memory, or Redis saturation appears first, treat the Phase 2 service boundary guidance as the next action.
4. If the same threshold trips on multiple runs across 7 days, open the Phase 2 or Phase 3 planning path instead of tuning in place.

Decision note: keep the smallest change that restores the target metric and record the reason in the run sheet.

## 13) Artifact-to-Worksheet Field Mapping
Use when filling the WebSocket and Admin API worksheets from helper-generated artifacts.

### WebSocket worksheet mapping
Artifact sources: `artifacts/<timestamp>/ws-chat-report.json`, `artifacts/<timestamp>/ws-chat-summary.txt`.

Field mapping:
- `Tool used` → `artillery` (or `scripts/loadtest/run-loadtests.sh -s ws`)
- `Raw results location` → artifact folder path for this run
- `Concurrency tiers reached` → `ws-chat.yml` phases (`warmup-10`, `sustain-100`, `sustain-500`, `sustain-1000`)
- `Error rate` → `vusers.failure_rate_percent` from `ws-chat-summary.txt`
- `Dropped connections` → derive from `vusers.created - vusers.completed` (use `vusers.failed` as failure signal)
- `p95 message latency` → first latency line under `summary.*` in `ws-chat-summary.txt` containing `p95=...`
- `p50 message latency` → first latency line under `summary.*` in `ws-chat-summary.txt` containing `median=...`

Notes: the exact `summary.*` metric names in Artillery output can vary by engine/plugins; if no latency summary field is emitted, record this explicitly and use server-side latency metrics as fallback evidence.

### Admin API worksheet mapping
Artifact sources: `artifacts/<timestamp>/admin-api-summary.json`, `artifacts/<timestamp>/admin-api-output.txt`, `artifacts/<timestamp>/admin-api-endpoint-summary.txt`.

Field mapping:
- `Tool used` → `k6` (or `scripts/loadtest/run-loadtests.sh -s admin`)
- `Raw results location` → artifact folder path for this run
- `/api/admin/overview|analytics|chat-log|users` p95 → `overview|analytics|chat_log|users` row `P95Ms` in `admin-api-endpoint-summary.txt`
- `Error rate` → `Global failure rate` in `admin-api-endpoint-summary.txt`
- `Slowest endpoint` → endpoint row with highest `P95Ms`

Notes: keep the raw `admin-api-summary.json` file for audits and re-parsing; if an endpoint row is missing, treat it as a test/data issue and re-run with full endpoint coverage.

Summary helper (bash): run `scripts/loadtest/summarize-artillery.sh <report.json>` after an Artillery WS run, and `scripts/loadtest/summarize-k6.sh <summary.json>` after a k6 admin run, to produce the artifact files used in the mappings above.