204 lines
4.7 KiB
Markdown
204 lines
4.7 KiB
Markdown
# Scaling and Performance Playbook
|
|
|
|
Purpose: define repeatable method to measure performance, capture baseline data, and decide scaling actions.
|
|
|
|
Scope: planning and execution guide for TODO items under "Scaling & Performance (planning only)".
|
|
|
|
## 1) Manifest TTL Tuning Plan
|
|
|
|
Goal: tune `.m3u8` TTL to about `0.5x` segment duration while keeping playback stable.
|
|
|
|
Inputs:
|
|
|
|
- Segment duration (`HLS_SEGMENT_DURATION`)
|
|
- Current manifest TTL (Cloudflare cache rule)
|
|
- Cache outcome distribution (`Cf-Cache-Status`)
|
|
|
|
Procedure:
|
|
|
|
1. Start from current manifest TTL (2s).
|
|
2. Capture 24h baseline headers from repeated `playlist.m3u8` requests.
|
|
3. Test candidate TTLs: 1s, 2s, 3s.
|
|
4. Compare:
|
|
- rebuffer events from player logs
|
|
- origin request increase/decrease
|
|
- HIT ratio changes
|
|
5. Select lowest TTL that keeps playback stable and origin load acceptable.
|
|
|
|
Acceptance:
|
|
|
|
- No material increase in rebuffer rate.
|
|
- Manifest staleness not observable by users.
|
|
|
|
## 2) TTFB Baseline Plan
|
|
|
|
Goal: measure warm-cache TTFB from 3+ US regions with target `< 4s`.
|
|
|
|
Suggested regions:
|
|
|
|
- US East (Virginia)
|
|
- US Central
|
|
- US West (California/Oregon)
|
|
|
|
Method:
|
|
|
|
1. Use WebPageTest with repeat view enabled.
|
|
2. Run at least 5 samples per region.
|
|
3. Record median and p95 TTFB.
|
|
4. Store results in table (date, region, median, p95, test URL).
|
|
|
|
Acceptance:
|
|
|
|
- Warm-cache median < 4s in all tested regions.
|
|
- Outliers documented with likely cause.
|
|
|
|
## 3) WebSocket Load-Test Plan
|
|
|
|
Goal: characterize chat service behavior across 10 -> 100 -> 500 -> 1000 concurrent clients.
|
|
|
|
Tooling option A (Artillery):
|
|
|
|
```bash
|
|
npx artillery run scripts/loadtest/ws-chat.yml
|
|
```
|
|
|
|
Tooling option B (custom Go harness):
|
|
|
|
- open N concurrent WebSocket clients
|
|
- send message every X seconds
|
|
- measure round-trip latency and failures
|
|
|
|
Metrics to capture:
|
|
|
|
- p50/p95 message latency
|
|
- error rate (connect, send, receive)
|
|
- dropped connections
|
|
- CPU/memory for `chat`, `redis`, `nginx`
|
|
|
|
Acceptance:
|
|
|
|
- p95 chat latency <= 1s at target concurrency tier.
|
|
- error rate <= 1% during steady state.
|
|
|
|
## 4) Admin API Benchmark Plan
|
|
|
|
Goal: measure admin API p95 and identify slow endpoints.
|
|
|
|
Tooling option A (`k6`):
|
|
|
|
```bash
|
|
k6 run scripts/loadtest/admin-api.js
|
|
```
|
|
|
|
Tooling option B (`wrk`):
|
|
|
|
```bash
|
|
wrk -t4 -c100 -d60s http://<host>/api/admin/overview
|
|
```
|
|
|
|
Endpoints:
|
|
|
|
- `/api/admin/overview`
|
|
- `/api/admin/analytics`
|
|
- `/api/admin/chat-log`
|
|
- `/api/admin/users`
|
|
|
|
Acceptance:
|
|
|
|
- p95 < 1s for primary admin endpoints.
|
|
- slow endpoint list with remediation notes.
|
|
|
|
## 5) Cache Hit Ratio Tracking Plan
|
|
|
|
Goal: track Cloudflare cache hit ratio and alert when below threshold.
|
|
|
|
Implementation outline:
|
|
|
|
1. Pull Cloudflare GraphQL analytics on schedule.
|
|
2. Export to Prometheus-compatible metric path.
|
|
3. Add Grafana panel with 5m/1h/24h windows.
|
|
4. Alert when hit ratio < 90% for 15m.
|
|
|
|
Metric target:
|
|
|
|
- `cache_hit_ratio >= 0.90`
|
|
|
|
## 6) Capacity Trigger Rules
|
|
|
|
Define operational triggers:
|
|
|
|
- Cache miss ratio > 10% for 15m
|
|
- Chat p95 latency > 1s for 5m
|
|
- Host CPU > 80% for 10m
|
|
- Redis memory > 85% sustained
|
|
|
|
Action mapping:
|
|
|
|
- Level 1: investigate + restart non-critical components
|
|
- Level 2: scale service resources (CPU/RAM)
|
|
- Level 3: activate standby planning path
|
|
|
|
## 7) Phase 2 Service Separation Plan
|
|
|
|
Objective: separate bottlenecked services while keeping simple ops.
|
|
|
|
Boundaries:
|
|
|
|
- `playout` isolated first (CPU-heavy)
|
|
- `chat` + `redis` grouped for latency
|
|
- `db` isolated for IO stability
|
|
- monitoring stack optionally separated after sustained load
|
|
|
|
Initial sizing guidance:
|
|
|
|
- playout host: 4 vCPU / 8 GB
|
|
- chat host: 2-4 vCPU / 4-8 GB
|
|
- db host: 2 vCPU / 4 GB + fast disk
|
|
|
|
Exit criteria for Phase 2:
|
|
|
|
- repeated trigger events over 7 days
|
|
- capacity saturation despite tuning
|
|
|
|
## 8) Phase 3 Warm Standby Plan
|
|
|
|
Objective: second-region resilience with DNS failover.
|
|
|
|
Candidate layout:
|
|
|
|
- Primary: Hetzner Ashburn (US-East)
|
|
- Standby: Hetzner Hillsboro (US-West)
|
|
|
|
Design:
|
|
|
|
- Warm standby stack deployed and health-checked
|
|
- Redis replication or fallback mode documented
|
|
- Postgres replication strategy selected (streaming replica or periodic snapshot restore)
|
|
- Cloudflare DNS failover policy defined (health-based or manual emergency switch)
|
|
|
|
Failover drill requirements:
|
|
|
|
- periodic simulated cutover
|
|
- measured RTO/RPO logged in incident records
|
|
|
|
## 9) Data Capture Template
|
|
|
|
For each benchmark run, capture:
|
|
|
|
- Date/time
|
|
- Environment (test-lab/staging/production)
|
|
- Commit SHA / image tag
|
|
- Test scenario and duration
|
|
- Raw metrics (latency, errors, resource usage)
|
|
- Conclusion and next action
|
|
|
|
## 10) Execution Order
|
|
|
|
1. TTFB baseline
|
|
2. WebSocket load profile
|
|
3. Admin API benchmark
|
|
4. Cache ratio panel + alerting
|
|
5. Trigger tuning
|
|
6. Phase 2 boundary decision
|
|
7. Phase 3 standby design review
|