443 lines
12 KiB
Markdown
443 lines
12 KiB
Markdown
# Scaling and Performance Playbook
|
|
|
|
Purpose: define repeatable method to measure performance, capture baseline data, and decide scaling actions.
|
|
|
|
Scope: planning and execution guide for TODO items under "Scaling & Performance (planning only)".
|
|
|
|
## 1) Manifest TTL Tuning Plan
|
|
|
|
Goal: tune `.m3u8` TTL to about `0.5x` segment duration while keeping playback stable.
|
|
|
|
Inputs:
|
|
|
|
- Segment duration (`HLS_SEGMENT_DURATION`)
|
|
- Current manifest TTL (Cloudflare cache rule)
|
|
- Cache outcome distribution (`Cf-Cache-Status`)
|
|
|
|
Procedure:
|
|
|
|
1. Start from current manifest TTL (2s).
|
|
2. Capture 24h baseline headers from repeated `playlist.m3u8` requests.
|
|
3. Test candidate TTLs: 1s, 2s, 3s.
|
|
4. Compare:
|
|
- rebuffer events from player logs
|
|
- origin request increase/decrease
|
|
- HIT ratio changes
|
|
5. Select lowest TTL that keeps playback stable and origin load acceptable.
|
|
|
|
Acceptance:
|
|
|
|
- No material increase in rebuffer rate.
|
|
- Manifest staleness not observable by users.
|
|
|
|
## 2) TTFB Baseline Plan
|
|
|
|
Goal: measure warm-cache TTFB from 3+ US regions with target `< 4s`.
|
|
|
|
Suggested regions:
|
|
|
|
- US East (Virginia)
|
|
- US Central
|
|
- US West (California/Oregon)
|
|
|
|
Method:
|
|
|
|
1. Use WebPageTest with repeat view enabled.
|
|
2. Run at least 5 samples per region.
|
|
3. Record median and p95 TTFB.
|
|
4. Store results in table (date, region, median, p95, test URL).
|
|
|
|
Acceptance:
|
|
|
|
- Warm-cache median < 4s in all tested regions.
|
|
- Outliers documented with likely cause.
|
|
|
|
### TTFB Baseline Worksheet
|
|
|
|
Use one row per region and keep the raw WebPageTest result URL with the summary so later runs can be compared against the same evidence.
|
|
|
|
- US East (Virginia)
|
|
- Date/time:
|
|
- Test URL: <https://linearvoid.tv>
|
|
- WebPageTest result URL:
|
|
- Median TTFB:
|
|
- p95 TTFB:
|
|
- Notes:
|
|
- US Central
|
|
- Date/time:
|
|
- Test URL: <https://linearvoid.tv>
|
|
- WebPageTest result URL:
|
|
- Median TTFB:
|
|
- p95 TTFB:
|
|
- Notes:
|
|
- US West (California/Oregon)
|
|
- Date/time:
|
|
- Test URL: <https://linearvoid.tv>
|
|
- WebPageTest result URL:
|
|
- Median TTFB:
|
|
- p95 TTFB:
|
|
- Notes:
|
|
|
|
Suggested capture notes:
|
|
|
|
- Use repeat view for the warm-cache measurement.
|
|
- Keep the same browser profile and test URL across regions.
|
|
- Note any CDN cache-rule changes or deploys that occurred between samples.
|
|
|
|
## 3) WebSocket Load-Test Plan
|
|
|
|
Goal: characterize chat service behavior across 10 -> 100 -> 500 -> 1000 concurrent clients.
|
|
|
|
Tooling option A (Artillery):
|
|
|
|
```bash
|
|
npx artillery run scripts/loadtest/ws-chat.yml
|
|
```
|
|
|
|
Tooling option B (custom Go harness):
|
|
|
|
- open N concurrent WebSocket clients
|
|
- send message every X seconds
|
|
- measure round-trip latency and failures
|
|
|
|
Metrics to capture:
|
|
|
|
- p50/p95 message latency
|
|
- error rate (connect, send, receive)
|
|
- dropped connections
|
|
- CPU/memory for `chat`, `redis`, `nginx`
|
|
|
|
Acceptance:
|
|
|
|
- p95 chat latency <= 1s at target concurrency tier.
|
|
- error rate <= 1% during steady state.
|
|
|
|
### WebSocket Load-Test Worksheet
|
|
|
|
- Date/time:
|
|
- Environment:
|
|
- Commit SHA / image tag:
|
|
- Test URL:
|
|
- Tool used:
|
|
- Scenario duration:
|
|
- Concurrency tiers reached:
|
|
- p50 message latency:
|
|
- p95 message latency:
|
|
- Error rate:
|
|
- Dropped connections:
|
|
- CPU/memory notes for chat:
|
|
- CPU/memory notes for redis:
|
|
- CPU/memory notes for nginx:
|
|
- Raw results location:
|
|
- Conclusion:
|
|
- Next action:
|
|
|
|
## 4) Admin API Benchmark Plan
|
|
|
|
Goal: measure admin API p95 and identify slow endpoints.
|
|
|
|
Tooling option A (`k6`):
|
|
|
|
```bash
|
|
k6 run scripts/loadtest/admin-api.js
|
|
```
|
|
|
|
Tooling option B (`wrk`):
|
|
|
|
```bash
|
|
wrk -t4 -c100 -d60s http://<host>/api/admin/overview
|
|
```
|
|
|
|
Endpoints:
|
|
|
|
- `/api/admin/overview`
|
|
- `/api/admin/analytics`
|
|
- `/api/admin/chat-log`
|
|
- `/api/admin/users`
|
|
|
|
Acceptance:
|
|
|
|
- p95 < 1s for primary admin endpoints.
|
|
- slow endpoint list with remediation notes.
|
|
|
|
### Admin API Benchmark Worksheet
|
|
|
|
- Date/time:
|
|
- Environment:
|
|
- Commit SHA / image tag:
|
|
- Test URL:
|
|
- Tool used:
|
|
- Scenario duration:
|
|
- Concurrency / virtual users:
|
|
- `/api/admin/overview` p95:
|
|
- `/api/admin/analytics` p95:
|
|
- `/api/admin/chat-log` p95:
|
|
- `/api/admin/users` p95:
|
|
- Error rate:
|
|
- Slowest endpoint:
|
|
- Cache or DB notes:
|
|
- Raw results location:
|
|
- Conclusion:
|
|
- Next action:
|
|
|
|
## 5) Cache Hit Ratio Tracking Plan
|
|
|
|
Goal: track Cloudflare cache hit ratio and alert when below threshold.
|
|
|
|
Implementation outline:
|
|
|
|
1. Pull Cloudflare GraphQL analytics on schedule.
|
|
2. Export to Prometheus-compatible metric path.
|
|
3. Add Grafana panel with 5m/1h/24h windows.
|
|
4. Alert when hit ratio < 90% for 15m.
|
|
|
|
Reference exporter:
|
|
|
|
- `scripts/cache-hit-ratio.js`
|
|
- `scripts/cache-hit-ratio.ps1` for Windows scheduled tasks / manual runs
|
|
- Example run: `CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_TAG=... CLOUDFLARE_HOSTNAME=media.linearvoid.tv node scripts/cache-hit-ratio.js`
|
|
- Windows example: `pwsh -File scripts/cache-hit-ratio.ps1`
|
|
- Intended output: Prometheus text format on stdout, suitable for cron, a Node wrapper, or a small scrape endpoint.
|
|
|
|
Metric target:
|
|
|
|
- `cache_hit_ratio >= 0.90`
|
|
|
|
### Cache Hit Ratio Worksheet
|
|
|
|
- Date/time:
|
|
- Environment:
|
|
- Commit SHA / image tag:
|
|
- Hostname:
|
|
- Window minutes:
|
|
- Cloudflare zone tag:
|
|
- Export command:
|
|
- Raw exporter output location:
|
|
- Hit requests:
|
|
- Total requests:
|
|
- Cache hit ratio:
|
|
- Threshold met:
|
|
- Notes:
|
|
|
|
## 6) Capacity Trigger Rules
|
|
|
|
Define operational triggers:
|
|
|
|
- Cache miss ratio > 10% for 15m
|
|
- Chat p95 latency > 1s for 5m
|
|
- Host CPU > 80% for 10m
|
|
- Redis memory usage ratio (`linearvoid_chat_redis_memory_usage_ratio`) > 0.85 for 10m
|
|
|
|
Action mapping:
|
|
|
|
- Level 1: investigate + restart non-critical components
|
|
- Level 2: scale service resources (CPU/RAM)
|
|
- Level 3: activate standby planning path
|
|
|
|
### Capacity Trigger Review
|
|
|
|
- Date/time:
|
|
- Environment:
|
|
- Commit SHA / image tag:
|
|
- Trigger hit:
|
|
- Threshold value:
|
|
- Observed value:
|
|
- Duration above threshold:
|
|
- Level selected:
|
|
- Immediate action taken:
|
|
- Services involved:
|
|
- Follow-up owner:
|
|
- Raw metrics / alert link:
|
|
- Notes:
|
|
|
|
## 7) Phase 2 Service Separation Plan
|
|
|
|
Objective: separate bottlenecked services while keeping simple ops.
|
|
|
|
Boundaries:
|
|
|
|
- `playout` isolated first (CPU-heavy)
|
|
- `chat` + `redis` grouped for latency
|
|
- `db` isolated for IO stability
|
|
- monitoring stack optionally separated after sustained load
|
|
|
|
Initial sizing guidance:
|
|
|
|
- playout host: 4 vCPU / 8 GB
|
|
- chat host: 2-4 vCPU / 4-8 GB
|
|
- db host: 2 vCPU / 4 GB + fast disk
|
|
|
|
Exit criteria for Phase 2:
|
|
|
|
- repeated trigger events over 7 days
|
|
- capacity saturation despite tuning
|
|
|
|
### Phase 2 Boundary Worksheet
|
|
|
|
- Date/time:
|
|
- Environment:
|
|
- Commit SHA / image tag:
|
|
- Trigger history summary:
|
|
- Repeated trigger count over 7 days:
|
|
- Primary bottleneck:
|
|
- Proposed split:
|
|
- `playout` target host size:
|
|
- `chat` target host size:
|
|
- `db` target host size:
|
|
- Monitoring separation needed:
|
|
- Data/state migration notes:
|
|
- Implementation owner:
|
|
- Review decision:
|
|
- Next action:
|
|
|
|
## 8) Phase 3 Warm Standby Plan
|
|
|
|
Objective: second-region resilience with DNS failover.
|
|
|
|
Candidate layout:
|
|
|
|
- Primary: Hetzner Ashburn (US-East)
|
|
- Standby: Hetzner Hillsboro (US-West)
|
|
|
|
Design:
|
|
|
|
- Warm standby stack deployed and health-checked
|
|
- Redis replication or fallback mode documented
|
|
- Postgres replication strategy selected (streaming replica or periodic snapshot restore)
|
|
- Cloudflare DNS failover policy defined (health-based or manual emergency switch)
|
|
|
|
Failover drill requirements:
|
|
|
|
### Phase 3 Warm Standby Worksheet
|
|
|
|
- Date/time:
|
|
- Environment:
|
|
- Commit SHA / image tag:
|
|
- Primary region:
|
|
- Standby region:
|
|
- DNS failover mode:
|
|
- Redis strategy:
|
|
- Postgres strategy:
|
|
- Health check endpoints:
|
|
- Simulated cutover performed:
|
|
- Measured RTO:
|
|
- Measured RPO:
|
|
- Open risks:
|
|
- Implementation owner:
|
|
- Review decision:
|
|
- Notes:
|
|
|
|
## 9) Data Capture Template
|
|
|
|
For each benchmark run, capture:
|
|
|
|
- Date/time
|
|
- Environment (test-lab/staging/production)
|
|
- Commit SHA / image tag
|
|
- Test scenario and duration
|
|
- Raw metrics (latency, errors, resource usage)
|
|
- Conclusion and next action
|
|
|
|
## 10) Execution Order
|
|
|
|
1. TTFB baseline
|
|
2. WebSocket load profile
|
|
3. Admin API benchmark
|
|
4. Cache ratio panel + alerting
|
|
5. Trigger tuning
|
|
6. Phase 2 boundary decision
|
|
7. Phase 3 standby design review
|
|
|
|
## 11) Benchmark Run Sheet
|
|
|
|
Use the same worksheet for every baseline or re-test so results stay comparable.
|
|
|
|
Preconditions:
|
|
|
|
- Confirm the target environment and commit SHA/image tag.
|
|
- Confirm Cloudflare cache rules are active when testing TTFB or manifest TTL.
|
|
- Confirm the chat, admin, and playout services are healthy before load testing.
|
|
|
|
Run template:
|
|
|
|
| Item | Value |
|
|
| -------------------------- | ----- |
|
|
| Date/time | |
|
|
| Environment | |
|
|
| Commit SHA / image tag | |
|
|
| Test scenario | |
|
|
| Concurrency / sample count | |
|
|
| Command or tool | |
|
|
| Raw results location | |
|
|
| Conclusion | |
|
|
| Next action | |
|
|
|
|
Suggested command mapping:
|
|
|
|
- TTFB baseline: WebPageTest repeat-view run against `https://linearvoid.tv`.
|
|
- WebSocket profile: `npx artillery run scripts/loadtest/ws-chat.yml` or the Go harness referenced above.
|
|
- Admin API benchmark: `k6 run scripts/loadtest/admin-api.js` or `wrk -t4 -c100 -d60s http://<host>/api/admin/overview`.
|
|
|
|
## 12) Scaling Decision Gate
|
|
|
|
Treat the following as the default review gate after any benchmark batch:
|
|
|
|
1. If warm-cache TTFB meets target but cache hit ratio falls below 90%, tune cache behavior before increasing capacity.
|
|
2. If chat or admin latency exceeds target while CPU and memory remain low, prioritize query or handler optimization.
|
|
3. If CPU, memory, or Redis saturation appears first, treat the service boundary guidance in Phase 2 as the next action.
|
|
4. If the same threshold trips on multiple runs across 7 days, open the Phase 2 or Phase 3 planning path instead of tuning in place.
|
|
|
|
Decision note:
|
|
|
|
- Keep the smallest change that restores the target metric and record the reason in the run sheet.
|
|
|
|
## 13) Artifact-to-Worksheet Field Mapping
|
|
|
|
Use this section when filling the WebSocket and Admin API worksheets from helper-generated artifacts.
|
|
|
|
### WebSocket worksheet mapping
|
|
|
|
Artifact sources:
|
|
|
|
- `artifacts/<timestamp>/ws-chat-report.json`
|
|
- `artifacts/<timestamp>/ws-chat-summary.txt`
|
|
|
|
Field mapping:
|
|
|
|
- Worksheet `Tool used` -> `artillery` (or `run-loadtests.ps1 -Scenario ws`)
|
|
- Worksheet `Raw results location` -> artifact folder path for this run
|
|
- Worksheet `Concurrency tiers reached` -> `ws-chat.yml` phases (`warmup-10`, `sustain-100`, `sustain-500`, `sustain-1000`)
|
|
- Worksheet `Error rate` -> `vusers.failure_rate_percent` from `ws-chat-summary.txt`
|
|
- Worksheet `Dropped connections` -> derive from `vusers.created - vusers.completed` (use `vusers.failed` as failure signal)
|
|
- Worksheet `p95 message latency` -> first available latency line under `summary.*` in `ws-chat-summary.txt` that includes `p95=...`
|
|
- Worksheet `p50 message latency` -> first available latency line under `summary.*` in `ws-chat-summary.txt` that includes `median=...`
|
|
|
|
Notes:
|
|
|
|
- The exact `summary.*` metric names in Artillery output can vary by engine/plugins.
|
|
- If no latency summary field is emitted, record this explicitly and use server-side latency metrics as fallback evidence.
|
|
|
|
### Admin API worksheet mapping
|
|
|
|
Artifact sources:
|
|
|
|
- `artifacts/<timestamp>/admin-api-summary.json`
|
|
- `artifacts/<timestamp>/admin-api-output.txt`
|
|
- `artifacts/<timestamp>/admin-api-endpoint-summary.txt`
|
|
|
|
Field mapping:
|
|
|
|
- Worksheet `Tool used` -> `k6` (or `run-loadtests.ps1 -Scenario admin`)
|
|
- Worksheet `Raw results location` -> artifact folder path for this run
|
|
- Worksheet `/api/admin/overview` p95 -> `overview` row `P95Ms` in `admin-api-endpoint-summary.txt`
|
|
- Worksheet `/api/admin/analytics` p95 -> `analytics` row `P95Ms` in `admin-api-endpoint-summary.txt`
|
|
- Worksheet `/api/admin/chat-log` p95 -> `chat_log` row `P95Ms` in `admin-api-endpoint-summary.txt`
|
|
- Worksheet `/api/admin/users` p95 -> `users` row `P95Ms` in `admin-api-endpoint-summary.txt`
|
|
- Worksheet `Error rate` -> `Global failure rate` in `admin-api-endpoint-summary.txt`
|
|
- Worksheet `Slowest endpoint` -> endpoint row with highest `P95Ms`
|
|
|
|
Notes:
|
|
|
|
- Keep the raw `admin-api-summary.json` file for audits and re-parsing.
|
|
- If an endpoint row is missing, treat as test/data issue and re-run with full endpoint coverage.
|