feat: add metrics, observability, test fixtures, and scaling playbook artifacts

docs/operations/scaling-performance-playbook.md

@@ -52,6 +52,38 @@ Acceptance:
 - Warm-cache median < 4s in all tested regions.
 - Outliers documented with likely cause.
 
+### TTFB Baseline Worksheet
+
+Use one row per region and keep the raw WebPageTest result URL with the summary so later runs can be compared against the same evidence.
+
+- US East (Virginia)
+  - Date/time:
+  - Test URL: <https://linearvoid.tv>
+  - WebPageTest result URL:
+  - Median TTFB:
+  - p95 TTFB:
+  - Notes:
+- US Central
+  - Date/time:
+  - Test URL: <https://linearvoid.tv>
+  - WebPageTest result URL:
+  - Median TTFB:
+  - p95 TTFB:
+  - Notes:
+- US West (California/Oregon)
+  - Date/time:
+  - Test URL: <https://linearvoid.tv>
+  - WebPageTest result URL:
+  - Median TTFB:
+  - p95 TTFB:
+  - Notes:
+
+Suggested capture notes:
+
+- Use repeat view for the warm-cache measurement.
+- Keep the same browser profile and test URL across regions.
+- Note any CDN cache-rule changes or deploys that occurred between samples.
+
 ## 3) WebSocket Load-Test Plan
 
 Goal: characterize chat service behavior across 10 -> 100 -> 500 -> 1000 concurrent clients.
@@ -80,6 +112,26 @@ Acceptance:
 - p95 chat latency <= 1s at target concurrency tier.
 - error rate <= 1% during steady state.
 
+### WebSocket Load-Test Worksheet
+
+- Date/time:
+- Environment:
+- Commit SHA / image tag:
+- Test URL:
+- Tool used:
+- Scenario duration:
+- Concurrency tiers reached:
+- p50 message latency:
+- p95 message latency:
+- Error rate:
+- Dropped connections:
+- CPU/memory notes for chat:
+- CPU/memory notes for redis:
+- CPU/memory notes for nginx:
+- Raw results location:
+- Conclusion:
+- Next action:
+
 ## 4) Admin API Benchmark Plan
 
 Goal: measure admin API p95 and identify slow endpoints.
@@ -108,6 +160,26 @@ Acceptance:
 - p95 < 1s for primary admin endpoints.
 - slow endpoint list with remediation notes.
 
+### Admin API Benchmark Worksheet
+
+- Date/time:
+- Environment:
+- Commit SHA / image tag:
+- Test URL:
+- Tool used:
+- Scenario duration:
+- Concurrency / virtual users:
+- `/api/admin/overview` p95:
+- `/api/admin/analytics` p95:
+- `/api/admin/chat-log` p95:
+- `/api/admin/users` p95:
+- Error rate:
+- Slowest endpoint:
+- Cache or DB notes:
+- Raw results location:
+- Conclusion:
+- Next action:
+
 ## 5) Cache Hit Ratio Tracking Plan
 
 Goal: track Cloudflare cache hit ratio and alert when below threshold.
@@ -119,10 +191,34 @@ Implementation outline:
 3. Add Grafana panel with 5m/1h/24h windows.
 4. Alert when hit ratio < 90% for 15m.
 
+Reference exporter:
+
+- `scripts/cache-hit-ratio.js`
+- `scripts/cache-hit-ratio.ps1` for Windows scheduled tasks / manual runs
+- Example run: `CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_TAG=... CLOUDFLARE_HOSTNAME=media.linearvoid.tv node scripts/cache-hit-ratio.js`
+- Windows example: `pwsh -File scripts/cache-hit-ratio.ps1`
+- Intended output: Prometheus text format on stdout, suitable for cron, a Node wrapper, or a small scrape endpoint.
+
 Metric target:
 
 - `cache_hit_ratio >= 0.90`
 
+### Cache Hit Ratio Worksheet
+
+- Date/time:
+- Environment:
+- Commit SHA / image tag:
+- Hostname:
+- Window minutes:
+- Cloudflare zone tag:
+- Export command:
+- Raw exporter output location:
+- Hit requests:
+- Total requests:
+- Cache hit ratio:
+- Threshold met:
+- Notes:
+
 ## 6) Capacity Trigger Rules
 
 Define operational triggers:
@@ -130,7 +226,7 @@ Define operational triggers:
 - Cache miss ratio > 10% for 15m
 - Chat p95 latency > 1s for 5m
 - Host CPU > 80% for 10m
-- Redis memory > 85% sustained
+- Redis memory usage ratio (`linearvoid_chat_redis_memory_usage_ratio`) > 0.85 for 10m
 
 Action mapping:
 
@@ -138,6 +234,22 @@ Action mapping:
 - Level 2: scale service resources (CPU/RAM)
 - Level 3: activate standby planning path
 
+### Capacity Trigger Review
+
+- Date/time:
+- Environment:
+- Commit SHA / image tag:
+- Trigger hit:
+- Threshold value:
+- Observed value:
+- Duration above threshold:
+- Level selected:
+- Immediate action taken:
+- Services involved:
+- Follow-up owner:
+- Raw metrics / alert link:
+- Notes:
+
 ## 7) Phase 2 Service Separation Plan
 
 Objective: separate bottlenecked services while keeping simple ops.
@@ -160,6 +272,24 @@ Exit criteria for Phase 2:
 - repeated trigger events over 7 days
 - capacity saturation despite tuning
 
+### Phase 2 Boundary Worksheet
+
+- Date/time:
+- Environment:
+- Commit SHA / image tag:
+- Trigger history summary:
+- Repeated trigger count over 7 days:
+- Primary bottleneck:
+- Proposed split:
+- `playout` target host size:
+- `chat` target host size:
+- `db` target host size:
+- Monitoring separation needed:
+- Data/state migration notes:
+- Implementation owner:
+- Review decision:
+- Next action:
+
 ## 8) Phase 3 Warm Standby Plan
 
 Objective: second-region resilience with DNS failover.
@@ -178,8 +308,24 @@ Design:
 
 Failover drill requirements:
 
-- periodic simulated cutover
-- measured RTO/RPO logged in incident records
+### Phase 3 Warm Standby Worksheet
+
+- Date/time:
+- Environment:
+- Commit SHA / image tag:
+- Primary region:
+- Standby region:
+- DNS failover mode:
+- Redis strategy:
+- Postgres strategy:
+- Health check endpoints:
+- Simulated cutover performed:
+- Measured RTO:
+- Measured RPO:
+- Open risks:
+- Implementation owner:
+- Review decision:
+- Notes:
 
 ## 9) Data Capture Template
 
@@ -201,3 +347,96 @@ For each benchmark run, capture:
 5. Trigger tuning
 6. Phase 2 boundary decision
 7. Phase 3 standby design review
+
+## 11) Benchmark Run Sheet
+
+Use the same worksheet for every baseline or re-test so results stay comparable.
+
+Preconditions:
+
+- Confirm the target environment and commit SHA/image tag.
+- Confirm Cloudflare cache rules are active when testing TTFB or manifest TTL.
+- Confirm the chat, admin, and playout services are healthy before load testing.
+
+Run template:
+
+| Item                       | Value |
+| -------------------------- | ----- |
+| Date/time                  |       |
+| Environment                |       |
+| Commit SHA / image tag     |       |
+| Test scenario              |       |
+| Concurrency / sample count |       |
+| Command or tool            |       |
+| Raw results location       |       |
+| Conclusion                 |       |
+| Next action                |       |
+
+Suggested command mapping:
+
+- TTFB baseline: WebPageTest repeat-view run against `https://linearvoid.tv`.
+- WebSocket profile: `npx artillery run scripts/loadtest/ws-chat.yml` or the Go harness referenced above.
+- Admin API benchmark: `k6 run scripts/loadtest/admin-api.js` or `wrk -t4 -c100 -d60s http://<host>/api/admin/overview`.
+
+## 12) Scaling Decision Gate
+
+Treat the following as the default review gate after any benchmark batch:
+
+1. If warm-cache TTFB meets target but cache hit ratio falls below 90%, tune cache behavior before increasing capacity.
+2. If chat or admin latency exceeds target while CPU and memory remain low, prioritize query or handler optimization.
+3. If CPU, memory, or Redis saturation appears first, treat the service boundary guidance in Phase 2 as the next action.
+4. If the same threshold trips on multiple runs across 7 days, open the Phase 2 or Phase 3 planning path instead of tuning in place.
+
+Decision note:
+
+- Keep the smallest change that restores the target metric and record the reason in the run sheet.
+
+## 13) Artifact-to-Worksheet Field Mapping
+
+Use this section when filling the WebSocket and Admin API worksheets from helper-generated artifacts.
+
+### WebSocket worksheet mapping
+
+Artifact sources:
+
+- `artifacts/<timestamp>/ws-chat-report.json`
+- `artifacts/<timestamp>/ws-chat-summary.txt`
+
+Field mapping:
+
+- Worksheet `Tool used` -> `artillery` (or `run-loadtests.ps1 -Scenario ws`)
+- Worksheet `Raw results location` -> artifact folder path for this run
+- Worksheet `Concurrency tiers reached` -> `ws-chat.yml` phases (`warmup-10`, `sustain-100`, `sustain-500`, `sustain-1000`)
+- Worksheet `Error rate` -> `vusers.failure_rate_percent` from `ws-chat-summary.txt`
+- Worksheet `Dropped connections` -> derive from `vusers.created - vusers.completed` (use `vusers.failed` as failure signal)
+- Worksheet `p95 message latency` -> first available latency line under `summary.*` in `ws-chat-summary.txt` that includes `p95=...`
+- Worksheet `p50 message latency` -> first available latency line under `summary.*` in `ws-chat-summary.txt` that includes `median=...`
+
+Notes:
+
+- The exact `summary.*` metric names in Artillery output can vary by engine/plugins.
+- If no latency summary field is emitted, record this explicitly and use server-side latency metrics as fallback evidence.
+
+### Admin API worksheet mapping
+
+Artifact sources:
+
+- `artifacts/<timestamp>/admin-api-summary.json`
+- `artifacts/<timestamp>/admin-api-output.txt`
+- `artifacts/<timestamp>/admin-api-endpoint-summary.txt`
+
+Field mapping:
+
+- Worksheet `Tool used` -> `k6` (or `run-loadtests.ps1 -Scenario admin`)
+- Worksheet `Raw results location` -> artifact folder path for this run
+- Worksheet `/api/admin/overview` p95 -> `overview` row `P95Ms` in `admin-api-endpoint-summary.txt`
+- Worksheet `/api/admin/analytics` p95 -> `analytics` row `P95Ms` in `admin-api-endpoint-summary.txt`
+- Worksheet `/api/admin/chat-log` p95 -> `chat_log` row `P95Ms` in `admin-api-endpoint-summary.txt`
+- Worksheet `/api/admin/users` p95 -> `users` row `P95Ms` in `admin-api-endpoint-summary.txt`
+- Worksheet `Error rate` -> `Global failure rate` in `admin-api-endpoint-summary.txt`
+- Worksheet `Slowest endpoint` -> endpoint row with highest `P95Ms`
+
+Notes:
+
+- Keep the raw `admin-api-summary.json` file for audits and re-parsing.
+- If an endpoint row is missing, treat as test/data issue and re-run with full endpoint coverage.