feat: add metrics, observability, test fixtures, and scaling playbook artifacts

docs/DEPLOYMENT.md

@@ -8,6 +8,7 @@
   - [Production Target (Hetzner)](#production-target-hetzner)
     - [Planned Infrastructure](#planned-infrastructure)
     - [Cutover Plan](#cutover-plan)
+    - [Production Preflight](#production-preflight)
     - [Firewall / Security Group](#firewall--security-group)
     - [Pipeline Readiness](#pipeline-readiness)
   - [Target Topology](#target-topology)
@@ -64,6 +65,19 @@ Linear Void runs in two separate environments:
 6. Enable HTTPS, WAF, cache rules on production zone
 7. Decommission test lab as public target
 
+### Production Preflight
+
+Before the cutover in the plan above, treat the following as the minimum readiness gate:
+
+1. Verify the Hetzner host is reachable over SSH and Docker remote API TLS is working for the Gitea runner.
+2. Confirm all required production secrets are present under the `CI_PROD_*` prefix.
+3. Confirm `linearvoid.tv`, `www.linearvoid.tv`, and `media.linearvoid.tv` DNS targets are correct for the intended phase.
+4. Complete the manual Cloudflare/UptimeRobot checklist and attach evidence in `docs/operations/cloudflare-uptimerobot-checklist.md`.
+5. Run the production smoke path end to end: viewer SPA, HLS playback, chat connect, admin login, upload, playlist save, and playout health.
+6. Verify rollback is still possible before the final DNS switch.
+
+Readiness is not complete until the checklist above and the manual evidence tracker are both satisfied.
+
 ### Firewall / Security Group
 
 | Port | Source                | Purpose                        |

docs/DEVELOPMENT.md

@@ -1,49 +1,53 @@
 # DEVELOPMENT.md — Linear Void Remaining Tasks
 
-## Test Fixtures (BLOCKED — requires ffmpeg)
+## Test Fixtures (COMPLETE)
 
-- [ ] Create test media fixtures in `backend/playout/testdata/`:
+- [x] Create test media fixtures in `backend/playout/testdata/`:
   - `valid_720p30.mp4` — small (1-2s) H.264/AAC compliant file
   - `invalid_h265.mp4` — wrong codec
   - `invalid_24fps.mp4` — wrong framerate
   - Generate via `ffmpeg` commands documented in test file headers
+  - Windows dev path: `C:\Users\zwits\AppData\Local\Microsoft\WinGet\Packages\Gyan.FFmpeg_Microsoft.Winget.Source_8wekyb3d8bbwe\ffmpeg-9.0-full_build\bin\ffmpeg.exe`
 
-- [ ] Create test playlist fixtures in `backend/playout/testdata/`:
+- [x] Create test playlist fixtures in `backend/playout/testdata/`:
   - `valid_playlist.txt`, `empty_playlist.txt`, `comments_only.txt`, `mixed_valid_invalid.txt`
 
-## User Frontend Tests
+- [x] Document ffmpeg fixture-generation commands in `playlist_test.go` header
 
-- [ ] Video player behavior (play, pause, seek, buffering, error handling)
-- [ ] Chat functionality (send/receive messages, presence, rate limiting, moderation)
+## User Frontend Tests (PARTIALLY COMPLETE)
+
+- [x] Video player behavior (play, pause, seek, buffering, error handling) - HLS player startup, native/fallback, URL resolution covered
+- [x] Chat functionality (send/receive messages, presence, rate limiting, moderation) - send, receive, presence join/leave, DM flow, reconnect covered
 - [ ] State management and data flow
 - [ ] Responsive design and layout across different screen sizes
 - [ ] Accessibility testing (keyboard navigation, screen reader support)
-- [ ] Integration with backend APIs (mocking API responses for testing)
+- [x] Integration with backend APIs (mocking API responses for testing)
 - [ ] Test coverage for all critical frontend components and interactions
 - [ ] Add code coverage reporting and enforce minimum coverage threshold in CI/CD pipeline
 - [ ] Document test cases and expected outcomes
 
-## Admin Frontend Tests
+## Admin Frontend Tests (PARTIALLY COMPLETE)
 
-- [ ] Playlist management UI (add/remove videos, reorder, normalize)
+- [x] Playlist management UI (add/remove videos, reorder, normalize) - save/error handling, sortable rendering
 - [ ] User moderation tools (mute/ban/report users, view moderation logs)
-- [ ] Metrics and monitoring dashboards (Prometheus/Grafana integration)
+- [x] Metrics and monitoring dashboards (Prometheus/Grafana integration) - overview + analytics rendering
 - [ ] State management and data flow
 - [ ] Responsive design and layout across different screen sizes
 - [ ] Accessibility testing (keyboard navigation, screen reader support)
-- [ ] Integration with backend APIs (mocking API responses for testing)
+- [x] Integration with backend APIs (mocking API responses for testing)
 - [ ] Test coverage for all critical admin UI components and interactions
 - [ ] Add code coverage reporting and enforce minimum coverage threshold in CI/CD pipeline
 - [ ] Document test cases and expected outcomes
 
-## Cloudflare R2 Integration Test
+## Cloudflare R2 Integration Test (PARTIALLY COMPLETE)
 
-- [ ] Upload test HLS segments and manifests to R2 bucket
-- [ ] Verify correct cache headers are applied (immutable segments, TTLs)
+- [x] Upload test HLS segments and manifests to R2 bucket (SyncToR2 with mock, verified bucket/key/body)
+- [x] Verify correct cache headers are applied (immutable segments via segmentCacheControl, short-lived manifests via manifestCacheControl)
 - [ ] Test retrieval of segments and manifests from R2 via Cloudflare edge
 - [ ] Validate cache hit/miss behavior and performance metrics
-- [ ] Ensure integration test can run in CI/CD pipeline with proper mocking/stubbing of R2 API
-- [ ] Document test cases and expected outcomes
+- [x] Ensure integration test can run in CI/CD pipeline with proper mocking/stubbing of R2 API (putObject mock, no network calls)
+- [x] Document test cases and expected outcomes (TestSyncToR2 + TestSyncToR2_ManifestCacheHeaders + TestSyncToR2_NoCacheHeadersForUnknownExt)
+- [x] Document required stubs or mocks (putObject variable injection pattern)
 
 ## Cloudflare Configuration (manual — Cloudflare dashboard)
 
@@ -83,9 +87,9 @@
 
 - [ ] Tune manifest TTL: measure `Cf-Cache-Status` distribution, adjust to 0.5× segment duration
 - [ ] Measure TTFB: WebPageTest from 3+ US regions, target <4s warm cache, document baseline
-- [ ] Load-test WebSocket: `artillery` or Go script, ramp 10→100→500→1000 connections, measure p95 latency + errors
-- [ ] Benchmark admin API: `wrk`/`k6` against `/api/admin/*`, target p95 <1s, identify slow queries
+- [ ] Load-test WebSocket: `artillery` or Go script, ramp 10→100→500→1000 connections, measure p95 latency + errors (scaffolding complete; benchmark execution pending)
+- [ ] Benchmark admin API: `wrk`/`k6` against `/api/admin/*`, target p95 <1s, identify slow queries (scaffolding complete; benchmark execution pending)
 - [ ] Track cache hit ratio: Cloudflare GraphQL → Grafana panel, alert <90%
-- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, chat p95 >1s/5min, CPU >80%/10min
+- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, chat p95 >1s/5min, CPU >80%/10min, Redis memory >85%/10min
 - [ ] Phase 2 plan: document boundaries (playout CPU, chat latency, DB state, monitoring), estimate per-service resources
 - [ ] Phase 3 plan: research second Hetzner location, design warm standby + DNS failover + DB/Redis replication

docs/operations/cloudflare-uptimerobot-checklist.md

@@ -2,6 +2,14 @@
 
 Purpose: execute and record all manual dashboard tasks that cannot be automated from repository workflows.
 
+Recommended execution order:
+
+1. Bind `media.linearvoid.tv` and confirm the HLS origin is separate from the app origin.
+2. Enable Pages, Tiered Cache, and cache behavior for the media zone.
+3. Apply WAF, bot, rate limiting, HTTPS, and HSTS settings.
+4. Verify monitor targets and alert delivery in UptimeRobot.
+5. Capture evidence and fill in the sign-off block.
+
 ## Status Legend
 
 - [ ] Pending
@@ -18,6 +26,7 @@ Evidence to capture:
 
 - Screenshot of Pages deployment settings
 - Deployment URL and latest deployment timestamp
+- Confirmed production build output path in the Pages project
 
 ## 2) Tiered Cache Enablement and Validation
 
@@ -37,6 +46,7 @@ Evidence to capture:
 
 - Headers from first request (expected MISS)
 - Headers from repeated request (expected HIT)
+- Screenshot or export showing Smart Tiered Cache enabled
 
 ## 3) WAF and Bot Protection
 
@@ -50,6 +60,7 @@ Evidence to capture:
 
 - WAF ruleset status screenshot
 - Count of blocked/challenged events after cutover
+- Note on any false positives and the final allow/exception rules applied
 
 ## 4) Rate Limiting Rules
 
@@ -67,6 +78,7 @@ Evidence to capture:
 
 - Rule definitions screenshot
 - Test result notes (challenge seen / no lockout)
+- Keep a copy of the test source IP and timestamps used during validation
 
 ## 5) HTTPS and WSS Hardening
 
@@ -89,6 +101,7 @@ Evidence to capture:
 
 - Response headers showing HSTS
 - Successful browser WebSocket connect in devtools
+- TLS mode screenshot from Cloudflare showing Full (strict)
 
 ## 6) UptimeRobot Monitors
 
@@ -101,6 +114,7 @@ Evidence to capture:
 
 - Monitor dashboard screenshot
 - Test alert delivery confirmation
+- List of recipients/webhook destinations verified during the alert test
 
 ## 7) Sign-Off
 
@@ -109,6 +123,15 @@ Evidence to capture:
 - [ ] TODO items updated to done
 - [ ] Changelog entry added for completion batch
 
+Completion summary:
+
+- Pages build URL:
+- Cache validation result:
+- WAF/rate-limit status:
+- HTTPS/HSTS status:
+- UptimeRobot monitor status:
+- Evidence folder or note link:
+
 Sign-off fields:
 
 - Owner:

docs/operations/scaling-performance-playbook.md

@@ -52,6 +52,38 @@ Acceptance:
 - Warm-cache median < 4s in all tested regions.
 - Outliers documented with likely cause.
 
+### TTFB Baseline Worksheet
+
+Use one row per region and keep the raw WebPageTest result URL with the summary so later runs can be compared against the same evidence.
+
+- US East (Virginia)
+  - Date/time:
+  - Test URL: <https://linearvoid.tv>
+  - WebPageTest result URL:
+  - Median TTFB:
+  - p95 TTFB:
+  - Notes:
+- US Central
+  - Date/time:
+  - Test URL: <https://linearvoid.tv>
+  - WebPageTest result URL:
+  - Median TTFB:
+  - p95 TTFB:
+  - Notes:
+- US West (California/Oregon)
+  - Date/time:
+  - Test URL: <https://linearvoid.tv>
+  - WebPageTest result URL:
+  - Median TTFB:
+  - p95 TTFB:
+  - Notes:
+
+Suggested capture notes:
+
+- Use repeat view for the warm-cache measurement.
+- Keep the same browser profile and test URL across regions.
+- Note any CDN cache-rule changes or deploys that occurred between samples.
+
 ## 3) WebSocket Load-Test Plan
 
 Goal: characterize chat service behavior across 10 -> 100 -> 500 -> 1000 concurrent clients.
@@ -80,6 +112,26 @@ Acceptance:
 - p95 chat latency <= 1s at target concurrency tier.
 - error rate <= 1% during steady state.
 
+### WebSocket Load-Test Worksheet
+
+- Date/time:
+- Environment:
+- Commit SHA / image tag:
+- Test URL:
+- Tool used:
+- Scenario duration:
+- Concurrency tiers reached:
+- p50 message latency:
+- p95 message latency:
+- Error rate:
+- Dropped connections:
+- CPU/memory notes for chat:
+- CPU/memory notes for redis:
+- CPU/memory notes for nginx:
+- Raw results location:
+- Conclusion:
+- Next action:
+
 ## 4) Admin API Benchmark Plan
 
 Goal: measure admin API p95 and identify slow endpoints.
@@ -108,6 +160,26 @@ Acceptance:
 - p95 < 1s for primary admin endpoints.
 - slow endpoint list with remediation notes.
 
+### Admin API Benchmark Worksheet
+
+- Date/time:
+- Environment:
+- Commit SHA / image tag:
+- Test URL:
+- Tool used:
+- Scenario duration:
+- Concurrency / virtual users:
+- `/api/admin/overview` p95:
+- `/api/admin/analytics` p95:
+- `/api/admin/chat-log` p95:
+- `/api/admin/users` p95:
+- Error rate:
+- Slowest endpoint:
+- Cache or DB notes:
+- Raw results location:
+- Conclusion:
+- Next action:
+
 ## 5) Cache Hit Ratio Tracking Plan
 
 Goal: track Cloudflare cache hit ratio and alert when below threshold.
@@ -119,10 +191,34 @@ Implementation outline:
 3. Add Grafana panel with 5m/1h/24h windows.
 4. Alert when hit ratio < 90% for 15m.
 
+Reference exporter:
+
+- `scripts/cache-hit-ratio.js`
+- `scripts/cache-hit-ratio.ps1` for Windows scheduled tasks / manual runs
+- Example run: `CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_TAG=... CLOUDFLARE_HOSTNAME=media.linearvoid.tv node scripts/cache-hit-ratio.js`
+- Windows example: `pwsh -File scripts/cache-hit-ratio.ps1`
+- Intended output: Prometheus text format on stdout, suitable for cron, a Node wrapper, or a small scrape endpoint.
+
 Metric target:
 
 - `cache_hit_ratio >= 0.90`
 
+### Cache Hit Ratio Worksheet
+
+- Date/time:
+- Environment:
+- Commit SHA / image tag:
+- Hostname:
+- Window minutes:
+- Cloudflare zone tag:
+- Export command:
+- Raw exporter output location:
+- Hit requests:
+- Total requests:
+- Cache hit ratio:
+- Threshold met:
+- Notes:
+
 ## 6) Capacity Trigger Rules
 
 Define operational triggers:
@@ -130,7 +226,7 @@ Define operational triggers:
 - Cache miss ratio > 10% for 15m
 - Chat p95 latency > 1s for 5m
 - Host CPU > 80% for 10m
-- Redis memory > 85% sustained
+- Redis memory usage ratio (`linearvoid_chat_redis_memory_usage_ratio`) > 0.85 for 10m
 
 Action mapping:
 
@@ -138,6 +234,22 @@ Action mapping:
 - Level 2: scale service resources (CPU/RAM)
 - Level 3: activate standby planning path
 
+### Capacity Trigger Review
+
+- Date/time:
+- Environment:
+- Commit SHA / image tag:
+- Trigger hit:
+- Threshold value:
+- Observed value:
+- Duration above threshold:
+- Level selected:
+- Immediate action taken:
+- Services involved:
+- Follow-up owner:
+- Raw metrics / alert link:
+- Notes:
+
 ## 7) Phase 2 Service Separation Plan
 
 Objective: separate bottlenecked services while keeping simple ops.
@@ -160,6 +272,24 @@ Exit criteria for Phase 2:
 - repeated trigger events over 7 days
 - capacity saturation despite tuning
 
+### Phase 2 Boundary Worksheet
+
+- Date/time:
+- Environment:
+- Commit SHA / image tag:
+- Trigger history summary:
+- Repeated trigger count over 7 days:
+- Primary bottleneck:
+- Proposed split:
+- `playout` target host size:
+- `chat` target host size:
+- `db` target host size:
+- Monitoring separation needed:
+- Data/state migration notes:
+- Implementation owner:
+- Review decision:
+- Next action:
+
 ## 8) Phase 3 Warm Standby Plan
 
 Objective: second-region resilience with DNS failover.
@@ -178,8 +308,24 @@ Design:
 
 Failover drill requirements:
 
-- periodic simulated cutover
-- measured RTO/RPO logged in incident records
+### Phase 3 Warm Standby Worksheet
+
+- Date/time:
+- Environment:
+- Commit SHA / image tag:
+- Primary region:
+- Standby region:
+- DNS failover mode:
+- Redis strategy:
+- Postgres strategy:
+- Health check endpoints:
+- Simulated cutover performed:
+- Measured RTO:
+- Measured RPO:
+- Open risks:
+- Implementation owner:
+- Review decision:
+- Notes:
 
 ## 9) Data Capture Template
 
@@ -201,3 +347,96 @@ For each benchmark run, capture:
 5. Trigger tuning
 6. Phase 2 boundary decision
 7. Phase 3 standby design review
+
+## 11) Benchmark Run Sheet
+
+Use the same worksheet for every baseline or re-test so results stay comparable.
+
+Preconditions:
+
+- Confirm the target environment and commit SHA/image tag.
+- Confirm Cloudflare cache rules are active when testing TTFB or manifest TTL.
+- Confirm the chat, admin, and playout services are healthy before load testing.
+
+Run template:
+
+| Item                       | Value |
+| -------------------------- | ----- |
+| Date/time                  |       |
+| Environment                |       |
+| Commit SHA / image tag     |       |
+| Test scenario              |       |
+| Concurrency / sample count |       |
+| Command or tool            |       |
+| Raw results location       |       |
+| Conclusion                 |       |
+| Next action                |       |
+
+Suggested command mapping:
+
+- TTFB baseline: WebPageTest repeat-view run against `https://linearvoid.tv`.
+- WebSocket profile: `npx artillery run scripts/loadtest/ws-chat.yml` or the Go harness referenced above.
+- Admin API benchmark: `k6 run scripts/loadtest/admin-api.js` or `wrk -t4 -c100 -d60s http://<host>/api/admin/overview`.
+
+## 12) Scaling Decision Gate
+
+Treat the following as the default review gate after any benchmark batch:
+
+1. If warm-cache TTFB meets target but cache hit ratio falls below 90%, tune cache behavior before increasing capacity.
+2. If chat or admin latency exceeds target while CPU and memory remain low, prioritize query or handler optimization.
+3. If CPU, memory, or Redis saturation appears first, treat the service boundary guidance in Phase 2 as the next action.
+4. If the same threshold trips on multiple runs across 7 days, open the Phase 2 or Phase 3 planning path instead of tuning in place.
+
+Decision note:
+
+- Keep the smallest change that restores the target metric and record the reason in the run sheet.
+
+## 13) Artifact-to-Worksheet Field Mapping
+
+Use this section when filling the WebSocket and Admin API worksheets from helper-generated artifacts.
+
+### WebSocket worksheet mapping
+
+Artifact sources:
+
+- `artifacts/<timestamp>/ws-chat-report.json`
+- `artifacts/<timestamp>/ws-chat-summary.txt`
+
+Field mapping:
+
+- Worksheet `Tool used` -> `artillery` (or `run-loadtests.ps1 -Scenario ws`)
+- Worksheet `Raw results location` -> artifact folder path for this run
+- Worksheet `Concurrency tiers reached` -> `ws-chat.yml` phases (`warmup-10`, `sustain-100`, `sustain-500`, `sustain-1000`)
+- Worksheet `Error rate` -> `vusers.failure_rate_percent` from `ws-chat-summary.txt`
+- Worksheet `Dropped connections` -> derive from `vusers.created - vusers.completed` (use `vusers.failed` as failure signal)
+- Worksheet `p95 message latency` -> first available latency line under `summary.*` in `ws-chat-summary.txt` that includes `p95=...`
+- Worksheet `p50 message latency` -> first available latency line under `summary.*` in `ws-chat-summary.txt` that includes `median=...`
+
+Notes:
+
+- The exact `summary.*` metric names in Artillery output can vary by engine/plugins.
+- If no latency summary field is emitted, record this explicitly and use server-side latency metrics as fallback evidence.
+
+### Admin API worksheet mapping
+
+Artifact sources:
+
+- `artifacts/<timestamp>/admin-api-summary.json`
+- `artifacts/<timestamp>/admin-api-output.txt`
+- `artifacts/<timestamp>/admin-api-endpoint-summary.txt`
+
+Field mapping:
+
+- Worksheet `Tool used` -> `k6` (or `run-loadtests.ps1 -Scenario admin`)
+- Worksheet `Raw results location` -> artifact folder path for this run
+- Worksheet `/api/admin/overview` p95 -> `overview` row `P95Ms` in `admin-api-endpoint-summary.txt`
+- Worksheet `/api/admin/analytics` p95 -> `analytics` row `P95Ms` in `admin-api-endpoint-summary.txt`
+- Worksheet `/api/admin/chat-log` p95 -> `chat_log` row `P95Ms` in `admin-api-endpoint-summary.txt`
+- Worksheet `/api/admin/users` p95 -> `users` row `P95Ms` in `admin-api-endpoint-summary.txt`
+- Worksheet `Error rate` -> `Global failure rate` in `admin-api-endpoint-summary.txt`
+- Worksheet `Slowest endpoint` -> endpoint row with highest `P95Ms`
+
+Notes:
+
+- Keep the raw `admin-api-summary.json` file for audits and re-parsing.
+- If an endpoint row is missing, treat as test/data issue and re-run with full endpoint coverage.