feat: add metrics, observability, test fixtures, and scaling playbook artifacts

docs/DEVELOPMENT.md

@@ -1,49 +1,53 @@
 # DEVELOPMENT.md — Linear Void Remaining Tasks
 
-## Test Fixtures (BLOCKED — requires ffmpeg)
+## Test Fixtures (COMPLETE)
 
-- [ ] Create test media fixtures in `backend/playout/testdata/`:
+- [x] Create test media fixtures in `backend/playout/testdata/`:
   - `valid_720p30.mp4` — small (1-2s) H.264/AAC compliant file
   - `invalid_h265.mp4` — wrong codec
   - `invalid_24fps.mp4` — wrong framerate
   - Generate via `ffmpeg` commands documented in test file headers
+  - Windows dev path: `C:\Users\zwits\AppData\Local\Microsoft\WinGet\Packages\Gyan.FFmpeg_Microsoft.Winget.Source_8wekyb3d8bbwe\ffmpeg-9.0-full_build\bin\ffmpeg.exe`
 
-- [ ] Create test playlist fixtures in `backend/playout/testdata/`:
+- [x] Create test playlist fixtures in `backend/playout/testdata/`:
   - `valid_playlist.txt`, `empty_playlist.txt`, `comments_only.txt`, `mixed_valid_invalid.txt`
 
-## User Frontend Tests
+- [x] Document ffmpeg fixture-generation commands in `playlist_test.go` header
 
-- [ ] Video player behavior (play, pause, seek, buffering, error handling)
-- [ ] Chat functionality (send/receive messages, presence, rate limiting, moderation)
+## User Frontend Tests (PARTIALLY COMPLETE)
+
+- [x] Video player behavior (play, pause, seek, buffering, error handling) - HLS player startup, native/fallback, URL resolution covered
+- [x] Chat functionality (send/receive messages, presence, rate limiting, moderation) - send, receive, presence join/leave, DM flow, reconnect covered
 - [ ] State management and data flow
 - [ ] Responsive design and layout across different screen sizes
 - [ ] Accessibility testing (keyboard navigation, screen reader support)
-- [ ] Integration with backend APIs (mocking API responses for testing)
+- [x] Integration with backend APIs (mocking API responses for testing)
 - [ ] Test coverage for all critical frontend components and interactions
 - [ ] Add code coverage reporting and enforce minimum coverage threshold in CI/CD pipeline
 - [ ] Document test cases and expected outcomes
 
-## Admin Frontend Tests
+## Admin Frontend Tests (PARTIALLY COMPLETE)
 
-- [ ] Playlist management UI (add/remove videos, reorder, normalize)
+- [x] Playlist management UI (add/remove videos, reorder, normalize) - save/error handling, sortable rendering
 - [ ] User moderation tools (mute/ban/report users, view moderation logs)
-- [ ] Metrics and monitoring dashboards (Prometheus/Grafana integration)
+- [x] Metrics and monitoring dashboards (Prometheus/Grafana integration) - overview + analytics rendering
 - [ ] State management and data flow
 - [ ] Responsive design and layout across different screen sizes
 - [ ] Accessibility testing (keyboard navigation, screen reader support)
-- [ ] Integration with backend APIs (mocking API responses for testing)
+- [x] Integration with backend APIs (mocking API responses for testing)
 - [ ] Test coverage for all critical admin UI components and interactions
 - [ ] Add code coverage reporting and enforce minimum coverage threshold in CI/CD pipeline
 - [ ] Document test cases and expected outcomes
 
-## Cloudflare R2 Integration Test
+## Cloudflare R2 Integration Test (PARTIALLY COMPLETE)
 
-- [ ] Upload test HLS segments and manifests to R2 bucket
-- [ ] Verify correct cache headers are applied (immutable segments, TTLs)
+- [x] Upload test HLS segments and manifests to R2 bucket (SyncToR2 with mock, verified bucket/key/body)
+- [x] Verify correct cache headers are applied (immutable segments via segmentCacheControl, short-lived manifests via manifestCacheControl)
 - [ ] Test retrieval of segments and manifests from R2 via Cloudflare edge
 - [ ] Validate cache hit/miss behavior and performance metrics
-- [ ] Ensure integration test can run in CI/CD pipeline with proper mocking/stubbing of R2 API
-- [ ] Document test cases and expected outcomes
+- [x] Ensure integration test can run in CI/CD pipeline with proper mocking/stubbing of R2 API (putObject mock, no network calls)
+- [x] Document test cases and expected outcomes (TestSyncToR2 + TestSyncToR2_ManifestCacheHeaders + TestSyncToR2_NoCacheHeadersForUnknownExt)
+- [x] Document required stubs or mocks (putObject variable injection pattern)
 
 ## Cloudflare Configuration (manual — Cloudflare dashboard)
 
@@ -83,9 +87,9 @@
 
 - [ ] Tune manifest TTL: measure `Cf-Cache-Status` distribution, adjust to 0.5× segment duration
 - [ ] Measure TTFB: WebPageTest from 3+ US regions, target <4s warm cache, document baseline
-- [ ] Load-test WebSocket: `artillery` or Go script, ramp 10→100→500→1000 connections, measure p95 latency + errors
-- [ ] Benchmark admin API: `wrk`/`k6` against `/api/admin/*`, target p95 <1s, identify slow queries
+- [ ] Load-test WebSocket: `artillery` or Go script, ramp 10→100→500→1000 connections, measure p95 latency + errors (scaffolding complete; benchmark execution pending)
+- [ ] Benchmark admin API: `wrk`/`k6` against `/api/admin/*`, target p95 <1s, identify slow queries (scaffolding complete; benchmark execution pending)
 - [ ] Track cache hit ratio: Cloudflare GraphQL → Grafana panel, alert <90%
-- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, chat p95 >1s/5min, CPU >80%/10min
+- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, chat p95 >1s/5min, CPU >80%/10min, Redis memory >85%/10min
 - [ ] Phase 2 plan: document boundaries (playout CPU, chat latency, DB state, monitoring), estimate per-service resources
 - [ ] Phase 3 plan: research second Hetzner location, design warm standby + DNS failover + DB/Redis replication