linearvoid/docs/DEVELOPMENT.md

# DEVELOPMENT.md — Linear Void Remaining Tasks

## Test Fixtures (COMPLETE)

- [x] Create test media fixtures in `backend/playout/testdata/`:
  - `valid_720p30.mp4` — small (1-2s) H.264/AAC compliant file
  - `invalid_h265.mp4` — wrong codec
  - `invalid_24fps.mp4` — wrong framerate
  - Generate via `ffmpeg` commands documented in test file headers
  - Windows dev path: `C:\Users\zwits\AppData\Local\Microsoft\WinGet\Packages\Gyan.FFmpeg_Microsoft.Winget.Source_8wekyb3d8bbwe\ffmpeg-9.0-full_build\bin\ffmpeg.exe`

- [x] Create test playlist fixtures in `backend/playout/testdata/`:
  - `valid_playlist.txt`, `empty_playlist.txt`, `comments_only.txt`, `mixed_valid_invalid.txt`

- [x] Document ffmpeg fixture-generation commands in `playlist_test.go` header

## User Frontend Tests (PARTIALLY COMPLETE)

- [x] Video player behavior (play, pause, seek, buffering, error handling) - HLS player startup, native/fallback, URL resolution covered
- [x] Chat functionality (send/receive messages, presence, rate limiting, moderation) - send, receive, presence join/leave, DM flow, reconnect covered
- [ ] State management and data flow
- [ ] Responsive design and layout across different screen sizes
- [ ] Accessibility testing (keyboard navigation, screen reader support)
- [x] Integration with backend APIs (mocking API responses for testing)
- [ ] Test coverage for all critical frontend components and interactions
- [ ] Add code coverage reporting and enforce minimum coverage threshold in CI/CD pipeline
- [ ] Document test cases and expected outcomes

## Admin Frontend Tests (PARTIALLY COMPLETE)

- [x] Playlist management UI (add/remove videos, reorder, normalize) - save/error handling, sortable rendering
- [ ] User moderation tools (mute/ban/report users, view moderation logs)
- [x] Metrics and monitoring dashboards (Prometheus/Grafana integration) - overview + analytics rendering
- [ ] State management and data flow
- [ ] Responsive design and layout across different screen sizes
- [ ] Accessibility testing (keyboard navigation, screen reader support)
- [x] Integration with backend APIs (mocking API responses for testing)
- [ ] Test coverage for all critical admin UI components and interactions
- [ ] Add code coverage reporting and enforce minimum coverage threshold in CI/CD pipeline
- [ ] Document test cases and expected outcomes

## Cloudflare R2 Integration Test (PARTIALLY COMPLETE)

- [x] Upload test HLS segments and manifests to R2 bucket (SyncToR2 with mock, verified bucket/key/body)
- [x] Verify correct cache headers are applied (immutable segments via segmentCacheControl, short-lived manifests via manifestCacheControl)
- [ ] Test retrieval of segments and manifests from R2 via Cloudflare edge
- [ ] Validate cache hit/miss behavior and performance metrics
- [x] Ensure integration test can run in CI/CD pipeline with proper mocking/stubbing of R2 API (putObject mock, no network calls)
- [x] Document test cases and expected outcomes (TestSyncToR2 + TestSyncToR2_ManifestCacheHeaders + TestSyncToR2_NoCacheHeadersForUnknownExt)
- [x] Document required stubs or mocks (putObject variable injection pattern)

## Cloudflare Configuration (manual — Cloudflare dashboard)

> Evidence tracker: `docs/operations/cloudflare-uptimerobot-checklist.md`
> Cloudflare Pages deploy workflow created (`.gitea/workflows/pages-deploy.yml`); setup docs written (`docs/operations/cloudflare-pages-setup.md`, `docs/operations/gitea-cloudflare.pages.md`).

- [ ] Cloudflare Pages: create project in dashboard, link custom domain, or deploy via Gitea Actions Direct Upload
- [ ] Tiered Cache: enable Smart Tiered Cache, validate `Cf-Cache-Status: HIT` from upper tier
- [ ] WAF: enable OWASP Managed Ruleset, set to Simulate → review 48h → Block, enable Bot Fight Mode
- [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action
- [ ] HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2
- [ ] HTTPS/WSS: enforce at edge (Cloudflare) and origin (nginx), redirect HTTP → HTTPS

## External Services

> Evidence tracker: `docs/operations/cloudflare-uptimerobot-checklist.md`

- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword: `#EXTM3U`, 5min)
- [ ] Alert on 2 consecutive failures, notify via email + Discord/Slack webhook

## Production Deployment (Hetzner — not provisioned)

- [ ] Provision Hetzner CX22 (2 vCPU, 4 GB, 40 GB) in Ashburn VA, Ubuntu 24.04
- [ ] Install Docker Engine + Compose plugin, open firewall: 80/443 (public), 2375 (runner IP only)
- [ ] Configure Docker daemon for remote TLS access (`-H tcp://0.0.0.0:2376` with certs)
- [ ] Create `/srv/linearvoid/data/{playlist,media,hls,chat,slate}` directories
- [ ] Add `CI_PROD_*` secrets to Gitea: DOCKER_HOST, R2 keys, admin password, DB password, Grafana password
- [ ] Trigger CD workflow with `DOCKER_HOST` pointing to Hetzner IP
- [ ] Verify all 11 containers start healthy, check `/health`, `/hls/playlist.m3u8`, WebSocket
- [ ] Update Cloudflare A record `linearvoid.tv` → Hetzner IP, SSL Full (strict), test HTTPS + WSS
- [ ] Smoke test: viewer SPA loads, HLS plays, chat connects, admin login → upload → playlist → playout
- [ ] Set up Prometheus alerting notifications + UptimeRobot on production URL
- [ ] Run `scripts/backup-db.sh` + `scripts/verify-backups.sh` on production host
- [ ] Document production IP, SSH access, rollback procedure, scaling thresholds

## Scaling & Performance (planning only)

- [ ] Tune manifest TTL: measure `Cf-Cache-Status` distribution, adjust to 0.5× segment duration
- [ ] Measure TTFB: WebPageTest from 3+ US regions, target <4s warm cache, document baseline
- [ ] Load-test WebSocket: `artillery` or Go script, ramp 10→100→500→1000 connections, measure p95 latency + errors (scaffolding complete; benchmark execution pending)
- [ ] Benchmark admin API: `wrk`/`k6` against `/api/admin/*`, target p95 <1s, identify slow queries (scaffolding complete; benchmark execution pending)
- [x] Load-test scaffolding: ws-chat.yml, admin-api.js, run-loadtests.ps1, summarizers, README
- [x] Cache-hit ratio exporter (scripts/cache-hit-ratio.js + .ps1)
- [x] Saturation alerts: chat p95 latency, Redis memory usage, cache-hit ratio
- [ ] Track cache hit ratio: Cloudflare GraphQL → Grafana panel, alert <90%
- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min (chat p95, Redis memory, cache-hit already done)
- [ ] Phase 2 plan: document boundaries (playout CPU, chat latency, DB state, monitoring), estimate per-service resources
- [ ] Phase 3 plan: research second Hetzner location, design warm standby + DNS failover + DB/Redis replication

## Streaming Pipeline (2026-08-10 audit)

Gaps between STREAMING.md contract and playlist.go/main.ts implementation:

- [ ] **Continuous R2 sync during playout (HIGH)**: `syncLatestToR2()` runs only after FFmpeg exits. Needs per-segment upload via filesystem watch or FFmpeg progress pipe.
- [ ] **Globally unique segment naming (HIGH)**: Add run/session UUID + monotonic sequence. Currently `strftime` only — collisions possible.
- [ ] **Atomic manifest publication (MEDIUM)**: Upload to temp key then atomically replace `live/playlist.m3u8`.
- [ ] **Manifest Cache-Control alignment (LOW)**: Add `must-revalidate` and `s-maxage`/`max-age` split.
- [ ] **CORS headers on HLS responses (MEDIUM)**: Add `Access-Control-Allow-Origin` to Worker or R2 custom domain.
- [ ] **Player error recovery (MEDIUM)**: Add `hls.on(Hls.Events.ERROR, ...)` with retries, drift, reconnecting state.
- [ ] **Health endpoint detail (LOW)**: Add active item, playlist revision, run ID, segment info, R2 pub timestamp.
- [x] Source media contract + validation — implemented
- [x] fMP4 HLS packaging (6s, 24 window) — implemented
- [x] Slate fallback + recovery — implemented
- [x] R2 cache headers (immutable segments, short-lived manifest) — implemented
- [x] Cloudflare Worker immutable `.m4s` headers — implemented