feat: add automatic media normalization, R2 retention cleanup, and seamless playout loops

.github/instructions/TODO.md

@@ -1,80 +1,90 @@
 # TODO
 
-> Completed tasks moved to [DONE](./DONE.md). Full remaining task list in [BACKLOG](./BACKLOG.md).
+> Completed tasks moved to [DONE](./DONE.md). Remaining tasks in [BACKLOG](./BACKLOG.md).
 
-## Plan
+## Enhancements
 
-### Phase 1: unblock playout test fixtures and lock down the source-media contract
+### Application configuration
 
-- [x] Create small compliant fixture media in `backend/playout/testdata/` for the validator and playout tests
-- [x] Create invalid fixture media for codec, framerate, and audio rejection paths
-- [x] Create playlist fixtures for empty, comment-only, mixed-validity, and valid playlists
-- [x] Document the ffmpeg fixture-generation commands in the relevant test headers
+- [ ] Configuration: create simple getter/setter for configuration values (e.g. `MATOMO_URL`, `MATOMO_SITE_ID`) in `backend/admin/config.go`
+- [ ] Configuration: expose configuration values to frontend via `/api/admin/config` endpoint
+- [ ] Configuration: make configuration values editable:
+  - [ ] HLS_SEGMENT_DURATION
+  - [ ] HLS_WINDOW_SIZE
+  - [ ] MATOMO_URL
+  - [ ] MATOMO_SITE_ID
 
-### Phase 2: finish frontend coverage for the live viewer experience
+### Admin frontend
 
-- [x] Add tests for HLS player startup, manifest resolution, and cross-origin/local fallback behavior
-- [x] Add tests for chat send/receive, presence updates, private DM flow, and reconnect behavior
-- [x] Add responsive-layout and accessibility checks for the viewer shell and chat sidebar
-- [x] Add coverage reporting and enforce a minimum threshold in CI
+- [ ] Playlist: show `needs_normalization` badge and block/queue playlist-add until `ready`
+- [ ] Playlist: show `ready` badge and allow playlist-add when `ready`
+- [ ] Extend `system.html` with a section for configuration settings on top of page (HLS segment duration, window size, Matomo URL/site ID)
+- [ ] Add functions to `admin.ts` to call `/api/admin/config` for getting and updating configuration values
 
-### Phase 3: finish frontend coverage for the admin experience
+### User frontend
 
-- [x] Add tests for playlist management, moderation flows, and dashboard rendering
-- [x] Add tests for API integration points with mocked backend responses
-- [x] Add responsive-layout and accessibility checks for admin screens
-- [x] Add coverage reporting and enforce a minimum threshold in CI
+- [ ] Player frontend: Embed JavaScript snippet in `index.html` to send pageview events to Matomo
 
-### Phase 4: validate the R2 and edge-delivery path
+### Chat user features
 
-- [x] Add an integration test for upload and retrieval of HLS objects through the R2 path
-- [x] Verify segment and manifest cache headers, including immutable media and live manifest freshness
-- [x] Validate edge cache hit/miss behavior with repeat requests
-- [x] Document any required stubs or mocks so the test can run in CI (putObject mock pattern documented in test)
+- [ ] Chat: add user registration and login flows
+- [ ] Chat: private DM history and search
+- [ ] Chat UI: add login/logout button, show username in chat, show/toggle user list, show/toggle online status, show typing indicator
+
+### Monitoring
+
+- [ ] Prometheus TSDB retention policy: delete metrics older than 30d
+
+## Setup phases remaining
 
 ### Phase 5: complete manual Cloudflare and external-service setup
 
 - [ ] Apply Cloudflare Pages, Tiered Cache, WAF, rate limiting, and HTTPS/HSTS settings in the dashboard
-- [ ] Create or verify UptimeRobot monitors for health and HLS manifest availability
+- [ ] Cloudflare Pages: link custom domain and cut over frontend hosting
+- [ ] Enable Smart Tiered Cache; validate `Cf-Cache-Status: HIT` from upper tier
+- [ ] WAF: enable OWASP Managed Ruleset (Simulate → review 48h → Block), enable Bot Fight Mode
+- [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action
+- [ ] HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2; enforce at edge and origin
+- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
+- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword `#EXTM3U`, 5min)
+- [ ] UptimeRobot: alert on 2 consecutive failures via email + Discord/Slack webhook
 - [ ] Capture evidence in `docs/operations/cloudflare-uptimerobot-checklist.md`
 
 ### Phase 6: rehearse production deployment end to end
 
-- [ ] Provision the Hetzner host and prepare the Docker/runtime baseline
-- [ ] Configure production secrets, remote deployment, and DNS cutover
-- [ ] Run health, HLS, chat, and admin smoke tests against production-like infrastructure
-- [ ] Verify backup creation, restore, and integrity checks on the production host
+- [ ] Provision Hetzner CX22 (2 vCPU, 4 GB, 40 GB) in Ashburn VA, Ubuntu 24.04
+- [ ] Install Docker Engine + Compose plugin; open firewall: 80/443 public, 2375 runner-only
+- [ ] Configure Docker daemon for remote TLS access (`-H tcp://0.0.0.0:2376` with certs)
+- [ ] Create `/srv/linearvoid/data/{playlist,media,hls,chat,slate}` directories
+- [ ] Add `CI_PROD_*` secrets to Gitea (DOCKER_HOST, R2 keys, admin password, DB password, Grafana password)
+- [ ] Trigger CD workflow with `DOCKER_HOST` pointing to Hetzner IP
+- [ ] Verify all 11 containers healthy: `/health`, `/hls/playlist.m3u8`, WebSocket
+- [ ] Update Cloudflare A record `linearvoid.tv` → Hetzner IP, SSL Full (strict); test HTTPS + WSS
+- [ ] Smoke test: viewer SPA loads, HLS plays, chat connects, admin login → upload → playlist → playout
+- [ ] Set up Prometheus alerting notifications + UptimeRobot on production URL
+- [ ] Run `scripts/backup-db.sh` + `scripts/verify-backups.sh` on production host
+- [ ] Document production IP, SSH access, rollback procedure, scaling thresholds
 
 ### Phase 7: baseline scaling and performance
 
-- [ ] Measure manifest TTFB and cache-hit distribution from multiple US regions
-- [x] Prepare WebSocket load-test scaffolding and summary artifacts (`ws-chat.yml`, `run-loadtests.ps1`, `summarize-artillery.ps1`)
-- [ ] Execute WebSocket throughput and chat-latency benchmarks under growing connection counts
-- [x] Prepare Admin API benchmark scaffolding and endpoint summary artifacts (`admin-api.js`, `run-loadtests.ps1`, `summarize-k6.ps1`)
-- [ ] Execute Admin API benchmarks and identify slow paths
-- [x] Add saturation thresholds to monitoring (chat p95 latency + Redis memory usage)
-- [x] Add cache-hit threshold alert operationalization to monitoring
+- [ ] Execute WebSocket load tests — ramp 10→100→500→1000 connections; measure p95 latency + errors
+- [ ] Execute admin API benchmarks — `wrk`/`k6` against `/api/admin/*`; target p95 <1s
+- [ ] Measure manifest TTFB and cache-hit distribution from multiple US regions (WebPageTest 3+ regions, <4s warm cache)
+- [ ] Tune manifest TTL to 0.5× segment duration; validate `Cf-Cache-Status` distribution
+- [ ] Track cache-hit ratio and R2 ops in production (Cloudflare GraphQL → Grafana panel, alert <90%)
+- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min
+- [ ] Plan Phase 2 service separation (playout CPU, chat latency, DB state, monitoring)
+- [ ] Plan Phase 3 multi-origin resilience (second Hetzner location, warm standby, DNS failover, DB/Redis replication)
 
-## Status Update (2026-08-10)
+### Phase 8: close out remaining test and validation gaps
 
-- [x] Phase 1 (test fixtures) complete.
-- [x] Phase 2 (viewer frontend): player startup, URL resolution, chat send/receive, reconnect, presence, DM flow, message rendering all covered (27 tests).
-- [x] Phase 3 (admin frontend): dashboard rendering, API integration, playlist management all covered (31 tests total).
-- [x] Phase 4 (R2 validation) partially complete: SyncToR2 cache headers + integration tests done; edge cache hit/miss remains manual.
-- [ ] Phase 5 (Cloudflare): Pages deploy workflow (`.gitea/workflows/pages-deploy.yml`) created; Pages/Workers setup docs written; manual dashboard tasks remain.
-- [ ] Phase 6 (production deployment) pending: Hetzner provisioning, secrets, DNS cutover, smoke tests, backup/restore.
-- [ ] Phase 7: Loadtest scaffolding complete; `run-loadtests.ps1` auto-generates ws-chat-summary.txt and admin-api-endpoint-summary.txt; live benchmark execution pending.
-- Version tagged 1.0.11; go vet clean.
+- [ ] Admin frontend: user moderation tools tests (mute/ban/report, view moderation logs)
+- [ ] Frontend: coverage for all critical user + admin components and interactions
+- [ ] Frontend: document test cases and expected outcomes
+- [ ] Backend R2: test retrieval of segments/manifests via Cloudflare edge
+- [ ] Backend R2: validate edge cache hit/miss behavior with repeat requests (`scripts/validate-edge-cache.sh`)
 
-## Streaming Pipeline Gaps (2026-08-10 audit → 2026-08-10 implemented)
+### Playout
 
-STREAMING.md contract vs playlist.go/main.ts implementation. See Backlog for full details.
-
-- [x] Continuous R2 sync during playout (HIGH) — implemented: fsnotify watcher uploads segments within ~7s
-- [x] Globally unique segment naming with run ID (HIGH) — implemented: `r-{ISOTIME}/seg-{seq}.m4s`
-- [x] Atomic manifest publication with temp key (MEDIUM) — implemented: upload to .tmp, then CopyObject
-- [x] Manifest Cache-Control: add must-revalidate + s-maxage split (LOW) — implemented: `public, max-age=1, s-maxage=2, must-revalidate`
-- [ ] CORS headers on HLS responses from media.linearvoid.tv (MEDIUM) — infrastructure, needs Cloudflare R2 CORS config
-- [x] Player error recovery: retries, backoff, reconnecting state (MEDIUM) — implemented: hls.on(ERROR) handler with exponential backoff
-- [x] Health endpoint: add active item, revision, run ID, segment info (LOW) — implemented: full STREAMING.md contract fields
-- [x] Phase 4 edge cache validation — script `scripts/validate-edge-cache.sh` created
+- [ ] Configure an R2 lifecycle rule on the `hls/` prefix as defense-in-depth (manual)
+- [ ] Full-loop E2E (2-file playlist, assert `Looping...` with no `CRASHED` across the boundary) — manual via `scripts/test-playout-e2e.sh`