linearvoid/.github/instructions/BACKLOG.md

BACKLOG

All current work is planned in TODO (2026-08-13). This file lists backlog items; new unplanned work is added here.

Setup phases remaining

Phase 5: complete manual Cloudflare and external-service setup

• Apply Cloudflare Workers, Tiered Cache, WAF, rate limiting, and HTTPS/HSTS settings in the dashboard
• Cloudflare Workers: link custom domain and cut over frontend hosting
• Enable Smart Tiered Cache; validate Cf-Cache-Status: HIT from upper tier
• WAF: enable OWASP Managed Ruleset (Simulate → review 48h → Block), enable Bot Fight Mode
• Rate limiting: /api/admin/login + /api/admin/* → 10 req/10s per IP, Challenge action
• HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2; enforce at edge and origin
• UptimeRobot: add HTTP monitor for https://linearvoid.tv/health (5min interval)
• UptimeRobot: add keyword monitor for https://linearvoid.tv/hls/playlist.m3u8 (keyword #EXTM3U, 5min)
• UptimeRobot: alert on 2 consecutive failures via email + Discord/Slack webhook
• Capture evidence in docs/operations/cloudflare-uptimerobot-checklist.md

Phase 6: rehearse production deployment end to end

• Provision Hetzner CX22 (2 vCPU, 4 GB, 40 GB) in Ashburn VA, Ubuntu 24.04
• Install Docker Engine + Compose plugin; open firewall: 80/443 public, 2375 runner-only
• Configure Docker daemon for remote TLS access (-H tcp://0.0.0.0:2376 with certs)
• Create /srv/linearvoid/data/{playlist,media,hls,chat,slate} directories
• Add CI_PROD_* secrets to Gitea (DOCKER_HOST, R2 keys, admin password, DB password, Grafana password)
• Trigger CD workflow with DOCKER_HOST pointing to Hetzner IP
• Verify all 11 containers healthy: /health, /hls/playlist.m3u8, WebSocket
• Update Cloudflare A record linearvoid.tv → Hetzner IP, SSL Full (strict); test HTTPS + WSS
• Smoke test: viewer SPA loads, HLS plays, chat connects, admin login → upload → playlist → playout
• Set up Prometheus alerting notifications + UptimeRobot on production URL
• Run scripts/backup-db.sh + scripts/verify-backups.sh on production host
• Document production IP, SSH access, rollback procedure, scaling thresholds

Phase 7: baseline scaling and performance

• Execute WebSocket load tests — ramp 10→100→500→1000 connections; measure p95 latency + errors
• Execute admin API benchmarks — wrk/k6 against /api/admin/*; target p95 <1s
• Measure manifest TTFB and cache-hit distribution from multiple US regions (WebPageTest 3+ regions, <4s warm cache)
• Tune manifest TTL to 0.5× segment duration; validate Cf-Cache-Status distribution
• Track cache-hit ratio and R2 ops in production (Cloudflare GraphQL → Grafana panel, alert <90%)
• Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min
• Plan Phase 2 service separation (playout CPU, chat latency, DB state, monitoring)
• Plan Phase 3 multi-origin resilience (second Hetzner location, warm standby, DNS failover, DB/Redis replication)

Phase 8: close out remaining test and validation gaps

• Admin frontend: user moderation tools tests (mute/ban/report, view moderation logs)
• Frontend: coverage for all critical user + admin components and interactions
• Frontend: document test cases and expected outcomes
• Backend R2: test retrieval of segments/manifests via Cloudflare edge
• Backend R2: validate edge cache hit/miss behavior with repeat requests (scripts/validate-edge-cache.sh)

Playout

• Configure an R2 lifecycle rule on the hls/ prefix as defense-in-depth (manual)
• Full-loop E2E (2-file playlist, assert Looping... with no CRASHED across the boundary) — manual via scripts/test-playout-e2e.sh

Cloudflare Configuration

DNS & Workers

• Cloudflare Workers: link custom domain (manual)
• Create Cloudflare Worker and cut over frontend hosting (manual)

Security & TLS

• Enable Tiered Cache, WAF, rate limiting (manual)
• Tiered Cache: enable Smart Tiered Cache, validate Cf-Cache-Status: HIT from upper tier
• WAF: enable OWASP Managed Ruleset, set to Simulate → review 48h → Block, enable Bot Fight Mode
• Rate limiting: /api/admin/login + /api/admin/* → 10 req/10s per IP, Challenge action
• HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2
• HTTPS/WSS: enforce at edge (Cloudflare) and origin (nginx), redirect HTTP → HTTPS

Monitoring & Observability

External Monitoring

• Set up UptimeRobot external monitoring
• UptimeRobot: add HTTP monitor for https://linearvoid.tv/health (5min interval)
• UptimeRobot: add keyword monitor for https://linearvoid.tv/hls/playlist.m3u8 (keyword: #EXTM3U, 5min)
• Alert on 2 consecutive failures, notify via email + Discord/Slack webhook

External Services

Evidence tracker: docs/operations/cloudflare-uptimerobot-checklist.md

• UptimeRobot: add HTTP monitor for https://linearvoid.tv/health (5min interval)
• UptimeRobot: add keyword monitor for https://linearvoid.tv/hls/playlist.m3u8 (keyword: #EXTM3U, 5min)
• Alert on 2 consecutive failures, notify via email + Discord/Slack webhook

Production Deployment

Host Provisioning (Hetzner — not provisioned)

• Provision Hetzner CX22 (2 vCPU, 4 GB, 40 GB) in Ashburn VA, Ubuntu 24.04
• Install Docker Engine + Compose plugin, open firewall: 80/443 (public), 2375 (runner IP only)
• Configure Docker daemon for remote TLS access (-H tcp://0.0.0.0:2376 with certs)
• Create /srv/linearvoid/data/{playlist,media,hls,chat,slate} directories

CI/CD & Secrets

• Add CI_PROD_* secrets to Gitea: DOCKER_HOST, R2 keys, admin password, DB password, Grafana password
• Trigger CD workflow with DOCKER_HOST pointing to Hetzner IP

Smoke Test & Cutover

• Verify all 11 containers start healthy, check /health, /hls/playlist.m3u8, WebSocket
• Update Cloudflare A record linearvoid.tv → Hetzner IP, SSL Full (strict), test HTTPS + WSS
• Smoke test: viewer SPA loads, HLS plays, chat connects, admin login → upload → playlist → playout

Production Operations

• Set up Prometheus alerting notifications + UptimeRobot on production URL
• Run scripts/backup-db.sh + scripts/verify-backups.sh on production host
• Document production IP, SSH access, rollback procedure, scaling thresholds

Scaling & Performance

Load Testing

• Execute WebSocket load tests with real infrastructure — ramp 10→100→500→1000 connections, measure p95 latency + errors
• Execute admin API benchmarks with real infrastructure — wrk/k6 against /api/admin/*, target p95 <1s

Cache & TTFB Optimization

• Optimize segment immutability and manifest TTL for cache efficiency
• Tune manifest TTL: measure Cf-Cache-Status distribution, adjust to 0.5× segment duration
• Measure playback TTFB from multiple US regions — WebPageTest from 3+ regions, target <4s warm cache
• Track cache hit ratio and R2 ops in production — Cloudflare GraphQL → Grafana panel, alert <90%

Capacity & Future Planning

• Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min
• Plan Phase 2 service separation — document boundaries (playout CPU, chat latency, DB state, monitoring), estimate per-service resources
• Plan Phase 3 multi-origin resilience — research second Hetzner location, design warm standby + DNS failover + DB/Redis replication