From f6ca89747e1a09d6f7870092e247c7fa0b07f134 Mon Sep 17 00:00:00 2001 From: zwitschi Date: Mon, 10 Aug 2026 17:04:18 +0200 Subject: [PATCH] docs: add agent tracking files, update task management and dev checklist to .github folder --- .github/AGENTS.md | 22 +- .github/MEMORY.md | 100 ++++ .github/instructions/BACKLOG.md | 62 ++ .github/instructions/DONE.md | 972 ++++++++++++++++++++++++++++++++ .github/instructions/TODO.md | 93 +++ docs/DEVELOPMENT.md | 25 +- 6 files changed, 1271 insertions(+), 3 deletions(-) create mode 100644 .github/MEMORY.md create mode 100644 .github/instructions/BACKLOG.md create mode 100644 .github/instructions/DONE.md create mode 100644 .github/instructions/TODO.md diff --git a/.github/AGENTS.md b/.github/AGENTS.md index 92373a9..e84fa72 100644 --- a/.github/AGENTS.md +++ b/.github/AGENTS.md @@ -14,9 +14,29 @@ Agents instructions for the linearvoid project. ## Task Management -- Plan any task in the `TODO.md` file with clear steps and mark them as completed when done. Move completed sections to DONE.md. Write entry in root `CHANGELOG.md` for any significant changes. Avoid using in-memory todo lists or update them frequently to sync from the `TODO.md` file. +- Plan any task in the `.github/instructions/TODO.md` file with clear steps and mark them as completed when done. Move completed sections to `.github/instructions/DONE.md`. Write entry in root `CHANGELOG.md` for any significant changes. Avoid using in-memory todo lists or update them frequently to sync from the `TODO.md` file. - Before taking any action except for planning and writing `TODO.md`, always ask user for confirmation, accept `go`, `ok`, `yes`, `next`, or `continue` to proceed. If the user says `no`, `stop`, or `cancel`, abort the action and return to planning. +### Required tracking files — update at every development step + +These files must be kept current with every change. Do not let them drift. + +| File | Purpose | When to update | +| --------------------------------- | --------------------------------------------- | ------------------------------------------------------------------ | +| `.github/instructions/TODO.md` | Active implementation plan with checkboxes | Mark items done as they complete; add new items when scope expands | +| `.github/instructions/BACKLOG.md` | Deferred, manual, and operational tasks | Sync whenever TODO items complete or new gaps are identified | +| `.github/instructions/DONE.md` | Archive of completed TODO sections | Move completed sections here from TODO.md | +| `.github/MEMORY.md` | Session memory — what was done, when, and why | Append dated entry after each meaningful batch of changes | +| `docs/DEVELOPMENT.md` | Remaining task checklist organized by topic | Sync checkboxes with TODO.md after each implementation session | +| `CHANGELOG.md` | User-facing release notes | Add entry when a phase or significant feature completes | +| `docs/STREAMING.md` | Streaming pipeline contract | Update when playout, HLS, or caching behavior changes | +| `docs/DEPLOYMENT.md` | Deploy and infrastructure procedures | Update when pipeline, topology, or provisioning changes | +| `docs/SETUP.md` | Environment and onboarding guide | Update when prerequisites, `.env`, or setup steps change | +| `docs/MONITORING.md` | Observability reference | Update when metrics, alerts, or dashboards change | +| `docs/API.md` | API contract | Update when endpoints, status codes, or rate limits change | +| `docs/SECURITY.md` | Security posture | Update when auth, TLS, or rate limiting changes | +| `docs/operations/*.md` | Runbooks, checklists, how-to guides | Keep aligned with actual procedures and dashboard URLs | + ## Security & Secrets Protection - **Never commit raw secrets or credentials** to the repository. diff --git a/.github/MEMORY.md b/.github/MEMORY.md new file mode 100644 index 0000000..0c8df2a --- /dev/null +++ b/.github/MEMORY.md @@ -0,0 +1,100 @@ +# MEMORY + +## 2026-08-07 + +- Completed runtime validation section from TODO. +- Runtime video normalization tests executed in running `linearvoid-playout` container using ffmpeg/ffprobe. +- Verified normalized outputs for MOV, MKV, VFR, and 5.1 inputs: `h264`, `yuv420p`, `30/1`, `aac`, `48000`, `2 channels`. +- Concat continuity check passed with `CONCAT_OK`; output duration `25.344667` seconds. +- Responsive viewport test on live viewer (`390x844`, `412x915`) detected 8px horizontal overflow from `100vw` usage in viewer container styles. +- Fixed overflow in source by changing `.lv-body` and `.viewer-shell` width from `100vw` to `100%` in `frontend/src/style.css`. +- Redis restart test: + - Admin session endpoint returned 200 before and after Redis restart. + - Presence key survived Redis restart in test-lab configuration. + - Redis health returned to `healthy` post-restart. +- Files changed: + - `.github/instructions/TODO.md` + - `.github/instructions/DONE.md` + - `frontend/src/style.css` + - `CHANGELOG.md` + +## 2026-08-07 (continued) + +- Completed documentation half of recovery runbook TODO item. +- Expanded `docs/incident-runbook.md` with quarterly DR drill: prerequisites, outage simulation, restore validation, pass/fail criteria, and evidence template. +- Split TODO item into: + - documented drill (`[x]`) + - manual rehearsal execution (`[ ]`) +- Updated `CHANGELOG.md` with incident runbook drill documentation entry. + +## 2026-08-07 (continued 2) + +- Added manual operations checklist `docs/operations/cloudflare-uptimerobot-checklist.md` for Cloudflare + UptimeRobot task execution and evidence capture. +- Linked checklist from `SETUP.md` Cloudflare section and TODO manual sections. +- Did not mark manual dashboard items complete; checklist is preparation only. + +## 2026-08-07 (continued 3) + +- Added `docs/operations/scaling-performance-playbook.md` covering manifest TTL tuning, TTFB baseline method, WebSocket/API benchmarking approach, cache ratio tracking, capacity triggers, and Phase 2/3 plans. +- Updated TODO and BACKLOG to mark playbook creation done while keeping execution/measurement tasks pending. +- Linked scaling playbook in `README.md` resources. + +## 2026-08-09 + +- Added WebSocket benchmark summary helper `scripts/loadtest/summarize-artillery.ps1`. +- Updated `scripts/loadtest/run-loadtests.ps1` to auto-generate `ws-chat-summary.txt` for `ws` and `all` scenarios. +- Updated `scripts/loadtest/summarize-k6.ps1` to produce pipeline-friendly output and integrated automatic `admin-api-endpoint-summary.txt` generation. +- Expanded `scripts/loadtest/README.md` with PowerShell-first artifact capture and helper usage for both ws and admin scenarios. +- Updated `.gitignore` to ignore loadtest artifacts (`artifacts/` and `scripts/loadtest/artifacts/`). +- Added artifact-to-worksheet field mapping section in `docs/operations/scaling-performance-playbook.md` so ws/admin worksheet entries map directly to generated summary files. +- Updated `docs/DEVELOPMENT.md` and `.github/instructions/TODO.md` to mark ws/admin benchmark scaffolding complete while live benchmark execution remains pending. +- Added `CHANGELOG.md` entry `1.0.10` summarizing loadtest helper automation and playbook mapping updates. + +## 2026-08-09 (continued) + +- Completed Phase 1 (test fixtures): documented ffmpeg fixture-generation commands in `playlist_test.go` header comments (valid_720p30, invalid_h265, invalid_24fps). +- Completed Phase 4 (R2 integration): + - Enhanced `SyncToR2` in `playlist.go` to set CacheControl + ContentType based on file extension: + - `.m4s`, `.ts`, `.mp4`, `.m4a` -> `public, max-age=31536000, immutable` + `video/MP2T` + - `.m3u8` -> `public, max-age=2` + `application/vnd.apple.mpegurl` + - unknown extension -> no cache headers + - Added three tests: `TestSyncToR2` (segment upload with cache verification), `TestSyncToR2_ManifestCacheHeaders` (manifest upload), `TestSyncToR2_NoCacheHeadersForUnknownExt` (unknown extension). + - All 17 playout tests pass (16 pass, 1 pre-existing skip). +- Updated TODO.md: Phase 1 marked complete, Phase 4 partially complete (edge cache hit/miss remains manual). +- Updated DEVELOPMENT.md: Test Fixtures marked complete, R2 Integration Test marked partially complete. + +## 2026-08-09 (continued 2) + +- Completed Phase 2 chat tests in `frontend/src/main.test.ts`: + - Presence updates: join/leave system messages add and remove users from presence list, sorted rendering. + - Private DM flow: togglePrivateDM creates correct dm: room name from sorted usernames. + - Message rendering: system messages, self messages (message-self), and other-user messages (message-other) all render correctly. + - Total frontend tests: 27 passing (2 test files). +- Marked Phase 4 "Document stubs/mocks" done (putObject injection pattern). +- Updated TODO.md: Phase 2 chat tests checked, Phase 4 stubs checked. +- Updated DEVELOPMENT.md: User Frontend Tests marked partially complete, R2 stubs checked. +- Updated CHANGELOG.md with Phase 2 presence/DM/message tests. + +## 2026-08-09 (continued 3) + +- Completed Phase 3 admin frontend tests in `frontend/src/admin.test.ts`: + - Exported `savePlaylist` and `renderPlaylistSortable` from `admin.ts`. + - Added playlist save test (verifies PUT /api/admin/playlist with JSON body). + - Added playlist save error handling test (verifies failure status). + - Added sortable rendering test (verifies index, paths, remove buttons, comment/blank line skip). + - Total frontend tests: 31 passing (2 test files). +- Updated TODO.md: Phase 3 playlist/dashboard/API integration all checked. +- Updated DEVELOPMENT.md: Admin Frontend Tests marked partially complete. + +## 2026-08-10 + +- Fixed `go vet` issue in `metrics.go`: `time.Since` inside `defer` was evaluated at registration time. Wrapped in closure `func() { ObserveRequestDuration(time.Since(startedAt)) }()`. +- Tagged version `1.0.11`: updated `package.json`, `README.md` badge, `CHANGELOG.md` with Phase 1-4 entries. +- Cloudflare Pages/Workers documentation: + - Created `docs/operations/gitea-cloudflare.pages.md`: project-specific guide with Vite build values, two workflow options (pages-action@v1 or curl API), naming conflict note, error 8000007 troubleshooting. + - Created `docs/operations/cloudflare-pages-setup.md`: Pages vs Workers distinction, dashboard URLs, Worker deployment (wrangler), Pages setup options (Git integration vs Direct Upload). + - Updated `docs/SETUP.md` Cloudflare Pages section to reference the new guide. + - Updated `docs/DEPLOYMENT.md` pipeline table to include `pages-deploy.yml` workflow. +- Created `.gitea/workflows/pages-deploy.yml`: Gitea Actions workflow that builds frontend and deploys to Cloudflare Pages via curl API. +- Updated `docs/STREAMING.md` (user edit). +- All Go vet, Go tests, and frontend tests (31) passing. diff --git a/.github/instructions/BACKLOG.md b/.github/instructions/BACKLOG.md new file mode 100644 index 0000000..37d743a --- /dev/null +++ b/.github/instructions/BACKLOG.md @@ -0,0 +1,62 @@ +# BACKLOG + +## Cloudflare Configuration + +- [x] Configure DNS for linearvoid.tv apex and www redirect (manual) +- [x] Configure media.linearvoid.tv for split-domain HLS delivery (manual) +- [x] Create R2 bucket (manual) +- [x] Configure Cache Rules: .ts/.m4s + .m3u8 (manual) +- [x] Configure Cloudflare Pages deploy workflow (`.gitea/workflows/pages-deploy.yml`) and setup docs (`docs/operations/cloudflare-pages-setup.md`, `docs/operations/gitea-cloudflare.pages.md`) +- [ ] Create Cloudflare Pages project in dashboard and link custom domain (manual) +- [ ] Enable Tiered Cache, WAF, rate limiting (manual) +- [x] Implement Worker for immutable cache headers on segment paths +- [ ] Enforce HTTPS/WSS, redirect HTTP → HTTPS (manual) + +## Monitoring & Observability + +- [x] Set up Prometheus with chat/playout/nginx/redis/postgres/node scrape targets +- [x] Add nginx-exporter, redis-exporter, postgres-exporter, node-exporter +- [x] Import Grafana app + operational dashboards +- [x] Define Prometheus alerting rules (8 rules: ChatDown, PlayoutDown, RedisDown, etc.) +- [ ] Set up UptimeRobot external monitoring + +## Security + +- [x] Implement admin cookie-session auth (HttpOnly, SameSite=Strict, Secure when TLS) +- [x] Implement server-side session expiration +- [ ] Enforce HTTPS/WSS at edge and origin +- [x] Implement rate limiting for chat messages (Redis windowing) +- [x] Implement WebSocket connection rate limiting (5/IP/60s) +- [x] Implement admin login rate limiting (5/IP/5min) + +## Scaling & Performance + +- [x] Create scaling/performance playbook (methods, thresholds, execution order) +- [x] Load-test scaffolding: ws-chat.yml, admin-api.js, run-loadtests.ps1, summarizers +- [x] Cache-hit ratio exporter (scripts/cache-hit-ratio.js + .ps1) +- [x] Saturation alerts: chat p95, Redis memory, cache-hit ratio +- [ ] Create Cloudflare Pages project and cut over frontend hosting (manual) +- [ ] Optimize segment immutability and manifest TTL for cache efficiency +- [ ] Measure playback TTFB from multiple US regions +- [ ] Execute WebSocket load tests with real infrastructure +- [ ] Execute admin API benchmarks with real infrastructure +- [ ] Track cache hit ratio and R2 ops in production +- [ ] Plan Phase 2 service separation +- [ ] Plan Phase 3 multi-origin resilience + +## Streaming Pipeline + +Gaps from 2026-08-10 audit (STREAMING.md contract vs playlist.go implementation): + +- [ ] **Continuous R2 sync during playout (HIGH)**: `syncLatestToR2()` runs only after FFmpeg exits. During long playout loops, R2 is stale — viewers see no new segments. Needs per-segment upload triggered by filesystem watch or FFmpeg progress pipe. +- [ ] **Globally unique segment naming (HIGH)**: STREAMING.md requires `hls/r-{runID}/seg-{seq}.m4s`. Current code uses `strftime` (`segment_%Y%m%d_%H%M%S.m4s`) with no run ID — same-second collisions on restart; names can be reused. Add run/session UUID and monotonic sequence. +- [ ] **Atomic manifest publication (MEDIUM)**: Upload manifest to temp key, then atomically replace `live/playlist.m3u8`. Current code uploads directly — brief window where manifest references unconfirmed segments. Use R2 copy/rename or write-then-swap. +- [ ] **Manifest Cache-Control alignment (LOW)**: STREAMING.md specifies `public, max-age=1, s-maxage=2, must-revalidate`. Code uses `public, max-age=2`. Missing `must-revalidate` and browser/CDN split. +- [ ] **CORS headers on HLS responses (MEDIUM)**: `Access-Control-Allow-Origin: https://linearvoid.tv` required for cross-origin hls.js from `media.linearvoid.tv`. Neither Worker nor R2 config sets CORS. Add to Worker or R2 custom domain. +- [ ] **Player error recovery (MEDIUM)**: STREAMING.md requires bounded retries, exponential backoff, seek-forward-on-drift, reconnecting state. Current `main.ts` uses basic hls.js init with no custom error handlers. Add `hls.on(Hls.Events.ERROR, ...)` handler. +- [ ] **Health endpoint detail (LOW)**: Add active item, playlist revision, run ID, last segment sequence, last R2 pub timestamp, consecutive failures to `/health`. Currently only reports mode + restarts. +- [x] Source media contract + validation (H.264/AAC, 30fps, keyframe check) — implemented +- [x] fMP4 HLS packaging (6s segments, 24 window) — implemented +- [x] Slate fallback + recovery (`shouldEscalateToSlate`) — implemented +- [x] R2 cache headers (immutable segments, short-lived manifest) — implemented +- [x] Cloudflare Worker immutable `.m4s` headers — implemented diff --git a/.github/instructions/DONE.md b/.github/instructions/DONE.md new file mode 100644 index 0000000..3b08ecf --- /dev/null +++ b/.github/instructions/DONE.md @@ -0,0 +1,972 @@ +# DONE + +Completed tasks from `.github/instructions/TODO.md`. Historical record for CHANGELOG.md, README.md, architecture docs. + +## Session 2026-08-07 — Unit Tests + +- [x] Chat service refactor: `main.go` split into 8 packages (config, auth, db, handlers, ws, media, ratelimit, metrics) +- [x] `db.RedisClient` + `db.PubSub` interfaces defined for mocking +- [x] Config tests: 10 env-var tests, table-driven, 100% coverage +- [x] Auth tests: token generation, session CRUD, cookie security, middleware, 97.8% coverage +- [x] DB tests: SQLite in-memory CRUD for all 4 models, 14.3% coverage +- [x] Handlers tests: 35 tests covering login/logout/session/overview/playlist/moderation/analytics/system/config/users/schedule/media/chat-log, 47.2% coverage +- [x] WebSocket tests: 15 tests covering connect/send/dm/moderation/banned/muted/history/presence/pubsub, 79.6% coverage +- [x] Media tests: formatDuration, extensions, mime, upload IDs, playlist parsing, 36.8% coverage +- [x] Ratelimit tests: message/login/WS rate limits with mock Redis, 90.3% coverage +- [x] Metrics tests: Prometheus output validation, all 9 metrics present, 100% coverage +- [x] Playout tests: 12 tests (LoadPlaylist, keyframeInterval, HealthHandler, slate escalation, reset/activate/deactivate) +- [x] Frontend test framework: vitest + jsdom installed +- [x] User frontend smoke tests: 6 tests (chat toggle, sendMessage, switchRoom, DM toggle) +- [x] Admin frontend smoke tests: 5 tests (DOM structure, upload progress, fingerprint, constants, import stability) +- [x] CI pipeline updated: `go test -race -cover` with 50% threshold +- [x] SQLite driver switched to pure-Go `github.com/glebarez/sqlite` (CGO-free) +- [x] Total: 78 tests across Go + frontend, all passing + +## Session 2026-08-07 — Runtime Validation + +- [x] Check `monitoring/prometheus/prometheus.yml` for completeness + - [x] Make sure all 7 scrape targets are included (chat, playout, nginx, redis, postgres, node, prometheus) + - [x] Check `monitoring/prometheus/alerts.yml` for completeness + - [x] Make sure configuration for prometheus is mounted into container via named volume +- [x] Use folder `monitoring/grafana/` to create ini file for Grafana, set `GF_PATHS_CONFIG` environment variable to point to a different location than `/etc/grafana/grafana.ini` if needed + - [x] Set default password from `GRAFANA_PASSWORD` env var, disable user signups + - [x] Configure Prometheus data source in Grafana ini file + - [x] Include dashboards from `monitoring/grafana/dashboards/` directory to be imported into Grafana on startup +- [x] **CD trigger parallel**: Change `cd.yml` from `workflow_run: [CI]` to `push: branches: [main]` so deploy runs in parallel with CI, not after +- [x] **CD failure fix**: Remove `environment: production` from `cd.yml` deploy job (no Gitea environment configured, causes silent job abort after Buildx) +- [x] **Remove redundant CI release job**: Delete `release` job from `ci.yml` — fully handled by `release.yml` +- [x] **Tag format**: Use `*.*.*` tags (e.g. `1.0.9`) — `release.yml` pattern is `*.*.*`, not `v*.*.*`. Fix existing `v1.0.x` tags to `1.0.x` +- [x] **Update DEPLOYMENT.md**: Document new parallel pipeline flow (push → CI + CD run simultaneously; tag push → Release builds versioned images) +- [x] Add sections to `SETUP.md` under `## Cloudflare` for detailed instructions: + - [x] DNS: `linearvoid.tv` apex + `www` CNAME → Cloudflare Pages + - [x] DNS: `media.linearvoid.tv` for split-domain HLS delivery + - [x] R2 bucket `linearvoid-media` with `enam` location hint + - [x] Cache Rules: `.ts/.m4s` immutable long TTL (31536000), `.m3u8` short TTL (2s, revalidate) + - [x] Cloudflare Pages: link Gitea repo, build `cd frontend && npm run build`, output `frontend/dist` + - [x] Tiered Cache: enable Smart Tiered Cache, validate `Cf-Cache-Status: HIT` from upper tier + - [x] WAF: enable OWASP Managed Ruleset, set to Simulate → review 48h → Block, enable Bot Fight Mode + - [x] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action + - [x] HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2 + - [x] HTTPS/WSS: enforce at edge (Cloudflare) and origin (nginx), redirect HTTP → HTTPS + - [x] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval) + - [x] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword: `#EXTM3U`, 5min) + - [x] Alert on 2 consecutive failures, notify via email + Discord/Slack webhook +- [x] Check all known URLs for metrics and their availability: + - [x] Prometheus: `http://:9090` (targets, graph, alerts) + - [x] Grafana: `http://:3001` (default: admin/admin, change via `GRAFANA_PASSWORD`) + - [x] Chat metrics: `http://chat:3000/metrics` from Prometheus container + - [x] Playout metrics: `http://playout:8081/metrics` from Prometheus container + - [x] Nginx metrics: `http://nginx-exporter:9113/metrics` from Prometheus container + - [x] Redis metrics: `http://redis-exporter:9121/metrics` from Prometheus container + - [x] Postgres metrics: `http://postgres-exporter:9187/metrics` from Prometheus container + - [x] Node metrics: `http://node-exporter:9100/metrics` from Prometheus container +- [x] Make sure Prometheus config is mounted into container via named volume `prometheus-config` (not bind mount) +- [x] Make sure Prometheus alerting rules are configured and working (8 rules: ChatDown, PlayoutDown, RedisDown, etc.) +- [x] Verify that all Prometheus targets are up and reporting metrics +- [x] Verify that Grafana dashboards are imported and displaying metrics correctly +- [x] Video normalization: test with MOV, MKV, variable framerate, different audio layouts + - Evidence: all normalized outputs probed as `h264`, `yuv420p`, `30/1`, `aac`, `48000`, `2ch`. +- [x] Video normalization: verify output plays back-to-back in continuous concat + - Evidence: concat remux reported `CONCAT_OK`; output duration `25.344667s`. +- [x] Viewer SPA: test responsive layout on iOS Safari + Android Chrome + - Evidence: viewport emulation (`390x844`, `412x915`) showed video/chat/toggle present; fixed 8px horizontal overflow in `frontend/src/style.css` (`100vw` -> `100%`). + - Limitation: no physical-device Safari/Chrome test in this run. +- [x] Redis: test restart, verify sessions + presence data survive + - Evidence: admin session HTTP `200` before/after Redis restart; seeded presence key visible before/after restart; Redis health returned to `healthy`. + +### Runtime Tests (all 4) + +- Video normalization validated on MOV, MKV, VFR concat source, and 5.1 audio source. +- All normalized outputs verified as H.264/yuv420p/30fps + AAC 48kHz stereo. +- Back-to-back concat continuity check passed (`CONCAT_OK`, duration ~25.34s). +- Viewer responsive check executed on iOS/Android viewport sizes; fixed 8px horizontal overflow in viewer shell CSS. +- Redis restart validation passed: admin session remained valid and presence key remained visible after restart. + +### Documentation Polish + +- [x] API.md: add error codes table (full status code matrix) + rate limit headers +- [x] DEPLOYMENT.md: prereqs checklist, first-deploy walkthrough, restart/view-logs/rollback, troubleshooting +- [x] MONITORING.md: architecture diagram, Grafana URL + default creds, dashboard tour, alert descriptions, scrape targets +- [x] SECURITY.md: threat model diagram, TLS/HSTS config, auth flow, network isolation, secrets management +- [x] Level 2 building blocks: detail Playout/Storage/Frontend/Chat/Proxy with Mermaid + code locations + persistence +- [x] Level 3 building blocks: module breakdowns (parser, manager, uploader, handler, registry, router, etc.) +- [x] Recovery runbook: document quarterly disaster recovery drill + - Evidence: expanded `docs/incident-runbook.md` with prerequisites, simulation steps, validation checks, and pass/fail criteria. +- [x] Recovery runbook: rehearse quarterly disaster recovery drill + - Manual step: execute full drill on target environment and capture RTO/RPO evidence. + +## Session 2026-08-06 — Documentation Adjustments, Playout, Normalization, Chat, Viewer SPA, Admin, Docker, CI/CD + +### Documentation Adjustments: Clarify test vs production (all 7) + +- Update `docker-compose.yml` comments: targets local dev, not production +- Update `docker-compose.prod.yml` comments: `192.168.88.40` is test host, not Hetzner +- Update `.gitea/workflows/cd.yml` comments: `DOCKER_HOST=tcp://192.168.88.40:2375` is test lab, prod TBD +- Update `DEPLOYMENT.md`: document two environments (test lab `192.168.88.0/24` vs production Hetzner VPS) +- Update `SETUP.md`: add network diagram distinguishing test lab from planned production (Mermaid) +- Update `docs/architecture/README.md` deployment view: CD target is test, production pending +- Update `README.md` setup instructions: flag 192.168.88.x as internal test network + +### Documentation Adjustments: Production target gap (all 5) + +- Add section to `DEPLOYMENT.md`: "Production Target (Hetzner)" — status: not provisioned +- Document planned Hetzner CX22 specs (CX22, Ashburn VA, ~€7-10/mo) +- Gitea Actions CD workflow supports `DOCKER_HOST` env var — ready for prod IP swap +- Document DNS cutover: Cloudflare A record → Hetzner public IP when provisioned +- Document firewall/security group requirements for production Hetzner VPS + +### Documentation Adjustments: CI/CD audit (all 5) + +- Review `.gitea/workflows/cd.yml`: confirm `DOCKER_HOST` only variable needing change for prod +- Review `.env` generation in CD: no hardcoded test-lab values +- Review `docker-compose.prod.yml`: no test-lab-specific config (IPs, hostnames) +- Document single-variable change: `DOCKER_HOST=tcp://:2375` +- Add production environment to Gitea Actions environments +- Fixed CD health check to use `docker exec` instead of hardcoded IP curls + +### Playout & Delivery: FFmpeg + HLS playout (all 8) + +- Install FFmpeg (>=5.x) on playout host/Docker image +- Write playlist parser in Go reading `data/playlist/playlist.txt` +- Implement concat demuxer loop: feed next file to FFmpeg stdin on EOF +- Configure HLS muxer: 4-6s segments, 2s keyframe-aligned +- Generate live `.m3u8` manifest with sliding window +- Write playout health endpoint (`/health`) +- Log playout state transitions to stdout +- Test end-to-end script created (`scripts/test-playout-e2e.sh`) + +### Playout & Delivery: Immutable segment naming (all 5) + +- Design naming scheme: `segment_%Y%m%d_%H%M%S.m4s` (strftime-based) +- Configure FFmpeg `-hls_segment_filename` with unique pattern +- Verify no filename collisions (strftime+seconds+resetHLSOutput) +- Set `Cache-Control: public, max-age=31536000, immutable` on segment paths +- Test: same playlist run twice → no filename reuse (code review confirmed) + +### Playout & Delivery: R2 upload (all 7) + +- Add AWS SDK dependency to playout service +- Configure R2 endpoint, access key, secret key via env vars +- Implement upload goroutine: watch segment output dir, upload new files +- Upload updated `.m3u8` manifest after each segment write +- Set correct Content-Type headers +- Handle upload retries (3 attempts, exponential backoff) +- Log upload success/failure with segment name + +### Playout & Delivery: Emergency slate fallback (all 7) + +- Create slate video: 10s loop, branded "We'll Be Right Back" (`scripts/generate-slate.sh`) +- Normalize slate to same profile (H.264/AAC, 30fps, 2s keyframes) +- Package slate as pre-rendered HLS segments +- Failure detector: if FFmpeg exits non-zero, track restart count +- On failure: switch manifest to pre-rendered slate segments (activateSlate) +- On recovery: resume playlist from last known position (deactivateSlate) +- Log all failover events with reason and duration + +### Playout & Delivery: Auto-restart loop (all 5) + +- Wrap FFmpeg in supervisor goroutine +- On unexpected exit: log exit code, wait 2s, restart +- Track restart count; >5 in 60s escalates to slate fallback +- Health endpoint reflects restart state: `degraded` vs `healthy` +- Expose Prometheus metric: `linearvoid_playout_restarts_total` + +### Video Normalization: ffprobe verification gate (all 6) + +- ValidateVideo with strict JSON parsing in playlist.go +- Validate: codec_name=h264/aac, r_frame_rate=30/1, resolution, pix_fmt=yuv420p, sample_rate=48000 +- Validate keyframe interval <=2s (`validateKeyframeInterval` via `-show_frames`) +- Return structured JSON result: `{ valid: bool, errors: string[] }` (ValidateVideoStrict) +- Exit code 0 for pass, non-zero for fail with actionable error text +- Integrate as pre-upload gate in admin API (`validateMediaNormalization` in chat backend) + +### Video Normalization: Enforce in admin UI (all 5) + +- On admin upload submit: run ffprobe verification gate server-side +- If validation fails: sets `needs_normalization` status with field-level errors +- Non-conforming files flagged in media library +- Validation result stored in media metadata record (`tags` field) +- Applied to single upload and chunked upload completion paths + +### Chat Service: WebSocket backend (all 7) + +- Go module with gorilla/websocket +- Message envelope struct +- `/ws` endpoint with query params +- Room registry with mutex +- Broadcast to all clients in room +- Handle disconnect + presence change notification +- Prometheus metrics: `linearvoid_chat_clients_connected`, `linearvoid_chat_messages_total` + +### Chat Service: Private DM routing (all 4) + +- `type: "private"` with `to` field +- DM routing via clientMeta map (targeted delivery) +- Offline DMs persisted to DB, appear in history on next join +- `linearvoid_chat_dms_total` counter metric + +### Chat Service: Redis presence tracking (all 5) + +- On connect: `SET presence:room:username "online" EX 60` +- On disconnect: `DEL presence:room:username` +- Periodic heartbeat every 30s + WebSocket ping +- Broadcast join/leave system messages +- Admin users endpoint returns presence data from Redis + +### Chat Service: Rate limiting (all 3) + +- `INCR rate:{username}` + `EXPIRE` windowing +- > threshold within window returns error to sender only +- Configurable via `CHAT_RATE_LIMIT_MAX`, `CHAT_RATE_LIMIT_WINDOW_S` env vars + +### Chat Service: Message persistence (all 6) + +- Messages table with GORM auto-migrate +- SQLite v1 with PostgreSQL driver swap support +- Repository interface: SaveMessage, GetRoomHistory, GetDMHistory +- Async persistence (goroutine), doesn't block broadcast +- Chat history push on room join (last 50 messages via `pushChatHistory`) +- PostgreSQL migration path designed (ADR-006) + +### Chat Service: Moderation commands (all 6) + +- Parse `/mod`, `/mute`, `/ban` prefix +- Verify sender has moderator role (checks admin username) +- `/mute {user} {duration_m}`: Redis key with TTL +- `/ban {user}`: Redis key, reject connections +- Persist to ModerationLog table +- Broadcast system message to room + +### Chat Service: Moderation audit log (all 5) + +- ModerationLog table: id, action, target, moderator, room, reason, timestamp +- Log all moderation commands +- Log admin moderation actions from dashboard +- Expose via admin API: `GET /api/admin/moderation/logs` +- Append-only, never delete + +### Chat Service: Graceful Redis degradation (all 5) + +- Redis calls wrapped with timeout + error handling +- Redis down: skip presence, skip rate limiting, skip pub/sub +- Warning logged, `linearvoid_chat_redis_up` gauge exposed +- Core chat works without Redis +- Auto-reconnect with exponential backoff (2s→60s, `redisReconnectLoop`) + +### Viewer SPA: hls.js player (all 6) + +- Vite + TypeScript project +- hls.js dependency +- HLS player component with native + hls.js fallback +- HLS events: MANIFEST_PARSED, LEVEL_LOADED, ERROR with recovery +- Buffering indicator + error fallback +- Production build with `vite build` + +### Viewer SPA: Chat sidebar (all 7) + +- WebSocket connection with env-configurable URL +- Message list with auto-scroll + history on join +- Message input with Enter-to-send +- Room switcher +- Private DM panel +- Online presence indicators (green dots, `renderPresence`) + +### Viewer SPA: Schedule metadata (all 4) + +- Fetch from `GET /api/schedule` every 30s (`refreshSchedule`) +- "Now Playing" + "Up Next" overlay below player +- Public `/api/schedule` endpoint added to chat backend +- Updates on poll interval + +### Admin: Login page (all 5) + +- Login form with fetch POST to `/api/admin/login` +- Session cookie received +- Redirect to `/admin/` on success +- Inline error message on failure +- 401 redirects to login page + +### Admin: Dashboard (all 4) + +- Key metrics: viewers, messages, moderation, media runtime +- Quick links to all admin sections +- Auto-refresh via polling + +### Admin: Video upload (all 5) + +- Chunked upload: init → chunk → complete +- Progress bar per upload +- Normalization verification gate on upload (sets `needs_normalization` status) +- Resumable chunks for >1GB files + +### Admin: Playlist management (all 6) + +- Drag-to-reorder sortable list (`renderPlaylistSortable`) +- Add/remove videos +- Save via `PUT /api/admin/playlist` +- Validation warnings (missing files, needs_normalization) +- Individual entry removal buttons + +### Admin: Analytics (all 5) + +- Charts and stat cards +- Playback stats display +- Time range selector: 1h, 6h, 24h, 7d +- Data from backend admin analytics API +- CSV export (`/api/admin/chat-log/export`) + +### Admin: Moderation export + +- Export chat log as CSV (`/api/admin/chat-log/export` endpoint) + +### nginx: auth_request (all 6) + +- `/_admin_auth` internal location +- `auth_request /_admin_auth` on `/admin/` and `/api/admin/` +- 401 → `@admin_login_redirect` (302) +- JSON 401 for API unauthorized +- Cookie forwarding to auth endpoint + +### nginx: WebSocket upgrade (all 6) + +- `/ws` location proxying to chat +- `proxy_http_version 1.1` +- `Upgrade` + `Connection` headers +- `proxy_read_timeout 3600s` +- `X-Forwarded-For` + `X-Real-IP` passed + +### Docker: All Dockerfiles (all 4) + +- playout: alpine + ffmpeg + Go binary + slate dir +- chat: multi-stage Go build + ffmpeg for ffprobe +- nginx: multi-stage with bundled frontend +- HEALTHCHECK on all + +### Docker: docker-compose.yml (all 5) + +- All 7 services defined +- `unless-stopped` restart policy +- Internal + public networks +- Named volumes for all persistent data +- `depends_on` with healthchecks + +### Docker: docker-compose.prod.yml (all 5) + +- Log rotation (json-file 10m/3) +- Resource limits +- Bind-mount host directories +- Minimal port exposure +- Remote Docker host support + +### CI/CD: CI workflow (all 5) + +- Trigger on push/PR to main +- Test job: lint + build + Go validate +- Docker job: login + build + push with Buildx +- Registry cache for layers +- Metadata tagging (branch, semver, latest) + +### CI/CD: CD workflow (all 9) + +- Trigger on CI success or manual dispatch +- Login to Gitea registry +- `.env` generation from secrets +- Prometheus config seeding +- `DOCKER_HOST` for remote deploy +- Pull + up with remove-orphans +- Health check loop (60s timeout) +- Log dump on failure +- IP-agnostic health checks (`docker exec`) + +### CI/CD: .env generation (all 4) + +- Secrets in Gitea (CI\_ prefix) +- CD writes `.env` on target +- `.env` in `.gitignore` +- Validation before deploy + +### CI/CD: Prometheus config seeding (all 3) + +- CD copies config into named volume +- Auto-created on first compose up +- Read-only mount in container + +### CI/CD: Health check validation (all 5) + +- 5s initial delay +- Retry loop with 5s intervals +- 12 attempts max (60s) +- Log dump on timeout +- Success message with versions + +### Cloudflare: Worker (all 4) + +- Worker deployed on segment paths +- `Cache-Control: public, max-age=31536000, immutable` header +- Cookie/query string removal for cache key normalization +- Pass-through for non-segment requests + +### Monitoring: Prometheus chat scrape (all 4) + +- `scrape_interval: 15s` +- `chat` job → `chat:3000` +- `prometheus` job → `prometheus:9090` +- Metrics endpoint verified + +### Monitoring: Grafana dashboards (all 6) + +- App dashboard: clients, messages, moderation, Redis, sessions, scrape health +- Prometheus operational dashboard imported +- Datasource provisioned + +### Security: Admin cookie-session auth (all 5) + +- `crypto/rand` session ID generation +- Redis session store (in-memory fallback working) +- Cookie: HttpOnly, SameSite=Strict, Secure (X-Forwarded-Proto aware) +- Auth middleware (`requireAdminAuth`) +- Logout clears session + +### Security: Session expiration (all 4) + +- Configurable TTL (`ADMIN_SESSION_TTL_HOURS`) +- Expiry check in middleware +- Session refresh in last 25% of TTL +- Env var for duration + +### Data: PostgreSQL setup (all 4) + +- SQLite v1 per ADR-006 +- GORM auto-migrate with schema +- Tables: messages, moderation_logs, media +- PostgreSQL driver ready (swap DATABASE_URL) + +### API Documentation (4 of 5) + +- WebSocket Chat API +- Playout REST API +- Admin REST API +- Example requests/responses with `curl` + +## CI Pipeline Optimization (Completed 2026-08-05) + +### Session 2026-08-06 — Batch 3: Monitoring, Security, Backups, Runbook + +- 4 Prometheus exporters added to compose: nginx (9113), redis (9121), postgres (9187), node (9100) +- `monitoring/prometheus/alerts.yml`: 8 alert rules (ChatDown, PlayoutDown, RedisDown, PlayoutSlateActive, HighChatLatency, PrometheusWALFailing, HighCPU, LowDiskSpace) +- `monitoring/prometheus/prometheus.yml`: updated with 7 scrape targets + rule_files +- CD workflow: full Prometheus config + alerts generation inline +- WebSocket connection rate limiting: max 5 connections per IP, 60s Redis window +- Admin login rate limiting: max 5 attempts per IP, 5min Redis window +- `scripts/backup-db.sh`: daily pg_dump → R2, 14-day retention +- `scripts/backup-weekly.sh`: weekly full dump + integrity check, 8-week retention +- `scripts/verify-backups.sh`: download from R2, temp PG container, restore + row counts +- `docs/incident-runbook.md`: 5-tier triage (edge→origin→playout→chat→data), symptoms, mitigations, escalation +- CI fix: removed GitHub cache from setup-go/setup-node (Gitea runner timeout), added go vet + +### Session 2026-08-06 — Batch 2: Admin completion, nginx polish, schedule overrides + +- Schedule override CRUD: `ScheduleOverride` model, `GET/POST/DELETE /api/admin/schedule/overrides`, conflict detection +- User management actions: `POST /api/admin/users` mute/unmute/ban/unban, UI buttons per user +- Inline moderation: mute/ban buttons on each chat log entry +- System config editor: `GET/PUT /api/admin/config`, restart buttons with confirmation +- nginx OPTIONS preflight: 204 on all 4 HLS locations +- nginx health: JSON response, stub_status on /nginx-status +- Redis maxmemory: 256mb allkeys-lru in both compose files +- Schedule metadata: "Now Playing" + "Up Next" overlay, public /api/schedule endpoint +- Presence indicators: green dots in chat sidebar via join/leave tracking +- BACKLOG.md synced with current implementation state + +## CI Pipeline Optimization Batch 1 (Completed 2026-08-05) + +- Read latest logs in `.gitea/logs/`, analyze cache config + runner setup, provide recommendations. +- Remove `--no-cache` from `docker compose build` in test job — rebuilds all 3 images from scratch every run. +- Remove `docker compose build` step from test job entirely — images already built in `docker` job (duplicate work). +- Fix Go module cache — `actions/setup-go` cache always misses. Fixed: removed missing `backend/playout/go.sum` from cache-dependency-path, added `go mod download` to seed cache. +- Fix or remove playlist validation step — `go run src/playlist.go` fails with missing `go.mod`, always falls through to `|| echo`. Removed dead step. +- Add Docker Buildx registry cache to `docker` and `release` jobs — cache layers in Gitea registry. + +**Details:** + +- `ci.yml`: removed `docker compose build --no-cache` from test job; removed broken playlist validation step; fixed `cache-dependency-path` to only `backend/chat/go.sum`; added `go mod download` step; added `cache-from`/`cache-to` (registry type) to all 3 `docker/build-push-action` steps. +- `release.yml`: added `cache-from`/`cache-to` (registry type) to all 3 `docker/build-push-action` steps. +- Converted nginx build in `ci.yml` from custom shell `docker buildx build` to `docker/build-push-action@v5` for consistency and cache support. +- Logs analyzed: `ci-test-678.log`, `ci-test-685.log`, `ci-docker-679.log`, `ci-docker-686.log`, `cd-deploy-688.log`. Key findings: npm cache works on gar v0.2.13+, Go cache misses due to missing go.sum path, docker-sans-cache rebuilds waste time, playlist validation dead code. + +## Color Scheme Migration (Completed 2026-08-05) + +New frontend color palette applied: `#32292f` (dark bg), `#b4c4ae` (light bg), `#587792` (brand), `#2ebfa5` (focus), `#04650d` (ok). + +- Derive full dark-theme CSS custom properties from `#32292f` base — generated `--bg-0` through `--badge-end`. +- Derive full light-theme CSS custom properties from `#b4c4ae` base — light variants in `@media (prefers-color-scheme: light)` block. +- Set `--brand: #587792`, `--brand-strong: #4a6780`, `--brand-contrast: #f4f8fb`. +- Set `--focus: #2ebfa5`, `--badge-start: #2ebfa5`, `--badge-end: #1a8c78`. +- Set `--ok: #04650d`, derived `--warn: #d49624`, `--danger: #c44a5e`. +- Set `--ink-0/1/soft` for readable contrast on dark (`#ede8e6/#c4bdb9/#8a827e`) and light (`#1a1f18/#3a4037/#5c6358`). +- Updated `--live-dot`, `--overlay-bg`, `--input-bg`, `--placeholder` for new tones. +- Updated light-theme `@media (prefers-color-scheme: light)` block. +- Replaced hardcoded admin sidebar colors (`#102a2f`, `#ecfff6`, `#ddf8ee`) with CSS custom properties. +- Updated `.live-label` to use `var(--live-dot)` instead of hardcoded `#d62828`. + +**Details:** + +- `frontend/src/style.css`: full `:root` and light-theme `@media` block replacement with 22 color tokens from new palette. +- Dark backgrounds shifted from `#0d1417` (teal-black) to `#32292f` (warm purple-brown); light from `#f4f6ef` to `#b4c4ae` (sage green). +- Brand changed teal `#2ec4b6` → steel blue `#587792`; focus `#78f2e0` → teal `#2ebfa5`; success `#5add94` → forest green `#04650d`. +- Admin sidebar `.admin-sidebar` and `.tab-trigger` hardcoded colors replaced with `var(--bg-2)`, `var(--ink-0)`, `var(--ink-1)`. + +## CI/CD Improvements (Completed 2026-08-05) + +### Build Optimization + +- Fix Go cache in CI: `go.sum` lives inside `backend/playout/` and `backend/chat/`, not workspace root. Update cache key pattern. +- Add `npm run lint` script to `frontend/package.json` (was skipped via `|| true`). +- Define missing Vite env vars (`VITE_PUBLIC_ADMIN_DEFAULT_MODERATOR`, `VITE_PUBLIC_ADMIN_DEFAULT_ROOM`, `VITE_PUBLIC_ADMIN_MODERATION_LOG_LIMIT`) or remove from HTML templates. + +### Deployment Hardening + +- Add `start_period: 30s` to chat healthcheck (needs time to connect DB + Redis). +- Add nginx health check to CD workflow (verify `http://192.168.88.40:8080` serves frontend). +- Add container log dump on health check failure for all services (not just playout). +- Test `docker image prune -f` cleanup step does not remove in-use images. + +**Details:** + +- `ci.yml`: `setup-go` cache-dependency-path now lists both `backend/playout/go.sum` and `backend/chat/go.sum` +- `package.json`: added `"lint": "tsc --noEmit"` script +- `index.html`: removed `data-admin-default-moderator`, `data-admin-default-room`, `data-admin-moderation-log-limit` attrs (JS has fallbacks via `adminConfig`) +- `docker-compose.prod.yml`: chat healthcheck `start_period` bumped 10s → 30s +- `cd.yml`: health check now verifies chat (3000) + nginx (8080) + playout (8081); logs dumped per service on failure +- `docker image prune -f --filter "until=24h"`: verified safe — Docker never removes images used by running containers + +## Chat Message Admin Improvements (Completed 2026-08-05) + +- Add `GET /api/admin/chat-log` endpoint for all chat log entries for a room, optional `?limit=` and `?before=` params. Returns JSON array. +- Add `GET /api/admin/chat-log/:id` endpoint for single chat log entry by ID. Returns JSON object. +- Add `DELETE /api/admin/chat-log/:id` endpoint. Returns 204 No Content. +- Split `frontend/admin/index.html` into main template + separate files per admin page. +- Extend `frontend/admin/chat-log.html` to display chat log entries with delete capability. +- Confirmation dialog before deleting chat log entry. +- Pagination in chat log list. +- Search/filter in chat log list. +- Add `DELETE /api/admin/chat-log/bulk` endpoint for bulk deletion with checkbox select-all UI. + +**Details:** + +- Backend: `handleAdminChatLog`, `handleAdminChatLogByID`, `handleAdminChatLogBulk` in `main.go` +- Types: `ChatLogEntry`, `ChatLogListResponse`, `ChatLogBulkDeleteRequest` +- Frontend: 9 page HTML files + dynamic `index.html` template with `loadPageContent` / `switchTab` / `bindPageEvents` +- `admin.ts`: `refreshChatLog`, `deleteChatEntry`, `bulkDeleteChatLog`, pagination (prev/next), filter (room/username/text), `chatLogState` management + +## Security Improvements (Completed 2026-08-05) + +- Added catch-all `/` route to backend router returning JSON 404 for unhandled requests. +- Verified all admin API + frontend admin endpoints require auth: Go `requireAdminAuth` middleware guards `/api/admin/*` (except login/logout/session), nginx `auth_request /_admin_auth` guards `/admin/` frontend routes. No gaps. + +## Documentation Polish (Completed 2026-08-04) + +### Architecture Documentation (docs/architecture/README.md) Updates + +- Add monitoring and alerting details (Prometheus rules, Grafana dashboards). +- Add security considerations section (auth, rate limiting, WAF). +- Add performance benchmarks / capacity planning section. +- Add disaster recovery / backup procedures. +- Add scaling strategy documentation. +- Add API documentation section (chat WebSocket, playout REST, admin REST). +- Add video normalization specification details. +- Add Cloudflare configuration reference. + +### Additional Documentation Files + +- Create API.md with WebSocket and REST API specifications. +- Create DEPLOYMENT.md with production deployment guide. +- Create MONITORING.md with Prometheus/Grafana setup guide. +- Create SECURITY.md with security considerations and best practices. +- Keep `CHANGELOG.md` aligned with documentation and release changes. + +## Frontend CSS and JS Rewrite (No Tailwind) (Completed 2026-08-04) + +### Remove Tailwind and Replace with Native CSS/JS + +- Removed Tailwind from frontend dependencies and build pipeline (`package.json`, PostCSS/Tailwind config, CSS directives). +- Removed Tailwind CDN usage from admin HTML pages and template references. +- Replaced utility-class styling with custom CSS architecture using design tokens. +- Organized styles into clear sections for viewer, admin, and shared layout. +- Responsive behavior for mobile, tablet, desktop breakpoints. +- Replaced Tailwind-dependent interaction patterns with plain JavaScript. +- Kept and improved UI behavior in navigation, forms, tables, status messaging. + +### Accessibility (Latest Guidelines) + +- Applied WCAG 2.2 AA baseline across rewritten UI. +- Semantic structure and landmarks (`header`, `nav`, `main`, `section`, proper heading order). +- Full keyboard operability for controls, tab views, interaction flows. +- Visible focus indicators with sufficient contrast and logical tab sequence. +- Text/background and component contrast meets AA thresholds. +- Accessible labels, helper text, error messaging for form controls. +- Validated ARIA usage for dynamic states only where native semantics insufficient. +- Respected reduced-motion preferences. + +### Validation and Documentation + +- Ran frontend production build, resolved all Tailwind/PostCSS directive warnings. +- Local preview smoke checks for viewer chat send, admin tab navigation, admin login states. +- Manually tested key user flows in admin and viewer pages after rewrite. +- Updated `CHANGELOG.md` with rewrite summary. + +### Dark Mode First + +- Dark-mode-first design with CSS custom properties and light-scheme override path. + +### Player Page Layout + +- Refactored viewer to single header with logo/title/subtitle and chat toggle. +- Consolidated viewer into one main content area: responsive video/chat split on desktop, stacked on mobile. +- Viewer uses full viewport dimensions, removed desktop outer scrollbars. +- Video player visible and responsive with viewport changes. +- Removed legacy "Now Playing" UI controls and related runtime update behavior. + +## Admin UI Features (Completed 2026-08-04) + +### Media Library - Full CRUD + +- Documentation: ADR-006 revised to implemented Postgres-first storage and media model decisions. +- Planning: TODO and CHANGELOG aligned with implementation progress. +- Backend: Media GORM model and AutoMigrate added. +- Backend: `ffprobe` duration extraction on upload, persisted duration metadata. +- Backend: full CRUD for `/api/admin/media` with GET/POST and GET/PUT/DELETE by id. +- Backend: schedule and dashboard endpoints extended with media duration totals and counts. +- Frontend: admin media library updated with runtime display and edit/delete/upload controls. +- Frontend: runtime display added to Schedule and Dashboard (total and per-item runtime). + +### Schedule Page - Video Runtime Display + +- Implemented after backend duration support. + +### Dashboard - Total Video Runtime + +- Implemented after backend stats support. + +### Implementation Notes Applied + +- Used existing GORM + Postgres setup. +- Installed `ffmpeg` in chat Dockerfile, used `ffprobe` via `exec.Command`. +- Media metadata stored in same DB as messages/moderation; media binaries remain on filesystem. +- Admin endpoints protected with `requireAdminAuth` middleware. +- Media listing uses DB-backed records. +- Added `formatDuration(seconds int) string` helper for HH:MM:SS formatting. + +## Upload functionality + +- Large file uploads chunked and resumable, with progress feedback. +- Uploads up to 2GB with server-side validation of file type and size. + +## Documentation Polish - Architecture README (Partial, Completed Items) + +- Updated Deployment View with CI/CD pipeline architecture and Mermaid diagram. + +## Release Validation (Completed 2026-08-04) + +### Release Workflow + +- Reviewed `.gitea/workflows/release.yml` — triggers on `v*` tags + `workflow_dispatch`, builds all 3 images. +- Fixed Nginx build context: `dockerfile:` → `file:` in `release.yml`. +- Replaced `softprops/action-gh-release@v1` (GitHub-only) with `curl` → Gitea API `POST /repos/{owner}/{repo}/releases`. +- Added VERSION empty-guard to `release.yml` — fails gracefully with error message. +- Confirmed curl API working: `POST /repos/zwitschi/linearvoid/releases` returns 201, release at `https://git.allucanget.biz/zwitschi/linearvoid/releases/tag/1.0.3`. +- Semver tags verified: `type=semver,pattern={{version}}`, `type=semver,pattern={{major}}.{{minor}}`, `type=raw,value=latest`. + +### Dependency Update Workflow + +- Replaced `peter-evans/create-pull-request@v5` (GitHub-only) with Gitea API `POST /repos/{owner}/{repo}/pulls` via `curl`. +- Added `git commit` + `git push origin HEAD:dependency-updates --force` step before PR creation. +- Handles HTTP 201 (new PR) and 409 (PR exists, force-push updated commits). + +## Documentation Polish — Root README.md (Completed 2026-08-04) + +- Consolidated badges: build status, license, version, release notes. +- Quick start section with `docker compose up --build` and viewer/admin/metrics URLs. +- Inline Mermaid architecture diagram. +- Troubleshooting table + environment variables reference table. +- API endpoint reference table (Chat, Playout, Nginx). +- Contributing summary with links to `CONTRIBUTING.md` and `LICENSE`. + +## Fix CI Docker Build Failures (Completed 2026-08-04) + +- **Root Cause**: CI `docker` job used `context: .` (repo root) for Playout build, but Dockerfile COPY paths relative to `backend/playout/`. Changed `context: .` → `context: ./backend/playout` in `.gitea/workflows/ci.yml`. +- Verified Chat image context (`./backend/chat`) correct. +- Verified Nginx image context (`.`) correct (uses `frontend/` and `infrastructure/nginx/` from root). +- Build cache resolved automatically with fresh push after context fix. + +## Metrics Endpoint (Completed) + +- Added `/metrics` to chat backend with Prometheus exposition: connected clients, stored messages, moderation actions, Redis state, active admin sessions, build info. +- Exposed `/metrics` through nginx for browser and domain access. +- Updated Prometheus scrape config to target implemented metrics endpoints only. +- Replaced placeholder Grafana dashboard queries with live chat metrics and scrape-health panels. +- Validated metrics access: `http://localhost/metrics`, `https://linearvoid.tv/metrics`, Prometheus target discovery, inside Prometheus container. + +## Admin Panel (Completed) + +- Replaced static admin mock data with real API-backed data loading. +- Added backend admin endpoints for playlist read/write, moderation actions, audit-log queries, analytics, system-health summaries, users, schedule, media management. +- Wired admin UI forms and tabs to backend endpoints. +- Session-based admin auth for `/admin/` and `/api/admin/*`, including login/logout flows. +- Moved admin panel config to `.env` variables, documented required auth settings. + +## Local Docker Validation (Completed) + +- `docker compose build` passes for `nginx`, `chat`, `playout`. +- `docker compose up --build -d` starts all services cleanly, playout healthy, nginx serving built frontend. +- Fixed local compose-network runtime defaults: `REDIS_URL=redis://redis:6379/0`, `DATABASE_URL=postgres://linearvoid:linearvoid@db:5432/linearvoid?sslmode=disable` work inside containers. +- Chat honors `DATABASE_URL`, supports SQLite and Postgres, validated against compose Postgres service. +- Local sample media and mounted playlists generate HLS output correctly. +- Fixed concat-playlist parsing: playout validation no longer passes literal `file '...'` entries to `ffprobe`. +- Fixed HLS segment naming: fresh local output uses timestamped immutable segment names. +- Admin production build emits `dist/admin/index.html`, nginx serves `/admin/` as own entrypoint, stale hashed assets return `404` instead of HTML fallback. +- Restored nginx MIME mappings: built JS/CSS assets served with browser-safe content types. + +Notes: + +- Preferred local access: `https://linearvoid.tv` and `https://linearvoid.tv/admin` through Cloudflare and Nginx Proxy Manager. +- Direct `localhost` access is fallback when public proxy path intentionally bypassed. + +## Base URL Rollout: linearvoid.tv (Completed) + +- Production URL model: app, admin, WebSocket on `https://linearvoid.tv`; HLS on dedicated `MEDIA_DOMAIN`, target `https://media.linearvoid.tv`. +- Introduced explicit public URL env vars for viewer, WebSocket, media endpoints; localhost-friendly defaults preserved for dev. +- Refactored frontend to stop hardcoding `localhost` URLs, use same-origin or env-driven endpoint resolution. +- Updated nginx config: host handling, SPA routing, `/ws`, `/hls/` match `linearvoid.tv` deployment model. +- Updated `.env.example`, `SETUP.md`, deployment docs with final DNS, Cloudflare, TLS, runtime values. +- Validated localhost dev and production domain behavior via frontend production builds using local and production public URL variables. + +Notes: + +- Viewer SPA, admin UI, chat WebSocket use same origin on `linearvoid.tv`. +- Media delivery split from app origin for Cloudflare R2 + cache-rule alignment; production target `media.linearvoid.tv` via `MEDIA_DOMAIN`. +- Local dev keeps localhost defaults; production swaps only public hostnames, not internal compose service names. +- Public frontend URL vars in root `.env` / `.env.example`: `VITE_PUBLIC_APP_URL`, `VITE_PUBLIC_WS_URL`, `VITE_PUBLIC_HLS_URL`; Vite reads from repo root through `frontend/vite.config.ts`. +- Frontend resolves WebSocket and HLS from Vite env first, falls back to same-origin `/ws` and `/hls/playlist.m3u8`. +- Nginx treats `linearvoid.tv` as canonical app host, redirects `www.linearvoid.tv`, proxies `/ws` with forwarded headers, keeps `/hls/` as local fallback. + +## Task List Maintenance - 2026-08-03 + +- Moved completed TODO items out of active queue. +- Updated architecture docs table of contents to include ADR-001 through ADR-007. +- Added ADR-002 through ADR-007 to architecture decisions section. +- Updated `SETUP.md` with latest docker-compose and environment guidance. +- Consolidated redundant active TODO items, reprioritized remaining work around `linearvoid.tv` rollout, release validation, documentation polish. + +## Project Setup & Architecture Documentation + +- Analyzed arc42 modules. +- Created clean project folder structure (frontend, backend/playout, backend/chat, backend/shared, infrastructure, scripts, monitoring). +- Created consolidated ADR-001 (merged previous drafts). +- Created ADR-002 (module structure) and ADR-003 (tech stack, languages, implicit assumptions). +- Fully populated `docs/architecture/README.md` (arc42 template with Mermaid diagrams, quality goals, deployment view, cross-cutting concepts, risks, glossary). +- Updated root `README.md` with Project Structure section and links to all ADRs. +- Initialized git repository. +- Created `.github/instructions/TODO.md` with detailed tracking. + +## Foundational Configuration Files + +- Created `.env.example` with R2, Cloudflare, Redis, PostgreSQL, ports, playout settings. +- Enhanced `.gitignore` for Node.js, Go, Docker, FFmpeg, data, build artifacts. +- Created root `docker-compose.yml` orchestrating playout, chat, nginx, redis, db services. +- Added `backend/playout/Dockerfile`, `start-playout.sh`, `.dockerignore`. +- Added `backend/chat/Dockerfile` (multi-stage Go build). + +## Create ADRs for Data Storage, Caching, and Admin Capabilities + +- Created ADR-006: Data Storage Strategy (SQLite vs PostgreSQL v1, migration path, admin interface to view/edit settings). +- Created ADR-007: Redis Usage (use cases, TTL policies, persistence, fallback behavior, admin interface to view stats, adjust rate limits/TTLs, flush keys, monitor health). +- Extended admin interface plans (from ADR-004/ADR-005) with "System Configuration" tab for Redis and storage management, full audit logging. + +**Status**: Data storage and caching decisions documented. Admin interface capabilities to adjust Redis config values planned. + +## Create ADRs for Admin Interfaces (Completed) + +- Created ADR-004: Video Admin Web Interface (upload, playlist management, schedule overrides, normalization enforcement, Redis config tab from ADR-007). +- Created ADR-005: Web Admin Interfaces for user management, moderation dashboard, analytics, system configuration. +- Extended admin UI plans with shared layout, HTMX + Tailwind, auth, audit logging, Redis/storage management. + +**Status**: Admin interface docs and implementation complete. Protected `/admin` UI included in frontend. + +## Infrastructure & Monitoring (Completed) + +- Nginx config for static serving, proxying, HLS delivery. +- Cloudflare cache rule examples and Worker templates. +- Prometheus and Grafana dashboards. +- Health checks and UptimeRobot configuration. +- Updated `docker-compose.yml` to include monitoring services. + +**Overall Status**: All tasks completed. Fully functional, self-hosted 24/7 linear video channel with synchronized HLS playback, custom chat, R2 delivery, moderation, audit logging, admin interface, monitoring, complete docs (arc42 template + ADRs 001-007). + +## Implement Playout Service (`backend/playout/`) (Completed) + +- Playlist management, normalization validation (ffprobe), continuous HLS with immutable segment naming, automatic R2 sync, health check endpoint, Docker integration. +- Updated `docker-compose.yml`, Dockerfile, start script. + +## Implement Chat Backend (`backend/chat/`) (Completed) + +- Go + Gorilla WebSocket, Redis for presence/rate limiting/pub/sub, SQLite persistence for message history and ModerationLog. +- Public rooms, private DM routing, moderation commands with immutable audit logging, health check endpoint. +- Updated `docker-compose.yml` with volume for SQLite. + +## Frontend (`frontend/`) (Completed) + +- Vite + TypeScript + Tailwind + Alpine.js. +- HLS player with hls.js, chat sidebar with room switching and private messaging, "Now Playing" overlay. +- Modern UI with Tailwind styling and Alpine.js reactivity. + +**Overall Status**: Backend and Frontend complete. Fully functional 24/7 linear video channel with synchronized playback, custom chat, R2 delivery, moderation, audit logging. Remaining: Infrastructure & Monitoring, admin interfaces (ADR-004/ADR-005), CI/CD, final polish. + +## Create SETUP.md Guide (Completed) + +- Prerequisites: Docker, Node.js, FFmpeg, Git. +- Environment Setup: `.env.example` to `.env`, R2 credentials, Cloudflare tokens, ports, Redis/PostgreSQL settings. +- Running: `docker compose up --build`, frontend dev server, health checks, monitoring (Prometheus/Grafana). +- Production Deployment: Hetzner VPS, Cloudflare cache rules, R2 bucket, systemd services. +- Troubleshooting: Common issues (R2 credentials, normalization errors, cache rules, Redis connection). + +**Status**: SETUP.md complete. Comprehensive setup docs. + +## Final Project Completion + +- All tasks completed: Infrastructure & Monitoring, admin interface, documentation, CI/CD, final polish. +- Fully functional, self-hosted 24/7 linear video channel with synchronized HLS playback, custom chat, R2 delivery, moderation, audit logging, admin interface, monitoring, complete docs (arc42 template + ADRs 001-007). + +**Overall Status**: Linear Void complete and ready for production. Run `docker compose up --build`. See `SETUP.md`. + +## CI/CD Pipeline — Full Deployment (Completed 2026-08-04) + +### CI Pipeline + +- CI workflow on Gitea Actions runner — **passed** (task 569). +- `npm ci` — 38 packages, 0 vulnerabilities. +- `npm run build` — all outputs built (index.html, admin pages, assets). +- `docker compose build --no-cache` — all 3 images (playout, chat, nginx) built. +- Multi-stage nginx Dockerfile (frontend baked in). +- Removed frontend-only image from CI & release workflows. +- All images pushed to Gitea registry. + +### CD Pipeline — Registry Login Fixed + +- Renamed all `GITEA_*` → `CI_*` vars in `.env`, `.env.example`, `docker-compose.yml`, all workflows, `SETUP.md`, `DONE.md`. +- New secrets `CI_REGISTRY_USER` and `CI_REGISTRY_TOKEN` created in Gitea. +- Old `GITEA_USERNAME` and `GITEA_TOKEN` removed. +- **Login confirmed** — `cd-deploy-568.log` shows `Login Succeeded!` using new `CI_*` secrets. + +### CD Pipeline — Push Images to Registry — FIXED ✅ + +- Added `Prepare environment` step to `cd.yml`: generates minimal `.env` inline. +- Configured `DOCKER_HOST: tcp://192.168.88.40:2375` in `cd.yml` — directs deploy/cleanup to test server at `192.168.88.40`. +- Updated health check to query `http://192.168.88.40` instead of `localhost`. +- CD login works + images pull. +- Added `.env` creation step to CD workflow. +- Documented Gitea Actions secrets setup in `SETUP.md`. +- Added secrets protection guidelines to `AGENTS.md`. +- **Fixed Prometheus mount failure**: switched to named volume `prometheus-config` + CD seeds config at deploy time. +- **Fixed playout crash-loop**: changed playlist volume from `:ro` to `:rw` so startup script can create demo playlist. +- **Fixed chat unhealthy status**: added `curl` to chat runtime Alpine image for healthcheck. +- **CD triggers after CI success**: changed CD trigger from `push` to `workflow_run` on CI completion. +- **Images built before deployment**: CI builds and pushes images, CD pulls and deploys. +- Containers start and pass health checks on `192.168.88.40`. +- `docker push git.allucanget.biz/zwitschi/linearvoid/playout:latest` +- `docker push git.allucanget.biz/zwitschi/linearvoid/chat:latest` +- `docker push git.allucanget.biz/zwitschi/linearvoid/nginx:latest` — **rebuilt with multi-stage Dockerfile, frontend bundled** ✅ + +### CD Pipeline — Track C: Run CD Workflow + +- Images pulled on runner ✅. +- `.env` fix applied — `cd.yml` creates inline config before deploy. +- Health checks pass after containers start. +- `docker image prune -f` cleanup does not remove in-use images. + +### Local Test Environment Deployment (Remote Docker Host) + +- **Configure Alpine Docker Host** (`192.168.88.40`) to listen on TCP `2375`: + - Edit `/etc/conf.d/docker`, set `DOCKER_OPTS="-H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375"`. + - Restart Docker: `rc-service docker restart`. + - Verify: `netstat -tlnp | grep 2375`. +- **Configure local env** to point to remote Docker host at `192.168.88.40:2375` via `DOCKER_HOST`. +- **Verify connection**: `docker info` confirms communication with remote host. +- **Test remote build and deploy** from source: `docker compose up -d --build` — **succeeded after removing prometheus/grafana from gitignore** ✅. +- **Test remote production-like deploy** from Gitea registry: `docker compose -f docker-compose.prod.yml pull && docker compose -f docker-compose.prod.yml up -d`. +- **Verify remote health checks**: + - Playout: `http://192.168.88.40:8081/health` + - Chat: `http://192.168.88.40:3000/health` + - Nginx: `http://192.168.88.40:8080` +- **Reset local env** by clearing `DOCKER_HOST`. + +### CI Docker Job (push to registry) + +- CI `docker` job triggers after `test` passes on `main` push. +- CI pushes `playout`, `chat`, `nginx` images to Gitea Container Registry. +- Metadata tagging validated (branch-based + `latest`). + +### Workflow Fixes & Adjustments + +- **Fix `.env` heredoc indentation in `cd.yml`**: Removed leading spaces inside heredoc so Docker Compose parses correctly. +- **Remove `type=gha` cache from `ci.yml`**: Gitea Actions doesn't support GitHub Actions cache (`type=gha`). Removed. +- **Verify health check endpoints in `cd.yml`**: Confirmed `curl` targets match actual ports (`chat:3000`, `playout:8081`). +- **Fix nginx build in CI**: Switched from `docker/build-push-action` (ignored `dockerfile` param) to shell-based `docker buildx build --file`. + +### CI/CD Pipeline Architecture Documentation + +- Added CI/CD Pipeline Architecture section to `docs/architecture/README.md` with Mermaid diagram. +- Documented pipeline sequence: CI builds/pushes → CD pulls/deploy. +- Updated Deployment View with docker-compose services and remote host details. + +## Gitea Self-Hosting Migration (Completed) + +### Gitea Instance Setup & Repository Migration + +- Verified Gitea instance at accessible and configured — Gitea 1.27.1 +- Created org/user `zwitschi` on Gitea +- Created repo `linearvoid` under `zwitschi` (`git.allucanget.biz/zwitschi/linearvoid`) +- Configured repo settings (visibility, default branch `main`, branch protection) +- Pushed existing code to Gitea remote +- Configured SSH keys for Gitea access (deploy keys for CI/CD) +- Set up Gitea Actions runner (self-hosted) +- Configured Gitea Container Registry (enable in app.ini, set up storage) + +### CI/CD Migration to Gitea Actions + +- Created `.gitea/workflows/` directory structure +- Moved `.github/workflows/ci.yml` to `.gitea/workflows/ci.yml` +- Adjusted CI workflow for Gitea Actions runner: + - Replaced `actions/checkout@v4` with Gitea-compatible checkout + - Updated `runs-on` to self-hosted runner labels + - Replaced `actions/setup-node@v4` with Gitea-compatible install + - Updated Docker image references to Gitea Container Registry + - Configured registry auth for Gitea Container Registry + - Added Gitea-specific env vars (CI_SERVER_URL, CI_REGISTRY_TOKEN) +- Created separate workflows for CI, CD, Release, Dependency updates + +### Gitea Container Registry Setup + +- Enabled Container Registry in Gitea config (app.ini) +- Configured storage backend +- Set up registry auth (token-based, OAuth2) +- Image naming convention: `git.allucanget.biz/zwitschi/linearvoid/:` +- Image retention policies (keep last N tags, cleanup old images) +- Registry cleanup cron job + +### Repository & Documentation Updates + +- Updated all GitHub references in docs to Gitea URLs (README.md, CONTRIBUTING.md, SETUP.md, docs/architecture/README.md) +- Updated docker-compose.yml to use Gitea registry images +- Updated .env.example with Gitea registry variables +- Created .gitea/ directory for workflows and config +- Updated docker-compose.prod.yml for production with Gitea registry +- Removed .github/workflows/ after migration complete + +### Infrastructure & Runner Setup + +- Provisioned self-hosted Gitea Actions runner (VM or container) +- Installed Docker, Node.js, Go, FFmpeg on runner +- Registered runner with Gitea instance +- Configured runner labels (linux) +- Set up runner auto-scaling (optional) +- Configured runner logging and monitoring diff --git a/.github/instructions/TODO.md b/.github/instructions/TODO.md new file mode 100644 index 0000000..1d4c993 --- /dev/null +++ b/.github/instructions/TODO.md @@ -0,0 +1,93 @@ +# TODO + +> Completed tasks moved to `.github/instructions/DONE.md`. Full remaining task list in `DEVELOPMENT.md`. + +## Detailed Implementation Plan + +- [x] Phase 1: unblock playout test fixtures and lock down the source-media contract + - [x] Create small compliant fixture media in `backend/playout/testdata/` for the validator and playout tests + - [x] Create invalid fixture media for codec, framerate, and audio rejection paths + - [x] Create playlist fixtures for empty, comment-only, mixed-validity, and valid playlists + - [x] Document the ffmpeg fixture-generation commands in the relevant test headers + +- [ ] Phase 2: finish frontend coverage for the live viewer experience + - [x] Add tests for HLS player startup, manifest resolution, and cross-origin/local fallback behavior + - [x] Add tests for chat send/receive, presence updates, private DM flow, and reconnect behavior + - [ ] Add responsive-layout and accessibility checks for the viewer shell and chat sidebar + - [ ] Add coverage reporting and enforce a minimum threshold in CI + +- [ ] Phase 3: finish admin frontend coverage + - [x] Add tests for playlist management, moderation flows, and dashboard rendering + - [x] Add tests for API integration points with mocked backend responses + - [ ] Add responsive-layout and accessibility checks for admin screens + - [ ] Add coverage reporting and enforce a minimum threshold in CI + +- [x] Phase 4: validate the R2 and edge-delivery path + - [x] Add an integration test for upload and retrieval of HLS objects through the R2 path + - [x] Verify segment and manifest cache headers, including immutable media and live manifest freshness + - [ ] Validate edge cache hit/miss behavior with repeat requests + - [x] Document any required stubs or mocks so the test can run in CI (putObject mock pattern documented in test) + +- [ ] Phase 5: complete manual Cloudflare and external-service setup + - [ ] Apply Cloudflare Pages, Tiered Cache, WAF, rate limiting, and HTTPS/HSTS settings in the dashboard + - [ ] Create or verify UptimeRobot monitors for health and HLS manifest availability + - [ ] Capture evidence in `docs/operations/cloudflare-uptimerobot-checklist.md` + +- [ ] Phase 6: rehearse production deployment end to end + - [ ] Provision the Hetzner host and prepare the Docker/runtime baseline + - [ ] Configure production secrets, remote deployment, and DNS cutover + - [ ] Run health, HLS, chat, and admin smoke tests against production-like infrastructure + - [ ] Verify backup creation, restore, and integrity checks on the production host + +- [ ] Phase 7: baseline scaling and performance + - [ ] Measure manifest TTFB and cache-hit distribution from multiple US regions + - [x] Prepare WebSocket load-test scaffolding and summary artifacts (`ws-chat.yml`, `run-loadtests.ps1`, `summarize-artillery.ps1`) + - [ ] Execute WebSocket throughput and chat-latency benchmarks under growing connection counts + - [x] Prepare Admin API benchmark scaffolding and endpoint summary artifacts (`admin-api.js`, `run-loadtests.ps1`, `summarize-k6.ps1`) + - [ ] Execute Admin API benchmarks and identify slow paths + - [x] Add saturation thresholds to monitoring (chat p95 latency + Redis memory usage) + - [x] Add cache-hit threshold alert operationalization to monitoring + +## Status Update (2026-08-10) + +- Phase 1 (test fixtures) complete. +- Phase 2 (viewer frontend): player startup, URL resolution, chat send/receive, reconnect, presence, DM flow, message rendering all covered (27 tests). +- Phase 3 (admin frontend): dashboard rendering, API integration, playlist management all covered (31 tests total). +- Phase 4 (R2 validation) partially complete: SyncToR2 cache headers + integration tests done; edge cache hit/miss remains manual. +- Phase 5 (Cloudflare): Pages deploy workflow (`.gitea/workflows/pages-deploy.yml`) created; Pages/Workers setup docs written; manual dashboard tasks remain. +- Phase 7: Loadtest scaffolding complete; `run-loadtests.ps1` auto-generates ws-chat-summary.txt and admin-api-endpoint-summary.txt; live benchmark execution pending. +- Version tagged 1.0.11; go vet clean. + +## Remaining (summary) + +- **Test Fixtures**: media + playlist fixtures created; ffmpeg generation commands documented in test headers +- **User Frontend Tests**: video player, chat integration, responsive, accessibility, coverage +- **Admin Frontend Tests**: playlist UI, moderation UI, metrics dashboards, coverage +- **Cloudflare R2 Integration Test**: upload + cache header validation done (SyncToR2 with CacheControl + ContentType); edge cache hit/miss validation remains manual +- **Cloudflare Config**: Cloudflare Pages deploy workflow and setup docs created; manual dashboard setup (Tiered Cache, WAF, rate limiting, HTTPS/HSTS) remains pending; Pages project must be created in dashboard first. +- **External Services**: UptimeRobot monitors + alerts +- **Production Deployment**: Hetzner provisioning, CD pipeline, DNS cutover, smoke test +- **Scaling & Performance**: TTFB baseline and live benchmark runs still pending; loadtest scaffolding now includes Artillery/k6 helpers with auto summaries plus worksheet field mapping; chat p95, Redis saturation, and cache-hit threshold alerting are implemented + +See `DEVELOPMENT.md` for detailed checklist. + +## Streaming Pipeline Gaps (2026-08-10 audit) + +STREAMING.md contract vs playlist.go/main.ts implementation. See Backlog for full details. + +- [ ] Continuous R2 sync during playout (HIGH) — currently only on FFmpeg exit +- [ ] Globally unique segment naming with run ID (HIGH) — currently strftime only +- [ ] Atomic manifest publication with temp key (MEDIUM) +- [ ] CORS headers on HLS responses from media.linearvoid.tv (MEDIUM) +- [ ] Player error recovery: retries, drift, reconnecting state (MEDIUM) +- [ ] Manifest Cache-Control: add must-revalidate + s-maxage split (LOW) +- [ ] Health endpoint: add active item, revision, run ID, segment info (LOW) + +## Feature List (Future Enhancements) + +- [ ] Admin backend + - [ ] Analytics: implement a settings entry to set matomo url and page id +- [ ] User frontend + - [ ] Chat: add user registration and login flows + - [ ] Chat: private DM history and search + - [ ] Analytics: add javascript snippet to send pageview events to matomo diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index 64d983b..e3b0f9d 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -52,8 +52,9 @@ ## Cloudflare Configuration (manual — Cloudflare dashboard) > Evidence tracker: `docs/operations/cloudflare-uptimerobot-checklist.md` +> Cloudflare Pages deploy workflow created (`.gitea/workflows/pages-deploy.yml`); setup docs written (`docs/operations/cloudflare-pages-setup.md`, `docs/operations/gitea-cloudflare.pages.md`). -- [ ] Cloudflare Pages: link Gitea repo, build `cd frontend && npm run build`, output `frontend/dist` +- [ ] Cloudflare Pages: create project in dashboard, link custom domain, or deploy via Gitea Actions Direct Upload - [ ] Tiered Cache: enable Smart Tiered Cache, validate `Cf-Cache-Status: HIT` from upper tier - [ ] WAF: enable OWASP Managed Ruleset, set to Simulate → review 48h → Block, enable Bot Fight Mode - [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action @@ -89,7 +90,27 @@ - [ ] Measure TTFB: WebPageTest from 3+ US regions, target <4s warm cache, document baseline - [ ] Load-test WebSocket: `artillery` or Go script, ramp 10→100→500→1000 connections, measure p95 latency + errors (scaffolding complete; benchmark execution pending) - [ ] Benchmark admin API: `wrk`/`k6` against `/api/admin/*`, target p95 <1s, identify slow queries (scaffolding complete; benchmark execution pending) +- [x] Load-test scaffolding: ws-chat.yml, admin-api.js, run-loadtests.ps1, summarizers, README +- [x] Cache-hit ratio exporter (scripts/cache-hit-ratio.js + .ps1) +- [x] Saturation alerts: chat p95 latency, Redis memory usage, cache-hit ratio - [ ] Track cache hit ratio: Cloudflare GraphQL → Grafana panel, alert <90% -- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, chat p95 >1s/5min, CPU >80%/10min, Redis memory >85%/10min +- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min (chat p95, Redis memory, cache-hit already done) - [ ] Phase 2 plan: document boundaries (playout CPU, chat latency, DB state, monitoring), estimate per-service resources - [ ] Phase 3 plan: research second Hetzner location, design warm standby + DNS failover + DB/Redis replication + +## Streaming Pipeline (2026-08-10 audit) + +Gaps between STREAMING.md contract and playlist.go/main.ts implementation: + +- [ ] **Continuous R2 sync during playout (HIGH)**: `syncLatestToR2()` runs only after FFmpeg exits. Needs per-segment upload via filesystem watch or FFmpeg progress pipe. +- [ ] **Globally unique segment naming (HIGH)**: Add run/session UUID + monotonic sequence. Currently `strftime` only — collisions possible. +- [ ] **Atomic manifest publication (MEDIUM)**: Upload to temp key then atomically replace `live/playlist.m3u8`. +- [ ] **Manifest Cache-Control alignment (LOW)**: Add `must-revalidate` and `s-maxage`/`max-age` split. +- [ ] **CORS headers on HLS responses (MEDIUM)**: Add `Access-Control-Allow-Origin` to Worker or R2 custom domain. +- [ ] **Player error recovery (MEDIUM)**: Add `hls.on(Hls.Events.ERROR, ...)` with retries, drift, reconnecting state. +- [ ] **Health endpoint detail (LOW)**: Add active item, playlist revision, run ID, segment info, R2 pub timestamp. +- [x] Source media contract + validation — implemented +- [x] fMP4 HLS packaging (6s, 24 window) — implemented +- [x] Slate fallback + recovery — implemented +- [x] R2 cache headers (immutable segments, short-lived manifest) — implemented +- [x] Cloudflare Worker immutable `.m4s` headers — implemented