From 1cc44282bc1826d55dfcbcbc694b0cd906c64472 Mon Sep 17 00:00:00 2001 From: zwitschi Date: Sat, 15 Aug 2026 16:36:17 +0200 Subject: [PATCH] docs: clean up TODO and DONE, updating BACKLOG --- .github/instructions/BACKLOG.md | 73 +- .github/instructions/DONE.md | 1189 +------------------------------ .github/instructions/TODO.md | 64 -- 3 files changed, 81 insertions(+), 1245 deletions(-) diff --git a/.github/instructions/BACKLOG.md b/.github/instructions/BACKLOG.md index c01a6af..e2ccc4d 100644 --- a/.github/instructions/BACKLOG.md +++ b/.github/instructions/BACKLOG.md @@ -2,6 +2,70 @@ > All current work is planned in [TODO](./TODO.md) (2026-08-13). This file lists backlog items; new unplanned work is added here. +## Chat improvements (remaining) + +- [ ] Manual smoke: viewer join/leave shows a 3s flash (not a chat row); reconnect <5s shows no duplicate; admin chat-log table has no system rows + +## Playout improvements (remaining) + +- [ ] Test in Firefox: live playback, rebuffer/recovery, chat sync; record quirks in `docs/STREAMING.md` +- [ ] Rebuild + run playout; confirm `playlist.m3u8` lists ~12 segments at 2s each +- [ ] Manual: verify viewer latency and cache-hit behavior after the change + +## Setup phases remaining + +### Phase 5: complete manual Cloudflare and external-service setup + +- [ ] Apply Cloudflare Pages, Tiered Cache, WAF, rate limiting, and HTTPS/HSTS settings in the dashboard +- [ ] Cloudflare Pages: link custom domain and cut over frontend hosting +- [ ] Enable Smart Tiered Cache; validate `Cf-Cache-Status: HIT` from upper tier +- [ ] WAF: enable OWASP Managed Ruleset (Simulate → review 48h → Block), enable Bot Fight Mode +- [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action +- [ ] HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2; enforce at edge and origin +- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval) +- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword `#EXTM3U`, 5min) +- [ ] UptimeRobot: alert on 2 consecutive failures via email + Discord/Slack webhook +- [ ] Capture evidence in `docs/operations/cloudflare-uptimerobot-checklist.md` + +### Phase 6: rehearse production deployment end to end + +- [ ] Provision Hetzner CX22 (2 vCPU, 4 GB, 40 GB) in Ashburn VA, Ubuntu 24.04 +- [ ] Install Docker Engine + Compose plugin; open firewall: 80/443 public, 2375 runner-only +- [ ] Configure Docker daemon for remote TLS access (`-H tcp://0.0.0.0:2376` with certs) +- [ ] Create `/srv/linearvoid/data/{playlist,media,hls,chat,slate}` directories +- [ ] Add `CI_PROD_*` secrets to Gitea (DOCKER_HOST, R2 keys, admin password, DB password, Grafana password) +- [ ] Trigger CD workflow with `DOCKER_HOST` pointing to Hetzner IP +- [ ] Verify all 11 containers healthy: `/health`, `/hls/playlist.m3u8`, WebSocket +- [ ] Update Cloudflare A record `linearvoid.tv` → Hetzner IP, SSL Full (strict); test HTTPS + WSS +- [ ] Smoke test: viewer SPA loads, HLS plays, chat connects, admin login → upload → playlist → playout +- [ ] Set up Prometheus alerting notifications + UptimeRobot on production URL +- [ ] Run `scripts/backup-db.sh` + `scripts/verify-backups.sh` on production host +- [ ] Document production IP, SSH access, rollback procedure, scaling thresholds + +### Phase 7: baseline scaling and performance + +- [ ] Execute WebSocket load tests — ramp 10→100→500→1000 connections; measure p95 latency + errors +- [ ] Execute admin API benchmarks — `wrk`/`k6` against `/api/admin/*`; target p95 <1s +- [ ] Measure manifest TTFB and cache-hit distribution from multiple US regions (WebPageTest 3+ regions, <4s warm cache) +- [ ] Tune manifest TTL to 0.5× segment duration; validate `Cf-Cache-Status` distribution +- [ ] Track cache-hit ratio and R2 ops in production (Cloudflare GraphQL → Grafana panel, alert <90%) +- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min +- [ ] Plan Phase 2 service separation (playout CPU, chat latency, DB state, monitoring) +- [ ] Plan Phase 3 multi-origin resilience (second Hetzner location, warm standby, DNS failover, DB/Redis replication) + +### Phase 8: close out remaining test and validation gaps + +- [ ] Admin frontend: user moderation tools tests (mute/ban/report, view moderation logs) +- [ ] Frontend: coverage for all critical user + admin components and interactions +- [ ] Frontend: document test cases and expected outcomes +- [ ] Backend R2: test retrieval of segments/manifests via Cloudflare edge +- [ ] Backend R2: validate edge cache hit/miss behavior with repeat requests (`scripts/validate-edge-cache.sh`) + +### Playout + +- [ ] Configure an R2 lifecycle rule on the `hls/` prefix as defense-in-depth (manual) +- [ ] Full-loop E2E (2-file playlist, assert `Looping...` with no `CRASHED` across the boundary) — manual via `scripts/test-playout-e2e.sh` + ## Cloudflare Configuration ### DNS & Pages @@ -102,12 +166,3 @@ - [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min - [ ] Plan Phase 2 service separation — document boundaries (playout CPU, chat latency, DB state, monitoring), estimate per-service resources - [ ] Plan Phase 3 multi-origin resilience — research second Hetzner location, design warm standby + DNS failover + DB/Redis replication - -## Feature List (Future Enhancements) - -- [ ] Admin backend - - [ ] Analytics: implement a settings entry to set matomo url and page id -- [ ] User frontend - - [ ] Chat: add user registration and login flows - - [ ] Chat: private DM history and search - - [ ] Analytics: add javascript snippet to send pageview events to matomo diff --git a/.github/instructions/DONE.md b/.github/instructions/DONE.md index a058700..4a4d9ab 100644 --- a/.github/instructions/DONE.md +++ b/.github/instructions/DONE.md @@ -1,1184 +1,29 @@ # DONE -> Completed tasks from [TODO](./TODO.md). Historical record for [CHANGELOG](./CHANGELOG.md), [README](./README.md), [architecture docs](./architecture/README.md). +> Completed tasks from [TODO](./TODO.md). Product, feature, and release history lives in [CHANGELOG](./CHANGELOG.md); this file keeps only engineering and ops notes not captured there. -## Session 2026-08-07 — Unit Tests +## Test infrastructure & refactors (2026-08-07) -- Chat service refactor: `main.go` split into 8 packages (config, auth, db, handlers, ws, media, ratelimit, metrics) -- `db.RedisClient` + `db.PubSub` interfaces defined for mocking -- Config tests: 10 env-var tests, table-driven, 100% coverage -- Auth tests: token generation, session CRUD, cookie security, middleware, 97.8% coverage -- DB tests: SQLite in-memory CRUD for all 4 models, 14.3% coverage -- Handlers tests: 35 tests covering login/logout/session/overview/playlist/moderation/analytics/system/config/users/schedule/media/chat-log, 47.2% coverage -- WebSocket tests: 15 tests covering connect/send/dm/moderation/banned/muted/history/presence/pubsub, 79.6% coverage -- Media tests: formatDuration, extensions, mime, upload IDs, playlist parsing, 36.8% coverage -- Ratelimit tests: message/login/WS rate limits with mock Redis, 90.3% coverage -- Metrics tests: Prometheus output validation, all 9 metrics present, 100% coverage -- Playout tests: 12 tests (LoadPlaylist, keyframeInterval, HealthHandler, slate escalation, reset/activate/deactivate) -- Frontend test framework: vitest + jsdom installed -- User frontend smoke tests: 6 tests (chat toggle, sendMessage, switchRoom, DM toggle) -- Admin frontend smoke tests: 5 tests (DOM structure, upload progress, fingerprint, constants, import stability) -- CI pipeline updated: `go test -race -cover` with 50% threshold -- SQLite driver switched to pure-Go `github.com/glebarez/sqlite` (CGO-free) -- Total: 78 tests across Go + frontend, all passing +- Chat service `main.go` split into 8 packages (config, auth, db, handlers, ws, media, ratelimit, metrics); `db.RedisClient` + `db.PubSub` interfaces added for mocking. +- 78 tests total (Go + frontend), all passing. Coverage: auth 97.8%, config 100%, metrics 100%, ratelimit 90.3%, ws 79.6%, handlers 47.2%, media 36.8%, db 14.3%. +- Playout: 12 tests. Frontend: vitest + jsdom, 6 user + 5 admin smoke tests. +- CI: `go test -race -cover` with a 50% coverage threshold; SQLite switched to pure-Go `github.com/glebarez/sqlite` (CGO-free, works on Windows). +- Frontend coverage enforcement: vitest v8 provider, 40% statements/lines + 30% branches/functions thresholds, `test:coverage` script + CI step. -## Session 2026-08-07 — Runtime Validation +## Backups -- Check `monitoring/prometheus/prometheus.yml` for completeness - - Make sure all 7 scrape targets are included (chat, playout, nginx, redis, postgres, node, prometheus) - - Check `monitoring/prometheus/alerts.yml` for completeness - - Make sure configuration for prometheus is mounted into container via named volume -- Use folder `monitoring/grafana/` to create ini file for Grafana, set `GF_PATHS_CONFIG` environment variable to point to a different location than `/etc/grafana/grafana.ini` if needed - - Set default password from `GRAFANA_PASSWORD` env var, disable user signups - - Configure Prometheus data source in Grafana ini file - - Include dashboards from `monitoring/grafana/dashboards/` directory to be imported into Grafana on startup -- **CD trigger parallel**: Change `cd.yml` from `workflow_run: [CI]` to `push: branches: [main]` so deploy runs in parallel with CI, not after -- **CD failure fix**: Remove `environment: production` from `cd.yml` deploy job (no Gitea environment configured, causes silent job abort after Buildx) -- **Remove redundant CI release job**: Delete `release` job from `ci.yml` — fully handled by `release.yml` -- **Tag format**: Use `*.*.*` tags (e.g. `1.0.9`) — `release.yml` pattern is `*.*.*`, not `v*.*.*`. Fix existing `v1.0.x` tags to `1.0.x` -- **Update DEPLOYMENT.md**: Document new parallel pipeline flow (push → CI + CD run simultaneously; tag push → Release builds versioned images) -- Add sections to `SETUP.md` under `## Cloudflare` for detailed instructions: - - DNS: `linearvoid.tv` apex + `www` CNAME → Cloudflare Pages - - DNS: `media.linearvoid.tv` for split-domain HLS delivery - - R2 bucket `linearvoid-media` with `enam` location hint - - Cache Rules: `.ts/.m4s` immutable long TTL (31536000), `.m3u8` short TTL (2s, revalidate) - - Cloudflare Pages: link Gitea repo, build `cd frontend && npm run build`, output `frontend/dist` - - Tiered Cache: enable Smart Tiered Cache, validate `Cf-Cache-Status: HIT` from upper tier - - WAF: enable OWASP Managed Ruleset, set to Simulate → review 48h → Block, enable Bot Fight Mode - - Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action - - HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2 - - HTTPS/WSS: enforce at edge (Cloudflare) and origin (nginx), redirect HTTP → HTTPS - - UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval) - - UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword: `#EXTM3U`, 5min) - - Alert on 2 consecutive failures, notify via email + Discord/Slack webhook -- Check all known URLs for metrics and their availability: - - Prometheus: `http://:9090` (targets, graph, alerts) - - Grafana: `http://:3001` (default: admin/admin, change via `GRAFANA_PASSWORD`) - - Chat metrics: `http://chat:3000/metrics` from Prometheus container - - Playout metrics: `http://playout:8081/metrics` from Prometheus container - - Nginx metrics: `http://nginx-exporter:9113/metrics` from Prometheus container - - Redis metrics: `http://redis-exporter:9121/metrics` from Prometheus container - - Postgres metrics: `http://postgres-exporter:9187/metrics` from Prometheus container - - Node metrics: `http://node-exporter:9100/metrics` from Prometheus container -- Make sure Prometheus config is mounted into container via named volume `prometheus-config` (not bind mount) -- Make sure Prometheus alerting rules are configured and working (8 rules: ChatDown, PlayoutDown, RedisDown, etc.) -- Verify that all Prometheus targets are up and reporting metrics -- Verify that Grafana dashboards are imported and displaying metrics correctly -- Video normalization: test with MOV, MKV, variable framerate, different audio layouts - - Evidence: all normalized outputs probed as `h264`, `yuv420p`, `30/1`, `aac`, `48000`, `2ch`. -- Video normalization: verify output plays back-to-back in continuous concat - - Evidence: concat remux reported `CONCAT_OK`; output duration `25.344667s`. -- Viewer SPA: test responsive layout on iOS Safari + Android Chrome - - Evidence: viewport emulation (`390x844`, `412x915`) showed video/chat/toggle present; fixed 8px horizontal overflow in `frontend/src/style.css` (`100vw` -> `100%`). - - Limitation: no physical-device Safari/Chrome test in this run. -- Redis: test restart, verify sessions + presence data survive - - Evidence: admin session HTTP `200` before/after Redis restart; seeded presence key visible before/after restart; Redis health returned to `healthy`. +- `scripts/backup-db.sh` (daily pg_dump → R2, 14-day retention), `scripts/backup-weekly.sh` (weekly full dump + integrity check, 8-week retention), `scripts/verify-backups.sh` (download → temp Postgres → restore + row counts). -### Runtime Tests (all 4) +## Gitea self-hosting migration -- Video normalization validated on MOV, MKV, VFR concat source, and 5.1 audio source. -- All normalized outputs verified as H.264/yuv420p/30fps + AAC 48kHz stereo. -- Back-to-back concat continuity check passed (`CONCAT_OK`, duration ~25.34s). -- Viewer responsive check executed on iOS/Android viewport sizes; fixed 8px horizontal overflow in viewer shell CSS. -- Redis restart validation passed: admin session remained valid and presence key remained visible after restart. +- Migrated repo + CI/CD from GitHub to Gitea Actions (`git.allucanget.biz/zwitschi/linearvoid`): self-hosted runner, Container Registry convention `git.allucanget.biz/zwitschi/linearvoid/:`, secrets renamed `GITEA_*` → `CI_*`. -### Documentation Polish +## Streaming pipeline gaps (2026-08-10) -- API.md: add error codes table (full status code matrix) + rate limit headers -- DEPLOYMENT.md: prereqs checklist, first-deploy walkthrough, restart/view-logs/rollback, troubleshooting -- MONITORING.md: architecture diagram, Grafana URL + default creds, dashboard tour, alert descriptions, scrape targets -- SECURITY.md: threat model diagram, TLS/HSTS config, auth flow, network isolation, secrets management -- Level 2 building blocks: detail Playout/Storage/Frontend/Chat/Proxy with Mermaid + code locations + persistence -- Level 3 building blocks: module breakdowns (parser, manager, uploader, handler, registry, router, etc.) -- Recovery runbook: document quarterly disaster recovery drill - - Evidence: expanded `docs/incident-runbook.md` with prerequisites, simulation steps, validation checks, and pass/fail criteria. -- Recovery runbook: rehearse quarterly disaster recovery drill - - Manual step: execute full drill on target environment and capture RTO/RPO evidence. +- Continuous R2 sync via `fsnotify` (`watchAndPublishSegments`, ~7s delay); run-ID + monotonic segment naming (`r-{ISOTIME}/seg-{seq}.m4s`); atomic manifest publication (upload `.tmp` → `CopyObject`); enriched `/health` (run_id, active_item, playlist_revision, last_segment_seq, last_r2_pub, consecutive_failures); player error recovery (5 retries, 2s→32s exponential backoff, reconnecting overlay). +- CORS on `media.linearvoid.tv` remains infrastructure-only (Cloudflare R2 config, not code). -## Session 2026-08-06 — Documentation Adjustments, Playout, Normalization, Chat, Viewer SPA, Admin, Docker, CI/CD +## Bug fixes not in CHANGELOG -### Documentation Adjustments: Clarify test vs production (all 7) - -- Update `docker-compose.yml` comments: targets local dev, not production -- Update `docker-compose.prod.yml` comments: `192.168.88.40` is test host, not Hetzner -- Update `.gitea/workflows/cd.yml` comments: `DOCKER_HOST=tcp://192.168.88.40:2375` is test lab, prod TBD -- Update `DEPLOYMENT.md`: document two environments (test lab `192.168.88.0/24` vs production Hetzner VPS) -- Update `SETUP.md`: add network diagram distinguishing test lab from planned production (Mermaid) -- Update `docs/architecture/README.md` deployment view: CD target is test, production pending -- Update `README.md` setup instructions: flag 192.168.88.x as internal test network - -### Documentation Adjustments: Production target gap (all 5) - -- Add section to `DEPLOYMENT.md`: "Production Target (Hetzner)" — status: not provisioned -- Document planned Hetzner CX22 specs (CX22, Ashburn VA, ~€7-10/mo) -- Gitea Actions CD workflow supports `DOCKER_HOST` env var — ready for prod IP swap -- Document DNS cutover: Cloudflare A record → Hetzner public IP when provisioned -- Document firewall/security group requirements for production Hetzner VPS - -### Documentation Adjustments: CI/CD audit (all 5) - -- Review `.gitea/workflows/cd.yml`: confirm `DOCKER_HOST` only variable needing change for prod -- Review `.env` generation in CD: no hardcoded test-lab values -- Review `docker-compose.prod.yml`: no test-lab-specific config (IPs, hostnames) -- Document single-variable change: `DOCKER_HOST=tcp://:2375` -- Add production environment to Gitea Actions environments -- Fixed CD health check to use `docker exec` instead of hardcoded IP curls - -### Playout & Delivery: FFmpeg + HLS playout (all 8) - -- Install FFmpeg (>=5.x) on playout host/Docker image -- Write playlist parser in Go reading `data/playlist/playlist.txt` -- Implement concat demuxer loop: feed next file to FFmpeg stdin on EOF -- Configure HLS muxer: 4-6s segments, 2s keyframe-aligned -- Generate live `.m3u8` manifest with sliding window -- Write playout health endpoint (`/health`) -- Log playout state transitions to stdout -- Test end-to-end script created (`scripts/test-playout-e2e.sh`) - -### Playout & Delivery: Immutable segment naming (all 5) - -- Design naming scheme: `segment_%Y%m%d_%H%M%S.m4s` (strftime-based) -- Configure FFmpeg `-hls_segment_filename` with unique pattern -- Verify no filename collisions (strftime+seconds+resetHLSOutput) -- Set `Cache-Control: public, max-age=31536000, immutable` on segment paths -- Test: same playlist run twice → no filename reuse (code review confirmed) - -### Playout & Delivery: R2 upload (all 7) - -- Add AWS SDK dependency to playout service -- Configure R2 endpoint, access key, secret key via env vars -- Implement upload goroutine: watch segment output dir, upload new files -- Upload updated `.m3u8` manifest after each segment write -- Set correct Content-Type headers -- Handle upload retries (3 attempts, exponential backoff) -- Log upload success/failure with segment name - -### Playout & Delivery: Emergency slate fallback (all 7) - -- Create slate video: 10s loop, branded "We'll Be Right Back" (`scripts/generate-slate.sh`) -- Normalize slate to same profile (H.264/AAC, 30fps, 2s keyframes) -- Package slate as pre-rendered HLS segments -- Failure detector: if FFmpeg exits non-zero, track restart count -- On failure: switch manifest to pre-rendered slate segments (activateSlate) -- On recovery: resume playlist from last known position (deactivateSlate) -- Log all failover events with reason and duration - -### Playout & Delivery: Auto-restart loop (all 5) - -- Wrap FFmpeg in supervisor goroutine -- On unexpected exit: log exit code, wait 2s, restart -- Track restart count; >5 in 60s escalates to slate fallback -- Health endpoint reflects restart state: `degraded` vs `healthy` -- Expose Prometheus metric: `linearvoid_playout_restarts_total` - -### Video Normalization: ffprobe verification gate (all 6) - -- ValidateVideo with strict JSON parsing in playlist.go -- Validate: codec_name=h264/aac, r_frame_rate=30/1, resolution, pix_fmt=yuv420p, sample_rate=48000 -- Validate keyframe interval <=2s (`validateKeyframeInterval` via `-show_frames`) -- Return structured JSON result: `{ valid: bool, errors: string[] }` (ValidateVideoStrict) -- Exit code 0 for pass, non-zero for fail with actionable error text -- Integrate as pre-upload gate in admin API (`validateMediaNormalization` in chat backend) - -### Video Normalization: Enforce in admin UI (all 5) - -- On admin upload submit: run ffprobe verification gate server-side -- If validation fails: sets `needs_normalization` status with field-level errors -- Non-conforming files flagged in media library -- Validation result stored in media metadata record (`tags` field) -- Applied to single upload and chunked upload completion paths - -### Chat Service: WebSocket backend (all 7) - -- Go module with gorilla/websocket -- Message envelope struct -- `/ws` endpoint with query params -- Room registry with mutex -- Broadcast to all clients in room -- Handle disconnect + presence change notification -- Prometheus metrics: `linearvoid_chat_clients_connected`, `linearvoid_chat_messages_total` - -### Chat Service: Private DM routing (all 4) - -- `type: "private"` with `to` field -- DM routing via clientMeta map (targeted delivery) -- Offline DMs persisted to DB, appear in history on next join -- `linearvoid_chat_dms_total` counter metric - -### Chat Service: Redis presence tracking (all 5) - -- On connect: `SET presence:room:username "online" EX 60` -- On disconnect: `DEL presence:room:username` -- Periodic heartbeat every 30s + WebSocket ping -- Broadcast join/leave system messages -- Admin users endpoint returns presence data from Redis - -### Chat Service: Rate limiting (all 3) - -- `INCR rate:{username}` + `EXPIRE` windowing -- > threshold within window returns error to sender only -- Configurable via `CHAT_RATE_LIMIT_MAX`, `CHAT_RATE_LIMIT_WINDOW_S` env vars - -### Chat Service: Message persistence (all 6) - -- Messages table with GORM auto-migrate -- SQLite v1 with PostgreSQL driver swap support -- Repository interface: SaveMessage, GetRoomHistory, GetDMHistory -- Async persistence (goroutine), doesn't block broadcast -- Chat history push on room join (last 50 messages via `pushChatHistory`) -- PostgreSQL migration path designed (ADR-006) - -### Chat Service: Moderation commands (all 6) - -- Parse `/mod`, `/mute`, `/ban` prefix -- Verify sender has moderator role (checks admin username) -- `/mute {user} {duration_m}`: Redis key with TTL -- `/ban {user}`: Redis key, reject connections -- Persist to ModerationLog table -- Broadcast system message to room - -### Chat Service: Moderation audit log (all 5) - -- ModerationLog table: id, action, target, moderator, room, reason, timestamp -- Log all moderation commands -- Log admin moderation actions from dashboard -- Expose via admin API: `GET /api/admin/moderation/logs` -- Append-only, never delete - -### Chat Service: Graceful Redis degradation (all 5) - -- Redis calls wrapped with timeout + error handling -- Redis down: skip presence, skip rate limiting, skip pub/sub -- Warning logged, `linearvoid_chat_redis_up` gauge exposed -- Core chat works without Redis -- Auto-reconnect with exponential backoff (2s→60s, `redisReconnectLoop`) - -### Viewer SPA: hls.js player (all 6) - -- Vite + TypeScript project -- hls.js dependency -- HLS player component with native + hls.js fallback -- HLS events: MANIFEST_PARSED, LEVEL_LOADED, ERROR with recovery -- Buffering indicator + error fallback -- Production build with `vite build` - -### Viewer SPA: Chat sidebar (all 7) - -- WebSocket connection with env-configurable URL -- Message list with auto-scroll + history on join -- Message input with Enter-to-send -- Room switcher -- Private DM panel -- Online presence indicators (green dots, `renderPresence`) - -### Viewer SPA: Schedule metadata (all 4) - -- Fetch from `GET /api/schedule` every 30s (`refreshSchedule`) -- "Now Playing" + "Up Next" overlay below player -- Public `/api/schedule` endpoint added to chat backend -- Updates on poll interval - -### Admin: Login page (all 5) - -- Login form with fetch POST to `/api/admin/login` -- Session cookie received -- Redirect to `/admin/` on success -- Inline error message on failure -- 401 redirects to login page - -### Admin: Dashboard (all 4) - -- Key metrics: viewers, messages, moderation, media runtime -- Quick links to all admin sections -- Auto-refresh via polling - -### Admin: Video upload (all 5) - -- Chunked upload: init → chunk → complete -- Progress bar per upload -- Normalization verification gate on upload (sets `needs_normalization` status) -- Resumable chunks for >1GB files - -### Admin: Playlist management (all 6) - -- Drag-to-reorder sortable list (`renderPlaylistSortable`) -- Add/remove videos -- Save via `PUT /api/admin/playlist` -- Validation warnings (missing files, needs_normalization) -- Individual entry removal buttons - -### Admin: Analytics (all 5) - -- Charts and stat cards -- Playback stats display -- Time range selector: 1h, 6h, 24h, 7d -- Data from backend admin analytics API -- CSV export (`/api/admin/chat-log/export`) - -### Admin: Moderation export - -- Export chat log as CSV (`/api/admin/chat-log/export` endpoint) - -### nginx: auth_request (all 6) - -- `/_admin_auth` internal location -- `auth_request /_admin_auth` on `/admin/` and `/api/admin/` -- 401 → `@admin_login_redirect` (302) -- JSON 401 for API unauthorized -- Cookie forwarding to auth endpoint - -### nginx: WebSocket upgrade (all 6) - -- `/ws` location proxying to chat -- `proxy_http_version 1.1` -- `Upgrade` + `Connection` headers -- `proxy_read_timeout 3600s` -- `X-Forwarded-For` + `X-Real-IP` passed - -### Docker: All Dockerfiles (all 4) - -- playout: alpine + ffmpeg + Go binary + slate dir -- chat: multi-stage Go build + ffmpeg for ffprobe -- nginx: multi-stage with bundled frontend -- HEALTHCHECK on all - -### Docker: docker-compose.yml (all 5) - -- All 7 services defined -- `unless-stopped` restart policy -- Internal + public networks -- Named volumes for all persistent data -- `depends_on` with healthchecks - -### Docker: docker-compose.prod.yml (all 5) - -- Log rotation (json-file 10m/3) -- Resource limits -- Bind-mount host directories -- Minimal port exposure -- Remote Docker host support - -### CI/CD: CI workflow (all 5) - -- Trigger on push/PR to main -- Test job: lint + build + Go validate -- Docker job: login + build + push with Buildx -- Registry cache for layers -- Metadata tagging (branch, semver, latest) - -### CI/CD: CD workflow (all 9) - -- Trigger on CI success or manual dispatch -- Login to Gitea registry -- `.env` generation from secrets -- Prometheus config seeding -- `DOCKER_HOST` for remote deploy -- Pull + up with remove-orphans -- Health check loop (60s timeout) -- Log dump on failure -- IP-agnostic health checks (`docker exec`) - -### CI/CD: .env generation (all 4) - -- Secrets in Gitea (CI\_ prefix) -- CD writes `.env` on target -- `.env` in `.gitignore` -- Validation before deploy - -### CI/CD: Prometheus config seeding (all 3) - -- CD copies config into named volume -- Auto-created on first compose up -- Read-only mount in container - -### CI/CD: Health check validation (all 5) - -- 5s initial delay -- Retry loop with 5s intervals -- 12 attempts max (60s) -- Log dump on timeout -- Success message with versions - -### Cloudflare: Worker (all 4) - -- Worker deployed on segment paths -- `Cache-Control: public, max-age=31536000, immutable` header -- Cookie/query string removal for cache key normalization -- Pass-through for non-segment requests - -### Monitoring: Prometheus chat scrape (all 4) - -- `scrape_interval: 15s` -- `chat` job → `chat:3000` -- `prometheus` job → `prometheus:9090` -- Metrics endpoint verified - -### Monitoring: Grafana dashboards (all 6) - -- App dashboard: clients, messages, moderation, Redis, sessions, scrape health -- Prometheus operational dashboard imported -- Datasource provisioned - -### Security: Admin cookie-session auth (all 5) - -- `crypto/rand` session ID generation -- Redis session store (in-memory fallback working) -- Cookie: HttpOnly, SameSite=Strict, Secure (X-Forwarded-Proto aware) -- Auth middleware (`requireAdminAuth`) -- Logout clears session - -### Security: Session expiration (all 4) - -- Configurable TTL (`ADMIN_SESSION_TTL_HOURS`) -- Expiry check in middleware -- Session refresh in last 25% of TTL -- Env var for duration - -### Data: PostgreSQL setup (all 4) - -- SQLite v1 per ADR-006 -- GORM auto-migrate with schema -- Tables: messages, moderation_logs, media -- PostgreSQL driver ready (swap DATABASE_URL) - -### API Documentation (4 of 5) - -- WebSocket Chat API -- Playout REST API -- Admin REST API -- Example requests/responses with `curl` - -## CI Pipeline Optimization (Completed 2026-08-05) - -### Session 2026-08-06 — Batch 3: Monitoring, Security, Backups, Runbook - -- 4 Prometheus exporters added to compose: nginx (9113), redis (9121), postgres (9187), node (9100) -- `monitoring/prometheus/alerts.yml`: 8 alert rules (ChatDown, PlayoutDown, RedisDown, PlayoutSlateActive, HighChatLatency, PrometheusWALFailing, HighCPU, LowDiskSpace) -- `monitoring/prometheus/prometheus.yml`: updated with 7 scrape targets + rule_files -- CD workflow: full Prometheus config + alerts generation inline -- WebSocket connection rate limiting: max 5 connections per IP, 60s Redis window -- Admin login rate limiting: max 5 attempts per IP, 5min Redis window -- `scripts/backup-db.sh`: daily pg_dump → R2, 14-day retention -- `scripts/backup-weekly.sh`: weekly full dump + integrity check, 8-week retention -- `scripts/verify-backups.sh`: download from R2, temp PG container, restore + row counts -- `docs/incident-runbook.md`: 5-tier triage (edge→origin→playout→chat→data), symptoms, mitigations, escalation -- CI fix: removed GitHub cache from setup-go/setup-node (Gitea runner timeout), added go vet - -### Session 2026-08-06 — Batch 2: Admin completion, nginx polish, schedule overrides - -- Schedule override CRUD: `ScheduleOverride` model, `GET/POST/DELETE /api/admin/schedule/overrides`, conflict detection -- User management actions: `POST /api/admin/users` mute/unmute/ban/unban, UI buttons per user -- Inline moderation: mute/ban buttons on each chat log entry -- System config editor: `GET/PUT /api/admin/config`, restart buttons with confirmation -- nginx OPTIONS preflight: 204 on all 4 HLS locations -- nginx health: JSON response, stub_status on /nginx-status -- Redis maxmemory: 256mb allkeys-lru in both compose files -- Schedule metadata: "Now Playing" + "Up Next" overlay, public /api/schedule endpoint -- Presence indicators: green dots in chat sidebar via join/leave tracking -- BACKLOG.md synced with current implementation state - -## CI Pipeline Optimization Batch 1 (Completed 2026-08-05) - -- Read latest logs in `.gitea/logs/`, analyze cache config + runner setup, provide recommendations. -- Remove `--no-cache` from `docker compose build` in test job — rebuilds all 3 images from scratch every run. -- Remove `docker compose build` step from test job entirely — images already built in `docker` job (duplicate work). -- Fix Go module cache — `actions/setup-go` cache always misses. Fixed: removed missing `backend/playout/go.sum` from cache-dependency-path, added `go mod download` to seed cache. -- Fix or remove playlist validation step — `go run src/playlist.go` fails with missing `go.mod`, always falls through to `|| echo`. Removed dead step. -- Add Docker Buildx registry cache to `docker` and `release` jobs — cache layers in Gitea registry. - -**Details:** - -- `ci.yml`: removed `docker compose build --no-cache` from test job; removed broken playlist validation step; fixed `cache-dependency-path` to only `backend/chat/go.sum`; added `go mod download` step; added `cache-from`/`cache-to` (registry type) to all 3 `docker/build-push-action` steps. -- `release.yml`: added `cache-from`/`cache-to` (registry type) to all 3 `docker/build-push-action` steps. -- Converted nginx build in `ci.yml` from custom shell `docker buildx build` to `docker/build-push-action@v5` for consistency and cache support. -- Logs analyzed: `ci-test-678.log`, `ci-test-685.log`, `ci-docker-679.log`, `ci-docker-686.log`, `cd-deploy-688.log`. Key findings: npm cache works on gar v0.2.13+, Go cache misses due to missing go.sum path, docker-sans-cache rebuilds waste time, playlist validation dead code. - -## Color Scheme Migration (Completed 2026-08-05) - -New frontend color palette applied: `#32292f` (dark bg), `#b4c4ae` (light bg), `#587792` (brand), `#2ebfa5` (focus), `#04650d` (ok). - -- Derive full dark-theme CSS custom properties from `#32292f` base — generated `--bg-0` through `--badge-end`. -- Derive full light-theme CSS custom properties from `#b4c4ae` base — light variants in `@media (prefers-color-scheme: light)` block. -- Set `--brand: #587792`, `--brand-strong: #4a6780`, `--brand-contrast: #f4f8fb`. -- Set `--focus: #2ebfa5`, `--badge-start: #2ebfa5`, `--badge-end: #1a8c78`. -- Set `--ok: #04650d`, derived `--warn: #d49624`, `--danger: #c44a5e`. -- Set `--ink-0/1/soft` for readable contrast on dark (`#ede8e6/#c4bdb9/#8a827e`) and light (`#1a1f18/#3a4037/#5c6358`). -- Updated `--live-dot`, `--overlay-bg`, `--input-bg`, `--placeholder` for new tones. -- Updated light-theme `@media (prefers-color-scheme: light)` block. -- Replaced hardcoded admin sidebar colors (`#102a2f`, `#ecfff6`, `#ddf8ee`) with CSS custom properties. -- Updated `.live-label` to use `var(--live-dot)` instead of hardcoded `#d62828`. - -**Details:** - -- `frontend/src/style.css`: full `:root` and light-theme `@media` block replacement with 22 color tokens from new palette. -- Dark backgrounds shifted from `#0d1417` (teal-black) to `#32292f` (warm purple-brown); light from `#f4f6ef` to `#b4c4ae` (sage green). -- Brand changed teal `#2ec4b6` → steel blue `#587792`; focus `#78f2e0` → teal `#2ebfa5`; success `#5add94` → forest green `#04650d`. -- Admin sidebar `.admin-sidebar` and `.tab-trigger` hardcoded colors replaced with `var(--bg-2)`, `var(--ink-0)`, `var(--ink-1)`. - -## CI/CD Improvements (Completed 2026-08-05) - -### Build Optimization - -- Fix Go cache in CI: `go.sum` lives inside `backend/playout/` and `backend/chat/`, not workspace root. Update cache key pattern. -- Add `npm run lint` script to `frontend/package.json` (was skipped via `|| true`). -- Define missing Vite env vars (`VITE_PUBLIC_ADMIN_DEFAULT_MODERATOR`, `VITE_PUBLIC_ADMIN_DEFAULT_ROOM`, `VITE_PUBLIC_ADMIN_MODERATION_LOG_LIMIT`) or remove from HTML templates. - -### Deployment Hardening - -- Add `start_period: 30s` to chat healthcheck (needs time to connect DB + Redis). -- Add nginx health check to CD workflow (verify `http://192.168.88.40:8080` serves frontend). -- Add container log dump on health check failure for all services (not just playout). -- Test `docker image prune -f` cleanup step does not remove in-use images. - -**Details:** - -- `ci.yml`: `setup-go` cache-dependency-path now lists both `backend/playout/go.sum` and `backend/chat/go.sum` -- `package.json`: added `"lint": "tsc --noEmit"` script -- `index.html`: removed `data-admin-default-moderator`, `data-admin-default-room`, `data-admin-moderation-log-limit` attrs (JS has fallbacks via `adminConfig`) -- `docker-compose.prod.yml`: chat healthcheck `start_period` bumped 10s → 30s -- `cd.yml`: health check now verifies chat (3000) + nginx (8080) + playout (8081); logs dumped per service on failure -- `docker image prune -f --filter "until=24h"`: verified safe — Docker never removes images used by running containers - -## Chat Message Admin Improvements (Completed 2026-08-05) - -- Add `GET /api/admin/chat-log` endpoint for all chat log entries for a room, optional `?limit=` and `?before=` params. Returns JSON array. -- Add `GET /api/admin/chat-log/:id` endpoint for single chat log entry by ID. Returns JSON object. -- Add `DELETE /api/admin/chat-log/:id` endpoint. Returns 204 No Content. -- Split `frontend/admin/index.html` into main template + separate files per admin page. -- Extend `frontend/admin/chat-log.html` to display chat log entries with delete capability. -- Confirmation dialog before deleting chat log entry. -- Pagination in chat log list. -- Search/filter in chat log list. -- Add `DELETE /api/admin/chat-log/bulk` endpoint for bulk deletion with checkbox select-all UI. - -**Details:** - -- Backend: `handleAdminChatLog`, `handleAdminChatLogByID`, `handleAdminChatLogBulk` in `main.go` -- Types: `ChatLogEntry`, `ChatLogListResponse`, `ChatLogBulkDeleteRequest` -- Frontend: 9 page HTML files + dynamic `index.html` template with `loadPageContent` / `switchTab` / `bindPageEvents` -- `admin.ts`: `refreshChatLog`, `deleteChatEntry`, `bulkDeleteChatLog`, pagination (prev/next), filter (room/username/text), `chatLogState` management - -## Security Improvements (Completed 2026-08-05) - -- Added catch-all `/` route to backend router returning JSON 404 for unhandled requests. -- Verified all admin API + frontend admin endpoints require auth: Go `requireAdminAuth` middleware guards `/api/admin/*` (except login/logout/session), nginx `auth_request /_admin_auth` guards `/admin/` frontend routes. No gaps. - -## Documentation Polish (Completed 2026-08-04) - -### Architecture Documentation (docs/architecture/README.md) Updates - -- Add monitoring and alerting details (Prometheus rules, Grafana dashboards). -- Add security considerations section (auth, rate limiting, WAF). -- Add performance benchmarks / capacity planning section. -- Add disaster recovery / backup procedures. -- Add scaling strategy documentation. -- Add API documentation section (chat WebSocket, playout REST, admin REST). -- Add video normalization specification details. -- Add Cloudflare configuration reference. - -### Additional Documentation Files - -- Create API.md with WebSocket and REST API specifications. -- Create DEPLOYMENT.md with production deployment guide. -- Create MONITORING.md with Prometheus/Grafana setup guide. -- Create SECURITY.md with security considerations and best practices. -- Keep `CHANGELOG.md` aligned with documentation and release changes. - -## Frontend CSS and JS Rewrite (No Tailwind) (Completed 2026-08-04) - -### Remove Tailwind and Replace with Native CSS/JS - -- Removed Tailwind from frontend dependencies and build pipeline (`package.json`, PostCSS/Tailwind config, CSS directives). -- Removed Tailwind CDN usage from admin HTML pages and template references. -- Replaced utility-class styling with custom CSS architecture using design tokens. -- Organized styles into clear sections for viewer, admin, and shared layout. -- Responsive behavior for mobile, tablet, desktop breakpoints. -- Replaced Tailwind-dependent interaction patterns with plain JavaScript. -- Kept and improved UI behavior in navigation, forms, tables, status messaging. - -### Accessibility (Latest Guidelines) - -- Applied WCAG 2.2 AA baseline across rewritten UI. -- Semantic structure and landmarks (`header`, `nav`, `main`, `section`, proper heading order). -- Full keyboard operability for controls, tab views, interaction flows. -- Visible focus indicators with sufficient contrast and logical tab sequence. -- Text/background and component contrast meets AA thresholds. -- Accessible labels, helper text, error messaging for form controls. -- Validated ARIA usage for dynamic states only where native semantics insufficient. -- Respected reduced-motion preferences. - -### Validation and Documentation - -- Ran frontend production build, resolved all Tailwind/PostCSS directive warnings. -- Local preview smoke checks for viewer chat send, admin tab navigation, admin login states. -- Manually tested key user flows in admin and viewer pages after rewrite. -- Updated `CHANGELOG.md` with rewrite summary. - -### Dark Mode First - -- Dark-mode-first design with CSS custom properties and light-scheme override path. - -### Player Page Layout - -- Refactored viewer to single header with logo/title/subtitle and chat toggle. -- Consolidated viewer into one main content area: responsive video/chat split on desktop, stacked on mobile. -- Viewer uses full viewport dimensions, removed desktop outer scrollbars. -- Video player visible and responsive with viewport changes. -- Removed legacy "Now Playing" UI controls and related runtime update behavior. - -## Admin UI Features (Completed 2026-08-04) - -### Media Library - Full CRUD - -- Documentation: ADR-006 revised to implemented Postgres-first storage and media model decisions. -- Planning: TODO and CHANGELOG aligned with implementation progress. -- Backend: Media GORM model and AutoMigrate added. -- Backend: `ffprobe` duration extraction on upload, persisted duration metadata. -- Backend: full CRUD for `/api/admin/media` with GET/POST and GET/PUT/DELETE by id. -- Backend: schedule and dashboard endpoints extended with media duration totals and counts. -- Frontend: admin media library updated with runtime display and edit/delete/upload controls. -- Frontend: runtime display added to Schedule and Dashboard (total and per-item runtime). - -### Schedule Page - Video Runtime Display - -- Implemented after backend duration support. - -### Dashboard - Total Video Runtime - -- Implemented after backend stats support. - -### Implementation Notes Applied - -- Used existing GORM + Postgres setup. -- Installed `ffmpeg` in chat Dockerfile, used `ffprobe` via `exec.Command`. -- Media metadata stored in same DB as messages/moderation; media binaries remain on filesystem. -- Admin endpoints protected with `requireAdminAuth` middleware. -- Media listing uses DB-backed records. -- Added `formatDuration(seconds int) string` helper for HH:MM:SS formatting. - -## Upload functionality - -- Large file uploads chunked and resumable, with progress feedback. -- Uploads up to 2GB with server-side validation of file type and size. - -## Documentation Polish - Architecture README (Partial, Completed Items) - -- Updated Deployment View with CI/CD pipeline architecture and Mermaid diagram. - -## Release Validation (Completed 2026-08-04) - -### Release Workflow - -- Reviewed `.gitea/workflows/release.yml` — triggers on `v*` tags + `workflow_dispatch`, builds all 3 images. -- Fixed Nginx build context: `dockerfile:` → `file:` in `release.yml`. -- Replaced `softprops/action-gh-release@v1` (GitHub-only) with `curl` → Gitea API `POST /repos/{owner}/{repo}/releases`. -- Added VERSION empty-guard to `release.yml` — fails gracefully with error message. -- Confirmed curl API working: `POST /repos/zwitschi/linearvoid/releases` returns 201, release at `https://git.allucanget.biz/zwitschi/linearvoid/releases/tag/1.0.3`. -- Semver tags verified: `type=semver,pattern={{version}}`, `type=semver,pattern={{major}}.{{minor}}`, `type=raw,value=latest`. - -### Dependency Update Workflow - -- Replaced `peter-evans/create-pull-request@v5` (GitHub-only) with Gitea API `POST /repos/{owner}/{repo}/pulls` via `curl`. -- Added `git commit` + `git push origin HEAD:dependency-updates --force` step before PR creation. -- Handles HTTP 201 (new PR) and 409 (PR exists, force-push updated commits). - -## Documentation Polish — Root README.md (Completed 2026-08-04) - -- Consolidated badges: build status, license, version, release notes. -- Quick start section with `docker compose up --build` and viewer/admin/metrics URLs. -- Inline Mermaid architecture diagram. -- Troubleshooting table + environment variables reference table. -- API endpoint reference table (Chat, Playout, Nginx). -- Contributing summary with links to `CONTRIBUTING.md` and `LICENSE`. - -## Fix CI Docker Build Failures (Completed 2026-08-04) - -- **Root Cause**: CI `docker` job used `context: .` (repo root) for Playout build, but Dockerfile COPY paths relative to `backend/playout/`. Changed `context: .` → `context: ./backend/playout` in `.gitea/workflows/ci.yml`. -- Verified Chat image context (`./backend/chat`) correct. -- Verified Nginx image context (`.`) correct (uses `frontend/` and `infrastructure/nginx/` from root). -- Build cache resolved automatically with fresh push after context fix. - -## Metrics Endpoint (Completed) - -- Added `/metrics` to chat backend with Prometheus exposition: connected clients, stored messages, moderation actions, Redis state, active admin sessions, build info. -- Exposed `/metrics` through nginx for browser and domain access. -- Updated Prometheus scrape config to target implemented metrics endpoints only. -- Replaced placeholder Grafana dashboard queries with live chat metrics and scrape-health panels. -- Validated metrics access: `http://localhost/metrics`, `https://linearvoid.tv/metrics`, Prometheus target discovery, inside Prometheus container. - -## Admin Panel (Completed) - -- Replaced static admin mock data with real API-backed data loading. -- Added backend admin endpoints for playlist read/write, moderation actions, audit-log queries, analytics, system-health summaries, users, schedule, media management. -- Wired admin UI forms and tabs to backend endpoints. -- Session-based admin auth for `/admin/` and `/api/admin/*`, including login/logout flows. -- Moved admin panel config to `.env` variables, documented required auth settings. - -## Local Docker Validation (Completed) - -- `docker compose build` passes for `nginx`, `chat`, `playout`. -- `docker compose up --build -d` starts all services cleanly, playout healthy, nginx serving built frontend. -- Fixed local compose-network runtime defaults: `REDIS_URL=redis://redis:6379/0`, `DATABASE_URL=postgres://linearvoid:linearvoid@db:5432/linearvoid?sslmode=disable` work inside containers. -- Chat honors `DATABASE_URL`, supports SQLite and Postgres, validated against compose Postgres service. -- Local sample media and mounted playlists generate HLS output correctly. -- Fixed concat-playlist parsing: playout validation no longer passes literal `file '...'` entries to `ffprobe`. -- Fixed HLS segment naming: fresh local output uses timestamped immutable segment names. -- Admin production build emits `dist/admin/index.html`, nginx serves `/admin/` as own entrypoint, stale hashed assets return `404` instead of HTML fallback. -- Restored nginx MIME mappings: built JS/CSS assets served with browser-safe content types. - -Notes: - -- Preferred local access: `https://linearvoid.tv` and `https://linearvoid.tv/admin` through Cloudflare and Nginx Proxy Manager. -- Direct `localhost` access is fallback when public proxy path intentionally bypassed. - -## Base URL Rollout: linearvoid.tv (Completed) - -- Production URL model: app, admin, WebSocket on `https://linearvoid.tv`; HLS on dedicated `MEDIA_DOMAIN`, target `https://media.linearvoid.tv`. -- Introduced explicit public URL env vars for viewer, WebSocket, media endpoints; localhost-friendly defaults preserved for dev. -- Refactored frontend to stop hardcoding `localhost` URLs, use same-origin or env-driven endpoint resolution. -- Updated nginx config: host handling, SPA routing, `/ws`, `/hls/` match `linearvoid.tv` deployment model. -- Updated `.env.example`, `SETUP.md`, deployment docs with final DNS, Cloudflare, TLS, runtime values. -- Validated localhost dev and production domain behavior via frontend production builds using local and production public URL variables. - -Notes: - -- Viewer SPA, admin UI, chat WebSocket use same origin on `linearvoid.tv`. -- Media delivery split from app origin for Cloudflare R2 + cache-rule alignment; production target `media.linearvoid.tv` via `MEDIA_DOMAIN`. -- Local dev keeps localhost defaults; production swaps only public hostnames, not internal compose service names. -- Public frontend URL vars in root `.env` / `.env.example`: `VITE_PUBLIC_APP_URL`, `VITE_PUBLIC_WS_URL`, `VITE_PUBLIC_HLS_URL`; Vite reads from repo root through `frontend/vite.config.ts`. -- Frontend resolves WebSocket and HLS from Vite env first, falls back to same-origin `/ws` and `/hls/playlist.m3u8`. -- Nginx treats `linearvoid.tv` as canonical app host, redirects `www.linearvoid.tv`, proxies `/ws` with forwarded headers, keeps `/hls/` as local fallback. - -## Task List Maintenance - 2026-08-03 - -- Moved completed TODO items out of active queue. -- Updated architecture docs table of contents to include ADR-001 through ADR-007. -- Added ADR-002 through ADR-007 to architecture decisions section. -- Updated `SETUP.md` with latest docker-compose and environment guidance. -- Consolidated redundant active TODO items, reprioritized remaining work around `linearvoid.tv` rollout, release validation, documentation polish. - -## Project Setup & Architecture Documentation - -- Analyzed arc42 modules. -- Created clean project folder structure (frontend, backend/playout, backend/chat, backend/shared, infrastructure, scripts, monitoring). -- Created consolidated ADR-001 (merged previous drafts). -- Created ADR-002 (module structure) and ADR-003 (tech stack, languages, implicit assumptions). -- Fully populated `docs/architecture/README.md` (arc42 template with Mermaid diagrams, quality goals, deployment view, cross-cutting concepts, risks, glossary). -- Updated root `README.md` with Project Structure section and links to all ADRs. -- Initialized git repository. -- Created `.github/instructions/TODO.md` with detailed tracking. - -## Foundational Configuration Files - -- Created `.env.example` with R2, Cloudflare, Redis, PostgreSQL, ports, playout settings. -- Enhanced `.gitignore` for Node.js, Go, Docker, FFmpeg, data, build artifacts. -- Created root `docker-compose.yml` orchestrating playout, chat, nginx, redis, db services. -- Added `backend/playout/Dockerfile`, `start-playout.sh`, `.dockerignore`. -- Added `backend/chat/Dockerfile` (multi-stage Go build). - -## Create ADRs for Data Storage, Caching, and Admin Capabilities - -- Created ADR-006: Data Storage Strategy (SQLite vs PostgreSQL v1, migration path, admin interface to view/edit settings). -- Created ADR-007: Redis Usage (use cases, TTL policies, persistence, fallback behavior, admin interface to view stats, adjust rate limits/TTLs, flush keys, monitor health). -- Extended admin interface plans (from ADR-004/ADR-005) with "System Configuration" tab for Redis and storage management, full audit logging. - -**Status**: Data storage and caching decisions documented. Admin interface capabilities to adjust Redis config values planned. - -## Create ADRs for Admin Interfaces (Completed) - -- Created ADR-004: Video Admin Web Interface (upload, playlist management, schedule overrides, normalization enforcement, Redis config tab from ADR-007). -- Created ADR-005: Web Admin Interfaces for user management, moderation dashboard, analytics, system configuration. -- Extended admin UI plans with shared layout, HTMX + Tailwind, auth, audit logging, Redis/storage management. - -**Status**: Admin interface docs and implementation complete. Protected `/admin` UI included in frontend. - -## Infrastructure & Monitoring (Completed) - -- Nginx config for static serving, proxying, HLS delivery. -- Cloudflare cache rule examples and Worker templates. -- Prometheus and Grafana dashboards. -- Health checks and UptimeRobot configuration. -- Updated `docker-compose.yml` to include monitoring services. - -**Overall Status**: All tasks completed. Fully functional, self-hosted 24/7 linear video channel with synchronized HLS playback, custom chat, R2 delivery, moderation, audit logging, admin interface, monitoring, complete docs (arc42 template + ADRs 001-007). - -## Implement Playout Service (`backend/playout/`) (Completed) - -- Playlist management, normalization validation (ffprobe), continuous HLS with immutable segment naming, automatic R2 sync, health check endpoint, Docker integration. -- Updated `docker-compose.yml`, Dockerfile, start script. - -## Implement Chat Backend (`backend/chat/`) (Completed) - -- Go + Gorilla WebSocket, Redis for presence/rate limiting/pub/sub, SQLite persistence for message history and ModerationLog. -- Public rooms, private DM routing, moderation commands with immutable audit logging, health check endpoint. -- Updated `docker-compose.yml` with volume for SQLite. - -## Frontend (`frontend/`) (Completed) - -- Vite + TypeScript + Tailwind + Alpine.js. -- HLS player with hls.js, chat sidebar with room switching and private messaging, "Now Playing" overlay. -- Modern UI with Tailwind styling and Alpine.js reactivity. - -**Overall Status**: Backend and Frontend complete. Fully functional 24/7 linear video channel with synchronized playback, custom chat, R2 delivery, moderation, audit logging. Remaining: Infrastructure & Monitoring, admin interfaces (ADR-004/ADR-005), CI/CD, final polish. - -## Create SETUP.md Guide (Completed) - -- Prerequisites: Docker, Node.js, FFmpeg, Git. -- Environment Setup: `.env.example` to `.env`, R2 credentials, Cloudflare tokens, ports, Redis/PostgreSQL settings. -- Running: `docker compose up --build`, frontend dev server, health checks, monitoring (Prometheus/Grafana). -- Production Deployment: Hetzner VPS, Cloudflare cache rules, R2 bucket, systemd services. -- Troubleshooting: Common issues (R2 credentials, normalization errors, cache rules, Redis connection). - -**Status**: SETUP.md complete. Comprehensive setup docs. - -## Final Project Completion - -- All tasks completed: Infrastructure & Monitoring, admin interface, documentation, CI/CD, final polish. -- Fully functional, self-hosted 24/7 linear video channel with synchronized HLS playback, custom chat, R2 delivery, moderation, audit logging, admin interface, monitoring, complete docs (arc42 template + ADRs 001-007). - -**Overall Status**: Linear Void complete and ready for production. Run `docker compose up --build`. See `SETUP.md`. - -## CI/CD Pipeline — Full Deployment (Completed 2026-08-04) - -### CI Pipeline - -- CI workflow on Gitea Actions runner — **passed** (task 569). -- `npm ci` — 38 packages, 0 vulnerabilities. -- `npm run build` — all outputs built (index.html, admin pages, assets). -- `docker compose build --no-cache` — all 3 images (playout, chat, nginx) built. -- Multi-stage nginx Dockerfile (frontend baked in). -- Removed frontend-only image from CI & release workflows. -- All images pushed to Gitea registry. - -### CD Pipeline — Registry Login Fixed - -- Renamed all `GITEA_*` → `CI_*` vars in `.env`, `.env.example`, `docker-compose.yml`, all workflows, `SETUP.md`, `DONE.md`. -- New secrets `CI_REGISTRY_USER` and `CI_REGISTRY_TOKEN` created in Gitea. -- Old `GITEA_USERNAME` and `GITEA_TOKEN` removed. -- **Login confirmed** — `cd-deploy-568.log` shows `Login Succeeded!` using new `CI_*` secrets. - -### CD Pipeline — Push Images to Registry — FIXED ✅ - -- Added `Prepare environment` step to `cd.yml`: generates minimal `.env` inline. -- Configured `DOCKER_HOST: tcp://192.168.88.40:2375` in `cd.yml` — directs deploy/cleanup to test server at `192.168.88.40`. -- Updated health check to query `http://192.168.88.40` instead of `localhost`. -- CD login works + images pull. -- Added `.env` creation step to CD workflow. -- Documented Gitea Actions secrets setup in `SETUP.md`. -- Added secrets protection guidelines to `AGENTS.md`. -- **Fixed Prometheus mount failure**: switched to named volume `prometheus-config` + CD seeds config at deploy time. -- **Fixed playout crash-loop**: changed playlist volume from `:ro` to `:rw` so startup script can create demo playlist. -- **Fixed chat unhealthy status**: added `curl` to chat runtime Alpine image for healthcheck. -- **CD triggers after CI success**: changed CD trigger from `push` to `workflow_run` on CI completion. -- **Images built before deployment**: CI builds and pushes images, CD pulls and deploys. -- Containers start and pass health checks on `192.168.88.40`. -- `docker push git.allucanget.biz/zwitschi/linearvoid/playout:latest` -- `docker push git.allucanget.biz/zwitschi/linearvoid/chat:latest` -- `docker push git.allucanget.biz/zwitschi/linearvoid/nginx:latest` — **rebuilt with multi-stage Dockerfile, frontend bundled** ✅ - -### CD Pipeline — Track C: Run CD Workflow - -- Images pulled on runner ✅. -- `.env` fix applied — `cd.yml` creates inline config before deploy. -- Health checks pass after containers start. -- `docker image prune -f` cleanup does not remove in-use images. - -### Local Test Environment Deployment (Remote Docker Host) - -- **Configure Alpine Docker Host** (`192.168.88.40`) to listen on TCP `2375`: - - Edit `/etc/conf.d/docker`, set `DOCKER_OPTS="-H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375"`. - - Restart Docker: `rc-service docker restart`. - - Verify: `netstat -tlnp | grep 2375`. -- **Configure local env** to point to remote Docker host at `192.168.88.40:2375` via `DOCKER_HOST`. -- **Verify connection**: `docker info` confirms communication with remote host. -- **Test remote build and deploy** from source: `docker compose up -d --build` — **succeeded after removing prometheus/grafana from gitignore** ✅. -- **Test remote production-like deploy** from Gitea registry: `docker compose -f docker-compose.prod.yml pull && docker compose -f docker-compose.prod.yml up -d`. -- **Verify remote health checks**: - - Playout: `http://192.168.88.40:8081/health` - - Chat: `http://192.168.88.40:3000/health` - - Nginx: `http://192.168.88.40:8080` -- **Reset local env** by clearing `DOCKER_HOST`. - -### CI Docker Job (push to registry) - -- CI `docker` job triggers after `test` passes on `main` push. -- CI pushes `playout`, `chat`, `nginx` images to Gitea Container Registry. -- Metadata tagging validated (branch-based + `latest`). - -### Workflow Fixes & Adjustments - -- **Fix `.env` heredoc indentation in `cd.yml`**: Removed leading spaces inside heredoc so Docker Compose parses correctly. -- **Remove `type=gha` cache from `ci.yml`**: Gitea Actions doesn't support GitHub Actions cache (`type=gha`). Removed. -- **Verify health check endpoints in `cd.yml`**: Confirmed `curl` targets match actual ports (`chat:3000`, `playout:8081`). -- **Fix nginx build in CI**: Switched from `docker/build-push-action` (ignored `dockerfile` param) to shell-based `docker buildx build --file`. - -### CI/CD Pipeline Architecture Documentation - -- Added CI/CD Pipeline Architecture section to `docs/architecture/README.md` with Mermaid diagram. -- Documented pipeline sequence: CI builds/pushes → CD pulls/deploy. -- Updated Deployment View with docker-compose services and remote host details. - -## Gitea Self-Hosting Migration (Completed) - -### Gitea Instance Setup & Repository Migration - -- Verified Gitea instance at accessible and configured — Gitea 1.27.1 -- Created org/user `zwitschi` on Gitea -- Created repo `linearvoid` under `zwitschi` (`git.allucanget.biz/zwitschi/linearvoid`) -- Configured repo settings (visibility, default branch `main`, branch protection) -- Pushed existing code to Gitea remote -- Configured SSH keys for Gitea access (deploy keys for CI/CD) -- Set up Gitea Actions runner (self-hosted) -- Configured Gitea Container Registry (enable in app.ini, set up storage) - -### CI/CD Migration to Gitea Actions - -- Created `.gitea/workflows/` directory structure -- Moved `.github/workflows/ci.yml` to `.gitea/workflows/ci.yml` -- Adjusted CI workflow for Gitea Actions runner: - - Replaced `actions/checkout@v4` with Gitea-compatible checkout - - Updated `runs-on` to self-hosted runner labels - - Replaced `actions/setup-node@v4` with Gitea-compatible install - - Updated Docker image references to Gitea Container Registry - - Configured registry auth for Gitea Container Registry - - Added Gitea-specific env vars (CI_SERVER_URL, CI_REGISTRY_TOKEN) -- Created separate workflows for CI, CD, Release, Dependency updates - -### Gitea Container Registry Setup - -- Enabled Container Registry in Gitea config (app.ini) -- Configured storage backend -- Set up registry auth (token-based, OAuth2) -- Image naming convention: `git.allucanget.biz/zwitschi/linearvoid/:` -- Image retention policies (keep last N tags, cleanup old images) -- Registry cleanup cron job - -### Repository & Documentation Updates - -- Updated all GitHub references in docs to Gitea URLs (README.md, CONTRIBUTING.md, SETUP.md, docs/architecture/README.md) -- Updated docker-compose.yml to use Gitea registry images -- Updated .env.example with Gitea registry variables -- Created .gitea/ directory for workflows and config -- Updated docker-compose.prod.yml for production with Gitea registry -- Removed .github/workflows/ after migration complete - -### Infrastructure & Runner Setup - -- Provisioned self-hosted Gitea Actions runner (VM or container) -- Installed Docker, Node.js, Go, FFmpeg on runner -- Registered runner with Gitea instance -- Configured runner labels (linux) -- Set up runner auto-scaling (optional) -- Configured runner logging and monitoring - -## Session 2026-08-10 — STREAMING.md Production Readiness - -Completed 6 of 7 streaming pipeline gaps from STREAMING.md contract audit: - -- **Continuous R2 sync (HIGH)** — Added `watchAndPublishSegments()` using `fsnotify` to upload segments in real-time (~7s delay), replacing batch-on-exit pattern. -- **Globally unique segment naming (HIGH)** — Added `initRunID()` generating `r-ISOTIME` IDs and `seg-{seq}.m4s` monotonic naming. Replaced `strftime` collision-prone pattern. -- **Atomic manifest publication (MEDIUM)** — Added `publishManifest()` uploading to `playlist.m3u8.tmp` then `CopyObject` to final key. -- **Manifest Cache-Control contract (LOW)** — Updated to `public, max-age=1, s-maxage=2, must-revalidate`. -- **Player error recovery (MEDIUM)** — Added `hls.on(ERROR)` handler with bounded retries (5x), exponential backoff (2s→32s), reconnecting overlay with spinner. -- **Health endpoint detail (LOW)** — Enriched `/health` with `run_id`, `active_item`, `playlist_revision`, `last_segment_seq`, `last_r2_pub`, `consecutive_failures`, and `unhealthy` detection. -- [ ] CORS on `media.linearvoid.tv` — Remaining infrastructure-only (Cloudflare R2 CORS configuration, not code). - -**Files changed:** `backend/playout/src/playlist.go` (+171/-35), `playlist_test.go` (+51/-7), `go.mod` (+2), `frontend/src/main.ts` (+76/-2), `main.test.ts` (+43/-1), `style.css` (+37), `index.html` (+5). -**Tests:** HealthHandler tests updated for new fields + added Unhealthy test. Player error recovery test added. All existing tests preserved. - -## Session 2026-08-10 — Phase 2+3 Frontend Coverage - -Completed Phase 2 (viewer) and Phase 3 (admin) responsive/accessibility tests + coverage: - -- **Viewer accessibility (5 tests):** video controls/autoplay/playsinline attributes, chat panel aria-labels (live region + panel label), chat toggle aria-expanded state, message input label association, reconnecting overlay aria-live=assertive + role=alert. -- **Viewer responsive (3 tests):** chat-collapsed CSS class toggle, viewer-content CSS grid display, now-playing aria-live=polite. -- **Admin accessibility (3 tests):** nav aria-label, active tab aria-current=page, nav button focusability (no tabindex/disabled). -- **Admin responsive (2 tests):** tab content visibility (display not none), essential tab presence. -- **Coverage enforcement:** Added vitest config with v8 provider, 40% statements/lines + 30% branches/functions thresholds. Added `test:coverage` script and CI step. - -**Files:** `frontend/src/main.test.ts` (+65), `frontend/src/admin.test.ts` (+55), `frontend/vitest.config.ts` (+12), `frontend/package.json` (+1), `.gitea/workflows/ci.yml` (+3). -**Total:** 44 frontend tests (31 existing + 13 new). - -## Session 2026-08-13 — Plan consolidation (TODO → DONE) - -Completed tasks moved out of `.github/instructions/TODO.md`. Remaining work replanned in TODO from `BACKLOG.md`. - -### Phase 1 — playout test fixtures + source-media contract - -- Create small compliant fixture media in `backend/playout/testdata/` for the validator and playout tests -- Create invalid fixture media for codec, framerate, and audio rejection paths -- Create playlist fixtures for empty, comment-only, mixed-validity, and valid playlists -- Document the ffmpeg fixture-generation commands in the relevant test headers - -### Phase 2 — viewer frontend coverage - -- HLS player startup, manifest resolution, and cross-origin/local fallback tests -- Chat send/receive, presence updates, private DM flow, and reconnect tests -- Responsive-layout and accessibility checks for the viewer shell and chat sidebar -- Coverage reporting added and minimum threshold enforced in CI - -### Phase 3 — admin frontend coverage - -- Playlist management, moderation flows, and dashboard rendering tests -- API integration points with mocked backend responses -- Responsive-layout and accessibility checks for admin screens -- Coverage reporting added and minimum threshold enforced in CI - -### Phase 4 — R2 and edge-delivery validation - -- Integration test for upload and retrieval of HLS objects through the R2 path -- Segment and manifest cache headers verified (immutable media, live manifest freshness) -- Edge cache hit/miss validation scripted (`scripts/validate-edge-cache.sh`) -- Stubs/mocks documented (putObject variable-injection pattern) - -### Phase 7 (partial) — load-test scaffolding + saturation alerts - -- WebSocket load-test scaffolding and summary artifacts (`ws-chat.yml`, `run-loadtests.ps1`, `summarize-artillery.ps1`) -- Admin API benchmark scaffolding and endpoint summary artifacts (`admin-api.js`, `run-loadtests.ps1`, `summarize-k6.ps1`) -- Saturation thresholds added to monitoring (chat p95 latency + Redis memory usage) -- Cache-hit threshold alert operationalization added to monitoring - -### Streaming pipeline gaps (audit → implemented) - -- Continuous R2 sync during playout (fsnotify watcher, ~7s delay) -- Globally unique segment naming with run ID (`r-{ISOTIME}/seg-{seq}.m4s`) -- Atomic manifest publication (upload to `.tmp`, then CopyObject) -- Manifest Cache-Control: `public, max-age=1, s-maxage=2, must-revalidate` -- Player error recovery (bounded retries, exponential backoff, reconnecting overlay) -- Health endpoint detail (run ID, active item, playlist revision, segment seq, R2 pub, consecutive failures) -- CORS headers — documented as Cloudflare R2 infrastructure config -- Phase 4 edge cache validation — `scripts/validate-edge-cache.sh` created - -## Session 2026-08-13 — Playout Enhancements - -Completed the three Playout enhancement workstreams from `TODO.md`. Remaining polish items kept in TODO. - -### 1. Automatic normalization of source media on upload (H.264/AAC, 30fps, keyframe check) - -- Implement `media.NormalizeVideo` in `backend/chat/src/media/media.go` (ffmpeg exec, Linear Void profile, bounded 2h timeout, stderr capture) -- Write normalized output to `*.normalized.mp4` and re-verify with ffprobe (`ValidateNormalization` + `ProbeDurationSeconds`) -- Extend media status states: `normalizing`, `normalization_failed` (alongside `ready` / `needs_normalization`) -- Integrate into single-upload path — `HandleAdminMedia`: persist `needs_normalization` then `go normalizeMediaRecord(...)` -- Integrate into chunked-upload completion — `HandleAdminMediaUploads` `"complete"`: same async flow after assembly + validation -- Add `POST /api/admin/media/{id}/normalize` (behind `requireAdminAuth`, returns 202 + `normalizing` record) -- Admin UI (`admin.ts`): "Normalize" action + status polling until `ready`/`normalization_failed` -- Tests: `NormalizeVideo` args + ffmpeg-error path; handler `normalizeMediaRecord` success/failure; `/normalize` endpoint 202 -- Retire hardcoded `backend/playout/normalize-video.sh` (superseded by in-Go normalization) - file deleted - -### 2. R2 data retention: delete old segments after 24h - -- Add `cleanupOldSegments(ctx)` to `backend/playout/src/playlist.go` -- Use `s3.ListObjectsV2` on prefix `hls/` to enumerate run directories -- Delete `hls/r-*/*` older than `now - R2_SEGMENT_RETENTION_HOURS` (default 24) -- Never delete `hls/playlist.m3u8` or `hls/playlist.m3u8.tmp` -- Run cleanup as a ticker goroutine (hourly) plus once at startup -- Add `R2_SEGMENT_RETENTION_HOURS` (default `24`) to `.env.example` -- Expose `linearvoid_playout_r2_cleanup_objects_total` metric and log deleted keys -- Tests: mock `ListObjectsV2`/`DeleteObjects` via injectable vars (mirror the existing `putObject` pattern) - -### 3. Continuous playout: confirm ffmpeg loops without interruption - -- Resolve the dual-loop ambiguity between `main()` and `start-playout.sh` -- Add a `-validate` mode to `main()`: run `ValidatePlaylist` and exit 0/1 without the ffmpeg loop -- Update `start-playout.sh` to invoke `-validate` and `exec` the Go binary (single source of truth) -- Minimize the restart gap: skip full `resetHLSOutput` when the playlist is unchanged -- Track `playlistRevision` (SHA-256 fingerprint); reset only when the playlist changed, on crash/slate recovery, or first run -- Continue segment numbering via `-start_number` + prune stale segments (`hlsNextStartNumber`) -- Verify `init.mp4` + manifest are correctly published on clean completion (existing `publishManifest` path runs after every run) -- Add metrics `linearvoid_playout_loops_total` (clean completions) and `linearvoid_playout_run_duration_seconds` -- Tests: `TestPlaylistFingerprint`, `TestHLSNextStartNumber` - -## Session 2026-08-13 — Application Configuration, Matomo, Chat Accounts - -### Application configuration (runtime config store) - -- Runtime config store in `backend/chat/src/config/runtime.go` (thread-safe `Get`/`Set`, JSON persist, typed getters/setters with validation) -- `ConfigPath()` default `/app/playlist/config.json` (shared with playout via the playlist volume) -- Typed getters: `MatomoURL()`, `MatomoSiteID()`, `HLSSegmentDuration()`, `HLSWindowSize()` -- `GET/PUT /api/admin/config` now reads/persists `matomoUrl`, `matomoSiteId`, `hlsSegmentDuration`, `hlsWindowSize` -- Admin UI config form extended (`frontend/public/admin/system.html` + `frontend/src/admin.ts`) -- Playout reads the shared `config.json` (`loadSharedHLSConfig` + `PLAYOUT_CONFIG_PATH`) -- Tests: `runtime_test.go`, config handler tests, `TestLoadSharedHLSConfig` - -### Matomo analytics - -- Public `GET /api/matomo` endpoint -- Viewer pageview snippet in `frontend/index.html` -- Test: `TestHandlePublicMatomo` - -### Admin frontend (media status + playlist gating) - -- Media library: status badges (`ready`/`needs_normalization`/`normalizing`/`failed`) + "Add to playlist" gated to `ready` media -- Playlist entries: per-item status badges -- `statusBadge`, `fetchMediaStatusMap`, `addMediaToPlaylist` in `admin.ts` - -### Chat user accounts - -- `db.User` model (bcrypt password hash, unique username) -- `auth.UserSessionStore` (token → username with expiry) -- Endpoints: `POST /api/auth/register`, `POST /api/auth/login`, `GET /api/auth/session`, `POST /api/auth/logout` -- Viewer sign-in UI (`frontend/index.html` + `frontend/src/main.ts`) -- Tests: `TestUserSessionStore`, register/login/session handler tests - -### Private DM history & search - -- `GET /api/dm/history` (paginated) + `GET /api/dm/search` (case-insensitive) -- DM search UI in viewer -- Tests: `TestHandleUserDMHistory`, `TestHandleUserDMHistory_Unauthorized`, `TestHandleUserDMSearch` - -### Chat UI (typing indicator + user list) - -- `Hub.BroadcastTyping` (ephemeral typing events, not persisted) -- Viewer typing indicator (`showTyping`/`hideTyping`/`sendTyping`) + user-list toggle - -## Session 2026-08-13 — Prometheus TSDB Retention - -- Added `--storage.tsdb.retention.time=30d` (delete metrics older than 30d) + `--storage.tsdb.retention.size=15GB` (disk cap) to the `prometheus` service `command` in `docker-compose.yml` and `docker-compose.prod.yml` -- Validated: `docker compose config --quiet` (both files), `promtool check config` (SUCCESS, 10 rules), Prometheus v3.13.2 accepts both retention flags -- Documented the retention policy in `docs/MONITORING.md` → Operational Recommendations - -## Session 2026-08-13 — Architecture Documentation Update - -- Updated `docs/architecture/README.md` to reflect the `[Unreleased]` changes: - - Automatic media normalization on upload (`media.NormalizeVideo`, statuses, `POST /api/admin/media/{id}/normalize`) - - Playout: seamless looping, R2 segment retention, `-validate` mode, shared runtime config, new loop/cleanup metrics - - Prometheus: TSDB retention (`30d` + `15GB`), alert rule count fixed 8 → 10 - - New Cross-cutting Concepts sections: Runtime Configuration Store, Analytics (Matomo) - - Viewer user accounts (register/login/logout/session, bcrypt, `linearvoid_user_session`) - - Chat: DM history/search, typing indicator, user-list toggle - - API reference: `GET/PUT /api/admin/config`, `POST /api/admin/media/{id}/normalize`, new Viewer REST API section - - Frontend + Chat Service module tables updated - -## Session 2026-08-13 — Release 1.0.13 - -- Promoted `[Unreleased]` → `## [1.0.13] - 2026-08-13` in `CHANGELOG.md` -- Bumped `frontend/package.json` to `1.0.13` and README version badge to `1.0.13` -- Committed remaining work + `chore: bump version to 1.0.13`, tagged `1.0.13` (annotated) -- Pushed `main` + `1.0.13` to origin → CI/CD + `release.yml` builds semver images and creates the Gitea release - -## Session 2026-08-13 — Frontend Error Fix - -- Fixed `Uncaught TypeError: Cannot read properties of null (reading 'appendChild')` in the viewer bundle: `index.html` was missing the `#messages` chat container (and had a stray ``), so `addMessage` crashed on WebSocket open/message. -- Restored chat panel structure: added `
` and wrapped `#presence-list` in `
`. -- Made `addMessage` defensive (returns early when `#messages` is absent), matching the null-guards in `renderPresence`/`showTyping`/`hideTyping`. - -## Session 2026-08-13 — Chat improvements - -### Transient presence flash + system-message cleanup - -- Added `#chat-flash` container (`index.html`) + `.chat-flash` CSS (top of chat panel, `0.83rem`, 3s auto-hide with 0.5s fade-out). -- `main.ts` `ws.onmessage` routes join/leave system messages to `showFlash()` + presence list instead of `addMessage()`. -- Backend: `Broadcast` no longer persists `type: "system"` messages; `PushChatHistory` excludes system rows; `HandleAdminChatLog` filters `type <> 'system'`. -- 5s reconnect grace: disconnect schedules a delayed "left" via `time.AfterFunc` (`Hub.PresenceGracePeriod`); a reconnect within grace cancels it and skips "joined". -- Font sizing: `.message-text` set to `0.95rem` (between default and system `0.83rem`); message alignment already correct. -- Tests: `TestHub_Broadcast_SystemNotPersisted`, `TestHub_ReconnectGracePeriod`, `TestHandleAdminChatLog_ExcludesSystem`, frontend `presence flash`; ws test DB fixed to `SetMaxOpenConns(1)`. - -## Session 2026-08-13 — Playout improvements - -### HLS segment duration 2s + window 12 - -- `backend/playout/src/playlist.go`: `HLS_SEGMENT_DURATION` default `"6"→"2"`, `HLS_WINDOW_SIZE` `"24"→"12"`. -- `backend/chat/src/config/runtime.go`: `defaultHLSSegmentDuration 6→2`, `defaultHLSWindowSize 24→12`. -- ffmpeg GOP already 2s (`-g 60 -keyint_min 60` @30fps), so segments cut cleanly on keyframes — no encoder change needed. -- Updated `runtime_test.go` defaults + env-fallback tests, `.env.example`, `docs/STREAMING.md` (2s rationale + Firefox notes), `docs/operations/cloudflare-pages-setup.md`, `docs/architecture/README.md`. -- Verified: chat + playout `go test ./...` and `go vet ./...` all pass. +- **2026-08-13 viewer crash**: `index.html` was missing the `#messages` container, so `addMessage` threw `Cannot read properties of null (reading 'appendChild')` on WebSocket open/message. Restored the chat panel structure and made `addMessage` defensive. +- **2026-08-15 chat reconnect loop**: after a registered user logged in, the chat panel repeatedly filled with old messages. `connectWebSocket` left the replaced socket's `onclose` handler attached, so each intentional reconnect scheduled another reconnect and every connection re-pulled the room's 50-message history on a 3s loop. Fixed by detaching the old socket's `onclose` before closing and guarding the handler against stale sockets. Regression test added; `npm --prefix frontend test` → 58 tests passing. diff --git a/.github/instructions/TODO.md b/.github/instructions/TODO.md index 23109ce..9ae3cec 100644 --- a/.github/instructions/TODO.md +++ b/.github/instructions/TODO.md @@ -1,67 +1,3 @@ # TODO > Completed tasks moved to [DONE](./DONE.md). Remaining tasks in [BACKLOG](./BACKLOG.md). - -## Chat improvements (remaining) - -- [ ] Manual smoke: viewer join/leave shows a 3s flash (not a chat row); reconnect <5s shows no duplicate; admin chat-log table has no system rows - -## Playout improvements (remaining) - -- [ ] Test in Firefox: live playback, rebuffer/recovery, chat sync; record quirks in `docs/STREAMING.md` -- [ ] Rebuild + run playout; confirm `playlist.m3u8` lists ~12 segments at 2s each -- [ ] Manual: verify viewer latency and cache-hit behavior after the change - -## Setup phases remaining - -### Phase 5: complete manual Cloudflare and external-service setup - -- [ ] Apply Cloudflare Pages, Tiered Cache, WAF, rate limiting, and HTTPS/HSTS settings in the dashboard -- [ ] Cloudflare Pages: link custom domain and cut over frontend hosting -- [ ] Enable Smart Tiered Cache; validate `Cf-Cache-Status: HIT` from upper tier -- [ ] WAF: enable OWASP Managed Ruleset (Simulate → review 48h → Block), enable Bot Fight Mode -- [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action -- [ ] HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2; enforce at edge and origin -- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval) -- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword `#EXTM3U`, 5min) -- [ ] UptimeRobot: alert on 2 consecutive failures via email + Discord/Slack webhook -- [ ] Capture evidence in `docs/operations/cloudflare-uptimerobot-checklist.md` - -### Phase 6: rehearse production deployment end to end - -- [ ] Provision Hetzner CX22 (2 vCPU, 4 GB, 40 GB) in Ashburn VA, Ubuntu 24.04 -- [ ] Install Docker Engine + Compose plugin; open firewall: 80/443 public, 2375 runner-only -- [ ] Configure Docker daemon for remote TLS access (`-H tcp://0.0.0.0:2376` with certs) -- [ ] Create `/srv/linearvoid/data/{playlist,media,hls,chat,slate}` directories -- [ ] Add `CI_PROD_*` secrets to Gitea (DOCKER_HOST, R2 keys, admin password, DB password, Grafana password) -- [ ] Trigger CD workflow with `DOCKER_HOST` pointing to Hetzner IP -- [ ] Verify all 11 containers healthy: `/health`, `/hls/playlist.m3u8`, WebSocket -- [ ] Update Cloudflare A record `linearvoid.tv` → Hetzner IP, SSL Full (strict); test HTTPS + WSS -- [ ] Smoke test: viewer SPA loads, HLS plays, chat connects, admin login → upload → playlist → playout -- [ ] Set up Prometheus alerting notifications + UptimeRobot on production URL -- [ ] Run `scripts/backup-db.sh` + `scripts/verify-backups.sh` on production host -- [ ] Document production IP, SSH access, rollback procedure, scaling thresholds - -### Phase 7: baseline scaling and performance - -- [ ] Execute WebSocket load tests — ramp 10→100→500→1000 connections; measure p95 latency + errors -- [ ] Execute admin API benchmarks — `wrk`/`k6` against `/api/admin/*`; target p95 <1s -- [ ] Measure manifest TTFB and cache-hit distribution from multiple US regions (WebPageTest 3+ regions, <4s warm cache) -- [ ] Tune manifest TTL to 0.5× segment duration; validate `Cf-Cache-Status` distribution -- [ ] Track cache-hit ratio and R2 ops in production (Cloudflare GraphQL → Grafana panel, alert <90%) -- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min -- [ ] Plan Phase 2 service separation (playout CPU, chat latency, DB state, monitoring) -- [ ] Plan Phase 3 multi-origin resilience (second Hetzner location, warm standby, DNS failover, DB/Redis replication) - -### Phase 8: close out remaining test and validation gaps - -- [ ] Admin frontend: user moderation tools tests (mute/ban/report, view moderation logs) -- [ ] Frontend: coverage for all critical user + admin components and interactions -- [ ] Frontend: document test cases and expected outcomes -- [ ] Backend R2: test retrieval of segments/manifests via Cloudflare edge -- [ ] Backend R2: validate edge cache hit/miss behavior with repeat requests (`scripts/validate-edge-cache.sh`) - -### Playout - -- [ ] Configure an R2 lifecycle rule on the `hls/` prefix as defense-in-depth (manual) -- [ ] Full-loop E2E (2-file playlist, assert `Looping...` with no `CRASHED` across the boundary) — manual via `scripts/test-playout-e2e.sh`