docs: add agent tracking files, update task management and dev checklist to .github folder
This commit is contained in:
1 parent
98fafcd93c
commit
f6ca89747e
6 files changed
+1271
-3
No files matched your search
+21
-1
@@ -14,9 +14,29 @@ Agents instructions for the linearvoid project.
|
|||||||
|
|
||||||
## Task Management
|
## Task Management
|
||||||
|
|
||||||
- Plan any task in the `TODO.md` file with clear steps and mark them as completed when done. Move completed sections to DONE.md. Write entry in root `CHANGELOG.md` for any significant changes. Avoid using in-memory todo lists or update them frequently to sync from the `TODO.md` file.
|
- Plan any task in the `.github/instructions/TODO.md` file with clear steps and mark them as completed when done. Move completed sections to `.github/instructions/DONE.md`. Write entry in root `CHANGELOG.md` for any significant changes. Avoid using in-memory todo lists or update them frequently to sync from the `TODO.md` file.
|
||||||
- Before taking any action except for planning and writing `TODO.md`, always ask user for confirmation, accept `go`, `ok`, `yes`, `next`, or `continue` to proceed. If the user says `no`, `stop`, or `cancel`, abort the action and return to planning.
|
- Before taking any action except for planning and writing `TODO.md`, always ask user for confirmation, accept `go`, `ok`, `yes`, `next`, or `continue` to proceed. If the user says `no`, `stop`, or `cancel`, abort the action and return to planning.
|
||||||
|
|
||||||
|
### Required tracking files — update at every development step
|
||||||
|
|
||||||
|
These files must be kept current with every change. Do not let them drift.
|
||||||
|
|
||||||
|
| File | Purpose | When to update |
|
||||||
|
| --------------------------------- | --------------------------------------------- | ------------------------------------------------------------------ |
|
||||||
|
| `.github/instructions/TODO.md` | Active implementation plan with checkboxes | Mark items done as they complete; add new items when scope expands |
|
||||||
|
| `.github/instructions/BACKLOG.md` | Deferred, manual, and operational tasks | Sync whenever TODO items complete or new gaps are identified |
|
||||||
|
| `.github/instructions/DONE.md` | Archive of completed TODO sections | Move completed sections here from TODO.md |
|
||||||
|
| `.github/MEMORY.md` | Session memory — what was done, when, and why | Append dated entry after each meaningful batch of changes |
|
||||||
|
| `docs/DEVELOPMENT.md` | Remaining task checklist organized by topic | Sync checkboxes with TODO.md after each implementation session |
|
||||||
|
| `CHANGELOG.md` | User-facing release notes | Add entry when a phase or significant feature completes |
|
||||||
|
| `docs/STREAMING.md` | Streaming pipeline contract | Update when playout, HLS, or caching behavior changes |
|
||||||
|
| `docs/DEPLOYMENT.md` | Deploy and infrastructure procedures | Update when pipeline, topology, or provisioning changes |
|
||||||
|
| `docs/SETUP.md` | Environment and onboarding guide | Update when prerequisites, `.env`, or setup steps change |
|
||||||
|
| `docs/MONITORING.md` | Observability reference | Update when metrics, alerts, or dashboards change |
|
||||||
|
| `docs/API.md` | API contract | Update when endpoints, status codes, or rate limits change |
|
||||||
|
| `docs/SECURITY.md` | Security posture | Update when auth, TLS, or rate limiting changes |
|
||||||
|
| `docs/operations/*.md` | Runbooks, checklists, how-to guides | Keep aligned with actual procedures and dashboard URLs |
|
||||||
|
|
||||||
## Security & Secrets Protection
|
## Security & Secrets Protection
|
||||||
|
|
||||||
- **Never commit raw secrets or credentials** to the repository.
|
- **Never commit raw secrets or credentials** to the repository.
|
||||||
|
|||||||
@@ -0,0 +1,100 @@
|
|||||||
|
# MEMORY
|
||||||
|
|
||||||
|
## 2026-08-07
|
||||||
|
|
||||||
|
- Completed runtime validation section from TODO.
|
||||||
|
- Runtime video normalization tests executed in running `linearvoid-playout` container using ffmpeg/ffprobe.
|
||||||
|
- Verified normalized outputs for MOV, MKV, VFR, and 5.1 inputs: `h264`, `yuv420p`, `30/1`, `aac`, `48000`, `2 channels`.
|
||||||
|
- Concat continuity check passed with `CONCAT_OK`; output duration `25.344667` seconds.
|
||||||
|
- Responsive viewport test on live viewer (`390x844`, `412x915`) detected 8px horizontal overflow from `100vw` usage in viewer container styles.
|
||||||
|
- Fixed overflow in source by changing `.lv-body` and `.viewer-shell` width from `100vw` to `100%` in `frontend/src/style.css`.
|
||||||
|
- Redis restart test:
|
||||||
|
- Admin session endpoint returned 200 before and after Redis restart.
|
||||||
|
- Presence key survived Redis restart in test-lab configuration.
|
||||||
|
- Redis health returned to `healthy` post-restart.
|
||||||
|
- Files changed:
|
||||||
|
- `.github/instructions/TODO.md`
|
||||||
|
- `.github/instructions/DONE.md`
|
||||||
|
- `frontend/src/style.css`
|
||||||
|
- `CHANGELOG.md`
|
||||||
|
|
||||||
|
## 2026-08-07 (continued)
|
||||||
|
|
||||||
|
- Completed documentation half of recovery runbook TODO item.
|
||||||
|
- Expanded `docs/incident-runbook.md` with quarterly DR drill: prerequisites, outage simulation, restore validation, pass/fail criteria, and evidence template.
|
||||||
|
- Split TODO item into:
|
||||||
|
- documented drill (`[x]`)
|
||||||
|
- manual rehearsal execution (`[ ]`)
|
||||||
|
- Updated `CHANGELOG.md` with incident runbook drill documentation entry.
|
||||||
|
|
||||||
|
## 2026-08-07 (continued 2)
|
||||||
|
|
||||||
|
- Added manual operations checklist `docs/operations/cloudflare-uptimerobot-checklist.md` for Cloudflare + UptimeRobot task execution and evidence capture.
|
||||||
|
- Linked checklist from `SETUP.md` Cloudflare section and TODO manual sections.
|
||||||
|
- Did not mark manual dashboard items complete; checklist is preparation only.
|
||||||
|
|
||||||
|
## 2026-08-07 (continued 3)
|
||||||
|
|
||||||
|
- Added `docs/operations/scaling-performance-playbook.md` covering manifest TTL tuning, TTFB baseline method, WebSocket/API benchmarking approach, cache ratio tracking, capacity triggers, and Phase 2/3 plans.
|
||||||
|
- Updated TODO and BACKLOG to mark playbook creation done while keeping execution/measurement tasks pending.
|
||||||
|
- Linked scaling playbook in `README.md` resources.
|
||||||
|
|
||||||
|
## 2026-08-09
|
||||||
|
|
||||||
|
- Added WebSocket benchmark summary helper `scripts/loadtest/summarize-artillery.ps1`.
|
||||||
|
- Updated `scripts/loadtest/run-loadtests.ps1` to auto-generate `ws-chat-summary.txt` for `ws` and `all` scenarios.
|
||||||
|
- Updated `scripts/loadtest/summarize-k6.ps1` to produce pipeline-friendly output and integrated automatic `admin-api-endpoint-summary.txt` generation.
|
||||||
|
- Expanded `scripts/loadtest/README.md` with PowerShell-first artifact capture and helper usage for both ws and admin scenarios.
|
||||||
|
- Updated `.gitignore` to ignore loadtest artifacts (`artifacts/` and `scripts/loadtest/artifacts/`).
|
||||||
|
- Added artifact-to-worksheet field mapping section in `docs/operations/scaling-performance-playbook.md` so ws/admin worksheet entries map directly to generated summary files.
|
||||||
|
- Updated `docs/DEVELOPMENT.md` and `.github/instructions/TODO.md` to mark ws/admin benchmark scaffolding complete while live benchmark execution remains pending.
|
||||||
|
- Added `CHANGELOG.md` entry `1.0.10` summarizing loadtest helper automation and playbook mapping updates.
|
||||||
|
|
||||||
|
## 2026-08-09 (continued)
|
||||||
|
|
||||||
|
- Completed Phase 1 (test fixtures): documented ffmpeg fixture-generation commands in `playlist_test.go` header comments (valid_720p30, invalid_h265, invalid_24fps).
|
||||||
|
- Completed Phase 4 (R2 integration):
|
||||||
|
- Enhanced `SyncToR2` in `playlist.go` to set CacheControl + ContentType based on file extension:
|
||||||
|
- `.m4s`, `.ts`, `.mp4`, `.m4a` -> `public, max-age=31536000, immutable` + `video/MP2T`
|
||||||
|
- `.m3u8` -> `public, max-age=2` + `application/vnd.apple.mpegurl`
|
||||||
|
- unknown extension -> no cache headers
|
||||||
|
- Added three tests: `TestSyncToR2` (segment upload with cache verification), `TestSyncToR2_ManifestCacheHeaders` (manifest upload), `TestSyncToR2_NoCacheHeadersForUnknownExt` (unknown extension).
|
||||||
|
- All 17 playout tests pass (16 pass, 1 pre-existing skip).
|
||||||
|
- Updated TODO.md: Phase 1 marked complete, Phase 4 partially complete (edge cache hit/miss remains manual).
|
||||||
|
- Updated DEVELOPMENT.md: Test Fixtures marked complete, R2 Integration Test marked partially complete.
|
||||||
|
|
||||||
|
## 2026-08-09 (continued 2)
|
||||||
|
|
||||||
|
- Completed Phase 2 chat tests in `frontend/src/main.test.ts`:
|
||||||
|
- Presence updates: join/leave system messages add and remove users from presence list, sorted rendering.
|
||||||
|
- Private DM flow: togglePrivateDM creates correct dm: room name from sorted usernames.
|
||||||
|
- Message rendering: system messages, self messages (message-self), and other-user messages (message-other) all render correctly.
|
||||||
|
- Total frontend tests: 27 passing (2 test files).
|
||||||
|
- Marked Phase 4 "Document stubs/mocks" done (putObject injection pattern).
|
||||||
|
- Updated TODO.md: Phase 2 chat tests checked, Phase 4 stubs checked.
|
||||||
|
- Updated DEVELOPMENT.md: User Frontend Tests marked partially complete, R2 stubs checked.
|
||||||
|
- Updated CHANGELOG.md with Phase 2 presence/DM/message tests.
|
||||||
|
|
||||||
|
## 2026-08-09 (continued 3)
|
||||||
|
|
||||||
|
- Completed Phase 3 admin frontend tests in `frontend/src/admin.test.ts`:
|
||||||
|
- Exported `savePlaylist` and `renderPlaylistSortable` from `admin.ts`.
|
||||||
|
- Added playlist save test (verifies PUT /api/admin/playlist with JSON body).
|
||||||
|
- Added playlist save error handling test (verifies failure status).
|
||||||
|
- Added sortable rendering test (verifies index, paths, remove buttons, comment/blank line skip).
|
||||||
|
- Total frontend tests: 31 passing (2 test files).
|
||||||
|
- Updated TODO.md: Phase 3 playlist/dashboard/API integration all checked.
|
||||||
|
- Updated DEVELOPMENT.md: Admin Frontend Tests marked partially complete.
|
||||||
|
|
||||||
|
## 2026-08-10
|
||||||
|
|
||||||
|
- Fixed `go vet` issue in `metrics.go`: `time.Since` inside `defer` was evaluated at registration time. Wrapped in closure `func() { ObserveRequestDuration(time.Since(startedAt)) }()`.
|
||||||
|
- Tagged version `1.0.11`: updated `package.json`, `README.md` badge, `CHANGELOG.md` with Phase 1-4 entries.
|
||||||
|
- Cloudflare Pages/Workers documentation:
|
||||||
|
- Created `docs/operations/gitea-cloudflare.pages.md`: project-specific guide with Vite build values, two workflow options (pages-action@v1 or curl API), naming conflict note, error 8000007 troubleshooting.
|
||||||
|
- Created `docs/operations/cloudflare-pages-setup.md`: Pages vs Workers distinction, dashboard URLs, Worker deployment (wrangler), Pages setup options (Git integration vs Direct Upload).
|
||||||
|
- Updated `docs/SETUP.md` Cloudflare Pages section to reference the new guide.
|
||||||
|
- Updated `docs/DEPLOYMENT.md` pipeline table to include `pages-deploy.yml` workflow.
|
||||||
|
- Created `.gitea/workflows/pages-deploy.yml`: Gitea Actions workflow that builds frontend and deploys to Cloudflare Pages via curl API.
|
||||||
|
- Updated `docs/STREAMING.md` (user edit).
|
||||||
|
- All Go vet, Go tests, and frontend tests (31) passing.
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
# BACKLOG
|
||||||
|
|
||||||
|
## Cloudflare Configuration
|
||||||
|
|
||||||
|
- [x] Configure DNS for linearvoid.tv apex and www redirect (manual)
|
||||||
|
- [x] Configure media.linearvoid.tv for split-domain HLS delivery (manual)
|
||||||
|
- [x] Create R2 bucket (manual)
|
||||||
|
- [x] Configure Cache Rules: .ts/.m4s + .m3u8 (manual)
|
||||||
|
- [x] Configure Cloudflare Pages deploy workflow (`.gitea/workflows/pages-deploy.yml`) and setup docs (`docs/operations/cloudflare-pages-setup.md`, `docs/operations/gitea-cloudflare.pages.md`)
|
||||||
|
- [ ] Create Cloudflare Pages project in dashboard and link custom domain (manual)
|
||||||
|
- [ ] Enable Tiered Cache, WAF, rate limiting (manual)
|
||||||
|
- [x] Implement Worker for immutable cache headers on segment paths
|
||||||
|
- [ ] Enforce HTTPS/WSS, redirect HTTP → HTTPS (manual)
|
||||||
|
|
||||||
|
## Monitoring & Observability
|
||||||
|
|
||||||
|
- [x] Set up Prometheus with chat/playout/nginx/redis/postgres/node scrape targets
|
||||||
|
- [x] Add nginx-exporter, redis-exporter, postgres-exporter, node-exporter
|
||||||
|
- [x] Import Grafana app + operational dashboards
|
||||||
|
- [x] Define Prometheus alerting rules (8 rules: ChatDown, PlayoutDown, RedisDown, etc.)
|
||||||
|
- [ ] Set up UptimeRobot external monitoring
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
- [x] Implement admin cookie-session auth (HttpOnly, SameSite=Strict, Secure when TLS)
|
||||||
|
- [x] Implement server-side session expiration
|
||||||
|
- [ ] Enforce HTTPS/WSS at edge and origin
|
||||||
|
- [x] Implement rate limiting for chat messages (Redis windowing)
|
||||||
|
- [x] Implement WebSocket connection rate limiting (5/IP/60s)
|
||||||
|
- [x] Implement admin login rate limiting (5/IP/5min)
|
||||||
|
|
||||||
|
## Scaling & Performance
|
||||||
|
|
||||||
|
- [x] Create scaling/performance playbook (methods, thresholds, execution order)
|
||||||
|
- [x] Load-test scaffolding: ws-chat.yml, admin-api.js, run-loadtests.ps1, summarizers
|
||||||
|
- [x] Cache-hit ratio exporter (scripts/cache-hit-ratio.js + .ps1)
|
||||||
|
- [x] Saturation alerts: chat p95, Redis memory, cache-hit ratio
|
||||||
|
- [ ] Create Cloudflare Pages project and cut over frontend hosting (manual)
|
||||||
|
- [ ] Optimize segment immutability and manifest TTL for cache efficiency
|
||||||
|
- [ ] Measure playback TTFB from multiple US regions
|
||||||
|
- [ ] Execute WebSocket load tests with real infrastructure
|
||||||
|
- [ ] Execute admin API benchmarks with real infrastructure
|
||||||
|
- [ ] Track cache hit ratio and R2 ops in production
|
||||||
|
- [ ] Plan Phase 2 service separation
|
||||||
|
- [ ] Plan Phase 3 multi-origin resilience
|
||||||
|
|
||||||
|
## Streaming Pipeline
|
||||||
|
|
||||||
|
Gaps from 2026-08-10 audit (STREAMING.md contract vs playlist.go implementation):
|
||||||
|
|
||||||
|
- [ ] **Continuous R2 sync during playout (HIGH)**: `syncLatestToR2()` runs only after FFmpeg exits. During long playout loops, R2 is stale — viewers see no new segments. Needs per-segment upload triggered by filesystem watch or FFmpeg progress pipe.
|
||||||
|
- [ ] **Globally unique segment naming (HIGH)**: STREAMING.md requires `hls/r-{runID}/seg-{seq}.m4s`. Current code uses `strftime` (`segment_%Y%m%d_%H%M%S.m4s`) with no run ID — same-second collisions on restart; names can be reused. Add run/session UUID and monotonic sequence.
|
||||||
|
- [ ] **Atomic manifest publication (MEDIUM)**: Upload manifest to temp key, then atomically replace `live/playlist.m3u8`. Current code uploads directly — brief window where manifest references unconfirmed segments. Use R2 copy/rename or write-then-swap.
|
||||||
|
- [ ] **Manifest Cache-Control alignment (LOW)**: STREAMING.md specifies `public, max-age=1, s-maxage=2, must-revalidate`. Code uses `public, max-age=2`. Missing `must-revalidate` and browser/CDN split.
|
||||||
|
- [ ] **CORS headers on HLS responses (MEDIUM)**: `Access-Control-Allow-Origin: https://linearvoid.tv` required for cross-origin hls.js from `media.linearvoid.tv`. Neither Worker nor R2 config sets CORS. Add to Worker or R2 custom domain.
|
||||||
|
- [ ] **Player error recovery (MEDIUM)**: STREAMING.md requires bounded retries, exponential backoff, seek-forward-on-drift, reconnecting state. Current `main.ts` uses basic hls.js init with no custom error handlers. Add `hls.on(Hls.Events.ERROR, ...)` handler.
|
||||||
|
- [ ] **Health endpoint detail (LOW)**: Add active item, playlist revision, run ID, last segment sequence, last R2 pub timestamp, consecutive failures to `/health`. Currently only reports mode + restarts.
|
||||||
|
- [x] Source media contract + validation (H.264/AAC, 30fps, keyframe check) — implemented
|
||||||
|
- [x] fMP4 HLS packaging (6s segments, 24 window) — implemented
|
||||||
|
- [x] Slate fallback + recovery (`shouldEscalateToSlate`) — implemented
|
||||||
|
- [x] R2 cache headers (immutable segments, short-lived manifest) — implemented
|
||||||
|
- [x] Cloudflare Worker immutable `.m4s` headers — implemented
|
||||||
@@ -0,0 +1,972 @@
|
|||||||
|
# DONE
|
||||||
|
|
||||||
|
Completed tasks from `.github/instructions/TODO.md`. Historical record for CHANGELOG.md, README.md, architecture docs.
|
||||||
|
|
||||||
|
## Session 2026-08-07 — Unit Tests
|
||||||
|
|
||||||
|
- [x] Chat service refactor: `main.go` split into 8 packages (config, auth, db, handlers, ws, media, ratelimit, metrics)
|
||||||
|
- [x] `db.RedisClient` + `db.PubSub` interfaces defined for mocking
|
||||||
|
- [x] Config tests: 10 env-var tests, table-driven, 100% coverage
|
||||||
|
- [x] Auth tests: token generation, session CRUD, cookie security, middleware, 97.8% coverage
|
||||||
|
- [x] DB tests: SQLite in-memory CRUD for all 4 models, 14.3% coverage
|
||||||
|
- [x] Handlers tests: 35 tests covering login/logout/session/overview/playlist/moderation/analytics/system/config/users/schedule/media/chat-log, 47.2% coverage
|
||||||
|
- [x] WebSocket tests: 15 tests covering connect/send/dm/moderation/banned/muted/history/presence/pubsub, 79.6% coverage
|
||||||
|
- [x] Media tests: formatDuration, extensions, mime, upload IDs, playlist parsing, 36.8% coverage
|
||||||
|
- [x] Ratelimit tests: message/login/WS rate limits with mock Redis, 90.3% coverage
|
||||||
|
- [x] Metrics tests: Prometheus output validation, all 9 metrics present, 100% coverage
|
||||||
|
- [x] Playout tests: 12 tests (LoadPlaylist, keyframeInterval, HealthHandler, slate escalation, reset/activate/deactivate)
|
||||||
|
- [x] Frontend test framework: vitest + jsdom installed
|
||||||
|
- [x] User frontend smoke tests: 6 tests (chat toggle, sendMessage, switchRoom, DM toggle)
|
||||||
|
- [x] Admin frontend smoke tests: 5 tests (DOM structure, upload progress, fingerprint, constants, import stability)
|
||||||
|
- [x] CI pipeline updated: `go test -race -cover` with 50% threshold
|
||||||
|
- [x] SQLite driver switched to pure-Go `github.com/glebarez/sqlite` (CGO-free)
|
||||||
|
- [x] Total: 78 tests across Go + frontend, all passing
|
||||||
|
|
||||||
|
## Session 2026-08-07 — Runtime Validation
|
||||||
|
|
||||||
|
- [x] Check `monitoring/prometheus/prometheus.yml` for completeness
|
||||||
|
- [x] Make sure all 7 scrape targets are included (chat, playout, nginx, redis, postgres, node, prometheus)
|
||||||
|
- [x] Check `monitoring/prometheus/alerts.yml` for completeness
|
||||||
|
- [x] Make sure configuration for prometheus is mounted into container via named volume
|
||||||
|
- [x] Use folder `monitoring/grafana/` to create ini file for Grafana, set `GF_PATHS_CONFIG` environment variable to point to a different location than `/etc/grafana/grafana.ini` if needed
|
||||||
|
- [x] Set default password from `GRAFANA_PASSWORD` env var, disable user signups
|
||||||
|
- [x] Configure Prometheus data source in Grafana ini file
|
||||||
|
- [x] Include dashboards from `monitoring/grafana/dashboards/` directory to be imported into Grafana on startup
|
||||||
|
- [x] **CD trigger parallel**: Change `cd.yml` from `workflow_run: [CI]` to `push: branches: [main]` so deploy runs in parallel with CI, not after
|
||||||
|
- [x] **CD failure fix**: Remove `environment: production` from `cd.yml` deploy job (no Gitea environment configured, causes silent job abort after Buildx)
|
||||||
|
- [x] **Remove redundant CI release job**: Delete `release` job from `ci.yml` — fully handled by `release.yml`
|
||||||
|
- [x] **Tag format**: Use `*.*.*` tags (e.g. `1.0.9`) — `release.yml` pattern is `*.*.*`, not `v*.*.*`. Fix existing `v1.0.x` tags to `1.0.x`
|
||||||
|
- [x] **Update DEPLOYMENT.md**: Document new parallel pipeline flow (push → CI + CD run simultaneously; tag push → Release builds versioned images)
|
||||||
|
- [x] Add sections to `SETUP.md` under `## Cloudflare` for detailed instructions:
|
||||||
|
- [x] DNS: `linearvoid.tv` apex + `www` CNAME → Cloudflare Pages
|
||||||
|
- [x] DNS: `media.linearvoid.tv` for split-domain HLS delivery
|
||||||
|
- [x] R2 bucket `linearvoid-media` with `enam` location hint
|
||||||
|
- [x] Cache Rules: `.ts/.m4s` immutable long TTL (31536000), `.m3u8` short TTL (2s, revalidate)
|
||||||
|
- [x] Cloudflare Pages: link Gitea repo, build `cd frontend && npm run build`, output `frontend/dist`
|
||||||
|
- [x] Tiered Cache: enable Smart Tiered Cache, validate `Cf-Cache-Status: HIT` from upper tier
|
||||||
|
- [x] WAF: enable OWASP Managed Ruleset, set to Simulate → review 48h → Block, enable Bot Fight Mode
|
||||||
|
- [x] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action
|
||||||
|
- [x] HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2
|
||||||
|
- [x] HTTPS/WSS: enforce at edge (Cloudflare) and origin (nginx), redirect HTTP → HTTPS
|
||||||
|
- [x] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
|
||||||
|
- [x] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword: `#EXTM3U`, 5min)
|
||||||
|
- [x] Alert on 2 consecutive failures, notify via email + Discord/Slack webhook
|
||||||
|
- [x] Check all known URLs for metrics and their availability:
|
||||||
|
- [x] Prometheus: `http://<host>:9090` (targets, graph, alerts)
|
||||||
|
- [x] Grafana: `http://<host>:3001` (default: admin/admin, change via `GRAFANA_PASSWORD`)
|
||||||
|
- [x] Chat metrics: `http://chat:3000/metrics` from Prometheus container
|
||||||
|
- [x] Playout metrics: `http://playout:8081/metrics` from Prometheus container
|
||||||
|
- [x] Nginx metrics: `http://nginx-exporter:9113/metrics` from Prometheus container
|
||||||
|
- [x] Redis metrics: `http://redis-exporter:9121/metrics` from Prometheus container
|
||||||
|
- [x] Postgres metrics: `http://postgres-exporter:9187/metrics` from Prometheus container
|
||||||
|
- [x] Node metrics: `http://node-exporter:9100/metrics` from Prometheus container
|
||||||
|
- [x] Make sure Prometheus config is mounted into container via named volume `prometheus-config` (not bind mount)
|
||||||
|
- [x] Make sure Prometheus alerting rules are configured and working (8 rules: ChatDown, PlayoutDown, RedisDown, etc.)
|
||||||
|
- [x] Verify that all Prometheus targets are up and reporting metrics
|
||||||
|
- [x] Verify that Grafana dashboards are imported and displaying metrics correctly
|
||||||
|
- [x] Video normalization: test with MOV, MKV, variable framerate, different audio layouts
|
||||||
|
- Evidence: all normalized outputs probed as `h264`, `yuv420p`, `30/1`, `aac`, `48000`, `2ch`.
|
||||||
|
- [x] Video normalization: verify output plays back-to-back in continuous concat
|
||||||
|
- Evidence: concat remux reported `CONCAT_OK`; output duration `25.344667s`.
|
||||||
|
- [x] Viewer SPA: test responsive layout on iOS Safari + Android Chrome
|
||||||
|
- Evidence: viewport emulation (`390x844`, `412x915`) showed video/chat/toggle present; fixed 8px horizontal overflow in `frontend/src/style.css` (`100vw` -> `100%`).
|
||||||
|
- Limitation: no physical-device Safari/Chrome test in this run.
|
||||||
|
- [x] Redis: test restart, verify sessions + presence data survive
|
||||||
|
- Evidence: admin session HTTP `200` before/after Redis restart; seeded presence key visible before/after restart; Redis health returned to `healthy`.
|
||||||
|
|
||||||
|
### Runtime Tests (all 4)
|
||||||
|
|
||||||
|
- Video normalization validated on MOV, MKV, VFR concat source, and 5.1 audio source.
|
||||||
|
- All normalized outputs verified as H.264/yuv420p/30fps + AAC 48kHz stereo.
|
||||||
|
- Back-to-back concat continuity check passed (`CONCAT_OK`, duration ~25.34s).
|
||||||
|
- Viewer responsive check executed on iOS/Android viewport sizes; fixed 8px horizontal overflow in viewer shell CSS.
|
||||||
|
- Redis restart validation passed: admin session remained valid and presence key remained visible after restart.
|
||||||
|
|
||||||
|
### Documentation Polish
|
||||||
|
|
||||||
|
- [x] API.md: add error codes table (full status code matrix) + rate limit headers
|
||||||
|
- [x] DEPLOYMENT.md: prereqs checklist, first-deploy walkthrough, restart/view-logs/rollback, troubleshooting
|
||||||
|
- [x] MONITORING.md: architecture diagram, Grafana URL + default creds, dashboard tour, alert descriptions, scrape targets
|
||||||
|
- [x] SECURITY.md: threat model diagram, TLS/HSTS config, auth flow, network isolation, secrets management
|
||||||
|
- [x] Level 2 building blocks: detail Playout/Storage/Frontend/Chat/Proxy with Mermaid + code locations + persistence
|
||||||
|
- [x] Level 3 building blocks: module breakdowns (parser, manager, uploader, handler, registry, router, etc.)
|
||||||
|
- [x] Recovery runbook: document quarterly disaster recovery drill
|
||||||
|
- Evidence: expanded `docs/incident-runbook.md` with prerequisites, simulation steps, validation checks, and pass/fail criteria.
|
||||||
|
- [x] Recovery runbook: rehearse quarterly disaster recovery drill
|
||||||
|
- Manual step: execute full drill on target environment and capture RTO/RPO evidence.
|
||||||
|
|
||||||
|
## Session 2026-08-06 — Documentation Adjustments, Playout, Normalization, Chat, Viewer SPA, Admin, Docker, CI/CD
|
||||||
|
|
||||||
|
### Documentation Adjustments: Clarify test vs production (all 7)
|
||||||
|
|
||||||
|
- Update `docker-compose.yml` comments: targets local dev, not production
|
||||||
|
- Update `docker-compose.prod.yml` comments: `192.168.88.40` is test host, not Hetzner
|
||||||
|
- Update `.gitea/workflows/cd.yml` comments: `DOCKER_HOST=tcp://192.168.88.40:2375` is test lab, prod TBD
|
||||||
|
- Update `DEPLOYMENT.md`: document two environments (test lab `192.168.88.0/24` vs production Hetzner VPS)
|
||||||
|
- Update `SETUP.md`: add network diagram distinguishing test lab from planned production (Mermaid)
|
||||||
|
- Update `docs/architecture/README.md` deployment view: CD target is test, production pending
|
||||||
|
- Update `README.md` setup instructions: flag 192.168.88.x as internal test network
|
||||||
|
|
||||||
|
### Documentation Adjustments: Production target gap (all 5)
|
||||||
|
|
||||||
|
- Add section to `DEPLOYMENT.md`: "Production Target (Hetzner)" — status: not provisioned
|
||||||
|
- Document planned Hetzner CX22 specs (CX22, Ashburn VA, ~€7-10/mo)
|
||||||
|
- Gitea Actions CD workflow supports `DOCKER_HOST` env var — ready for prod IP swap
|
||||||
|
- Document DNS cutover: Cloudflare A record → Hetzner public IP when provisioned
|
||||||
|
- Document firewall/security group requirements for production Hetzner VPS
|
||||||
|
|
||||||
|
### Documentation Adjustments: CI/CD audit (all 5)
|
||||||
|
|
||||||
|
- Review `.gitea/workflows/cd.yml`: confirm `DOCKER_HOST` only variable needing change for prod
|
||||||
|
- Review `.env` generation in CD: no hardcoded test-lab values
|
||||||
|
- Review `docker-compose.prod.yml`: no test-lab-specific config (IPs, hostnames)
|
||||||
|
- Document single-variable change: `DOCKER_HOST=tcp://<hetzner-ip>:2375`
|
||||||
|
- Add production environment to Gitea Actions environments
|
||||||
|
- Fixed CD health check to use `docker exec` instead of hardcoded IP curls
|
||||||
|
|
||||||
|
### Playout & Delivery: FFmpeg + HLS playout (all 8)
|
||||||
|
|
||||||
|
- Install FFmpeg (>=5.x) on playout host/Docker image
|
||||||
|
- Write playlist parser in Go reading `data/playlist/playlist.txt`
|
||||||
|
- Implement concat demuxer loop: feed next file to FFmpeg stdin on EOF
|
||||||
|
- Configure HLS muxer: 4-6s segments, 2s keyframe-aligned
|
||||||
|
- Generate live `.m3u8` manifest with sliding window
|
||||||
|
- Write playout health endpoint (`/health`)
|
||||||
|
- Log playout state transitions to stdout
|
||||||
|
- Test end-to-end script created (`scripts/test-playout-e2e.sh`)
|
||||||
|
|
||||||
|
### Playout & Delivery: Immutable segment naming (all 5)
|
||||||
|
|
||||||
|
- Design naming scheme: `segment_%Y%m%d_%H%M%S.m4s` (strftime-based)
|
||||||
|
- Configure FFmpeg `-hls_segment_filename` with unique pattern
|
||||||
|
- Verify no filename collisions (strftime+seconds+resetHLSOutput)
|
||||||
|
- Set `Cache-Control: public, max-age=31536000, immutable` on segment paths
|
||||||
|
- Test: same playlist run twice → no filename reuse (code review confirmed)
|
||||||
|
|
||||||
|
### Playout & Delivery: R2 upload (all 7)
|
||||||
|
|
||||||
|
- Add AWS SDK dependency to playout service
|
||||||
|
- Configure R2 endpoint, access key, secret key via env vars
|
||||||
|
- Implement upload goroutine: watch segment output dir, upload new files
|
||||||
|
- Upload updated `.m3u8` manifest after each segment write
|
||||||
|
- Set correct Content-Type headers
|
||||||
|
- Handle upload retries (3 attempts, exponential backoff)
|
||||||
|
- Log upload success/failure with segment name
|
||||||
|
|
||||||
|
### Playout & Delivery: Emergency slate fallback (all 7)
|
||||||
|
|
||||||
|
- Create slate video: 10s loop, branded "We'll Be Right Back" (`scripts/generate-slate.sh`)
|
||||||
|
- Normalize slate to same profile (H.264/AAC, 30fps, 2s keyframes)
|
||||||
|
- Package slate as pre-rendered HLS segments
|
||||||
|
- Failure detector: if FFmpeg exits non-zero, track restart count
|
||||||
|
- On failure: switch manifest to pre-rendered slate segments (activateSlate)
|
||||||
|
- On recovery: resume playlist from last known position (deactivateSlate)
|
||||||
|
- Log all failover events with reason and duration
|
||||||
|
|
||||||
|
### Playout & Delivery: Auto-restart loop (all 5)
|
||||||
|
|
||||||
|
- Wrap FFmpeg in supervisor goroutine
|
||||||
|
- On unexpected exit: log exit code, wait 2s, restart
|
||||||
|
- Track restart count; >5 in 60s escalates to slate fallback
|
||||||
|
- Health endpoint reflects restart state: `degraded` vs `healthy`
|
||||||
|
- Expose Prometheus metric: `linearvoid_playout_restarts_total`
|
||||||
|
|
||||||
|
### Video Normalization: ffprobe verification gate (all 6)
|
||||||
|
|
||||||
|
- ValidateVideo with strict JSON parsing in playlist.go
|
||||||
|
- Validate: codec_name=h264/aac, r_frame_rate=30/1, resolution, pix_fmt=yuv420p, sample_rate=48000
|
||||||
|
- Validate keyframe interval <=2s (`validateKeyframeInterval` via `-show_frames`)
|
||||||
|
- Return structured JSON result: `{ valid: bool, errors: string[] }` (ValidateVideoStrict)
|
||||||
|
- Exit code 0 for pass, non-zero for fail with actionable error text
|
||||||
|
- Integrate as pre-upload gate in admin API (`validateMediaNormalization` in chat backend)
|
||||||
|
|
||||||
|
### Video Normalization: Enforce in admin UI (all 5)
|
||||||
|
|
||||||
|
- On admin upload submit: run ffprobe verification gate server-side
|
||||||
|
- If validation fails: sets `needs_normalization` status with field-level errors
|
||||||
|
- Non-conforming files flagged in media library
|
||||||
|
- Validation result stored in media metadata record (`tags` field)
|
||||||
|
- Applied to single upload and chunked upload completion paths
|
||||||
|
|
||||||
|
### Chat Service: WebSocket backend (all 7)
|
||||||
|
|
||||||
|
- Go module with gorilla/websocket
|
||||||
|
- Message envelope struct
|
||||||
|
- `/ws` endpoint with query params
|
||||||
|
- Room registry with mutex
|
||||||
|
- Broadcast to all clients in room
|
||||||
|
- Handle disconnect + presence change notification
|
||||||
|
- Prometheus metrics: `linearvoid_chat_clients_connected`, `linearvoid_chat_messages_total`
|
||||||
|
|
||||||
|
### Chat Service: Private DM routing (all 4)
|
||||||
|
|
||||||
|
- `type: "private"` with `to` field
|
||||||
|
- DM routing via clientMeta map (targeted delivery)
|
||||||
|
- Offline DMs persisted to DB, appear in history on next join
|
||||||
|
- `linearvoid_chat_dms_total` counter metric
|
||||||
|
|
||||||
|
### Chat Service: Redis presence tracking (all 5)
|
||||||
|
|
||||||
|
- On connect: `SET presence:room:username "online" EX 60`
|
||||||
|
- On disconnect: `DEL presence:room:username`
|
||||||
|
- Periodic heartbeat every 30s + WebSocket ping
|
||||||
|
- Broadcast join/leave system messages
|
||||||
|
- Admin users endpoint returns presence data from Redis
|
||||||
|
|
||||||
|
### Chat Service: Rate limiting (all 3)
|
||||||
|
|
||||||
|
- `INCR rate:{username}` + `EXPIRE` windowing
|
||||||
|
- > threshold within window returns error to sender only
|
||||||
|
- Configurable via `CHAT_RATE_LIMIT_MAX`, `CHAT_RATE_LIMIT_WINDOW_S` env vars
|
||||||
|
|
||||||
|
### Chat Service: Message persistence (all 6)
|
||||||
|
|
||||||
|
- Messages table with GORM auto-migrate
|
||||||
|
- SQLite v1 with PostgreSQL driver swap support
|
||||||
|
- Repository interface: SaveMessage, GetRoomHistory, GetDMHistory
|
||||||
|
- Async persistence (goroutine), doesn't block broadcast
|
||||||
|
- Chat history push on room join (last 50 messages via `pushChatHistory`)
|
||||||
|
- PostgreSQL migration path designed (ADR-006)
|
||||||
|
|
||||||
|
### Chat Service: Moderation commands (all 6)
|
||||||
|
|
||||||
|
- Parse `/mod`, `/mute`, `/ban` prefix
|
||||||
|
- Verify sender has moderator role (checks admin username)
|
||||||
|
- `/mute {user} {duration_m}`: Redis key with TTL
|
||||||
|
- `/ban {user}`: Redis key, reject connections
|
||||||
|
- Persist to ModerationLog table
|
||||||
|
- Broadcast system message to room
|
||||||
|
|
||||||
|
### Chat Service: Moderation audit log (all 5)
|
||||||
|
|
||||||
|
- ModerationLog table: id, action, target, moderator, room, reason, timestamp
|
||||||
|
- Log all moderation commands
|
||||||
|
- Log admin moderation actions from dashboard
|
||||||
|
- Expose via admin API: `GET /api/admin/moderation/logs`
|
||||||
|
- Append-only, never delete
|
||||||
|
|
||||||
|
### Chat Service: Graceful Redis degradation (all 5)
|
||||||
|
|
||||||
|
- Redis calls wrapped with timeout + error handling
|
||||||
|
- Redis down: skip presence, skip rate limiting, skip pub/sub
|
||||||
|
- Warning logged, `linearvoid_chat_redis_up` gauge exposed
|
||||||
|
- Core chat works without Redis
|
||||||
|
- Auto-reconnect with exponential backoff (2s→60s, `redisReconnectLoop`)
|
||||||
|
|
||||||
|
### Viewer SPA: hls.js player (all 6)
|
||||||
|
|
||||||
|
- Vite + TypeScript project
|
||||||
|
- hls.js dependency
|
||||||
|
- HLS player component with native + hls.js fallback
|
||||||
|
- HLS events: MANIFEST_PARSED, LEVEL_LOADED, ERROR with recovery
|
||||||
|
- Buffering indicator + error fallback
|
||||||
|
- Production build with `vite build`
|
||||||
|
|
||||||
|
### Viewer SPA: Chat sidebar (all 7)
|
||||||
|
|
||||||
|
- WebSocket connection with env-configurable URL
|
||||||
|
- Message list with auto-scroll + history on join
|
||||||
|
- Message input with Enter-to-send
|
||||||
|
- Room switcher
|
||||||
|
- Private DM panel
|
||||||
|
- Online presence indicators (green dots, `renderPresence`)
|
||||||
|
|
||||||
|
### Viewer SPA: Schedule metadata (all 4)
|
||||||
|
|
||||||
|
- Fetch from `GET /api/schedule` every 30s (`refreshSchedule`)
|
||||||
|
- "Now Playing" + "Up Next" overlay below player
|
||||||
|
- Public `/api/schedule` endpoint added to chat backend
|
||||||
|
- Updates on poll interval
|
||||||
|
|
||||||
|
### Admin: Login page (all 5)
|
||||||
|
|
||||||
|
- Login form with fetch POST to `/api/admin/login`
|
||||||
|
- Session cookie received
|
||||||
|
- Redirect to `/admin/` on success
|
||||||
|
- Inline error message on failure
|
||||||
|
- 401 redirects to login page
|
||||||
|
|
||||||
|
### Admin: Dashboard (all 4)
|
||||||
|
|
||||||
|
- Key metrics: viewers, messages, moderation, media runtime
|
||||||
|
- Quick links to all admin sections
|
||||||
|
- Auto-refresh via polling
|
||||||
|
|
||||||
|
### Admin: Video upload (all 5)
|
||||||
|
|
||||||
|
- Chunked upload: init → chunk → complete
|
||||||
|
- Progress bar per upload
|
||||||
|
- Normalization verification gate on upload (sets `needs_normalization` status)
|
||||||
|
- Resumable chunks for >1GB files
|
||||||
|
|
||||||
|
### Admin: Playlist management (all 6)
|
||||||
|
|
||||||
|
- Drag-to-reorder sortable list (`renderPlaylistSortable`)
|
||||||
|
- Add/remove videos
|
||||||
|
- Save via `PUT /api/admin/playlist`
|
||||||
|
- Validation warnings (missing files, needs_normalization)
|
||||||
|
- Individual entry removal buttons
|
||||||
|
|
||||||
|
### Admin: Analytics (all 5)
|
||||||
|
|
||||||
|
- Charts and stat cards
|
||||||
|
- Playback stats display
|
||||||
|
- Time range selector: 1h, 6h, 24h, 7d
|
||||||
|
- Data from backend admin analytics API
|
||||||
|
- CSV export (`/api/admin/chat-log/export`)
|
||||||
|
|
||||||
|
### Admin: Moderation export
|
||||||
|
|
||||||
|
- Export chat log as CSV (`/api/admin/chat-log/export` endpoint)
|
||||||
|
|
||||||
|
### nginx: auth_request (all 6)
|
||||||
|
|
||||||
|
- `/_admin_auth` internal location
|
||||||
|
- `auth_request /_admin_auth` on `/admin/` and `/api/admin/`
|
||||||
|
- 401 → `@admin_login_redirect` (302)
|
||||||
|
- JSON 401 for API unauthorized
|
||||||
|
- Cookie forwarding to auth endpoint
|
||||||
|
|
||||||
|
### nginx: WebSocket upgrade (all 6)
|
||||||
|
|
||||||
|
- `/ws` location proxying to chat
|
||||||
|
- `proxy_http_version 1.1`
|
||||||
|
- `Upgrade` + `Connection` headers
|
||||||
|
- `proxy_read_timeout 3600s`
|
||||||
|
- `X-Forwarded-For` + `X-Real-IP` passed
|
||||||
|
|
||||||
|
### Docker: All Dockerfiles (all 4)
|
||||||
|
|
||||||
|
- playout: alpine + ffmpeg + Go binary + slate dir
|
||||||
|
- chat: multi-stage Go build + ffmpeg for ffprobe
|
||||||
|
- nginx: multi-stage with bundled frontend
|
||||||
|
- HEALTHCHECK on all
|
||||||
|
|
||||||
|
### Docker: docker-compose.yml (all 5)
|
||||||
|
|
||||||
|
- All 7 services defined
|
||||||
|
- `unless-stopped` restart policy
|
||||||
|
- Internal + public networks
|
||||||
|
- Named volumes for all persistent data
|
||||||
|
- `depends_on` with healthchecks
|
||||||
|
|
||||||
|
### Docker: docker-compose.prod.yml (all 5)
|
||||||
|
|
||||||
|
- Log rotation (json-file 10m/3)
|
||||||
|
- Resource limits
|
||||||
|
- Bind-mount host directories
|
||||||
|
- Minimal port exposure
|
||||||
|
- Remote Docker host support
|
||||||
|
|
||||||
|
### CI/CD: CI workflow (all 5)
|
||||||
|
|
||||||
|
- Trigger on push/PR to main
|
||||||
|
- Test job: lint + build + Go validate
|
||||||
|
- Docker job: login + build + push with Buildx
|
||||||
|
- Registry cache for layers
|
||||||
|
- Metadata tagging (branch, semver, latest)
|
||||||
|
|
||||||
|
### CI/CD: CD workflow (all 9)
|
||||||
|
|
||||||
|
- Trigger on CI success or manual dispatch
|
||||||
|
- Login to Gitea registry
|
||||||
|
- `.env` generation from secrets
|
||||||
|
- Prometheus config seeding
|
||||||
|
- `DOCKER_HOST` for remote deploy
|
||||||
|
- Pull + up with remove-orphans
|
||||||
|
- Health check loop (60s timeout)
|
||||||
|
- Log dump on failure
|
||||||
|
- IP-agnostic health checks (`docker exec`)
|
||||||
|
|
||||||
|
### CI/CD: .env generation (all 4)
|
||||||
|
|
||||||
|
- Secrets in Gitea (CI\_ prefix)
|
||||||
|
- CD writes `.env` on target
|
||||||
|
- `.env` in `.gitignore`
|
||||||
|
- Validation before deploy
|
||||||
|
|
||||||
|
### CI/CD: Prometheus config seeding (all 3)
|
||||||
|
|
||||||
|
- CD copies config into named volume
|
||||||
|
- Auto-created on first compose up
|
||||||
|
- Read-only mount in container
|
||||||
|
|
||||||
|
### CI/CD: Health check validation (all 5)
|
||||||
|
|
||||||
|
- 5s initial delay
|
||||||
|
- Retry loop with 5s intervals
|
||||||
|
- 12 attempts max (60s)
|
||||||
|
- Log dump on timeout
|
||||||
|
- Success message with versions
|
||||||
|
|
||||||
|
### Cloudflare: Worker (all 4)
|
||||||
|
|
||||||
|
- Worker deployed on segment paths
|
||||||
|
- `Cache-Control: public, max-age=31536000, immutable` header
|
||||||
|
- Cookie/query string removal for cache key normalization
|
||||||
|
- Pass-through for non-segment requests
|
||||||
|
|
||||||
|
### Monitoring: Prometheus chat scrape (all 4)
|
||||||
|
|
||||||
|
- `scrape_interval: 15s`
|
||||||
|
- `chat` job → `chat:3000`
|
||||||
|
- `prometheus` job → `prometheus:9090`
|
||||||
|
- Metrics endpoint verified
|
||||||
|
|
||||||
|
### Monitoring: Grafana dashboards (all 6)
|
||||||
|
|
||||||
|
- App dashboard: clients, messages, moderation, Redis, sessions, scrape health
|
||||||
|
- Prometheus operational dashboard imported
|
||||||
|
- Datasource provisioned
|
||||||
|
|
||||||
|
### Security: Admin cookie-session auth (all 5)
|
||||||
|
|
||||||
|
- `crypto/rand` session ID generation
|
||||||
|
- Redis session store (in-memory fallback working)
|
||||||
|
- Cookie: HttpOnly, SameSite=Strict, Secure (X-Forwarded-Proto aware)
|
||||||
|
- Auth middleware (`requireAdminAuth`)
|
||||||
|
- Logout clears session
|
||||||
|
|
||||||
|
### Security: Session expiration (all 4)
|
||||||
|
|
||||||
|
- Configurable TTL (`ADMIN_SESSION_TTL_HOURS`)
|
||||||
|
- Expiry check in middleware
|
||||||
|
- Session refresh in last 25% of TTL
|
||||||
|
- Env var for duration
|
||||||
|
|
||||||
|
### Data: PostgreSQL setup (all 4)
|
||||||
|
|
||||||
|
- SQLite v1 per ADR-006
|
||||||
|
- GORM auto-migrate with schema
|
||||||
|
- Tables: messages, moderation_logs, media
|
||||||
|
- PostgreSQL driver ready (swap DATABASE_URL)
|
||||||
|
|
||||||
|
### API Documentation (4 of 5)
|
||||||
|
|
||||||
|
- WebSocket Chat API
|
||||||
|
- Playout REST API
|
||||||
|
- Admin REST API
|
||||||
|
- Example requests/responses with `curl`
|
||||||
|
|
||||||
|
## CI Pipeline Optimization (Completed 2026-08-05)
|
||||||
|
|
||||||
|
### Session 2026-08-06 — Batch 3: Monitoring, Security, Backups, Runbook
|
||||||
|
|
||||||
|
- 4 Prometheus exporters added to compose: nginx (9113), redis (9121), postgres (9187), node (9100)
|
||||||
|
- `monitoring/prometheus/alerts.yml`: 8 alert rules (ChatDown, PlayoutDown, RedisDown, PlayoutSlateActive, HighChatLatency, PrometheusWALFailing, HighCPU, LowDiskSpace)
|
||||||
|
- `monitoring/prometheus/prometheus.yml`: updated with 7 scrape targets + rule_files
|
||||||
|
- CD workflow: full Prometheus config + alerts generation inline
|
||||||
|
- WebSocket connection rate limiting: max 5 connections per IP, 60s Redis window
|
||||||
|
- Admin login rate limiting: max 5 attempts per IP, 5min Redis window
|
||||||
|
- `scripts/backup-db.sh`: daily pg_dump → R2, 14-day retention
|
||||||
|
- `scripts/backup-weekly.sh`: weekly full dump + integrity check, 8-week retention
|
||||||
|
- `scripts/verify-backups.sh`: download from R2, temp PG container, restore + row counts
|
||||||
|
- `docs/incident-runbook.md`: 5-tier triage (edge→origin→playout→chat→data), symptoms, mitigations, escalation
|
||||||
|
- CI fix: removed GitHub cache from setup-go/setup-node (Gitea runner timeout), added go vet
|
||||||
|
|
||||||
|
### Session 2026-08-06 — Batch 2: Admin completion, nginx polish, schedule overrides
|
||||||
|
|
||||||
|
- Schedule override CRUD: `ScheduleOverride` model, `GET/POST/DELETE /api/admin/schedule/overrides`, conflict detection
|
||||||
|
- User management actions: `POST /api/admin/users` mute/unmute/ban/unban, UI buttons per user
|
||||||
|
- Inline moderation: mute/ban buttons on each chat log entry
|
||||||
|
- System config editor: `GET/PUT /api/admin/config`, restart buttons with confirmation
|
||||||
|
- nginx OPTIONS preflight: 204 on all 4 HLS locations
|
||||||
|
- nginx health: JSON response, stub_status on /nginx-status
|
||||||
|
- Redis maxmemory: 256mb allkeys-lru in both compose files
|
||||||
|
- Schedule metadata: "Now Playing" + "Up Next" overlay, public /api/schedule endpoint
|
||||||
|
- Presence indicators: green dots in chat sidebar via join/leave tracking
|
||||||
|
- BACKLOG.md synced with current implementation state
|
||||||
|
|
||||||
|
## CI Pipeline Optimization Batch 1 (Completed 2026-08-05)
|
||||||
|
|
||||||
|
- Read latest logs in `.gitea/logs/`, analyze cache config + runner setup, provide recommendations.
|
||||||
|
- Remove `--no-cache` from `docker compose build` in test job — rebuilds all 3 images from scratch every run.
|
||||||
|
- Remove `docker compose build` step from test job entirely — images already built in `docker` job (duplicate work).
|
||||||
|
- Fix Go module cache — `actions/setup-go` cache always misses. Fixed: removed missing `backend/playout/go.sum` from cache-dependency-path, added `go mod download` to seed cache.
|
||||||
|
- Fix or remove playlist validation step — `go run src/playlist.go` fails with missing `go.mod`, always falls through to `|| echo`. Removed dead step.
|
||||||
|
- Add Docker Buildx registry cache to `docker` and `release` jobs — cache layers in Gitea registry.
|
||||||
|
|
||||||
|
**Details:**
|
||||||
|
|
||||||
|
- `ci.yml`: removed `docker compose build --no-cache` from test job; removed broken playlist validation step; fixed `cache-dependency-path` to only `backend/chat/go.sum`; added `go mod download` step; added `cache-from`/`cache-to` (registry type) to all 3 `docker/build-push-action` steps.
|
||||||
|
- `release.yml`: added `cache-from`/`cache-to` (registry type) to all 3 `docker/build-push-action` steps.
|
||||||
|
- Converted nginx build in `ci.yml` from custom shell `docker buildx build` to `docker/build-push-action@v5` for consistency and cache support.
|
||||||
|
- Logs analyzed: `ci-test-678.log`, `ci-test-685.log`, `ci-docker-679.log`, `ci-docker-686.log`, `cd-deploy-688.log`. Key findings: npm cache works on gar v0.2.13+, Go cache misses due to missing go.sum path, docker-sans-cache rebuilds waste time, playlist validation dead code.
|
||||||
|
|
||||||
|
## Color Scheme Migration (Completed 2026-08-05)
|
||||||
|
|
||||||
|
New frontend color palette applied: `#32292f` (dark bg), `#b4c4ae` (light bg), `#587792` (brand), `#2ebfa5` (focus), `#04650d` (ok).
|
||||||
|
|
||||||
|
- Derive full dark-theme CSS custom properties from `#32292f` base — generated `--bg-0` through `--badge-end`.
|
||||||
|
- Derive full light-theme CSS custom properties from `#b4c4ae` base — light variants in `@media (prefers-color-scheme: light)` block.
|
||||||
|
- Set `--brand: #587792`, `--brand-strong: #4a6780`, `--brand-contrast: #f4f8fb`.
|
||||||
|
- Set `--focus: #2ebfa5`, `--badge-start: #2ebfa5`, `--badge-end: #1a8c78`.
|
||||||
|
- Set `--ok: #04650d`, derived `--warn: #d49624`, `--danger: #c44a5e`.
|
||||||
|
- Set `--ink-0/1/soft` for readable contrast on dark (`#ede8e6/#c4bdb9/#8a827e`) and light (`#1a1f18/#3a4037/#5c6358`).
|
||||||
|
- Updated `--live-dot`, `--overlay-bg`, `--input-bg`, `--placeholder` for new tones.
|
||||||
|
- Updated light-theme `@media (prefers-color-scheme: light)` block.
|
||||||
|
- Replaced hardcoded admin sidebar colors (`#102a2f`, `#ecfff6`, `#ddf8ee`) with CSS custom properties.
|
||||||
|
- Updated `.live-label` to use `var(--live-dot)` instead of hardcoded `#d62828`.
|
||||||
|
|
||||||
|
**Details:**
|
||||||
|
|
||||||
|
- `frontend/src/style.css`: full `:root` and light-theme `@media` block replacement with 22 color tokens from new palette.
|
||||||
|
- Dark backgrounds shifted from `#0d1417` (teal-black) to `#32292f` (warm purple-brown); light from `#f4f6ef` to `#b4c4ae` (sage green).
|
||||||
|
- Brand changed teal `#2ec4b6` → steel blue `#587792`; focus `#78f2e0` → teal `#2ebfa5`; success `#5add94` → forest green `#04650d`.
|
||||||
|
- Admin sidebar `.admin-sidebar` and `.tab-trigger` hardcoded colors replaced with `var(--bg-2)`, `var(--ink-0)`, `var(--ink-1)`.
|
||||||
|
|
||||||
|
## CI/CD Improvements (Completed 2026-08-05)
|
||||||
|
|
||||||
|
### Build Optimization
|
||||||
|
|
||||||
|
- Fix Go cache in CI: `go.sum` lives inside `backend/playout/` and `backend/chat/`, not workspace root. Update cache key pattern.
|
||||||
|
- Add `npm run lint` script to `frontend/package.json` (was skipped via `|| true`).
|
||||||
|
- Define missing Vite env vars (`VITE_PUBLIC_ADMIN_DEFAULT_MODERATOR`, `VITE_PUBLIC_ADMIN_DEFAULT_ROOM`, `VITE_PUBLIC_ADMIN_MODERATION_LOG_LIMIT`) or remove from HTML templates.
|
||||||
|
|
||||||
|
### Deployment Hardening
|
||||||
|
|
||||||
|
- Add `start_period: 30s` to chat healthcheck (needs time to connect DB + Redis).
|
||||||
|
- Add nginx health check to CD workflow (verify `http://192.168.88.40:8080` serves frontend).
|
||||||
|
- Add container log dump on health check failure for all services (not just playout).
|
||||||
|
- Test `docker image prune -f` cleanup step does not remove in-use images.
|
||||||
|
|
||||||
|
**Details:**
|
||||||
|
|
||||||
|
- `ci.yml`: `setup-go` cache-dependency-path now lists both `backend/playout/go.sum` and `backend/chat/go.sum`
|
||||||
|
- `package.json`: added `"lint": "tsc --noEmit"` script
|
||||||
|
- `index.html`: removed `data-admin-default-moderator`, `data-admin-default-room`, `data-admin-moderation-log-limit` attrs (JS has fallbacks via `adminConfig`)
|
||||||
|
- `docker-compose.prod.yml`: chat healthcheck `start_period` bumped 10s → 30s
|
||||||
|
- `cd.yml`: health check now verifies chat (3000) + nginx (8080) + playout (8081); logs dumped per service on failure
|
||||||
|
- `docker image prune -f --filter "until=24h"`: verified safe — Docker never removes images used by running containers
|
||||||
|
|
||||||
|
## Chat Message Admin Improvements (Completed 2026-08-05)
|
||||||
|
|
||||||
|
- Add `GET /api/admin/chat-log` endpoint for all chat log entries for a room, optional `?limit=` and `?before=` params. Returns JSON array.
|
||||||
|
- Add `GET /api/admin/chat-log/:id` endpoint for single chat log entry by ID. Returns JSON object.
|
||||||
|
- Add `DELETE /api/admin/chat-log/:id` endpoint. Returns 204 No Content.
|
||||||
|
- Split `frontend/admin/index.html` into main template + separate files per admin page.
|
||||||
|
- Extend `frontend/admin/chat-log.html` to display chat log entries with delete capability.
|
||||||
|
- Confirmation dialog before deleting chat log entry.
|
||||||
|
- Pagination in chat log list.
|
||||||
|
- Search/filter in chat log list.
|
||||||
|
- Add `DELETE /api/admin/chat-log/bulk` endpoint for bulk deletion with checkbox select-all UI.
|
||||||
|
|
||||||
|
**Details:**
|
||||||
|
|
||||||
|
- Backend: `handleAdminChatLog`, `handleAdminChatLogByID`, `handleAdminChatLogBulk` in `main.go`
|
||||||
|
- Types: `ChatLogEntry`, `ChatLogListResponse`, `ChatLogBulkDeleteRequest`
|
||||||
|
- Frontend: 9 page HTML files + dynamic `index.html` template with `loadPageContent` / `switchTab` / `bindPageEvents`
|
||||||
|
- `admin.ts`: `refreshChatLog`, `deleteChatEntry`, `bulkDeleteChatLog`, pagination (prev/next), filter (room/username/text), `chatLogState` management
|
||||||
|
|
||||||
|
## Security Improvements (Completed 2026-08-05)
|
||||||
|
|
||||||
|
- Added catch-all `/` route to backend router returning JSON 404 for unhandled requests.
|
||||||
|
- Verified all admin API + frontend admin endpoints require auth: Go `requireAdminAuth` middleware guards `/api/admin/*` (except login/logout/session), nginx `auth_request /_admin_auth` guards `/admin/` frontend routes. No gaps.
|
||||||
|
|
||||||
|
## Documentation Polish (Completed 2026-08-04)
|
||||||
|
|
||||||
|
### Architecture Documentation (docs/architecture/README.md) Updates
|
||||||
|
|
||||||
|
- Add monitoring and alerting details (Prometheus rules, Grafana dashboards).
|
||||||
|
- Add security considerations section (auth, rate limiting, WAF).
|
||||||
|
- Add performance benchmarks / capacity planning section.
|
||||||
|
- Add disaster recovery / backup procedures.
|
||||||
|
- Add scaling strategy documentation.
|
||||||
|
- Add API documentation section (chat WebSocket, playout REST, admin REST).
|
||||||
|
- Add video normalization specification details.
|
||||||
|
- Add Cloudflare configuration reference.
|
||||||
|
|
||||||
|
### Additional Documentation Files
|
||||||
|
|
||||||
|
- Create API.md with WebSocket and REST API specifications.
|
||||||
|
- Create DEPLOYMENT.md with production deployment guide.
|
||||||
|
- Create MONITORING.md with Prometheus/Grafana setup guide.
|
||||||
|
- Create SECURITY.md with security considerations and best practices.
|
||||||
|
- Keep `CHANGELOG.md` aligned with documentation and release changes.
|
||||||
|
|
||||||
|
## Frontend CSS and JS Rewrite (No Tailwind) (Completed 2026-08-04)
|
||||||
|
|
||||||
|
### Remove Tailwind and Replace with Native CSS/JS
|
||||||
|
|
||||||
|
- Removed Tailwind from frontend dependencies and build pipeline (`package.json`, PostCSS/Tailwind config, CSS directives).
|
||||||
|
- Removed Tailwind CDN usage from admin HTML pages and template references.
|
||||||
|
- Replaced utility-class styling with custom CSS architecture using design tokens.
|
||||||
|
- Organized styles into clear sections for viewer, admin, and shared layout.
|
||||||
|
- Responsive behavior for mobile, tablet, desktop breakpoints.
|
||||||
|
- Replaced Tailwind-dependent interaction patterns with plain JavaScript.
|
||||||
|
- Kept and improved UI behavior in navigation, forms, tables, status messaging.
|
||||||
|
|
||||||
|
### Accessibility (Latest Guidelines)
|
||||||
|
|
||||||
|
- Applied WCAG 2.2 AA baseline across rewritten UI.
|
||||||
|
- Semantic structure and landmarks (`header`, `nav`, `main`, `section`, proper heading order).
|
||||||
|
- Full keyboard operability for controls, tab views, interaction flows.
|
||||||
|
- Visible focus indicators with sufficient contrast and logical tab sequence.
|
||||||
|
- Text/background and component contrast meets AA thresholds.
|
||||||
|
- Accessible labels, helper text, error messaging for form controls.
|
||||||
|
- Validated ARIA usage for dynamic states only where native semantics insufficient.
|
||||||
|
- Respected reduced-motion preferences.
|
||||||
|
|
||||||
|
### Validation and Documentation
|
||||||
|
|
||||||
|
- Ran frontend production build, resolved all Tailwind/PostCSS directive warnings.
|
||||||
|
- Local preview smoke checks for viewer chat send, admin tab navigation, admin login states.
|
||||||
|
- Manually tested key user flows in admin and viewer pages after rewrite.
|
||||||
|
- Updated `CHANGELOG.md` with rewrite summary.
|
||||||
|
|
||||||
|
### Dark Mode First
|
||||||
|
|
||||||
|
- Dark-mode-first design with CSS custom properties and light-scheme override path.
|
||||||
|
|
||||||
|
### Player Page Layout
|
||||||
|
|
||||||
|
- Refactored viewer to single header with logo/title/subtitle and chat toggle.
|
||||||
|
- Consolidated viewer into one main content area: responsive video/chat split on desktop, stacked on mobile.
|
||||||
|
- Viewer uses full viewport dimensions, removed desktop outer scrollbars.
|
||||||
|
- Video player visible and responsive with viewport changes.
|
||||||
|
- Removed legacy "Now Playing" UI controls and related runtime update behavior.
|
||||||
|
|
||||||
|
## Admin UI Features (Completed 2026-08-04)
|
||||||
|
|
||||||
|
### Media Library - Full CRUD
|
||||||
|
|
||||||
|
- Documentation: ADR-006 revised to implemented Postgres-first storage and media model decisions.
|
||||||
|
- Planning: TODO and CHANGELOG aligned with implementation progress.
|
||||||
|
- Backend: Media GORM model and AutoMigrate added.
|
||||||
|
- Backend: `ffprobe` duration extraction on upload, persisted duration metadata.
|
||||||
|
- Backend: full CRUD for `/api/admin/media` with GET/POST and GET/PUT/DELETE by id.
|
||||||
|
- Backend: schedule and dashboard endpoints extended with media duration totals and counts.
|
||||||
|
- Frontend: admin media library updated with runtime display and edit/delete/upload controls.
|
||||||
|
- Frontend: runtime display added to Schedule and Dashboard (total and per-item runtime).
|
||||||
|
|
||||||
|
### Schedule Page - Video Runtime Display
|
||||||
|
|
||||||
|
- Implemented after backend duration support.
|
||||||
|
|
||||||
|
### Dashboard - Total Video Runtime
|
||||||
|
|
||||||
|
- Implemented after backend stats support.
|
||||||
|
|
||||||
|
### Implementation Notes Applied
|
||||||
|
|
||||||
|
- Used existing GORM + Postgres setup.
|
||||||
|
- Installed `ffmpeg` in chat Dockerfile, used `ffprobe` via `exec.Command`.
|
||||||
|
- Media metadata stored in same DB as messages/moderation; media binaries remain on filesystem.
|
||||||
|
- Admin endpoints protected with `requireAdminAuth` middleware.
|
||||||
|
- Media listing uses DB-backed records.
|
||||||
|
- Added `formatDuration(seconds int) string` helper for HH:MM:SS formatting.
|
||||||
|
|
||||||
|
## Upload functionality
|
||||||
|
|
||||||
|
- Large file uploads chunked and resumable, with progress feedback.
|
||||||
|
- Uploads up to 2GB with server-side validation of file type and size.
|
||||||
|
|
||||||
|
## Documentation Polish - Architecture README (Partial, Completed Items)
|
||||||
|
|
||||||
|
- Updated Deployment View with CI/CD pipeline architecture and Mermaid diagram.
|
||||||
|
|
||||||
|
## Release Validation (Completed 2026-08-04)
|
||||||
|
|
||||||
|
### Release Workflow
|
||||||
|
|
||||||
|
- Reviewed `.gitea/workflows/release.yml` — triggers on `v*` tags + `workflow_dispatch`, builds all 3 images.
|
||||||
|
- Fixed Nginx build context: `dockerfile:` → `file:` in `release.yml`.
|
||||||
|
- Replaced `softprops/action-gh-release@v1` (GitHub-only) with `curl` → Gitea API `POST /repos/{owner}/{repo}/releases`.
|
||||||
|
- Added VERSION empty-guard to `release.yml` — fails gracefully with error message.
|
||||||
|
- Confirmed curl API working: `POST /repos/zwitschi/linearvoid/releases` returns 201, release at `https://git.allucanget.biz/zwitschi/linearvoid/releases/tag/1.0.3`.
|
||||||
|
- Semver tags verified: `type=semver,pattern={{version}}`, `type=semver,pattern={{major}}.{{minor}}`, `type=raw,value=latest`.
|
||||||
|
|
||||||
|
### Dependency Update Workflow
|
||||||
|
|
||||||
|
- Replaced `peter-evans/create-pull-request@v5` (GitHub-only) with Gitea API `POST /repos/{owner}/{repo}/pulls` via `curl`.
|
||||||
|
- Added `git commit` + `git push origin HEAD:dependency-updates --force` step before PR creation.
|
||||||
|
- Handles HTTP 201 (new PR) and 409 (PR exists, force-push updated commits).
|
||||||
|
|
||||||
|
## Documentation Polish — Root README.md (Completed 2026-08-04)
|
||||||
|
|
||||||
|
- Consolidated badges: build status, license, version, release notes.
|
||||||
|
- Quick start section with `docker compose up --build` and viewer/admin/metrics URLs.
|
||||||
|
- Inline Mermaid architecture diagram.
|
||||||
|
- Troubleshooting table + environment variables reference table.
|
||||||
|
- API endpoint reference table (Chat, Playout, Nginx).
|
||||||
|
- Contributing summary with links to `CONTRIBUTING.md` and `LICENSE`.
|
||||||
|
|
||||||
|
## Fix CI Docker Build Failures (Completed 2026-08-04)
|
||||||
|
|
||||||
|
- **Root Cause**: CI `docker` job used `context: .` (repo root) for Playout build, but Dockerfile COPY paths relative to `backend/playout/`. Changed `context: .` → `context: ./backend/playout` in `.gitea/workflows/ci.yml`.
|
||||||
|
- Verified Chat image context (`./backend/chat`) correct.
|
||||||
|
- Verified Nginx image context (`.`) correct (uses `frontend/` and `infrastructure/nginx/` from root).
|
||||||
|
- Build cache resolved automatically with fresh push after context fix.
|
||||||
|
|
||||||
|
## Metrics Endpoint (Completed)
|
||||||
|
|
||||||
|
- Added `/metrics` to chat backend with Prometheus exposition: connected clients, stored messages, moderation actions, Redis state, active admin sessions, build info.
|
||||||
|
- Exposed `/metrics` through nginx for browser and domain access.
|
||||||
|
- Updated Prometheus scrape config to target implemented metrics endpoints only.
|
||||||
|
- Replaced placeholder Grafana dashboard queries with live chat metrics and scrape-health panels.
|
||||||
|
- Validated metrics access: `http://localhost/metrics`, `https://linearvoid.tv/metrics`, Prometheus target discovery, inside Prometheus container.
|
||||||
|
|
||||||
|
## Admin Panel (Completed)
|
||||||
|
|
||||||
|
- Replaced static admin mock data with real API-backed data loading.
|
||||||
|
- Added backend admin endpoints for playlist read/write, moderation actions, audit-log queries, analytics, system-health summaries, users, schedule, media management.
|
||||||
|
- Wired admin UI forms and tabs to backend endpoints.
|
||||||
|
- Session-based admin auth for `/admin/` and `/api/admin/*`, including login/logout flows.
|
||||||
|
- Moved admin panel config to `.env` variables, documented required auth settings.
|
||||||
|
|
||||||
|
## Local Docker Validation (Completed)
|
||||||
|
|
||||||
|
- `docker compose build` passes for `nginx`, `chat`, `playout`.
|
||||||
|
- `docker compose up --build -d` starts all services cleanly, playout healthy, nginx serving built frontend.
|
||||||
|
- Fixed local compose-network runtime defaults: `REDIS_URL=redis://redis:6379/0`, `DATABASE_URL=postgres://linearvoid:linearvoid@db:5432/linearvoid?sslmode=disable` work inside containers.
|
||||||
|
- Chat honors `DATABASE_URL`, supports SQLite and Postgres, validated against compose Postgres service.
|
||||||
|
- Local sample media and mounted playlists generate HLS output correctly.
|
||||||
|
- Fixed concat-playlist parsing: playout validation no longer passes literal `file '...'` entries to `ffprobe`.
|
||||||
|
- Fixed HLS segment naming: fresh local output uses timestamped immutable segment names.
|
||||||
|
- Admin production build emits `dist/admin/index.html`, nginx serves `/admin/` as own entrypoint, stale hashed assets return `404` instead of HTML fallback.
|
||||||
|
- Restored nginx MIME mappings: built JS/CSS assets served with browser-safe content types.
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
- Preferred local access: `https://linearvoid.tv` and `https://linearvoid.tv/admin` through Cloudflare and Nginx Proxy Manager.
|
||||||
|
- Direct `localhost` access is fallback when public proxy path intentionally bypassed.
|
||||||
|
|
||||||
|
## Base URL Rollout: linearvoid.tv (Completed)
|
||||||
|
|
||||||
|
- Production URL model: app, admin, WebSocket on `https://linearvoid.tv`; HLS on dedicated `MEDIA_DOMAIN`, target `https://media.linearvoid.tv`.
|
||||||
|
- Introduced explicit public URL env vars for viewer, WebSocket, media endpoints; localhost-friendly defaults preserved for dev.
|
||||||
|
- Refactored frontend to stop hardcoding `localhost` URLs, use same-origin or env-driven endpoint resolution.
|
||||||
|
- Updated nginx config: host handling, SPA routing, `/ws`, `/hls/` match `linearvoid.tv` deployment model.
|
||||||
|
- Updated `.env.example`, `SETUP.md`, deployment docs with final DNS, Cloudflare, TLS, runtime values.
|
||||||
|
- Validated localhost dev and production domain behavior via frontend production builds using local and production public URL variables.
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
- Viewer SPA, admin UI, chat WebSocket use same origin on `linearvoid.tv`.
|
||||||
|
- Media delivery split from app origin for Cloudflare R2 + cache-rule alignment; production target `media.linearvoid.tv` via `MEDIA_DOMAIN`.
|
||||||
|
- Local dev keeps localhost defaults; production swaps only public hostnames, not internal compose service names.
|
||||||
|
- Public frontend URL vars in root `.env` / `.env.example`: `VITE_PUBLIC_APP_URL`, `VITE_PUBLIC_WS_URL`, `VITE_PUBLIC_HLS_URL`; Vite reads from repo root through `frontend/vite.config.ts`.
|
||||||
|
- Frontend resolves WebSocket and HLS from Vite env first, falls back to same-origin `/ws` and `/hls/playlist.m3u8`.
|
||||||
|
- Nginx treats `linearvoid.tv` as canonical app host, redirects `www.linearvoid.tv`, proxies `/ws` with forwarded headers, keeps `/hls/` as local fallback.
|
||||||
|
|
||||||
|
## Task List Maintenance - 2026-08-03
|
||||||
|
|
||||||
|
- Moved completed TODO items out of active queue.
|
||||||
|
- Updated architecture docs table of contents to include ADR-001 through ADR-007.
|
||||||
|
- Added ADR-002 through ADR-007 to architecture decisions section.
|
||||||
|
- Updated `SETUP.md` with latest docker-compose and environment guidance.
|
||||||
|
- Consolidated redundant active TODO items, reprioritized remaining work around `linearvoid.tv` rollout, release validation, documentation polish.
|
||||||
|
|
||||||
|
## Project Setup & Architecture Documentation
|
||||||
|
|
||||||
|
- Analyzed arc42 modules.
|
||||||
|
- Created clean project folder structure (frontend, backend/playout, backend/chat, backend/shared, infrastructure, scripts, monitoring).
|
||||||
|
- Created consolidated ADR-001 (merged previous drafts).
|
||||||
|
- Created ADR-002 (module structure) and ADR-003 (tech stack, languages, implicit assumptions).
|
||||||
|
- Fully populated `docs/architecture/README.md` (arc42 template with Mermaid diagrams, quality goals, deployment view, cross-cutting concepts, risks, glossary).
|
||||||
|
- Updated root `README.md` with Project Structure section and links to all ADRs.
|
||||||
|
- Initialized git repository.
|
||||||
|
- Created `.github/instructions/TODO.md` with detailed tracking.
|
||||||
|
|
||||||
|
## Foundational Configuration Files
|
||||||
|
|
||||||
|
- Created `.env.example` with R2, Cloudflare, Redis, PostgreSQL, ports, playout settings.
|
||||||
|
- Enhanced `.gitignore` for Node.js, Go, Docker, FFmpeg, data, build artifacts.
|
||||||
|
- Created root `docker-compose.yml` orchestrating playout, chat, nginx, redis, db services.
|
||||||
|
- Added `backend/playout/Dockerfile`, `start-playout.sh`, `.dockerignore`.
|
||||||
|
- Added `backend/chat/Dockerfile` (multi-stage Go build).
|
||||||
|
|
||||||
|
## Create ADRs for Data Storage, Caching, and Admin Capabilities
|
||||||
|
|
||||||
|
- Created ADR-006: Data Storage Strategy (SQLite vs PostgreSQL v1, migration path, admin interface to view/edit settings).
|
||||||
|
- Created ADR-007: Redis Usage (use cases, TTL policies, persistence, fallback behavior, admin interface to view stats, adjust rate limits/TTLs, flush keys, monitor health).
|
||||||
|
- Extended admin interface plans (from ADR-004/ADR-005) with "System Configuration" tab for Redis and storage management, full audit logging.
|
||||||
|
|
||||||
|
**Status**: Data storage and caching decisions documented. Admin interface capabilities to adjust Redis config values planned.
|
||||||
|
|
||||||
|
## Create ADRs for Admin Interfaces (Completed)
|
||||||
|
|
||||||
|
- Created ADR-004: Video Admin Web Interface (upload, playlist management, schedule overrides, normalization enforcement, Redis config tab from ADR-007).
|
||||||
|
- Created ADR-005: Web Admin Interfaces for user management, moderation dashboard, analytics, system configuration.
|
||||||
|
- Extended admin UI plans with shared layout, HTMX + Tailwind, auth, audit logging, Redis/storage management.
|
||||||
|
|
||||||
|
**Status**: Admin interface docs and implementation complete. Protected `/admin` UI included in frontend.
|
||||||
|
|
||||||
|
## Infrastructure & Monitoring (Completed)
|
||||||
|
|
||||||
|
- Nginx config for static serving, proxying, HLS delivery.
|
||||||
|
- Cloudflare cache rule examples and Worker templates.
|
||||||
|
- Prometheus and Grafana dashboards.
|
||||||
|
- Health checks and UptimeRobot configuration.
|
||||||
|
- Updated `docker-compose.yml` to include monitoring services.
|
||||||
|
|
||||||
|
**Overall Status**: All tasks completed. Fully functional, self-hosted 24/7 linear video channel with synchronized HLS playback, custom chat, R2 delivery, moderation, audit logging, admin interface, monitoring, complete docs (arc42 template + ADRs 001-007).
|
||||||
|
|
||||||
|
## Implement Playout Service (`backend/playout/`) (Completed)
|
||||||
|
|
||||||
|
- Playlist management, normalization validation (ffprobe), continuous HLS with immutable segment naming, automatic R2 sync, health check endpoint, Docker integration.
|
||||||
|
- Updated `docker-compose.yml`, Dockerfile, start script.
|
||||||
|
|
||||||
|
## Implement Chat Backend (`backend/chat/`) (Completed)
|
||||||
|
|
||||||
|
- Go + Gorilla WebSocket, Redis for presence/rate limiting/pub/sub, SQLite persistence for message history and ModerationLog.
|
||||||
|
- Public rooms, private DM routing, moderation commands with immutable audit logging, health check endpoint.
|
||||||
|
- Updated `docker-compose.yml` with volume for SQLite.
|
||||||
|
|
||||||
|
## Frontend (`frontend/`) (Completed)
|
||||||
|
|
||||||
|
- Vite + TypeScript + Tailwind + Alpine.js.
|
||||||
|
- HLS player with hls.js, chat sidebar with room switching and private messaging, "Now Playing" overlay.
|
||||||
|
- Modern UI with Tailwind styling and Alpine.js reactivity.
|
||||||
|
|
||||||
|
**Overall Status**: Backend and Frontend complete. Fully functional 24/7 linear video channel with synchronized playback, custom chat, R2 delivery, moderation, audit logging. Remaining: Infrastructure & Monitoring, admin interfaces (ADR-004/ADR-005), CI/CD, final polish.
|
||||||
|
|
||||||
|
## Create SETUP.md Guide (Completed)
|
||||||
|
|
||||||
|
- Prerequisites: Docker, Node.js, FFmpeg, Git.
|
||||||
|
- Environment Setup: `.env.example` to `.env`, R2 credentials, Cloudflare tokens, ports, Redis/PostgreSQL settings.
|
||||||
|
- Running: `docker compose up --build`, frontend dev server, health checks, monitoring (Prometheus/Grafana).
|
||||||
|
- Production Deployment: Hetzner VPS, Cloudflare cache rules, R2 bucket, systemd services.
|
||||||
|
- Troubleshooting: Common issues (R2 credentials, normalization errors, cache rules, Redis connection).
|
||||||
|
|
||||||
|
**Status**: SETUP.md complete. Comprehensive setup docs.
|
||||||
|
|
||||||
|
## Final Project Completion
|
||||||
|
|
||||||
|
- All tasks completed: Infrastructure & Monitoring, admin interface, documentation, CI/CD, final polish.
|
||||||
|
- Fully functional, self-hosted 24/7 linear video channel with synchronized HLS playback, custom chat, R2 delivery, moderation, audit logging, admin interface, monitoring, complete docs (arc42 template + ADRs 001-007).
|
||||||
|
|
||||||
|
**Overall Status**: Linear Void complete and ready for production. Run `docker compose up --build`. See `SETUP.md`.
|
||||||
|
|
||||||
|
## CI/CD Pipeline — Full Deployment (Completed 2026-08-04)
|
||||||
|
|
||||||
|
### CI Pipeline
|
||||||
|
|
||||||
|
- CI workflow on Gitea Actions runner — **passed** (task 569).
|
||||||
|
- `npm ci` — 38 packages, 0 vulnerabilities.
|
||||||
|
- `npm run build` — all outputs built (index.html, admin pages, assets).
|
||||||
|
- `docker compose build --no-cache` — all 3 images (playout, chat, nginx) built.
|
||||||
|
- Multi-stage nginx Dockerfile (frontend baked in).
|
||||||
|
- Removed frontend-only image from CI & release workflows.
|
||||||
|
- All images pushed to Gitea registry.
|
||||||
|
|
||||||
|
### CD Pipeline — Registry Login Fixed
|
||||||
|
|
||||||
|
- Renamed all `GITEA_*` → `CI_*` vars in `.env`, `.env.example`, `docker-compose.yml`, all workflows, `SETUP.md`, `DONE.md`.
|
||||||
|
- New secrets `CI_REGISTRY_USER` and `CI_REGISTRY_TOKEN` created in Gitea.
|
||||||
|
- Old `GITEA_USERNAME` and `GITEA_TOKEN` removed.
|
||||||
|
- **Login confirmed** — `cd-deploy-568.log` shows `Login Succeeded!` using new `CI_*` secrets.
|
||||||
|
|
||||||
|
### CD Pipeline — Push Images to Registry — FIXED ✅
|
||||||
|
|
||||||
|
- Added `Prepare environment` step to `cd.yml`: generates minimal `.env` inline.
|
||||||
|
- Configured `DOCKER_HOST: tcp://192.168.88.40:2375` in `cd.yml` — directs deploy/cleanup to test server at `192.168.88.40`.
|
||||||
|
- Updated health check to query `http://192.168.88.40` instead of `localhost`.
|
||||||
|
- CD login works + images pull.
|
||||||
|
- Added `.env` creation step to CD workflow.
|
||||||
|
- Documented Gitea Actions secrets setup in `SETUP.md`.
|
||||||
|
- Added secrets protection guidelines to `AGENTS.md`.
|
||||||
|
- **Fixed Prometheus mount failure**: switched to named volume `prometheus-config` + CD seeds config at deploy time.
|
||||||
|
- **Fixed playout crash-loop**: changed playlist volume from `:ro` to `:rw` so startup script can create demo playlist.
|
||||||
|
- **Fixed chat unhealthy status**: added `curl` to chat runtime Alpine image for healthcheck.
|
||||||
|
- **CD triggers after CI success**: changed CD trigger from `push` to `workflow_run` on CI completion.
|
||||||
|
- **Images built before deployment**: CI builds and pushes images, CD pulls and deploys.
|
||||||
|
- Containers start and pass health checks on `192.168.88.40`.
|
||||||
|
- `docker push git.allucanget.biz/zwitschi/linearvoid/playout:latest`
|
||||||
|
- `docker push git.allucanget.biz/zwitschi/linearvoid/chat:latest`
|
||||||
|
- `docker push git.allucanget.biz/zwitschi/linearvoid/nginx:latest` — **rebuilt with multi-stage Dockerfile, frontend bundled** ✅
|
||||||
|
|
||||||
|
### CD Pipeline — Track C: Run CD Workflow
|
||||||
|
|
||||||
|
- Images pulled on runner ✅.
|
||||||
|
- `.env` fix applied — `cd.yml` creates inline config before deploy.
|
||||||
|
- Health checks pass after containers start.
|
||||||
|
- `docker image prune -f` cleanup does not remove in-use images.
|
||||||
|
|
||||||
|
### Local Test Environment Deployment (Remote Docker Host)
|
||||||
|
|
||||||
|
- **Configure Alpine Docker Host** (`192.168.88.40`) to listen on TCP `2375`:
|
||||||
|
- Edit `/etc/conf.d/docker`, set `DOCKER_OPTS="-H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375"`.
|
||||||
|
- Restart Docker: `rc-service docker restart`.
|
||||||
|
- Verify: `netstat -tlnp | grep 2375`.
|
||||||
|
- **Configure local env** to point to remote Docker host at `192.168.88.40:2375` via `DOCKER_HOST`.
|
||||||
|
- **Verify connection**: `docker info` confirms communication with remote host.
|
||||||
|
- **Test remote build and deploy** from source: `docker compose up -d --build` — **succeeded after removing prometheus/grafana from gitignore** ✅.
|
||||||
|
- **Test remote production-like deploy** from Gitea registry: `docker compose -f docker-compose.prod.yml pull && docker compose -f docker-compose.prod.yml up -d`.
|
||||||
|
- **Verify remote health checks**:
|
||||||
|
- Playout: `http://192.168.88.40:8081/health`
|
||||||
|
- Chat: `http://192.168.88.40:3000/health`
|
||||||
|
- Nginx: `http://192.168.88.40:8080`
|
||||||
|
- **Reset local env** by clearing `DOCKER_HOST`.
|
||||||
|
|
||||||
|
### CI Docker Job (push to registry)
|
||||||
|
|
||||||
|
- CI `docker` job triggers after `test` passes on `main` push.
|
||||||
|
- CI pushes `playout`, `chat`, `nginx` images to Gitea Container Registry.
|
||||||
|
- Metadata tagging validated (branch-based + `latest`).
|
||||||
|
|
||||||
|
### Workflow Fixes & Adjustments
|
||||||
|
|
||||||
|
- **Fix `.env` heredoc indentation in `cd.yml`**: Removed leading spaces inside heredoc so Docker Compose parses correctly.
|
||||||
|
- **Remove `type=gha` cache from `ci.yml`**: Gitea Actions doesn't support GitHub Actions cache (`type=gha`). Removed.
|
||||||
|
- **Verify health check endpoints in `cd.yml`**: Confirmed `curl` targets match actual ports (`chat:3000`, `playout:8081`).
|
||||||
|
- **Fix nginx build in CI**: Switched from `docker/build-push-action` (ignored `dockerfile` param) to shell-based `docker buildx build --file`.
|
||||||
|
|
||||||
|
### CI/CD Pipeline Architecture Documentation
|
||||||
|
|
||||||
|
- Added CI/CD Pipeline Architecture section to `docs/architecture/README.md` with Mermaid diagram.
|
||||||
|
- Documented pipeline sequence: CI builds/pushes → CD pulls/deploy.
|
||||||
|
- Updated Deployment View with docker-compose services and remote host details.
|
||||||
|
|
||||||
|
## Gitea Self-Hosting Migration (Completed)
|
||||||
|
|
||||||
|
### Gitea Instance Setup & Repository Migration
|
||||||
|
|
||||||
|
- Verified Gitea instance at <https://git.allucanget.biz> accessible and configured — Gitea 1.27.1
|
||||||
|
- Created org/user `zwitschi` on Gitea
|
||||||
|
- Created repo `linearvoid` under `zwitschi` (`git.allucanget.biz/zwitschi/linearvoid`)
|
||||||
|
- Configured repo settings (visibility, default branch `main`, branch protection)
|
||||||
|
- Pushed existing code to Gitea remote
|
||||||
|
- Configured SSH keys for Gitea access (deploy keys for CI/CD)
|
||||||
|
- Set up Gitea Actions runner (self-hosted)
|
||||||
|
- Configured Gitea Container Registry (enable in app.ini, set up storage)
|
||||||
|
|
||||||
|
### CI/CD Migration to Gitea Actions
|
||||||
|
|
||||||
|
- Created `.gitea/workflows/` directory structure
|
||||||
|
- Moved `.github/workflows/ci.yml` to `.gitea/workflows/ci.yml`
|
||||||
|
- Adjusted CI workflow for Gitea Actions runner:
|
||||||
|
- Replaced `actions/checkout@v4` with Gitea-compatible checkout
|
||||||
|
- Updated `runs-on` to self-hosted runner labels
|
||||||
|
- Replaced `actions/setup-node@v4` with Gitea-compatible install
|
||||||
|
- Updated Docker image references to Gitea Container Registry
|
||||||
|
- Configured registry auth for Gitea Container Registry
|
||||||
|
- Added Gitea-specific env vars (CI_SERVER_URL, CI_REGISTRY_TOKEN)
|
||||||
|
- Created separate workflows for CI, CD, Release, Dependency updates
|
||||||
|
|
||||||
|
### Gitea Container Registry Setup
|
||||||
|
|
||||||
|
- Enabled Container Registry in Gitea config (app.ini)
|
||||||
|
- Configured storage backend
|
||||||
|
- Set up registry auth (token-based, OAuth2)
|
||||||
|
- Image naming convention: `git.allucanget.biz/zwitschi/linearvoid/<service>:<tag>`
|
||||||
|
- Image retention policies (keep last N tags, cleanup old images)
|
||||||
|
- Registry cleanup cron job
|
||||||
|
|
||||||
|
### Repository & Documentation Updates
|
||||||
|
|
||||||
|
- Updated all GitHub references in docs to Gitea URLs (README.md, CONTRIBUTING.md, SETUP.md, docs/architecture/README.md)
|
||||||
|
- Updated docker-compose.yml to use Gitea registry images
|
||||||
|
- Updated .env.example with Gitea registry variables
|
||||||
|
- Created .gitea/ directory for workflows and config
|
||||||
|
- Updated docker-compose.prod.yml for production with Gitea registry
|
||||||
|
- Removed .github/workflows/ after migration complete
|
||||||
|
|
||||||
|
### Infrastructure & Runner Setup
|
||||||
|
|
||||||
|
- Provisioned self-hosted Gitea Actions runner (VM or container)
|
||||||
|
- Installed Docker, Node.js, Go, FFmpeg on runner
|
||||||
|
- Registered runner with Gitea instance
|
||||||
|
- Configured runner labels (linux)
|
||||||
|
- Set up runner auto-scaling (optional)
|
||||||
|
- Configured runner logging and monitoring
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
# TODO
|
||||||
|
|
||||||
|
> Completed tasks moved to `.github/instructions/DONE.md`. Full remaining task list in `DEVELOPMENT.md`.
|
||||||
|
|
||||||
|
## Detailed Implementation Plan
|
||||||
|
|
||||||
|
- [x] Phase 1: unblock playout test fixtures and lock down the source-media contract
|
||||||
|
- [x] Create small compliant fixture media in `backend/playout/testdata/` for the validator and playout tests
|
||||||
|
- [x] Create invalid fixture media for codec, framerate, and audio rejection paths
|
||||||
|
- [x] Create playlist fixtures for empty, comment-only, mixed-validity, and valid playlists
|
||||||
|
- [x] Document the ffmpeg fixture-generation commands in the relevant test headers
|
||||||
|
|
||||||
|
- [ ] Phase 2: finish frontend coverage for the live viewer experience
|
||||||
|
- [x] Add tests for HLS player startup, manifest resolution, and cross-origin/local fallback behavior
|
||||||
|
- [x] Add tests for chat send/receive, presence updates, private DM flow, and reconnect behavior
|
||||||
|
- [ ] Add responsive-layout and accessibility checks for the viewer shell and chat sidebar
|
||||||
|
- [ ] Add coverage reporting and enforce a minimum threshold in CI
|
||||||
|
|
||||||
|
- [ ] Phase 3: finish admin frontend coverage
|
||||||
|
- [x] Add tests for playlist management, moderation flows, and dashboard rendering
|
||||||
|
- [x] Add tests for API integration points with mocked backend responses
|
||||||
|
- [ ] Add responsive-layout and accessibility checks for admin screens
|
||||||
|
- [ ] Add coverage reporting and enforce a minimum threshold in CI
|
||||||
|
|
||||||
|
- [x] Phase 4: validate the R2 and edge-delivery path
|
||||||
|
- [x] Add an integration test for upload and retrieval of HLS objects through the R2 path
|
||||||
|
- [x] Verify segment and manifest cache headers, including immutable media and live manifest freshness
|
||||||
|
- [ ] Validate edge cache hit/miss behavior with repeat requests
|
||||||
|
- [x] Document any required stubs or mocks so the test can run in CI (putObject mock pattern documented in test)
|
||||||
|
|
||||||
|
- [ ] Phase 5: complete manual Cloudflare and external-service setup
|
||||||
|
- [ ] Apply Cloudflare Pages, Tiered Cache, WAF, rate limiting, and HTTPS/HSTS settings in the dashboard
|
||||||
|
- [ ] Create or verify UptimeRobot monitors for health and HLS manifest availability
|
||||||
|
- [ ] Capture evidence in `docs/operations/cloudflare-uptimerobot-checklist.md`
|
||||||
|
|
||||||
|
- [ ] Phase 6: rehearse production deployment end to end
|
||||||
|
- [ ] Provision the Hetzner host and prepare the Docker/runtime baseline
|
||||||
|
- [ ] Configure production secrets, remote deployment, and DNS cutover
|
||||||
|
- [ ] Run health, HLS, chat, and admin smoke tests against production-like infrastructure
|
||||||
|
- [ ] Verify backup creation, restore, and integrity checks on the production host
|
||||||
|
|
||||||
|
- [ ] Phase 7: baseline scaling and performance
|
||||||
|
- [ ] Measure manifest TTFB and cache-hit distribution from multiple US regions
|
||||||
|
- [x] Prepare WebSocket load-test scaffolding and summary artifacts (`ws-chat.yml`, `run-loadtests.ps1`, `summarize-artillery.ps1`)
|
||||||
|
- [ ] Execute WebSocket throughput and chat-latency benchmarks under growing connection counts
|
||||||
|
- [x] Prepare Admin API benchmark scaffolding and endpoint summary artifacts (`admin-api.js`, `run-loadtests.ps1`, `summarize-k6.ps1`)
|
||||||
|
- [ ] Execute Admin API benchmarks and identify slow paths
|
||||||
|
- [x] Add saturation thresholds to monitoring (chat p95 latency + Redis memory usage)
|
||||||
|
- [x] Add cache-hit threshold alert operationalization to monitoring
|
||||||
|
|
||||||
|
## Status Update (2026-08-10)
|
||||||
|
|
||||||
|
- Phase 1 (test fixtures) complete.
|
||||||
|
- Phase 2 (viewer frontend): player startup, URL resolution, chat send/receive, reconnect, presence, DM flow, message rendering all covered (27 tests).
|
||||||
|
- Phase 3 (admin frontend): dashboard rendering, API integration, playlist management all covered (31 tests total).
|
||||||
|
- Phase 4 (R2 validation) partially complete: SyncToR2 cache headers + integration tests done; edge cache hit/miss remains manual.
|
||||||
|
- Phase 5 (Cloudflare): Pages deploy workflow (`.gitea/workflows/pages-deploy.yml`) created; Pages/Workers setup docs written; manual dashboard tasks remain.
|
||||||
|
- Phase 7: Loadtest scaffolding complete; `run-loadtests.ps1` auto-generates ws-chat-summary.txt and admin-api-endpoint-summary.txt; live benchmark execution pending.
|
||||||
|
- Version tagged 1.0.11; go vet clean.
|
||||||
|
|
||||||
|
## Remaining (summary)
|
||||||
|
|
||||||
|
- **Test Fixtures**: media + playlist fixtures created; ffmpeg generation commands documented in test headers
|
||||||
|
- **User Frontend Tests**: video player, chat integration, responsive, accessibility, coverage
|
||||||
|
- **Admin Frontend Tests**: playlist UI, moderation UI, metrics dashboards, coverage
|
||||||
|
- **Cloudflare R2 Integration Test**: upload + cache header validation done (SyncToR2 with CacheControl + ContentType); edge cache hit/miss validation remains manual
|
||||||
|
- **Cloudflare Config**: Cloudflare Pages deploy workflow and setup docs created; manual dashboard setup (Tiered Cache, WAF, rate limiting, HTTPS/HSTS) remains pending; Pages project must be created in dashboard first.
|
||||||
|
- **External Services**: UptimeRobot monitors + alerts
|
||||||
|
- **Production Deployment**: Hetzner provisioning, CD pipeline, DNS cutover, smoke test
|
||||||
|
- **Scaling & Performance**: TTFB baseline and live benchmark runs still pending; loadtest scaffolding now includes Artillery/k6 helpers with auto summaries plus worksheet field mapping; chat p95, Redis saturation, and cache-hit threshold alerting are implemented
|
||||||
|
|
||||||
|
See `DEVELOPMENT.md` for detailed checklist.
|
||||||
|
|
||||||
|
## Streaming Pipeline Gaps (2026-08-10 audit)
|
||||||
|
|
||||||
|
STREAMING.md contract vs playlist.go/main.ts implementation. See Backlog for full details.
|
||||||
|
|
||||||
|
- [ ] Continuous R2 sync during playout (HIGH) — currently only on FFmpeg exit
|
||||||
|
- [ ] Globally unique segment naming with run ID (HIGH) — currently strftime only
|
||||||
|
- [ ] Atomic manifest publication with temp key (MEDIUM)
|
||||||
|
- [ ] CORS headers on HLS responses from media.linearvoid.tv (MEDIUM)
|
||||||
|
- [ ] Player error recovery: retries, drift, reconnecting state (MEDIUM)
|
||||||
|
- [ ] Manifest Cache-Control: add must-revalidate + s-maxage split (LOW)
|
||||||
|
- [ ] Health endpoint: add active item, revision, run ID, segment info (LOW)
|
||||||
|
|
||||||
|
## Feature List (Future Enhancements)
|
||||||
|
|
||||||
|
- [ ] Admin backend
|
||||||
|
- [ ] Analytics: implement a settings entry to set matomo url and page id
|
||||||
|
- [ ] User frontend
|
||||||
|
- [ ] Chat: add user registration and login flows
|
||||||
|
- [ ] Chat: private DM history and search
|
||||||
|
- [ ] Analytics: add javascript snippet to send pageview events to matomo
|
||||||
+23
-2
@@ -52,8 +52,9 @@
|
|||||||
## Cloudflare Configuration (manual — Cloudflare dashboard)
|
## Cloudflare Configuration (manual — Cloudflare dashboard)
|
||||||
|
|
||||||
> Evidence tracker: `docs/operations/cloudflare-uptimerobot-checklist.md`
|
> Evidence tracker: `docs/operations/cloudflare-uptimerobot-checklist.md`
|
||||||
|
> Cloudflare Pages deploy workflow created (`.gitea/workflows/pages-deploy.yml`); setup docs written (`docs/operations/cloudflare-pages-setup.md`, `docs/operations/gitea-cloudflare.pages.md`).
|
||||||
|
|
||||||
- [ ] Cloudflare Pages: link Gitea repo, build `cd frontend && npm run build`, output `frontend/dist`
|
- [ ] Cloudflare Pages: create project in dashboard, link custom domain, or deploy via Gitea Actions Direct Upload
|
||||||
- [ ] Tiered Cache: enable Smart Tiered Cache, validate `Cf-Cache-Status: HIT` from upper tier
|
- [ ] Tiered Cache: enable Smart Tiered Cache, validate `Cf-Cache-Status: HIT` from upper tier
|
||||||
- [ ] WAF: enable OWASP Managed Ruleset, set to Simulate → review 48h → Block, enable Bot Fight Mode
|
- [ ] WAF: enable OWASP Managed Ruleset, set to Simulate → review 48h → Block, enable Bot Fight Mode
|
||||||
- [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action
|
- [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action
|
||||||
@@ -89,7 +90,27 @@
|
|||||||
- [ ] Measure TTFB: WebPageTest from 3+ US regions, target <4s warm cache, document baseline
|
- [ ] Measure TTFB: WebPageTest from 3+ US regions, target <4s warm cache, document baseline
|
||||||
- [ ] Load-test WebSocket: `artillery` or Go script, ramp 10→100→500→1000 connections, measure p95 latency + errors (scaffolding complete; benchmark execution pending)
|
- [ ] Load-test WebSocket: `artillery` or Go script, ramp 10→100→500→1000 connections, measure p95 latency + errors (scaffolding complete; benchmark execution pending)
|
||||||
- [ ] Benchmark admin API: `wrk`/`k6` against `/api/admin/*`, target p95 <1s, identify slow queries (scaffolding complete; benchmark execution pending)
|
- [ ] Benchmark admin API: `wrk`/`k6` against `/api/admin/*`, target p95 <1s, identify slow queries (scaffolding complete; benchmark execution pending)
|
||||||
|
- [x] Load-test scaffolding: ws-chat.yml, admin-api.js, run-loadtests.ps1, summarizers, README
|
||||||
|
- [x] Cache-hit ratio exporter (scripts/cache-hit-ratio.js + .ps1)
|
||||||
|
- [x] Saturation alerts: chat p95 latency, Redis memory usage, cache-hit ratio
|
||||||
- [ ] Track cache hit ratio: Cloudflare GraphQL → Grafana panel, alert <90%
|
- [ ] Track cache hit ratio: Cloudflare GraphQL → Grafana panel, alert <90%
|
||||||
- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, chat p95 >1s/5min, CPU >80%/10min, Redis memory >85%/10min
|
- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, CPU >80%/10min (chat p95, Redis memory, cache-hit already done)
|
||||||
- [ ] Phase 2 plan: document boundaries (playout CPU, chat latency, DB state, monitoring), estimate per-service resources
|
- [ ] Phase 2 plan: document boundaries (playout CPU, chat latency, DB state, monitoring), estimate per-service resources
|
||||||
- [ ] Phase 3 plan: research second Hetzner location, design warm standby + DNS failover + DB/Redis replication
|
- [ ] Phase 3 plan: research second Hetzner location, design warm standby + DNS failover + DB/Redis replication
|
||||||
|
|
||||||
|
## Streaming Pipeline (2026-08-10 audit)
|
||||||
|
|
||||||
|
Gaps between STREAMING.md contract and playlist.go/main.ts implementation:
|
||||||
|
|
||||||
|
- [ ] **Continuous R2 sync during playout (HIGH)**: `syncLatestToR2()` runs only after FFmpeg exits. Needs per-segment upload via filesystem watch or FFmpeg progress pipe.
|
||||||
|
- [ ] **Globally unique segment naming (HIGH)**: Add run/session UUID + monotonic sequence. Currently `strftime` only — collisions possible.
|
||||||
|
- [ ] **Atomic manifest publication (MEDIUM)**: Upload to temp key then atomically replace `live/playlist.m3u8`.
|
||||||
|
- [ ] **Manifest Cache-Control alignment (LOW)**: Add `must-revalidate` and `s-maxage`/`max-age` split.
|
||||||
|
- [ ] **CORS headers on HLS responses (MEDIUM)**: Add `Access-Control-Allow-Origin` to Worker or R2 custom domain.
|
||||||
|
- [ ] **Player error recovery (MEDIUM)**: Add `hls.on(Hls.Events.ERROR, ...)` with retries, drift, reconnecting state.
|
||||||
|
- [ ] **Health endpoint detail (LOW)**: Add active item, playlist revision, run ID, segment info, R2 pub timestamp.
|
||||||
|
- [x] Source media contract + validation — implemented
|
||||||
|
- [x] fMP4 HLS packaging (6s, 24 window) — implemented
|
||||||
|
- [x] Slate fallback + recovery — implemented
|
||||||
|
- [x] R2 cache headers (immutable segments, short-lived manifest) — implemented
|
||||||
|
- [x] Cloudflare Worker immutable `.m4s` headers — implemented
|
||||||
Reference in new issue
Block a user