From 528a7fffc08a23e3737e4286c621fe891e025097 Mon Sep 17 00:00:00 2001 From: zwitschi Date: Tue, 11 Aug 2026 15:54:00 +0200 Subject: [PATCH] chore: remove obsolete DEVELOPMENT.md and consolidate task tracking docs --- .github/AGENTS.md | 1 - .github/MEMORY.md | 12 ++++ .github/instructions/BACKLOG.md | 11 +++- .github/instructions/TODO.md | 111 +++++++++++++++----------------- CHANGELOG.md | 14 ++++ docs/DEVELOPMENT.md | 5 -- docs/README.md | 23 +++++-- 7 files changed, 108 insertions(+), 69 deletions(-) delete mode 100644 docs/DEVELOPMENT.md diff --git a/.github/AGENTS.md b/.github/AGENTS.md index 4b8c282..ee73dea 100644 --- a/.github/AGENTS.md +++ b/.github/AGENTS.md @@ -23,7 +23,6 @@ These files must be kept current with every change. Do not let them drift. | `.github/instructions/BACKLOG.md` | Deferred, manual, and operational tasks | Sync whenever TODO items complete or new gaps are identified | | `.github/instructions/DONE.md` | Archive of completed TODO sections | Move completed sections here from TODO.md | | `.github/MEMORY.md` | Session memory — what was done, when, and why | Append dated entry after each meaningful batch of changes | -| `docs/DEVELOPMENT.md` | Remaining task checklist organized by topic | Sync checkboxes with TODO.md after each implementation session | | `CHANGELOG.md` | User-facing release notes | Add entry when a phase or significant feature completes | | `docs/STREAMING.md` | Streaming pipeline contract | Update when playout, HLS, or caching behavior changes | | `docs/DEPLOYMENT.md` | Deploy and infrastructure procedures | Update when pipeline, topology, or provisioning changes | diff --git a/.github/MEMORY.md b/.github/MEMORY.md index 855bb92..c804ac0 100644 --- a/.github/MEMORY.md +++ b/.github/MEMORY.md @@ -122,3 +122,15 @@ - Frontend: 45 tests pass (2 files), test:coverage meets thresholds (22.92% / 18.44% / 19.01% / 25.42%) - Edge cache validation script `scripts/validate-edge-cache.sh` exists. - Remaining: CORS on media.linearvoid.tv (Cloudflare R2 bucket CORS config, not a code change). Phases 5-7 are manual/infrastructure only (Cloudflare dashboard, Hetzner provisioning, live benchmarks). + +## 2026-08-11 + +- Consolidated all development tasks from `DEVELOPMENT.md` and `.github/instructions/TODO.md` into `.github/instructions/BACKLOG.md` as the single source of truth. `DEVELOPMENT.md` now obsolete and removed. `TODO.md` references `BACKLOG.md` and `DONE.md`. +- Fixed R2 endpoint validation in playout service (`backend/playout/src/playlist.go`): + - Added `isR2Configured()` checking all four required env vars (`R2_ENDPOINT`, `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_BUCKET_NAME`). + - `initR2Client()` now returns a clear error if R2 is not configured instead of creating an S3 client with an empty endpoint. + - Segment watcher (`watchAndPublishSegments`), `syncLatestToR2()`, and `publishManifest()` are now guarded by `r2Client != nil` — no error spam when R2 is unavailable. + - Playout gracefully degrades: local HLS via nginx works without R2. +- Updated `.gitea/workflows/cd.yml` "Prepare environment" step to include all R2* variables sourced from Gitea secrets (`R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_BUCKET_NAME`, `R2_ENDPOINT`, `R2_ADMIN*\*`), Cloudflare variables (`CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`), and frontend public URL variables. +- Updated `docs/DEPLOYMENT.md` "Environment File" section: replaced vague "Cloudflare/R2 variables" bullet with explicit variable lists and added a new "Gitea Secrets" table documenting all 15 required secrets. +- All playout tests pass (20 tests), `go vet` clean. diff --git a/.github/instructions/BACKLOG.md b/.github/instructions/BACKLOG.md index b16a2f2..66cbe15 100644 --- a/.github/instructions/BACKLOG.md +++ b/.github/instructions/BACKLOG.md @@ -1,6 +1,15 @@ # BACKLOG -> Consolidated from `BACKLOG.md`, `DEVELOPMENT.md`, and `TODO.md` on 2026-08-11. +## Remaining from 2026-08-11 (summary) + +- **Test Fixtures**: media + playlist fixtures created; ffmpeg generation commands documented in test headers +- **User Frontend Tests**: video player, chat integration, responsive, accessibility, coverage +- **Admin Frontend Tests**: playlist UI, moderation UI, metrics dashboards, coverage +- **Cloudflare R2 Integration Test**: upload + cache header validation done (SyncToR2 with CacheControl + ContentType); edge cache hit/miss validation remains manual +- **Cloudflare Config**: Cloudflare Pages deploy workflow and setup docs created; manual dashboard setup (Tiered Cache, WAF, rate limiting, HTTPS/HSTS) remains pending; Pages project must be created in dashboard first. +- **External Services**: UptimeRobot monitors + alerts +- **Production Deployment**: Hetzner provisioning, CD pipeline, DNS cutover, smoke test +- **Scaling & Performance**: TTFB baseline and live benchmark runs still pending; loadtest scaffolding now includes Artillery/k6 helpers with auto summaries plus worksheet field mapping; chat p95, Redis saturation, and cache-hit threshold alerting are implemented ## Cloudflare Configuration diff --git a/.github/instructions/TODO.md b/.github/instructions/TODO.md index f38b229..3a30040 100644 --- a/.github/instructions/TODO.md +++ b/.github/instructions/TODO.md @@ -1,76 +1,71 @@ # TODO -> Completed tasks moved to `.github/instructions/DONE.md`. Full remaining task list in `DEVELOPMENT.md`. +> Completed tasks moved to [DONE](./DONE.md). Full remaining task list in [BACKLOG](./BACKLOG.md). -## Detailed Implementation Plan +## Plan -- [x] Phase 1: unblock playout test fixtures and lock down the source-media contract - - [x] Create small compliant fixture media in `backend/playout/testdata/` for the validator and playout tests - - [x] Create invalid fixture media for codec, framerate, and audio rejection paths - - [x] Create playlist fixtures for empty, comment-only, mixed-validity, and valid playlists - - [x] Document the ffmpeg fixture-generation commands in the relevant test headers +### Phase 1: unblock playout test fixtures and lock down the source-media contract -- [x] Phase 2: finish frontend coverage for the live viewer experience - - [x] Add tests for HLS player startup, manifest resolution, and cross-origin/local fallback behavior - - [x] Add tests for chat send/receive, presence updates, private DM flow, and reconnect behavior - - [x] Add responsive-layout and accessibility checks for the viewer shell and chat sidebar - - [x] Add coverage reporting and enforce a minimum threshold in CI +- [x] Create small compliant fixture media in `backend/playout/testdata/` for the validator and playout tests +- [x] Create invalid fixture media for codec, framerate, and audio rejection paths +- [x] Create playlist fixtures for empty, comment-only, mixed-validity, and valid playlists +- [x] Document the ffmpeg fixture-generation commands in the relevant test headers -- [x] Phase 3: finish admin frontend coverage - - [x] Add tests for playlist management, moderation flows, and dashboard rendering - - [x] Add tests for API integration points with mocked backend responses - - [x] Add responsive-layout and accessibility checks for admin screens - - [x] Add coverage reporting and enforce a minimum threshold in CI +### Phase 2: finish frontend coverage for the live viewer experience -- [x] Phase 4: validate the R2 and edge-delivery path - - [x] Add an integration test for upload and retrieval of HLS objects through the R2 path - - [x] Verify segment and manifest cache headers, including immutable media and live manifest freshness - - [x] Validate edge cache hit/miss behavior with repeat requests - - [x] Document any required stubs or mocks so the test can run in CI (putObject mock pattern documented in test) +- [x] Add tests for HLS player startup, manifest resolution, and cross-origin/local fallback behavior +- [x] Add tests for chat send/receive, presence updates, private DM flow, and reconnect behavior +- [x] Add responsive-layout and accessibility checks for the viewer shell and chat sidebar +- [x] Add coverage reporting and enforce a minimum threshold in CI -- [ ] Phase 5: complete manual Cloudflare and external-service setup - - [ ] Apply Cloudflare Pages, Tiered Cache, WAF, rate limiting, and HTTPS/HSTS settings in the dashboard - - [ ] Create or verify UptimeRobot monitors for health and HLS manifest availability - - [ ] Capture evidence in `docs/operations/cloudflare-uptimerobot-checklist.md` +### Phase 3: finish frontend coverage for the admin experience -- [ ] Phase 6: rehearse production deployment end to end - - [ ] Provision the Hetzner host and prepare the Docker/runtime baseline - - [ ] Configure production secrets, remote deployment, and DNS cutover - - [ ] Run health, HLS, chat, and admin smoke tests against production-like infrastructure - - [ ] Verify backup creation, restore, and integrity checks on the production host +- [x] Add tests for playlist management, moderation flows, and dashboard rendering +- [x] Add tests for API integration points with mocked backend responses +- [x] Add responsive-layout and accessibility checks for admin screens +- [x] Add coverage reporting and enforce a minimum threshold in CI -- [ ] Phase 7: baseline scaling and performance - - [ ] Measure manifest TTFB and cache-hit distribution from multiple US regions - - [x] Prepare WebSocket load-test scaffolding and summary artifacts (`ws-chat.yml`, `run-loadtests.ps1`, `summarize-artillery.ps1`) - - [ ] Execute WebSocket throughput and chat-latency benchmarks under growing connection counts - - [x] Prepare Admin API benchmark scaffolding and endpoint summary artifacts (`admin-api.js`, `run-loadtests.ps1`, `summarize-k6.ps1`) - - [ ] Execute Admin API benchmarks and identify slow paths - - [x] Add saturation thresholds to monitoring (chat p95 latency + Redis memory usage) - - [x] Add cache-hit threshold alert operationalization to monitoring +### Phase 4: validate the R2 and edge-delivery path + +- [x] Add an integration test for upload and retrieval of HLS objects through the R2 path +- [x] Verify segment and manifest cache headers, including immutable media and live manifest freshness +- [x] Validate edge cache hit/miss behavior with repeat requests +- [x] Document any required stubs or mocks so the test can run in CI (putObject mock pattern documented in test) + +### Phase 5: complete manual Cloudflare and external-service setup + +- [ ] Apply Cloudflare Pages, Tiered Cache, WAF, rate limiting, and HTTPS/HSTS settings in the dashboard +- [ ] Create or verify UptimeRobot monitors for health and HLS manifest availability +- [ ] Capture evidence in `docs/operations/cloudflare-uptimerobot-checklist.md` + +### Phase 6: rehearse production deployment end to end + +- [ ] Provision the Hetzner host and prepare the Docker/runtime baseline +- [ ] Configure production secrets, remote deployment, and DNS cutover +- [ ] Run health, HLS, chat, and admin smoke tests against production-like infrastructure +- [ ] Verify backup creation, restore, and integrity checks on the production host + +### Phase 7: baseline scaling and performance + +- [ ] Measure manifest TTFB and cache-hit distribution from multiple US regions +- [x] Prepare WebSocket load-test scaffolding and summary artifacts (`ws-chat.yml`, `run-loadtests.ps1`, `summarize-artillery.ps1`) +- [ ] Execute WebSocket throughput and chat-latency benchmarks under growing connection counts +- [x] Prepare Admin API benchmark scaffolding and endpoint summary artifacts (`admin-api.js`, `run-loadtests.ps1`, `summarize-k6.ps1`) +- [ ] Execute Admin API benchmarks and identify slow paths +- [x] Add saturation thresholds to monitoring (chat p95 latency + Redis memory usage) +- [x] Add cache-hit threshold alert operationalization to monitoring ## Status Update (2026-08-10) -- Phase 1 (test fixtures) complete. -- Phase 2 (viewer frontend): player startup, URL resolution, chat send/receive, reconnect, presence, DM flow, message rendering all covered (27 tests). -- Phase 3 (admin frontend): dashboard rendering, API integration, playlist management all covered (31 tests total). -- Phase 4 (R2 validation) partially complete: SyncToR2 cache headers + integration tests done; edge cache hit/miss remains manual. -- Phase 5 (Cloudflare): Pages deploy workflow (`.gitea/workflows/pages-deploy.yml`) created; Pages/Workers setup docs written; manual dashboard tasks remain. -- Phase 7: Loadtest scaffolding complete; `run-loadtests.ps1` auto-generates ws-chat-summary.txt and admin-api-endpoint-summary.txt; live benchmark execution pending. +- [x] Phase 1 (test fixtures) complete. +- [x] Phase 2 (viewer frontend): player startup, URL resolution, chat send/receive, reconnect, presence, DM flow, message rendering all covered (27 tests). +- [x] Phase 3 (admin frontend): dashboard rendering, API integration, playlist management all covered (31 tests total). +- [x] Phase 4 (R2 validation) partially complete: SyncToR2 cache headers + integration tests done; edge cache hit/miss remains manual. +- [ ] Phase 5 (Cloudflare): Pages deploy workflow (`.gitea/workflows/pages-deploy.yml`) created; Pages/Workers setup docs written; manual dashboard tasks remain. +- [ ] Phase 6 (production deployment) pending: Hetzner provisioning, secrets, DNS cutover, smoke tests, backup/restore. +- [ ] Phase 7: Loadtest scaffolding complete; `run-loadtests.ps1` auto-generates ws-chat-summary.txt and admin-api-endpoint-summary.txt; live benchmark execution pending. - Version tagged 1.0.11; go vet clean. -## Remaining (summary) - -- **Test Fixtures**: media + playlist fixtures created; ffmpeg generation commands documented in test headers -- **User Frontend Tests**: video player, chat integration, responsive, accessibility, coverage -- **Admin Frontend Tests**: playlist UI, moderation UI, metrics dashboards, coverage -- **Cloudflare R2 Integration Test**: upload + cache header validation done (SyncToR2 with CacheControl + ContentType); edge cache hit/miss validation remains manual -- **Cloudflare Config**: Cloudflare Pages deploy workflow and setup docs created; manual dashboard setup (Tiered Cache, WAF, rate limiting, HTTPS/HSTS) remains pending; Pages project must be created in dashboard first. -- **External Services**: UptimeRobot monitors + alerts -- **Production Deployment**: Hetzner provisioning, CD pipeline, DNS cutover, smoke test -- **Scaling & Performance**: TTFB baseline and live benchmark runs still pending; loadtest scaffolding now includes Artillery/k6 helpers with auto summaries plus worksheet field mapping; chat p95, Redis saturation, and cache-hit threshold alerting are implemented - -See `DEVELOPMENT.md` for detailed checklist. - ## Streaming Pipeline Gaps (2026-08-10 audit → 2026-08-10 implemented) STREAMING.md contract vs playlist.go/main.ts implementation. See Backlog for full details. diff --git a/CHANGELOG.md b/CHANGELOG.md index 5d3afa7..3c0b61e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,20 @@ +## [Unreleased] + +## [1.0.12] - 2026-08-11 + +### Changed + +- **Task tracking consolidated**: Removed `DEVELOPMENT.md` and moved all development tasks to `.github/instructions/BACKLOG.md` as the single source of truth. `TODO.md` updated to reference `BACKLOG.md` and `DONE.md`. + +### Fixed + +- **Playout R2 endpoint validation**: `initR2Client()` now validates all required R2 environment variables (`R2_ENDPOINT`, `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_BUCKET_NAME`) before creating the S3 client. Previously, an empty `R2_ENDPOINT` caused cryptic `Custom endpoint '' was not a valid URI` errors on every segment upload attempt. Playout now gracefully degrades when R2 is not configured — local HLS via nginx continues to work. +- **CD workflow missing R2 secrets**: `.gitea/workflows/cd.yml` "Prepare environment" step now injects all R2_ variables (`R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_BUCKET_NAME`, `R2_ENDPOINT`, plus admin variants) from Gitea secrets. Previously the generated `.env` was missing R2 entirely, causing the playout container to start with an empty endpoint. +- **Deployment docs**: `docs/DEPLOYMENT.md` now includes explicit R2 and Cloudflare variable lists plus a "Gitea Secrets" table documenting all 15 required secrets. + ## [1.0.11] - 2026-08-10 ### Added diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md deleted file mode 100644 index 201bfe4..0000000 --- a/docs/DEVELOPMENT.md +++ /dev/null @@ -1,5 +0,0 @@ -# DEVELOPMENT.md - -> **All task lists have been consolidated into [`.github/instructions/BACKLOG.md`](../.github/instructions/BACKLOG.md).** -> -> This file is kept as a reference for development setup instructions. See [`SETUP.md`](SETUP.md) for environment setup. diff --git a/docs/README.md b/docs/README.md index 6dd9152..9de72cd 100644 --- a/docs/README.md +++ b/docs/README.md @@ -6,6 +6,15 @@ This folder contains the documentation for the Linear Void project. The document - [Architecture](architecture/README.md) - High-level system architecture and component interactions. +## Infrastructure + +- Cloudflare + - [Cache Rules](./infrastructure/cloudflare/cache-rules.md) - Cache rules for Cloudflare Workers and Pages. + - [Account Troubleshooting](./infrastructure/cloudflare/cloudflare-account-troubleshooting.md) - Troubleshooting Cloudflare account issues. +- Monitoring + - [Prometheus CURL](./infrastructure/monitoring/prometheus_curl.md) - Prometheus metrics endpoint testing with curl. + - [Prometheus CURL Windows](./infrastructure/monitoring/prometheus_curl_win.md) - Prometheus metrics endpoint testing with curl on Windows. + ## API - [API](API.md) - Implemented WebSocket and REST API reference. @@ -30,12 +39,18 @@ This folder contains the documentation for the Linear Void project. The document - [Streaming](STREAMING.md) - Streaming architecture, playout, and delivery details. -## Development - -- [Development](DEVELOPMENT.md) - Remaining tasks and test fixtures for development. - ## Additional Resources - [Marketing](marketing/README.md) - Marketing materials and resources. - [Operations](operations/README.md) - Operational procedures and checklists. - [Incident Runbook](incident-runbook.md) - Incident response and disaster recovery procedures. + +## Other Documents + +- [CHANGELOG](../CHANGELOG.md) - Release notes and version history. +- [CONTRIBUTING](../CONTRIBUTING.md) - Contribution guidelines and code of conduct. +- [TODO](../.github/instructions/TODO.md) - Active implementation plan with checkboxes. +- [Backlog](../.github/instructions/BACKLOG.md) - Consolidated list of all development tasks. +- [Agents](../.github/AGENTS.md) - Instructions for agents and task management. +- [Memory](../.github/MEMORY.md) - Session memory and development progress tracking. +- [README](../README.md) - Project overview and high-level information.