From 4c02ab9d55d80e1263967666774f100516701594 Mon Sep 17 00:00:00 2001 From: zwitschi Date: Thu, 6 Aug 2026 22:01:29 +0200 Subject: [PATCH] fix: parallel CD trigger, remove env blocker, cleanup release pipeline --- .gitea/workflows/cd.yml | 16 +------ .gitea/workflows/ci.yml | 19 +------- CHANGELOG.md | 6 ++- CONTRIBUTING.md | 30 ++++++------ DEPLOYMENT.md | 48 +++++++++++++++++-- README.md | 6 +-- SETUP.md | 2 +- docker-compose.prod.yml | 2 +- docs/architecture/README.md | 9 +++- docs/architecture/adr/ADR-004-admin-video.md | 2 - .../adr/ADR-005-admin-moderation-analytics.md | 4 -- docs/architecture/adr/ADR-007-redis-usage.md | 2 - docs/incident-runbook.md | 40 +++++++--------- 13 files changed, 95 insertions(+), 91 deletions(-) diff --git a/.gitea/workflows/cd.yml b/.gitea/workflows/cd.yml index 194ab9b..291b58c 100644 --- a/.gitea/workflows/cd.yml +++ b/.gitea/workflows/cd.yml @@ -1,23 +1,18 @@ # CD workflow — TEST LAB DEPLOYMENT # Deploys to test lab host 192.168.88.40:2375. # Production Hetzner target TBD — switch DOCKER_HOST when provisioned. -# See TODO.md Production Deployment section for cutover plan. +# See DEPLOYMENT.md Production Deployment section for cutover plan. name: CD on: - workflow_run: - workflows: ["CI"] - types: - - completed + push: branches: [main] workflow_dispatch: jobs: deploy: runs-on: [ubuntu-latest] - if: ${{ github.event.workflow_run.conclusion == 'success' || github.event_name == 'workflow_dispatch' }} - environment: production steps: - name: Checkout code uses: actions/checkout@v4 @@ -27,13 +22,6 @@ jobs: - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 - - name: Log in to Gitea Container Registry - uses: docker/login-action@v3 - with: - registry: git.allucanget.biz - username: ${{ secrets.CI_REGISTRY_USER }} - password: ${{ secrets.CI_REGISTRY_TOKEN }} - - name: Prepare environment run: | cat > .env << 'EOF' diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index f09f50e..03659f2 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -43,7 +43,7 @@ jobs: run: cd backend/chat && GOTOOLCHAIN=local go vet ./... - name: Run Tests - run: echo "Tests will be added in future CI steps (per TODO.md)" + run: echo "Tests will be added in future CI steps" docker: needs: test @@ -132,20 +132,3 @@ jobs: labels: ${{ steps.meta-nginx.outputs.labels }} cache-from: type=registry,ref=git.allucanget.biz/zwitschi/linearvoid/buildcache:nginx cache-to: type=registry,ref=git.allucanget.biz/zwitschi/linearvoid/buildcache:nginx,mode=max - - release: - needs: docker - runs-on: [ubuntu-latest] - if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/') - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - - name: Create Release - uses: softprops/action-gh-release@v1 - with: - generate_release_notes: true - env: - GITHUB_TOKEN: ${{ secrets.CI_REGISTRY_TOKEN }} diff --git a/CHANGELOG.md b/CHANGELOG.md index 3192f9a..2fb90c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,10 @@ ### Changed - **Prod prometheus**: Switched from empty named volume to bind mounts for config files (consistent with dev). +- **CD pipeline**: Trigger on `push` (parallel with CI), removed `environment: production` (was causing silent abort). +- **CI pipeline**: Removed redundant `release` job (handled by `release.yml`). +- **Tag format**: Standardized on `*.*.*` (e.g. `1.0.9`), removed `v` prefix from existing tags. +- **DEPLOYMENT.md**: Documented parallel CI/CD pipeline flow with workflow table. ### Removed @@ -154,7 +158,7 @@ Full backend and frontend implementation complete. - **Frontend** (`frontend/`): Vite + TypeScript + Tailwind + Alpine.js, HLS player with hls.js, chat sidebar with room switching and private messaging, "Now Playing" overlay, and modern UI (ADR-002, ADR-003). - Complete `docker-compose.yml` for local development with all services, volumes, and healthchecks. - All ADRs (001-007) created and linked in documentation. -- `TODO.md`, `DONE.md`, and `CHANGELOG.md` updated with full traceability. +- `CHANGELOG.md` updated with full traceability. - Git repository initialized with comprehensive initial commit. The project is a fully functional, self-hosted 24/7 linear video channel with synchronized playback, custom chat, R2 delivery, moderation, and admin-ready structure. It matches the architecture defined in the arc42 template and all ADRs. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 11a409c..24a4974 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -7,9 +7,8 @@ Thank you for considering contributing to Linear Void! 1. **Read the Architecture First**: All design decisions are documented in `docs/architecture/README.md` (arc42 template) and the ADRs in `docs/architecture/adr/` (ADR-001 to ADR-007). 2. **Follow the Structure**: Use the folder layout defined in ADR-002 (`frontend/`, `backend/playout/`, `backend/chat/`, `infrastructure/`, etc.). 3. **Update Documentation Before Code**: Create or update an ADR before implementing significant changes. -4. **Update TODO.md**: Plan tasks in `.github/instructions/TODO.md`, mark as completed, and move finished sections to `DONE.md`. -5. **Commit Messages**: Reference the relevant ADR (e.g., "feat(playout): add R2 sync per ADR-001"). -6. **Test with Docker**: Run `docker compose up --build` to test changes. +4. **Commit Messages**: Reference the relevant ADR (e.g., "feat(playout): add R2 sync per ADR-001"). +5. **Test with Docker**: Run `docker compose up --build` to test changes. ## Repository @@ -40,23 +39,22 @@ Linear Void follows **Semantic Versioning** ([SemVer](https://semver.org)): `MAJ - **MINOR** — new features (endpoints, admin tabs, UI capabilities) that are backward-compatible. - **PATCH** — bug fixes, optimizations, documentation polish. -Git tags use a `v` prefix: `1.0.2`, `1.1.0`, `v2.0.0`. +Git tags use no prefix: `1.0.2`, `1.1.0`, `2.0.0`. ### Where Versions Are Declared -| File | Token | Example | -| ------------------------------ | -------------- | ------------------------- | -| `frontend/package.json` | `"version"` | `"1.0.2"` | -| `CHANGELOG.md` | `## [X.Y.Z]` | `## [1.0.2] - 2026-08-05` | -| `README.md` | Version badge | `version-1.0.2-blue` | -| `.github/instructions/TODO.md` | Section header | `## Version 1.0.2` | +| File | Token | Example | +| ----------------------- | ------------- | ------------------------- | +| `frontend/package.json` | `"version"` | `"1.0.2"` | +| `CHANGELOG.md` | `## [X.Y.Z]` | `## [1.0.2] - 2026-08-05` | +| `README.md` | Version badge | `version-1.0.2-blue` | ### Tag Triggers -| Tag Pattern | Workflow | What Happens | -| ------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------ | -| `v*` (e.g. `1.0.2`) | `release.yml` | Builds all 3 images, pushes to registry with semver tags (`1.0.2`, `1.0`, `latest`), creates Gitea Release via API | -| Push to `main` | `ci.yml` → `cd.yml` | CI builds + pushes images with `main`/`latest` tags, CD deploys to production | +| Tag Pattern | Workflow | What Happens | +| ---------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------ | +| `*.*.*` (e.g. `1.0.2`) | `release.yml` | Builds all 3 images, pushes to registry with semver tags (`1.0.2`, `1.0`, `latest`), creates Gitea Release via API | +| Push to `main` | `ci.yml` → `cd.yml` | CI builds + pushes images with `main`/`latest` tags, CD deploys to production | ### Full Release Procedure @@ -87,7 +85,7 @@ Add entries under `## [X.Y.Z]` in `CHANGELOG.md` following [Keep a Changelog](ht ```bash # Stage all version bumps + changelog -git add frontend/package.json CHANGELOG.md README.md .github/instructions/TODO.md +git add frontend/package.json CHANGELOG.md README.md # Commit git commit -m "chore: bump version to X.Y.Z" @@ -145,6 +143,6 @@ git.allucanget.biz/zwitschi/linearvoid/playout:latest ### Moving Completed Tasks -After release, move completed sections from `TODO.md` to `DONE.md` and update `CHANGELOG.md` if needed. The `DONE.md` file serves as the historical record linking tasks to releases. +After release, update `CHANGELOG.md` if needed. See the ADRs for architectural decisions and `docs/architecture/README.md` for the big picture. Thank you for contributing! diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index 252533e..db6d532 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -1,5 +1,33 @@ # Deployment Guide +## Table of Contents + +- [Deployment Guide](#deployment-guide) + - [Table of Contents](#table-of-contents) + - [Environments](#environments) + - [Production Target (Hetzner)](#production-target-hetzner) + - [Planned Infrastructure](#planned-infrastructure) + - [Cutover Plan](#cutover-plan) + - [Firewall / Security Group](#firewall--security-group) + - [Pipeline Readiness](#pipeline-readiness) + - [Target Topology](#target-topology) + - [Prerequisites](#prerequisites) + - [Environment File](#environment-file) + - [Pull-and-Run Deployment](#pull-and-run-deployment) + - [First-Time or Recovery Provisioning](#first-time-or-recovery-provisioning) + - [Health Verification](#health-verification) + - [Zero/Low-Downtime Update Pattern](#zerolow-downtime-update-pattern) + - [Rollback](#rollback) + - [Operational Tasks](#operational-tasks) + - [View logs](#view-logs) + - [Restart one service](#restart-one-service) + - [Stop stack](#stop-stack) + - [Keep volumes (recommended) vs remove volumes (destructive)](#keep-volumes-recommended-vs-remove-volumes-destructive) + - [Nginx and Proxy Considerations](#nginx-and-proxy-considerations) + - [Security Baseline](#security-baseline) + - [Common Deployment Pitfalls](#common-deployment-pitfalls) + - [Troubleshooting](#troubleshooting) + ## Environments Linear Void runs in two separate environments: @@ -9,7 +37,7 @@ Linear Void runs in two separate environments: | **Test Lab** | `192.168.88.40` (internal) | `tcp://192.168.88.40:2375` | Integration testing, staging, CI/CD validation | | **Production** | Hetzner VPS (TBD) | `tcp://:2375` | Public linearvoid.tv serving real viewers | -**Current state (2026-08-06):** Test lab is active and receives automated CD deploys. Production Hetzner VPS is **not yet provisioned**. The CD workflow (`cd.yml`) and `docker-compose.prod.yml` currently target the test lab. Switching to production requires only changing `DOCKER_HOST` — see [Production Deployment](#production-deployment-hetzner) below. +**Current state (2026-08-06):** Test lab is active and receives automated CD deploys. Production Hetzner VPS is **not yet provisioned**. The CD workflow (`cd.yml`) and `docker-compose.prod.yml` currently target the test lab. Switching to production requires only changing `DOCKER_HOST` — see [Production Deployment](#production-target-hetzner) below. ## Production Target (Hetzner) @@ -28,7 +56,7 @@ Linear Void runs in two separate environments: ### Cutover Plan -1. Provision Hetzner VPS (see `TODO.md` #92) +1. Provision Hetzner VPS (see [Production Deployment](#production-target-hetzner) section) 2. Add production secrets to Gitea (`CI_PROD_*` prefix) 3. Update `DOCKER_HOST` in CD workflow to Hetzner public IP 4. Trigger deploy → validate health checks @@ -49,7 +77,17 @@ Linear Void runs in two separate environments: ### Pipeline Readiness -The Gitea Actions CD workflow (`cd.yml`) already supports `DOCKER_HOST` as a configurable variable. To switch targets: +Three Gitea Actions workflows drive the full pipeline: + +| Workflow | Trigger | Purpose | +| ------------- | ---------------- | ---------------------------------------------------------------------------------------- | +| `ci.yml` | Push to `main` | Lint, build frontend, vet Go, build & push Docker images (`:latest`, `:main`) | +| `cd.yml` | Push to `main` | Deploy to test lab (`192.168.88.40:2375`) — runs **in parallel** with CI | +| `release.yml` | Tag push `*.*.*` | Build & push versioned Docker images (`:X.Y.Z`, `:X.Y`, `:latest`), create Gitea Release | + +CI and CD run simultaneously on every push to `main`. The deploy pulls whatever `:latest` images are available in the registry at that moment. + +The CD workflow (`cd.yml`) uses `DOCKER_HOST` to target the deployment host. To switch targets: ```bash # Test lab (current) @@ -59,7 +97,7 @@ DOCKER_HOST=tcp://192.168.88.40:2375 DOCKER_HOST=tcp://:2375 ``` -No code changes needed in the pipeline. See `TODO.md` #91 and #92–#99 for full production deployment checklist. +No code changes needed in the pipeline. See `DEPLOYMENT.md` Production Deployment section for full checklist. ## Target Topology @@ -99,7 +137,7 @@ See .env.example and SETUP.md for full variable reference. Images are built and pushed automatically by the CI and release workflows (see [CONTRIBUTING.md](CONTRIBUTING.md#versioning-and-releases) for the release procedure). -Every push to `main` produces `:main` and `:latest` tags. Semantic version tags (`v*`) produce `:X.Y.Z`, `:X.Y`, and `:latest` tags. +Every push to `main` triggers CI (builds `:main` and `:latest` tags) and CD (deploys `:latest` to test lab) in parallel. Semantic version tags (`*.*.*`, e.g. `1.0.9`) trigger the release workflow which builds `:X.Y.Z`, `:X.Y`, and `:latest` tags and creates a Gitea Release. From repository root: diff --git a/README.md b/README.md index 867e635..8d41efa 100644 --- a/README.md +++ b/README.md @@ -112,7 +112,7 @@ This structure directly reflects the building blocks, runtime flows, deployment ## Getting Started -> **Note:** The `docker compose up` below targets local development. The test lab (192.168.88.40) receives automated CD deploys. Production Hetzner VPS is not yet provisioned — see `TODO.md`. +> **Note:** The `docker compose up` below targets local development. The test lab (192.168.88.40) receives automated CD deploys. Production Hetzner VPS is not yet provisioned — see `DEPLOYMENT.md`. ```bash git clone https://git.allucanget.biz/zwitschi/linearvoid.git @@ -223,11 +223,11 @@ Contributions, issues, and further evolution (LL-HLS, ABR, multi-channel, E2EE) Contributions are welcome! See [`CONTRIBUTING.md`](CONTRIBUTING.md) for guidelines. Quick overview: 1. Read the [architecture docs](docs/architecture/README.md) and ADRs first. -2. Plan tasks in `.github/instructions/TODO.md`. +2. Code against the defined module structure (frontend, backend/playout, backend/chat, infrastructure, scripts, monitoring). 3. Open PRs on [Gitea](https://git.allucanget.biz/zwitschi/linearvoid). 4. CI validates builds and tests automatically. -Since this a linear project, we also rely on a linear development and release model: Only main branch is used for releases, and all features are merged into main after review and CI validation. Releases are tagged with semantic versioning (1.0.3, 1.0.4, etc.) and automatically built/pushed to the container registry. +Since this a linear project, we also rely on a linear development and release model: Only main branch is used for releases, and all features are merged into main after review and CI validation. Releases are tagged with semantic versioning (1.0.3, 1.0.4, etc.) and automatically built/pushed to the container registry. Version numbers are declared in `frontend/package.json`, `CHANGELOG.md`, and the version badge in this README. For patches, we use even numbers for stable versions (1.0.2, 1.0.4) and odd numbers for pre-release/unstable versions (1.0.3, 1.0.5). [MIT License](LICENSE) diff --git a/SETUP.md b/SETUP.md index e26e586..f0c3c80 100644 --- a/SETUP.md +++ b/SETUP.md @@ -55,7 +55,7 @@ flowchart LR **Test Lab (active):** Internal network `192.168.88.0/24`, host `192.168.88.40`. Receives automated CD deploys from Gitea Actions. Used for integration testing and staging validation. -**Production (pending):** Hetzner Cloud VPS, Ashburn VA. Not yet provisioned. CD workflow and compose files are ready — only `DOCKER_HOST` needs changing. See `TODO.md` Production Deployment section. +**Production (pending):** Hetzner Cloud VPS, Ashburn VA. Not yet provisioned. CD workflow and compose files are ready — only `DOCKER_HOST` needs changing. See Production Deployment section in [DEPLOYMENT.md](./DEPLOYMENT.md) for cutover plan. ## Development Prerequisites diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index 8a0c64b..23e78d0 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -1,6 +1,6 @@ # docker-compose.prod.yml — TEST LAB DEPLOYMENT # Target: test lab host 192.168.88.40 (NOT Hetzner production). -# Production Hetzner VPS not yet provisioned — see TODO.md Production Deployment section. +# Production Hetzner VPS not yet provisioned — see DEPLOYMENT.md Production Deployment section. # Uses pre-built images from Gitea Container Registry (no local build). # Override DOCKER_HOST in CD workflow to switch target. diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 4924aab..46291ef 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -20,7 +20,14 @@ - [Building Block View](#building-block-view) - [Whitebox Overall System](#whitebox-overall-system) - [Level 2](#level-2) + - [Playout Service (`backend/playout/src/playlist.go`)](#playout-service-backendplayoutsrcplaylistgo) + - [Chat Service (`backend/chat/src/main.go`)](#chat-service-backendchatsrcmaingo) + - [Frontend SPA (`frontend/`)](#frontend-spa-frontend) + - [Reverse Proxy (`infrastructure/nginx/nginx.conf`)](#reverse-proxy-infrastructurenginxnginxconf) - [Level 3](#level-3) + - [Chat Service Modules](#chat-service-modules) + - [Playout Service Modules](#playout-service-modules) + - [Frontend Modules](#frontend-modules) - [Runtime View](#runtime-view) - [Playout \& Delivery Scenario](#playout--delivery-scenario) - [Deployment View](#deployment-view) @@ -453,7 +460,7 @@ flowchart LR ### Infrastructure Level 2 -**Hetzner VPS (planned):** Will run playout (FFmpeg/MediaMTX), chat backend, Nginx. Monitoring via dashboards + UptimeRobot. Backups to R2. Currently test lab (`192.168.88.40`) serves this role. Production Hetzner instance not yet provisioned — see `TODO.md` Production Deployment section. +**Hetzner VPS (planned):** Will run playout (FFmpeg/MediaMTX), chat backend, Nginx. Monitoring via dashboards + UptimeRobot. Backups to R2. Currently test lab (`192.168.88.40`) serves this role. Production Hetzner instance not yet provisioned — see Production Deployment section in [`DEPLOYMENT.md`](../../DEPLOYMENT.md). **Cloudflare Layer**: diff --git a/docs/architecture/adr/ADR-004-admin-video.md b/docs/architecture/adr/ADR-004-admin-video.md index 6e87c2c..11c92fa 100644 --- a/docs/architecture/adr/ADR-004-admin-video.md +++ b/docs/architecture/adr/ADR-004-admin-video.md @@ -60,8 +60,6 @@ Create a dedicated **admin web interface** in a new `frontend/admin/` directory - Integration with existing playout service and R2 keeps the architecture consistent. - Admin actions will be fully auditable, paving the way for ADR-005 (moderation and user management). -This decision was planned in `TODO.md` before implementation, per AGENTS.md. - ## Consequences **Positive**: diff --git a/docs/architecture/adr/ADR-005-admin-moderation-analytics.md b/docs/architecture/adr/ADR-005-admin-moderation-analytics.md index 3fc8dff..d744ae7 100644 --- a/docs/architecture/adr/ADR-005-admin-moderation-analytics.md +++ b/docs/architecture/adr/ADR-005-admin-moderation-analytics.md @@ -59,8 +59,6 @@ Extend the admin interface started in ADR-004 with additional protected sections - Leveraging Prometheus/Grafana for analytics avoids duplicating metrics collection. - Immutable audit logging supports future compliance and moderation review (key requirement for a public chat system). -This ADR was planned in `TODO.md` before any implementation, per AGENTS.md. - ## Consequences **Positive**: @@ -92,8 +90,6 @@ This ADR was planned in `TODO.md` before any implementation, per AGENTS.md. 7. Update authentication middleware and tests. 8. Document API contracts and update `docker-compose.yml` to expose the admin interface securely. -**Next Steps**: After both admin ADRs are complete, move to implementation of the Playout Service (highest priority in the core modules section of TODO.md). - ## References - ADR-001: Chat features, moderation requirements, monitoring needs. diff --git a/docs/architecture/adr/ADR-007-redis-usage.md b/docs/architecture/adr/ADR-007-redis-usage.md index 63abda9..4a0da35 100644 --- a/docs/architecture/adr/ADR-007-redis-usage.md +++ b/docs/architecture/adr/ADR-007-redis-usage.md @@ -95,8 +95,6 @@ We must explicitly document: 6. Add monitoring (Prometheus exporter) and admin audit logging for configuration changes. 7. Write tests for rate limiting, presence, and admin configuration flows. -**Next Steps**: After ADR-006 and ADR-007 are reviewed, mark the "Create ADRs for Data Storage, Caching, and Admin Capabilities" section as completed in TODO.md and move it to DONE.md. Then proceed with implementation of the Playout Service. - ## References - ADR-001: Chat features requiring presence, rate limiting, and pub/sub. diff --git a/docs/incident-runbook.md b/docs/incident-runbook.md index 5cf7d81..48d0516 100644 --- a/docs/incident-runbook.md +++ b/docs/incident-runbook.md @@ -4,15 +4,11 @@ Triage order for Linear Void incidents. Work top-to-bottom — each tier depends ## Triage Order -``` 1. Edge Reachability (Cloudflare) 2. Origin Health (VPS/Docker) 3. Playout Progression (HLS) 4. Chat Connectivity (WebSocket) 5. Data-Plane (Redis/DB) -``` - ---- ## 1. Edge Reachability (Cloudflare) @@ -28,17 +24,17 @@ curl -I https://linearvoid.tv/health ``` **Common causes:** + - DNS misconfiguration (A record pointing to wrong IP) - Cloudflare proxy disabled (grey cloud → origin exposed directly) - SSL certificate expired (check Cloudflare dashboard → SSL/TLS) -- Cloudflare outage → https://www.cloudflarestatus.com +- Cloudflare outage → check [https://www.cloudflarestatus.com](https://www.cloudflarestatus.com) **Mitigation:** + - Verify A record: `linearvoid.tv` → origin IP (proxied, orange cloud) - Bypass Cloudflare temporarily: `curl -H "Host: linearvoid.tv" http:///health` ---- - ## 2. Origin Health (VPS/Docker) **Symptoms:** Cloudflare returns 502/504, containers down. @@ -53,18 +49,18 @@ curl http://192.168.88.40/health ``` **Common causes:** + - Host out of disk space: `df -h` - Docker daemon stopped: `systemctl status docker` - Container restart loop: `docker ps -a --filter "status=exited"` - Resource exhaustion (OOM): `dmesg | grep -i oom` **Mitigation:** + - Restart stack: `docker compose -f docker-compose.prod.yml up -d` - Free disk: `docker system prune -f` - Check logs per service: `docker logs linearvoid- --tail 100` ---- - ## 3. Playout Progression (HLS) **Symptoms:** Viewer sees slate or buffering, playlist not advancing. @@ -83,19 +79,19 @@ docker logs linearvoid-playout --tail 100 ``` **Common causes:** + - FFmpeg crash → slate fallback active (health shows `degraded`) - Playlist file empty/missing: `cat data/playlist/playlist.txt` - Media file missing from playlist: `ls data/media/` - Disk full → HLS segments can't write **Mitigation:** + - Verify playlist has valid entries - Check media files exist at paths in playlist - Restart playout: `docker compose restart playout` - Generate demo playlist if empty: see `backend/playout/start-playout.sh` ---- - ## 4. Chat Connectivity (WebSocket) **Symptoms:** Chat not loading, messages not sending, presence broken. @@ -112,18 +108,18 @@ docker logs linearvoid-chat --tail 100 ``` **Common causes:** + - Redis down → presence/rate-limiting degraded (chat still works) - Database down → messages not persisting - Port not exposed: `docker port linearvoid-chat` **Mitigation:** + - Restart chat: `docker compose restart chat` - Check Redis: `docker exec linearvoid-redis redis-cli ping` - Check DB: `docker exec linearvoid-db pg_isready -U linearvoid` - Chat works without Redis (graceful degradation) — check logs for "Redis connection warning" ---- - ## 5. Data-Plane (Redis/DB) **Symptoms:** Chat works but no persistence/presence, admin stats empty. @@ -139,32 +135,30 @@ docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;" ``` **Common causes:** + - Redis OOM (check `maxmemory` setting) - PostgreSQL connection limit reached - Disk full on volume mount **Mitigation:** + - Redis: `docker compose restart redis` (AOF persist enabled) - PostgreSQL: `docker compose restart db` - Restore from backup: see `scripts/verify-backups.sh` for procedure ---- - ## Escalation -| Severity | When | Action | -|---|---|---| -| Critical | Slate active >30min, complete outage | Trigger DR plan, provision fallback host | -| Warning | Redis down, high CPU >30min | Investigate during business hours, restart service | -| Info | Single service blip | Monitor, no action unless recurring | +| Severity | When | Action | +| -------- | ------------------------------------ | -------------------------------------------------- | +| Critical | Slate active >30min, complete outage | Trigger DR plan, provision fallback host | +| Warning | Redis down, high CPU >30min | Investigate during business hours, restart service | +| Info | Single service blip | Monitor, no action unless recurring | ## Post-Incident 1. Document timeline in incident channel 2. Identify root cause -3. Add prevention to TODO.md if actionable +3. Add prevention to docs if actionable 4. Update this runbook if new failure mode discovered ---- - **Recovery targets:** RPO ≤24h, RTO ≤60min. Quarterly DR drill: simulate host loss, restore from backup.