linearvoid/docs/architecture/README.md

# Architecture Documentation for Linear Void

Linear Void architecture is documented using the [arc42 template](https://arc42.org/).

> Companion docs: see [Documentation README](../README.md) and `adr/` for full decision rationale, alternatives, and consequences.

- [Architecture Documentation for Linear Void](#architecture-documentation-for-linear-void)
  - [Introduction and Goals](#introduction-and-goals)
  - [Architecture Constraints](#architecture-constraints)
  - [Context and Scope](#context-and-scope)
  - [Solution Strategy](#solution-strategy)
  - [Building Block View](#building-block-view)
  - [Runtime View](#runtime-view)
  - [Deployment View](#deployment-view)
  - [Cross-cutting Concepts](#cross-cutting-concepts)
  - [Architecture Decisions](#architecture-decisions)
  - [Quality Requirements](#quality-requirements)
  - [Risks and Technical Debts](#risks-and-technical-debts)
  - [Glossary](#glossary)

## Introduction and Goals

**Requirements Overview** — Linear Void is an independent **24/7 linear video channel** broadcasting self-owned pre-recorded videos to a primarily US audience via a custom lightweight HTML/CSS/JS SPA with synchronized HLS playback, schedule metadata, public chat rooms, and private 1:1 messaging. Fully independent of third-party video platforms (YouTube, Twitch, etc.).

**Core Requirements** (merged from all ADRs):

- Continuous synchronized playlist playback (live-style HLS, not VOD).
- Self-hosted source media, frontend, schedule, and chat data.
- Primary US audience/origin (East Coast preferred for low latency/peering).
- Extreme cost efficiency (near-zero egress via Cloudflare R2 + edge caching).
- Custom chat with rooms, DMs, presence, moderation, persistence.
- Storage for 48h of 720p30/1080p30 content.
- Browser-compatible playback (`hls.js` + native HLS).
- Operational simplicity and scalability.

Full rationale in [`adr/ADR-001.md`](adr/ADR-001.md).

**Quality Goals**

| #   | Attribute           | Goal                                             | Scenario                                                                             |
| --- | ------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------ |
| 1   | Cost Efficiency     | <$25/month at moderate scale (100s concurrent)   | R2 $0 egress + immutable `.m4s`/`.ts` edge caching removes per-viewer bandwidth cost |
| 2   | Performance/Latency | Low US latency; acceptable HLS delay             | Hetzner Ashburn origin + Cloudflare US POPs (15–35s end-to-end)                      |
| 3   | Availability        | 24/7 with minimal downtime                       | FFmpeg/MediaMTX auto-restart, fallback slate, monitoring                             |
| 4   | Scalability         | Grow 10s→1000s viewers without proportional cost | Edge caching + R2; only request volume limits                                        |
| 5   | Maintainability     | Lightweight, clear configs, low ops              | Static SPA, Go/Node chat, SQLite/PostgreSQL+Redis                                    |
| 6   | Security/Usability  | Secure transport, moderation, good UX            | HTTPS/WSS, Cloudflare WAF, rate limits, intuitive chat/player                        |

**Stakeholders**: Product Owner, Engineering Team, US Viewers, Moderators/Admins, Operations — all expecting a cost-effective, reliable, low-latency, low-TCO independent channel with rich chat and easy management.

## Architecture Constraints

- No YouTube/Twitch/paid Cloudflare Stream dependency.
- US-centric (origin + audience).
- Cost stays low even at scale (bandwidth dominant).
- Standard browser playback only (no plugins).
- Cloudflare ToS compliance for video (R2 as origin for large immutable files).
- HLS primary (standard, not LL-HLS initially).
- Self-hosted playlist normalization and continuous playout.

## Context and Scope

**Business Context** — Linear Void runs a continuous scheduled, playlist-driven program to a US audience. The custom viewer site replaces traditional platforms, giving full control over content, schedule, UI, and real-time chat (public rooms + DMs).

**High-Level Context Diagram**

```mermaid
flowchart TD
    subgraph "External Users"
        Viewers[US Viewers<br/>Browser SPA + hls.js + Chat UI]
        Admins[Admins / Moderators<br/>Playlist & Schedule UI]
    end

    subgraph "Cloudflare Global Edge"
        CDN[Cloudflare CDN<br/>Cache Rules + WAF + TLS]
        Workers[Cloudflare Workers<br/>Static SPA Hosting]
        R2[Cloudflare R2<br/>HLS Segments + Source MP4s<br/>enam location hint]
    end

    subgraph "US-East Origin (Hetzner Ashburn VPS)"
        VPS[VPS CX22<br/>Nginx Proxy + Monitoring]
        Playout[Playout Service<br/>FFmpeg + MediaMTX/SRS<br/>Continuous HLS from Playlist]
        Chat[Chat Backend<br/>WebSockets + Redis + DB<br/>Public rooms + Private DMs]
    end

    Admins -->|HTTPS - Schedule/Playlist| Playout
    Playout -->|S3-compatible Upload| R2
    R2 -->|Cached HLS| CDN
    Viewers -->|HTTPS| Workers
    Viewers -->|HLS Requests| CDN
    Viewers -->|WSS Chat| Chat
    Chat --> VPS
    VPS -->|Proxy / Static| Workers
    CDN -->|Origin Pull on Miss| R2
```

**External Interfaces**: Cloudflare (CDN cache rules for `.m4s`/`.ts` vs `.m3u8`, R2 $0 egress, Workers static hosting, WAF/TLS); Hetzner VPS (US-East compute, good peering); Browsers (native HLS + WebSocket); Source (pre-normalized MP4s in R2).

**Technical Context**

- **Playout**: FFmpeg consuming normalized playlist → continuous HLS.
- **Packaging/Delivery**: MediaMTX/SRS/Nginx → immutable `.m4s`/`.ts` to R2; edge cache.
- **Frontend**: Static SPA with hls.js.
- **Chat**: Lightweight WebSocket service + Redis (presence/rate limits) + PostgreSQL/SQLite (history/moderation).
- **Normalization**: All sources H.264/AAC, constant 30 fps, 2s keyframes.

**Mapping**: Input = normalized MP4s in R2 + playlist; Output = live `master.m3u8` + segments via Cloudflare, chat via WSS.

## Solution Strategy

1. **US-East Playout on Hetzner Ashburn VPS** (CX22+): FFmpeg + MediaMTX/SRS continuous playlist→HLS (`-c copy` where possible).
2. **Immutable HLS to Cloudflare R2**: unique segment names; R2 origin with $0 egress.
3. **Aggressive Edge Caching**: long TTL for immutable `.m4s`/`.ts`; short (1–10s) for live `.m3u8`; Tiered Cache enabled — viewer delivery effectively free.
4. **Lightweight Custom Frontend**: static SPA + hls.js + chat sidebar.
5. **Separate co-located Chat Service**: WebSockets with persistence/moderation (low bandwidth).
6. **Normalization & Monitoring**: pre-process all videos; dashboards + alerts for playout health, cache hit ratio, R2 ops.

**Accepted Trade-off**: standard HLS latency (15–35s) acceptable for linear programming/chat; LL-HLS/WebRTC deferred.

## Building Block View

**Whitebox Overall System**

```mermaid
graph TD
    subgraph Cloudflare
        CDN[CDN + Cache Rules<br/>.m4s and .ts immutable long TTL<br/>.m3u8 short TTL]
        R2[R2 - HLS + Source]
        Workers[Workers - Static SPA]
    end
    VPS[VPS Hetzner Ashburn]
    Playout[FFmpeg + MediaMTX/SRS<br/>Playlist → HLS]
    Chat[WebSocket Chat<br/>Node/Go + Redis + DB]
    Viewer[US Viewers]
    Admin[Admin UI]

    Admin --> Playout
    Playout --> R2
    R2 --> CDN
    Viewer --> Workers
    Viewer --> CDN
    Viewer --> Chat
    Chat --> VPS
    VPS --> Nginx[Nginx Proxy]
```

**Motivation**: decouple playout (continuous linear), delivery (cached immutable segments), interaction (chat). R2 + edge caching solves bandwidth cost while keeping synchronization and independence.

**Contained blocks**: Playout (FFmpeg + MediaMTX/SRS); Storage/Delivery (R2 + CDN); Frontend (SPA); Chat Service (WebSocket); Proxy/Static (Nginx).

**Important interfaces**: HTTP(S) for HLS, WSS for chat, S3-compatible for R2 uploads.

**Level 2 — components with code locations and persistence**

```mermaid
graph TB
    subgraph "Cloudflare Edge"
        CF[CDN + WAF<br/>Cache Rules]
        R2["R2 Bucket<br/>media + HLS<br/>hls/r-{runID}/… run-scoped"]
        MEDIA[linearvoid-media Worker<br/>R2-bound · media.linearvoid.tv]
        WORKERS[linearvoid-frontend Worker<br/>SPA hosting]
    end

    subgraph "Docker Host"
        NGINX[nginx<br/>:80/:443<br/>proxy + auth]
        CHAT[chat service<br/>:3000<br/>Go + gorilla/websocket]
        PLAYOUT[playout service<br/>:8081<br/>Go + FFmpeg]
        REDIS[Redis 7<br/>:6379<br/>presence/rate/pubsub]
        DB[PostgreSQL 15<br/>:5432<br/>chat history/moderation]
    end

    subgraph "External"
        VIEWER[Viewer Browser<br/>hls.js + WebSocket]
        ADMIN[Admin Browser<br/>SPA + uploads]
    end

    VIEWER -->|HTTPS SPA| CF
    VIEWER -->|WSS Chat| NGINX
    VIEWER -->|HTTPS /hls| MEDIA
    ADMIN -->|HTTPS| CF
    CF --> NGINX
    MEDIA --> R2
    NGINX --> CHAT
    NGINX --> PLAYOUT
    CHAT --> REDIS
    CHAT --> DB
    PLAYOUT --> R2
    PLAYOUT -->|reads| R2
```

**Playout Service** (`backend/playout/src/playlist.go`)

- Role: continuous HLS encoding loop via FFmpeg concat demuxer.
- Input: `data/playlist/playlist.txt` (ordered MP4 paths); Output: HLS segments (`.m4s`, `.ts`) + live `.m3u8` to `data/hls/`, mirrored to R2 run-scoped under `hls/r-{runID}/…`.
- Run-scoped manifest: before publishing the live `hls/playlist.m3u8`, `publishManifest` rewrites bare media URIs to the current run folder (`rewriteManifestToRun`), so manifest → segment links resolve on `media.linearvoid.tv`. See [`STREAMING.md`](../STREAMING.md).
- Resilience: restart counter (>5/60s → slate); Prometheus on 8081.
- Validation: ffprobe gate (codec/fps/resolution/keyframe); `-validate` mode checks playlist without running.
- Seamless looping: resets HLS only when playlist fingerprint (SHA-256) changes, crash/slate recovery, or first run; else continues via `-start_number`.
- R2 retention: `cleanupOldRuns` keeps newest `R2_KEEP_RUNS` (default 2) capped at `R2_MAX_RETENTION_BYTES` (default 10 GiB), after each run + hourly backstop.
- Shared config: reads runtime `config.json` (`loadSharedHLSConfig`) as env fallback for HLS segment duration/window.

**Chat Service** (`backend/chat/src/main.go`)

- Role: WebSocket chat (rooms, DMs, moderation, admin API).
- Persistence: PostgreSQL 15 via GORM (messages, moderation, media metadata); SQLite dev fallback.
- Redis: presence, rate limiting, pub/sub multi-instance fan-out.
- Auth: session-cookie admin auth, Redis-backed expiry.
- Metrics on `/metrics`: connected clients, stored messages, DMs, moderation actions.

**Frontend SPA** (`frontend/`)

- Viewer `src/main.ts`: hls.js player, WS chat, presence, schedule overlay.
- Admin `src/admin.ts`: 9 tabs (dashboard, upload, playlist, schedule, moderation, chat-log, users, analytics, system).
- Build: Vite → `frontend/dist/`.

**Reverse Proxy** (`infrastructure/nginx/nginx.conf`)

- `auth_request` gate on `/admin` and `/api/admin`; WS upgrade for `/ws`; HLS CORS + OPTIONS preflight; `/health` JSON + `/nginx-status` stub_status.

**Level 3 — Chat Service Modules**

| Module             | Location                                                | Responsibility                                                             |
| ------------------ | ------------------------------------------------------- | -------------------------------------------------------------------------- |
| Connection Manager | `main.go` (hub, clients map)                            | Track active WS connections per room, fan-out                              |
| Message Parser     | `main.go` (readPump, handleMessage)                     | Parse JSON envelope `{type, room?, ...}`, validate, route                  |
| Room Registry      | `main.go` (rooms map)                                   | Room name → client set, join/leave/close                                   |
| DM Router          | `main.go` (handleDirectMessage)                         | Resolve user, deliver private message                                      |
| Persistence Layer  | `main.go` (GORM: Message, DirectMessage, ModerationLog) | Store/retrieve messages, DMs, moderation                                   |
| Redis Subsystem    | `main.go` (redisClient)                                 | INCR+EXPIRE rate limit, SETEX presence, PSUBSCRIBE fan-out                 |
| Auth Handler       | `main.go` (handleAdminLogin, adminAuthMiddleware)       | Session cookie create/validate, login rate limit                           |
| User Auth Handler  | `handlers/handlers.go`                                  | Viewer register/login/logout/session (bcrypt, user cookie)                 |
| DM History/Search  | `handlers/handlers.go`                                  | Paginated DM history + search                                              |
| Runtime Config     | `config/runtime.go`                                     | Thread-safe JSON runtime config (HLS, Matomo)                              |
| Admin API Router   | `main.go` (handleAdmin\*)                               | Upload, playlist CRUD, schedule overrides, moderation, analytics, system   |
| Metrics Exporter   | `main.go`                                               | connected_clients, messages_total, dms_total, moderation_actions, redis_up |

**Level 3 — Playout Service Modules**

| Module           | Location                                                | Responsibility                                                                    |
| ---------------- | ------------------------------------------------------- | --------------------------------------------------------------------------------- |
| Playlist Parser  | `playlist.go` (parsePlaylist)                           | Read playlist, skip comments/blanks, resolve paths                                |
| FFmpeg Manager   | `playlist.go` (runPlayout)                              | Spawn/monitor FFmpeg concat, restart on crash                                     |
| Loop Controller  | `playlist.go` (playlistFingerprint, hlsNextStartNumber) | Detect changes (SHA-256), continue numbering                                      |
| Slate Controller | `playlist.go` (activateSlate/deactivateSlate)           | Switch to slate on excessive failures                                             |
| R2 Sync          | `playlist.go` (syncToR2)                                | Upload segments + manifest to R2                                                  |
| R2 Cleanup       | `playlist.go` (cleanupOldRuns)                          | Keep last N run folders under size cap; delete `hls/r-*/*`                        |
| Health Endpoint  | `playlist.go` (healthHandler)                           | `/health` JSON: mode, segment count, uptime                                       |
| Metrics Exporter | `playlist.go`                                           | restarts_total, mode, loops_total, run_duration_seconds, r2_cleanup_objects_total |

**Level 3 — Frontend Modules**

| Module              | File                                            | Responsibility                                                      |
| ------------------- | ----------------------------------------------- | ------------------------------------------------------------------- |
| HLS Player          | `src/main.ts`                                   | hls.js init, attach, error recovery                                 |
| WebSocket Client    | `src/main.ts`                                   | Connect/parse envelope/dispatch                                     |
| Chat UI             | `src/main.ts`                                   | Message list, room tabs, DM panel, presence dots, typing, user-list |
| DM Search           | `src/main.ts`                                   | `/api/dm/search` + render                                           |
| Auth UI             | `src/main.ts` + `index.html`                    | Viewer login/register/logout, session, user label                   |
| Schedule Overlay    | `src/main.ts`                                   | Poll `/api/schedule`, render current/next                           |
| Admin Router        | `src/admin.ts`                                  | Tab nav, dynamic pages                                              |
| Upload Manager      | `src/admin.ts`                                  | Multipart chunk upload, normalization badges                        |
| Playlist Editor     | `src/admin.ts`                                  | Drag-reorder (gated to `ready` media), save                         |
| Moderation Panel    | `src/admin.ts` + `public/admin/moderation.html` | Mute/ban, action log                                                |
| Analytics Dashboard | `src/admin.ts` + `public/admin/analytics.html`  | Message counts, top rooms, moderation stats                         |
| System Config       | `src/admin.ts` + `public/admin/system.html`     | Health, runtime config form, restart                                |

## Runtime View

**Playout & Delivery Scenario**

1. Admin adds normalized MP4 to R2 + updates playlist.
2. Playout worker (FFmpeg concat loop) reads next file, packages HLS segments (2s).
3. Segments uploaded to R2 with unique immutable, run-scoped names (`hls/r-{runID}/…`); the live `.m3u8` is published referencing those run-prefixed URIs (see `STREAMING.md`).
4. Viewer loads SPA → requests manifest → hls.js follows current edge.
5. Cloudflare serves `.m4s`/`.ts` from nearest edge cache (origin hit once per POP/region).
6. Parallel WebSocket to chat service for synchronized interaction.

Manifest freshness keeps all viewers on the same timeline; cache hit ratio drives cost/latency. Fallback slate activates on failure. (Moderation events, scale spikes, failover in ADR.)

## Deployment View

**CI/CD Pipeline** — Parallel workflows on each push to `main`: **CI builds/pushes images** while **CD deploys** to the test lab (`192.168.88.40`); a dedicated release workflow handles semantic version tags. Production Hetzner VPS planned but not yet provisioned — same pipeline deploys there by changing `DOCKER_HOST`.

```mermaid
flowchart LR
    subgraph "Gitea Actions"
        CI[CI Workflow<br/>Build + Push Images]
        CD[CD Workflow<br/>Pull + Deploy]
    end
    subgraph "Gitea Container Registry"
        REGISTRY[git.allucanget.biz<br/>playout:latest<br/>chat:latest<br/>nginx:latest]
    end
    subgraph "Test Lab Host (192.168.88.40) — Production Hetzner TBD"
        DOCKER[Docker Engine<br/>via TCP :2375]
        PLAYOUT[linearvoid-playout]
        CHAT[linearvoid-chat]
        NGINX[linearvoid-nginx]
        REDIS[linearvoid-redis]
        DB[linearvoid-db]
        PROM[linearvoid-prometheus]
        GRAF[linearvoid-grafana]
    end

    CI -->|push| REGISTRY
    REGISTRY -->|pull| CD
    CD -->|deploy via DOCKER_HOST| DOCKER
    DOCKER --> PLAYOUT
    DOCKER --> CHAT
    DOCKER --> NGINX
    DOCKER --> REDIS
    DOCKER --> DB
    DOCKER --> PROM
    DOCKER --> GRAF
```

**Pipeline sequence**

1. **CI** (`.gitea/workflows/ci.yml`): on `push` to `main` + `pull_request`; `test` job (frontend lint/build, Go validation); `docker` job (after test, main only) builds `playout`/`chat`/`nginx` with Buildx, pushes `main` + `latest` tags to Gitea registry.
2. **CD** (`.gitea/workflows/cd.yml`): on `push` to `main` (parallel) or `workflow_dispatch`; logs into remote Docker host, generates `.env` from secrets, seeds Prometheus/Grafana volumes, deploys via `docker compose -f docker-compose.prod.yml up -d --remove-orphans --force-recreate` to `192.168.88.40:2375` (switch to Hetzner IP for prod); health checks with retry for playout init.
3. **Release** (`.gitea/workflows/release.yml`): on tags `*.*.*` (e.g. `1.0.9`); pushes `X.Y.Z`, `X.Y`, `latest`; creates Gitea release.

**Infrastructure Level 1** — Single low-cost US-East origin (Hetzner) with global edge caching. High cache-hit ratio, low US latency, edge-scale WAF/DDoS. Mapping: Playout + Chat + Nginx → Hetzner CX22 (Ashburn, VA); HLS storage/caching/static SPA → Cloudflare R2 + CDN + Workers (`enam` location hint); all public traffic through Cloudflare.

**Infrastructure Level 2**

- **Hetzner VPS (planned)**: playout (FFmpeg/MediaMTX), chat backend, Nginx; monitoring dashboards + UptimeRobot; backups to R2. Test lab currently fills this role (see `DEPLOYMENT.md`).
- **Cloudflare**: R2 bucket; Cache Rules (`.m4s`/`.ts` Cache Everything/immutable/long TTL; `.m3u8` short TTL/must-revalidate); WAF, rate limiting, Tiered Cache; Workers for SPA.
- Future: Docker Compose, managed DB, secondary regions (later ADRs).

## Cross-cutting Concepts

**Caching & Cost Optimization** — Immutable segment naming + targeted Cloudflare Cache Rules + R2 ($0 egress) = near-zero viewer bandwidth. Long TTL on `.m4s`/`.ts`, short on live manifests. Monitor hit ratio and R2 Class B ops.

**Video Normalization & Playout** — All sources must meet the Linear Void source profile before production playlist acceptance. The chat backend **auto-normalizes** non-conforming uploads on ingest (`media.NormalizeVideo`), removing manual pre-processing. Prevents decode resets, timestamp drift, AV discontinuities during concat playout.

_Required Source Profile (v1)_

| Field             | Required Value              | Notes                                            |
| ----------------- | --------------------------- | ------------------------------------------------ |
| Container         | MP4                         | Single-video-asset input                         |
| Video codec       | H.264/AVC (`libx264`)       | Baseline delivery codec                          |
| Audio codec       | AAC-LC                      | Stereo preferred                                 |
| Resolution        | 1280x720 or 1920x1080       | Keep single res per playlist block when possible |
| Framerate         | Constant 30 fps (`-r 30`)   | VFR must be normalized                           |
| GOP/keyframe      | 2s (`-g 60 -keyint_min 60`) | Matches HLS segment boundaries                   |
| Pixel format      | `yuv420p`                   | Browser baseline                                 |
| Audio sample rate | 48 kHz                      | Stable concat timing                             |
| Audio bitrate     | 128k–192k                   | Predictable envelope                             |

_Reference normalization command_

```bash
ffmpeg -i input.mov \
  -c:v libx264 -preset medium -crf 21 \
  -r 30 -g 60 -keyint_min 60 -sc_threshold 0 \
  -pix_fmt yuv420p \
  -c:a aac -b:a 160k -ar 48000 -ac 2 \
  -movflags +faststart \
  output-normalized.mp4
```

_Automatic normalization & verification (ffprobe)_

- `media.NormalizeVideo` runs ffmpeg with the profile, writes `*.normalized.mp4`, re-verifies with ffprobe.
- Media status: `ready`, `needs_normalization`, `normalizing`, `normalization_failed`; admin UI "Normalize" action with polling.
- `POST /api/admin/media/{id}/normalize` re-triggers manually.
- Keep verification aligned with runtime playout assumptions to preserve `-c copy` efficiency.

**Runtime Configuration Store** — Thread-safe, JSON-persisted store in `backend/chat/src/config/runtime.go`, shared with playout via the playlist volume (`/app/playlist/config.json`).

- Editable via `GET/PUT /api/admin/config`.
- Fields: `hlsSegmentDuration` (2–15s, default 2), `hlsWindowSize` (3–100, default 12), `matomoUrl`, `matomoSiteId`.
- Playout reads same file (`loadSharedHLSConfig`) as env fallback.
- Typed setters validate; invalid → `400`.

**Analytics (Matomo)** — Viewer sends pageviews via public `GET /api/matomo`, returning `matomoUrl` + `matomoSiteId` from the runtime config. Optional per deployment (empty `matomoUrl` disables).

**Monitoring & Resilience** — Two-layer model: (1) application layer (chat + platform health) via Prometheus + lightweight Grafana; (2) infrastructure/runtime layer (Prometheus self-observability, TSDB, Go runtime) via detailed Grafana dashboard.

_Prometheus baseline_ (`monitoring/prometheus/prometheus.yml`): 15s scrape; 7 jobs — `chat`→`chat:3000`, `playout`→`playout:8081`, `nginx`→`nginx-exporter:9113`, `redis`→`redis-exporter:9121`, `postgres`→`postgres-exporter:9187`, `node`→`node-exporter:9100`, `prometheus`→`localhost:9090`. TSDB retention `30d` + `15GB` disk cap (both compose files).

_Grafana dashboards_: `monitoring/grafana/dashboard.json` (app metrics: connected clients, messages, moderation, Redis reachability, admin sessions, scrape health); `monitoring/grafana/dashboards/dashboard.json` (Prometheus ops: status, uptime, CPU/mem, TSDB ingestion/churn/errors, HTTP latencies, query load, GC/scheduler). Use app dashboard for service behavior; Prometheus overview for platform diagnostics/capacity.

_Alerting rules_ (`monitoring/prometheus/alerts.yml`, loaded via `rule_files`): 10 rules — `ChatDown`, `PlayoutDown`, `RedisDown`, `PlayoutSlateActive`, `HighChatLatency`, `PrometheusWALFailing`, `HighCPU`, `LowCacheHitRatio`, `HighRedisMemoryUsage`, `LowDiskSpace`. Route warning/critical to ops channel; attach runbook links.

_Resilience baseline_: emergency slate fallback; auto-restart for playout/chat/nginx; weekly backup/restore drills for stateful data; incident runbook triage: edge reachability → origin health → playout progression → chat connectivity → data-plane dependencies.

**API Reference** — Three operational API surfaces (canonical contract in `API.md`).

_Chat WebSocket API_: endpoint `/ws`; `wss://` prod / `ws://` local; query params `username`, `room`; envelope `{room, username, text, type: public|private|system}`; responsibilities — room fan-out, DM routing, moderation commands, rate limiting/presence.

_Playout & Delivery REST_: playout `/health`; edge/origin `/health` via nginx; HLS manifest/segment namespace under `/hls/`. Liveness for orchestration + stable HLS path contract.

_Admin REST_: auth/session `/api/admin/login|logout|session`; domains overview/analytics/users/moderation/schedule/playlist/media; runtime config `GET/PUT /api/admin/config`; normalization `POST /api/admin/media/{id}/normalize`; uploads `/api/admin/media/uploads` (init/resume), `/uploads/{uploadId}/chunk`, `/complete`. Protected control plane; media metadata + upload orchestration; moderation/audit + system status.

_Viewer REST_: auth `POST /api/auth/register|login|logout`, `GET /api/auth/session`; DMs `GET /api/dm/history` (paginated), `GET /api/dm/search`; analytics `GET /api/matomo` (public).

**Cloudflare Configuration Reference** — Cloudflare is part of the production control plane, not only a CDN.

_DNS & zone routing_: `linearvoid.tv` → app origin (nginx); `www.linearvoid.tv` → redirect apex; `media.linearvoid.tv` → media domain for HLS/object delivery when split-domain enabled.

_Cache rules baseline_: `.ts`/`.m4s` Cache Everything, long TTL, immutable; `.m3u8` short TTL (1–10s), revalidation-friendly.

_Edge security/controls_: enforce HTTPS/WSS + redirect HTTP; WAF baseline; rate limits on abusive patterns (auth/admin); minimize origin exposure.

_Worker & header policy_: the `linearvoid-media` Worker ([`infrastructure/cloudflare/`](../../infrastructure/cloudflare/), R2-bound) owns `media.linearvoid.tv` and enforces immutable caching for segments, short TTL for live manifests, plus HLS CORS/Range; it is a faithful path→R2 mapper and does **not** rewrite manifest URIs at serve time (playout rewrites them to run-scoped paths at publish — see [`STREAMING.md`](../STREAMING.md)). The `linearvoid-frontend` Worker ([`frontend/wrangler.toml`](../../frontend/wrangler.toml)) serves the static SPA on `linearvoid.tv`. Automated deploys: `workers-deploy.yml` (SPA) and `workers-media-deploy.yml` (media). Ops detail: [`cloudflare-workers-setup.md`](../operations/cloudflare-workers-setup.md).

_Operations & validation_: verify cache status progression (MISS→HIT), manifest freshness under active playout; track hit ratio, request/error rates, ToS/cost posture.

**Security & Moderation** — Split across edge (Cloudflare), proxy (nginx), and application (chat/admin).

- _Transport/edge_: HTTPS/WSS; WAF + rate limiting; Worker enforces immutable headers.
- _nginx_: `auth_request` on `/admin/`, `/api/admin/`; admin UI → redirect `/admin/login/`; admin API → JSON `401`; WS upgrade forwarded for `/ws`.
- _Admin auth_: cookie session (`linearvoid_admin_session`); `HttpOnly`, `SameSite=Strict`, `Secure` when HTTPS (from TLS or `X-Forwarded-Proto`); server-side expiry.
- _Viewer accounts_: register/login/logout; bcrypt passwords; `linearvoid_user_session` cookie, 30-day default TTL (`USER_SESSION_TTL_HOURS`); `SameSite=None; Secure` over HTTPS (cross-site from Workers SPA), `SameSite=Lax` over plain HTTP (local dev); sign-in UI in `frontend/index.html`.
- _Rate limiting/abuse_: Redis `INCR + EXPIRE`; baseline `>10` messages in `10s` → throttle. Redis down → presence/rate-limit degrade gracefully (warn logged; core chat available).
- _CORS_: HLS endpoints `Access-Control-Allow-Origin: *`; API routes reflect allowlisted origin + `Access-Control-Allow-Credentials: true`; viewer cookie `SameSite=None; Secure` cross-site; admin cookie stays `SameSite=Strict`.
- _Moderation/privacy_: commands `/mod`, `/mute`, `/ban` handled in chat flow, persisted. DMs are application-level private, **not** end-to-end encrypted. Formalize moderation/audit retention in runbook/ops docs for production hardening.

**Chat** — WebSockets + Redis (presence, pub/sub, rate limits) + DB (history, audit). Public rooms + on-demand private chats; scales independently of video. DM history/search persisted; viewer UI has typing indicator, user-list toggle, DM search. (Details/cache tables/bitrate/cost model/configs in ADR-001.)

## Architecture Decisions

Consolidated ADRs (in `adr/`):

- **ADR-001**: overall architecture, HLS + R2 caching, storage calcs, cost model, risks.
- **ADR-002**: project folder/module structure and rationale.
- **ADR-003**: tech stack (TypeScript frontend; Go for chat/playout wrapper; Bash/Python scripts), Docker-first, documented assumptions.
- **ADR-004**: video admin web interface (upload, playlist, schedule overrides, normalization).
- **ADR-005**: admin UI for users, moderation dashboard, analytics, system config.
- **ADR-006**: data storage (SQLite vs PostgreSQL for v1, migration path).
- **ADR-007**: Redis usage (use cases, TTL, persistence, fallback, admin config).

**Key decisions**: self-managed US-East playout (Hetzner Ashburn) FFmpeg + MediaMTX/SRS → HLS → R2 + edge cache; module structure mirroring arc42 blocks; Go for perf-critical services, TypeScript frontend, Docker everywhere; admin interfaces (HTMX + custom CSS); SQLite primary for v1 with PostgreSQL migration path; Redis for rate limiting/presence/pub/sub/session with defined TTL and fallback.

## Quality Requirements

See **Quality Goals**. Primary: cost (near-zero egress), availability (auto-restart + fallback), scalability (edge caching), usability (synchronized player + responsive chat).

**Performance Benchmarks & Capacity Planning**

_Baseline SLO targets (v1)_: playback startup < 4s after first manifest request on warm cache; no segment starvation in steady state; end-to-end HLS 15–35s; chat delivery p95 < 500ms same region; admin read APIs p95 < 1s.

_Workload assumptions_: US audience, US-East origin; edge serves most segments, origin handles manifest churn + cache misses; chat low-bandwidth vs video.

_Initial capacity model_: bitrates — 720p30 ~2.6–4.2 Mbps, 1080p30 ~5.1–8.2 Mbps (with audio). 48h library — 720p30 ~57–90 GB raw (alloc ~80–110 GB); 1080p30 ~110–175 GB raw (alloc ~130–180 GB). Cost driven by cache efficiency + request volume, not direct origin egress.

_Benchmark plan_: playback (time-to-first-frame, rebuffers across US regions; manifest freshness/segment availability); chat (concurrent WS + fan-out; p50/p95 latency, Redis fallback); admin/API (p95 latency, error rate); platform (TSDB ingestion, query p99, GC in Grafana).

_Capacity triggers/actions_: viewer growth + rising cache-miss → tighten cache, enable/verify Tiered Cache, review manifest TTL; chat p95 drift/WS errors → scale chat, tune Redis, isolate moderation; Prometheus WAL/query saturation → tune retention/cardinality, add resources, split queries; origin CPU/mem pressure → optimize FFmpeg, isolate services, add secondary origin.

**Scaling Strategy** — preserve cost efficiency first, then throughput/resilience in phases:

- **Phase 1 — Vertical + Cache-First (current)**: single US-East origin, strong edge caching; optimize segment immutability + manifest TTL before adding infra; scale compute vertically on pressure.
- **Phase 2 — Service Separation**: split playout/chat/observability across nodes; isolate noisy workloads from latency-sensitive chat; introduce managed PostgreSQL/Redis or dedicated state nodes where justified.
- **Phase 3 — Multi-Origin Resilience**: warm standby or active-secondary origin; health-aware DNS/origin failover; keep HLS/cache contract stable.
- **Phase 4 — Regional/Throughput Expansion**: add origin region only when audience/latency justify; explicit chat affinity strategy; evaluate cache-miss economics vs origin complexity.
- _Scaling triggers_: rising cache-miss with sustained growth; chat p95/p99 drift or WS errors; persistent origin saturation; increased blast-radius from single-host dependency.

**Disaster Recovery & Backup**

_Recovery targets_: RPO ≤ 24h (chat/moderation/admin metadata); RTO ≤ 60 min (restore baseline streaming + chat).

_Backup scope_: persistent volumes — Postgres (`postgres-data`), Redis append-only (`redis-data`) where enabled, Prometheus TSDB (`prometheus-data`) if retention required, Grafana state (`grafana-data`); host-mounted dirs — `data/chat`, `data/playlist`, `data/media` (authoritative may be R2), `data/hls` (optional, regenerable); source of truth — compose, nginx, monitoring configs, env template.

_Cadence_: daily logical Postgres backups (+ weekly full snapshot); daily sync of chat/playlist dirs; weekly restore integrity check in isolated env. Retention: daily 14 days, weekly 8 weeks, monthly 6 months.

_Recovery runbook (minimum)_:

1. Restore infra config (compose, nginx, env, monitoring).
2. Restore Postgres + Redis state.
3. Restore `data/playlist` + chat persistence dirs.
4. Rehydrate media from authoritative object storage if missing.
5. Start stack; validate in order: playout `/health`, chat `/health`, nginx `/health`, HLS availability, WS chat connectivity.
6. Verify admin auth + moderation audit visibility.
7. Announce recovery; elevated monitoring ≥ one playlist cycle.

_Failure modes_: single-host loss → redeploy on standby, restore backups, repoint DNS/edge; storage corruption → restore last good snapshot + replay logical backups; partial failure → replace only impacted service, restore scoped data. Rehearse quarterly.

## Risks and Technical Debts

- Cloudflare ToS risk for heavy video on free tier (use R2; fallback to BunnyCDN).
- HLS latency (15–35s) may affect perceived chat sync.
- Single VPS = SPOF (mitigate with DNS failover, Phase 3).
- Upfront effort for normalization + cache tuning.
- Moderation/abuse operational load at scale.
- R2 request costs on cache misses (optimize Tiered Cache; measure in prod).
- Deferred: LL-HLS, ABR, E2EE, paid access controls, full Docker Compose.
- _Debt_: detailed service configs, CI/CD, advanced resilience left for implementation ADRs.

## Glossary

| Term              | Definition                                                                       |
| ----------------- | -------------------------------------------------------------------------------- |
| HLS               | HTTP Live Streaming — `.m3u8` manifests + media segments (fMP4).                 |
| Immutable Segment | HLS chunk with unique, unchanging name enabling long-lived caching.              |
| R2                | Cloudflare object storage ($0 egress; ideal origin for cached HLS).              |
| Edge Cache / POP  | Cloudflare points-of-presence caching/serving content near viewers.              |
| Playout           | Continuous generation of a scheduled linear program from a playlist.             |
| Normalization     | Pre-processing to consistent codec/fps/keyframe settings for seamless playback.  |
| Cache Rule        | Cloudflare config for caching behavior, TTL, headers per URL pattern.            |
| SPOF              | Single Point of Failure (current single VPS origin).                             |
| Tiered Cache      | Cloudflare feature reducing origin requests by checking upper-tier caches first. |

**References**: Cloudflare R2/Cache docs, MediaMTX/SRS/FFmpeg HLS guides, Hetzner pricing, hls.js docs, video bitrate calculators. Original root `ADR-001-*.md` files consolidated and can be archived/deleted.