# Architecture Documentation for Linear Void

> The architecture documentation for the Linear Void project is structured according to the [arc42 template](https://arc42.org/). This template provides a comprehensive framework for documenting software architecture, ensuring that all relevant aspects of the system are covered.

## Table of Contents

- [Architecture Documentation for Linear Void](#architecture-documentation-for-linear-void)
  - [Table of Contents](#table-of-contents)
  - [Companion Documentation](#companion-documentation)
  - [Introduction and Goals](#introduction-and-goals)
    - [Requirements Overview](#requirements-overview)
    - [Quality Goals](#quality-goals)
    - [Stakeholders](#stakeholders)
  - [Architecture Constraints](#architecture-constraints)
  - [Context and Scope](#context-and-scope)
    - [Business Context](#business-context)
    - [High-Level Context Diagram (Mermaid)](#high-level-context-diagram-mermaid)
    - [Technical Context](#technical-context)
  - [Solution Strategy](#solution-strategy)
  - [Building Block View](#building-block-view)
    - [Whitebox Overall System](#whitebox-overall-system)
    - [Level 2](#level-2)
    - [Level 3](#level-3)
  - [Runtime View](#runtime-view)
    - [Playout \& Delivery Scenario](#playout--delivery-scenario)
  - [Deployment View](#deployment-view)
    - [CI/CD Pipeline Architecture](#cicd-pipeline-architecture)
    - [Infrastructure Level 1](#infrastructure-level-1)
    - [Infrastructure Level 2](#infrastructure-level-2)
  - [Cross-cutting Concepts](#cross-cutting-concepts)
    - [Caching \& Cost Optimization](#caching--cost-optimization)
    - [Video Normalization \& Playout](#video-normalization--playout)
      - [Required Source Profile (Linear Void v1)](#required-source-profile-linear-void-v1)
      - [Reference Normalization Command](#reference-normalization-command)
      - [Verification Gate (ffprobe)](#verification-gate-ffprobe)
    - [Monitoring \& Resilience](#monitoring--resilience)
      - [Prometheus Baseline](#prometheus-baseline)
      - [Grafana Dashboards](#grafana-dashboards)
      - [Alerting Rules (Prometheus Rule Groups)](#alerting-rules-prometheus-rule-groups)
    - [API Documentation Reference](#api-documentation-reference)
      - [Chat WebSocket API](#chat-websocket-api)
      - [Playout and Delivery REST API](#playout-and-delivery-rest-api)
      - [Admin REST API](#admin-rest-api)
    - [Cloudflare Configuration Reference](#cloudflare-configuration-reference)
      - [DNS and Zone Routing](#dns-and-zone-routing)
      - [Cache Rules Baseline](#cache-rules-baseline)
      - [Edge Security and Controls](#edge-security-and-controls)
      - [Worker and Header Policy](#worker-and-header-policy)
      - [Operations and Validation](#operations-and-validation)
    - [Security \& Moderation](#security--moderation)
      - [Transport and Edge Protection](#transport-and-edge-protection)
      - [Reverse Proxy Enforcement (nginx)](#reverse-proxy-enforcement-nginx)
      - [Admin Authentication and Session Security](#admin-authentication-and-session-security)
      - [Rate Limiting and Abuse Controls](#rate-limiting-and-abuse-controls)
      - [CORS and Cross-Origin Behavior](#cors-and-cross-origin-behavior)
      - [Moderation and Privacy](#moderation-and-privacy)
    - [Chat](#chat)
  - [Architecture Decisions](#architecture-decisions)
  - [Quality Requirements](#quality-requirements)
    - [Performance Benchmarks and Capacity Planning](#performance-benchmarks-and-capacity-planning)
      - [Baseline SLO Targets (v1)](#baseline-slo-targets-v1)
      - [Workload Assumptions](#workload-assumptions)
      - [Initial Capacity Model](#initial-capacity-model)
      - [Benchmark Plan](#benchmark-plan)
      - [Capacity Triggers and Actions](#capacity-triggers-and-actions)
    - [Scaling Strategy](#scaling-strategy)
      - [Phase 1: Vertical + Cache-First (Current Baseline)](#phase-1-vertical--cache-first-current-baseline)
      - [Phase 2: Service Separation](#phase-2-service-separation)
      - [Phase 3: Multi-Origin Resilience](#phase-3-multi-origin-resilience)
      - [Phase 4: Regional and Throughput Expansion](#phase-4-regional-and-throughput-expansion)
      - [Scaling Triggers](#scaling-triggers)
    - [Disaster Recovery and Backup Procedures](#disaster-recovery-and-backup-procedures)
      - [Recovery Targets](#recovery-targets)
      - [Backup Scope](#backup-scope)
      - [Backup Cadence](#backup-cadence)
      - [Recovery Runbook (Minimum)](#recovery-runbook-minimum)
      - [Failure Modes and Strategy](#failure-modes-and-strategy)
  - [Risks and Technical Debts](#risks-and-technical-debts)
  - [Glossary](#glossary)

## Companion Documentation

- [API.md](../../API.md) - Implemented WebSocket and REST API reference.
- [DEPLOYMENT.md](../../DEPLOYMENT.md) - Production deployment and operational runbook.
- [MONITORING.md](../../MONITORING.md) - Metrics, dashboards, and monitoring troubleshooting.
- [SECURITY.md](../../SECURITY.md) - Security model, controls, and hardening baseline.

## Introduction and Goals

### Requirements Overview

Linear Void is an independent **24/7 linear video channel** broadcasting a playlist of self-owned pre-recorded videos. Viewers access it via a custom lightweight HTML/CSS/JavaScript SPA with synchronized HLS playback, schedule metadata, public chat rooms, and private 1:1 messaging. Complete independence from third-party video platforms (YouTube, Twitch, etc.).

**Core Requirements** (merged from all ADRs):

- Continuous synchronized playback from playlist (live-style HLS, not VOD).
- Self-hosted source media, frontend, schedule, and chat data.
- Primary US audience/origin (East Coast preferred for low latency/peering).
- Extreme cost efficiency (bandwidth dominant; target near-zero egress via Cloudflare R2 + edge caching).
- Custom chat with rooms, DMs, presence, moderation, persistence.
- Storage planning for 48h of 720p30/1080p30 content.
- Browser-compatible playback (`hls.js` + native HLS).
- Operational simplicity and scalability.

Full decision rationale, alternatives, and consequences in [`adr/ADR-001.md`](adr/ADR-001.md).

### Quality Goals

| Priority | Quality Attribute         | Goal                                                     | Scenario                                                                         |
| -------- | ------------------------- | -------------------------------------------------------- | -------------------------------------------------------------------------------- |
| 1        | **Cost Efficiency**       | <$25/month for moderate scale (100s concurrent)          | R2 $0 egress + immutable `.ts` edge caching eliminates per-viewer bandwidth cost |
| 2        | **Performance / Latency** | Low US latency; acceptable HLS delay                     | Hetzner Ashburn origin + Cloudflare US POPs (15–35s end-to-end)                  |
| 3        | **Availability**          | 24/7 with minimal downtime                               | FFmpeg/MediaMTX auto-restart loop, fallback slate, monitoring                    |
| 4        | **Scalability**           | Grow from 10s to 1000s viewers without proportional cost | Edge caching + R2; only request volume limits                                    |
| 5        | **Maintainability**       | Lightweight, clear configs, low ops overhead             | Static SPA, Go/Node chat, SQLite/PostgreSQL+Redis                                |
| 6        | **Security / Usability**  | Secure transport, moderation tools, good UX              | HTTPS/WSS, Cloudflare WAF, rate limits, intuitive chat/player                    |

### Stakeholders

| Role/Name           | Contact | Expectations                                                                     |
| ------------------- | ------- | -------------------------------------------------------------------------------- |
| Product Owner       | -       | Cost-effective independent 24/7 channel with synchronized viewing and rich chat  |
| Engineering Team    | -       | Simple, evolvable stack (FFmpeg, HLS, Cloudflare R2, lightweight WebSocket chat) |
| US Viewers          | -       | Reliable low-latency playback, responsive synchronized chat experience           |
| Moderators / Admins | -       | Easy playlist management, moderation dashboard, monitoring/alerting              |
| Operations          | -       | Low TCO, clear cost model, automated failover, backups                           |

(See `adr/ADR-001.md` for detailed forces, decision drivers, and rationale.)

## Architecture Constraints

- Independence from video hosting platforms (no YouTube, Twitch, paid Cloudflare Stream).
- US-centric (origin + audience).
- Cost must remain low even at scale (bandwidth dominant).
- Standard browser playback only (no plugins).
- Cloudflare ToS compliance for video delivery (use R2 as origin for large immutable files).
- HLS as primary protocol (standard, not LL-HLS initially).
- Self-hosted playlist normalization and continuous playout required.

(See `adr/ADR-001.md` for full forces and trade-offs.)

## Context and Scope

### Business Context

Linear Void provides a continuously running scheduled video program (playlist-driven) to a primarily US audience. The custom viewer site replaces traditional platforms, giving full control over content, schedule, UI, and real-time chat (public rooms + private DMs).

### High-Level Context Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "External Users"
        Viewers[US Viewers<br/>Browser SPA + hls.js + Chat UI]
        Admins[Admins / Moderators<br/>Playlist & Schedule UI]
    end

    subgraph "Cloudflare Global Edge"
        CDN[Cloudflare CDN<br/>Cache Rules + WAF + TLS]
        Pages[Cloudflare Pages<br/>Static SPA Hosting]
        R2[Cloudflare R2<br/>HLS Segments + Source MP4s<br/>enam location hint]
    end

    subgraph "US-East Origin (Hetzner Ashburn VPS)"
        VPS[VPS CX22<br/>Nginx Proxy + Monitoring]
        Playout[Playout Service<br/>FFmpeg + MediaMTX/SRS<br/>Continuous HLS from Playlist]
        Chat[Chat Backend<br/>WebSockets + Redis + DB<br/>Public rooms + Private DMs]
    end

    Admins -->|HTTPS - Schedule/Playlist| Playout
    Playout -->|S3-compatible Upload| R2
    R2 -->|Cached HLS| CDN
    Viewers -->|HTTPS| Pages
    Viewers -->|HLS Requests| CDN
    Viewers -->|WSS Chat| Chat
    Chat --> VPS
    VPS -->|Proxy / Static| Pages
    CDN -->|Origin Pull on Miss| R2
```

**External Interfaces**:

- **Cloudflare**: CDN (cache rules for `.ts` vs `.m3u8`), R2 (object storage, $0 egress), Pages (static hosting), WAF/TLS.
- **Hetzner VPS**: Compute for playout and chat (low-cost US-East with good peering).
- **Browsers**: Native HLS (`<video>` + hls.js) and WebSocket APIs.
- **Source**: Pre-normalized MP4 files uploaded to R2.

This diagram synthesizes the context views from all original ADRs. Full decision details, cache rules, storage calculations, and rationale are in [`adr/ADR-001.md`](adr/ADR-001.md).

### Technical Context

- **Playout**: FFmpeg consuming normalized playlist → continuous HLS.
- **Packaging/Delivery**: MediaMTX/SRS/Nginx → immutable `.ts` segments to R2; Cloudflare edge cache.
- **Frontend**: Static SPA with hls.js.
- **Chat**: Lightweight WebSocket service (Node/Go) + Redis (presence/rate limits) + PostgreSQL/SQLite (history/moderation).
- **Normalization**: All source videos must be H.264/AAC, 30 fps constant, 2s keyframes.

**Mapping**:

- Input: Normalized MP4s in R2 + playlist.
- Output: Live `master.m3u8` + segments served via Cloudflare; chat via WSS.

## Solution Strategy

**Core Strategy** (synthesized & accepted from all ADRs):

1. **US-East Playout on Hetzner Ashburn VPS** (`CX22` or equivalent): FFmpeg + MediaMTX/SRS for continuous playlist-to-HLS conversion (`-c copy` where possible).
2. **Immutable HLS to Cloudflare R2**: Segments use unique names; R2 acts as origin with $0 egress.
3. **Aggressive Edge Caching**: Long TTL for `.ts` (immutable), short TTL (1–10s) for live `.m3u8`. Tiered Cache enabled. This makes viewer delivery essentially free.
4. **Lightweight Custom Frontend**: Static HTML/JS SPA + hls.js player + embedded chat sidebar.
5. **Separate but co-located Chat Service**: WebSockets with persistence and moderation (low bandwidth).
6. **Normalization & Monitoring**: Pre-process all videos; use dashboards + alerts for playout health, cache hit ratio, R2 ops.

This eliminates egress economics, ensures synchronization via live playlist, keeps ops simple, and scales well. See `adr/ADR-001.md` for detailed cache rules, storage tables, cost model, configs (FFmpeg/Nginx examples), and phase-by-phase implementation plan.

**Key Trade-off Accepted**: Standard HLS latency (15–35s) is acceptable for linear programming/chat. LL-HLS/WebRTC deferred.

## Building Block View

### Whitebox Overall System

Overview (Mermaid diagram):

```mermaid
graph TD
    subgraph Cloudflare
        CDN[CDN + Cache Rules<br/>.ts immutable long TTL<br/>.m3u8 short TTL]
        R2[R2 - HLS + Source]
        Pages[Pages - Static SPA]
    end
    VPS[VPS Hetzner Ashburn]
    Playout[FFmpeg + MediaMTX/SRS<br/>Playlist → HLS]
    Chat[WebSocket Chat<br/>Node/Go + Redis + DB]
    Viewer[US Viewers]
    Admin[Admin UI]

    Admin --> Playout
    Playout --> R2
    R2 --> CDN
    Viewer --> Pages
    Viewer --> CDN
    Viewer --> Chat
    Chat --> VPS
    VPS --> Nginx[Nginx Proxy]
```

**Motivation**: Decouple playout (continuous linear), delivery (cached immutable segments), and interaction (chat). Cloudflare R2 + edge caching solves the bandwidth cost problem while maintaining synchronization and independence.

**Contained Building Blocks** (high-level):

- **Playout**: FFmpeg + MediaMTX/SRS (continuous HLS from playlist).
- **Storage/Delivery**: Cloudflare R2 + CDN (immutable segments).
- **Frontend**: Static SPA (hls.js player + chat UI).
- **Chat Service**: WebSocket backend with persistence/moderation.
- **Proxy/Static**: Nginx on VPS.

**Important Interfaces**: HTTP(S) for HLS, WSS for chat, S3-compatible for R2 uploads.

Detailed black-box descriptions, level-2/3 views, and code locations are in `adr/ADR-001.md` and future implementation ADRs.

### Level 2

Deferred to implementation (see ADR for component responsibilities, interfaces, quality attributes, and risks). Playout enforces normalization; Delivery layer focuses on cache rules and immutable naming.

### Level 3

Deferred to implementation (see ADR for component responsibilities, interfaces, quality attributes, and risks). Playout enforces normalization; Delivery layer focuses on cache rules and immutable naming.

## Runtime View

### Playout & Delivery Scenario

1. Admin adds normalized MP4 to R2 + updates playlist.
2. Playout worker (FFmpeg loop with concat demuxer) reads next file, packages as HLS segments (4-6s).
3. Segments uploaded to R2 with unique immutable names; live `.m3u8` updated.
4. Viewer loads SPA → requests master/live manifest → hls.js player follows current edge.
5. Cloudflare serves `.ts` from nearest edge cache (origin/R2 hit only once per POP/region).
6. Parallel WebSocket connection to chat service for synchronized interaction.

Notable aspects: Manifest freshness ensures all viewers stay on same timeline; cache hit ratio directly impacts cost and latency. Fallback slate activates on playout failure. Full cache policy, example configs (FFmpeg bash loop, Nginx RTMP HLS), and storage tables in `adr/ADR-001.md`.

(Additional scenarios: moderation events, scale spikes, failover covered in the ADR.)

## Deployment View

### CI/CD Pipeline Architecture

The deployment follows a two-stage pipeline: **CI builds and pushes images**, then **CD pulls and deploys** them to the production host.

```mermaid
flowchart LR
    subgraph "Gitea Actions"
        CI[CI Workflow<br/>Build + Push Images]
        CD[CD Workflow<br/>Pull + Deploy]
    end
    subgraph "Gitea Container Registry"
        REGISTRY[git.allucanget.biz<br/>playout:latest<br/>chat:latest<br/>nginx:latest]
    end
    subgraph "Production Host (192.168.88.40)"
        DOCKER[Docker Engine<br/>via TCP :2375]
        PLAYOUT[linearvoid-playout]
        CHAT[linearvoid-chat]
        NGINX[linearvoid-nginx]
        REDIS[linearvoid-redis]
        DB[linearvoid-db]
        PROM[linearvoid-prometheus]
        GRAF[linearvoid-grafana]
    end

    CI -->|push| REGISTRY
    REGISTRY -->|pull| CD
    CD -->|deploy via DOCKER_HOST| DOCKER
    DOCKER --> PLAYOUT
    DOCKER --> CHAT
    DOCKER --> NGINX
    DOCKER --> REDIS
    DOCKER --> DB
    DOCKER --> PROM
    DOCKER --> GRAF
```

**Pipeline Sequence:**

1. **CI Workflow** (`.gitea/workflows/ci.yml`):
   - Triggers on `push` to `main` or `pull_request`
   - Runs `test` job: frontend lint/build, Go validation, `docker compose build`
   - Runs `docker` job (after `test` passes, only on `main` push):
     - Builds `playout`, `chat`, `nginx` images with Buildx
     - Pushes to Gitea Container Registry with `main` and `latest` tags
2. **CD Workflow** (`.gitea/workflows/cd.yml`):
   - Triggers on `workflow_run` when CI completes successfully, or manually via `workflow_dispatch`
   - Pulls latest images from registry
   - Generates `.env` from secrets
   - Seeds Prometheus config into named volume
   - Deploys via `docker compose -f docker-compose.prod.yml up -d` to remote host (`192.168.88.40:2375`)
   - Runs health checks with retry logic for playout initialization

### Infrastructure Level 1

**Overview Diagram**: See Mermaid in Building Block section (VPS + Cloudflare R2/Pages/CDN).

**Motivation**: Low-cost US-East origin with global edge caching. Hetzner provides excellent price/performance and included traffic; Cloudflare eliminates egress fees.

**Quality/Performance Features**: High cache-hit ratio for video, low US latency, automatic scaling at edge, built-in WAF/DDoS protection.

**Mapping of Building Blocks to Infrastructure**:

- Playout + Chat + Nginx → Hetzner Cloud CX22 (Ashburn, VA).
- HLS storage, caching, static SPA → Cloudflare R2 + CDN + Pages (custom domains, `enam` location hint).
- All public traffic routed through Cloudflare.

### Infrastructure Level 2

**Hetzner VPS**: Runs playout (FFmpeg/MediaMTX), chat backend, Nginx. Monitoring via dashboards + UptimeRobot. Backups to R2.

**Cloudflare Layer**:

- R2 bucket for media.
- Cache Rules (`.ts`: Cache Everything, immutable, long TTL; `.m3u8`: short TTL/must-revalidate).
- WAF, rate limiting, Tiered Cache.
- Pages for SPA hosting.

Future evolution: Docker Compose, managed DB, secondary regions (detailed in subsequent ADRs).

## Cross-cutting Concepts

### Caching & Cost Optimization

Immutable segment naming + targeted Cloudflare Cache Rules + R2 ($0 egress) = near-zero viewer bandwidth cost. Long TTL on `.ts`, short on live manifests. Monitor hit ratio and R2 Class B ops.

### Video Normalization & Playout

All source assets must be pre-normalized before they are accepted into production playlists. This prevents decode resets, timestamp drift, and audio/video discontinuities during continuous concat playout.

#### Required Source Profile (Linear Void v1)

| Field                   | Required Value                     | Notes                                                   |
| ----------------------- | ---------------------------------- | ------------------------------------------------------- |
| Container               | MP4                                | Single-video-asset input format                         |
| Video codec             | H.264/AVC (`libx264`)              | Baseline delivery codec for broad compatibility         |
| Audio codec             | AAC-LC                             | Stereo preferred                                        |
| Resolution              | 1280x720 or 1920x1080              | Keep single resolution per playlist block when possible |
| Framerate               | Constant 30 fps (`-r 30`)          | Variable frame rate content must be normalized          |
| GOP / keyframe interval | 2 seconds (`-g 60 -keyint_min 60`) | Matches HLS segment boundaries better                   |
| Pixel format            | `yuv420p`                          | Browser compatibility baseline                          |
| Audio sample rate       | 48 kHz                             | Stable timing for concat playout                        |
| Audio bitrate           | 128k-192k                          | Predictable quality envelope                            |

#### Reference Normalization Command

```bash
ffmpeg -i input.mov \
  -c:v libx264 -preset medium -crf 21 \
  -r 30 -g 60 -keyint_min 60 -sc_threshold 0 \
  -pix_fmt yuv420p \
  -c:a aac -b:a 160k -ar 48000 -ac 2 \
  -movflags +faststart \
  output-normalized.mp4
```

#### Verification Gate (ffprobe)

- Validate codec, fps, resolution, and keyframe cadence before upload acceptance.
- Reject non-conforming files from admin ingest with actionable error feedback.
- Keep verification logic aligned with runtime playout assumptions to preserve `-c copy` efficiency where possible.

### Monitoring & Resilience

Linear Void uses a two-layer monitoring model:

1. **Application layer (chat + platform health)** via Prometheus metrics and a lightweight Grafana dashboard.
2. **Infrastructure/runtime layer (Prometheus self-observability, TSDB, query engine, Go runtime)** via a detailed Grafana dashboard.

#### Prometheus Baseline

- Config file: `monitoring/prometheus/prometheus.yml`
- Default scrape interval: `15s`
- Current scrape jobs: `chat` -> `chat:3000`, `prometheus` -> `prometheus:9090`.

Recommended next-step scrape targets (if/when exporters are introduced):

- `nginx` (via nginx-exporter)
- `redis` (via redis-exporter)
- `postgres` (via postgres-exporter)
- `node` (via node-exporter)

#### Grafana Dashboards

- `monitoring/grafana/dashboard.json` (Linear Void app metrics: connected clients, messages, moderation actions, Redis reachability, admin sessions, scrape health).
- `monitoring/grafana/dashboards/dashboard.json` (Prometheus operational visibility: status, uptime, CPU/memory, TSDB ingestion/churn/errors, HTTP latencies, query load, GC/scheduler).

Use the app dashboard for service-level behavior and the Prometheus overview dashboard for platform diagnostics and capacity planning.

#### Alerting Rules (Prometheus Rule Groups)

Define alert rules as grouped policies (for example in `monitoring/prometheus/alerts.yml`) and include them from `prometheus.yml`.

Minimum recommended alert set:

- **Availability**
  `up{job="chat"} == 0` for `2m`; `up{job="prometheus"} == 0` for `2m`.
- **Chat service health**
  `linearvoid_chat_redis_up == 0` for `2m`; sudden disconnect spike / connected clients collapse.
- **Playout continuity**
  No HLS segment progression for `30-60s` (from playout-specific metric when available).
- **Prometheus capacity and ingestion health**
  Sustained growth in `prometheus_tsdb_wal_writes_failed_total` or `prometheus_tsdb_compactions_failed_total`, abnormal jump in `prometheus_tsdb_out_of_order_samples_total`, and high `prometheus_engine_queries` near concurrent limit.
- **Latency and user-impact**
  Elevated p99 handler latency from `prometheus_http_request_duration_seconds_bucket` and elevated p99 query execution latency from `prometheus_engine_query_duration_histogram_seconds_bucket`.

Route warning/critical severities to the team channel and keep runbook links attached to each alert definition.

Operational resilience baseline:

- Emergency slate fallback for playout failure.
- Auto-restart policies for playout/chat/nginx containers.
- Weekly backup and restore drills for stateful data (`data/chat`, playlist metadata, Postgres).
- Incident runbook with triage order: edge reachability -> origin health -> playout progression -> chat connectivity -> data-plane dependencies.

### API Documentation Reference

Linear Void currently exposes three operational API surfaces. The canonical detailed contract should live in `API.md`, while this section provides the architecture-level map.

#### Chat WebSocket API

- Endpoint: `/ws`
- Transport: `wss://` in production, `ws://` locally
- Query parameters:
  - `username`
  - `room`
- Message envelope (typical):
  - `room`
  - `username`
  - `text`
  - `type` (`public`, `private`, `system`)
- Responsibilities:
  - room fan-out
  - private DM channel routing
  - moderation command handling
  - rate limiting and presence integration

#### Playout and Delivery REST API

- Health endpoints:
  - playout health (`/health` on playout service)
  - edge/origin health (`/health` via nginx)
- Stream delivery:
  - HLS manifest and segment path namespace under `/hls/`
- Responsibilities:
  - expose liveness for orchestration and monitoring
  - provide stable HLS path contract for viewer and CDN

#### Admin REST API

- Auth/session:
  - `/api/admin/login`
  - `/api/admin/logout`
  - `/api/admin/session`
- Operational/admin domains:
  - overview, analytics, users, moderation, schedule, playlist, media
- Media upload domain:
  - `/api/admin/media/uploads` (init/resume)
  - `/api/admin/media/uploads/{uploadId}/chunk`
  - `/api/admin/media/uploads/{uploadId}/complete`
- Responsibilities:
  - protected control plane for operator workflows
  - media metadata + upload orchestration
  - moderation/audit and system status management

### Cloudflare Configuration Reference

Cloudflare is part of the production control plane, not only a CDN edge. The following configuration set is required to maintain cost, latency, and reliability targets.

#### DNS and Zone Routing

- `linearvoid.tv` -> application origin (nginx)
- `www.linearvoid.tv` -> redirect to apex
- `media.linearvoid.tv` -> media domain for HLS/object delivery when split-domain mode is enabled

#### Cache Rules Baseline

- HLS segments (`.ts`, `.m4s`):
  - Cache Everything
  - long TTL (immutable)
  - respect immutable semantics
- Manifests (`.m3u8`):
  - short TTL (1-10s)
  - revalidation-friendly policy for timeline freshness

#### Edge Security and Controls

- Enforce HTTPS/WSS and redirect HTTP to HTTPS.
- Enable WAF baseline protections for common web exploits.
- Apply rate limits for abusive request patterns (especially auth/admin paths).
- Keep origin exposure minimized (prefer Cloudflare-only public ingress where feasible).

#### Worker and Header Policy

- Use Worker logic for immutable cache headers and request normalization where cache rules alone are insufficient.
- Keep CORS behavior explicit for HLS resources to support browser playback.

#### Operations and Validation

- Validate cache behavior after deployment:
  - expected cache status progression (MISS -> HIT)
  - manifest freshness under active playout
- Track edge metrics and adjust:
  - cache hit ratio
  - request volume and error rates
  - ToS/cost posture for sustained traffic profiles

### Security & Moderation

Security is split between edge controls (Cloudflare), reverse proxy controls (nginx), and application controls (chat/admin backend).

#### Transport and Edge Protection

- Public traffic uses HTTPS/WSS at the edge.
- Cloudflare WAF and rate limiting are part of the baseline (see `infrastructure/cloudflare/cache-rules.md`).
- Cloudflare Worker can enforce immutable cache headers for segment paths (`infrastructure/cloudflare/worker.js`).

#### Reverse Proxy Enforcement (nginx)

- Admin UI and admin API paths are protected by `auth_request` before reaching backend handlers (`/admin/`, `/api/admin/`).
- Unauthorized admin UI requests are redirected to `/admin/login/`.
- Unauthorized admin API requests return JSON `401` from a dedicated location handler.
- WebSocket upgrade headers are explicitly forwarded for `/ws`.

#### Admin Authentication and Session Security

- Admin authentication is cookie-session based (`linearvoid_admin_session`).
- Session cookie controls in backend include `HttpOnly`, `SameSite=Strict`, and `Secure` when request is HTTPS (derived from TLS or `X-Forwarded-Proto`).
- Session expiration is enforced server-side.

#### Rate Limiting and Abuse Controls

- Chat message rate limiting uses Redis `INCR + EXPIRE` windowing.
- Current baseline in chat handler: `>10` messages within `10s` triggers a user-facing throttle response.
- If Redis is unavailable, presence/rate-limit features degrade gracefully (warning logged; core chat remains available).

#### CORS and Cross-Origin Behavior

- HLS endpoints exposed by nginx include explicit `Access-Control-Allow-Origin: *` for player compatibility.
- Admin/authenticated routes are expected to run same-origin behind nginx auth gates.

#### Moderation and Privacy

- Moderation commands (`/mod`, `/mute`, `/ban`) are handled in backend chat flow and persisted through chat storage paths.
- Private DMs are application-level private channels, not end-to-end encrypted in current architecture.
- Moderation and audit retention policies should be formalized in runbook/ops docs as part of production hardening.

### Chat

WebSockets + Redis (presence, pub/sub, rate limits) + DB (history, audit). Public rooms + on-demand private chats. Scales independently of video.

Full details, cache rule table, bitrate/storage calculations, cost model, example configs, ToS notes, and implementation phases are in [`adr/ADR-001.md`](adr/ADR-001.md).

## Architecture Decisions

The primary decisions are captured in the following consolidated ADRs (all located in `adr/`):

- **[ADR-001](./adr/ADR-001.md)**: Overall architecture, HLS + Cloudflare R2 caching strategy, storage calculations, cost model, risks.
- **[ADR-002](./adr/ADR-002-module-structure.md)**: Chosen project folder/module structure (frontend, backend/playout, backend/chat, infrastructure, scripts, monitoring) and rationale.
- **[ADR-003](./adr/ADR-003-tech-stack.md)**: Technology stack, programming language choices per module (TypeScript frontend, Go for chat/playout wrapper, Bash/Python scripts), Docker-first approach, and explicit documentation of all implicit assumptions.
- **[ADR-004](./adr/ADR-004-admin-video.md)**: Video admin web interface (upload, playlist management, schedule overrides, normalization enforcement).
- **[ADR-005](./adr/ADR-005-admin-moderation-analytics.md)**: Web admin interfaces for user management, moderation dashboard, analytics, and system configuration.
- **[ADR-006](./adr/ADR-006-data-storage.md)**: Data storage strategy (SQLite vs PostgreSQL for v1, migration path, admin interface capabilities).
- **[ADR-007](./adr/ADR-007-redis-usage.md)**: Redis usage (exact use cases, TTL policies, persistence, fallback behavior, admin configuration).

These ADRs together form the canonical decision log.

**Summary of Key Decisions** (from ADR-001 through ADR-007):

- Self-managed US-East playout (Hetzner Ashburn) with FFmpeg + MediaMTX/SRS → HLS → Cloudflare R2 + edge caching.
- Module structure that directly reflects the arc42 building blocks and context diagram.
- Go preferred for performance-critical services; TypeScript for frontend; Docker for all deployment.
- Admin interfaces for video management, moderation, analytics, and system configuration using HTMX + Tailwind.
- SQLite as primary database for v1 with PostgreSQL migration path if needed.
- Redis for rate limiting, presence, pub/sub, session storage with defined TTL policies and fallback behavior.

## Quality Requirements

See **Quality Goals** table in the Introduction section. Primary scenarios focus on cost (near-zero egress), availability (auto-restart + fallback), scalability (edge caching), and usability (synchronized player + responsive chat).

### Performance Benchmarks and Capacity Planning

#### Baseline SLO Targets (v1)

- Playback startup: target < 4s after first manifest request on warm cache.
- Steady-state playback: no segment starvation during normal operation.
- End-to-end HLS latency: expected 15-35s (standard HLS trade-off).
- Chat delivery latency: p95 < 500ms inside same region under normal load.
- Admin page responsiveness: p95 API response < 1s for common read operations.

#### Workload Assumptions

- Primary audience in US; origin in US-East.
- Edge cache serves most segment traffic; origin handles manifest churn and cache misses.
- Chat traffic scales with concurrent viewers but remains low-bandwidth compared to video.

#### Initial Capacity Model

- Video bitrate envelopes (from ADR-001):
  - 720p30: ~2.6-4.2 Mbps total with audio.
  - 1080p30: ~5.1-8.2 Mbps total with audio.
- Storage planning for 48h rolling library:
  - 720p30: ~57-90 GB raw (allocate ~80-110 GB with headroom).
  - 1080p30: ~110-175 GB raw (allocate ~130-180 GB with headroom).
- Cost behavior:
  - Viewer scale is mostly constrained by cache efficiency and request volume, not by direct origin egress.

#### Benchmark Plan

- Playback path:
  - Measure time-to-first-frame and rebuffer events from multiple US regions.
  - Track manifest freshness and segment availability at edge.
- Chat path:
  - Load-test concurrent websocket connections and message fan-out.
  - Measure p50/p95 send-to-receive latency and Redis fallback behavior.
- Admin/API path:
  - Benchmark playlist/system endpoints for p95 latency and error rate.
- Platform path:
  - Observe TSDB ingestion, query p99, and GC pause behavior in Grafana.

#### Capacity Triggers and Actions

- Trigger: sustained viewer growth with rising cache-miss ratio.
  - Action: tighten cache policy, enable/verify Tiered Cache, review manifest TTL.
- Trigger: chat p95 latency drift or websocket error growth.
  - Action: scale chat service, tune Redis, separate noisy moderation workloads.
- Trigger: Prometheus WAL/compaction failures or query saturation.
  - Action: tune retention/cardinality, increase resources, split dashboards/queries.
- Trigger: origin CPU/memory pressure on playout host.
  - Action: optimize FFmpeg profile, isolate services, add secondary origin.

### Scaling Strategy

Scaling is designed to preserve cost efficiency first, then increase throughput and resilience in controlled phases.

#### Phase 1: Vertical + Cache-First (Current Baseline)

- Single primary origin in US-East with strong Cloudflare edge caching.
- Optimize segment immutability and manifest TTL before adding infrastructure.
- Scale compute vertically when CPU/memory pressure appears on playout/chat host.

#### Phase 2: Service Separation

- Split playout, chat, and observability workloads across dedicated nodes/VMs.
- Isolate noisy workloads (analytics, heavy admin tasks) from latency-sensitive chat paths.
- Introduce managed PostgreSQL/Redis or dedicated state nodes where operationally justified.

#### Phase 3: Multi-Origin Resilience

- Add warm standby or active-secondary origin.
- Use health-aware DNS/origin failover strategy.
- Keep HLS path and cache contract stable so viewer impact during failover is minimized.

#### Phase 4: Regional and Throughput Expansion

- Add additional origin region only when audience distribution and latency justify it.
- Keep chat affinity strategy explicit (regional routing, state replication considerations).
- Continuously evaluate cache miss economics vs additional origin complexity.

#### Scaling Triggers

- Rising cache miss ratio with sustained traffic growth.
- Chat p95/p99 latency drift or websocket error growth.
- Persistent origin resource saturation despite optimization.
- Increased blast-radius concerns from single-host dependency.

### Disaster Recovery and Backup Procedures

#### Recovery Targets

- RPO target: <= 24h for chat/moderation/admin metadata.
- RTO target: <= 60 minutes for restoring baseline streaming + chat operations.

#### Backup Scope

- Persistent service volumes:
  - Postgres data (`postgres-data` volume).
  - Redis append-only persistence (`redis-data` volume) where enabled.
  - Prometheus long-term TSDB (`prometheus-data` volume) if historical metrics retention is required.
  - Grafana state (`grafana-data` volume) if dashboards/users are managed in-instance.
- Host-mounted state directories:
  - `data/chat`
  - `data/playlist`
  - `data/media` (authoritative source may be R2/object storage)
  - `data/hls` (optional; can be regenerated by playout)
- Configuration and infra source of truth:
  - Compose files, nginx config, monitoring configs, and environment template.

#### Backup Cadence

- Daily logical Postgres backups (plus weekly full snapshot).
- Daily sync of chat/playlist state directories.
- Weekly integrity check by restoring backups into an isolated environment.
- Retention policy baseline:
  - Daily backups: 14 days
  - Weekly backups: 8 weeks
  - Monthly backups: 6 months

#### Recovery Runbook (Minimum)

1. Restore infrastructure configuration (compose, nginx, env, monitoring config).
2. Restore Postgres and Redis persisted state.
3. Restore `data/playlist` and chat persistence directories.
4. Rehydrate media from authoritative object storage if local media is missing.
5. Start stack and validate in order:
   - `playout` health endpoint
   - `chat` health endpoint
   - nginx `/health`
   - HLS playlist availability
   - websocket chat connectivity
6. Verify admin authentication and moderation audit visibility.
7. Announce recovery and continue elevated monitoring for at least one playlist cycle.

#### Failure Modes and Strategy

- Single-host loss: redeploy stack on standby host, restore data backups, repoint DNS/edge origin.
- Storage corruption: restore last known-good snapshot and replay latest logical backups.
- Partial service failure: replace only impacted service and restore scoped data (for example Postgres only).

Document and rehearse this runbook quarterly; recovery that is not exercised is not reliable.

## Risks and Technical Debts

- Cloudflare ToS risk for heavy video on free tier (use R2 path; fallback to BunnyCDN if challenged).
- HLS latency (15–35s) may affect perceived chat synchronization.
- Single VPS as SPOF (mitigate with DNS failover in Phase 3).
- Upfront effort for video normalization and cache tuning.
- Moderation/abuse prevention operational load at scale.
- R2 request costs on cache misses (optimize with Tiered Cache; measure in prod).
- Deferred items: LL-HLS, ABR, E2EE, paid access controls, full Docker Compose.

**Technical Debt**: Detailed service configs, CI/CD, and advanced resilience are left for implementation ADRs.

## Glossary

| Term              | Definition                                                                                 |
| ----------------- | ------------------------------------------------------------------------------------------ |
| HLS               | HTTP Live Streaming – protocol using `.m3u8` manifests and media segments (`.ts` or fMP4). |
| Immutable Segment | HLS chunk with unique, unchanging name enabling long-lived caching.                        |
| R2                | Cloudflare object storage ($0 egress, ideal origin for cached HLS).                        |
| Edge Cache / POP  | Cloudflare points-of-presence that cache and serve content close to viewers.               |
| Playout           | Continuous generation of a scheduled linear program from a playlist.                       |
| Normalization     | Pre-processing videos to consistent codec/fps/keyframe settings for seamless playback.     |
| Cache Rule        | Cloudflare configuration defining caching behavior, TTL, and headers per URL pattern.      |
| SPOF              | Single Point of Failure (current single VPS origin).                                       |
| Tiered Cache      | Cloudflare feature reducing origin requests by checking upper-tier caches first.           |

**References**: Cloudflare R2/Cache docs, MediaMTX/SRS/FFmpeg HLS guides, Hetzner pricing, hls.js documentation, standard video bitrate calculators. All original root `ADR-001-*.md` files have been consolidated and can be archived/deleted.