902 lines
47 KiB
Markdown
902 lines
47 KiB
Markdown
# Architecture Documentation for Linear Void
|
||
|
||
> The architecture documentation for the Linear Void project is structured according to the [arc42 template](https://arc42.org/). This template provides a comprehensive framework for documenting software architecture, ensuring that all relevant aspects of the system are covered.
|
||
|
||
## Table of Contents
|
||
|
||
- [Architecture Documentation for Linear Void](#architecture-documentation-for-linear-void)
|
||
- [Table of Contents](#table-of-contents)
|
||
- [Companion Documentation](#companion-documentation)
|
||
- [Introduction and Goals](#introduction-and-goals)
|
||
- [Requirements Overview](#requirements-overview)
|
||
- [Quality Goals](#quality-goals)
|
||
- [Stakeholders](#stakeholders)
|
||
- [Architecture Constraints](#architecture-constraints)
|
||
- [Context and Scope](#context-and-scope)
|
||
- [Business Context](#business-context)
|
||
- [High-Level Context Diagram (Mermaid)](#high-level-context-diagram-mermaid)
|
||
- [Technical Context](#technical-context)
|
||
- [Solution Strategy](#solution-strategy)
|
||
- [Building Block View](#building-block-view)
|
||
- [Whitebox Overall System](#whitebox-overall-system)
|
||
- [Level 2](#level-2)
|
||
- [Playout Service (`backend/playout/src/playlist.go`)](#playout-service-backendplayoutsrcplaylistgo)
|
||
- [Chat Service (`backend/chat/src/main.go`)](#chat-service-backendchatsrcmaingo)
|
||
- [Frontend SPA (`frontend/`)](#frontend-spa-frontend)
|
||
- [Reverse Proxy (`infrastructure/nginx/nginx.conf`)](#reverse-proxy-infrastructurenginxnginxconf)
|
||
- [Level 3](#level-3)
|
||
- [Chat Service Modules](#chat-service-modules)
|
||
- [Playout Service Modules](#playout-service-modules)
|
||
- [Frontend Modules](#frontend-modules)
|
||
- [Runtime View](#runtime-view)
|
||
- [Playout \& Delivery Scenario](#playout--delivery-scenario)
|
||
- [Deployment View](#deployment-view)
|
||
- [CI/CD Pipeline Architecture](#cicd-pipeline-architecture)
|
||
- [Infrastructure Level 1](#infrastructure-level-1)
|
||
- [Infrastructure Level 2](#infrastructure-level-2)
|
||
- [Cross-cutting Concepts](#cross-cutting-concepts)
|
||
- [Caching \& Cost Optimization](#caching--cost-optimization)
|
||
- [Video Normalization \& Playout](#video-normalization--playout)
|
||
- [Required Source Profile (Linear Void v1)](#required-source-profile-linear-void-v1)
|
||
- [Reference Normalization Command](#reference-normalization-command)
|
||
- [Verification Gate (ffprobe)](#verification-gate-ffprobe)
|
||
- [Monitoring \& Resilience](#monitoring--resilience)
|
||
- [Prometheus Baseline](#prometheus-baseline)
|
||
- [Grafana Dashboards](#grafana-dashboards)
|
||
- [Alerting Rules (Prometheus Rule Groups)](#alerting-rules-prometheus-rule-groups)
|
||
- [API Documentation Reference](#api-documentation-reference)
|
||
- [Chat WebSocket API](#chat-websocket-api)
|
||
- [Playout and Delivery REST API](#playout-and-delivery-rest-api)
|
||
- [Admin REST API](#admin-rest-api)
|
||
- [Cloudflare Configuration Reference](#cloudflare-configuration-reference)
|
||
- [DNS and Zone Routing](#dns-and-zone-routing)
|
||
- [Cache Rules Baseline](#cache-rules-baseline)
|
||
- [Edge Security and Controls](#edge-security-and-controls)
|
||
- [Worker and Header Policy](#worker-and-header-policy)
|
||
- [Operations and Validation](#operations-and-validation)
|
||
- [Security \& Moderation](#security--moderation)
|
||
- [Transport and Edge Protection](#transport-and-edge-protection)
|
||
- [Reverse Proxy Enforcement (nginx)](#reverse-proxy-enforcement-nginx)
|
||
- [Admin Authentication and Session Security](#admin-authentication-and-session-security)
|
||
- [Rate Limiting and Abuse Controls](#rate-limiting-and-abuse-controls)
|
||
- [CORS and Cross-Origin Behavior](#cors-and-cross-origin-behavior)
|
||
- [Moderation and Privacy](#moderation-and-privacy)
|
||
- [Chat](#chat)
|
||
- [Architecture Decisions](#architecture-decisions)
|
||
- [Quality Requirements](#quality-requirements)
|
||
- [Performance Benchmarks and Capacity Planning](#performance-benchmarks-and-capacity-planning)
|
||
- [Baseline SLO Targets (v1)](#baseline-slo-targets-v1)
|
||
- [Workload Assumptions](#workload-assumptions)
|
||
- [Initial Capacity Model](#initial-capacity-model)
|
||
- [Benchmark Plan](#benchmark-plan)
|
||
- [Capacity Triggers and Actions](#capacity-triggers-and-actions)
|
||
- [Scaling Strategy](#scaling-strategy)
|
||
- [Phase 1: Vertical + Cache-First (Current Baseline)](#phase-1-vertical--cache-first-current-baseline)
|
||
- [Phase 2: Service Separation](#phase-2-service-separation)
|
||
- [Phase 3: Multi-Origin Resilience](#phase-3-multi-origin-resilience)
|
||
- [Phase 4: Regional and Throughput Expansion](#phase-4-regional-and-throughput-expansion)
|
||
- [Scaling Triggers](#scaling-triggers)
|
||
- [Disaster Recovery and Backup Procedures](#disaster-recovery-and-backup-procedures)
|
||
- [Recovery Targets](#recovery-targets)
|
||
- [Backup Scope](#backup-scope)
|
||
- [Backup Cadence](#backup-cadence)
|
||
- [Recovery Runbook (Minimum)](#recovery-runbook-minimum)
|
||
- [Failure Modes and Strategy](#failure-modes-and-strategy)
|
||
- [Risks and Technical Debts](#risks-and-technical-debts)
|
||
- [Glossary](#glossary)
|
||
|
||
## Companion Documentation
|
||
|
||
- [API.md](../../API.md) - Implemented WebSocket and REST API reference.
|
||
- [DEPLOYMENT.md](../../DEPLOYMENT.md) - Production deployment and operational runbook.
|
||
- [MONITORING.md](../../MONITORING.md) - Metrics, dashboards, and monitoring troubleshooting.
|
||
- [SECURITY.md](../../SECURITY.md) - Security model, controls, and hardening baseline.
|
||
|
||
## Introduction and Goals
|
||
|
||
### Requirements Overview
|
||
|
||
Linear Void is an independent **24/7 linear video channel** broadcasting a playlist of self-owned pre-recorded videos. Viewers access it via a custom lightweight HTML/CSS/JavaScript SPA with synchronized HLS playback, schedule metadata, public chat rooms, and private 1:1 messaging. Complete independence from third-party video platforms (YouTube, Twitch, etc.).
|
||
|
||
**Core Requirements** (merged from all ADRs):
|
||
|
||
- Continuous synchronized playback from playlist (live-style HLS, not VOD).
|
||
- Self-hosted source media, frontend, schedule, and chat data.
|
||
- Primary US audience/origin (East Coast preferred for low latency/peering).
|
||
- Extreme cost efficiency (bandwidth dominant; target near-zero egress via Cloudflare R2 + edge caching).
|
||
- Custom chat with rooms, DMs, presence, moderation, persistence.
|
||
- Storage planning for 48h of 720p30/1080p30 content.
|
||
- Browser-compatible playback (`hls.js` + native HLS).
|
||
- Operational simplicity and scalability.
|
||
|
||
Full decision rationale, alternatives, and consequences in [`adr/ADR-001.md`](adr/ADR-001.md).
|
||
|
||
### Quality Goals
|
||
|
||
| Priority | Quality Attribute | Goal | Scenario |
|
||
| -------- | ------------------------- | -------------------------------------------------------- | -------------------------------------------------------------------------------- |
|
||
| 1 | **Cost Efficiency** | <$25/month for moderate scale (100s concurrent) | R2 $0 egress + immutable `.ts` edge caching eliminates per-viewer bandwidth cost |
|
||
| 2 | **Performance / Latency** | Low US latency; acceptable HLS delay | Hetzner Ashburn origin + Cloudflare US POPs (15–35s end-to-end) |
|
||
| 3 | **Availability** | 24/7 with minimal downtime | FFmpeg/MediaMTX auto-restart loop, fallback slate, monitoring |
|
||
| 4 | **Scalability** | Grow from 10s to 1000s viewers without proportional cost | Edge caching + R2; only request volume limits |
|
||
| 5 | **Maintainability** | Lightweight, clear configs, low ops overhead | Static SPA, Go/Node chat, SQLite/PostgreSQL+Redis |
|
||
| 6 | **Security / Usability** | Secure transport, moderation tools, good UX | HTTPS/WSS, Cloudflare WAF, rate limits, intuitive chat/player |
|
||
|
||
### Stakeholders
|
||
|
||
| Role/Name | Contact | Expectations |
|
||
| ------------------- | ------- | -------------------------------------------------------------------------------- |
|
||
| Product Owner | - | Cost-effective independent 24/7 channel with synchronized viewing and rich chat |
|
||
| Engineering Team | - | Simple, evolvable stack (FFmpeg, HLS, Cloudflare R2, lightweight WebSocket chat) |
|
||
| US Viewers | - | Reliable low-latency playback, responsive synchronized chat experience |
|
||
| Moderators / Admins | - | Easy playlist management, moderation dashboard, monitoring/alerting |
|
||
| Operations | - | Low TCO, clear cost model, automated failover, backups |
|
||
|
||
(See `adr/ADR-001.md` for detailed forces, decision drivers, and rationale.)
|
||
|
||
## Architecture Constraints
|
||
|
||
- Independence from video hosting platforms (no YouTube, Twitch, paid Cloudflare Stream).
|
||
- US-centric (origin + audience).
|
||
- Cost must remain low even at scale (bandwidth dominant).
|
||
- Standard browser playback only (no plugins).
|
||
- Cloudflare ToS compliance for video delivery (use R2 as origin for large immutable files).
|
||
- HLS as primary protocol (standard, not LL-HLS initially).
|
||
- Self-hosted playlist normalization and continuous playout required.
|
||
|
||
(See `adr/ADR-001.md` for full forces and trade-offs.)
|
||
|
||
## Context and Scope
|
||
|
||
### Business Context
|
||
|
||
Linear Void provides a continuously running scheduled video program (playlist-driven) to a primarily US audience. The custom viewer site replaces traditional platforms, giving full control over content, schedule, UI, and real-time chat (public rooms + private DMs).
|
||
|
||
### High-Level Context Diagram (Mermaid)
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
subgraph "External Users"
|
||
Viewers[US Viewers<br/>Browser SPA + hls.js + Chat UI]
|
||
Admins[Admins / Moderators<br/>Playlist & Schedule UI]
|
||
end
|
||
|
||
subgraph "Cloudflare Global Edge"
|
||
CDN[Cloudflare CDN<br/>Cache Rules + WAF + TLS]
|
||
Pages[Cloudflare Pages<br/>Static SPA Hosting]
|
||
R2[Cloudflare R2<br/>HLS Segments + Source MP4s<br/>enam location hint]
|
||
end
|
||
|
||
subgraph "US-East Origin (Hetzner Ashburn VPS)"
|
||
VPS[VPS CX22<br/>Nginx Proxy + Monitoring]
|
||
Playout[Playout Service<br/>FFmpeg + MediaMTX/SRS<br/>Continuous HLS from Playlist]
|
||
Chat[Chat Backend<br/>WebSockets + Redis + DB<br/>Public rooms + Private DMs]
|
||
end
|
||
|
||
Admins -->|HTTPS - Schedule/Playlist| Playout
|
||
Playout -->|S3-compatible Upload| R2
|
||
R2 -->|Cached HLS| CDN
|
||
Viewers -->|HTTPS| Pages
|
||
Viewers -->|HLS Requests| CDN
|
||
Viewers -->|WSS Chat| Chat
|
||
Chat --> VPS
|
||
VPS -->|Proxy / Static| Pages
|
||
CDN -->|Origin Pull on Miss| R2
|
||
```
|
||
|
||
**External Interfaces**:
|
||
|
||
- **Cloudflare**: CDN (cache rules for `.ts` vs `.m3u8`), R2 (object storage, $0 egress), Pages (static hosting), WAF/TLS.
|
||
- **Hetzner VPS**: Compute for playout and chat (low-cost US-East with good peering).
|
||
- **Browsers**: Native HLS (`<video>` + hls.js) and WebSocket APIs.
|
||
- **Source**: Pre-normalized MP4 files uploaded to R2.
|
||
|
||
This diagram synthesizes the context views from all original ADRs. Full decision details, cache rules, storage calculations, and rationale are in [`adr/ADR-001.md`](adr/ADR-001.md).
|
||
|
||
### Technical Context
|
||
|
||
- **Playout**: FFmpeg consuming normalized playlist → continuous HLS.
|
||
- **Packaging/Delivery**: MediaMTX/SRS/Nginx → immutable `.ts` segments to R2; Cloudflare edge cache.
|
||
- **Frontend**: Static SPA with hls.js.
|
||
- **Chat**: Lightweight WebSocket service (Node/Go) + Redis (presence/rate limits) + PostgreSQL/SQLite (history/moderation).
|
||
- **Normalization**: All source videos must be H.264/AAC, 30 fps constant, 2s keyframes.
|
||
|
||
**Mapping**:
|
||
|
||
- Input: Normalized MP4s in R2 + playlist.
|
||
- Output: Live `master.m3u8` + segments served via Cloudflare; chat via WSS.
|
||
|
||
## Solution Strategy
|
||
|
||
**Core Strategy** (synthesized & accepted from all ADRs):
|
||
|
||
1. **US-East Playout on Hetzner Ashburn VPS** (`CX22` or equivalent): FFmpeg + MediaMTX/SRS for continuous playlist-to-HLS conversion (`-c copy` where possible).
|
||
2. **Immutable HLS to Cloudflare R2**: Segments use unique names; R2 acts as origin with $0 egress.
|
||
3. **Aggressive Edge Caching**: Long TTL for `.ts` (immutable), short TTL (1–10s) for live `.m3u8`. Tiered Cache enabled. This makes viewer delivery essentially free.
|
||
4. **Lightweight Custom Frontend**: Static HTML/JS SPA + hls.js player + embedded chat sidebar.
|
||
5. **Separate but co-located Chat Service**: WebSockets with persistence and moderation (low bandwidth).
|
||
6. **Normalization & Monitoring**: Pre-process all videos; use dashboards + alerts for playout health, cache hit ratio, R2 ops.
|
||
|
||
This eliminates egress economics, ensures synchronization via live playlist, keeps ops simple, and scales well. See `adr/ADR-001.md` for detailed cache rules, storage tables, cost model, configs (FFmpeg/Nginx examples), and phase-by-phase implementation plan.
|
||
|
||
**Key Trade-off Accepted**: Standard HLS latency (15–35s) is acceptable for linear programming/chat. LL-HLS/WebRTC deferred.
|
||
|
||
## Building Block View
|
||
|
||
### Whitebox Overall System
|
||
|
||
Overview (Mermaid diagram):
|
||
|
||
```mermaid
|
||
graph TD
|
||
subgraph Cloudflare
|
||
CDN[CDN + Cache Rules<br/>.ts immutable long TTL<br/>.m3u8 short TTL]
|
||
R2[R2 - HLS + Source]
|
||
Pages[Pages - Static SPA]
|
||
end
|
||
VPS[VPS Hetzner Ashburn]
|
||
Playout[FFmpeg + MediaMTX/SRS<br/>Playlist → HLS]
|
||
Chat[WebSocket Chat<br/>Node/Go + Redis + DB]
|
||
Viewer[US Viewers]
|
||
Admin[Admin UI]
|
||
|
||
Admin --> Playout
|
||
Playout --> R2
|
||
R2 --> CDN
|
||
Viewer --> Pages
|
||
Viewer --> CDN
|
||
Viewer --> Chat
|
||
Chat --> VPS
|
||
VPS --> Nginx[Nginx Proxy]
|
||
```
|
||
|
||
**Motivation**: Decouple playout (continuous linear), delivery (cached immutable segments), and interaction (chat). Cloudflare R2 + edge caching solves the bandwidth cost problem while maintaining synchronization and independence.
|
||
|
||
**Contained Building Blocks** (high-level):
|
||
|
||
- **Playout**: FFmpeg + MediaMTX/SRS (continuous HLS from playlist).
|
||
- **Storage/Delivery**: Cloudflare R2 + CDN (immutable segments).
|
||
- **Frontend**: Static SPA (hls.js player + chat UI).
|
||
- **Chat Service**: WebSocket backend with persistence/moderation.
|
||
- **Proxy/Static**: Nginx on VPS.
|
||
|
||
**Important Interfaces**: HTTP(S) for HLS, WSS for chat, S3-compatible for R2 uploads.
|
||
|
||
Detailed black-box descriptions, level-2/3 views, and code locations are in `adr/ADR-001.md` and future implementation ADRs.
|
||
|
||
### Level 2
|
||
|
||
Detailed component diagram with code locations and persistence.
|
||
|
||
```mermaid
|
||
graph TB
|
||
subgraph "Cloudflare Edge"
|
||
CF[CDN + WAF<br/>Cache Rules]
|
||
R2[R2 Bucket<br/>media + HLS]
|
||
PAGES[Pages<br/>SPA hosting]
|
||
end
|
||
|
||
subgraph "Docker Host (Hetzner CX22)"
|
||
NGINX[nginx<br/>:80/:443<br/>proxy + auth]
|
||
CHAT[chat service<br/>:3000<br/>Go + gorilla/websocket]
|
||
PLAYOUT[playout service<br/>:8081<br/>Go + FFmpeg]
|
||
REDIS[Redis 7<br/>:6379<br/>presence/rate/pubsub]
|
||
DB[PostgreSQL 15<br/>:5432<br/>chat history/moderation]
|
||
end
|
||
|
||
subgraph "External"
|
||
VIEWER[Viewer Browser<br/>hls.js + WebSocket]
|
||
ADMIN[Admin Browser<br/>SPA + uploads]
|
||
end
|
||
|
||
VIEWER -->|HTTPS| CF
|
||
VIEWER -->|WSS| NGINX
|
||
ADMIN -->|HTTPS| CF
|
||
CF --> NGINX
|
||
NGINX --> CHAT
|
||
NGINX --> PLAYOUT
|
||
NGINX -->|/hls/*| R2
|
||
CHAT --> REDIS
|
||
CHAT --> DB
|
||
PLAYOUT --> R2
|
||
PLAYOUT -->|reads| R2
|
||
```
|
||
|
||
#### Playout Service (`backend/playout/src/playlist.go`)
|
||
|
||
- **Role**: Continuous HLS encoding loop using FFmpeg concat demuxer
|
||
- **Input**: `data/playlist/playlist.txt` — ordered list of MP4 file paths
|
||
- **Output**: HLS segments (`.ts`) + live manifest (`.m3u8`) written to `data/hls/`, mirrored to R2
|
||
- **Resilience**: Restart counter (>5/60s → activate slate), Prometheus metrics on port 8081
|
||
- **Validation**: ffprobe gate on all ingested files (codec, fps, resolution, keyframe interval)
|
||
|
||
#### Chat Service (`backend/chat/src/main.go`)
|
||
|
||
- **Role**: WebSocket chat with rooms, DMs, moderation, admin API
|
||
- **Persistence**: PostgreSQL 15 via GORM (messages, moderation logs, media metadata). SQLite for dev fallback
|
||
- **Redis**: Presence tracking, rate limiting, pub/sub for multi-instance fan-out
|
||
- **Auth**: Session-cookie admin auth with Redis-backed expiry
|
||
- **Metrics**: Connected clients, stored messages, DMs, moderation actions on `/metrics`
|
||
|
||
#### Frontend SPA (`frontend/`)
|
||
|
||
- **Viewer**: `src/main.ts` — hls.js player, WebSocket chat, presence indicators, schedule overlay
|
||
- **Admin**: `src/admin.ts` — 9 dynamic tabs (dashboard, upload, playlist, schedule, moderation, chat-log, users, analytics, system)
|
||
- **Build**: Vite → static output in `frontend/dist/`
|
||
|
||
#### Reverse Proxy (`infrastructure/nginx/nginx.conf`)
|
||
|
||
- auth_request gate on `/admin` and `/api/admin`
|
||
- WebSocket upgrade for `/ws`
|
||
- HLS CORS headers + OPTIONS preflight
|
||
- `/health` JSON endpoint, `/nginx-status` stub_status
|
||
|
||
### Level 3
|
||
|
||
Module-level breakdown per service.
|
||
|
||
#### Chat Service Modules
|
||
|
||
| Module | File/Lines | Responsibility |
|
||
| ---------------------- | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
|
||
| **Connection Manager** | `main.go` (hub, clients map) | Track active WebSocket connections per room, fan-out messages |
|
||
| **Message Parser** | `main.go` (readPump, handleMessage) | Parse JSON envelope `{type, room?, ...}`, validate fields, route to handler |
|
||
| **Room Registry** | `main.go` (rooms map, room broadcasts) | Map room name → client set, handle join/leave/close |
|
||
| **DM Router** | `main.go` (handleDirectMessage) | Resolve target user in clients map, deliver private message |
|
||
| **Persistence Layer** | `main.go` (GORM models: Message, DirectMessage, ModerationLog) | Store/retrieve chat messages, DMs, moderation events |
|
||
| **Redis Subsystem** | `main.go` (redisClient, rate limiting, pub/sub, presence) | INCR+EXPIRE rate limit, SETEX presence, PSUBSCRIBE room fan-out |
|
||
| **Auth Handler** | `main.go` (handleAdminLogin, adminAuthMiddleware) | Session cookie creation/validation, login rate limiting |
|
||
| **Admin API Router** | `main.go` (handleAdmin\*) | Upload, playlist CRUD, schedule overrides, moderation actions, analytics, system summary |
|
||
| **Metrics Exporter** | `main.go` (prometheus.MustRegister) | Expose connected_clients, messages_total, dms_total, moderation_actions, redis_up |
|
||
|
||
#### Playout Service Modules
|
||
|
||
| Module | File/Lines | Responsibility |
|
||
| -------------------- | ---------------------------------------------- | ------------------------------------------------------------------ |
|
||
| **Playlist Parser** | `playlist.go` (parsePlaylist) | Read `playlist.txt`, skip comments/blank lines, resolve file paths |
|
||
| **FFmpeg Manager** | `playlist.go` (runPlayout, restart logic) | Spawn/monitor FFmpeg concat process, restart on crash |
|
||
| **Slate Controller** | `playlist.go` (activateSlate, deactivateSlate) | Switch output to slate loop on excessive failures |
|
||
| **R2 Sync** | `playlist.go` (syncToR2) | Upload HLS segments + manifest to R2 bucket |
|
||
| **Health Endpoint** | `playlist.go` (healthHandler) | `/health` → JSON with mode, segment count, uptime |
|
||
| **Metrics Exporter** | `playlist.go` (prometheus) | Expose restarts_total, mode (live/slate) |
|
||
|
||
#### Frontend Modules
|
||
|
||
| Module | File | Responsibility |
|
||
| ----------------------- | ----------------------------------------------- | ----------------------------------------------------------------- |
|
||
| **HLS Player** | `src/main.ts` | hls.js initialization, attach to video element, error recovery |
|
||
| **WebSocket Client** | `src/main.ts` | Connect/disconnect, parse envelope, dispatch to UI handlers |
|
||
| **Chat UI** | `src/main.ts` | Message list rendering, room tabs, DM panel, presence dots |
|
||
| **Schedule Overlay** | `src/main.ts` | Poll `/api/schedule`, render current/next program |
|
||
| **Admin Router** | `src/admin.ts` | Tab-based navigation, dynamic page loading |
|
||
| **Upload Manager** | `src/admin.ts` | Multipart upload with chunk progress, ffprobe validation feedback |
|
||
| **Playlist Editor** | `src/admin.ts` | Drag-reorder, add/remove items, save to `/api/admin/playlist` |
|
||
| **Moderation Panel** | `src/admin.ts` + `public/admin/moderation.html` | Inline mute/ban, action log viewer |
|
||
| **Analytics Dashboard** | `src/admin.ts` + `public/admin/analytics.html` | Message counts, top rooms, moderation stats |
|
||
| **System Config** | `src/admin.ts` + `public/admin/system.html` | Service health, config metadata, restart buttons |
|
||
|
||
## Runtime View
|
||
|
||
### Playout & Delivery Scenario
|
||
|
||
1. Admin adds normalized MP4 to R2 + updates playlist.
|
||
2. Playout worker (FFmpeg loop with concat demuxer) reads next file, packages as HLS segments (4-6s).
|
||
3. Segments uploaded to R2 with unique immutable names; live `.m3u8` updated.
|
||
4. Viewer loads SPA → requests master/live manifest → hls.js player follows current edge.
|
||
5. Cloudflare serves `.ts` from nearest edge cache (origin/R2 hit only once per POP/region).
|
||
6. Parallel WebSocket connection to chat service for synchronized interaction.
|
||
|
||
Notable aspects: Manifest freshness ensures all viewers stay on same timeline; cache hit ratio directly impacts cost and latency. Fallback slate activates on playout failure. Full cache policy, example configs (FFmpeg bash loop, Nginx RTMP HLS), and storage tables in `adr/ADR-001.md`.
|
||
|
||
(Additional scenarios: moderation events, scale spikes, failover covered in the ADR.)
|
||
|
||
## Deployment View
|
||
|
||
### CI/CD Pipeline Architecture
|
||
|
||
The deployment follows a two-stage pipeline: **CI builds and pushes images**, then **CD pulls and deploys** them. Currently targeting the **test lab** (192.168.88.40). Production Hetzner VPS is planned but not yet provisioned — the same pipeline will deploy there by changing `DOCKER_HOST`.
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
subgraph "Gitea Actions"
|
||
CI[CI Workflow<br/>Build + Push Images]
|
||
CD[CD Workflow<br/>Pull + Deploy]
|
||
end
|
||
subgraph "Gitea Container Registry"
|
||
REGISTRY[git.allucanget.biz<br/>playout:latest<br/>chat:latest<br/>nginx:latest]
|
||
end
|
||
subgraph "Test Lab Host (192.168.88.40) — Production Hetzner TBD"
|
||
DOCKER[Docker Engine<br/>via TCP :2375]
|
||
PLAYOUT[linearvoid-playout]
|
||
CHAT[linearvoid-chat]
|
||
NGINX[linearvoid-nginx]
|
||
REDIS[linearvoid-redis]
|
||
DB[linearvoid-db]
|
||
PROM[linearvoid-prometheus]
|
||
GRAF[linearvoid-grafana]
|
||
end
|
||
|
||
CI -->|push| REGISTRY
|
||
REGISTRY -->|pull| CD
|
||
CD -->|deploy via DOCKER_HOST| DOCKER
|
||
DOCKER --> PLAYOUT
|
||
DOCKER --> CHAT
|
||
DOCKER --> NGINX
|
||
DOCKER --> REDIS
|
||
DOCKER --> DB
|
||
DOCKER --> PROM
|
||
DOCKER --> GRAF
|
||
```
|
||
|
||
**Pipeline Sequence:**
|
||
|
||
1. **CI Workflow** (`.gitea/workflows/ci.yml`):
|
||
- Triggers on `push` to `main` or `pull_request`
|
||
- Runs `test` job: frontend lint/build, Go validation, `docker compose build`
|
||
- Runs `docker` job (after `test` passes, only on `main` push):
|
||
- Builds `playout`, `chat`, `nginx` images with Buildx
|
||
- Pushes to Gitea Container Registry with `main` and `latest` tags
|
||
2. **CD Workflow** (`.gitea/workflows/cd.yml`):
|
||
- Triggers on `workflow_run` when CI completes successfully, or manually via `workflow_dispatch`
|
||
- Pulls latest images from registry
|
||
- Generates `.env` from secrets
|
||
- Seeds Prometheus config into named volume
|
||
- Deploys via `docker compose -f docker-compose.prod.yml up -d` to remote host (`192.168.88.40:2375` — test lab; switch to Hetzner IP for production)
|
||
- Runs health checks with retry logic for playout initialization
|
||
|
||
### Infrastructure Level 1
|
||
|
||
**Overview Diagram**: See Mermaid in Building Block section (VPS + Cloudflare R2/Pages/CDN).
|
||
|
||
**Motivation**: Low-cost US-East origin with global edge caching. Hetzner provides excellent price/performance and included traffic; Cloudflare eliminates egress fees.
|
||
|
||
**Quality/Performance Features**: High cache-hit ratio for video, low US latency, automatic scaling at edge, built-in WAF/DDoS protection.
|
||
|
||
**Mapping of Building Blocks to Infrastructure**:
|
||
|
||
- Playout + Chat + Nginx → Hetzner Cloud CX22 (Ashburn, VA).
|
||
- HLS storage, caching, static SPA → Cloudflare R2 + CDN + Pages (custom domains, `enam` location hint).
|
||
- All public traffic routed through Cloudflare.
|
||
|
||
### Infrastructure Level 2
|
||
|
||
**Hetzner VPS (planned):** Will run playout (FFmpeg/MediaMTX), chat backend, Nginx. Monitoring via dashboards + UptimeRobot. Backups to R2. Currently test lab (`192.168.88.40`) serves this role. Production Hetzner instance not yet provisioned — see Production Deployment section in [`DEPLOYMENT.md`](../../DEPLOYMENT.md).
|
||
|
||
**Cloudflare Layer**:
|
||
|
||
- R2 bucket for media.
|
||
- Cache Rules (`.ts`: Cache Everything, immutable, long TTL; `.m3u8`: short TTL/must-revalidate).
|
||
- WAF, rate limiting, Tiered Cache.
|
||
- Pages for SPA hosting.
|
||
|
||
Future evolution: Docker Compose, managed DB, secondary regions (detailed in subsequent ADRs).
|
||
|
||
## Cross-cutting Concepts
|
||
|
||
### Caching & Cost Optimization
|
||
|
||
Immutable segment naming + targeted Cloudflare Cache Rules + R2 ($0 egress) = near-zero viewer bandwidth cost. Long TTL on `.ts`, short on live manifests. Monitor hit ratio and R2 Class B ops.
|
||
|
||
### Video Normalization & Playout
|
||
|
||
All source assets must be pre-normalized before they are accepted into production playlists. This prevents decode resets, timestamp drift, and audio/video discontinuities during continuous concat playout.
|
||
|
||
#### Required Source Profile (Linear Void v1)
|
||
|
||
| Field | Required Value | Notes |
|
||
| ----------------------- | ---------------------------------- | ------------------------------------------------------- |
|
||
| Container | MP4 | Single-video-asset input format |
|
||
| Video codec | H.264/AVC (`libx264`) | Baseline delivery codec for broad compatibility |
|
||
| Audio codec | AAC-LC | Stereo preferred |
|
||
| Resolution | 1280x720 or 1920x1080 | Keep single resolution per playlist block when possible |
|
||
| Framerate | Constant 30 fps (`-r 30`) | Variable frame rate content must be normalized |
|
||
| GOP / keyframe interval | 2 seconds (`-g 60 -keyint_min 60`) | Matches HLS segment boundaries better |
|
||
| Pixel format | `yuv420p` | Browser compatibility baseline |
|
||
| Audio sample rate | 48 kHz | Stable timing for concat playout |
|
||
| Audio bitrate | 128k-192k | Predictable quality envelope |
|
||
|
||
#### Reference Normalization Command
|
||
|
||
```bash
|
||
ffmpeg -i input.mov \
|
||
-c:v libx264 -preset medium -crf 21 \
|
||
-r 30 -g 60 -keyint_min 60 -sc_threshold 0 \
|
||
-pix_fmt yuv420p \
|
||
-c:a aac -b:a 160k -ar 48000 -ac 2 \
|
||
-movflags +faststart \
|
||
output-normalized.mp4
|
||
```
|
||
|
||
#### Verification Gate (ffprobe)
|
||
|
||
- Validate codec, fps, resolution, and keyframe cadence before upload acceptance.
|
||
- Reject non-conforming files from admin ingest with actionable error feedback.
|
||
- Keep verification logic aligned with runtime playout assumptions to preserve `-c copy` efficiency where possible.
|
||
|
||
### Monitoring & Resilience
|
||
|
||
Linear Void uses a two-layer monitoring model:
|
||
|
||
1. **Application layer (chat + platform health)** via Prometheus metrics and a lightweight Grafana dashboard.
|
||
2. **Infrastructure/runtime layer (Prometheus self-observability, TSDB, query engine, Go runtime)** via a detailed Grafana dashboard.
|
||
|
||
#### Prometheus Baseline
|
||
|
||
- Config file: `monitoring/prometheus/prometheus.yml`
|
||
- Default scrape interval: `15s`
|
||
- Current scrape jobs: `chat` -> `chat:3000`, `prometheus` -> `prometheus:9090`.
|
||
|
||
Recommended next-step scrape targets (if/when exporters are introduced):
|
||
|
||
- `nginx` (via nginx-exporter)
|
||
- `redis` (via redis-exporter)
|
||
- `postgres` (via postgres-exporter)
|
||
- `node` (via node-exporter)
|
||
|
||
#### Grafana Dashboards
|
||
|
||
- `monitoring/grafana/dashboard.json` (Linear Void app metrics: connected clients, messages, moderation actions, Redis reachability, admin sessions, scrape health).
|
||
- `monitoring/grafana/dashboards/dashboard.json` (Prometheus operational visibility: status, uptime, CPU/memory, TSDB ingestion/churn/errors, HTTP latencies, query load, GC/scheduler).
|
||
|
||
Use the app dashboard for service-level behavior and the Prometheus overview dashboard for platform diagnostics and capacity planning.
|
||
|
||
#### Alerting Rules (Prometheus Rule Groups)
|
||
|
||
Define alert rules as grouped policies (for example in `monitoring/prometheus/alerts.yml`) and include them from `prometheus.yml`.
|
||
|
||
Minimum recommended alert set:
|
||
|
||
- **Availability**
|
||
`up{job="chat"} == 0` for `2m`; `up{job="prometheus"} == 0` for `2m`.
|
||
- **Chat service health**
|
||
`linearvoid_chat_redis_up == 0` for `2m`; sudden disconnect spike / connected clients collapse.
|
||
- **Playout continuity**
|
||
No HLS segment progression for `30-60s` (from playout-specific metric when available).
|
||
- **Prometheus capacity and ingestion health**
|
||
Sustained growth in `prometheus_tsdb_wal_writes_failed_total` or `prometheus_tsdb_compactions_failed_total`, abnormal jump in `prometheus_tsdb_out_of_order_samples_total`, and high `prometheus_engine_queries` near concurrent limit.
|
||
- **Latency and user-impact**
|
||
Elevated p99 handler latency from `prometheus_http_request_duration_seconds_bucket` and elevated p99 query execution latency from `prometheus_engine_query_duration_histogram_seconds_bucket`.
|
||
|
||
Route warning/critical severities to the team channel and keep runbook links attached to each alert definition.
|
||
|
||
Operational resilience baseline:
|
||
|
||
- Emergency slate fallback for playout failure.
|
||
- Auto-restart policies for playout/chat/nginx containers.
|
||
- Weekly backup and restore drills for stateful data (`data/chat`, playlist metadata, Postgres).
|
||
- Incident runbook with triage order: edge reachability -> origin health -> playout progression -> chat connectivity -> data-plane dependencies.
|
||
|
||
### API Documentation Reference
|
||
|
||
Linear Void currently exposes three operational API surfaces. The canonical detailed contract should live in `API.md`, while this section provides the architecture-level map.
|
||
|
||
#### Chat WebSocket API
|
||
|
||
- Endpoint: `/ws`
|
||
- Transport: `wss://` in production, `ws://` locally
|
||
- Query parameters:
|
||
- `username`
|
||
- `room`
|
||
- Message envelope (typical):
|
||
- `room`
|
||
- `username`
|
||
- `text`
|
||
- `type` (`public`, `private`, `system`)
|
||
- Responsibilities:
|
||
- room fan-out
|
||
- private DM channel routing
|
||
- moderation command handling
|
||
- rate limiting and presence integration
|
||
|
||
#### Playout and Delivery REST API
|
||
|
||
- Health endpoints:
|
||
- playout health (`/health` on playout service)
|
||
- edge/origin health (`/health` via nginx)
|
||
- Stream delivery:
|
||
- HLS manifest and segment path namespace under `/hls/`
|
||
- Responsibilities:
|
||
- expose liveness for orchestration and monitoring
|
||
- provide stable HLS path contract for viewer and CDN
|
||
|
||
#### Admin REST API
|
||
|
||
- Auth/session:
|
||
- `/api/admin/login`
|
||
- `/api/admin/logout`
|
||
- `/api/admin/session`
|
||
- Operational/admin domains:
|
||
- overview, analytics, users, moderation, schedule, playlist, media
|
||
- Media upload domain:
|
||
- `/api/admin/media/uploads` (init/resume)
|
||
- `/api/admin/media/uploads/{uploadId}/chunk`
|
||
- `/api/admin/media/uploads/{uploadId}/complete`
|
||
- Responsibilities:
|
||
- protected control plane for operator workflows
|
||
- media metadata + upload orchestration
|
||
- moderation/audit and system status management
|
||
|
||
### Cloudflare Configuration Reference
|
||
|
||
Cloudflare is part of the production control plane, not only a CDN edge. The following configuration set is required to maintain cost, latency, and reliability targets.
|
||
|
||
#### DNS and Zone Routing
|
||
|
||
- `linearvoid.tv` -> application origin (nginx)
|
||
- `www.linearvoid.tv` -> redirect to apex
|
||
- `media.linearvoid.tv` -> media domain for HLS/object delivery when split-domain mode is enabled
|
||
|
||
#### Cache Rules Baseline
|
||
|
||
- HLS segments (`.ts`, `.m4s`):
|
||
- Cache Everything
|
||
- long TTL (immutable)
|
||
- respect immutable semantics
|
||
- Manifests (`.m3u8`):
|
||
- short TTL (1-10s)
|
||
- revalidation-friendly policy for timeline freshness
|
||
|
||
#### Edge Security and Controls
|
||
|
||
- Enforce HTTPS/WSS and redirect HTTP to HTTPS.
|
||
- Enable WAF baseline protections for common web exploits.
|
||
- Apply rate limits for abusive request patterns (especially auth/admin paths).
|
||
- Keep origin exposure minimized (prefer Cloudflare-only public ingress where feasible).
|
||
|
||
#### Worker and Header Policy
|
||
|
||
- Use Worker logic for immutable cache headers and request normalization where cache rules alone are insufficient.
|
||
- Keep CORS behavior explicit for HLS resources to support browser playback.
|
||
|
||
#### Operations and Validation
|
||
|
||
- Validate cache behavior after deployment:
|
||
- expected cache status progression (MISS -> HIT)
|
||
- manifest freshness under active playout
|
||
- Track edge metrics and adjust:
|
||
- cache hit ratio
|
||
- request volume and error rates
|
||
- ToS/cost posture for sustained traffic profiles
|
||
|
||
### Security & Moderation
|
||
|
||
Security is split between edge controls (Cloudflare), reverse proxy controls (nginx), and application controls (chat/admin backend).
|
||
|
||
#### Transport and Edge Protection
|
||
|
||
- Public traffic uses HTTPS/WSS at the edge.
|
||
- Cloudflare WAF and rate limiting are part of the baseline (see `infrastructure/cloudflare/cache-rules.md`).
|
||
- Cloudflare Worker can enforce immutable cache headers for segment paths (`infrastructure/cloudflare/worker.js`).
|
||
|
||
#### Reverse Proxy Enforcement (nginx)
|
||
|
||
- Admin UI and admin API paths are protected by `auth_request` before reaching backend handlers (`/admin/`, `/api/admin/`).
|
||
- Unauthorized admin UI requests are redirected to `/admin/login/`.
|
||
- Unauthorized admin API requests return JSON `401` from a dedicated location handler.
|
||
- WebSocket upgrade headers are explicitly forwarded for `/ws`.
|
||
|
||
#### Admin Authentication and Session Security
|
||
|
||
- Admin authentication is cookie-session based (`linearvoid_admin_session`).
|
||
- Session cookie controls in backend include `HttpOnly`, `SameSite=Strict`, and `Secure` when request is HTTPS (derived from TLS or `X-Forwarded-Proto`).
|
||
- Session expiration is enforced server-side.
|
||
|
||
#### Rate Limiting and Abuse Controls
|
||
|
||
- Chat message rate limiting uses Redis `INCR + EXPIRE` windowing.
|
||
- Current baseline in chat handler: `>10` messages within `10s` triggers a user-facing throttle response.
|
||
- If Redis is unavailable, presence/rate-limit features degrade gracefully (warning logged; core chat remains available).
|
||
|
||
#### CORS and Cross-Origin Behavior
|
||
|
||
- HLS endpoints exposed by nginx include explicit `Access-Control-Allow-Origin: *` for player compatibility.
|
||
- Admin/authenticated routes are expected to run same-origin behind nginx auth gates.
|
||
|
||
#### Moderation and Privacy
|
||
|
||
- Moderation commands (`/mod`, `/mute`, `/ban`) are handled in backend chat flow and persisted through chat storage paths.
|
||
- Private DMs are application-level private channels, not end-to-end encrypted in current architecture.
|
||
- Moderation and audit retention policies should be formalized in runbook/ops docs as part of production hardening.
|
||
|
||
### Chat
|
||
|
||
WebSockets + Redis (presence, pub/sub, rate limits) + DB (history, audit). Public rooms + on-demand private chats. Scales independently of video.
|
||
|
||
Full details, cache rule table, bitrate/storage calculations, cost model, example configs, ToS notes, and implementation phases are in [`adr/ADR-001.md`](adr/ADR-001.md).
|
||
|
||
## Architecture Decisions
|
||
|
||
The primary decisions are captured in the following consolidated ADRs (all located in `adr/`):
|
||
|
||
- **[ADR-001](./adr/ADR-001.md)**: Overall architecture, HLS + Cloudflare R2 caching strategy, storage calculations, cost model, risks.
|
||
- **[ADR-002](./adr/ADR-002-module-structure.md)**: Chosen project folder/module structure (frontend, backend/playout, backend/chat, infrastructure, scripts, monitoring) and rationale.
|
||
- **[ADR-003](./adr/ADR-003-tech-stack.md)**: Technology stack, programming language choices per module (TypeScript frontend, Go for chat/playout wrapper, Bash/Python scripts), Docker-first approach, and explicit documentation of all implicit assumptions.
|
||
- **[ADR-004](./adr/ADR-004-admin-video.md)**: Video admin web interface (upload, playlist management, schedule overrides, normalization enforcement).
|
||
- **[ADR-005](./adr/ADR-005-admin-moderation-analytics.md)**: Web admin interfaces for user management, moderation dashboard, analytics, and system configuration.
|
||
- **[ADR-006](./adr/ADR-006-data-storage.md)**: Data storage strategy (SQLite vs PostgreSQL for v1, migration path, admin interface capabilities).
|
||
- **[ADR-007](./adr/ADR-007-redis-usage.md)**: Redis usage (exact use cases, TTL policies, persistence, fallback behavior, admin configuration).
|
||
|
||
These ADRs together form the canonical decision log.
|
||
|
||
**Summary of Key Decisions** (from ADR-001 through ADR-007):
|
||
|
||
- Self-managed US-East playout (Hetzner Ashburn) with FFmpeg + MediaMTX/SRS → HLS → Cloudflare R2 + edge caching.
|
||
- Module structure that directly reflects the arc42 building blocks and context diagram.
|
||
- Go preferred for performance-critical services; TypeScript for frontend; Docker for all deployment.
|
||
- Admin interfaces for video management, moderation, analytics, and system configuration using HTMX + Tailwind.
|
||
- SQLite as primary database for v1 with PostgreSQL migration path if needed.
|
||
- Redis for rate limiting, presence, pub/sub, session storage with defined TTL policies and fallback behavior.
|
||
|
||
## Quality Requirements
|
||
|
||
See **Quality Goals** table in the Introduction section. Primary scenarios focus on cost (near-zero egress), availability (auto-restart + fallback), scalability (edge caching), and usability (synchronized player + responsive chat).
|
||
|
||
### Performance Benchmarks and Capacity Planning
|
||
|
||
#### Baseline SLO Targets (v1)
|
||
|
||
- Playback startup: target < 4s after first manifest request on warm cache.
|
||
- Steady-state playback: no segment starvation during normal operation.
|
||
- End-to-end HLS latency: expected 15-35s (standard HLS trade-off).
|
||
- Chat delivery latency: p95 < 500ms inside same region under normal load.
|
||
- Admin page responsiveness: p95 API response < 1s for common read operations.
|
||
|
||
#### Workload Assumptions
|
||
|
||
- Primary audience in US; origin in US-East.
|
||
- Edge cache serves most segment traffic; origin handles manifest churn and cache misses.
|
||
- Chat traffic scales with concurrent viewers but remains low-bandwidth compared to video.
|
||
|
||
#### Initial Capacity Model
|
||
|
||
- Video bitrate envelopes (from ADR-001):
|
||
- 720p30: ~2.6-4.2 Mbps total with audio.
|
||
- 1080p30: ~5.1-8.2 Mbps total with audio.
|
||
- Storage planning for 48h rolling library:
|
||
- 720p30: ~57-90 GB raw (allocate ~80-110 GB with headroom).
|
||
- 1080p30: ~110-175 GB raw (allocate ~130-180 GB with headroom).
|
||
- Cost behavior:
|
||
- Viewer scale is mostly constrained by cache efficiency and request volume, not by direct origin egress.
|
||
|
||
#### Benchmark Plan
|
||
|
||
- Playback path:
|
||
- Measure time-to-first-frame and rebuffer events from multiple US regions.
|
||
- Track manifest freshness and segment availability at edge.
|
||
- Chat path:
|
||
- Load-test concurrent websocket connections and message fan-out.
|
||
- Measure p50/p95 send-to-receive latency and Redis fallback behavior.
|
||
- Admin/API path:
|
||
- Benchmark playlist/system endpoints for p95 latency and error rate.
|
||
- Platform path:
|
||
- Observe TSDB ingestion, query p99, and GC pause behavior in Grafana.
|
||
|
||
#### Capacity Triggers and Actions
|
||
|
||
- Trigger: sustained viewer growth with rising cache-miss ratio.
|
||
- Action: tighten cache policy, enable/verify Tiered Cache, review manifest TTL.
|
||
- Trigger: chat p95 latency drift or websocket error growth.
|
||
- Action: scale chat service, tune Redis, separate noisy moderation workloads.
|
||
- Trigger: Prometheus WAL/compaction failures or query saturation.
|
||
- Action: tune retention/cardinality, increase resources, split dashboards/queries.
|
||
- Trigger: origin CPU/memory pressure on playout host.
|
||
- Action: optimize FFmpeg profile, isolate services, add secondary origin.
|
||
|
||
### Scaling Strategy
|
||
|
||
Scaling is designed to preserve cost efficiency first, then increase throughput and resilience in controlled phases.
|
||
|
||
#### Phase 1: Vertical + Cache-First (Current Baseline)
|
||
|
||
- Single primary origin in US-East with strong Cloudflare edge caching.
|
||
- Optimize segment immutability and manifest TTL before adding infrastructure.
|
||
- Scale compute vertically when CPU/memory pressure appears on playout/chat host.
|
||
|
||
#### Phase 2: Service Separation
|
||
|
||
- Split playout, chat, and observability workloads across dedicated nodes/VMs.
|
||
- Isolate noisy workloads (analytics, heavy admin tasks) from latency-sensitive chat paths.
|
||
- Introduce managed PostgreSQL/Redis or dedicated state nodes where operationally justified.
|
||
|
||
#### Phase 3: Multi-Origin Resilience
|
||
|
||
- Add warm standby or active-secondary origin.
|
||
- Use health-aware DNS/origin failover strategy.
|
||
- Keep HLS path and cache contract stable so viewer impact during failover is minimized.
|
||
|
||
#### Phase 4: Regional and Throughput Expansion
|
||
|
||
- Add additional origin region only when audience distribution and latency justify it.
|
||
- Keep chat affinity strategy explicit (regional routing, state replication considerations).
|
||
- Continuously evaluate cache miss economics vs additional origin complexity.
|
||
|
||
#### Scaling Triggers
|
||
|
||
- Rising cache miss ratio with sustained traffic growth.
|
||
- Chat p95/p99 latency drift or websocket error growth.
|
||
- Persistent origin resource saturation despite optimization.
|
||
- Increased blast-radius concerns from single-host dependency.
|
||
|
||
### Disaster Recovery and Backup Procedures
|
||
|
||
#### Recovery Targets
|
||
|
||
- RPO target: <= 24h for chat/moderation/admin metadata.
|
||
- RTO target: <= 60 minutes for restoring baseline streaming + chat operations.
|
||
|
||
#### Backup Scope
|
||
|
||
- Persistent service volumes:
|
||
- Postgres data (`postgres-data` volume).
|
||
- Redis append-only persistence (`redis-data` volume) where enabled.
|
||
- Prometheus long-term TSDB (`prometheus-data` volume) if historical metrics retention is required.
|
||
- Grafana state (`grafana-data` volume) if dashboards/users are managed in-instance.
|
||
- Host-mounted state directories:
|
||
- `data/chat`
|
||
- `data/playlist`
|
||
- `data/media` (authoritative source may be R2/object storage)
|
||
- `data/hls` (optional; can be regenerated by playout)
|
||
- Configuration and infra source of truth:
|
||
- Compose files, nginx config, monitoring configs, and environment template.
|
||
|
||
#### Backup Cadence
|
||
|
||
- Daily logical Postgres backups (plus weekly full snapshot).
|
||
- Daily sync of chat/playlist state directories.
|
||
- Weekly integrity check by restoring backups into an isolated environment.
|
||
- Retention policy baseline:
|
||
- Daily backups: 14 days
|
||
- Weekly backups: 8 weeks
|
||
- Monthly backups: 6 months
|
||
|
||
#### Recovery Runbook (Minimum)
|
||
|
||
1. Restore infrastructure configuration (compose, nginx, env, monitoring config).
|
||
2. Restore Postgres and Redis persisted state.
|
||
3. Restore `data/playlist` and chat persistence directories.
|
||
4. Rehydrate media from authoritative object storage if local media is missing.
|
||
5. Start stack and validate in order:
|
||
- `playout` health endpoint
|
||
- `chat` health endpoint
|
||
- nginx `/health`
|
||
- HLS playlist availability
|
||
- websocket chat connectivity
|
||
6. Verify admin authentication and moderation audit visibility.
|
||
7. Announce recovery and continue elevated monitoring for at least one playlist cycle.
|
||
|
||
#### Failure Modes and Strategy
|
||
|
||
- Single-host loss: redeploy stack on standby host, restore data backups, repoint DNS/edge origin.
|
||
- Storage corruption: restore last known-good snapshot and replay latest logical backups.
|
||
- Partial service failure: replace only impacted service and restore scoped data (for example Postgres only).
|
||
|
||
Document and rehearse this runbook quarterly; recovery that is not exercised is not reliable.
|
||
|
||
## Risks and Technical Debts
|
||
|
||
- Cloudflare ToS risk for heavy video on free tier (use R2 path; fallback to BunnyCDN if challenged).
|
||
- HLS latency (15–35s) may affect perceived chat synchronization.
|
||
- Single VPS as SPOF (mitigate with DNS failover in Phase 3).
|
||
- Upfront effort for video normalization and cache tuning.
|
||
- Moderation/abuse prevention operational load at scale.
|
||
- R2 request costs on cache misses (optimize with Tiered Cache; measure in prod).
|
||
- Deferred items: LL-HLS, ABR, E2EE, paid access controls, full Docker Compose.
|
||
|
||
**Technical Debt**: Detailed service configs, CI/CD, and advanced resilience are left for implementation ADRs.
|
||
|
||
## Glossary
|
||
|
||
| Term | Definition |
|
||
| ----------------- | ------------------------------------------------------------------------------------------ |
|
||
| HLS | HTTP Live Streaming – protocol using `.m3u8` manifests and media segments (`.ts` or fMP4). |
|
||
| Immutable Segment | HLS chunk with unique, unchanging name enabling long-lived caching. |
|
||
| R2 | Cloudflare object storage ($0 egress, ideal origin for cached HLS). |
|
||
| Edge Cache / POP | Cloudflare points-of-presence that cache and serve content close to viewers. |
|
||
| Playout | Continuous generation of a scheduled linear program from a playlist. |
|
||
| Normalization | Pre-processing videos to consistent codec/fps/keyframe settings for seamless playback. |
|
||
| Cache Rule | Cloudflare configuration defining caching behavior, TTL, and headers per URL pattern. |
|
||
| SPOF | Single Point of Failure (current single VPS origin). |
|
||
| Tiered Cache | Cloudflare feature reducing origin requests by checking upper-tier caches first. |
|
||
|
||
**References**: Cloudflare R2/Cache docs, MediaMTX/SRS/FFmpeg HLS guides, Hetzner pricing, hls.js documentation, standard video bitrate calculators. All original root `ADR-001-*.md` files have been consolidated and can be archived/deleted.
|