linearvoid/docs/architecture/README.md

Architecture Documentation for Linear Void

Linear Void architecture is documented using the arc42 template.

Companion docs: see Documentation README and adr/ for full decision rationale, alternatives, and consequences.

• Architecture Documentation for Linear Void
  • Introduction and Goals
  • Architecture Constraints
  • Context and Scope
  • Solution Strategy
  • Building Block View
  • Runtime View
  • Deployment View
  • Cross-cutting Concepts
  • Architecture Decisions
  • Quality Requirements
  • Risks and Technical Debts
  • Glossary

Introduction and Goals

Requirements Overview — Linear Void is an independent 24/7 linear video channel broadcasting self-owned pre-recorded videos to a primarily US audience via a custom lightweight HTML/CSS/JS SPA with synchronized HLS playback, schedule metadata, public chat rooms, and private 1:1 messaging. Fully independent of third-party video platforms (YouTube, Twitch, etc.).

Core Requirements (merged from all ADRs):

• Continuous synchronized playlist playback (live-style HLS, not VOD).
• Self-hosted source media, frontend, schedule, and chat data.
• Primary US audience/origin (East Coast preferred for low latency/peering).
• Extreme cost efficiency (near-zero egress via Cloudflare R2 + edge caching).
• Custom chat with rooms, DMs, presence, moderation, persistence.
• Storage for 48h of 720p30/1080p30 content.
• Browser-compatible playback (hls.js + native HLS).
• Operational simplicity and scalability.

Full rationale in adr/ADR-001.md.

Quality Goals

#	Attribute	Goal	Scenario
1	Cost Efficiency	<$25/month at moderate scale (100s concurrent)	R2 $0 egress + immutable .m4s/.ts edge caching removes per-viewer bandwidth cost
2	Performance/Latency	Low US latency; acceptable HLS delay	Hetzner Ashburn origin + Cloudflare US POPs (15–35s end-to-end)
3	Availability	24/7 with minimal downtime	FFmpeg/MediaMTX auto-restart, fallback slate, monitoring
4	Scalability	Grow 10s→1000s viewers without proportional cost	Edge caching + R2; only request volume limits
5	Maintainability	Lightweight, clear configs, low ops	Static SPA, Go/Node chat, SQLite/PostgreSQL+Redis
6	Security/Usability	Secure transport, moderation, good UX	HTTPS/WSS, Cloudflare WAF, rate limits, intuitive chat/player

Stakeholders: Product Owner, Engineering Team, US Viewers, Moderators/Admins, Operations — all expecting a cost-effective, reliable, low-latency, low-TCO independent channel with rich chat and easy management.

Architecture Constraints

• No YouTube/Twitch/paid Cloudflare Stream dependency.
• US-centric (origin + audience).
• Cost stays low even at scale (bandwidth dominant).
• Standard browser playback only (no plugins).
• Cloudflare ToS compliance for video (R2 as origin for large immutable files).
• HLS primary (standard, not LL-HLS initially).
• Self-hosted playlist normalization and continuous playout.

Context and Scope

Business Context — Linear Void runs a continuous scheduled, playlist-driven program to a US audience. The custom viewer site replaces traditional platforms, giving full control over content, schedule, UI, and real-time chat (public rooms + DMs).

High-Level Context Diagram

flowchart TD
    subgraph "External Users"
        Viewers[US Viewers<br/>Browser SPA + hls.js + Chat UI]
        Admins[Admins / Moderators<br/>Playlist & Schedule UI]
    end

    subgraph "Cloudflare Global Edge"
        CDN[Cloudflare CDN<br/>Cache Rules + WAF + TLS]
        Workers[Cloudflare Workers<br/>Static SPA Hosting]
        R2[Cloudflare R2<br/>HLS Segments + Source MP4s<br/>enam location hint]
    end

    subgraph "US-East Origin (Hetzner Ashburn VPS)"
        VPS[VPS CX22<br/>Nginx Proxy + Monitoring]
        Playout[Playout Service<br/>FFmpeg + MediaMTX/SRS<br/>Continuous HLS from Playlist]
        Chat[Chat Backend<br/>WebSockets + Redis + DB<br/>Public rooms + Private DMs]
    end

    Admins -->|HTTPS - Schedule/Playlist| Playout
    Playout -->|S3-compatible Upload| R2
    R2 -->|Cached HLS| CDN
    Viewers -->|HTTPS| Workers
    Viewers -->|HLS Requests| CDN
    Viewers -->|WSS Chat| Chat
    Chat --> VPS
    VPS -->|Proxy / Static| Workers
    CDN -->|Origin Pull on Miss| R2

External Interfaces: Cloudflare (CDN cache rules for .m4s/.ts vs .m3u8, R2 $0 egress, Workers static hosting, WAF/TLS); Hetzner VPS (US-East compute, good peering); Browsers (native HLS + WebSocket); Source (pre-normalized MP4s in R2).

Technical Context

• Playout: FFmpeg consuming normalized playlist → continuous HLS.
• Packaging/Delivery: MediaMTX/SRS/Nginx → immutable .m4s/.ts to R2; edge cache.
• Frontend: Static SPA with hls.js.
• Chat: Lightweight WebSocket service + Redis (presence/rate limits) + PostgreSQL/SQLite (history/moderation).
• Normalization: All sources H.264/AAC, constant 30 fps, 2s keyframes.

Mapping: Input = normalized MP4s in R2 + playlist; Output = live master.m3u8 + segments via Cloudflare, chat via WSS.

Solution Strategy

1. US-East Playout on Hetzner Ashburn VPS (CX22+): FFmpeg + MediaMTX/SRS continuous playlist→HLS (-c copy where possible).
2. Immutable HLS to Cloudflare R2: unique segment names; R2 origin with $0 egress.
3. Aggressive Edge Caching: long TTL for immutable .m4s/.ts; short (1–10s) for live .m3u8; Tiered Cache enabled — viewer delivery effectively free.
4. Lightweight Custom Frontend: static SPA + hls.js + chat sidebar.
5. Separate co-located Chat Service: WebSockets with persistence/moderation (low bandwidth).
6. Normalization & Monitoring: pre-process all videos; dashboards + alerts for playout health, cache hit ratio, R2 ops.

Accepted Trade-off: standard HLS latency (15–35s) acceptable for linear programming/chat; LL-HLS/WebRTC deferred.

Building Block View

Whitebox Overall System

graph TD
    subgraph Cloudflare
        CDN[CDN + Cache Rules<br/>.m4s and .ts immutable long TTL<br/>.m3u8 short TTL]
        R2[R2 - HLS + Source]
        Workers[Workers - Static SPA]
    end
    VPS[VPS Hetzner Ashburn]
    Playout[FFmpeg + MediaMTX/SRS<br/>Playlist → HLS]
    Chat[WebSocket Chat<br/>Node/Go + Redis + DB]
    Viewer[US Viewers]
    Admin[Admin UI]

    Admin --> Playout
    Playout --> R2
    R2 --> CDN
    Viewer --> Workers
    Viewer --> CDN
    Viewer --> Chat
    Chat --> VPS
    VPS --> Nginx[Nginx Proxy]

Motivation: decouple playout (continuous linear), delivery (cached immutable segments), interaction (chat). R2 + edge caching solves bandwidth cost while keeping synchronization and independence.

Contained blocks: Playout (FFmpeg + MediaMTX/SRS); Storage/Delivery (R2 + CDN); Frontend (SPA); Chat Service (WebSocket); Proxy/Static (Nginx).

Important interfaces: HTTP(S) for HLS, WSS for chat, S3-compatible for R2 uploads.

Level 2 — components with code locations and persistence

graph TB
    subgraph "Cloudflare Edge"
        CF[CDN + WAF<br/>Cache Rules]
        R2[R2 Bucket<br/>media + HLS]
        WORKERS[Workers<br/>SPA hosting]
    end

    subgraph "Docker Host (Hetzner CX22)"
        NGINX[nginx<br/>:80/:443<br/>proxy + auth]
        CHAT[chat service<br/>:3000<br/>Go + gorilla/websocket]
        PLAYOUT[playout service<br/>:8081<br/>Go + FFmpeg]
        REDIS[Redis 7<br/>:6379<br/>presence/rate/pubsub]
        DB[PostgreSQL 15<br/>:5432<br/>chat history/moderation]
    end

    subgraph "External"
        VIEWER[Viewer Browser<br/>hls.js + WebSocket]
        ADMIN[Admin Browser<br/>SPA + uploads]
    end

    VIEWER -->|HTTPS| CF
    VIEWER -->|WSS| NGINX
    ADMIN -->|HTTPS| CF
    CF --> NGINX
    NGINX --> CHAT
    NGINX --> PLAYOUT
    NGINX -->|/hls/*| R2
    CHAT --> REDIS
    CHAT --> DB
    PLAYOUT --> R2
    PLAYOUT -->|reads| R2

Playout Service (backend/playout/src/playlist.go)

• Role: continuous HLS encoding loop via FFmpeg concat demuxer.
• Input: data/playlist/playlist.txt (ordered MP4 paths); Output: HLS segments (.m4s, .ts) + live .m3u8 to data/hls/, mirrored to R2.
• Resilience: restart counter (>5/60s → slate); Prometheus on 8081.
• Validation: ffprobe gate (codec/fps/resolution/keyframe); -validate mode checks playlist without running.
• Seamless looping: resets HLS only when playlist fingerprint (SHA-256) changes, crash/slate recovery, or first run; else continues via -start_number.
• R2 retention: cleanupOldRuns keeps newest R2_KEEP_RUNS (default 2) capped at R2_MAX_RETENTION_BYTES (default 10 GiB), after each run + hourly backstop.
• Shared config: reads runtime config.json (loadSharedHLSConfig) as env fallback for HLS segment duration/window.

Chat Service (backend/chat/src/main.go)

• Role: WebSocket chat (rooms, DMs, moderation, admin API).
• Persistence: PostgreSQL 15 via GORM (messages, moderation, media metadata); SQLite dev fallback.
• Redis: presence, rate limiting, pub/sub multi-instance fan-out.
• Auth: session-cookie admin auth, Redis-backed expiry.
• Metrics on /metrics: connected clients, stored messages, DMs, moderation actions.

Frontend SPA (frontend/)

• Viewer src/main.ts: hls.js player, WS chat, presence, schedule overlay.
• Admin src/admin.ts: 9 tabs (dashboard, upload, playlist, schedule, moderation, chat-log, users, analytics, system).
• Build: Vite → frontend/dist/.

Reverse Proxy (infrastructure/nginx/nginx.conf)

• auth_request gate on /admin and /api/admin; WS upgrade for /ws; HLS CORS + OPTIONS preflight; /health JSON + /nginx-status stub_status.

Level 3 — Chat Service Modules

Module	Location	Responsibility
Connection Manager	main.go (hub, clients map)	Track active WS connections per room, fan-out
Message Parser	main.go (readPump, handleMessage)	Parse JSON envelope {type, room?, ...}, validate, route
Room Registry	main.go (rooms map)	Room name → client set, join/leave/close
DM Router	main.go (handleDirectMessage)	Resolve user, deliver private message
Persistence Layer	main.go (GORM: Message, DirectMessage, ModerationLog)	Store/retrieve messages, DMs, moderation
Redis Subsystem	main.go (redisClient)	INCR+EXPIRE rate limit, SETEX presence, PSUBSCRIBE fan-out
Auth Handler	main.go (handleAdminLogin, adminAuthMiddleware)	Session cookie create/validate, login rate limit
User Auth Handler	handlers/handlers.go	Viewer register/login/logout/session (bcrypt, user cookie)
DM History/Search	handlers/handlers.go	Paginated DM history + search
Runtime Config	config/runtime.go	Thread-safe JSON runtime config (HLS, Matomo)
Admin API Router	main.go (handleAdmin*)	Upload, playlist CRUD, schedule overrides, moderation, analytics, system
Metrics Exporter	main.go	connected_clients, messages_total, dms_total, moderation_actions, redis_up

Level 3 — Playout Service Modules

Module	Location	Responsibility
Playlist Parser	playlist.go (parsePlaylist)	Read playlist, skip comments/blanks, resolve paths
FFmpeg Manager	playlist.go (runPlayout)	Spawn/monitor FFmpeg concat, restart on crash
Loop Controller	playlist.go (playlistFingerprint, hlsNextStartNumber)	Detect changes (SHA-256), continue numbering
Slate Controller	playlist.go (activateSlate/deactivateSlate)	Switch to slate on excessive failures
R2 Sync	playlist.go (syncToR2)	Upload segments + manifest to R2
R2 Cleanup	playlist.go (cleanupOldRuns)	Keep last N run folders under size cap; delete hls/r-*/*
Health Endpoint	playlist.go (healthHandler)	/health JSON: mode, segment count, uptime
Metrics Exporter	playlist.go	restarts_total, mode, loops_total, run_duration_seconds, r2_cleanup_objects_total

Level 3 — Frontend Modules

Module	File	Responsibility
HLS Player	src/main.ts	hls.js init, attach, error recovery
WebSocket Client	src/main.ts	Connect/parse envelope/dispatch
Chat UI	src/main.ts	Message list, room tabs, DM panel, presence dots, typing, user-list
DM Search	src/main.ts	/api/dm/search + render
Auth UI	src/main.ts + index.html	Viewer login/register/logout, session, user label
Schedule Overlay	src/main.ts	Poll /api/schedule, render current/next
Admin Router	src/admin.ts	Tab nav, dynamic pages
Upload Manager	src/admin.ts	Multipart chunk upload, normalization badges
Playlist Editor	src/admin.ts	Drag-reorder (gated to ready media), save
Moderation Panel	src/admin.ts + public/admin/moderation.html	Mute/ban, action log
Analytics Dashboard	src/admin.ts + public/admin/analytics.html	Message counts, top rooms, moderation stats
System Config	src/admin.ts + public/admin/system.html	Health, runtime config form, restart

Runtime View

Playout & Delivery Scenario

1. Admin adds normalized MP4 to R2 + updates playlist.
2. Playout worker (FFmpeg concat loop) reads next file, packages HLS segments (2s).
3. Segments uploaded to R2 with unique immutable names; live .m3u8 updated.
4. Viewer loads SPA → requests manifest → hls.js follows current edge.
5. Cloudflare serves .m4s/.ts from nearest edge cache (origin hit once per POP/region).
6. Parallel WebSocket to chat service for synchronized interaction.

Manifest freshness keeps all viewers on the same timeline; cache hit ratio drives cost/latency. Fallback slate activates on failure. (Moderation events, scale spikes, failover in ADR.)

Deployment View

CI/CD Pipeline — Parallel workflows on each push to main: CI builds/pushes images while CD deploys to the test lab (192.168.88.40); a dedicated release workflow handles semantic version tags. Production Hetzner VPS planned but not yet provisioned — same pipeline deploys there by changing DOCKER_HOST.

flowchart LR
    subgraph "Gitea Actions"
        CI[CI Workflow<br/>Build + Push Images]
        CD[CD Workflow<br/>Pull + Deploy]
    end
    subgraph "Gitea Container Registry"
        REGISTRY[git.allucanget.biz<br/>playout:latest<br/>chat:latest<br/>nginx:latest]
    end
    subgraph "Test Lab Host (192.168.88.40) — Production Hetzner TBD"
        DOCKER[Docker Engine<br/>via TCP :2375]
        PLAYOUT[linearvoid-playout]
        CHAT[linearvoid-chat]
        NGINX[linearvoid-nginx]
        REDIS[linearvoid-redis]
        DB[linearvoid-db]
        PROM[linearvoid-prometheus]
        GRAF[linearvoid-grafana]
    end

    CI -->|push| REGISTRY
    REGISTRY -->|pull| CD
    CD -->|deploy via DOCKER_HOST| DOCKER
    DOCKER --> PLAYOUT
    DOCKER --> CHAT
    DOCKER --> NGINX
    DOCKER --> REDIS
    DOCKER --> DB
    DOCKER --> PROM
    DOCKER --> GRAF

Pipeline sequence

1. CI (.gitea/workflows/ci.yml): on push to main + pull_request; test job (frontend lint/build, Go validation); docker job (after test, main only) builds playout/chat/nginx with Buildx, pushes main + latest tags to Gitea registry.
2. CD (.gitea/workflows/cd.yml): on push to main (parallel) or workflow_dispatch; logs into remote Docker host, generates .env from secrets, seeds Prometheus/Grafana volumes, deploys via docker compose -f docker-compose.prod.yml up -d --remove-orphans --force-recreate to 192.168.88.40:2375 (switch to Hetzner IP for prod); health checks with retry for playout init.
3. Release (.gitea/workflows/release.yml): on tags *.*.* (e.g. 1.0.9); pushes X.Y.Z, X.Y, latest; creates Gitea release.

Infrastructure Level 1 — Single low-cost US-East origin (Hetzner) with global edge caching. High cache-hit ratio, low US latency, edge-scale WAF/DDoS. Mapping: Playout + Chat + Nginx → Hetzner CX22 (Ashburn, VA); HLS storage/caching/static SPA → Cloudflare R2 + CDN + Workers (enam location hint); all public traffic through Cloudflare.

Infrastructure Level 2

• Hetzner VPS (planned): playout (FFmpeg/MediaMTX), chat backend, Nginx; monitoring dashboards + UptimeRobot; backups to R2. Test lab currently fills this role (see DEPLOYMENT.md).
• Cloudflare: R2 bucket; Cache Rules (.m4s/.ts Cache Everything/immutable/long TTL; .m3u8 short TTL/must-revalidate); WAF, rate limiting, Tiered Cache; Workers for SPA.
• Future: Docker Compose, managed DB, secondary regions (later ADRs).

Cross-cutting Concepts

Caching & Cost Optimization — Immutable segment naming + targeted Cloudflare Cache Rules + R2 ($0 egress) = near-zero viewer bandwidth. Long TTL on .m4s/.ts, short on live manifests. Monitor hit ratio and R2 Class B ops.

Video Normalization & Playout — All sources must meet the Linear Void source profile before production playlist acceptance. The chat backend auto-normalizes non-conforming uploads on ingest (media.NormalizeVideo), removing manual pre-processing. Prevents decode resets, timestamp drift, AV discontinuities during concat playout.

Required Source Profile (v1)

Field	Required Value	Notes
Container	MP4	Single-video-asset input
Video codec	H.264/AVC (libx264)	Baseline delivery codec
Audio codec	AAC-LC	Stereo preferred
Resolution	1280x720 or 1920x1080	Keep single res per playlist block when possible
Framerate	Constant 30 fps (-r 30)	VFR must be normalized
GOP/keyframe	2s (-g 60 -keyint_min 60)	Matches HLS segment boundaries
Pixel format	yuv420p	Browser baseline
Audio sample rate	48 kHz	Stable concat timing
Audio bitrate	128k–192k	Predictable envelope

Reference normalization command

ffmpeg -i input.mov \
  -c:v libx264 -preset medium -crf 21 \
  -r 30 -g 60 -keyint_min 60 -sc_threshold 0 \
  -pix_fmt yuv420p \
  -c:a aac -b:a 160k -ar 48000 -ac 2 \
  -movflags +faststart \
  output-normalized.mp4

Automatic normalization & verification (ffprobe)

• media.NormalizeVideo runs ffmpeg with the profile, writes *.normalized.mp4, re-verifies with ffprobe.
• Media status: ready, needs_normalization, normalizing, normalization_failed; admin UI "Normalize" action with polling.
• POST /api/admin/media/{id}/normalize re-triggers manually.
• Keep verification aligned with runtime playout assumptions to preserve -c copy efficiency.

Runtime Configuration Store — Thread-safe, JSON-persisted store in backend/chat/src/config/runtime.go, shared with playout via the playlist volume (/app/playlist/config.json).

• Editable via GET/PUT /api/admin/config.
• Fields: hlsSegmentDuration (2–15s, default 2), hlsWindowSize (3–100, default 12), matomoUrl, matomoSiteId.
• Playout reads same file (loadSharedHLSConfig) as env fallback.
• Typed setters validate; invalid → 400.

Analytics (Matomo) — Viewer sends pageviews via public GET /api/matomo, returning matomoUrl + matomoSiteId from the runtime config. Optional per deployment (empty matomoUrl disables).

Monitoring & Resilience — Two-layer model: (1) application layer (chat + platform health) via Prometheus + lightweight Grafana; (2) infrastructure/runtime layer (Prometheus self-observability, TSDB, Go runtime) via detailed Grafana dashboard.

Prometheus baseline (monitoring/prometheus/prometheus.yml): 15s scrape; 7 jobs — chat→chat:3000, playout→playout:8081, nginx→nginx-exporter:9113, redis→redis-exporter:9121, postgres→postgres-exporter:9187, node→node-exporter:9100, prometheus→localhost:9090. TSDB retention 30d + 15GB disk cap (both compose files).

Grafana dashboards: monitoring/grafana/dashboard.json (app metrics: connected clients, messages, moderation, Redis reachability, admin sessions, scrape health); monitoring/grafana/dashboards/dashboard.json (Prometheus ops: status, uptime, CPU/mem, TSDB ingestion/churn/errors, HTTP latencies, query load, GC/scheduler). Use app dashboard for service behavior; Prometheus overview for platform diagnostics/capacity.

Alerting rules (monitoring/prometheus/alerts.yml, loaded via rule_files): 10 rules — ChatDown, PlayoutDown, RedisDown, PlayoutSlateActive, HighChatLatency, PrometheusWALFailing, HighCPU, LowCacheHitRatio, HighRedisMemoryUsage, LowDiskSpace. Route warning/critical to ops channel; attach runbook links.

Resilience baseline: emergency slate fallback; auto-restart for playout/chat/nginx; weekly backup/restore drills for stateful data; incident runbook triage: edge reachability → origin health → playout progression → chat connectivity → data-plane dependencies.

API Reference — Three operational API surfaces (canonical contract in API.md).

Chat WebSocket API: endpoint /ws; wss:// prod / ws:// local; query params username, room; envelope {room, username, text, type: public|private|system}; responsibilities — room fan-out, DM routing, moderation commands, rate limiting/presence.

Playout & Delivery REST: playout /health; edge/origin /health via nginx; HLS manifest/segment namespace under /hls/. Liveness for orchestration + stable HLS path contract.

Admin REST: auth/session /api/admin/login|logout|session; domains overview/analytics/users/moderation/schedule/playlist/media; runtime config GET/PUT /api/admin/config; normalization POST /api/admin/media/{id}/normalize; uploads /api/admin/media/uploads (init/resume), /uploads/{uploadId}/chunk, /complete. Protected control plane; media metadata + upload orchestration; moderation/audit + system status.

Viewer REST: auth POST /api/auth/register|login|logout, GET /api/auth/session; DMs GET /api/dm/history (paginated), GET /api/dm/search; analytics GET /api/matomo (public).

Cloudflare Configuration Reference — Cloudflare is part of the production control plane, not only a CDN.

DNS & zone routing: linearvoid.tv → app origin (nginx); www.linearvoid.tv → redirect apex; media.linearvoid.tv → media domain for HLS/object delivery when split-domain enabled.

Cache rules baseline: .ts/.m4s Cache Everything, long TTL, immutable; .m3u8 short TTL (1–10s), revalidation-friendly.

Edge security/controls: enforce HTTPS/WSS + redirect HTTP; WAF baseline; rate limits on abusive patterns (auth/admin); minimize origin exposure.

Worker & header policy: the linearvoid-media Worker (infrastructure/cloudflare/, R2-bound) owns media.linearvoid.tv and enforces immutable caching for segments, short TTL for live manifests, plus HLS CORS/Range; the linearvoid-frontend Worker (frontend/wrangler.toml) serves the static SPA on linearvoid.tv. Automated deploys: workers-deploy.yml (SPA) and workers-media-deploy.yml (media).

Operations & validation: verify cache status progression (MISS→HIT), manifest freshness under active playout; track hit ratio, request/error rates, ToS/cost posture.

Security & Moderation — Split across edge (Cloudflare), proxy (nginx), and application (chat/admin).

• Transport/edge: HTTPS/WSS; WAF + rate limiting; Worker enforces immutable headers.
• nginx: auth_request on /admin/, /api/admin/; admin UI → redirect /admin/login/; admin API → JSON 401; WS upgrade forwarded for /ws.
• Admin auth: cookie session (linearvoid_admin_session); HttpOnly, SameSite=Strict, Secure when HTTPS (from TLS or X-Forwarded-Proto); server-side expiry.
• Viewer accounts: register/login/logout; bcrypt passwords; linearvoid_user_session cookie, 30-day default TTL (USER_SESSION_TTL_HOURS); SameSite=None; Secure over HTTPS (cross-site from Workers SPA), SameSite=Lax over plain HTTP (local dev); sign-in UI in frontend/index.html.
• Rate limiting/abuse: Redis INCR + EXPIRE; baseline >10 messages in 10s → throttle. Redis down → presence/rate-limit degrade gracefully (warn logged; core chat available).
• CORS: HLS endpoints Access-Control-Allow-Origin: *; API routes reflect allowlisted origin + Access-Control-Allow-Credentials: true; viewer cookie SameSite=None; Secure cross-site; admin cookie stays SameSite=Strict.
• Moderation/privacy: commands /mod, /mute, /ban handled in chat flow, persisted. DMs are application-level private, not end-to-end encrypted. Formalize moderation/audit retention in runbook/ops docs for production hardening.

Chat — WebSockets + Redis (presence, pub/sub, rate limits) + DB (history, audit). Public rooms + on-demand private chats; scales independently of video. DM history/search persisted; viewer UI has typing indicator, user-list toggle, DM search. (Details/cache tables/bitrate/cost model/configs in ADR-001.)

Architecture Decisions

Consolidated ADRs (in adr/):

• ADR-001: overall architecture, HLS + R2 caching, storage calcs, cost model, risks.
• ADR-002: project folder/module structure and rationale.
• ADR-003: tech stack (TypeScript frontend; Go for chat/playout wrapper; Bash/Python scripts), Docker-first, documented assumptions.
• ADR-004: video admin web interface (upload, playlist, schedule overrides, normalization).
• ADR-005: admin UI for users, moderation dashboard, analytics, system config.
• ADR-006: data storage (SQLite vs PostgreSQL for v1, migration path).
• ADR-007: Redis usage (use cases, TTL, persistence, fallback, admin config).

Key decisions: self-managed US-East playout (Hetzner Ashburn) FFmpeg + MediaMTX/SRS → HLS → R2 + edge cache; module structure mirroring arc42 blocks; Go for perf-critical services, TypeScript frontend, Docker everywhere; admin interfaces (HTMX + custom CSS); SQLite primary for v1 with PostgreSQL migration path; Redis for rate limiting/presence/pub/sub/session with defined TTL and fallback.

Quality Requirements

See Quality Goals. Primary: cost (near-zero egress), availability (auto-restart + fallback), scalability (edge caching), usability (synchronized player + responsive chat).

Performance Benchmarks & Capacity Planning

Baseline SLO targets (v1): playback startup < 4s after first manifest request on warm cache; no segment starvation in steady state; end-to-end HLS 15–35s; chat delivery p95 < 500ms same region; admin read APIs p95 < 1s.

Workload assumptions: US audience, US-East origin; edge serves most segments, origin handles manifest churn + cache misses; chat low-bandwidth vs video.

Initial capacity model: bitrates — 720p30 ~2.6–4.2 Mbps, 1080p30 ~5.1–8.2 Mbps (with audio). 48h library — 720p30 ~57–90 GB raw (alloc ~80–110 GB); 1080p30 ~110–175 GB raw (alloc ~130–180 GB). Cost driven by cache efficiency + request volume, not direct origin egress.

Benchmark plan: playback (time-to-first-frame, rebuffers across US regions; manifest freshness/segment availability); chat (concurrent WS + fan-out; p50/p95 latency, Redis fallback); admin/API (p95 latency, error rate); platform (TSDB ingestion, query p99, GC in Grafana).

Capacity triggers/actions: viewer growth + rising cache-miss → tighten cache, enable/verify Tiered Cache, review manifest TTL; chat p95 drift/WS errors → scale chat, tune Redis, isolate moderation; Prometheus WAL/query saturation → tune retention/cardinality, add resources, split queries; origin CPU/mem pressure → optimize FFmpeg, isolate services, add secondary origin.

Scaling Strategy — preserve cost efficiency first, then throughput/resilience in phases:

• Phase 1 — Vertical + Cache-First (current): single US-East origin, strong edge caching; optimize segment immutability + manifest TTL before adding infra; scale compute vertically on pressure.
• Phase 2 — Service Separation: split playout/chat/observability across nodes; isolate noisy workloads from latency-sensitive chat; introduce managed PostgreSQL/Redis or dedicated state nodes where justified.
• Phase 3 — Multi-Origin Resilience: warm standby or active-secondary origin; health-aware DNS/origin failover; keep HLS/cache contract stable.
• Phase 4 — Regional/Throughput Expansion: add origin region only when audience/latency justify; explicit chat affinity strategy; evaluate cache-miss economics vs origin complexity.
• Scaling triggers: rising cache-miss with sustained growth; chat p95/p99 drift or WS errors; persistent origin saturation; increased blast-radius from single-host dependency.

Disaster Recovery & Backup

Recovery targets: RPO ≤ 24h (chat/moderation/admin metadata); RTO ≤ 60 min (restore baseline streaming + chat).

Backup scope: persistent volumes — Postgres (postgres-data), Redis append-only (redis-data) where enabled, Prometheus TSDB (prometheus-data) if retention required, Grafana state (grafana-data); host-mounted dirs — data/chat, data/playlist, data/media (authoritative may be R2), data/hls (optional, regenerable); source of truth — compose, nginx, monitoring configs, env template.

Cadence: daily logical Postgres backups (+ weekly full snapshot); daily sync of chat/playlist dirs; weekly restore integrity check in isolated env. Retention: daily 14 days, weekly 8 weeks, monthly 6 months.

Recovery runbook (minimum):

1. Restore infra config (compose, nginx, env, monitoring).
2. Restore Postgres + Redis state.
3. Restore data/playlist + chat persistence dirs.
4. Rehydrate media from authoritative object storage if missing.
5. Start stack; validate in order: playout /health, chat /health, nginx /health, HLS availability, WS chat connectivity.
6. Verify admin auth + moderation audit visibility.
7. Announce recovery; elevated monitoring ≥ one playlist cycle.

Failure modes: single-host loss → redeploy on standby, restore backups, repoint DNS/edge; storage corruption → restore last good snapshot + replay logical backups; partial failure → replace only impacted service, restore scoped data. Rehearse quarterly.

Risks and Technical Debts

• Cloudflare ToS risk for heavy video on free tier (use R2; fallback to BunnyCDN).
• HLS latency (15–35s) may affect perceived chat sync.
• Single VPS = SPOF (mitigate with DNS failover, Phase 3).
• Upfront effort for normalization + cache tuning.
• Moderation/abuse operational load at scale.
• R2 request costs on cache misses (optimize Tiered Cache; measure in prod).
• Deferred: LL-HLS, ABR, E2EE, paid access controls, full Docker Compose.
• Debt: detailed service configs, CI/CD, advanced resilience left for implementation ADRs.

Glossary

Term	Definition
HLS	HTTP Live Streaming — .m3u8 manifests + media segments (fMP4).
Immutable Segment	HLS chunk with unique, unchanging name enabling long-lived caching.
R2	Cloudflare object storage ($0 egress; ideal origin for cached HLS).
Edge Cache / POP	Cloudflare points-of-presence caching/serving content near viewers.
Playout	Continuous generation of a scheduled linear program from a playlist.
Normalization	Pre-processing to consistent codec/fps/keyframe settings for seamless playback.
Cache Rule	Cloudflare config for caching behavior, TTL, headers per URL pattern.
SPOF	Single Point of Failure (current single VPS origin).
Tiered Cache	Cloudflare feature reducing origin requests by checking upper-tier caches first.

References: Cloudflare R2/Cache docs, MediaMTX/SRS/FFmpeg HLS guides, Hetzner pricing, hls.js docs, video bitrate calculators. Original root ADR-001-*.md files consolidated and can be archived/deleted.