zwitschi/linearvoid
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 9 Wiki Activity
Files
887023a4919cb687b0f709d9e8fb801e9fb0412c
linearvoid/docs/architecture/README.md
T

47 KiB
Raw Blame History

Architecture Documentation for Linear Void

The architecture documentation for the Linear Void project is structured according to the arc42 template. This template provides a comprehensive framework for documenting software architecture, ensuring that all relevant aspects of the system are covered.

Table of Contents

Companion Documentation

  • API.md - Implemented WebSocket and REST API reference.
  • DEPLOYMENT.md - Production deployment and operational runbook.
  • MONITORING.md - Metrics, dashboards, and monitoring troubleshooting.
  • SECURITY.md - Security model, controls, and hardening baseline.

Introduction and Goals

Requirements Overview

Linear Void is an independent 24/7 linear video channel broadcasting a playlist of self-owned pre-recorded videos. Viewers access it via a custom lightweight HTML/CSS/JavaScript SPA with synchronized HLS playback, schedule metadata, public chat rooms, and private 1:1 messaging. Complete independence from third-party video platforms (YouTube, Twitch, etc.).

Core Requirements (merged from all ADRs):

  • Continuous synchronized playback from playlist (live-style HLS, not VOD).
  • Self-hosted source media, frontend, schedule, and chat data.
  • Primary US audience/origin (East Coast preferred for low latency/peering).
  • Extreme cost efficiency (bandwidth dominant; target near-zero egress via Cloudflare R2 + edge caching).
  • Custom chat with rooms, DMs, presence, moderation, persistence.
  • Storage planning for 48h of 720p30/1080p30 content.
  • Browser-compatible playback (hls.js + native HLS).
  • Operational simplicity and scalability.

Full decision rationale, alternatives, and consequences in adr/ADR-001.md.

Quality Goals

Priority Quality Attribute Goal Scenario
1 Cost Efficiency <$25/month for moderate scale (100s concurrent) R2 $0 egress + immutable .m4s and .ts edge caching eliminates per-viewer bandwidth cost
2 Performance / Latency Low US latency; acceptable HLS delay Hetzner Ashburn origin + Cloudflare US POPs (15–35s end-to-end)
3 Availability 24/7 with minimal downtime FFmpeg/MediaMTX auto-restart loop, fallback slate, monitoring
4 Scalability Grow from 10s to 1000s viewers without proportional cost Edge caching + R2; only request volume limits
5 Maintainability Lightweight, clear configs, low ops overhead Static SPA, Go/Node chat, SQLite/PostgreSQL+Redis
6 Security / Usability Secure transport, moderation tools, good UX HTTPS/WSS, Cloudflare WAF, rate limits, intuitive chat/player

Stakeholders

Role/Name Contact Expectations
Product Owner - Cost-effective independent 24/7 channel with synchronized viewing and rich chat
Engineering Team - Simple, evolvable stack (FFmpeg, HLS, Cloudflare R2, lightweight WebSocket chat)
US Viewers - Reliable low-latency playback, responsive synchronized chat experience
Moderators / Admins - Easy playlist management, moderation dashboard, monitoring/alerting
Operations - Low TCO, clear cost model, automated failover, backups

(See adr/ADR-001.md for detailed forces, decision drivers, and rationale.)

Architecture Constraints

  • Independence from video hosting platforms (no YouTube, Twitch, paid Cloudflare Stream).
  • US-centric (origin + audience).
  • Cost must remain low even at scale (bandwidth dominant).
  • Standard browser playback only (no plugins).
  • Cloudflare ToS compliance for video delivery (use R2 as origin for large immutable files).
  • HLS as primary protocol (standard, not LL-HLS initially).
  • Self-hosted playlist normalization and continuous playout required.

(See adr/ADR-001.md for full forces and trade-offs.)

Context and Scope

Business Context

Linear Void provides a continuously running scheduled video program (playlist-driven) to a primarily US audience. The custom viewer site replaces traditional platforms, giving full control over content, schedule, UI, and real-time chat (public rooms + private DMs).

High-Level Context Diagram (Mermaid)

flowchart TD
    subgraph "External Users"
        Viewers[US Viewers<br/>Browser SPA + hls.js + Chat UI]
        Admins[Admins / Moderators<br/>Playlist & Schedule UI]
    end

    subgraph "Cloudflare Global Edge"
        CDN[Cloudflare CDN<br/>Cache Rules + WAF + TLS]
        Pages[Cloudflare Pages<br/>Static SPA Hosting]
        R2[Cloudflare R2<br/>HLS Segments + Source MP4s<br/>enam location hint]
    end

    subgraph "US-East Origin (Hetzner Ashburn VPS)"
        VPS[VPS CX22<br/>Nginx Proxy + Monitoring]
        Playout[Playout Service<br/>FFmpeg + MediaMTX/SRS<br/>Continuous HLS from Playlist]
        Chat[Chat Backend<br/>WebSockets + Redis + DB<br/>Public rooms + Private DMs]
    end

    Admins -->|HTTPS - Schedule/Playlist| Playout
    Playout -->|S3-compatible Upload| R2
    R2 -->|Cached HLS| CDN
    Viewers -->|HTTPS| Pages
    Viewers -->|HLS Requests| CDN
    Viewers -->|WSS Chat| Chat
    Chat --> VPS
    VPS -->|Proxy / Static| Pages
    CDN -->|Origin Pull on Miss| R2

External Interfaces:

  • Cloudflare: CDN (cache rules for .m4s and .ts vs .m3u8), R2 (object storage, $0 egress), Pages (static hosting), WAF/TLS.
  • Hetzner VPS: Compute for playout and chat (low-cost US-East with good peering).
  • Browsers: Native HLS (<video> + hls.js) and WebSocket APIs.
  • Source: Pre-normalized MP4 files uploaded to R2.

This diagram synthesizes the context views from all original ADRs. Full decision details, cache rules, storage calculations, and rationale are in adr/ADR-001.md.

Technical Context

  • Playout: FFmpeg consuming normalized playlist → continuous HLS.
  • Packaging/Delivery: MediaMTX/SRS/Nginx → immutable .m4s and .ts segments to R2; Cloudflare edge cache.
  • Frontend: Static SPA with hls.js.
  • Chat: Lightweight WebSocket service (Node/Go) + Redis (presence/rate limits) + PostgreSQL/SQLite (history/moderation).
  • Normalization: All source videos must be H.264/AAC, 30 fps constant, 2s keyframes.

Mapping:

  • Input: Normalized MP4s in R2 + playlist.
  • Output: Live master.m3u8 + segments served via Cloudflare; chat via WSS.

Solution Strategy

Core Strategy (synthesized & accepted from all ADRs):

  1. US-East Playout on Hetzner Ashburn VPS (CX22 or equivalent): FFmpeg + MediaMTX/SRS for continuous playlist-to-HLS conversion (-c copy where possible).
  2. Immutable HLS to Cloudflare R2: Segments use unique names; R2 acts as origin with $0 egress.
  3. Aggressive Edge Caching: Long TTL for .m4s and .ts (immutable), short TTL (1–10s) for live .m3u8. Tiered Cache enabled. This makes viewer delivery essentially free.
  4. Lightweight Custom Frontend: Static HTML/JS SPA + hls.js player + embedded chat sidebar.
  5. Separate but co-located Chat Service: WebSockets with persistence and moderation (low bandwidth).
  6. Normalization & Monitoring: Pre-process all videos; use dashboards + alerts for playout health, cache hit ratio, R2 ops.

This eliminates egress economics, ensures synchronization via live playlist, keeps ops simple, and scales well. See adr/ADR-001.md for detailed cache rules, storage tables, cost model, configs (FFmpeg/Nginx examples), and phase-by-phase implementation plan.

Key Trade-off Accepted: Standard HLS latency (15–35s) is acceptable for linear programming/chat. LL-HLS/WebRTC deferred.

Building Block View

Whitebox Overall System

Overview (Mermaid diagram):

graph TD
    subgraph Cloudflare
        CDN[CDN + Cache Rules<br/>.m4s and .ts immutable long TTL<br/>.m3u8 short TTL]
        R2[R2 - HLS + Source]
        Pages[Pages - Static SPA]
    end
    VPS[VPS Hetzner Ashburn]
    Playout[FFmpeg + MediaMTX/SRS<br/>Playlist → HLS]
    Chat[WebSocket Chat<br/>Node/Go + Redis + DB]
    Viewer[US Viewers]
    Admin[Admin UI]

    Admin --> Playout
    Playout --> R2
    R2 --> CDN
    Viewer --> Pages
    Viewer --> CDN
    Viewer --> Chat
    Chat --> VPS
    VPS --> Nginx[Nginx Proxy]

Motivation: Decouple playout (continuous linear), delivery (cached immutable segments), and interaction (chat). Cloudflare R2 + edge caching solves the bandwidth cost problem while maintaining synchronization and independence.

Contained Building Blocks (high-level):

  • Playout: FFmpeg + MediaMTX/SRS (continuous HLS from playlist).
  • Storage/Delivery: Cloudflare R2 + CDN (immutable segments).
  • Frontend: Static SPA (hls.js player + chat UI).
  • Chat Service: WebSocket backend with persistence/moderation.
  • Proxy/Static: Nginx on VPS.

Important Interfaces: HTTP(S) for HLS, WSS for chat, S3-compatible for R2 uploads.

Detailed black-box descriptions, level-2/3 views, and code locations are in adr/ADR-001.md and future implementation ADRs.

Level 2

Detailed component diagram with code locations and persistence.

graph TB
    subgraph "Cloudflare Edge"
        CF[CDN + WAF<br/>Cache Rules]
        R2[R2 Bucket<br/>media + HLS]
        PAGES[Pages<br/>SPA hosting]
    end

    subgraph "Docker Host (Hetzner CX22)"
        NGINX[nginx<br/>:80/:443<br/>proxy + auth]
        CHAT[chat service<br/>:3000<br/>Go + gorilla/websocket]
        PLAYOUT[playout service<br/>:8081<br/>Go + FFmpeg]
        REDIS[Redis 7<br/>:6379<br/>presence/rate/pubsub]
        DB[PostgreSQL 15<br/>:5432<br/>chat history/moderation]
    end

    subgraph "External"
        VIEWER[Viewer Browser<br/>hls.js + WebSocket]
        ADMIN[Admin Browser<br/>SPA + uploads]
    end

    VIEWER -->|HTTPS| CF
    VIEWER -->|WSS| NGINX
    ADMIN -->|HTTPS| CF
    CF --> NGINX
    NGINX --> CHAT
    NGINX --> PLAYOUT
    NGINX -->|/hls/*| R2
    CHAT --> REDIS
    CHAT --> DB
    PLAYOUT --> R2
    PLAYOUT -->|reads| R2

Playout Service (backend/playout/src/playlist.go)

  • Role: Continuous HLS encoding loop using FFmpeg concat demuxer
  • Input: data/playlist/playlist.txt — ordered list of MP4 file paths
  • Output: HLS segments (.m4s and .ts) + live manifest (.m3u8) written to data/hls/, mirrored to R2
  • Resilience: Restart counter (>5/60s → activate slate), Prometheus metrics on port 8081
  • Validation: ffprobe gate on all ingested files (codec, fps, resolution, keyframe interval)

Chat Service (backend/chat/src/main.go)

  • Role: WebSocket chat with rooms, DMs, moderation, admin API
  • Persistence: PostgreSQL 15 via GORM (messages, moderation logs, media metadata). SQLite for dev fallback
  • Redis: Presence tracking, rate limiting, pub/sub for multi-instance fan-out
  • Auth: Session-cookie admin auth with Redis-backed expiry
  • Metrics: Connected clients, stored messages, DMs, moderation actions on /metrics

Frontend SPA (frontend/)

  • Viewer: src/main.ts — hls.js player, WebSocket chat, presence indicators, schedule overlay
  • Admin: src/admin.ts — 9 dynamic tabs (dashboard, upload, playlist, schedule, moderation, chat-log, users, analytics, system)
  • Build: Vite → static output in frontend/dist/

Reverse Proxy (infrastructure/nginx/nginx.conf)

  • auth_request gate on /admin and /api/admin
  • WebSocket upgrade for /ws
  • HLS CORS headers + OPTIONS preflight
  • /health JSON endpoint, /nginx-status stub_status

Level 3

Module-level breakdown per service.

Chat Service Modules

Module File/Lines Responsibility
Connection Manager main.go (hub, clients map) Track active WebSocket connections per room, fan-out messages
Message Parser main.go (readPump, handleMessage) Parse JSON envelope {type, room?, ...}, validate fields, route to handler
Room Registry main.go (rooms map, room broadcasts) Map room name → client set, handle join/leave/close
DM Router main.go (handleDirectMessage) Resolve target user in clients map, deliver private message
Persistence Layer main.go (GORM models: Message, DirectMessage, ModerationLog) Store/retrieve chat messages, DMs, moderation events
Redis Subsystem main.go (redisClient, rate limiting, pub/sub, presence) INCR+EXPIRE rate limit, SETEX presence, PSUBSCRIBE room fan-out
Auth Handler main.go (handleAdminLogin, adminAuthMiddleware) Session cookie creation/validation, login rate limiting
Admin API Router main.go (handleAdmin*) Upload, playlist CRUD, schedule overrides, moderation actions, analytics, system summary
Metrics Exporter main.go (prometheus.MustRegister) Expose connected_clients, messages_total, dms_total, moderation_actions, redis_up

Playout Service Modules

Module File/Lines Responsibility
Playlist Parser playlist.go (parsePlaylist) Read playlist.txt, skip comments/blank lines, resolve file paths
FFmpeg Manager playlist.go (runPlayout, restart logic) Spawn/monitor FFmpeg concat process, restart on crash
Slate Controller playlist.go (activateSlate, deactivateSlate) Switch output to slate loop on excessive failures
R2 Sync playlist.go (syncToR2) Upload HLS segments + manifest to R2 bucket
Health Endpoint playlist.go (healthHandler) /health → JSON with mode, segment count, uptime
Metrics Exporter playlist.go (prometheus) Expose restarts_total, mode (live/slate)

Frontend Modules

Module File Responsibility
HLS Player src/main.ts hls.js initialization, attach to video element, error recovery
WebSocket Client src/main.ts Connect/disconnect, parse envelope, dispatch to UI handlers
Chat UI src/main.ts Message list rendering, room tabs, DM panel, presence dots
Schedule Overlay src/main.ts Poll /api/schedule, render current/next program
Admin Router src/admin.ts Tab-based navigation, dynamic page loading
Upload Manager src/admin.ts Multipart upload with chunk progress, ffprobe validation feedback
Playlist Editor src/admin.ts Drag-reorder, add/remove items, save to /api/admin/playlist
Moderation Panel src/admin.ts + public/admin/moderation.html Inline mute/ban, action log viewer
Analytics Dashboard src/admin.ts + public/admin/analytics.html Message counts, top rooms, moderation stats
System Config src/admin.ts + public/admin/system.html Service health, config metadata, restart buttons

Runtime View

Playout & Delivery Scenario

  1. Admin adds normalized MP4 to R2 + updates playlist.
  2. Playout worker (FFmpeg loop with concat demuxer) reads next file, packages as HLS segments (4-6s).
  3. Segments uploaded to R2 with unique immutable names; live .m3u8 updated.
  4. Viewer loads SPA → requests master/live manifest → hls.js player follows current edge.
  5. Cloudflare serves .m4s and .ts from nearest edge cache (origin/R2 hit only once per POP/region).
  6. Parallel WebSocket connection to chat service for synchronized interaction.

Notable aspects: Manifest freshness ensures all viewers stay on same timeline; cache hit ratio directly impacts cost and latency. Fallback slate activates on playout failure. Full cache policy, example configs (FFmpeg bash loop, Nginx RTMP HLS), and storage tables in adr/ADR-001.md.

(Additional scenarios: moderation events, scale spikes, failover covered in the ADR.)

Deployment View

CI/CD Pipeline Architecture

The deployment uses parallel workflows on each push to main: CI builds and pushes images while CD deploys latest images to the test lab. A dedicated release workflow handles semantic version tags. Current runtime target is the test lab (192.168.88.40). Production Hetzner VPS is planned but not yet provisioned — the same pipeline will deploy there by changing DOCKER_HOST.

flowchart LR
    subgraph "Gitea Actions"
        CI[CI Workflow<br/>Build + Push Images]
        CD[CD Workflow<br/>Pull + Deploy]
    end
    subgraph "Gitea Container Registry"
        REGISTRY[git.allucanget.biz<br/>playout:latest<br/>chat:latest<br/>nginx:latest]
    end
    subgraph "Test Lab Host (192.168.88.40) — Production Hetzner TBD"
        DOCKER[Docker Engine<br/>via TCP :2375]
        PLAYOUT[linearvoid-playout]
        CHAT[linearvoid-chat]
        NGINX[linearvoid-nginx]
        REDIS[linearvoid-redis]
        DB[linearvoid-db]
        PROM[linearvoid-prometheus]
        GRAF[linearvoid-grafana]
    end

    CI -->|push| REGISTRY
    REGISTRY -->|pull| CD
    CD -->|deploy via DOCKER_HOST| DOCKER
    DOCKER --> PLAYOUT
    DOCKER --> CHAT
    DOCKER --> NGINX
    DOCKER --> REDIS
    DOCKER --> DB
    DOCKER --> PROM
    DOCKER --> GRAF

Pipeline Sequence:

  1. CI Workflow (.gitea/workflows/ci.yml):
    • Triggers on push to main and pull_request
    • Runs test job: frontend lint/build and Go validation
    • Runs docker job (after test passes, only on main push):
      • Builds playout, chat, nginx images with Buildx
      • Pushes to Gitea Container Registry with main and latest tags
  2. CD Workflow (.gitea/workflows/cd.yml):
    • Triggers on push to main (parallel to CI), or manually via workflow_dispatch
    • Logs in remote Docker host to private registry
    • Generates .env from secrets
    • Seeds Prometheus and Grafana provisioning into named volumes
    • Deploys via docker compose -f docker-compose.prod.yml up -d --remove-orphans --force-recreate to remote host (192.168.88.40:2375 — test lab; switch to Hetzner IP for production)
    • Runs health checks with retry logic for playout initialization
  3. Release Workflow (.gitea/workflows/release.yml):
    • Triggers on semantic tags in *.*.* format (for example 1.0.9)
    • Builds and pushes versioned image tags (X.Y.Z, X.Y, latest)
    • Creates release entry in Gitea

Infrastructure Level 1

Overview Diagram: See Mermaid in Building Block section (VPS + Cloudflare R2/Pages/CDN).

Motivation: Low-cost US-East origin with global edge caching. Hetzner provides excellent price/performance and included traffic; Cloudflare eliminates egress fees.

Quality/Performance Features: High cache-hit ratio for video, low US latency, automatic scaling at edge, built-in WAF/DDoS protection.

Mapping of Building Blocks to Infrastructure:

  • Playout + Chat + Nginx → Hetzner Cloud CX22 (Ashburn, VA).
  • HLS storage, caching, static SPA → Cloudflare R2 + CDN + Pages (custom domains, enam location hint).
  • All public traffic routed through Cloudflare.

Infrastructure Level 2

Hetzner VPS (planned): Will run playout (FFmpeg/MediaMTX), chat backend, Nginx. Monitoring via dashboards + UptimeRobot. Backups to R2. Currently test lab (192.168.88.40) serves this role. Production Hetzner instance not yet provisioned — see Production Deployment section in DEPLOYMENT.md.

Cloudflare Layer:

  • R2 bucket for media.
  • Cache Rules (.m4s and .ts: Cache Everything, immutable, long TTL; .m3u8: short TTL/must-revalidate).
  • WAF, rate limiting, Tiered Cache.
  • Pages for SPA hosting.

Future evolution: Docker Compose, managed DB, secondary regions (detailed in subsequent ADRs).

Cross-cutting Concepts

Caching & Cost Optimization

Immutable segment naming + targeted Cloudflare Cache Rules + R2 ($0 egress) = near-zero viewer bandwidth cost. Long TTL on .m4s and .ts, short on live manifests. Monitor hit ratio and R2 Class B ops.

Video Normalization & Playout

All source assets must be pre-normalized before they are accepted into production playlists. This prevents decode resets, timestamp drift, and audio/video discontinuities during continuous concat playout.

Required Source Profile (Linear Void v1)

Field Required Value Notes
Container MP4 Single-video-asset input format
Video codec H.264/AVC (libx264) Baseline delivery codec for broad compatibility
Audio codec AAC-LC Stereo preferred
Resolution 1280x720 or 1920x1080 Keep single resolution per playlist block when possible
Framerate Constant 30 fps (-r 30) Variable frame rate content must be normalized
GOP / keyframe interval 2 seconds (-g 60 -keyint_min 60) Matches HLS segment boundaries better
Pixel format yuv420p Browser compatibility baseline
Audio sample rate 48 kHz Stable timing for concat playout
Audio bitrate 128k-192k Predictable quality envelope

Reference Normalization Command

ffmpeg -i input.mov \
  -c:v libx264 -preset medium -crf 21 \
  -r 30 -g 60 -keyint_min 60 -sc_threshold 0 \
  -pix_fmt yuv420p \
  -c:a aac -b:a 160k -ar 48000 -ac 2 \
  -movflags +faststart \
  output-normalized.mp4

Verification Gate (ffprobe)

  • Validate codec, fps, resolution, and keyframe cadence before upload acceptance.
  • Reject non-conforming files from admin ingest with actionable error feedback.
  • Keep verification logic aligned with runtime playout assumptions to preserve -c copy efficiency where possible.

Monitoring & Resilience

Linear Void uses a two-layer monitoring model:

  1. Application layer (chat + platform health) via Prometheus metrics and a lightweight Grafana dashboard.
  2. Infrastructure/runtime layer (Prometheus self-observability, TSDB, query engine, Go runtime) via a detailed Grafana dashboard.

Prometheus Baseline

  • Config file: monitoring/prometheus/prometheus.yml
  • Default scrape interval: 15s
  • Current scrape jobs (7 total):
    • chat -> chat:3000
    • playout -> playout:8081
    • nginx -> nginx-exporter:9113
    • redis -> redis-exporter:9121
    • postgres -> postgres-exporter:9187
    • node -> node-exporter:9100
    • prometheus -> localhost:9090

Grafana Dashboards

  • monitoring/grafana/dashboard.json (Linear Void app metrics: connected clients, messages, moderation actions, Redis reachability, admin sessions, scrape health).
  • monitoring/grafana/dashboards/dashboard.json (Prometheus operational visibility: status, uptime, CPU/memory, TSDB ingestion/churn/errors, HTTP latencies, query load, GC/scheduler).

Use the app dashboard for service-level behavior and the Prometheus overview dashboard for platform diagnostics and capacity planning.

Alerting Rules (Prometheus Rule Groups)

Rules are implemented in monitoring/prometheus/alerts.yml and loaded via rule_files from prometheus.yml.

Current rule set (8):

  • ChatDown
  • PlayoutDown
  • RedisDown
  • PlayoutSlateActive
  • HighChatLatency
  • PrometheusWALFailing
  • HighCPU
  • LowDiskSpace

Route warning/critical severities to the operations channel and keep runbook links attached to each alert definition.

Operational resilience baseline:

  • Emergency slate fallback for playout failure.
  • Auto-restart policies for playout/chat/nginx containers.
  • Weekly backup and restore drills for stateful data (data/chat, playlist metadata, Postgres).
  • Incident runbook with triage order: edge reachability -> origin health -> playout progression -> chat connectivity -> data-plane dependencies.

API Documentation Reference

Linear Void currently exposes three operational API surfaces. The canonical detailed contract should live in API.md, while this section provides the architecture-level map.

Chat WebSocket API

  • Endpoint: /ws
  • Transport: wss:// in production, ws:// locally
  • Query parameters:
    • username
    • room
  • Message envelope (typical):
    • room
    • username
    • text
    • type (public, private, system)
  • Responsibilities:
    • room fan-out
    • private DM channel routing
    • moderation command handling
    • rate limiting and presence integration

Playout and Delivery REST API

  • Health endpoints:
    • playout health (/health on playout service)
    • edge/origin health (/health via nginx)
  • Stream delivery:
    • HLS manifest and segment path namespace under /hls/
  • Responsibilities:
    • expose liveness for orchestration and monitoring
    • provide stable HLS path contract for viewer and CDN

Admin REST API

  • Auth/session:
    • /api/admin/login
    • /api/admin/logout
    • /api/admin/session
  • Operational/admin domains:
    • overview, analytics, users, moderation, schedule, playlist, media
  • Media upload domain:
    • /api/admin/media/uploads (init/resume)
    • /api/admin/media/uploads/{uploadId}/chunk
    • /api/admin/media/uploads/{uploadId}/complete
  • Responsibilities:
    • protected control plane for operator workflows
    • media metadata + upload orchestration
    • moderation/audit and system status management

Cloudflare Configuration Reference

Cloudflare is part of the production control plane, not only a CDN edge. The following configuration set is required to maintain cost, latency, and reliability targets.

DNS and Zone Routing

  • linearvoid.tv -> application origin (nginx)
  • www.linearvoid.tv -> redirect to apex
  • media.linearvoid.tv -> media domain for HLS/object delivery when split-domain mode is enabled

Cache Rules Baseline

  • HLS segments (.ts, .m4s):
    • Cache Everything
    • long TTL (immutable)
    • respect immutable semantics
  • Manifests (.m3u8):
    • short TTL (1-10s)
    • revalidation-friendly policy for timeline freshness

Edge Security and Controls

  • Enforce HTTPS/WSS and redirect HTTP to HTTPS.
  • Enable WAF baseline protections for common web exploits.
  • Apply rate limits for abusive request patterns (especially auth/admin paths).
  • Keep origin exposure minimized (prefer Cloudflare-only public ingress where feasible).

Worker and Header Policy

  • Use Worker logic for immutable cache headers and request normalization where cache rules alone are insufficient.
  • Keep CORS behavior explicit for HLS resources to support browser playback.

Operations and Validation

  • Validate cache behavior after deployment:
    • expected cache status progression (MISS -> HIT)
    • manifest freshness under active playout
  • Track edge metrics and adjust:
    • cache hit ratio
    • request volume and error rates
    • ToS/cost posture for sustained traffic profiles

Security & Moderation

Security is split between edge controls (Cloudflare), reverse proxy controls (nginx), and application controls (chat/admin backend).

Transport and Edge Protection

  • Public traffic uses HTTPS/WSS at the edge.
  • Cloudflare WAF and rate limiting are part of the baseline (see infrastructure/cloudflare/cache-rules.md).
  • Cloudflare Worker can enforce immutable cache headers for segment paths (infrastructure/cloudflare/worker.js).

Reverse Proxy Enforcement (nginx)

  • Admin UI and admin API paths are protected by auth_request before reaching backend handlers (/admin/, /api/admin/).
  • Unauthorized admin UI requests are redirected to /admin/login/.
  • Unauthorized admin API requests return JSON 401 from a dedicated location handler.
  • WebSocket upgrade headers are explicitly forwarded for /ws.

Admin Authentication and Session Security

  • Admin authentication is cookie-session based (linearvoid_admin_session).
  • Session cookie controls in backend include HttpOnly, SameSite=Strict, and Secure when request is HTTPS (derived from TLS or X-Forwarded-Proto).
  • Session expiration is enforced server-side.

Rate Limiting and Abuse Controls

  • Chat message rate limiting uses Redis INCR + EXPIRE windowing.
  • Current baseline in chat handler: >10 messages within 10s triggers a user-facing throttle response.
  • If Redis is unavailable, presence/rate-limit features degrade gracefully (warning logged; core chat remains available).

CORS and Cross-Origin Behavior

  • HLS endpoints exposed by nginx include explicit Access-Control-Allow-Origin: * for player compatibility.
  • Admin/authenticated routes are expected to run same-origin behind nginx auth gates.

Moderation and Privacy

  • Moderation commands (/mod, /mute, /ban) are handled in backend chat flow and persisted through chat storage paths.
  • Private DMs are application-level private channels, not end-to-end encrypted in current architecture.
  • Moderation and audit retention policies should be formalized in runbook/ops docs as part of production hardening.

Chat

WebSockets + Redis (presence, pub/sub, rate limits) + DB (history, audit). Public rooms + on-demand private chats. Scales independently of video.

Full details, cache rule table, bitrate/storage calculations, cost model, example configs, ToS notes, and implementation phases are in adr/ADR-001.md.

Architecture Decisions

The primary decisions are captured in the following consolidated ADRs (all located in adr/):

  • ADR-001: Overall architecture, HLS + Cloudflare R2 caching strategy, storage calculations, cost model, risks.
  • ADR-002: Chosen project folder/module structure (frontend, backend/playout, backend/chat, infrastructure, scripts, monitoring) and rationale.
  • ADR-003: Technology stack, programming language choices per module (TypeScript frontend, Go for chat/playout wrapper, Bash/Python scripts), Docker-first approach, and explicit documentation of all implicit assumptions.
  • ADR-004: Video admin web interface (upload, playlist management, schedule overrides, normalization enforcement).
  • ADR-005: Web admin interfaces for user management, moderation dashboard, analytics, and system configuration.
  • ADR-006: Data storage strategy (SQLite vs PostgreSQL for v1, migration path, admin interface capabilities).
  • ADR-007: Redis usage (exact use cases, TTL policies, persistence, fallback behavior, admin configuration).

These ADRs together form the canonical decision log.

Summary of Key Decisions (from ADR-001 through ADR-007):

  • Self-managed US-East playout (Hetzner Ashburn) with FFmpeg + MediaMTX/SRS → HLS → Cloudflare R2 + edge caching.
  • Module structure that directly reflects the arc42 building blocks and context diagram.
  • Go preferred for performance-critical services; TypeScript for frontend; Docker for all deployment.
  • Admin interfaces for video management, moderation, analytics, and system configuration using HTMX + Tailwind.
  • SQLite as primary database for v1 with PostgreSQL migration path if needed.
  • Redis for rate limiting, presence, pub/sub, session storage with defined TTL policies and fallback behavior.

Quality Requirements

See Quality Goals table in the Introduction section. Primary scenarios focus on cost (near-zero egress), availability (auto-restart + fallback), scalability (edge caching), and usability (synchronized player + responsive chat).

Performance Benchmarks and Capacity Planning

Baseline SLO Targets (v1)

  • Playback startup: target < 4s after first manifest request on warm cache.
  • Steady-state playback: no segment starvation during normal operation.
  • End-to-end HLS latency: expected 15-35s (standard HLS trade-off).
  • Chat delivery latency: p95 < 500ms inside same region under normal load.
  • Admin page responsiveness: p95 API response < 1s for common read operations.

Workload Assumptions

  • Primary audience in US; origin in US-East.
  • Edge cache serves most segment traffic; origin handles manifest churn and cache misses.
  • Chat traffic scales with concurrent viewers but remains low-bandwidth compared to video.

Initial Capacity Model

  • Video bitrate envelopes (from ADR-001):
    • 720p30: ~2.6-4.2 Mbps total with audio.
    • 1080p30: ~5.1-8.2 Mbps total with audio.
  • Storage planning for 48h rolling library:
    • 720p30: ~57-90 GB raw (allocate ~80-110 GB with headroom).
    • 1080p30: ~110-175 GB raw (allocate ~130-180 GB with headroom).
  • Cost behavior:
    • Viewer scale is mostly constrained by cache efficiency and request volume, not by direct origin egress.

Benchmark Plan

  • Playback path:
    • Measure time-to-first-frame and rebuffer events from multiple US regions.
    • Track manifest freshness and segment availability at edge.
  • Chat path:
    • Load-test concurrent websocket connections and message fan-out.
    • Measure p50/p95 send-to-receive latency and Redis fallback behavior.
  • Admin/API path:
    • Benchmark playlist/system endpoints for p95 latency and error rate.
  • Platform path:
    • Observe TSDB ingestion, query p99, and GC pause behavior in Grafana.

Capacity Triggers and Actions

  • Trigger: sustained viewer growth with rising cache-miss ratio.
    • Action: tighten cache policy, enable/verify Tiered Cache, review manifest TTL.
  • Trigger: chat p95 latency drift or websocket error growth.
    • Action: scale chat service, tune Redis, separate noisy moderation workloads.
  • Trigger: Prometheus WAL/compaction failures or query saturation.
    • Action: tune retention/cardinality, increase resources, split dashboards/queries.
  • Trigger: origin CPU/memory pressure on playout host.
    • Action: optimize FFmpeg profile, isolate services, add secondary origin.

Scaling Strategy

Scaling is designed to preserve cost efficiency first, then increase throughput and resilience in controlled phases.

Phase 1: Vertical + Cache-First (Current Baseline)

  • Single primary origin in US-East with strong Cloudflare edge caching.
  • Optimize segment immutability and manifest TTL before adding infrastructure.
  • Scale compute vertically when CPU/memory pressure appears on playout/chat host.

Phase 2: Service Separation

  • Split playout, chat, and observability workloads across dedicated nodes/VMs.
  • Isolate noisy workloads (analytics, heavy admin tasks) from latency-sensitive chat paths.
  • Introduce managed PostgreSQL/Redis or dedicated state nodes where operationally justified.

Phase 3: Multi-Origin Resilience

  • Add warm standby or active-secondary origin.
  • Use health-aware DNS/origin failover strategy.
  • Keep HLS path and cache contract stable so viewer impact during failover is minimized.

Phase 4: Regional and Throughput Expansion

  • Add additional origin region only when audience distribution and latency justify it.
  • Keep chat affinity strategy explicit (regional routing, state replication considerations).
  • Continuously evaluate cache miss economics vs additional origin complexity.

Scaling Triggers

  • Rising cache miss ratio with sustained traffic growth.
  • Chat p95/p99 latency drift or websocket error growth.
  • Persistent origin resource saturation despite optimization.
  • Increased blast-radius concerns from single-host dependency.

Disaster Recovery and Backup Procedures

Recovery Targets

  • RPO target: <= 24h for chat/moderation/admin metadata.
  • RTO target: <= 60 minutes for restoring baseline streaming + chat operations.

Backup Scope

  • Persistent service volumes:
    • Postgres data (postgres-data volume).
    • Redis append-only persistence (redis-data volume) where enabled.
    • Prometheus long-term TSDB (prometheus-data volume) if historical metrics retention is required.
    • Grafana state (grafana-data volume) if dashboards/users are managed in-instance.
  • Host-mounted state directories:
    • data/chat
    • data/playlist
    • data/media (authoritative source may be R2/object storage)
    • data/hls (optional; can be regenerated by playout)
  • Configuration and infra source of truth:
    • Compose files, nginx config, monitoring configs, and environment template.

Backup Cadence

  • Daily logical Postgres backups (plus weekly full snapshot).
  • Daily sync of chat/playlist state directories.
  • Weekly integrity check by restoring backups into an isolated environment.
  • Retention policy baseline:
    • Daily backups: 14 days
    • Weekly backups: 8 weeks
    • Monthly backups: 6 months

Recovery Runbook (Minimum)

  1. Restore infrastructure configuration (compose, nginx, env, monitoring config).
  2. Restore Postgres and Redis persisted state.
  3. Restore data/playlist and chat persistence directories.
  4. Rehydrate media from authoritative object storage if local media is missing.
  5. Start stack and validate in order:
    • playout health endpoint
    • chat health endpoint
    • nginx /health
    • HLS playlist availability
    • websocket chat connectivity
  6. Verify admin authentication and moderation audit visibility.
  7. Announce recovery and continue elevated monitoring for at least one playlist cycle.

Failure Modes and Strategy

  • Single-host loss: redeploy stack on standby host, restore data backups, repoint DNS/edge origin.
  • Storage corruption: restore last known-good snapshot and replay latest logical backups.
  • Partial service failure: replace only impacted service and restore scoped data (for example Postgres only).

Document and rehearse this runbook quarterly; recovery that is not exercised is not reliable.

Risks and Technical Debts

  • Cloudflare ToS risk for heavy video on free tier (use R2 path; fallback to BunnyCDN if challenged).
  • HLS latency (15–35s) may affect perceived chat synchronization.
  • Single VPS as SPOF (mitigate with DNS failover in Phase 3).
  • Upfront effort for video normalization and cache tuning.
  • Moderation/abuse prevention operational load at scale.
  • R2 request costs on cache misses (optimize with Tiered Cache; measure in prod).
  • Deferred items: LL-HLS, ABR, E2EE, paid access controls, full Docker Compose.

Technical Debt: Detailed service configs, CI/CD, and advanced resilience are left for implementation ADRs.

Glossary

Term Definition
HLS HTTP Live Streaming – protocol using .m3u8 manifests and media segments (fMP4).
Immutable Segment HLS chunk with unique, unchanging name enabling long-lived caching.
R2 Cloudflare object storage ($0 egress, ideal origin for cached HLS).
Edge Cache / POP Cloudflare points-of-presence that cache and serve content close to viewers.
Playout Continuous generation of a scheduled linear program from a playlist.
Normalization Pre-processing videos to consistent codec/fps/keyframe settings for seamless playback.
Cache Rule Cloudflare configuration defining caching behavior, TTL, and headers per URL pattern.
SPOF Single Point of Failure (current single VPS origin).
Tiered Cache Cloudflare feature reducing origin requests by checking upper-tier caches first.

References: Cloudflare R2/Cache docs, MediaMTX/SRS/FFmpeg HLS guides, Hetzner pricing, hls.js documentation, standard video bitrate calculators. All original root ADR-001-*.md files have been consolidated and can be archived/deleted.

Reference in new issue View Git Blame Copy Permalink