zwitschi
dafcec1303
- arc42 template fully populated in docs/architecture/README.md with Mermaid diagrams, quality goals, deployment view, cross-cutting concepts, risks and glossary - Consolidated ADR-001 (merged all previous root ADR drafts) - ADR-002: Module/folder structure decision and rationale - ADR-003: Technology stack, language choices per module (TypeScript, Go, Bash/Python), Docker-first approach, and explicit implicit assumptions - Updated root README.md with comprehensive Project Structure section and links to all ADRs - Created .github/instructions/TODO.md with detailed tracking (architecture phase marked complete) - Folder structure created matching ADR-002 (frontend/, backend/playout/, backend/chat/, backend/shared/, infrastructure/, scripts/, monitoring/) - Git repository initialized with main branch This establishes the foundation defined in the arc42 documentation and the three ADRs.
171 lines
9.5 KiB
Markdown
171 lines
9.5 KiB
Markdown
# ADR-001: Cost-Efficient Self-Hosted 24/7 Synchronized Linear Video Streaming with Custom Chat (US-Centric, Cloudflare + R2 Optimized)
|
||
|
||
**Status:** Accepted
|
||
**Date:** 2026-08-02
|
||
**Authors:** Synthesized from Grok, GPT-Terra, DeepSeek, Gemini (research-assisted)
|
||
**Decision Owners:** Product & Engineering
|
||
**Scope:** One 24/7 linear video channel, custom viewer website, public/private chat, playlist-driven playout. Related ADRs: authentication, moderation, payments, multi-channel.
|
||
|
||
## Context
|
||
|
||
We require a completely independent 24/7 linear video channel that broadcasts a continuous playlist of self-owned pre-recorded videos. Viewers consume it via a lightweight custom HTML/CSS/JavaScript single-page application (no reliance on YouTube, Twitch, Vimeo, or paid Cloudflare Stream tiers). The experience must include:
|
||
|
||
- True (or near-true) synchronized playback — all viewers see the same program timeline.
|
||
- Embedded custom chat with public rooms, private 1:1/direct messages, presence, moderation (mute/ban/report), rate limiting.
|
||
- “Now playing” schedule/metadata.
|
||
- Full control over source media, frontend, schedule, and chat data.
|
||
- Primary audience and operational origin in the **United States** (East/Central preferred for uploads and low latency).
|
||
|
||
**Key distinctions**:
|
||
|
||
- **Playout**: Continuous scheduled program from playlist.
|
||
- **Packaging**: FFmpeg → standard HLS (`.m3u8` + `.ts` segments, 4–6s duration).
|
||
- **Delivery**: Immutable segments served via Cloudflare edge caching + R2 (zero egress cost).
|
||
- **Interaction**: Separate lightweight WebSocket service for chat.
|
||
|
||
Bandwidth (not storage or compute) is the dominant cost driver. A single viewer at ~4 Mbps consumes ~1.3–1.8 TB/month; 100 concurrent viewers = ~130–180 TB/month. Direct VPS delivery is only viable for PoC/small audiences.
|
||
|
||
## Decision Drivers
|
||
|
||
1. Minimize monthly cost (target <$25 for moderate scale).
|
||
2. Full independence and ownership (no video platform lock-in).
|
||
3. Strong US performance and low latency.
|
||
4. True/near-true synchronization via live-style HLS (not pure VOD).
|
||
5. Aggressive immutable caching of `.ts` segments to achieve near-zero egress.
|
||
6. Browser-native playback (`hls.js` + native HLS).
|
||
7. Lightweight, maintainable stack with modest operational complexity.
|
||
8. Support for custom multi-room + private chat with persistence and moderation.
|
||
9. Scalability from tens to thousands of concurrent viewers without linear cost growth.
|
||
10. Compliance with Cloudflare ToS (use R2 as origin for large video files).
|
||
|
||
## Decision
|
||
|
||
Adopt a **self-managed US-East playout + Cloudflare R2 + edge-cached HLS** architecture:
|
||
|
||
### 1. Video Pipeline & Playout (US-East VPS)
|
||
|
||
- **Location**: Hetzner Cloud CX22 (or equivalent) in **Ashburn, VA** (~$4–8/mo, 2 vCPU/4 GB, 40 GB NVMe, 20 TB included traffic). Excellent peering.
|
||
- **Tools**:
|
||
- FFmpeg (concat demuxer + `while true` loop for auto-restart) for playlist playout and normalization (`-c copy` where possible, CRF encoding).
|
||
- MediaMTX (preferred), SRS, or Nginx-RTMP as lightweight media server/packager for continuous HLS output.
|
||
- Sliding window HLS: 4–6s segments, playlist length ~24–60s.
|
||
- **Source Normalization** (mandatory before scheduling):
|
||
- Container: MP4
|
||
- Video: H.264/AVC, yuv420p, constant 30 fps, keyframe interval 2s
|
||
- Audio: AAC-LC, 48 kHz, stereo, 128–192 kbps
|
||
- Bitrate targets: 720p30 (2.5–5 Mbps), 1080p30 (5–8 Mbps)
|
||
- Dashboard for playlist management, current file, health.
|
||
|
||
### 2. Storage & Delivery (Cloudflare R2 + CDN)
|
||
|
||
- HLS segments (`.ts`) and manifests uploaded to **Cloudflare R2** (Eastern North America hint `enam`).
|
||
- Custom domain (`media.example.com`) fronted by Cloudflare with **Cache Rules**:
|
||
- `*.ts` (and init `.mp4` if using fMP4 later): `Cache Everything`, immutable, Edge TTL 24h–1yr (`Cache-Control: public, max-age=31536000, immutable`).
|
||
- `*.m3u8`: Short TTL (1–10s) or `must-revalidate` (`Cache-Control: public, max-age=1..10, must-revalidate`).
|
||
- Master playlist: TTL ~30s.
|
||
- Tiered Cache enabled where possible.
|
||
- **R2 as origin** satisfies ToS for large video; $0 egress, free 10 GB, then ~$0.015/GB-mo (Standard).
|
||
- Viewer player hits Cloudflare edge → vast majority of traffic served from cache (origin/R2 hit only once per region).
|
||
|
||
### 3. Frontend
|
||
|
||
- Static SPA (HTML/CSS/JS + Tailwind + Alpine.js or vanilla) hosted on Cloudflare Pages or same VPS/Nginx.
|
||
- Video player: `hls.js` (Chromium/Firefox) or native HLS (Safari). Single live `master.m3u8`.
|
||
- Embedded chat sidebar with room switcher, private DMs, now-playing overlay.
|
||
|
||
### 4. Custom Chat
|
||
|
||
- Lightweight backend (Go, Node.js + Socket.io, or Elixir) on same VPS.
|
||
- WebSockets (`wss://`), Redis (presence, pub/sub, rate limits), PostgreSQL/SQLite (message history, moderation log, users).
|
||
- Features: public rooms, private 1:1, presence, rate limiting, mute/timeout/ban/report, basic search.
|
||
- Persistence with retention policy. Moderation audit log. No E2EE by default (separate ADR if required).
|
||
|
||
### 5. Infrastructure Summary & Cost Model
|
||
|
||
- **Primary**: Hetzner Ashburn VPS (playout + chat + Nginx) + Cloudflare R2 + Pages.
|
||
- **Monitoring**: MediaMTX/SRS dashboard, UptimeRobot, systemd logs, Cloudflare analytics.
|
||
- **Backup**: Weekly R2/S3-compatible snapshots of playlist, DB, config.
|
||
- **Fallback**: Emergency slate/loop playlist, automated restart.
|
||
- **Monthly Cost (early production, moderate US audience)**: $5–25 (VPS $4–8 + R2 $1–3 + negligible requests). Scales to hundreds/thousands of viewers with near-zero additional egress.
|
||
|
||
**Initial Targets**:
|
||
|
||
- Protocol: Standard HLS (`.ts` segments). Defer LL-HLS/CMAF/WebRTC.
|
||
- Max rendition: 720p30 initially (upgrade to 1080p30 or ABR later).
|
||
- Latency: 15–35s (acceptable for linear programming/chat).
|
||
|
||
## Storage for 48 Hours of Source Footage
|
||
|
||
Formula: `GB ≈ (bitrate_Mbps × duration_seconds) / 8 / 1024` (+5–10% overhead).
|
||
|
||
| Resolution | Video Bitrate | Total w/ Audio | 48h Raw | Recommended Allocation | R2 Cost/mo (Standard) |
|
||
| ---------- | ------------- | -------------- | ---------- | ---------------------- | --------------------- |
|
||
| 720p30 | 2.5–4 Mbps | ~2.6–4.2 Mbps | 57–90 GB | ~80–110 GB | ~$1.20–$1.65 |
|
||
| 1080p30 | 5–8 Mbps | ~5.1–8.2 Mbps | 110–175 GB | ~130–180 GB | ~$1.95–$2.70 |
|
||
|
||
Plan 2–3× headroom for variants, backups, artwork, processing. Use R2 Infrequent Access if latency acceptable.
|
||
|
||
## Rationale
|
||
|
||
- **Hetzner Ashburn + Cloudflare R2** delivers unmatched price/performance for US traffic. R2’s $0 egress + edge caching eliminates the dominant bandwidth cost.
|
||
- HLS is the most compatible, cache-friendly, operationally simple protocol for one-to-many linear broadcast.
|
||
- FFmpeg + MediaMTX/SRS provides reliable playlist-driven continuous playout with low CPU.
|
||
- Immutable segment naming + targeted Cache Rules ensure high hit ratios while keeping live manifests fresh.
|
||
- Lightweight static frontend + separate lightweight chat service keeps complexity low.
|
||
- Starts simple (single VPS) and evolves (ABR, secondary origins, managed DBs, LL-HLS) without breaking changes.
|
||
|
||
This hybrid synthesizes the strongest elements from all evaluated ADRs while resolving minor variances (Hetzner vs generic VPS, direct origin vs R2, SQLite vs PostgreSQL+Redis, exact bitrates).
|
||
|
||
## Considered Alternatives
|
||
|
||
- **Pure VPS origin (no R2/CDN)**: Rejected for production (20 TB cap limits ~12–20 concurrent HD viewers).
|
||
- **Managed platforms (YouTube/Twitch/Cloudflare Stream)**: Rejected (loss of control, higher cost at scale, ToS restrictions).
|
||
- **WebRTC / P2P**: High complexity, poor caching, not suited for passive linear broadcast.
|
||
- **Owncast**: Good for turnkey but less flexible for custom playlist/schedule/chat.
|
||
- **LL-HLS or CMAF initially**: Deferred (added complexity; standard HLS sufficient for v1).
|
||
|
||
## Consequences
|
||
|
||
**Positive**:
|
||
|
||
- Near-zero egress cost → unlimited viewer scaling.
|
||
- Full ownership, US performance, synchronized linear experience.
|
||
- Extremely low TCO (~$5–25/mo).
|
||
- Cacheable, maintainable, evolvable architecture.
|
||
|
||
**Negative / Risks & Technical Debt**:
|
||
|
||
- Cloudflare ToS scrutiny for heavy video (mitigated by R2 path; monitor & have BunnyCDN fallback).
|
||
- HLS latency (15–35s) affects real-time chat feel.
|
||
- Single-origin SPOF (mitigate with DNS failover/secondary region later).
|
||
- Video normalization and cache-rule tuning required upfront.
|
||
- Chat moderation and abuse prevention operational burden.
|
||
- R2 request costs on cache misses (measure in production; Tiered Cache helps).
|
||
- No built-in E2EE for private chats (future ADR).
|
||
|
||
**Operational Requirements**:
|
||
|
||
- Monitor playout health, segment generation, cache hit ratio, R2 ops, WebSocket connections.
|
||
- Maintain fallback slate, emergency playlist, backup/restore procedures.
|
||
- Strict CORS, HTTPS, WAF, rate limiting, authenticated admin paths.
|
||
|
||
## Implementation Plan
|
||
|
||
**Phase 1 (Prototype)**: Hetzner VPS, normalized playlist, FFmpeg+MediaMTX → HLS to R2, basic hls.js page + Socket.io chat, cache rules, load test.
|
||
|
||
**Phase 2 (MVP)**: Scheduler/metadata, PostgreSQL+Redis, full moderation, dashboards/alerts, emergency fallback.
|
||
|
||
**Phase 3 (Scale)**: ABR, multi-region, LL-HLS evaluation, paid access controls.
|
||
|
||
## References
|
||
|
||
- Cloudflare R2 pricing, custom domains, cache rules, Tiered Cache.
|
||
- MediaMTX / SRS / FFmpeg HLS documentation.
|
||
- Hetzner Cloud Ashburn pricing (2026).
|
||
- hls.js, Video.js, standard HLS best practices.
|
||
- Video bitrate/storage calculators.
|
||
|
||
---
|
||
|
||
_This consolidated ADR merges all prior drafts. Future ADRs will detail Docker Compose, exact cache rules, playlist normalization scripts, chat schema, monitoring, and access control._
|