zwitschi/linearvoid
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 9 Wiki Activity

Add comprehensive documentation for Linear Void project

Browse Source 
- Created README.md in docs folder outlining project structure and documentation links.
- Added SECURITY.md detailing the security model, threat model, authentication, transport security, network isolation, and incident response procedures.
- Introduced SETUP.md providing a detailed setup guide for local and production environments, including prerequisites, environment setup, and Gitea integration.
- Added STREAMING.md outlining the streaming pipeline, including production flow, source-media contract, playout and HLS packaging, cache contract, player behavior, and monitoring.
- Updated architecture documentation to include links to new SETUP.md and STREAMING.md files for better navigation.
This commit is contained in:
zwitschi committed 2026-08-08 14:52:19 +02:00
1 parent 6b9830de05
commit 926a663c23
10 files changed
+54 -13
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/API.md
+307
View File
@@ -0,0 +1,307 @@
# API Reference
This document describes the currently implemented API surfaces in Linear Void.
## Base URLs
- Public app and admin: <https://linearvoid.tv>
- WebSocket endpoint: <wss://linearvoid.tv/ws>
- Local fallback: <http://localhost> and <ws://localhost/ws>
## Authentication Model
Admin REST endpoints use a cookie session.
- Cookie name: linearvoid_admin_session
- Cookie attributes: HttpOnly, SameSite=Strict, Secure when request is HTTPS (or X-Forwarded-Proto=https)
- Session TTL: controlled by ADMIN_SESSION_TTL_HOURS (default 12h)
## Chat WebSocket API
### Connect
Endpoint:
- GET /ws?username={name}&room={room}
Client sends JSON messages over the socket.
### Message Envelope (client -> server)
```json
{
"room": "main",
"username": "viewer123",
"text": "hello world",
"type": "public"
}
```
Notes:
- type is usually public for room chat and private for DM room routing.
- Private room names are formatted like dm:userA-userB (sorted usernames).
- Moderation/system events are sent as type=system messages.
### Message Envelope (server -> client)
```json
{
"room": "main",
"username": "viewer123",
"text": "hello world",
"type": "public",
"timestamp": "2026-08-04T12:34:56Z"
}
```
## Admin Session API
### Login
- POST /api/admin/login
- Auth: none
Request:
```json
{
"username": "admin",
"password": "your-password"
}
```
Success response:
```json
{
"authenticated": true,
"username": "admin"
}
```
### Logout
- POST /api/admin/logout
- Auth: optional (clears cookie either way)
Response:
```json
{
"authenticated": false
}
```
### Session Check
- GET /api/admin/session
- Auth: cookie required
Success response:
```json
{
"authenticated": true,
"username": "admin"
}
```
Failure:
- 401 unauthorized
## Admin Data API
All endpoints below require admin authentication.
### Overview
- GET /api/admin/overview
Returns aggregate counters and runtime/system snapshot fields.
### Playlist
- GET /api/admin/playlist
- PUT /api/admin/playlist
PUT request:
```json
{
"content": "file '/app/media/video1.mp4'\nfile '/app/media/video2.mp4'\n"
}
```
### Schedule
- GET /api/admin/schedule
Returns parsed playlist entries plus matched media metadata/runtime.
### Moderation
- GET /api/admin/moderation/logs?limit=50
- POST /api/admin/moderation/actions
Action request:
```json
{
"action": "mute",
"target": "username-or-id",
"moderator": "admin",
"reason": "spam",
"room": "main"
}
```
### Analytics
- GET /api/admin/analytics
Returns totals (messages, moderation actions, rooms) and top room summary.
### System
- GET /api/admin/system
Returns service health snapshot for chat/playout/redis/database and config metadata.
### Active Users
- GET /api/admin/users
Returns currently discovered presence users (room + username) when Redis is available.
## Media Library API
### Media Collection
- GET /api/admin/media
- POST /api/admin/media (multipart upload, legacy path)
Multipart form field name: file
### Media Item
- GET /api/admin/media/{id}
- PUT /api/admin/media/{id}
- DELETE /api/admin/media/{id}
PUT request:
```json
{
"status": "ready",
"title": "My Clip",
"tags": "music,live"
}
```
## Resumable Upload API
Maximum file size: 2GB
Default chunk size: 8MB
### Initialize or Resume Session
- POST /api/admin/media/uploads
Request:
```json
{
"uploadId": "optional-existing-id",
"fileName": "clip.mp4",
"fileSize": 123456789,
"mimeType": "video/mp4",
"chunkSize": 8388608,
"totalChunks": 15
}
```
Response:
```json
{
"uploadId": "hex-token",
"fileName": "clip.mp4",
"fileSize": 123456789,
"chunkSize": 8388608,
"totalChunks": 15,
"uploadedChunks": 3,
"progress": 20
}
```
### Check Upload Status
- GET /api/admin/media/uploads/{uploadId}
Returns same status envelope as initialize.
### Upload Chunk
- PUT /api/admin/media/uploads/{uploadId}/chunk?index={chunkIndex}
- Content-Type: application/octet-stream
Response returns updated upload status envelope.
### Complete Upload
- POST /api/admin/media/uploads/{uploadId}/complete
On success returns admin media listing payload (files + generatedAt).
## Metrics and Health
### Chat metrics
- GET /metrics
Prometheus exposition includes connected clients, stored messages, moderation actions, Redis indicators, active admin sessions, and build info.
### Health
- GET /health on chat service
- GET /health on playout service
- GET /health on nginx reverse proxy
## HTTP Status Conventions
| Code | Meaning | When |
| ---- | --------------------- | ----------------------------------------------------------- |
| 200 | OK | Successful GET, PUT, or action |
| 201 | Created | Successful POST (upload init, override created) |
| 204 | No Content | Successful DELETE, CORS preflight |
| 400 | Bad Request | Malformed JSON, missing required fields, invalid input |
| 401 | Unauthorized | Missing or expired admin session |
| 403 | Forbidden | Valid session but insufficient permissions |
| 404 | Not Found | Resource doesn't exist |
| 405 | Method Not Allowed | Wrong HTTP method for endpoint |
| 409 | Conflict | Schedule override overlap, upload ID mismatch |
| 413 | Payload Too Large | Upload exceeds 2 GB limit |
| 422 | Unprocessable | File fails normalization check (codec, fps, resolution) |
| 429 | Too Many Requests | Rate limit exceeded (chat messages, WS connections, login) |
| 500 | Internal Server Error | Unexpected server failure |
| 503 | Service Unavailable | Redis unavailable for user actions, playout health degraded |
## Rate Limiting
All limits are per-IP (from `X-Forwarded-For` or `RemoteAddr`) unless noted.
| Resource | Limit | Window | Response |
| --------------------- | ---------------------------------- | ---------------------------------------- | ------------------------------------------------------ |
| Chat messages | `CHAT_RATE_LIMIT_MAX` (default 10) | `CHAT_RATE_LIMIT_WINDOW_S` (default 10s) | 429 + `{"type":"system","text":"Rate limit exceeded"}` |
| WebSocket connections | 5 | 60s | 429 + `"too many connections from this IP"` |
| Admin login | 5 | 5min | 429 + `"too many login attempts, try again later"` |
Rate limit headers on applicable responses:
- `X-RateLimit-Limit`: maximum requests per window
- `Retry-After`: seconds until window resets
- 413: payload too large
- 500: internal server error
docs/DEPLOYMENT.md
+284
View File
@@ -0,0 +1,284 @@
# Deployment Guide
## Table of Contents
- [Deployment Guide](#deployment-guide)
- [Table of Contents](#table-of-contents)
- [Environments](#environments)
- [Production Target (Hetzner)](#production-target-hetzner)
- [Planned Infrastructure](#planned-infrastructure)
- [Cutover Plan](#cutover-plan)
- [Firewall / Security Group](#firewall--security-group)
- [Pipeline Readiness](#pipeline-readiness)
- [Target Topology](#target-topology)
- [Prerequisites](#prerequisites)
- [Environment File](#environment-file)
- [Pull-and-Run Deployment](#pull-and-run-deployment)
- [First-Time or Recovery Provisioning](#first-time-or-recovery-provisioning)
- [Health Verification](#health-verification)
- [Zero/Low-Downtime Update Pattern](#zerolow-downtime-update-pattern)
- [Rollback](#rollback)
- [Operational Tasks](#operational-tasks)
- [View logs](#view-logs)
- [Restart one service](#restart-one-service)
- [Stop stack](#stop-stack)
- [Keep volumes (recommended) vs remove volumes (destructive)](#keep-volumes-recommended-vs-remove-volumes-destructive)
- [Nginx and Proxy Considerations](#nginx-and-proxy-considerations)
- [Security Baseline](#security-baseline)
- [Common Deployment Pitfalls](#common-deployment-pitfalls)
- [Troubleshooting](#troubleshooting)
## Environments
Linear Void runs in two separate environments:
| Environment | Host | Docker Target | Purpose |
| -------------- | -------------------------- | -------------------------- | ---------------------------------------------- |
| **Test Lab** | `192.168.88.40` (internal) | `tcp://192.168.88.40:2375` | Integration testing, staging, CI/CD validation |
| **Production** | Hetzner VPS (TBD) | `tcp://<hetzner-ip>:2375` | Public linearvoid.tv serving real viewers |
**Current state (2026-08-06):** Test lab is active and receives automated CD deploys. Production Hetzner VPS is **not yet provisioned**. The CD workflow (`cd.yml`) and `docker-compose.prod.yml` currently target the test lab. Switching to production requires only changing `DOCKER_HOST` — see [Production Deployment](#production-target-hetzner) below.
## Production Target (Hetzner)
**Status: Not provisioned.** The production deployment plan is documented and the pipeline is ready — only the VPS needs to be created.
### Planned Infrastructure
| Resource | Spec |
| -------------- | ------------------------------------------------ |
| Provider | Hetzner Cloud |
| Region | Ashburn, VA (us-east) |
| Instance | CX22 (2 vCPU, 4 GB RAM, 40 GB SSD) or equivalent |
| OS | Ubuntu 24.04 LTS or Debian 12 |
| Bandwidth | 20 TB included traffic/month |
| Estimated cost | ~€7–10/month base |
### Cutover Plan
1. Provision Hetzner VPS (see [Production Deployment](#production-target-hetzner) section)
2. Add production secrets to Gitea (`CI_PROD_*` prefix)
3. Update `DOCKER_HOST` in CD workflow to Hetzner public IP
4. Trigger deploy → validate health checks
5. Update Cloudflare A record `linearvoid.tv` → Hetzner IP
6. Enable HTTPS, WAF, cache rules on production zone
7. Decommission test lab as public target
### Firewall / Security Group
| Port | Source | Purpose |
| ---- | --------------------- | ------------------------------ |
| 22 | Admin IPs | SSH management |
| 80 | 0.0.0.0/0 | HTTP (redirect to HTTPS) |
| 443 | 0.0.0.0/0 | HTTPS (nginx) |
| 2375 | Gitea runner IP only | Docker remote API (TLS) |
| 9090 | Admin IPs or internal | Prometheus (or tunnel via SSH) |
| 3001 | Admin IPs or internal | Grafana (or tunnel via SSH) |
### Pipeline Readiness
Three Gitea Actions workflows drive the full pipeline:
| Workflow | Trigger | Purpose |
| ------------- | ---------------- | ---------------------------------------------------------------------------------------- |
| `ci.yml` | Push to `main` | Lint, build frontend, vet Go, build & push Docker images (`:latest`, `:main`) |
| `cd.yml` | Push to `main` | Deploy to test lab (`192.168.88.40:2375`) — runs **in parallel** with CI |
| `release.yml` | Tag push `*.*.*` | Build & push versioned Docker images (`:X.Y.Z`, `:X.Y`, `:latest`), create Gitea Release |
CI and CD run simultaneously on every push to `main`. The deploy pulls whatever `:latest` images are available in the registry at that moment.
The CD workflow (`cd.yml`) uses `DOCKER_HOST` to target the deployment host. To switch targets:
```bash
# Test lab (current)
DOCKER_HOST=tcp://192.168.88.40:2375
# Production (when provisioned)
DOCKER_HOST=tcp://<hetzner-public-ip>:2375
```
No code changes needed in the pipeline. See `DEPLOYMENT.md` Production Deployment section for full checklist.
## Target Topology
- Host: Linux server with Docker Engine
- Reverse proxy: nginx container (linearvoid-nginx)
- Core services: playout, chat, redis, postgres
- Observability: prometheus, grafana
- CDN/edge: Cloudflare
## Prerequisites
- Docker Engine + Docker Compose plugin
- Network access to git.allucanget.biz
- A valid .env file with production values
- DNS configured for linearvoid.tv (and media.linearvoid.tv if split media domain is used)
## Environment File
Use .env with at least:
- CI_REGISTRY_URL
- CI_REGISTRY_NAMESPACE
- CHAT_PORT
- NGINX_PORT
- DATABASE_URL (if overridden)
- REDIS_URL (if overridden)
- ADMIN_USERNAME
- ADMIN_PASSWORD
- VITE_PUBLIC_APP_URL
- VITE_PUBLIC_WS_URL
- VITE_PUBLIC_HLS_URL
- Cloudflare/R2 variables when media sync is enabled
See .env.example and SETUP.md for full variable reference.
## Pull-and-Run Deployment
Images are built and pushed automatically by the CI and release workflows (see [CONTRIBUTING.md](../CONTRIBUTING.md#versioning-and-releases) for the release procedure).
Every push to `main` triggers CI (builds `:main` and `:latest` tags) and CD (deploys `:latest` to test lab) in parallel. Semantic version tags (`*.*.*`, e.g. `1.0.9`) trigger the release workflow which builds `:X.Y.Z`, `:X.Y`, and `:latest` tags and creates a Gitea Release.
From repository root:
```bash
docker login git.allucanget.biz
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d --remove-orphans
```
## First-Time or Recovery Provisioning
1. Ensure host paths exist:
```bash
mkdir -p data/chat data/playlist data/media data/hls
```
1. Confirm volumes and networks:
```bash
docker volume ls
docker network ls
```
1. Start stack:
```bash
docker compose -f docker-compose.prod.yml up -d
```
## Health Verification
After startup, check service status:
```bash
docker compose -f docker-compose.prod.yml ps
```
Endpoint checks:
- App: <http://{host}:8080/>
- Nginx health: <http://{host}:8080/health>
- Chat health: <http://{host}:3000/health>
- Playout health: <http://{host}:8081/health>
- Prometheus: <http://{host}:9090>
- Grafana: <http://{host}:3001>
Domain checks:
- <https://linearvoid.tv>
- <https://linearvoid.tv/admin>
- <https://linearvoid.tv/metrics>
## Zero/Low-Downtime Update Pattern
1. Pull newest images.
1. Recreate changed services only:
```bash
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d nginx chat playout
```
1. Re-run health checks.
## Rollback
If new images are unhealthy:
1. Pin previous tags in docker-compose.prod.yml.
2. Pull pinned tags.
3. Restart impacted services.
```bash
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d nginx chat playout
```
## Operational Tasks
### View logs
```bash
docker compose -f docker-compose.prod.yml logs -f nginx chat playout
```
### Restart one service
```bash
docker compose -f docker-compose.prod.yml restart chat
```
### Stop stack
```bash
docker compose -f docker-compose.prod.yml down
```
### Keep volumes (recommended) vs remove volumes (destructive)
```bash
# keep data
docker compose -f docker-compose.prod.yml down
# destructive (remove data)
docker compose -f docker-compose.prod.yml down -v
```
## Nginx and Proxy Considerations
- If an external proxy (for example Nginx Proxy Manager) fronts the stack, align request body limits with internal nginx.
- For resumable admin uploads, ensure client_max_body_size supports chunk requests in both proxy layers.
## Security Baseline
- Use strong ADMIN_PASSWORD and rotate regularly.
- Keep Docker host access restricted.
- Prefer HTTPS/WSS through Cloudflare.
- Restrict exposed ports if only proxy access is needed.
## Common Deployment Pitfalls
- Missing frontend build artifacts in local-only compose workflows.
- Wrong registry credentials or missing login.
- Misaligned domain env vars (app/ws/hls) causing mixed-origin errors.
## Troubleshooting
| Symptom | Check | Fix |
| -------------------------- | --------------------------------------------- | ------------------------------------------------------------- |
| Service not starting | `docker logs linearvoid-<svc> --tail 50` | Check `.env` vars, volume mounts, port conflicts |
| Playout stall / slate mode | `curl http://<host>:8081/health` — mode=slate | Check playlist file, media paths, disk space |
| Chat not connecting | `curl http://<host>:3000/health` | Check Redis: `docker exec linearvoid-redis redis-cli ping` |
| Admin 401 loop | Cookie not sent | Verify `X-Forwarded-Proto: https`, nginx auth_request headers |
| Upload fails (413) | File too large | Check nginx `client_max_body_size` (should be 64m+) |
| HLS not playing | `curl http://<host>/hls/playlist.m3u8` | Check playout health, HLS dir permissions |
| Prometheus no targets | `curl http://<host>:9090/targets` | Verify container network, scrape config |
| Grafana no data | Check datasource | Verify Prometheus URL `http://prometheus:9090` provisioned |
- Proxy-layer body size limits returning 413 on upload.
- Health checks failing before service warmup is complete.
docs/DEVELOPMENT.md
+91
View File
@@ -0,0 +1,91 @@
# DEVELOPMENT.md — Linear Void Remaining Tasks
## Test Fixtures (BLOCKED — requires ffmpeg)
- [ ] Create test media fixtures in `backend/playout/testdata/`:
- `valid_720p30.mp4` — small (1-2s) H.264/AAC compliant file
- `invalid_h265.mp4` — wrong codec
- `invalid_24fps.mp4` — wrong framerate
- Generate via `ffmpeg` commands documented in test file headers
- [ ] Create test playlist fixtures in `backend/playout/testdata/`:
- `valid_playlist.txt`, `empty_playlist.txt`, `comments_only.txt`, `mixed_valid_invalid.txt`
## User Frontend Tests
- [ ] Video player behavior (play, pause, seek, buffering, error handling)
- [ ] Chat functionality (send/receive messages, presence, rate limiting, moderation)
- [ ] State management and data flow
- [ ] Responsive design and layout across different screen sizes
- [ ] Accessibility testing (keyboard navigation, screen reader support)
- [ ] Integration with backend APIs (mocking API responses for testing)
- [ ] Test coverage for all critical frontend components and interactions
- [ ] Add code coverage reporting and enforce minimum coverage threshold in CI/CD pipeline
- [ ] Document test cases and expected outcomes
## Admin Frontend Tests
- [ ] Playlist management UI (add/remove videos, reorder, normalize)
- [ ] User moderation tools (mute/ban/report users, view moderation logs)
- [ ] Metrics and monitoring dashboards (Prometheus/Grafana integration)
- [ ] State management and data flow
- [ ] Responsive design and layout across different screen sizes
- [ ] Accessibility testing (keyboard navigation, screen reader support)
- [ ] Integration with backend APIs (mocking API responses for testing)
- [ ] Test coverage for all critical admin UI components and interactions
- [ ] Add code coverage reporting and enforce minimum coverage threshold in CI/CD pipeline
- [ ] Document test cases and expected outcomes
## Cloudflare R2 Integration Test
- [ ] Upload test HLS segments and manifests to R2 bucket
- [ ] Verify correct cache headers are applied (immutable segments, TTLs)
- [ ] Test retrieval of segments and manifests from R2 via Cloudflare edge
- [ ] Validate cache hit/miss behavior and performance metrics
- [ ] Ensure integration test can run in CI/CD pipeline with proper mocking/stubbing of R2 API
- [ ] Document test cases and expected outcomes
## Cloudflare Configuration (manual — Cloudflare dashboard)
> Evidence tracker: `docs/operations/cloudflare-uptimerobot-checklist.md`
- [ ] Cloudflare Pages: link Gitea repo, build `cd frontend && npm run build`, output `frontend/dist`
- [ ] Tiered Cache: enable Smart Tiered Cache, validate `Cf-Cache-Status: HIT` from upper tier
- [ ] WAF: enable OWASP Managed Ruleset, set to Simulate → review 48h → Block, enable Bot Fight Mode
- [ ] Rate limiting: `/api/admin/login` + `/api/admin/*` → 10 req/10s per IP, Challenge action
- [ ] HTTPS/WSS: Always Use HTTPS, HSTS max-age=31536000 includeSubDomains preload, min TLS 1.2
- [ ] HTTPS/WSS: enforce at edge (Cloudflare) and origin (nginx), redirect HTTP → HTTPS
## External Services
> Evidence tracker: `docs/operations/cloudflare-uptimerobot-checklist.md`
- [ ] UptimeRobot: add HTTP monitor for `https://linearvoid.tv/health` (5min interval)
- [ ] UptimeRobot: add keyword monitor for `https://linearvoid.tv/hls/playlist.m3u8` (keyword: `#EXTM3U`, 5min)
- [ ] Alert on 2 consecutive failures, notify via email + Discord/Slack webhook
## Production Deployment (Hetzner — not provisioned)
- [ ] Provision Hetzner CX22 (2 vCPU, 4 GB, 40 GB) in Ashburn VA, Ubuntu 24.04
- [ ] Install Docker Engine + Compose plugin, open firewall: 80/443 (public), 2375 (runner IP only)
- [ ] Configure Docker daemon for remote TLS access (`-H tcp://0.0.0.0:2376` with certs)
- [ ] Create `/srv/linearvoid/data/{playlist,media,hls,chat,slate}` directories
- [ ] Add `CI_PROD_*` secrets to Gitea: DOCKER_HOST, R2 keys, admin password, DB password, Grafana password
- [ ] Trigger CD workflow with `DOCKER_HOST` pointing to Hetzner IP
- [ ] Verify all 11 containers start healthy, check `/health`, `/hls/playlist.m3u8`, WebSocket
- [ ] Update Cloudflare A record `linearvoid.tv` → Hetzner IP, SSL Full (strict), test HTTPS + WSS
- [ ] Smoke test: viewer SPA loads, HLS plays, chat connects, admin login → upload → playlist → playout
- [ ] Set up Prometheus alerting notifications + UptimeRobot on production URL
- [ ] Run `scripts/backup-db.sh` + `scripts/verify-backups.sh` on production host
- [ ] Document production IP, SSH access, rollback procedure, scaling thresholds
## Scaling & Performance (planning only)
- [ ] Tune manifest TTL: measure `Cf-Cache-Status` distribution, adjust to 0.5× segment duration
- [ ] Measure TTFB: WebPageTest from 3+ US regions, target <4s warm cache, document baseline
- [ ] Load-test WebSocket: `artillery` or Go script, ramp 10→100→500→1000 connections, measure p95 latency + errors
- [ ] Benchmark admin API: `wrk`/`k6` against `/api/admin/*`, target p95 <1s, identify slow queries
- [ ] Track cache hit ratio: Cloudflare GraphQL → Grafana panel, alert <90%
- [ ] Capacity triggers: Prometheus rules for cache-miss >10%/15min, chat p95 >1s/5min, CPU >80%/10min
- [ ] Phase 2 plan: document boundaries (playout CPU, chat latency, DB state, monitoring), estimate per-service resources
- [ ] Phase 3 plan: research second Hetzner location, design warm standby + DNS failover + DB/Redis replication
docs/MONITORING.md
+91
View File
@@ -0,0 +1,91 @@
# Monitoring Guide
Prometheus + Grafana monitoring for Linear Void.
## Architecture
```text
┌──────────────┐ ┌──────────────┐ ┌───────────┐
│ Exporters │───> │ Prometheus │───>│ Grafana │
│ (7 targets) │ │ :9090 │ │ :3001 │
└──────────────┘ └───────┬──────┘ └───────────┘
│
alerts.yml (8 rules)
```
## Scrape Targets
| Job | Target | Port | Metrics |
| ---------- | ----------------- | ---- | ------------------------------------------------------------- |
| chat | chat | 3000 | WebSocket clients, messages, DMs, moderation, Redis, sessions |
| playout | playout | 8081 | Restarts, mode (live/slate) |
| nginx | nginx-exporter | 9113 | Requests, connections |
| redis | redis-exporter | 9121 | Memory, clients, commands |
| postgres | postgres-exporter | 9187 | Connections, queries, locks |
| node | node-exporter | 9100 | CPU, memory, disk, network |
| prometheus | prometheus | 9090 | Self-monitoring |
Config: `monitoring/prometheus/prometheus.yml` (15s scrape interval)
## Alerting Rules
8 rules in `monitoring/prometheus/alerts.yml`:
| Alert | Severity | Expression |
| -------------------- | -------- | -------------------------------------- |
| ChatDown | critical | `up{job="chat"} == 0` for 2m |
| PlayoutDown | critical | `up{job="playout"} == 0` for 2m |
| RedisDown | warning | `linearvoid_chat_redis_up == 0` for 2m |
| PlayoutSlateActive | warning | `linearvoid_playout_mode == 1` for 1m |
| HighChatLatency | warning | message rate <0.01/s for 5m |
| PrometheusWALFailing | warning | WAL write failures >0 for 5m |
| HighCPU | warning | CPU >80% for 10m |
| LowDiskSpace | warning | Disk <10% free for 5m |
## Access
- Prometheus: `http://<host>:9090` (targets, graph, alerts)
- Grafana: `http://<host>:3001` (default: admin/admin, change via `GRAFANA_PASSWORD`)
## Dashboards
- **Linear Void app**: `monitoring/grafana/dashboard.json` — clients, messages, moderation, Redis, sessions, scrape health
- **Prometheus ops**: `monitoring/grafana/dashboards/dashboard.json` — TSDB, CPU/memory, WAL, compactions, query latency
## Key Metrics
- `linearvoid_chat_connected_clients` — active WebSocket connections
- `linearvoid_chat_messages_total` — total stored messages
- `linearvoid_chat_dms_total` — private messages sent
- `linearvoid_chat_redis_up` — Redis reachable (0/1)
- `linearvoid_playout_restarts_total` — FFmpeg restart counter
- `linearvoid_playout_mode` — 0=live, 1=slate, 2=degraded
- `linearvoid_chat_admin_sessions_active` — active admin logins
## Troubleshooting
| Symptom | Check |
| ----------------------- | -------------------------------------------------------------------- |
| No chat metrics | `curl http://chat:3000/metrics` from Prometheus container |
| No exporter metrics | Verify exporter containers running: `docker ps \| grep exporter` |
| Grafana no data | Datasource URL must be `http://prometheus:9090` |
| Prometheus targets down | Check container network: all must be on `linearvoid-net` |
| Alert not firing | Check Prometheus → Alerts tab, verify `rule_files` in prometheus.yml |
### Grafana shows no data
- Ensure datasource points to <http://prometheus:9090>.
- Confirm Prometheus is healthy.
- Check dashboard query labels match active jobs.
### High cardinality or query slowness
- Reduce unneeded labels.
- Limit expensive dashboard panels.
- Tune retention and query windows.
## Operational Recommendations
- Keep alert thresholds tied to actual user impact.
- Review dashboards after each release that changes endpoints or metrics names.
- Periodically test alert routing and incident response flow.
docs/README.md
+41
View File
@@ -0,0 +1,41 @@
# Docs Folder
This folder contains the documentation for the Linear Void project. The documentation is organized into several markdown files, each covering different aspects of the project.
## Architecture
- [Architecture](architecture/README.md) - High-level system architecture and component interactions.
## API
- [API](API.md) - Implemented WebSocket and REST API reference.
## Deployment
- [Deployment](DEPLOYMENT.md) - Production deployment and operational runbook.
## Monitoring
- [Monitoring](MONITORING.md) - Metrics, dashboards, and monitoring troubleshooting.
## Security
- [Security](SECURITY.md) - Security model, controls, and hardening baseline.
## Setup
- [Setup](SETUP.md) - Test lab setup, local development, and CI/CD workflow.
## Streaming
- [Streaming](STREAMING.md) - Streaming architecture, playout, and delivery details.
## Development
- [Development](DEVELOPMENT.md) - Remaining tasks and test fixtures for development.
## Additional Resources
- [Marketing](marketing/README.md) - Marketing materials and resources.
- [Operations](operations/README.md) - Operational procedures and checklists.
- [Incident Runbook](incident-runbook.md) - Incident response and disaster recovery procedures.
docs/SECURITY.md
+93
View File
@@ -0,0 +1,93 @@
# Security Guide
Layered security model for Linear Void.
## Threat Model
```
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Client │───▶│Cloudflare│───▶│ nginx │───▶│ Backend │
│ (public) │ │ WAF+CDN │ │ proxy │ │ chat │
└──────────┘ └──────────┘ └────┬─────┘ └────┬─────┘
│ │
┌──────┴──────┐ ┌─────┴──────┐
│ /health │ │ /api/admin │
│ /metrics │ │ /ws (chat) │
│ /hls/* │ │ /api/chat/* │
│ (public) │ │ (auth req) │
└─────────────┘ └────────────┘
```
## Authentication & Sessions
- Login: `POST /api/admin/login` → cookie `linearvoid_admin_session`
- Cookie flags: `HttpOnly`, `SameSite=Strict`, `Secure` (over HTTPS)
- Server-side expiry via `ADMIN_SESSION_TTL_HOURS`
- Rate limit: 5 attempts/IP/5min via Redis
## Transport Security (TLS/HSTS)
- Cloudflare TLS mode: **Full (strict)** — valid origin cert required
- nginx listens on 80 internally; Cloudflare handles TLS termination
- HSTS header set by Cloudflare: `max-age=31536000; includeSubDomains; preload`
- WSS upgrade path: `/ws` through nginx → chat:3000
## Network Isolation
- All backend services on internal Docker network (`linearvoid-net`)
- Only nginx port exposed to host (443/80)
- Database, Redis, exporters inaccessible from outside
- `docker-compose.prod.yml` keeps ports internal via `expose:` not `ports:`
## Reverse Proxy Controls
- `auth_request /api/admin/auth` gate on all `/admin` and `/api/admin` paths
- Explicit `client_max_body_size 64m` for resumable upload chunks
- WebSocket upgrade headers: `Upgrade`, `Connection` preserved
- CORS headers on HLS: `Access-Control-Allow-Origin *`
- XSS protection: CSP headers served by Cloudflare
## Rate Limiting
| Resource | Limit | Window |
| ----------------- | ----- | ------ |
| WebSocket connect | 5/IP | 60s |
| Chat messages | 10/IP | 10s |
| Admin login | 5/IP | 5min |
Implemented via Redis `INCR` + `EXPIRE`.
## Secrets Management
- `.env` — local dev only, **gitignored**
- Gitea Actions secrets (`CI_*` prefix) for CI/CD
- Never commit: passwords, tokens, API keys, R2 credentials
- Production `.env` provisioned manually on host, never checked in
- Rotate admin password quarterly
## Data Protection
- PostgreSQL: chat history, moderation logs, media metadata
- Redis: append-only persistence for presence, rate limits, sessions
- HLS segments: ephemeral, regenerated by playout
- Backups: `scripts/backup-db.sh` (daily), `scripts/backup-weekly.sh`, `scripts/verify-backups.sh`
## Hardening Checklist
- [ ] HTTPS/WSS enforced end-to-end (Cloudflare Full strict)
- [ ] Strong admin credentials configured
- [ ] Secrets only in env/secrets stores (never committed)
- [ ] Proxy auth and websocket headers validated
- [ ] Upload limits and MIME validation active
- [ ] Rate limiting active (WS, messages, login)
- [ ] Cloudflare WAF/rate-limit rules active
- [ ] Backup/restore drills completed
- [ ] Security alerts routed (Prometheus alertmanager)
## Incident Response
1. Detect — alerts (ChatDown, RedisDown, HighChatLatency) or user reports
2. Triage — edge (Cloudflare), proxy (nginx), app (chat/playout), data (Redis/Postgres)
3. Contain — rate limit, credential rotate, service restart, slate mode activation
4. Recover — verify health endpoints, check Prometheus targets, validate HLS output
5. Record — timeline + corrective actions in `docs/incident-runbook.md`
docs/SETUP.md
+588
View File
@@ -0,0 +1,588 @@
# Linear Void Setup Guide
This guide covers prerequisites, environment setup, running the application locally and in production, and troubleshooting. It references the [architecture documentation](./docs/architecture/README.md)(`docs/architecture/README.md`).
## Prerequisites
- **Docker & Docker Compose**: Required for running the full stack (playout, chat, nginx, redis, db, prometheus, grafana). Install from [docker.com](https://www.docker.com).
- **Node.js 20+**: Required for frontend development and production builds (`cd frontend && npm install && npm run build`).
- **npm**: Installed with Node.js and used for the frontend dependency installation and Vite workflow.
- **Go 1.23+**: Recommended when working on the Go services outside Docker (`backend/chat`, `backend/playout`).
- **FFmpeg**: Used by the playout service for HLS generation. Install via `apt install ffmpeg` (Linux), Homebrew (macOS), or Winget (Windows).
- **Git**: For cloning and version control.
- **Cloudflare Account**: For R2 storage and CDN (free tier sufficient for most use cases).
- **Gitea Account**: Required if you want to push images to the self-hosted registry or use the repository's CI/CD flow.
- Set up the required Actions secrets in Gitea at: [https://git.allucanget.biz/zwitschi/linearvoid/settings/actions/secrets](https://git.allucanget.biz/zwitschi/linearvoid/settings/actions/secrets)
- Add the following secrets:
- `CI_REGISTRY_USER`: Your Gitea username (e.g., `zwitschi`)
- `CI_REGISTRY_TOKEN`: A Gitea Personal Access Token with `write:packages` scope
- `ADMIN_USERNAME`: The administrator username for the production admin panel (e.g., `admin`)
- `ADMIN_PASSWORD`: A strong password for the production admin panel
- `DATABASE_URL`: The production database connection string (e.g., `postgres://linearvoid:<password>@db:5432/linearvoid?sslmode=disable`)
- **Hetzner Cloud Account** (optional for production): For the US-East VPS (Ashburn, VA recommended per ADR-001).
## Network Environments
Linear Void operates in two distinct environments. Know which one you're targeting.
```mermaid
flowchart LR
subgraph Test_Lab["Test Lab (192.168.88.0/24)"]
TEST_HOST["192.168.88.40<br/>Docker Engine :2375"]
end
subgraph Prod["Production (Hetzner — NOT provisioned)"]
PROD_HOST["Hetzner CX22 Ashburn, VA<br/>Public IP TBD"]
end
subgraph Gitea["Gitea CI/CD"]
CI["CI: build + push images"]
CD["CD: deploy via DOCKER_HOST"]
end
subgraph CF["Cloudflare"]
DNS["linearvoid.tv"]
R2["R2 Bucket"]
end
CD -->|"DOCKER_HOST=tcp://192.168.88.40:2375"| TEST_HOST
CD -.->|"DOCKER_HOST=tcp://&lt;hetzner-ip&gt;:2375<br/>(when provisioned)"| PROD_HOST
TEST_HOST -->|"test traffic"| DNS
PROD_HOST -.->|"production traffic"| DNS
TEST_HOST --> R2
PROD_HOST -.-> R2
```
**Test Lab (active):** Internal network `192.168.88.0/24`, host `192.168.88.40`. Receives automated CD deploys from Gitea Actions. Used for integration testing and staging validation.
**Production (pending):** Hetzner Cloud VPS, Ashburn VA. Not yet provisioned. CD workflow and compose files are ready — only `DOCKER_HOST` needs changing. See Production Deployment section in [DEPLOYMENT.md](./DEPLOYMENT.md) for cutover plan.
## Development Prerequisites
- **Frontend dependencies**: Run `cd frontend && npm install` before `npm run dev` or `npm run build`.
- **Docker build support**: Local development expects Docker to be able to build the `chat`, `playout`, and `nginx` images referenced in `docker-compose.yml`.
- **Data directories**: Create `data/playlist`, `data/media`, `data/hls`, and `data/chat` before the first run so bind mounts resolve cleanly.
- **Database choice**: The stack defaults to PostgreSQL in `docker-compose.yml`. If you want SQLite for local-only development, update `DATABASE_URL` in `.env` accordingly.
- **Redis availability**: Chat presence, rate limiting, and pub/sub require Redis. The compose stack provides it automatically.
- **Video inputs**: Source videos should be normalized before playout. If the playlist contains invalid files, the playout container will fail its validation flow.
## Environment Setup
1. **Clone the Repository** (if not already done):
```bash
git clone https://git.allucanget.biz/zwitschi/linearvoid.git
cd linearvoid
```
2. **Copy Environment File**:
```bash
cp .env.example .env
```
3. **Configure `.env`** (edit with your values):
```env
# Cloudflare R2 (required for video storage and $0 egress)
R2_ACCESS_KEY_ID=your-r2-access-key
R2_SECRET_ACCESS_KEY=your-r2-secret-key
R2_BUCKET_NAME=linearvoid-media
R2_ENDPOINT=https://your-account-id.r2.cloudflarestorage.com
R2_REGION=auto
# Cloudflare (for custom domain and cache rules)
CLOUDFLARE_API_TOKEN=your-cloudflare-api-token
CLOUDFLARE_ACCOUNT_ID=your-account-id
MEDIA_DOMAIN=media.linearvoid.tv
# Public frontend URLs
# Preferred local setup uses the real public domains through Cloudflare and Nginx Proxy Manager.
VITE_PUBLIC_APP_URL=https://linearvoid.tv
VITE_PUBLIC_WS_URL=wss://linearvoid.tv/ws
VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8
VITE_PUBLIC_ADMIN_DEFAULT_MODERATOR=admin
VITE_PUBLIC_ADMIN_DEFAULT_ROOM=main
VITE_PUBLIC_ADMIN_MODERATION_LOG_LIMIT=50
ADMIN_USERNAME=admin
ADMIN_PASSWORD=change-me
ADMIN_SESSION_TTL_HOURS=12
# Optional direct-local fallback when bypassing the public proxy:
# VITE_PUBLIC_APP_URL=http://localhost
# VITE_PUBLIC_WS_URL=ws://localhost/ws
# VITE_PUBLIC_HLS_URL=http://localhost/hls/playlist.m3u8
# Gitea Container Registry (for self-hosted Docker images)
CI_REGISTRY_URL=git.allucanget.biz
CI_REGISTRY_USER=your-gitea-username
CI_REGISTRY_TOKEN=your-gitea-token
CI_REGISTRY_NAMESPACE=zwitschi/linearvoid
# Gitea API access (optional, for CI/release automation)
CI_SERVER_URL=https://git.allucanget.biz
# Chat / Database
DATABASE_URL=postgres://linearvoid:linearvoid@db:5432/linearvoid?sslmode=disable
# or for SQLite-only local development:
# DATABASE_URL=sqlite://./data/chat.db
REDIS_URL=redis://redis:6379/0
# Ports (for local development)
PLAYOUT_PORT=8080
CHAT_PORT=3000
NGINX_PORT=80
# Playout Configuration
PLAYLIST_PATH=/app/playlist.txt
HLS_SEGMENT_DURATION=6
HLS_WINDOW_SIZE=24
# Logging / Monitoring
LOG_LEVEL=info
PROMETHEUS_PORT=9090
GRAFANA_PORT=3001
# Development flags
NODE_ENV=development
GO_ENV=development
```
4. **Check local value overrides for Docker Compose**:
- `docker-compose.yml` builds images locally and maps the frontend build output from `frontend/dist` into nginx.
- `docker-compose.prod.yml` pulls prebuilt images from the Gitea registry.
- Internal container-to-container URLs should use compose hostnames, for example `REDIS_URL=redis://redis:6379/0` and `DATABASE_URL=postgres://linearvoid:<password>@db:5432/linearvoid?sslmode=disable`.
- For production public routing, set `VITE_PUBLIC_APP_URL=https://linearvoid.tv`, `VITE_PUBLIC_WS_URL=wss://linearvoid.tv/ws`, and `VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8`.
5. **Create Data Directories**:
```bash
mkdir -p data/playlist data/media data/hls data/chat
```
6. **Prepare Sample Videos** (optional):
- Place normalized MP4 files in `data/media/`.
- Create a `data/playlist/playlist.txt` with paths (see ADR-001 for normalization requirements).
## Gitea Setup
1. **Create or verify a Gitea user**:
- The user should have access to the repository and permission to read from the container registry.
2. **Create an access token**:
- Generate a token with the scopes needed for package/registry access and, if applicable, repository automation.
- Store it as `CI_REGISTRY_TOKEN` in `.env`.
3. **Configure Gitea Actions Secrets**:
- Go to the repository settings page: [https://git.allucanget.biz/zwitschi/linearvoid/settings/actions/secrets](https://git.allucanget.biz/zwitschi/linearvoid/settings/actions/secrets)
- Add the following secrets to enable automated CI/CD and production deployment:
- `CI_REGISTRY_USER`: Your Gitea username (e.g., `zwitschi`).
- `CI_REGISTRY_TOKEN`: Your Gitea Personal Access Token with `write:packages` scope.
- `ADMIN_USERNAME`: The administrator username for the production admin panel (e.g., `admin`).
- `ADMIN_PASSWORD`: A strong password for the production admin panel.
- `DATABASE_URL`: The production database connection string (e.g., `postgres://linearvoid:<password>@db:5432/linearvoid?sslmode=disable`).
4. **Configure registry variables**:
- Set `CI_REGISTRY_URL` to the Gitea host.
- Set `CI_REGISTRY_NAMESPACE` to the image namespace, for example `zwitschi/linearvoid`.
- Set `CI_REGISTRY_USER` to the account that will push or pull images.
5. **Authenticate Docker to the registry**:
```bash
docker login git.allucanget.biz
```
6. **Build and publish images when needed**:
- Local development uses `docker-compose.yml`, which builds from source.
- Production uses `docker-compose.prod.yml`, which expects the `playout`, `chat`, and `nginx` images to exist in the configured Gitea registry namespace.
7. **Validate image availability**:
- Before a production rollout, verify that the expected `:latest` tags can be pulled from the registry on the target server.
## Cloudflare R2 Setup
1. **Create an R2 bucket**:
- Create a bucket such as `linearvoid-media` in Cloudflare R2.
- Keep the bucket name aligned with `R2_BUCKET_NAME` in `.env`.
2. **Generate R2 API credentials**:
- Create an access key pair with read/write permissions for the bucket.
- Set `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, and `R2_REGION=auto` in `.env`.
3. **Find the S3 endpoint**:
- Use the account-scoped endpoint format `https://<account-id>.r2.cloudflarestorage.com`.
- Store the account ID in both `R2_ENDPOINT` and `CLOUDFLARE_ACCOUNT_ID` where applicable.
4. **Bind a custom media domain**:
- Attach a domain such as `media.linearvoid.tv` to the bucket.
- Set `MEDIA_DOMAIN` to that hostname.
5. **Configure app-domain DNS and TLS**:
- Point `linearvoid.tv` to the public entrypoint serving nginx.
- Point `www.linearvoid.tv` to the same target and redirect it to `https://linearvoid.tv`.
- Keep `media.linearvoid.tv` on the R2 custom domain path so HLS delivery stays split from the app origin.
- Use Cloudflare proxied DNS with Full (strict) TLS so browser traffic uses `https://linearvoid.tv`, `wss://linearvoid.tv/ws`, and `https://media.linearvoid.tv`.
6. **Apply cache behavior**:
- Configure the `.m4s` and `.m3u8` cache rules from `infrastructure/cloudflare/cache-rules.md`.
- Enable Smart Tiered Cache for the R2-backed hostname.
- If you need header normalization for immutable segment caching, deploy `infrastructure/cloudflare/worker.js` to the same zone.
7. **Verify access**:
- Confirm uploads with an S3-compatible client such as `aws s3 ls --endpoint-url <R2_ENDPOINT>`.
- After deployment, check Cloudflare analytics for cache hit ratio and origin request volume.
## Cloudflare
Use this section for full edge setup (DNS, Pages, cache, security, and monitoring).
Execution tracker: `docs/operations/cloudflare-uptimerobot-checklist.md`.
### 1. DNS Setup
1. Add apex record for `linearvoid.tv` pointing to Cloudflare Pages target.
2. Add `www` CNAME to Cloudflare Pages and configure redirect to `https://linearvoid.tv`.
3. Add `media.linearvoid.tv` for split-domain HLS delivery.
4. Keep all records proxied (orange cloud) unless troubleshooting requires DNS-only mode.
### 2. R2 Bucket
1. Create bucket `linearvoid-media`.
2. Set location hint to `enam` (US-East).
3. Attach custom domain `media.linearvoid.tv` to the bucket.
4. Verify object read works from the custom domain.
### 3. Cache Rules
Configure Cloudflare Cache Rules:
1. HLS segment files (`*.ts`, `*.m4s`):
- Cache: eligible
- Edge TTL: `31536000`
- Browser TTL: long-lived
- Cache key: include full path
2. HLS manifests (`*.m3u8`):
- Cache: eligible
- Edge TTL: `2s`
- Revalidation: enabled
3. Static assets (`*.js`, `*.css`):
- Cache: eligible
- Edge TTL: `1y`
- Browser TTL: long-lived
- Cache key: include full path
4. Validate behavior using response headers:
- `Cf-Cache-Status` should move from `MISS` to `HIT` for hot objects
Reference: `infrastructure/cloudflare/cache-rules.md`.
### 4. Cloudflare Pages
1. Connect Gitea repository to Cloudflare Pages.
2. Build command:
```bash
cd frontend && npm run build
```
3. Output directory:
```text
frontend/dist
```
4. Configure production domain to `linearvoid.tv`.
### 5. Tiered Cache
1. Enable Smart Tiered Cache in Cloudflare dashboard.
2. Warm a segment and manifest path.
3. Confirm tiered behavior through repeated requests and `Cf-Cache-Status: HIT`.
### 6. WAF and Bot Protection
1. Enable OWASP Managed Ruleset.
2. Start in `Simulate` mode.
3. Review events for 48 hours.
4. Switch to `Block` once false positives are addressed.
5. Enable Bot Fight Mode.
### 7. Rate Limiting
Create rules:
1. Path: `/api/admin/login`
2. Path: `/api/admin/*`
3. Threshold: `10 requests / 10 seconds / IP`
4. Action: `Managed Challenge`
### 8. HTTPS and WSS
1. Enable `Always Use HTTPS`.
2. Enable HSTS with:
- `max-age=31536000`
- `includeSubDomains`
- `preload`
3. Minimum TLS version: `1.2`.
4. Keep SSL mode at `Full (strict)` when origin certs are valid.
5. Enforce HTTPS redirect at both layers:
- Cloudflare edge
- nginx origin
6. Validate websocket endpoint uses `wss://linearvoid.tv/ws`.
### 9. UptimeRobot Monitors
Create external monitors:
1. HTTP monitor:
- URL: `https://linearvoid.tv/health`
- Interval: 5 minutes
2. Keyword monitor:
- URL: `https://linearvoid.tv/hls/playlist.m3u8`
- Keyword: `#EXTM3U`
- Interval: 5 minutes
3. Alert policy:
- Trigger on 2 consecutive failures
- Notify via email and Discord/Slack webhook
### 10. Verification Checklist
1. DNS resolves for `linearvoid.tv`, `www.linearvoid.tv`, and `media.linearvoid.tv`.
2. Pages serves frontend for apex and `www` redirects correctly.
3. HLS segments are cacheable long-term, manifests refresh every ~2s.
4. Admin paths are rate-limited and WAF events are visible.
5. HTTPS and WSS both succeed with valid certificates.
6. UptimeRobot alerts fire as expected when endpoints are intentionally made unavailable.
## Running the Application
### Local Development
1. **Start All Services**:
```bash
docker compose up --build
```
2. **Access the Application**:
- Preferred local access with the documented proxy setup: <https://linearvoid.tv>
- Admin Interface: <https://linearvoid.tv/admin> (redirects to `/admin/login/` until authenticated)
- Chat WebSocket: `wss://linearvoid.tv/ws`
- Direct-local fallback when bypassing Cloudflare/Nginx Proxy Manager: <http://localhost>, <http://localhost/admin>, and `ws://localhost/ws`
- Health Checks:
- Playout: <http://localhost:8081/health>
- Chat: <http://localhost:3000/health>
- Monitoring:
- Prometheus: <http://localhost:9090>
- Grafana: <http://localhost:3001> (default login: admin/admin)
3. **Frontend Development** (optional, for UI changes):
```bash
cd frontend
npm install
npm run dev
```
The dev server runs on <http://localhost:5173>. It proxies API calls to the backend services.
4. **Build frontend assets for nginx-backed local runs**:
```bash
cd frontend
npm install
npm run build
```
The local nginx container serves `frontend/dist`, so rebuilding the frontend is required before validating production-like static output.
5. **Stop Services**:
```bash
docker compose down
```
### Local Test Environment Deployment (Remote Docker Host)
For testing deployments on a local network server without setting up a full production VPS, you can deploy directly to a remote Docker host (e.g., a home lab server or test machine) using Docker's TCP socket.
#### Test Environment Prerequisites
- A remote server running Docker at IP `192.168.88.40` (running Alpine Linux).
- Docker TCP socket enabled and listening on port `2375` on the remote server.
##### Enabling Docker TCP Socket on Alpine Linux
To allow the Gitea Actions runner (or your local machine) to deploy to the remote Alpine host, you must configure the Docker daemon to listen on the TCP socket:
1. Edit `/etc/conf.d/docker` on the Alpine host:
```bash
vi /etc/conf.d/docker
```
2. Update the `DOCKER_OPTS` line to include both the unix socket and the TCP socket:
```bash
DOCKER_OPTS="-H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375"
```
3. Restart the Docker service:
```bash
rc-service docker restart
```
4. Verify that Docker is listening on port `2375`:
```bash
netstat -tlnp | grep 2375
```
5. Ensure any firewall (like `awall` or `iptables`) on the Alpine host allows incoming traffic on port `2375` from your local network subnet (e.g., `192.168.88.0/24`).
#### Deployment Steps
1. **Configure Docker CLI to use the Remote Host**:
You can point your local Docker CLI to the remote server by setting the `DOCKER_HOST` environment variable:
- **PowerShell (Windows)**:
```powershell
$env:DOCKER_HOST="tcp://192.168.88.40:2375"
```
- **Bash (Linux/macOS)**:
```bash
export DOCKER_HOST="tcp://192.168.88.40:2375"
```
2. **Verify Connection**:
Run `docker info` to confirm your local CLI is communicating with the remote server at `192.168.88.40`:
```bash
docker info
```
The output should show the remote server's hostname and system details.
3. **Deploy the Stack**:
With `DOCKER_HOST` set, any `docker` or `docker compose` commands you run locally will execute directly on the remote server.
- **Deploy from Registry (Production-like)**:
```bash
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d --remove-orphans
```
- **Build and Deploy from Source**:
```bash
docker compose up -d --build
```
4. **Access the Test Environment**:
- Playout Health: `http://192.168.88.40:8081/health`
- Chat Health: `http://192.168.88.40:3000/health`
- Nginx Entrypoint: `http://192.168.88.40:8080` (or the port mapped to `NGINX_PORT` in `.env`)
5. **Reset to Local Docker Host**:
Once testing is complete, clear the `DOCKER_HOST` variable to point back to your local machine:
- **PowerShell**:
```powershell
Remove-Item env:DOCKER_HOST
```
- **Bash**:
```bash
unset DOCKER_HOST
```
### Production Setup Guide
1. **Prepare the target host**:
- Provision a Linux VPS with Docker Engine, Docker Compose, Git, and enough disk for media, HLS output, PostgreSQL, Redis persistence, and monitoring data.
- Open the ports required for HTTP/HTTPS and any restricted monitoring access you intend to expose.
2. **Clone the repository and create the environment file**:
```bash
git clone https://git.allucanget.biz/zwitschi/linearvoid.git
cd linearvoid
cp .env.example .env
```
3. **Set production-safe `.env` values**:
- Replace placeholder secrets.
- Use compose-network hostnames for internal services, for example `redis` and `db`.
- Set strong values for `POSTGRES_PASSWORD` and `GRAFANA_PASSWORD`.
- Keep `MEDIA_DOMAIN`, R2 credentials, and Gitea registry values consistent with the external infrastructure.
- Set `MEDIA_DOMAIN=media.linearvoid.tv`.
- Set `VITE_PUBLIC_APP_URL=https://linearvoid.tv`, `VITE_PUBLIC_WS_URL=wss://linearvoid.tv/ws`, and `VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8`.
- If you intentionally bypass the public proxy for local-only testing, switch those values to `http://localhost`, `ws://localhost/ws`, and `http://localhost/hls/playlist.m3u8`.
4. **Create runtime directories on the server**:
```bash
mkdir -p data/playlist data/media data/hls data/chat
```
5. **Publish or verify required images in Gitea**:
- Ensure the `playout`, `chat`, and `nginx` images exist under `${CI_REGISTRY_URL}/${CI_REGISTRY_NAMESPACE}`.
- Log in to the registry from the VPS before pulling.
6. **Deploy the production stack**:
```bash
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d
```
7. **Run post-deploy verification**:
- Check `docker compose -f docker-compose.prod.yml ps`.
- Confirm nginx, chat, and playout health endpoints respond.
- Verify that HLS output is being written to `data/hls` and that the media domain serves manifests and segments through Cloudflare.
8. **Harden and operate the deployment**:
- Put HTTPS in front of nginx using Cloudflare or another reverse proxy, with `linearvoid.tv` as canonical app host and `www.linearvoid.tv` redirected.
- Restrict Grafana and Prometheus exposure to trusted IPs or a private network.
- Run the stack under systemd or another service manager if you need auto-start behavior outside Docker restart policies.
### Production Deployment (Hetzner VPS + Cloudflare)
1. **Provision VPS**: Use Hetzner Cloud CX22 in Ashburn, VA (per ADR-001).
2. **Configure Cloudflare**:
- Create R2 bucket with `enam` location hint.
- Set up custom media domain (`media.linearvoid.tv`) with the cache rules from `infrastructure/cloudflare/cache-rules.md`.
- Point `linearvoid.tv` and `www.linearvoid.tv` at the app entrypoint, then redirect `www` to the apex domain.
- Use Full (strict) TLS for both app and media hostnames.
- Apply the Worker from `infrastructure/cloudflare/worker.js` for advanced header management.
3. **Deploy**:
- Copy the project to the VPS.
- Use production compose: `docker compose -f docker-compose.prod.yml up -d`
- Or pull from Gitea registry: `docker compose -f docker-compose.prod.yml pull && docker compose -f docker-compose.prod.yml up -d`
- Configure systemd to manage the Docker Compose stack.
4. **Monitoring**: Access Grafana at the exposed port for dashboards (included in `monitoring/grafana/dashboard.json`).
See `docs/architecture/README.md` for the full arc42 documentation and the ADRs for architectural decisions.
## Troubleshooting
- **R2 Upload Fails**: Check credentials in `.env` and ensure the bucket exists. Verify with `aws s3 ls --endpoint-url=$R2_ENDPOINT`.
- **Normalization Errors**: Run `backend/playout/start-playout.sh` manually to see ffprobe output. All videos must be H.264/AAC, 30 fps constant, 2s keyframes (ADR-001).
- **Chat Not Connecting**: Check Redis connection and WebSocket URL. The health check at `/health` should return 200.
- **Cache Not Working**: Verify Cloudflare cache rules (`.m4s` = long TTL immutable, `.m3u8` = short TTL). Check cache hit ratio in Cloudflare dashboard.
- **Docker Issues**: Run `docker compose logs` for service-specific errors. Ensure ports are not in use.
- **Admin Interface**: Ensure `ADMIN_USERNAME`, `ADMIN_PASSWORD`, and `ADMIN_SESSION_TTL_HOURS` are set. `/admin/` and `/api/admin/*` are protected by a session-based login flow.
- **Production Pull Fails**: Confirm `docker login` to Gitea succeeded on the server and the image tags referenced by `docker-compose.prod.yml` exist.
## Further Reading
- Architecture: `docs/architecture/README.md` (arc42 template with diagrams).
- Decisions: `docs/architecture/adr/` (ADR-001 to ADR-007).
- Contributing: `CONTRIBUTING.md`.
- Project Structure: Root `README.md`.
The system is ready for use. For production, review ADR-001 for cost estimates and scaling guidance.
**Last Updated**: 2026-08-03
docs/STREAMING.md
+186
View File
@@ -0,0 +1,186 @@
# Streaming Pipeline
This document is the operational contract for Linear Void video ingest, playout, publication, caching, and browser playback. General deployment, chat, and admin details remain in `DEPLOYMENT.md`, `SETUP.md`, and `API.md`.
## Production flow
```mermaid
flowchart LR
A["Normalized MP4 assets"] --> B["Playlist and schedule"]
B --> C["Playout service and FFmpeg"]
C --> D["Local HLS staging"]
D --> E["R2 object storage"]
E --> F["Cloudflare edge"]
F --> G["hls.js or native HLS"]
```
The playout service is the timeline authority. It reads the ordered playlist, runs FFmpeg continuously, stages a live HLS window under `data/hls/`, and publishes complete objects to R2. Viewers read only the public R2 custom domain through Cloudflare:
```text
https://media.linearvoid.tv/hls/playlist.m3u8
```
Nginx `/hls/` is a local/test fallback, not the production media origin. The SPA, admin API, and chat use `linearvoid.tv`; media uses `media.linearvoid.tv`.
## Source-media contract
Only validated files may enter the production playlist.
| Property | Required profile |
| ---------- | ------------------------------------------------------------------------ |
| Container | MP4 |
| Video | H.264/AVC, `yuv420p` |
| Resolution | 1280x720 or 1920x1080; avoid changing resolution within a playlist block |
| Frame rate | Constant 30 fps |
| GOP | Closed, fixed 2 seconds: `-g 60 -keyint_min 60 -sc_threshold 0` |
| Audio | AAC-LC, stereo, 48 kHz, 128–192 kbit/s |
The ingest gate uses `ffprobe` to reject incompatible codecs, variable frame rate, missing audio, invalid dimensions, or incorrect keyframe cadence. Assets are immutable after acceptance; corrections create a new asset ID and object key.
Reference normalization:
```bash
ffmpeg -i input.mov \
-c:v libx264 -preset medium -crf 21 \
-r 30 -g 60 -keyint_min 60 -sc_threshold 0 \
-pix_fmt yuv420p \
-c:a aac -b:a 160k -ar 48000 -ac 2 \
-movflags +faststart \
output-normalized.mp4
```
## Playlist and timeline
- `data/playlist/playlist.txt` contains one local media path per line; blank lines and `#` comments are ignored.
- The service validates the complete playlist before activating a revision.
- Playlist edits are written atomically and receive a revision ID. The active FFmpeg run keeps its current item and adopts the new revision at the next item boundary.
- All viewers follow the same live manifest; clients do not calculate the schedule independently.
- Schedule metadata must be derived from the same active playlist revision and timeline so “now/next” agrees with the video.
- When the last item ends, playout returns to the first item unless an active schedule override says otherwise.
## Playout and HLS packaging
The playout service wraps FFmpeg, tracks the active item and playlist revision, restarts failed processes, and exposes `/health` plus Prometheus metrics.
Baseline packaging:
| Setting | Value |
| ---------------- | ------------------------------------------------------------------------------------- |
| Protocol | Standard live HLS, not LL-HLS |
| Segment format | fMP4 (`.m4s`) with an initialization segment; `.ts` may be supported during migration |
| Segment target | 6 seconds (`HLS_SEGMENT_DURATION=6`) |
| Live window | 24 segments (`HLS_WINDOW_SIZE=24`), about 144 seconds |
| Codec path | Stream copy for conforming assets; no per-viewer transcoding |
| Expected latency | 15–35 seconds |
Segment boundaries must align with keyframes. Timestamps must remain monotonic across item boundaries. If FFmpeg cannot preserve a seamless timeline, insert `#EXT-X-DISCONTINUITY` and publish the corresponding new initialization segment.
Every segment name is globally unique and never reused, including after a restart or playlist loop. Use a run/session ID plus a monotonically increasing media sequence, for example:
```text
hls/r-20260807T170412Z/init-000001.mp4
hls/r-20260807T170412Z/seg-000001234.m4s
```
This naming rule is essential: a URL advertised as immutable must never later contain different bytes.
## Safe publication to R2
Publication order prevents a manifest from referencing an unavailable segment:
1. FFmpeg writes a segment to a temporary local name and closes it.
2. The publisher uploads the initialization segment, if new, and media segment to their final immutable R2 keys.
3. It verifies successful upload and records object size/checksum.
4. It renders the next media manifest referencing only confirmed objects.
5. It uploads the manifest to a temporary key, then replaces `hls/playlist.m3u8` as the final commit step.
Never upload an actively written segment or publish the manifest first. Use a single publisher instance, or fencing/leader election, so two playout workers cannot update the live manifest concurrently.
The local `data/hls/` directory is staging data and may be regenerated. Source media, playlist revisions, and schedule state are authoritative; generated HLS is not backed up as application state.
## Cache contract
| Objects | Edge policy | Browser policy | Required headers/behavior |
| -------------------------- | ---------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------- |
| `.m4s`, `.ts`, init `.mp4` | Cache eligible; TTL 1 year | TTL 1 year | `Cache-Control: public, max-age=31536000, immutable`; full path in cache key |
| Live `.m3u8` | Cache eligible; edge TTL 2 seconds | Do not retain beyond freshness interval | `Cache-Control: public, max-age=1, s-maxage=2, must-revalidate` |
| Versioned JS/CSS | Cache eligible; TTL 1 year | TTL 1 year | Content-hashed filenames and `immutable` |
Further rules:
- Do not add query-string cache busters to segment URLs; uniqueness belongs in the path.
- Cache keys must preserve hostname and full path. Ignore only explicitly approved tracking parameters.
- HLS responses require `Access-Control-Allow-Origin: https://linearvoid.tv` (or the reviewed origin allowlist), `GET`, `HEAD`, and `OPTIONS` support, and correct MIME types.
- Enable Smart Tiered Cache for `media.linearvoid.tv` to reduce R2 reads on regional misses.
- Purge manifests when necessary, but never purge immutable segments during normal operation.
- Retain expired-window segments in R2 for at least the maximum manifest TTL plus client retry/grace time. A practical minimum is one hour; lifecycle deletion may run later according to storage policy.
Expected validation:
```bash
curl -I https://media.linearvoid.tv/hls/playlist.m3u8
curl -I https://media.linearvoid.tv/hls/<segment>.m4s
```
The manifest must refresh within a few seconds. Repeated segment requests should progress from `CF-Cache-Status: MISS` to `HIT`, retain the same ETag/content length, and carry the immutable cache header.
## Player behavior
The SPA uses native HLS where supported and hls.js elsewhere.
- Start at the live edge with a conservative buffer; do not seek to playlist start.
- Treat the manifest as the synchronization source and target roughly three segments behind the newest segment unless testing supports a smaller margin.
- Recover transient network/media errors with bounded retries and exponential backoff.
- On a fatal hls.js network error, reload the manifest; on a fatal media error, attempt media recovery once before recreating the player.
- When tab suspension or drift leaves playback materially behind the target live position, seek forward to the target rather than playing the backlog.
- Surface a clear reconnecting state. Never silently loop stale media.
- Schedule/chat timestamps should account for the measured player latency rather than assuming wall-clock playout is already visible.
## Failure and recovery
The service retries an item failure with a bounded backoff. After the configured failure threshold it switches to a pre-normalized looping slate and reports `mode=slate`. It periodically retries normal playout and returns only after a segment and committed manifest have been published successfully.
On process or host restart:
- create a new run/session ID and continue with globally unique segment paths;
- resume from the schedule-derived current position when reliable state exists, otherwise start the current playlist item;
- never overwrite old segment objects;
- keep the last valid manifest available until the replacement stream is playable;
- emit a discontinuity when timeline continuity cannot be guaranteed.
Health is not merely “process running.” `/health` should report at least mode (`live`, `slate`, `starting`, `degraded`), active item, playlist revision, run ID, last segment sequence/time, last successful R2 publication, consecutive failures, and uptime. Mark the service unhealthy if the manifest or media sequence has not advanced for more than two segment durations.
## Monitoring and acceptance checks
Monitor and alert on:
- manifest age and media-sequence advancement;
- time since last successfully published segment;
- FFmpeg restarts and consecutive item failures;
- live/slate mode;
- R2 upload latency and errors;
- segment 404/5xx rate and manifest/segment request latency;
- Cloudflare cache-hit ratio, origin requests, and R2 Class B operations;
- player startup time, fatal errors, rebuffer ratio, and live-edge drift;
- local staging disk usage.
A release is streaming-ready only when one full playlist boundary has played without timestamp errors, the manifest never references a missing object, a second viewer joins the same timeline, cache headers match the table above, a forced FFmpeg failure activates the slate, and normal playout recovers without reusing an object URL.
## Required configuration
| Variable | Purpose |
| ------------------------------------------ | -------------------------------------------------- |
| `PLAYLIST_PATH` | Active playlist file |
| `HLS_SEGMENT_DURATION` | Target segment duration; baseline `6` |
| `HLS_WINDOW_SIZE` | Manifest window; baseline `24` |
| `R2_ENDPOINT`, `R2_REGION` | S3-compatible publication endpoint |
| `R2_BUCKET_NAME` | Media bucket |
| `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY` | Write credentials, available only to the publisher |
| `MEDIA_DOMAIN` | Public R2 custom domain |
| `VITE_PUBLIC_HLS_URL` | Public live manifest URL |
Production values are `MEDIA_DOMAIN=media.linearvoid.tv` and `VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8`. Local fallback may use `http://localhost/hls/playlist.m3u8`.
## Explicitly deferred
Adaptive-bitrate ladders, LL-HLS, DVR/catch-up, multi-origin playout, and regional chat affinity are outside the v1 contract. Adding any of them requires an ADR and an update to this document.
docs/architecture/README.md
+6 -4
View File
@@ -87,10 +87,12 @@
## Companion Documentation
- [API.md](../../API.md) - Implemented WebSocket and REST API reference.
- [DEPLOYMENT.md](../../DEPLOYMENT.md) - Production deployment and operational runbook.
- [MONITORING.md](../../MONITORING.md) - Metrics, dashboards, and monitoring troubleshooting.
- [SECURITY.md](../../SECURITY.md) - Security model, controls, and hardening baseline.
- [API.md](../API.md) - Implemented WebSocket and REST API reference.
- [DEPLOYMENT.md](../DEPLOYMENT.md) - Production deployment and operational runbook.
- [MONITORING.md](../MONITORING.md) - Metrics, dashboards, and monitoring troubleshooting.
- [SECURITY.md](../SECURITY.md) - Security model, controls, and hardening baseline.
- [SETUP.md](../SETUP.md) - Test lab setup, local development, and CI/CD workflow.
- [STREAMING.md](../STREAMING.md) - Streaming architecture, playout, and delivery details.
## Introduction and Goals