zwitschi/linearvoid
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 9 Wiki Activity

docs: complete API error codes, deploy troubleshooting, monitoring/security rewrite, L2+L3 building blocks

Browse Source
This commit is contained in:
zwitschi committed 2026-08-06 20:00:45 +02:00
1 parent 0f81a81029
commit db3bf96847
5 files changed
+267 -205
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

API.md
+31 -6
View File
@@ -272,11 +272,36 @@ Prometheus exposition includes connected clients, stored messages, moderation ac
## HTTP Status Conventions
- 200/201: successful request
- 400: malformed payload, invalid input, missing required fields
- 401: unauthorized admin request
- 404: not found
- 405: method not allowed
- 409: logical conflict (for example, mismatched resumed upload)
| Code | Meaning | When |
| ---- | --------------------- | ----------------------------------------------------------- |
| 200 | OK | Successful GET, PUT, or action |
| 201 | Created | Successful POST (upload init, override created) |
| 204 | No Content | Successful DELETE, CORS preflight |
| 400 | Bad Request | Malformed JSON, missing required fields, invalid input |
| 401 | Unauthorized | Missing or expired admin session |
| 403 | Forbidden | Valid session but insufficient permissions |
| 404 | Not Found | Resource doesn't exist |
| 405 | Method Not Allowed | Wrong HTTP method for endpoint |
| 409 | Conflict | Schedule override overlap, upload ID mismatch |
| 413 | Payload Too Large | Upload exceeds 2 GB limit |
| 422 | Unprocessable | File fails normalization check (codec, fps, resolution) |
| 429 | Too Many Requests | Rate limit exceeded (chat messages, WS connections, login) |
| 500 | Internal Server Error | Unexpected server failure |
| 503 | Service Unavailable | Redis unavailable for user actions, playout health degraded |
## Rate Limiting
All limits are per-IP (from `X-Forwarded-For` or `RemoteAddr`) unless noted.
| Resource | Limit | Window | Response |
| --------------------- | ---------------------------------- | ---------------------------------------- | ------------------------------------------------------ |
| Chat messages | `CHAT_RATE_LIMIT_MAX` (default 10) | `CHAT_RATE_LIMIT_WINDOW_S` (default 10s) | 429 + `{"type":"system","text":"Rate limit exceeded"}` |
| WebSocket connections | 5 | 60s | 429 + `"too many connections from this IP"` |
| Admin login | 5 | 5min | 429 + `"too many login attempts, try again later"` |
Rate limit headers on applicable responses:
- `X-RateLimit-Limit`: maximum requests per window
- `Retry-After`: seconds until window resets
- 413: payload too large
- 500: internal server error
DEPLOYMENT.md
+14
View File
@@ -228,5 +228,19 @@ docker compose -f docker-compose.prod.yml down -v
- Missing frontend build artifacts in local-only compose workflows.
- Wrong registry credentials or missing login.
- Misaligned domain env vars (app/ws/hls) causing mixed-origin errors.
## Troubleshooting
| Symptom | Check | Fix |
| -------------------------- | --------------------------------------------- | ------------------------------------------------------------- |
| Service not starting | `docker logs linearvoid-<svc> --tail 50` | Check `.env` vars, volume mounts, port conflicts |
| Playout stall / slate mode | `curl http://<host>:8081/health` — mode=slate | Check playlist file, media paths, disk space |
| Chat not connecting | `curl http://<host>:3000/health` | Check Redis: `docker exec linearvoid-redis redis-cli ping` |
| Admin 401 loop | Cookie not sent | Verify `X-Forwarded-Proto: https`, nginx auth_request headers |
| Upload fails (413) | File too large | Check nginx `client_max_body_size` (should be 64m+) |
| HLS not playing | `curl http://<host>/hls/playlist.m3u8` | Check playout health, HLS dir permissions |
| Prometheus no targets | `curl http://<host>:9090/targets` | Verify container network, scrape config |
| Grafana no data | Check datasource | Verify Prometheus URL `http://prometheus:9090` provisioned |
- Proxy-layer body size limits returning 413 on upload.
- Health checks failing before service warmup is complete.
MONITORING.md
+53 -99
View File
@@ -1,122 +1,76 @@
# Monitoring Guide
This guide covers Prometheus and Grafana setup for Linear Void.
Prometheus + Grafana monitoring for Linear Void.
## Current Monitoring Components
## Architecture
- Prometheus container: linearvoid-prometheus
- Grafana container: linearvoid-grafana
- Metrics target: chat service /metrics
- Base scrape config: monitoring/prometheus/prometheus.yml
## Prometheus Configuration
Current scrape config:
- Job chat -> chat:3000
- Job prometheus -> prometheus:9090
- Global scrape interval: 15s
File:
- monitoring/prometheus/prometheus.yml
## Start Monitoring Stack
With compose stack up:
- Prometheus UI: <http://localhost:9090>
- Grafana UI: <http://localhost:3001>
Compose services include prometheus and grafana in both docker-compose.yml and docker-compose.prod.yml.
## Grafana Provisioning
Current mount paths:
- Dashboards directory: monitoring/grafana/dashboards
- Datasources directory: monitoring/grafana/datasources
Notes:
- The dashboards directory includes a Prometheus overview dashboard file.
- Datasource provisioning can be done with a YAML file in monitoring/grafana/datasources, or by configuring a datasource in Grafana UI.
Example datasource provisioning file (create if needed):
```yaml
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
```
┌─────────────┐ ┌──────────────┐ ┌───────────┐
│ Exporters │───▶│ Prometheus │───▶│ Grafana │
│ (7 targets) │ │ :9090 │ │ :3001 │
└─────────────┘ └──────┬───────┘ └───────────┘
│
alerts.yml (8 rules)
```
## Dashboard References
## Scrape Targets
- Provisioned dashboard: [monitoring/grafana/dashboards/dashboard.json](monitoring/grafana/dashboards/dashboard.json)
- Root dashboard snapshot: [monitoring/grafana/dashboard.json](monitoring/grafana/dashboard.json)
| Job | Target | Port | Metrics |
| ---------- | ----------------- | ---- | ------------------------------------------------------------- |
| chat | chat | 3000 | WebSocket clients, messages, DMs, moderation, Redis, sessions |
| playout | playout | 8081 | Restarts, mode (live/slate) |
| nginx | nginx-exporter | 9113 | Requests, connections |
| redis | redis-exporter | 9121 | Memory, clients, commands |
| postgres | postgres-exporter | 9187 | Connections, queries, locks |
| node | node-exporter | 9100 | CPU, memory, disk, network |
| prometheus | prometheus | 9090 | Self-monitoring |
Grafana screenshot:
Config: `monitoring/prometheus/prometheus.yml` (15s scrape interval)
![Grafana dashboard screenshot](docs/img/grafana-screenshot.png)
## Alerting Rules
## Key Metrics to Track
8 rules in `monitoring/prometheus/alerts.yml`:
### Chat/Application
| Alert | Severity | Expression |
| -------------------- | -------- | -------------------------------------- |
| ChatDown | critical | `up{job="chat"} == 0` for 2m |
| PlayoutDown | critical | `up{job="playout"} == 0` for 2m |
| RedisDown | warning | `linearvoid_chat_redis_up == 0` for 2m |
| PlayoutSlateActive | warning | `linearvoid_playout_mode == 1` for 1m |
| HighChatLatency | warning | message rate <0.01/s for 5m |
| PrometheusWALFailing | warning | WAL write failures >0 for 5m |
| HighCPU | warning | CPU >80% for 10m |
| LowDiskSpace | warning | Disk <10% free for 5m |
- linearvoid_chat_connected_clients
- linearvoid_chat_stored_messages_total
- linearvoid_chat_moderation_actions_total
- linearvoid_chat_redis_enabled
- linearvoid_chat_redis_up
- linearvoid_chat_active_admin_sessions
## Access
### Prometheus Runtime
- Prometheus: `http://<host>:9090` (targets, graph, alerts)
- Grafana: `http://<host>:3001` (default: admin/admin, change via `GRAFANA_PASSWORD`)
- up
- process_cpu_seconds_total
- process_resident_memory_bytes
- prometheus_tsdb_head_series
- prometheus_tsdb_wal_writes_failed_total
- prometheus_tsdb_compactions_failed_total
## Dashboards
## Suggested Alerts
- **Linear Void app**: `monitoring/grafana/dashboard.json` — clients, messages, moderation, Redis, sessions, scrape health
- **Prometheus ops**: `monitoring/grafana/dashboards/dashboard.json` — TSDB, CPU/memory, WAL, compactions, query latency
Create a Prometheus rules file (for example monitoring/prometheus/alerts.yml) and wire it into Prometheus.
## Key Metrics
Recommended baseline alerts:
- Chat target down for 2m
- Prometheus target down for 2m
- Redis indicator down for 2m
- TSDB WAL/compaction failure rate above zero
- Elevated handler/query latency p99
## Validate Monitoring End-to-End
1. Verify /metrics through nginx:
- <https://linearvoid.tv/metrics>
1. Confirm targets in Prometheus:
- Status -> Targets page in Prometheus UI
1. Confirm dashboard data in Grafana:
- Import or open provisioned dashboards
- `linearvoid_chat_connected_clients` — active WebSocket connections
- `linearvoid_chat_messages_total` — total stored messages
- `linearvoid_chat_dms_total` — private messages sent
- `linearvoid_chat_redis_up` — Redis reachable (0/1)
- `linearvoid_playout_restarts_total` — FFmpeg restart counter
- `linearvoid_playout_mode` — 0=live, 1=slate, 2=degraded
- `linearvoid_chat_admin_sessions_active` — active admin logins
## Troubleshooting
### No chat metrics in Prometheus
- Check chat container health and logs.
- Confirm /metrics is reachable from Prometheus container.
- Verify scrape target is chat:3000 in prometheus.yml.
| Symptom | Check |
| ----------------------- | -------------------------------------------------------------------- |
| No chat metrics | `curl http://chat:3000/metrics` from Prometheus container |
| No exporter metrics | Verify exporter containers running: `docker ps \| grep exporter` |
| Grafana no data | Datasource URL must be `http://prometheus:9090` |
| Prometheus targets down | Check container network: all must be on `linearvoid-net` |
| Alert not firing | Check Prometheus → Alerts tab, verify `rule_files` in prometheus.yml |
### Grafana shows no data
SECURITY.md
+64 -98
View File
@@ -1,127 +1,93 @@
# Security Guide
This document summarizes security controls and best practices for Linear Void.
Layered security model for Linear Void.
## Security Model Overview
## Threat Model
Security is layered across:
```
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Client │───▶│Cloudflare│───▶│ nginx │───▶│ Backend │
│ (public) │ │ WAF+CDN │ │ proxy │ │ chat │
└──────────┘ └──────────┘ └────┬─────┘ └────┬─────┘
│ │
┌──────┴──────┐ ┌─────┴──────┐
│ /health │ │ /api/admin │
│ /metrics │ │ /ws (chat) │
│ /hls/* │ │ /api/chat/* │
│ (public) │ │ (auth req) │
└─────────────┘ └────────────┘
```
- Cloudflare edge
- nginx reverse proxy
- chat/admin backend
- container and host operations
## Authentication & Sessions
## Authentication and Sessions
- Login: `POST /api/admin/login` → cookie `linearvoid_admin_session`
- Cookie flags: `HttpOnly`, `SameSite=Strict`, `Secure` (over HTTPS)
- Server-side expiry via `ADMIN_SESSION_TTL_HOURS`
- Rate limit: 5 attempts/IP/5min via Redis
Admin access is session-cookie based.
## Transport Security (TLS/HSTS)
- Login endpoint: /api/admin/login
- Session endpoint: /api/admin/session
- Logout endpoint: /api/admin/logout
- Cookie: linearvoid_admin_session
- Cookie flags: HttpOnly, SameSite=Strict, Secure over HTTPS
- Session expiry enforced server-side
- Cloudflare TLS mode: **Full (strict)** — valid origin cert required
- nginx listens on 80 internally; Cloudflare handles TLS termination
- HSTS header set by Cloudflare: `max-age=31536000; includeSubDomains; preload`
- WSS upgrade path: `/ws` through nginx → chat:3000
Best practices:
## Network Isolation
- Set strong ADMIN_USERNAME and ADMIN_PASSWORD values.
- Rotate admin credentials periodically.
- Keep ADMIN_SESSION_TTL_HOURS aligned with operational policy.
## Transport Security
- Enforce HTTPS for app/admin endpoints.
- Use WSS for chat WebSocket.
- Keep Cloudflare TLS mode strict and valid at origin.
- All backend services on internal Docker network (`linearvoid-net`)
- Only nginx port exposed to host (443/80)
- Database, Redis, exporters inaccessible from outside
- `docker-compose.prod.yml` keeps ports internal via `expose:` not `ports:`
## Reverse Proxy Controls
nginx controls include:
- `auth_request /api/admin/auth` gate on all `/admin` and `/api/admin` paths
- Explicit `client_max_body_size 64m` for resumable upload chunks
- WebSocket upgrade headers: `Upgrade`, `Connection` preserved
- CORS headers on HLS: `Access-Control-Allow-Origin *`
- XSS protection: CSP headers served by Cloudflare
- auth_request guard for /admin and /api/admin paths
- dedicated unauthorized behavior for admin API responses
- explicit websocket upgrade headers for /ws
- upload body-size limits for resumable media chunk uploads
## Rate Limiting
Best practices:
| Resource | Limit | Window |
| ----------------- | ----- | ------ |
| WebSocket connect | 5/IP | 60s |
| Chat messages | 10/IP | 10s |
| Admin login | 5/IP | 5min |
- Keep proxy body-size and timeout settings aligned across all proxy layers.
- Avoid exposing backend service ports publicly unless required.
Implemented via Redis `INCR` + `EXPIRE`.
## Input Validation and Upload Safety
## Secrets Management
Server-side protections include:
- `.env` — local dev only, **gitignored**
- Gitea Actions secrets (`CI_*` prefix) for CI/CD
- Never commit: passwords, tokens, API keys, R2 credentials
- Production `.env` provisioned manually on host, never checked in
- Rotate admin password quarterly
- extension allow-list for media uploads
- MIME checks
- file-size ceiling (2GB)
- resumable chunk bounds and chunk-index validation
- media path safety checks on delete
## Data Protection
Best practices:
- Keep upload chunk size bounded.
- Reject unsupported content types consistently.
- Monitor 4xx/5xx rates on upload routes.
## Rate Limiting and Abuse Controls
- Chat rate limiting uses Redis INCR + EXPIRE windows.
- Current baseline: more than 10 messages in 10 seconds is throttled.
- Moderation actions are logged for auditability.
Best practices:
- Tune thresholds by observed abuse patterns.
- Keep moderation logs reviewed and retained per policy.
## Infrastructure and Secrets
- Store secrets in .env (not in committed source).
- Use Gitea Actions secrets for CI/CD credentials and admin credentials.
- Restrict Docker host control plane exposure.
- Limit who can read deployment environment files.
## Cloudflare and Edge Security
Recommended baseline:
- WAF enabled
- Rate limits on sensitive paths
- Bot protection where appropriate
- Cache rules scoped carefully to avoid caching private/control-plane content
## Data Protection and Retention
- Persistent data: postgres, redis append-only, chat state, playlist/media metadata
- Backup and restore drills should be periodic
- Restore procedures should be tested in isolated environments
## Logging and Monitoring for Security
Track and alert on:
- repeated failed admin login attempts
- spikes in 401 and 403 responses
- sudden upload errors (413/400/500 patterns)
- unusual websocket disconnect or message-rate anomalies
- PostgreSQL: chat history, moderation logs, media metadata
- Redis: append-only persistence for presence, rate limits, sessions
- HLS segments: ephemeral, regenerated by playout
- Backups: `scripts/backup-db.sh` (daily), `scripts/backup-weekly.sh`, `scripts/verify-backups.sh`
## Hardening Checklist
- [ ] HTTPS/WSS enforced end-to-end
- [ ] HTTPS/WSS enforced end-to-end (Cloudflare Full strict)
- [ ] Strong admin credentials configured
- [ ] Secrets only in env/secrets stores
- [ ] Secrets only in env/secrets stores (never committed)
- [ ] Proxy auth and websocket headers validated
- [ ] Upload limits and validation active
- [ ] Rate limiting active and tested
- [ ] Upload limits and MIME validation active
- [ ] Rate limiting active (WS, messages, login)
- [ ] Cloudflare WAF/rate-limit rules active
- [ ] Backup/restore drills completed
- [ ] Security-related monitoring alerts routed
- [ ] Security alerts routed (Prometheus alertmanager)
## Incident Response Baseline
## Incident Response
1. Detect issue (alerts/logs/user reports).
2. Triage scope (edge, proxy, app, data plane).
3. Contain impact (rate limit, credential rotate, route disable).
4. Recover services and verify health.
5. Record timeline and corrective actions.
1. Detect — alerts (ChatDown, RedisDown, HighChatLatency) or user reports
2. Triage — edge (Cloudflare), proxy (nginx), app (chat/playout), data (Redis/Postgres)
3. Contain — rate limit, credential rotate, service restart, slate mode activation
4. Recover — verify health endpoints, check Prometheus targets, validate HLS output
5. Record — timeline + corrective actions in `docs/incident-runbook.md`
docs/architecture/README.md
+105 -2
View File
@@ -259,11 +259,114 @@ Detailed black-box descriptions, level-2/3 views, and code locations are in `adr
### Level 2
Deferred to implementation (see ADR for component responsibilities, interfaces, quality attributes, and risks). Playout enforces normalization; Delivery layer focuses on cache rules and immutable naming.
Detailed component diagram with code locations and persistence.
```mermaid
graph TB
subgraph "Cloudflare Edge"
CF[CDN + WAF<br/>Cache Rules]
R2[R2 Bucket<br/>media + HLS]
PAGES[Pages<br/>SPA hosting]
end
subgraph "Docker Host (Hetzner CX22)"
NGINX[nginx<br/>:80/:443<br/>proxy + auth]
CHAT[chat service<br/>:3000<br/>Go + gorilla/websocket]
PLAYOUT[playout service<br/>:8081<br/>Go + FFmpeg]
REDIS[Redis 7<br/>:6379<br/>presence/rate/pubsub]
DB[PostgreSQL 15<br/>:5432<br/>chat history/moderation]
end
subgraph "External"
VIEWER[Viewer Browser<br/>hls.js + WebSocket]
ADMIN[Admin Browser<br/>SPA + uploads]
end
VIEWER -->|HTTPS| CF
VIEWER -->|WSS| NGINX
ADMIN -->|HTTPS| CF
CF --> NGINX
NGINX --> CHAT
NGINX --> PLAYOUT
NGINX -->|/hls/*| R2
CHAT --> REDIS
CHAT --> DB
PLAYOUT --> R2
PLAYOUT -->|reads| R2
```
#### Playout Service (`backend/playout/src/playlist.go`)
- **Role**: Continuous HLS encoding loop using FFmpeg concat demuxer
- **Input**: `data/playlist/playlist.txt` — ordered list of MP4 file paths
- **Output**: HLS segments (`.ts`) + live manifest (`.m3u8`) written to `data/hls/`, mirrored to R2
- **Resilience**: Restart counter (>5/60s → activate slate), Prometheus metrics on port 8081
- **Validation**: ffprobe gate on all ingested files (codec, fps, resolution, keyframe interval)
#### Chat Service (`backend/chat/src/main.go`)
- **Role**: WebSocket chat with rooms, DMs, moderation, admin API
- **Persistence**: PostgreSQL 15 via GORM (messages, moderation logs, media metadata). SQLite for dev fallback
- **Redis**: Presence tracking, rate limiting, pub/sub for multi-instance fan-out
- **Auth**: Session-cookie admin auth with Redis-backed expiry
- **Metrics**: Connected clients, stored messages, DMs, moderation actions on `/metrics`
#### Frontend SPA (`frontend/`)
- **Viewer**: `src/main.ts` — hls.js player, WebSocket chat, presence indicators, schedule overlay
- **Admin**: `src/admin.ts` — 9 dynamic tabs (dashboard, upload, playlist, schedule, moderation, chat-log, users, analytics, system)
- **Build**: Vite → static output in `frontend/dist/`
#### Reverse Proxy (`infrastructure/nginx/nginx.conf`)
- auth_request gate on `/admin` and `/api/admin`
- WebSocket upgrade for `/ws`
- HLS CORS headers + OPTIONS preflight
- `/health` JSON endpoint, `/nginx-status` stub_status
### Level 3
Deferred to implementation (see ADR for component responsibilities, interfaces, quality attributes, and risks). Playout enforces normalization; Delivery layer focuses on cache rules and immutable naming.
Module-level breakdown per service.
#### Chat Service Modules
| Module | File/Lines | Responsibility |
| ---------------------- | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| **Connection Manager** | `main.go` (hub, clients map) | Track active WebSocket connections per room, fan-out messages |
| **Message Parser** | `main.go` (readPump, handleMessage) | Parse JSON envelope `{type, room?, ...}`, validate fields, route to handler |
| **Room Registry** | `main.go` (rooms map, room broadcasts) | Map room name → client set, handle join/leave/close |
| **DM Router** | `main.go` (handleDirectMessage) | Resolve target user in clients map, deliver private message |
| **Persistence Layer** | `main.go` (GORM models: Message, DirectMessage, ModerationLog) | Store/retrieve chat messages, DMs, moderation events |
| **Redis Subsystem** | `main.go` (redisClient, rate limiting, pub/sub, presence) | INCR+EXPIRE rate limit, SETEX presence, PSUBSCRIBE room fan-out |
| **Auth Handler** | `main.go` (handleAdminLogin, adminAuthMiddleware) | Session cookie creation/validation, login rate limiting |
| **Admin API Router** | `main.go` (handleAdmin\*) | Upload, playlist CRUD, schedule overrides, moderation actions, analytics, system summary |
| **Metrics Exporter** | `main.go` (prometheus.MustRegister) | Expose connected_clients, messages_total, dms_total, moderation_actions, redis_up |
#### Playout Service Modules
| Module | File/Lines | Responsibility |
| -------------------- | ---------------------------------------------- | ------------------------------------------------------------------ |
| **Playlist Parser** | `playlist.go` (parsePlaylist) | Read `playlist.txt`, skip comments/blank lines, resolve file paths |
| **FFmpeg Manager** | `playlist.go` (runPlayout, restart logic) | Spawn/monitor FFmpeg concat process, restart on crash |
| **Slate Controller** | `playlist.go` (activateSlate, deactivateSlate) | Switch output to slate loop on excessive failures |
| **R2 Sync** | `playlist.go` (syncToR2) | Upload HLS segments + manifest to R2 bucket |
| **Health Endpoint** | `playlist.go` (healthHandler) | `/health` → JSON with mode, segment count, uptime |
| **Metrics Exporter** | `playlist.go` (prometheus) | Expose restarts_total, mode (live/slate) |
#### Frontend Modules
| Module | File | Responsibility |
| ----------------------- | ----------------------------------------------- | ----------------------------------------------------------------- |
| **HLS Player** | `src/main.ts` | hls.js initialization, attach to video element, error recovery |
| **WebSocket Client** | `src/main.ts` | Connect/disconnect, parse envelope, dispatch to UI handlers |
| **Chat UI** | `src/main.ts` | Message list rendering, room tabs, DM panel, presence dots |
| **Schedule Overlay** | `src/main.ts` | Poll `/api/schedule`, render current/next program |
| **Admin Router** | `src/admin.ts` | Tab-based navigation, dynamic page loading |
| **Upload Manager** | `src/admin.ts` | Multipart upload with chunk progress, ffprobe validation feedback |
| **Playlist Editor** | `src/admin.ts` | Drag-reorder, add/remove items, save to `/api/admin/playlist` |
| **Moderation Panel** | `src/admin.ts` + `public/admin/moderation.html` | Inline mute/ban, action log viewer |
| **Analytics Dashboard** | `src/admin.ts` + `public/admin/analytics.html` | Message counts, top rooms, moderation stats |
| **System Config** | `src/admin.ts` + `public/admin/system.html` | Service health, config metadata, restart buttons |
## Runtime View