diff --git a/API.md b/API.md index 0010c58..a8a8be8 100644 --- a/API.md +++ b/API.md @@ -272,11 +272,36 @@ Prometheus exposition includes connected clients, stored messages, moderation ac ## HTTP Status Conventions -- 200/201: successful request -- 400: malformed payload, invalid input, missing required fields -- 401: unauthorized admin request -- 404: not found -- 405: method not allowed -- 409: logical conflict (for example, mismatched resumed upload) +| Code | Meaning | When | +| ---- | --------------------- | ----------------------------------------------------------- | +| 200 | OK | Successful GET, PUT, or action | +| 201 | Created | Successful POST (upload init, override created) | +| 204 | No Content | Successful DELETE, CORS preflight | +| 400 | Bad Request | Malformed JSON, missing required fields, invalid input | +| 401 | Unauthorized | Missing or expired admin session | +| 403 | Forbidden | Valid session but insufficient permissions | +| 404 | Not Found | Resource doesn't exist | +| 405 | Method Not Allowed | Wrong HTTP method for endpoint | +| 409 | Conflict | Schedule override overlap, upload ID mismatch | +| 413 | Payload Too Large | Upload exceeds 2 GB limit | +| 422 | Unprocessable | File fails normalization check (codec, fps, resolution) | +| 429 | Too Many Requests | Rate limit exceeded (chat messages, WS connections, login) | +| 500 | Internal Server Error | Unexpected server failure | +| 503 | Service Unavailable | Redis unavailable for user actions, playout health degraded | + +## Rate Limiting + +All limits are per-IP (from `X-Forwarded-For` or `RemoteAddr`) unless noted. + +| Resource | Limit | Window | Response | +| --------------------- | ---------------------------------- | ---------------------------------------- | ------------------------------------------------------ | +| Chat messages | `CHAT_RATE_LIMIT_MAX` (default 10) | `CHAT_RATE_LIMIT_WINDOW_S` (default 10s) | 429 + `{"type":"system","text":"Rate limit exceeded"}` | +| WebSocket connections | 5 | 60s | 429 + `"too many connections from this IP"` | +| Admin login | 5 | 5min | 429 + `"too many login attempts, try again later"` | + +Rate limit headers on applicable responses: + +- `X-RateLimit-Limit`: maximum requests per window +- `Retry-After`: seconds until window resets - 413: payload too large - 500: internal server error diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index a55aa92..252533e 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -228,5 +228,19 @@ docker compose -f docker-compose.prod.yml down -v - Missing frontend build artifacts in local-only compose workflows. - Wrong registry credentials or missing login. - Misaligned domain env vars (app/ws/hls) causing mixed-origin errors. + +## Troubleshooting + +| Symptom | Check | Fix | +| -------------------------- | --------------------------------------------- | ------------------------------------------------------------- | +| Service not starting | `docker logs linearvoid- --tail 50` | Check `.env` vars, volume mounts, port conflicts | +| Playout stall / slate mode | `curl http://:8081/health` — mode=slate | Check playlist file, media paths, disk space | +| Chat not connecting | `curl http://:3000/health` | Check Redis: `docker exec linearvoid-redis redis-cli ping` | +| Admin 401 loop | Cookie not sent | Verify `X-Forwarded-Proto: https`, nginx auth_request headers | +| Upload fails (413) | File too large | Check nginx `client_max_body_size` (should be 64m+) | +| HLS not playing | `curl http:///hls/playlist.m3u8` | Check playout health, HLS dir permissions | +| Prometheus no targets | `curl http://:9090/targets` | Verify container network, scrape config | +| Grafana no data | Check datasource | Verify Prometheus URL `http://prometheus:9090` provisioned | + - Proxy-layer body size limits returning 413 on upload. - Health checks failing before service warmup is complete. diff --git a/MONITORING.md b/MONITORING.md index b999879..8c333ac 100644 --- a/MONITORING.md +++ b/MONITORING.md @@ -1,122 +1,76 @@ # Monitoring Guide -This guide covers Prometheus and Grafana setup for Linear Void. +Prometheus + Grafana monitoring for Linear Void. -## Current Monitoring Components +## Architecture -- Prometheus container: linearvoid-prometheus -- Grafana container: linearvoid-grafana -- Metrics target: chat service /metrics -- Base scrape config: monitoring/prometheus/prometheus.yml - -## Prometheus Configuration - -Current scrape config: - -- Job chat -> chat:3000 -- Job prometheus -> prometheus:9090 -- Global scrape interval: 15s - -File: - -- monitoring/prometheus/prometheus.yml - -## Start Monitoring Stack - -With compose stack up: - -- Prometheus UI: -- Grafana UI: - -Compose services include prometheus and grafana in both docker-compose.yml and docker-compose.prod.yml. - -## Grafana Provisioning - -Current mount paths: - -- Dashboards directory: monitoring/grafana/dashboards -- Datasources directory: monitoring/grafana/datasources - -Notes: - -- The dashboards directory includes a Prometheus overview dashboard file. -- Datasource provisioning can be done with a YAML file in monitoring/grafana/datasources, or by configuring a datasource in Grafana UI. - -Example datasource provisioning file (create if needed): - -```yaml -apiVersion: 1 - -datasources: - - name: Prometheus - type: prometheus - access: proxy - url: http://prometheus:9090 - isDefault: true +``` +┌─────────────┐ ┌──────────────┐ ┌───────────┐ +│ Exporters │───▶│ Prometheus │───▶│ Grafana │ +│ (7 targets) │ │ :9090 │ │ :3001 │ +└─────────────┘ └──────┬───────┘ └───────────┘ + │ + alerts.yml (8 rules) ``` -## Dashboard References +## Scrape Targets -- Provisioned dashboard: [monitoring/grafana/dashboards/dashboard.json](monitoring/grafana/dashboards/dashboard.json) -- Root dashboard snapshot: [monitoring/grafana/dashboard.json](monitoring/grafana/dashboard.json) +| Job | Target | Port | Metrics | +| ---------- | ----------------- | ---- | ------------------------------------------------------------- | +| chat | chat | 3000 | WebSocket clients, messages, DMs, moderation, Redis, sessions | +| playout | playout | 8081 | Restarts, mode (live/slate) | +| nginx | nginx-exporter | 9113 | Requests, connections | +| redis | redis-exporter | 9121 | Memory, clients, commands | +| postgres | postgres-exporter | 9187 | Connections, queries, locks | +| node | node-exporter | 9100 | CPU, memory, disk, network | +| prometheus | prometheus | 9090 | Self-monitoring | -Grafana screenshot: +Config: `monitoring/prometheus/prometheus.yml` (15s scrape interval) -![Grafana dashboard screenshot](docs/img/grafana-screenshot.png) +## Alerting Rules -## Key Metrics to Track +8 rules in `monitoring/prometheus/alerts.yml`: -### Chat/Application +| Alert | Severity | Expression | +| -------------------- | -------- | -------------------------------------- | +| ChatDown | critical | `up{job="chat"} == 0` for 2m | +| PlayoutDown | critical | `up{job="playout"} == 0` for 2m | +| RedisDown | warning | `linearvoid_chat_redis_up == 0` for 2m | +| PlayoutSlateActive | warning | `linearvoid_playout_mode == 1` for 1m | +| HighChatLatency | warning | message rate <0.01/s for 5m | +| PrometheusWALFailing | warning | WAL write failures >0 for 5m | +| HighCPU | warning | CPU >80% for 10m | +| LowDiskSpace | warning | Disk <10% free for 5m | -- linearvoid_chat_connected_clients -- linearvoid_chat_stored_messages_total -- linearvoid_chat_moderation_actions_total -- linearvoid_chat_redis_enabled -- linearvoid_chat_redis_up -- linearvoid_chat_active_admin_sessions +## Access -### Prometheus Runtime +- Prometheus: `http://:9090` (targets, graph, alerts) +- Grafana: `http://:3001` (default: admin/admin, change via `GRAFANA_PASSWORD`) -- up -- process_cpu_seconds_total -- process_resident_memory_bytes -- prometheus_tsdb_head_series -- prometheus_tsdb_wal_writes_failed_total -- prometheus_tsdb_compactions_failed_total +## Dashboards -## Suggested Alerts +- **Linear Void app**: `monitoring/grafana/dashboard.json` — clients, messages, moderation, Redis, sessions, scrape health +- **Prometheus ops**: `monitoring/grafana/dashboards/dashboard.json` — TSDB, CPU/memory, WAL, compactions, query latency -Create a Prometheus rules file (for example monitoring/prometheus/alerts.yml) and wire it into Prometheus. +## Key Metrics -Recommended baseline alerts: - -- Chat target down for 2m -- Prometheus target down for 2m -- Redis indicator down for 2m -- TSDB WAL/compaction failure rate above zero -- Elevated handler/query latency p99 - -## Validate Monitoring End-to-End - -1. Verify /metrics through nginx: - -- - -1. Confirm targets in Prometheus: - -- Status -> Targets page in Prometheus UI - -1. Confirm dashboard data in Grafana: - -- Import or open provisioned dashboards +- `linearvoid_chat_connected_clients` — active WebSocket connections +- `linearvoid_chat_messages_total` — total stored messages +- `linearvoid_chat_dms_total` — private messages sent +- `linearvoid_chat_redis_up` — Redis reachable (0/1) +- `linearvoid_playout_restarts_total` — FFmpeg restart counter +- `linearvoid_playout_mode` — 0=live, 1=slate, 2=degraded +- `linearvoid_chat_admin_sessions_active` — active admin logins ## Troubleshooting -### No chat metrics in Prometheus - -- Check chat container health and logs. -- Confirm /metrics is reachable from Prometheus container. -- Verify scrape target is chat:3000 in prometheus.yml. +| Symptom | Check | +| ----------------------- | -------------------------------------------------------------------- | +| No chat metrics | `curl http://chat:3000/metrics` from Prometheus container | +| No exporter metrics | Verify exporter containers running: `docker ps \| grep exporter` | +| Grafana no data | Datasource URL must be `http://prometheus:9090` | +| Prometheus targets down | Check container network: all must be on `linearvoid-net` | +| Alert not firing | Check Prometheus → Alerts tab, verify `rule_files` in prometheus.yml | ### Grafana shows no data diff --git a/SECURITY.md b/SECURITY.md index 577008c..4bee662 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,127 +1,93 @@ # Security Guide -This document summarizes security controls and best practices for Linear Void. +Layered security model for Linear Void. -## Security Model Overview +## Threat Model -Security is layered across: +``` +┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ +│ Client │───▶│Cloudflare│───▶│ nginx │───▶│ Backend │ +│ (public) │ │ WAF+CDN │ │ proxy │ │ chat │ +└──────────┘ └──────────┘ └────┬─────┘ └────┬─────┘ + │ │ + ┌──────┴──────┐ ┌─────┴──────┐ + │ /health │ │ /api/admin │ + │ /metrics │ │ /ws (chat) │ + │ /hls/* │ │ /api/chat/* │ + │ (public) │ │ (auth req) │ + └─────────────┘ └────────────┘ +``` -- Cloudflare edge -- nginx reverse proxy -- chat/admin backend -- container and host operations +## Authentication & Sessions -## Authentication and Sessions +- Login: `POST /api/admin/login` → cookie `linearvoid_admin_session` +- Cookie flags: `HttpOnly`, `SameSite=Strict`, `Secure` (over HTTPS) +- Server-side expiry via `ADMIN_SESSION_TTL_HOURS` +- Rate limit: 5 attempts/IP/5min via Redis -Admin access is session-cookie based. +## Transport Security (TLS/HSTS) -- Login endpoint: /api/admin/login -- Session endpoint: /api/admin/session -- Logout endpoint: /api/admin/logout -- Cookie: linearvoid_admin_session -- Cookie flags: HttpOnly, SameSite=Strict, Secure over HTTPS -- Session expiry enforced server-side +- Cloudflare TLS mode: **Full (strict)** — valid origin cert required +- nginx listens on 80 internally; Cloudflare handles TLS termination +- HSTS header set by Cloudflare: `max-age=31536000; includeSubDomains; preload` +- WSS upgrade path: `/ws` through nginx → chat:3000 -Best practices: +## Network Isolation -- Set strong ADMIN_USERNAME and ADMIN_PASSWORD values. -- Rotate admin credentials periodically. -- Keep ADMIN_SESSION_TTL_HOURS aligned with operational policy. - -## Transport Security - -- Enforce HTTPS for app/admin endpoints. -- Use WSS for chat WebSocket. -- Keep Cloudflare TLS mode strict and valid at origin. +- All backend services on internal Docker network (`linearvoid-net`) +- Only nginx port exposed to host (443/80) +- Database, Redis, exporters inaccessible from outside +- `docker-compose.prod.yml` keeps ports internal via `expose:` not `ports:` ## Reverse Proxy Controls -nginx controls include: +- `auth_request /api/admin/auth` gate on all `/admin` and `/api/admin` paths +- Explicit `client_max_body_size 64m` for resumable upload chunks +- WebSocket upgrade headers: `Upgrade`, `Connection` preserved +- CORS headers on HLS: `Access-Control-Allow-Origin *` +- XSS protection: CSP headers served by Cloudflare -- auth_request guard for /admin and /api/admin paths -- dedicated unauthorized behavior for admin API responses -- explicit websocket upgrade headers for /ws -- upload body-size limits for resumable media chunk uploads +## Rate Limiting -Best practices: +| Resource | Limit | Window | +| ----------------- | ----- | ------ | +| WebSocket connect | 5/IP | 60s | +| Chat messages | 10/IP | 10s | +| Admin login | 5/IP | 5min | -- Keep proxy body-size and timeout settings aligned across all proxy layers. -- Avoid exposing backend service ports publicly unless required. +Implemented via Redis `INCR` + `EXPIRE`. -## Input Validation and Upload Safety +## Secrets Management -Server-side protections include: +- `.env` — local dev only, **gitignored** +- Gitea Actions secrets (`CI_*` prefix) for CI/CD +- Never commit: passwords, tokens, API keys, R2 credentials +- Production `.env` provisioned manually on host, never checked in +- Rotate admin password quarterly -- extension allow-list for media uploads -- MIME checks -- file-size ceiling (2GB) -- resumable chunk bounds and chunk-index validation -- media path safety checks on delete +## Data Protection -Best practices: - -- Keep upload chunk size bounded. -- Reject unsupported content types consistently. -- Monitor 4xx/5xx rates on upload routes. - -## Rate Limiting and Abuse Controls - -- Chat rate limiting uses Redis INCR + EXPIRE windows. -- Current baseline: more than 10 messages in 10 seconds is throttled. -- Moderation actions are logged for auditability. - -Best practices: - -- Tune thresholds by observed abuse patterns. -- Keep moderation logs reviewed and retained per policy. - -## Infrastructure and Secrets - -- Store secrets in .env (not in committed source). -- Use Gitea Actions secrets for CI/CD credentials and admin credentials. -- Restrict Docker host control plane exposure. -- Limit who can read deployment environment files. - -## Cloudflare and Edge Security - -Recommended baseline: - -- WAF enabled -- Rate limits on sensitive paths -- Bot protection where appropriate -- Cache rules scoped carefully to avoid caching private/control-plane content - -## Data Protection and Retention - -- Persistent data: postgres, redis append-only, chat state, playlist/media metadata -- Backup and restore drills should be periodic -- Restore procedures should be tested in isolated environments - -## Logging and Monitoring for Security - -Track and alert on: - -- repeated failed admin login attempts -- spikes in 401 and 403 responses -- sudden upload errors (413/400/500 patterns) -- unusual websocket disconnect or message-rate anomalies +- PostgreSQL: chat history, moderation logs, media metadata +- Redis: append-only persistence for presence, rate limits, sessions +- HLS segments: ephemeral, regenerated by playout +- Backups: `scripts/backup-db.sh` (daily), `scripts/backup-weekly.sh`, `scripts/verify-backups.sh` ## Hardening Checklist -- [ ] HTTPS/WSS enforced end-to-end +- [ ] HTTPS/WSS enforced end-to-end (Cloudflare Full strict) - [ ] Strong admin credentials configured -- [ ] Secrets only in env/secrets stores +- [ ] Secrets only in env/secrets stores (never committed) - [ ] Proxy auth and websocket headers validated -- [ ] Upload limits and validation active -- [ ] Rate limiting active and tested +- [ ] Upload limits and MIME validation active +- [ ] Rate limiting active (WS, messages, login) - [ ] Cloudflare WAF/rate-limit rules active - [ ] Backup/restore drills completed -- [ ] Security-related monitoring alerts routed +- [ ] Security alerts routed (Prometheus alertmanager) -## Incident Response Baseline +## Incident Response -1. Detect issue (alerts/logs/user reports). -2. Triage scope (edge, proxy, app, data plane). -3. Contain impact (rate limit, credential rotate, route disable). -4. Recover services and verify health. -5. Record timeline and corrective actions. +1. Detect — alerts (ChatDown, RedisDown, HighChatLatency) or user reports +2. Triage — edge (Cloudflare), proxy (nginx), app (chat/playout), data (Redis/Postgres) +3. Contain — rate limit, credential rotate, service restart, slate mode activation +4. Recover — verify health endpoints, check Prometheus targets, validate HLS output +5. Record — timeline + corrective actions in `docs/incident-runbook.md` diff --git a/docs/architecture/README.md b/docs/architecture/README.md index c6c18bf..4924aab 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -259,11 +259,114 @@ Detailed black-box descriptions, level-2/3 views, and code locations are in `adr ### Level 2 -Deferred to implementation (see ADR for component responsibilities, interfaces, quality attributes, and risks). Playout enforces normalization; Delivery layer focuses on cache rules and immutable naming. +Detailed component diagram with code locations and persistence. + +```mermaid +graph TB + subgraph "Cloudflare Edge" + CF[CDN + WAF
Cache Rules] + R2[R2 Bucket
media + HLS] + PAGES[Pages
SPA hosting] + end + + subgraph "Docker Host (Hetzner CX22)" + NGINX[nginx
:80/:443
proxy + auth] + CHAT[chat service
:3000
Go + gorilla/websocket] + PLAYOUT[playout service
:8081
Go + FFmpeg] + REDIS[Redis 7
:6379
presence/rate/pubsub] + DB[PostgreSQL 15
:5432
chat history/moderation] + end + + subgraph "External" + VIEWER[Viewer Browser
hls.js + WebSocket] + ADMIN[Admin Browser
SPA + uploads] + end + + VIEWER -->|HTTPS| CF + VIEWER -->|WSS| NGINX + ADMIN -->|HTTPS| CF + CF --> NGINX + NGINX --> CHAT + NGINX --> PLAYOUT + NGINX -->|/hls/*| R2 + CHAT --> REDIS + CHAT --> DB + PLAYOUT --> R2 + PLAYOUT -->|reads| R2 +``` + +#### Playout Service (`backend/playout/src/playlist.go`) + +- **Role**: Continuous HLS encoding loop using FFmpeg concat demuxer +- **Input**: `data/playlist/playlist.txt` — ordered list of MP4 file paths +- **Output**: HLS segments (`.ts`) + live manifest (`.m3u8`) written to `data/hls/`, mirrored to R2 +- **Resilience**: Restart counter (>5/60s → activate slate), Prometheus metrics on port 8081 +- **Validation**: ffprobe gate on all ingested files (codec, fps, resolution, keyframe interval) + +#### Chat Service (`backend/chat/src/main.go`) + +- **Role**: WebSocket chat with rooms, DMs, moderation, admin API +- **Persistence**: PostgreSQL 15 via GORM (messages, moderation logs, media metadata). SQLite for dev fallback +- **Redis**: Presence tracking, rate limiting, pub/sub for multi-instance fan-out +- **Auth**: Session-cookie admin auth with Redis-backed expiry +- **Metrics**: Connected clients, stored messages, DMs, moderation actions on `/metrics` + +#### Frontend SPA (`frontend/`) + +- **Viewer**: `src/main.ts` — hls.js player, WebSocket chat, presence indicators, schedule overlay +- **Admin**: `src/admin.ts` — 9 dynamic tabs (dashboard, upload, playlist, schedule, moderation, chat-log, users, analytics, system) +- **Build**: Vite → static output in `frontend/dist/` + +#### Reverse Proxy (`infrastructure/nginx/nginx.conf`) + +- auth_request gate on `/admin` and `/api/admin` +- WebSocket upgrade for `/ws` +- HLS CORS headers + OPTIONS preflight +- `/health` JSON endpoint, `/nginx-status` stub_status ### Level 3 -Deferred to implementation (see ADR for component responsibilities, interfaces, quality attributes, and risks). Playout enforces normalization; Delivery layer focuses on cache rules and immutable naming. +Module-level breakdown per service. + +#### Chat Service Modules + +| Module | File/Lines | Responsibility | +| ---------------------- | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | +| **Connection Manager** | `main.go` (hub, clients map) | Track active WebSocket connections per room, fan-out messages | +| **Message Parser** | `main.go` (readPump, handleMessage) | Parse JSON envelope `{type, room?, ...}`, validate fields, route to handler | +| **Room Registry** | `main.go` (rooms map, room broadcasts) | Map room name → client set, handle join/leave/close | +| **DM Router** | `main.go` (handleDirectMessage) | Resolve target user in clients map, deliver private message | +| **Persistence Layer** | `main.go` (GORM models: Message, DirectMessage, ModerationLog) | Store/retrieve chat messages, DMs, moderation events | +| **Redis Subsystem** | `main.go` (redisClient, rate limiting, pub/sub, presence) | INCR+EXPIRE rate limit, SETEX presence, PSUBSCRIBE room fan-out | +| **Auth Handler** | `main.go` (handleAdminLogin, adminAuthMiddleware) | Session cookie creation/validation, login rate limiting | +| **Admin API Router** | `main.go` (handleAdmin\*) | Upload, playlist CRUD, schedule overrides, moderation actions, analytics, system summary | +| **Metrics Exporter** | `main.go` (prometheus.MustRegister) | Expose connected_clients, messages_total, dms_total, moderation_actions, redis_up | + +#### Playout Service Modules + +| Module | File/Lines | Responsibility | +| -------------------- | ---------------------------------------------- | ------------------------------------------------------------------ | +| **Playlist Parser** | `playlist.go` (parsePlaylist) | Read `playlist.txt`, skip comments/blank lines, resolve file paths | +| **FFmpeg Manager** | `playlist.go` (runPlayout, restart logic) | Spawn/monitor FFmpeg concat process, restart on crash | +| **Slate Controller** | `playlist.go` (activateSlate, deactivateSlate) | Switch output to slate loop on excessive failures | +| **R2 Sync** | `playlist.go` (syncToR2) | Upload HLS segments + manifest to R2 bucket | +| **Health Endpoint** | `playlist.go` (healthHandler) | `/health` → JSON with mode, segment count, uptime | +| **Metrics Exporter** | `playlist.go` (prometheus) | Expose restarts_total, mode (live/slate) | + +#### Frontend Modules + +| Module | File | Responsibility | +| ----------------------- | ----------------------------------------------- | ----------------------------------------------------------------- | +| **HLS Player** | `src/main.ts` | hls.js initialization, attach to video element, error recovery | +| **WebSocket Client** | `src/main.ts` | Connect/disconnect, parse envelope, dispatch to UI handlers | +| **Chat UI** | `src/main.ts` | Message list rendering, room tabs, DM panel, presence dots | +| **Schedule Overlay** | `src/main.ts` | Poll `/api/schedule`, render current/next program | +| **Admin Router** | `src/admin.ts` | Tab-based navigation, dynamic page loading | +| **Upload Manager** | `src/admin.ts` | Multipart upload with chunk progress, ffprobe validation feedback | +| **Playlist Editor** | `src/admin.ts` | Drag-reorder, add/remove items, save to `/api/admin/playlist` | +| **Moderation Panel** | `src/admin.ts` + `public/admin/moderation.html` | Inline mute/ban, action log viewer | +| **Analytics Dashboard** | `src/admin.ts` + `public/admin/analytics.html` | Message counts, top rooms, moderation stats | +| **System Config** | `src/admin.ts` + `public/admin/system.html` | Service health, config metadata, restart buttons | ## Runtime View