Add comprehensive documentation for Linear Void project

- Created README.md in docs folder outlining project structure and documentation links. - Added SECURITY.md detailing the security model, threat model, authentication, transport security, network isolation, and incident response procedures. - Introduced SETUP.md providing a detailed setup guide for local and production environments, including prerequisites, environment setup, and Gitea integration. - Added STREAMING.md outlining the streaming pipeline, including production flow, source-media contract, playout and HLS packaging, cache contract, player behavior, and monitoring. - Updated architecture documentation to include links to new SETUP.md and STREAMING.md files for better navigation.
2026-08-08 14:52:19 +02:00
1 parent 6b9830de05
commit 926a663c23
@@ -0,0 +1,91 @@
+# Monitoring Guide
+
+Prometheus + Grafana monitoring for Linear Void.
+
+## Architecture
+
+```text
+┌──────────────┐     ┌──────────────┐    ┌───────────┐
+│  Exporters   │───> │  Prometheus  │───>│  Grafana  │
+│  (7 targets) │     │  :9090       │    │  :3001    │
+└──────────────┘     └───────┬──────┘    └───────────┘
+                             │
+                   alerts.yml (8 rules)
+```
+
+## Scrape Targets
+
+| Job        | Target            | Port | Metrics                                                       |
+| ---------- | ----------------- | ---- | ------------------------------------------------------------- |
+| chat       | chat              | 3000 | WebSocket clients, messages, DMs, moderation, Redis, sessions |
+| playout    | playout           | 8081 | Restarts, mode (live/slate)                                   |
+| nginx      | nginx-exporter    | 9113 | Requests, connections                                         |
+| redis      | redis-exporter    | 9121 | Memory, clients, commands                                     |
+| postgres   | postgres-exporter | 9187 | Connections, queries, locks                                   |
+| node       | node-exporter     | 9100 | CPU, memory, disk, network                                    |
+| prometheus | prometheus        | 9090 | Self-monitoring                                               |
+
+Config: `monitoring/prometheus/prometheus.yml` (15s scrape interval)
+
+## Alerting Rules
+
+8 rules in `monitoring/prometheus/alerts.yml`:
+
+| Alert                | Severity | Expression                             |
+| -------------------- | -------- | -------------------------------------- |
+| ChatDown             | critical | `up{job="chat"} == 0` for 2m           |
+| PlayoutDown          | critical | `up{job="playout"} == 0` for 2m        |
+| RedisDown            | warning  | `linearvoid_chat_redis_up == 0` for 2m |
+| PlayoutSlateActive   | warning  | `linearvoid_playout_mode == 1` for 1m  |
+| HighChatLatency      | warning  | message rate <0.01/s for 5m            |
+| PrometheusWALFailing | warning  | WAL write failures >0 for 5m           |
+| HighCPU              | warning  | CPU >80% for 10m                       |
+| LowDiskSpace         | warning  | Disk <10% free for 5m                  |
+
+## Access
+
+- Prometheus: `http://<host>:9090` (targets, graph, alerts)
+- Grafana: `http://<host>:3001` (default: admin/admin, change via `GRAFANA_PASSWORD`)
+
+## Dashboards
+
+- **Linear Void app**: `monitoring/grafana/dashboard.json` — clients, messages, moderation, Redis, sessions, scrape health
+- **Prometheus ops**: `monitoring/grafana/dashboards/dashboard.json` — TSDB, CPU/memory, WAL, compactions, query latency
+
+## Key Metrics
+
+- `linearvoid_chat_connected_clients` — active WebSocket connections
+- `linearvoid_chat_messages_total` — total stored messages
+- `linearvoid_chat_dms_total` — private messages sent
+- `linearvoid_chat_redis_up` — Redis reachable (0/1)
+- `linearvoid_playout_restarts_total` — FFmpeg restart counter
+- `linearvoid_playout_mode` — 0=live, 1=slate, 2=degraded
+- `linearvoid_chat_admin_sessions_active` — active admin logins
+
+## Troubleshooting
+
+| Symptom                 | Check                                                                |
+| ----------------------- | -------------------------------------------------------------------- |
+| No chat metrics         | `curl http://chat:3000/metrics` from Prometheus container            |
+| No exporter metrics     | Verify exporter containers running: `docker ps \| grep exporter`     |
+| Grafana no data         | Datasource URL must be `http://prometheus:9090`                      |
+| Prometheus targets down | Check container network: all must be on `linearvoid-net`             |
+| Alert not firing        | Check Prometheus → Alerts tab, verify `rule_files` in prometheus.yml |
+
+### Grafana shows no data
+
+- Ensure datasource points to <http://prometheus:9090>.
+- Confirm Prometheus is healthy.
+- Check dashboard query labels match active jobs.
+
+### High cardinality or query slowness
+
+- Reduce unneeded labels.
+- Limit expensive dashboard panels.
+- Tune retention and query windows.
+
+## Operational Recommendations
+
+- Keep alert thresholds tied to actual user impact.
+- Review dashboards after each release that changes endpoints or metrics names.
+- Periodically test alert routing and incident response flow.