zwitschi
926a663c23
- Created README.md in docs folder outlining project structure and documentation links. - Added SECURITY.md detailing the security model, threat model, authentication, transport security, network isolation, and incident response procedures. - Introduced SETUP.md providing a detailed setup guide for local and production environments, including prerequisites, environment setup, and Gitea integration. - Added STREAMING.md outlining the streaming pipeline, including production flow, source-media contract, playout and HLS packaging, cache contract, player behavior, and monitoring. - Updated architecture documentation to include links to new SETUP.md and STREAMING.md files for better navigation.
4.5 KiB
4.5 KiB
Monitoring Guide
Prometheus + Grafana monitoring for Linear Void.
Architecture
┌──────────────┐ ┌──────────────┐ ┌───────────┐
│ Exporters │───> │ Prometheus │───>│ Grafana │
│ (7 targets) │ │ :9090 │ │ :3001 │
└──────────────┘ └───────┬──────┘ └───────────┘
│
alerts.yml (8 rules)
Scrape Targets
| Job | Target | Port | Metrics |
|---|---|---|---|
| chat | chat | 3000 | WebSocket clients, messages, DMs, moderation, Redis, sessions |
| playout | playout | 8081 | Restarts, mode (live/slate) |
| nginx | nginx-exporter | 9113 | Requests, connections |
| redis | redis-exporter | 9121 | Memory, clients, commands |
| postgres | postgres-exporter | 9187 | Connections, queries, locks |
| node | node-exporter | 9100 | CPU, memory, disk, network |
| prometheus | prometheus | 9090 | Self-monitoring |
Config: monitoring/prometheus/prometheus.yml (15s scrape interval)
Alerting Rules
8 rules in monitoring/prometheus/alerts.yml:
| Alert | Severity | Expression |
|---|---|---|
| ChatDown | critical | up{job="chat"} == 0 for 2m |
| PlayoutDown | critical | up{job="playout"} == 0 for 2m |
| RedisDown | warning | linearvoid_chat_redis_up == 0 for 2m |
| PlayoutSlateActive | warning | linearvoid_playout_mode == 1 for 1m |
| HighChatLatency | warning | message rate <0.01/s for 5m |
| PrometheusWALFailing | warning | WAL write failures >0 for 5m |
| HighCPU | warning | CPU >80% for 10m |
| LowDiskSpace | warning | Disk <10% free for 5m |
Access
- Prometheus:
http://<host>:9090(targets, graph, alerts) - Grafana:
http://<host>:3001(default: admin/admin, change viaGRAFANA_PASSWORD)
Dashboards
- Linear Void app:
monitoring/grafana/dashboard.json— clients, messages, moderation, Redis, sessions, scrape health - Prometheus ops:
monitoring/grafana/dashboards/dashboard.json— TSDB, CPU/memory, WAL, compactions, query latency
Key Metrics
linearvoid_chat_connected_clients— active WebSocket connectionslinearvoid_chat_messages_total— total stored messageslinearvoid_chat_dms_total— private messages sentlinearvoid_chat_redis_up— Redis reachable (0/1)linearvoid_playout_restarts_total— FFmpeg restart counterlinearvoid_playout_mode— 0=live, 1=slate, 2=degradedlinearvoid_chat_admin_sessions_active— active admin logins
Troubleshooting
| Symptom | Check |
|---|---|
| No chat metrics | curl http://chat:3000/metrics from Prometheus container |
| No exporter metrics | Verify exporter containers running: docker ps | grep exporter |
| Grafana no data | Datasource URL must be http://prometheus:9090 |
| Prometheus targets down | Check container network: all must be on linearvoid-net |
| Alert not firing | Check Prometheus → Alerts tab, verify rule_files in prometheus.yml |
Grafana shows no data
- Ensure datasource points to http://prometheus:9090.
- Confirm Prometheus is healthy.
- Check dashboard query labels match active jobs.
High cardinality or query slowness
- Reduce unneeded labels.
- Limit expensive dashboard panels.
- Tune retention and query windows.
Operational Recommendations
- Keep alert thresholds tied to actual user impact.
- Review dashboards after each release that changes endpoints or metrics names.
- Periodically test alert routing and incident response flow.