zwitschi/linearvoid
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 9 Wiki Activity

docs: complete API error codes, deploy troubleshooting, monitoring/security rewrite, L2+L3 building blocks

Browse Source
This commit is contained in:
zwitschi committed 2026-08-06 20:00:45 +02:00
1 parent 0f81a81029
commit db3bf96847
5 files changed
+267 -205
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

MONITORING.md
+53 -99
View File
@@ -1,122 +1,76 @@
# Monitoring Guide
This guide covers Prometheus and Grafana setup for Linear Void.
Prometheus + Grafana monitoring for Linear Void.
## Current Monitoring Components
## Architecture
- Prometheus container: linearvoid-prometheus
- Grafana container: linearvoid-grafana
- Metrics target: chat service /metrics
- Base scrape config: monitoring/prometheus/prometheus.yml
## Prometheus Configuration
Current scrape config:
- Job chat -> chat:3000
- Job prometheus -> prometheus:9090
- Global scrape interval: 15s
File:
- monitoring/prometheus/prometheus.yml
## Start Monitoring Stack
With compose stack up:
- Prometheus UI: <http://localhost:9090>
- Grafana UI: <http://localhost:3001>
Compose services include prometheus and grafana in both docker-compose.yml and docker-compose.prod.yml.
## Grafana Provisioning
Current mount paths:
- Dashboards directory: monitoring/grafana/dashboards
- Datasources directory: monitoring/grafana/datasources
Notes:
- The dashboards directory includes a Prometheus overview dashboard file.
- Datasource provisioning can be done with a YAML file in monitoring/grafana/datasources, or by configuring a datasource in Grafana UI.
Example datasource provisioning file (create if needed):
```yaml
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
```
┌─────────────┐ ┌──────────────┐ ┌───────────┐
│ Exporters │───▶│ Prometheus │───▶│ Grafana │
│ (7 targets) │ │ :9090 │ │ :3001 │
└─────────────┘ └──────┬───────┘ └───────────┘
│
alerts.yml (8 rules)
```
## Dashboard References
## Scrape Targets
- Provisioned dashboard: [monitoring/grafana/dashboards/dashboard.json](monitoring/grafana/dashboards/dashboard.json)
- Root dashboard snapshot: [monitoring/grafana/dashboard.json](monitoring/grafana/dashboard.json)
| Job | Target | Port | Metrics |
| ---------- | ----------------- | ---- | ------------------------------------------------------------- |
| chat | chat | 3000 | WebSocket clients, messages, DMs, moderation, Redis, sessions |
| playout | playout | 8081 | Restarts, mode (live/slate) |
| nginx | nginx-exporter | 9113 | Requests, connections |
| redis | redis-exporter | 9121 | Memory, clients, commands |
| postgres | postgres-exporter | 9187 | Connections, queries, locks |
| node | node-exporter | 9100 | CPU, memory, disk, network |
| prometheus | prometheus | 9090 | Self-monitoring |
Grafana screenshot:
Config: `monitoring/prometheus/prometheus.yml` (15s scrape interval)
![Grafana dashboard screenshot](docs/img/grafana-screenshot.png)
## Alerting Rules
## Key Metrics to Track
8 rules in `monitoring/prometheus/alerts.yml`:
### Chat/Application
| Alert | Severity | Expression |
| -------------------- | -------- | -------------------------------------- |
| ChatDown | critical | `up{job="chat"} == 0` for 2m |
| PlayoutDown | critical | `up{job="playout"} == 0` for 2m |
| RedisDown | warning | `linearvoid_chat_redis_up == 0` for 2m |
| PlayoutSlateActive | warning | `linearvoid_playout_mode == 1` for 1m |
| HighChatLatency | warning | message rate <0.01/s for 5m |
| PrometheusWALFailing | warning | WAL write failures >0 for 5m |
| HighCPU | warning | CPU >80% for 10m |
| LowDiskSpace | warning | Disk <10% free for 5m |
- linearvoid_chat_connected_clients
- linearvoid_chat_stored_messages_total
- linearvoid_chat_moderation_actions_total
- linearvoid_chat_redis_enabled
- linearvoid_chat_redis_up
- linearvoid_chat_active_admin_sessions
## Access
### Prometheus Runtime
- Prometheus: `http://<host>:9090` (targets, graph, alerts)
- Grafana: `http://<host>:3001` (default: admin/admin, change via `GRAFANA_PASSWORD`)
- up
- process_cpu_seconds_total
- process_resident_memory_bytes
- prometheus_tsdb_head_series
- prometheus_tsdb_wal_writes_failed_total
- prometheus_tsdb_compactions_failed_total
## Dashboards
## Suggested Alerts
- **Linear Void app**: `monitoring/grafana/dashboard.json` — clients, messages, moderation, Redis, sessions, scrape health
- **Prometheus ops**: `monitoring/grafana/dashboards/dashboard.json` — TSDB, CPU/memory, WAL, compactions, query latency
Create a Prometheus rules file (for example monitoring/prometheus/alerts.yml) and wire it into Prometheus.
## Key Metrics
Recommended baseline alerts:
- Chat target down for 2m
- Prometheus target down for 2m
- Redis indicator down for 2m
- TSDB WAL/compaction failure rate above zero
- Elevated handler/query latency p99
## Validate Monitoring End-to-End
1. Verify /metrics through nginx:
- <https://linearvoid.tv/metrics>
1. Confirm targets in Prometheus:
- Status -> Targets page in Prometheus UI
1. Confirm dashboard data in Grafana:
- Import or open provisioned dashboards
- `linearvoid_chat_connected_clients` — active WebSocket connections
- `linearvoid_chat_messages_total` — total stored messages
- `linearvoid_chat_dms_total` — private messages sent
- `linearvoid_chat_redis_up` — Redis reachable (0/1)
- `linearvoid_playout_restarts_total` — FFmpeg restart counter
- `linearvoid_playout_mode` — 0=live, 1=slate, 2=degraded
- `linearvoid_chat_admin_sessions_active` — active admin logins
## Troubleshooting
### No chat metrics in Prometheus
- Check chat container health and logs.
- Confirm /metrics is reachable from Prometheus container.
- Verify scrape target is chat:3000 in prometheus.yml.
| Symptom | Check |
| ----------------------- | -------------------------------------------------------------------- |
| No chat metrics | `curl http://chat:3000/metrics` from Prometheus container |
| No exporter metrics | Verify exporter containers running: `docker ps \| grep exporter` |
| Grafana no data | Datasource URL must be `http://prometheus:9090` |
| Prometheus targets down | Check container network: all must be on `linearvoid-net` |
| Alert not firing | Check Prometheus → Alerts tab, verify `rule_files` in prometheus.yml |
### Grafana shows no data