zwitschi
33b6296c5e
- Updated README.md to reflect changes in load testing commands and added usage examples for Bash scripts. - Converted PowerShell load testing scripts to Bash equivalents for better cross-platform compatibility. - Added new Bash scripts: run-loadtests.sh, summarize-artillery.sh, and summarize-k6.sh to mirror existing PowerShell functionality. - Updated admin-api.js and ws-chat.yml for consistency and improved performance metrics reporting. - Enhanced error handling and output formatting in summary scripts for better readability.
91 lines
4.9 KiB
Markdown
91 lines
4.9 KiB
Markdown
# Monitoring Guide
|
|
|
|
Prometheus + Grafana monitoring for Linear Void.
|
|
|
|
## Architecture
|
|
|
|
```text
|
|
┌──────────────┐ ┌──────────────┐ ┌───────────┐
|
|
│ Exporters │───> │ Prometheus │───>│ Grafana │
|
|
│ (7 targets) │ │ :9090 │ │ :3001 │
|
|
└──────────────┘ └───────┬──────┘ └───────────┘
|
|
│
|
|
alerts.yml (8 rules)
|
|
```
|
|
|
|
## Scrape Targets
|
|
|
|
| Job | Target | Port | Metrics |
|
|
| ---------- | ----------------- | ---- | ------------------------------------------------------------- |
|
|
| chat | chat | 3000 | WebSocket clients, messages, DMs, moderation, Redis, sessions |
|
|
| playout | playout | 8081 | Restarts, mode (live/slate) |
|
|
| nginx | nginx-exporter | 9113 | Requests, connections |
|
|
| redis | redis-exporter | 9121 | Memory, clients, commands |
|
|
| postgres | postgres-exporter | 9187 | Connections, queries, locks |
|
|
| node | node-exporter | 9100 | CPU, memory, disk, network |
|
|
| prometheus | prometheus | 9090 | Self-monitoring |
|
|
|
|
Config: `monitoring/prometheus/prometheus.yml` (15s scrape interval)
|
|
|
|
## Alerting Rules
|
|
|
|
8 rules in `monitoring/prometheus/alerts.yml`:
|
|
|
|
| Alert | Severity | Expression |
|
|
| -------------------- | -------- | -------------------------------------- |
|
|
| ChatDown | critical | `up{job="chat"} == 0` for 2m |
|
|
| PlayoutDown | critical | `up{job="playout"} == 0` for 2m |
|
|
| RedisDown | warning | `linearvoid_chat_redis_up == 0` for 2m |
|
|
| PlayoutSlateActive | warning | `linearvoid_playout_mode == 1` for 1m |
|
|
| HighChatLatency | warning | message rate <0.01/s for 5m |
|
|
| PrometheusWALFailing | warning | WAL write failures >0 for 5m |
|
|
| HighCPU | warning | CPU >80% for 10m |
|
|
| LowDiskSpace | warning | Disk <10% free for 5m |
|
|
|
|
## Access
|
|
|
|
- Prometheus: `http://<host>:9090` (targets, graph, alerts)
|
|
- Grafana: `http://<host>:3001` (default: admin/admin, change via `GRAFANA_PASSWORD`)
|
|
|
|
## Dashboards
|
|
|
|
- **Linear Void app**: `monitoring/grafana/dashboard.json` — clients, messages, moderation, Redis, sessions, scrape health
|
|
- **Prometheus ops**: `monitoring/grafana/dashboards/dashboard.json` — TSDB, CPU/memory, WAL, compactions, query latency
|
|
|
|
## Metrics Checking
|
|
|
|
See detailed scripts for querying Prometheus scrape targets:
|
|
|
|
- [Linux/macOS](./infrastructure/monitoring/prometheus_curl.md)
|
|
|
|
## Key Metrics
|
|
|
|
- `linearvoid_chat_connected_clients` — active WebSocket connections
|
|
- `linearvoid_chat_messages_total` — total stored messages
|
|
- `linearvoid_chat_dms_total` — private messages sent
|
|
- `linearvoid_chat_redis_up` — Redis reachable (0/1)
|
|
- `linearvoid_playout_restarts_total` — FFmpeg restart counter
|
|
- `linearvoid_playout_mode` — 0=live, 1=slate, 2=degraded
|
|
- `linearvoid_chat_admin_sessions_active` — active admin logins
|
|
|
|
## Troubleshooting
|
|
|
|
| Symptom | Check |
|
|
| ----------------------- | -------------------------------------------------------------------- |
|
|
| No chat metrics | `curl http://chat:3000/metrics` from Prometheus container |
|
|
| No exporter metrics | Verify exporter containers running: `docker ps \| grep exporter` |
|
|
| Grafana no data | Datasource URL must be `http://prometheus:9090` |
|
|
| Prometheus targets down | Check container network: all must be on `linearvoid-net` |
|
|
| Alert not firing | Check Prometheus → Alerts tab, verify `rule_files` in prometheus.yml |
|
|
|
|
**Grafana no data**: ensure datasource points to `http://prometheus:9090`; confirm Prometheus is healthy; check dashboard query labels match active jobs.
|
|
|
|
**High cardinality / slow queries**: reduce unneeded labels; limit expensive panels; tune retention and query windows.
|
|
|
|
## Operational Recommendations
|
|
|
|
- Keep alert thresholds tied to actual user impact.
|
|
- Review dashboards after each release that changes endpoints or metrics names.
|
|
- Periodically test alert routing and incident response flow.
|
|
- Retention: **30 days** time-based plus **15GB** size cap (`--storage.tsdb.retention.time=30d --storage.tsdb.retention.size=15GB` on the `prometheus` service in `docker-compose.yml` and `docker-compose.prod.yml`). Increase the size cap before extending the time window so the TSDB does not hit the disk limit first.
|