linearvoid/docs/MONITORING.md

Monitoring Guide

Prometheus + Grafana monitoring for Linear Void.

Architecture

┌──────────────┐     ┌──────────────┐    ┌───────────┐
│  Exporters   │───> │  Prometheus  │───>│  Grafana  │
│  (7 targets) │     │  :9090       │    │  :3001    │
└──────────────┘     └───────┬──────┘    └───────────┘
                             │
                   alerts.yml (8 rules)

Scrape Targets

Job	Target	Port	Metrics
chat	chat	3000	WebSocket clients, messages, DMs, moderation, Redis, sessions
playout	playout	8081	Restarts, mode (live/slate)
nginx	nginx-exporter	9113	Requests, connections
redis	redis-exporter	9121	Memory, clients, commands
postgres	postgres-exporter	9187	Connections, queries, locks
node	node-exporter	9100	CPU, memory, disk, network
prometheus	prometheus	9090	Self-monitoring

Config: monitoring/prometheus/prometheus.yml (15s scrape interval)

Alerting Rules

8 rules in monitoring/prometheus/alerts.yml:

Alert	Severity	Expression
ChatDown	critical	up{job="chat"} == 0 for 2m
PlayoutDown	critical	up{job="playout"} == 0 for 2m
RedisDown	warning	linearvoid_chat_redis_up == 0 for 2m
PlayoutSlateActive	warning	linearvoid_playout_mode == 1 for 1m
HighChatLatency	warning	message rate <0.01/s for 5m
PrometheusWALFailing	warning	WAL write failures >0 for 5m
HighCPU	warning	CPU >80% for 10m
LowDiskSpace	warning	Disk <10% free for 5m

Access

• Prometheus: http://<host>:9090 (targets, graph, alerts)
• Grafana: http://<host>:3001 (default: admin/admin, change via GRAFANA_PASSWORD)

Dashboards

• Linear Void app: monitoring/grafana/dashboard.json — clients, messages, moderation, Redis, sessions, scrape health
• Prometheus ops: monitoring/grafana/dashboards/dashboard.json — TSDB, CPU/memory, WAL, compactions, query latency

Metrics Checking

See detailed scripts for querying Prometheus scrape targets:

• Linux/macOS

Key Metrics

• linearvoid_chat_connected_clients — active WebSocket connections
• linearvoid_chat_messages_total — total stored messages
• linearvoid_chat_dms_total — private messages sent
• linearvoid_chat_redis_up — Redis reachable (0/1)
• linearvoid_playout_restarts_total — FFmpeg restart counter
• linearvoid_playout_mode — 0=live, 1=slate, 2=degraded
• linearvoid_chat_admin_sessions_active — active admin logins

Troubleshooting

Symptom	Check
No chat metrics	curl http://chat:3000/metrics from Prometheus container
No exporter metrics	Verify exporter containers running: docker ps | grep exporter
Grafana no data	Datasource URL must be http://prometheus:9090
Prometheus targets down	Check container network: all must be on linearvoid-net
Alert not firing	Check Prometheus → Alerts tab, verify rule_files in prometheus.yml

Grafana no data: ensure datasource points to http://prometheus:9090; confirm Prometheus is healthy; check dashboard query labels match active jobs.

High cardinality / slow queries: reduce unneeded labels; limit expensive panels; tune retention and query windows.

Operational Recommendations

• Keep alert thresholds tied to actual user impact.
• Review dashboards after each release that changes endpoints or metrics names.
• Periodically test alert routing and incident response flow.
• Retention: 30 days time-based plus 15GB size cap (--storage.tsdb.retention.time=30d --storage.tsdb.retention.size=15GB on the prometheus service in docker-compose.yml and docker-compose.prod.yml). Increase the size cap before extending the time window so the TSDB does not hit the disk limit first.