Monitoring Guide

Prometheus + Grafana monitoring for Linear Void.

Architecture

┌──────────────┐     ┌──────────────┐    ┌───────────┐
│  Exporters   │───> │  Prometheus  │───>│  Grafana  │
│  (7 targets) │     │  :9090       │    │  :3001    │
└──────────────┘     └───────┬──────┘    └───────────┘
                             │
                   alerts.yml (8 rules)

Scrape Targets

Job	Target	Port	Metrics
chat	chat	3000	WebSocket clients, messages, DMs, moderation, Redis, sessions
playout	playout	8081	Restarts, mode (live/slate)
nginx	nginx-exporter	9113	Requests, connections
redis	redis-exporter	9121	Memory, clients, commands
postgres	postgres-exporter	9187	Connections, queries, locks
node	node-exporter	9100	CPU, memory, disk, network
prometheus	prometheus	9090	Self-monitoring

Config: monitoring/prometheus/prometheus.yml (15s scrape interval)

Alerting Rules

8 rules in monitoring/prometheus/alerts.yml:

Alert	Severity	Expression
ChatDown	critical	`up{job="chat"} == 0` for 2m
PlayoutDown	critical	`up{job="playout"} == 0` for 2m
RedisDown	warning	`linearvoid_chat_redis_up == 0` for 2m
PlayoutSlateActive	warning	`linearvoid_playout_mode == 1` for 1m
HighChatLatency	warning	message rate <0.01/s for 5m
PrometheusWALFailing	warning	WAL write failures >0 for 5m
HighCPU	warning	CPU >80% for 10m
LowDiskSpace	warning	Disk <10% free for 5m

Access

Prometheus: http://<host>:9090 (targets, graph, alerts)
Grafana: http://<host>:3001 (default: admin/admin, change via GRAFANA_PASSWORD)

Dashboards

Linear Void app: monitoring/grafana/dashboard.json — clients, messages, moderation, Redis, sessions, scrape health
Prometheus ops: monitoring/grafana/dashboards/dashboard.json — TSDB, CPU/memory, WAL, compactions, query latency

Metrics checking

See detailed scripts for querying Prometheus scrape targets:

Linux/macOS

Key Metrics

linearvoid_chat_connected_clients — active WebSocket connections
linearvoid_chat_messages_total — total stored messages
linearvoid_chat_dms_total — private messages sent
linearvoid_chat_redis_up — Redis reachable (0/1)
linearvoid_playout_restarts_total — FFmpeg restart counter
linearvoid_playout_mode — 0=live, 1=slate, 2=degraded
linearvoid_chat_admin_sessions_active — active admin logins

Troubleshooting

Symptom	Check
No chat metrics	`curl http://chat:3000/metrics` from Prometheus container
No exporter metrics	Verify exporter containers running: `docker ps \| grep exporter`
Grafana no data	Datasource URL must be `http://prometheus:9090`
Prometheus targets down	Check container network: all must be on `linearvoid-net`
Alert not firing	Check Prometheus → Alerts tab, verify `rule_files` in prometheus.yml

Grafana shows no data

Ensure datasource points to http://prometheus:9090.
Confirm Prometheus is healthy.
Check dashboard query labels match active jobs.

High cardinality or query slowness

Reduce unneeded labels.
Limit expensive dashboard panels.
Tune retention and query windows.

Operational Recommendations

Keep alert thresholds tied to actual user impact.
Review dashboards after each release that changes endpoints or metrics names.
Periodically test alert routing and incident response flow.
Metrics retention is set to 30 days time-based plus a 15GB size cap (--storage.tsdb.retention.time=30d --storage.tsdb.retention.size=15GB on the prometheus service in docker-compose.yml and docker-compose.prod.yml). Increase the size cap before extending the time window so the TSDB does not hit the disk limit first.

5.0 KiB Raw Blame History