# Monitoring Guide

Prometheus + Grafana monitoring for Linear Void.

## Architecture

```text
┌──────────────┐     ┌──────────────┐    ┌───────────┐
│  Exporters   │───> │  Prometheus  │───>│  Grafana  │
│  (7 targets) │     │  :9090       │    │  :3001    │
└──────────────┘     └───────┬──────┘    └───────────┘
                             │
                   alerts.yml (8 rules)
```

## Scrape Targets

| Job        | Target            | Port | Metrics                                                       |
| ---------- | ----------------- | ---- | ------------------------------------------------------------- |
| chat       | chat              | 3000 | WebSocket clients, messages, DMs, moderation, Redis, sessions |
| playout    | playout           | 8081 | Restarts, mode (live/slate)                                   |
| nginx      | nginx-exporter    | 9113 | Requests, connections                                         |
| redis      | redis-exporter    | 9121 | Memory, clients, commands                                     |
| postgres   | postgres-exporter | 9187 | Connections, queries, locks                                   |
| node       | node-exporter     | 9100 | CPU, memory, disk, network                                    |
| prometheus | prometheus        | 9090 | Self-monitoring                                               |

Config: `monitoring/prometheus/prometheus.yml` (15s scrape interval)

## Alerting Rules

8 rules in `monitoring/prometheus/alerts.yml`:

| Alert                | Severity | Expression                             |
| -------------------- | -------- | -------------------------------------- |
| ChatDown             | critical | `up{job="chat"} == 0` for 2m           |
| PlayoutDown          | critical | `up{job="playout"} == 0` for 2m        |
| RedisDown            | warning  | `linearvoid_chat_redis_up == 0` for 2m |
| PlayoutSlateActive   | warning  | `linearvoid_playout_mode == 1` for 1m  |
| HighChatLatency      | warning  | message rate <0.01/s for 5m            |
| PrometheusWALFailing | warning  | WAL write failures >0 for 5m           |
| HighCPU              | warning  | CPU >80% for 10m                       |
| LowDiskSpace         | warning  | Disk <10% free for 5m                  |

## Access

- Prometheus: `http://<host>:9090` (targets, graph, alerts)
- Grafana: `http://<host>:3001` (default: admin/admin, change via `GRAFANA_PASSWORD`)

## Dashboards

- **Linear Void app**: `monitoring/grafana/dashboard.json` — clients, messages, moderation, Redis, sessions, scrape health
- **Prometheus ops**: `monitoring/grafana/dashboards/dashboard.json` — TSDB, CPU/memory, WAL, compactions, query latency

## Metrics Checking

See detailed scripts for querying Prometheus scrape targets:

- [Linux/macOS](./infrastructure/monitoring/prometheus_curl.md)

## Key Metrics

- `linearvoid_chat_connected_clients` — active WebSocket connections
- `linearvoid_chat_messages_total` — total stored messages
- `linearvoid_chat_dms_total` — private messages sent
- `linearvoid_chat_redis_up` — Redis reachable (0/1)
- `linearvoid_playout_restarts_total` — FFmpeg restart counter
- `linearvoid_playout_mode` — 0=live, 1=slate, 2=degraded
- `linearvoid_chat_admin_sessions_active` — active admin logins

## Troubleshooting

| Symptom                 | Check                                                                |
| ----------------------- | -------------------------------------------------------------------- |
| No chat metrics         | `curl http://chat:3000/metrics` from Prometheus container            |
| No exporter metrics     | Verify exporter containers running: `docker ps \| grep exporter`     |
| Grafana no data         | Datasource URL must be `http://prometheus:9090`                      |
| Prometheus targets down | Check container network: all must be on `linearvoid-net`             |
| Alert not firing        | Check Prometheus → Alerts tab, verify `rule_files` in prometheus.yml |

**Grafana no data**: ensure datasource points to `http://prometheus:9090`; confirm Prometheus is healthy; check dashboard query labels match active jobs.

**High cardinality / slow queries**: reduce unneeded labels; limit expensive panels; tune retention and query windows.

## Operational Recommendations

- Keep alert thresholds tied to actual user impact.
- Review dashboards after each release that changes endpoints or metrics names.
- Periodically test alert routing and incident response flow.
- Retention: **30 days** time-based plus **15GB** size cap (`--storage.tsdb.retention.time=30d --storage.tsdb.retention.size=15GB` on the `prometheus` service in `docker-compose.yml` and `docker-compose.prod.yml`). Increase the size cap before extending the time window so the TSDB does not hit the disk limit first.