# Monitoring Guide

This guide covers Prometheus and Grafana setup for Linear Void.

## Current Monitoring Components

- Prometheus container: linearvoid-prometheus
- Grafana container: linearvoid-grafana
- Metrics target: chat service /metrics
- Base scrape config: monitoring/prometheus/prometheus.yml

## Prometheus Configuration

Current scrape config:

- Job chat -> chat:3000
- Job prometheus -> prometheus:9090
- Global scrape interval: 15s

File:

- monitoring/prometheus/prometheus.yml

## Start Monitoring Stack

With compose stack up:

- Prometheus UI: <http://localhost:9090>
- Grafana UI: <http://localhost:3001>

Compose services include prometheus and grafana in both docker-compose.yml and docker-compose.prod.yml.

## Grafana Provisioning

Current mount paths:

- Dashboards directory: monitoring/grafana/dashboards
- Datasources directory: monitoring/grafana/datasources

Notes:

- The dashboards directory includes a Prometheus overview dashboard file.
- Datasource provisioning can be done with a YAML file in monitoring/grafana/datasources, or by configuring a datasource in Grafana UI.

Example datasource provisioning file (create if needed):

```yaml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
  url: http://prometheus:9090
    isDefault: true
```

## Dashboard References

- Provisioned dashboard: [monitoring/grafana/dashboards/dashboard.json](monitoring/grafana/dashboards/dashboard.json)
- Root dashboard snapshot: [monitoring/grafana/dashboard.json](monitoring/grafana/dashboard.json)

Grafana screenshot:

![Grafana dashboard screenshot](docs/img/grafana-screenshot.png)

## Key Metrics to Track

### Chat/Application

- linearvoid_chat_connected_clients
- linearvoid_chat_stored_messages_total
- linearvoid_chat_moderation_actions_total
- linearvoid_chat_redis_enabled
- linearvoid_chat_redis_up
- linearvoid_chat_active_admin_sessions

### Prometheus Runtime

- up
- process_cpu_seconds_total
- process_resident_memory_bytes
- prometheus_tsdb_head_series
- prometheus_tsdb_wal_writes_failed_total
- prometheus_tsdb_compactions_failed_total

## Suggested Alerts

Create a Prometheus rules file (for example monitoring/prometheus/alerts.yml) and wire it into Prometheus.

Recommended baseline alerts:

- Chat target down for 2m
- Prometheus target down for 2m
- Redis indicator down for 2m
- TSDB WAL/compaction failure rate above zero
- Elevated handler/query latency p99

## Validate Monitoring End-to-End

1. Verify /metrics through nginx:

- <https://linearvoid.tv/metrics>

1. Confirm targets in Prometheus:

- Status -> Targets page in Prometheus UI

1. Confirm dashboard data in Grafana:

- Import or open provisioned dashboards

## Troubleshooting

### No chat metrics in Prometheus

- Check chat container health and logs.
- Confirm /metrics is reachable from Prometheus container.
- Verify scrape target is chat:3000 in prometheus.yml.

### Grafana shows no data

- Ensure datasource points to <http://prometheus:9090>.
- Confirm Prometheus is healthy.
- Check dashboard query labels match active jobs.

### High cardinality or query slowness

- Reduce unneeded labels.
- Limit expensive dashboard panels.
- Tune retention and query windows.

## Operational Recommendations

- Keep alert thresholds tied to actual user impact.
- Review dashboards after each release that changes endpoints or metrics names.
- Periodically test alert routing and incident response flow.