linearvoid/MONITORING.md

Monitoring Guide

This guide covers Prometheus and Grafana setup for Linear Void.

Current Monitoring Components

• Prometheus container: linearvoid-prometheus
• Grafana container: linearvoid-grafana
• Metrics target: chat service /metrics
• Base scrape config: monitoring/prometheus/prometheus.yml

Prometheus Configuration

Current scrape config:

• Job chat -> chat:3000
• Job prometheus -> prometheus:9090
• Global scrape interval: 15s

File:

• monitoring/prometheus/prometheus.yml

Start Monitoring Stack

With compose stack up:

• Prometheus UI: http://localhost:9090
• Grafana UI: http://localhost:3001

Compose services include prometheus and grafana in both docker-compose.yml and docker-compose.prod.yml.

Grafana Provisioning

Current mount paths:

• Dashboards directory: monitoring/grafana/dashboards
• Datasources directory: monitoring/grafana/datasources

Notes:

• The dashboards directory includes a Prometheus overview dashboard file.
• Datasource provisioning can be done with a YAML file in monitoring/grafana/datasources, or by configuring a datasource in Grafana UI.

Example datasource provisioning file (create if needed):

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
  url: http://prometheus:9090
    isDefault: true

Dashboard References

• Provisioned dashboard: monitoring/grafana/dashboards/dashboard.json
• Root dashboard snapshot: monitoring/grafana/dashboard.json

Grafana screenshot:

Key Metrics to Track

Chat/Application

• linearvoid_chat_connected_clients
• linearvoid_chat_stored_messages_total
• linearvoid_chat_moderation_actions_total
• linearvoid_chat_redis_enabled
• linearvoid_chat_redis_up
• linearvoid_chat_active_admin_sessions

Prometheus Runtime

• up
• process_cpu_seconds_total
• process_resident_memory_bytes
• prometheus_tsdb_head_series
• prometheus_tsdb_wal_writes_failed_total
• prometheus_tsdb_compactions_failed_total

Suggested Alerts

Create a Prometheus rules file (for example monitoring/prometheus/alerts.yml) and wire it into Prometheus.

Recommended baseline alerts:

• Chat target down for 2m
• Prometheus target down for 2m
• Redis indicator down for 2m
• TSDB WAL/compaction failure rate above zero
• Elevated handler/query latency p99

Validate Monitoring End-to-End

1. Verify /metrics through nginx:

• https://linearvoid.tv/metrics

1. Confirm targets in Prometheus:

• Status -> Targets page in Prometheus UI

1. Confirm dashboard data in Grafana:

• Import or open provisioned dashboards

Troubleshooting

No chat metrics in Prometheus

• Check chat container health and logs.
• Confirm /metrics is reachable from Prometheus container.
• Verify scrape target is chat:3000 in prometheus.yml.

Grafana shows no data

• Ensure datasource points to http://prometheus:9090.
• Confirm Prometheus is healthy.
• Check dashboard query labels match active jobs.

High cardinality or query slowness

• Reduce unneeded labels.
• Limit expensive dashboard panels.
• Tune retention and query windows.

Operational Recommendations

• Keep alert thresholds tied to actual user impact.
• Review dashboards after each release that changes endpoints or metrics names.
• Periodically test alert routing and incident response flow.