138 lines
3.4 KiB
Markdown
138 lines
3.4 KiB
Markdown
# Monitoring Guide
|
|
|
|
This guide covers Prometheus and Grafana setup for Linear Void.
|
|
|
|
## Current Monitoring Components
|
|
|
|
- Prometheus container: linearvoid-prometheus
|
|
- Grafana container: linearvoid-grafana
|
|
- Metrics target: chat service /metrics
|
|
- Base scrape config: monitoring/prometheus/prometheus.yml
|
|
|
|
## Prometheus Configuration
|
|
|
|
Current scrape config:
|
|
|
|
- Job chat -> chat:3000
|
|
- Job prometheus -> prometheus:9090
|
|
- Global scrape interval: 15s
|
|
|
|
File:
|
|
|
|
- monitoring/prometheus/prometheus.yml
|
|
|
|
## Start Monitoring Stack
|
|
|
|
With compose stack up:
|
|
|
|
- Prometheus UI: <http://localhost:9090>
|
|
- Grafana UI: <http://localhost:3001>
|
|
|
|
Compose services include prometheus and grafana in both docker-compose.yml and docker-compose.prod.yml.
|
|
|
|
## Grafana Provisioning
|
|
|
|
Current mount paths:
|
|
|
|
- Dashboards directory: monitoring/grafana/dashboards
|
|
- Datasources directory: monitoring/grafana/datasources
|
|
|
|
Notes:
|
|
|
|
- The dashboards directory includes a Prometheus overview dashboard file.
|
|
- Datasource provisioning can be done with a YAML file in monitoring/grafana/datasources, or by configuring a datasource in Grafana UI.
|
|
|
|
Example datasource provisioning file (create if needed):
|
|
|
|
```yaml
|
|
apiVersion: 1
|
|
|
|
datasources:
|
|
- name: Prometheus
|
|
type: prometheus
|
|
access: proxy
|
|
url: http://prometheus:9090
|
|
isDefault: true
|
|
```
|
|
|
|
## Dashboard References
|
|
|
|
- Provisioned dashboard: [monitoring/grafana/dashboards/dashboard.json](monitoring/grafana/dashboards/dashboard.json)
|
|
- Root dashboard snapshot: [monitoring/grafana/dashboard.json](monitoring/grafana/dashboard.json)
|
|
|
|
Grafana screenshot:
|
|
|
|
|
|
|
|
## Key Metrics to Track
|
|
|
|
### Chat/Application
|
|
|
|
- linearvoid_chat_connected_clients
|
|
- linearvoid_chat_stored_messages_total
|
|
- linearvoid_chat_moderation_actions_total
|
|
- linearvoid_chat_redis_enabled
|
|
- linearvoid_chat_redis_up
|
|
- linearvoid_chat_active_admin_sessions
|
|
|
|
### Prometheus Runtime
|
|
|
|
- up
|
|
- process_cpu_seconds_total
|
|
- process_resident_memory_bytes
|
|
- prometheus_tsdb_head_series
|
|
- prometheus_tsdb_wal_writes_failed_total
|
|
- prometheus_tsdb_compactions_failed_total
|
|
|
|
## Suggested Alerts
|
|
|
|
Create a Prometheus rules file (for example monitoring/prometheus/alerts.yml) and wire it into Prometheus.
|
|
|
|
Recommended baseline alerts:
|
|
|
|
- Chat target down for 2m
|
|
- Prometheus target down for 2m
|
|
- Redis indicator down for 2m
|
|
- TSDB WAL/compaction failure rate above zero
|
|
- Elevated handler/query latency p99
|
|
|
|
## Validate Monitoring End-to-End
|
|
|
|
1. Verify /metrics through nginx:
|
|
|
|
- <https://linearvoid.tv/metrics>
|
|
|
|
1. Confirm targets in Prometheus:
|
|
|
|
- Status -> Targets page in Prometheus UI
|
|
|
|
1. Confirm dashboard data in Grafana:
|
|
|
|
- Import or open provisioned dashboards
|
|
|
|
## Troubleshooting
|
|
|
|
### No chat metrics in Prometheus
|
|
|
|
- Check chat container health and logs.
|
|
- Confirm /metrics is reachable from Prometheus container.
|
|
- Verify scrape target is chat:3000 in prometheus.yml.
|
|
|
|
### Grafana shows no data
|
|
|
|
- Ensure datasource points to <http://prometheus:9090>.
|
|
- Confirm Prometheus is healthy.
|
|
- Check dashboard query labels match active jobs.
|
|
|
|
### High cardinality or query slowness
|
|
|
|
- Reduce unneeded labels.
|
|
- Limit expensive dashboard panels.
|
|
- Tune retention and query windows.
|
|
|
|
## Operational Recommendations
|
|
|
|
- Keep alert thresholds tied to actual user impact.
|
|
- Review dashboards after each release that changes endpoints or metrics names.
|
|
- Periodically test alert routing and incident response flow.
|