Refactor code structure for improved readability and maintainability
This commit is contained in:
1 parent
207401b5fb
commit
c10d98f3b4
9 files changed
+2358
No files matched your search
+137
@@ -0,0 +1,137 @@
|
||||
# Monitoring Guide
|
||||
|
||||
This guide covers Prometheus and Grafana setup for Linear Void.
|
||||
|
||||
## Current Monitoring Components
|
||||
|
||||
- Prometheus container: linearvoid-prometheus
|
||||
- Grafana container: linearvoid-grafana
|
||||
- Metrics target: chat service /metrics
|
||||
- Base scrape config: monitoring/prometheus/prometheus.yml
|
||||
|
||||
## Prometheus Configuration
|
||||
|
||||
Current scrape config:
|
||||
|
||||
- Job chat -> chat:3000
|
||||
- Job prometheus -> prometheus:9090
|
||||
- Global scrape interval: 15s
|
||||
|
||||
File:
|
||||
|
||||
- monitoring/prometheus/prometheus.yml
|
||||
|
||||
## Start Monitoring Stack
|
||||
|
||||
With compose stack up:
|
||||
|
||||
- Prometheus UI: <http://localhost:9090>
|
||||
- Grafana UI: <http://localhost:3001>
|
||||
|
||||
Compose services include prometheus and grafana in both docker-compose.yml and docker-compose.prod.yml.
|
||||
|
||||
## Grafana Provisioning
|
||||
|
||||
Current mount paths:
|
||||
|
||||
- Dashboards directory: monitoring/grafana/dashboards
|
||||
- Datasources directory: monitoring/grafana/datasources
|
||||
|
||||
Notes:
|
||||
|
||||
- The dashboards directory includes a Prometheus overview dashboard file.
|
||||
- Datasource provisioning can be done with a YAML file in monitoring/grafana/datasources, or by configuring a datasource in Grafana UI.
|
||||
|
||||
Example datasource provisioning file (create if needed):
|
||||
|
||||
```yaml
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
```
|
||||
|
||||
## Dashboard References
|
||||
|
||||
- Provisioned dashboard: [monitoring/grafana/dashboards/dashboard.json](monitoring/grafana/dashboards/dashboard.json)
|
||||
- Root dashboard snapshot: [monitoring/grafana/dashboard.json](monitoring/grafana/dashboard.json)
|
||||
|
||||
Grafana screenshot:
|
||||
|
||||

|
||||
|
||||
## Key Metrics to Track
|
||||
|
||||
### Chat/Application
|
||||
|
||||
- linearvoid_chat_connected_clients
|
||||
- linearvoid_chat_stored_messages_total
|
||||
- linearvoid_chat_moderation_actions_total
|
||||
- linearvoid_chat_redis_enabled
|
||||
- linearvoid_chat_redis_up
|
||||
- linearvoid_chat_active_admin_sessions
|
||||
|
||||
### Prometheus Runtime
|
||||
|
||||
- up
|
||||
- process_cpu_seconds_total
|
||||
- process_resident_memory_bytes
|
||||
- prometheus_tsdb_head_series
|
||||
- prometheus_tsdb_wal_writes_failed_total
|
||||
- prometheus_tsdb_compactions_failed_total
|
||||
|
||||
## Suggested Alerts
|
||||
|
||||
Create a Prometheus rules file (for example monitoring/prometheus/alerts.yml) and wire it into Prometheus.
|
||||
|
||||
Recommended baseline alerts:
|
||||
|
||||
- Chat target down for 2m
|
||||
- Prometheus target down for 2m
|
||||
- Redis indicator down for 2m
|
||||
- TSDB WAL/compaction failure rate above zero
|
||||
- Elevated handler/query latency p99
|
||||
|
||||
## Validate Monitoring End-to-End
|
||||
|
||||
1. Verify /metrics through nginx:
|
||||
|
||||
- <https://linearvoid.tv/metrics>
|
||||
|
||||
1. Confirm targets in Prometheus:
|
||||
|
||||
- Status -> Targets page in Prometheus UI
|
||||
|
||||
1. Confirm dashboard data in Grafana:
|
||||
|
||||
- Import or open provisioned dashboards
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### No chat metrics in Prometheus
|
||||
|
||||
- Check chat container health and logs.
|
||||
- Confirm /metrics is reachable from Prometheus container.
|
||||
- Verify scrape target is chat:3000 in prometheus.yml.
|
||||
|
||||
### Grafana shows no data
|
||||
|
||||
- Ensure datasource points to <http://prometheus:9090>.
|
||||
- Confirm Prometheus is healthy.
|
||||
- Check dashboard query labels match active jobs.
|
||||
|
||||
### High cardinality or query slowness
|
||||
|
||||
- Reduce unneeded labels.
|
||||
- Limit expensive dashboard panels.
|
||||
- Tune retention and query windows.
|
||||
|
||||
## Operational Recommendations
|
||||
|
||||
- Keep alert thresholds tied to actual user impact.
|
||||
- Review dashboards after each release that changes endpoints or metrics names.
|
||||
- Periodically test alert routing and incident response flow.
|
||||
Reference in new issue
Block a user