Add Grafana dashboards for monitoring various services
- Created a new dashboard for NGINX monitoring with panels for status, active connections, request rates, and connection drop ratio. - Added a Node dashboard to monitor CPU usage, memory usage, disk I/O, and network traffic. - Introduced a Playout dashboard to track service status, mode, FFmpeg restarts, and scrape duration. - Implemented a PostgreSQL dashboard featuring active connections, database size, transaction rates, and cache hit ratio. - Developed a Prometheus dashboard to visualize scrape duration, targets status, and rule evaluation metrics. - Established a Redis dashboard to monitor connected clients, memory usage, command latency, and keyspace statistics.
This commit is contained in:
zwitschi committed
1 parent
926d70dc3c
commit
044851a43e
11 files changed
+10065
-123
No files matched your search
@@ -57,7 +57,6 @@ Config: `monitoring/prometheus/prometheus.yml` (15s scrape interval)
|
||||
See detailed scripts for querying Prometheus scrape targets:
|
||||
|
||||
- [Linux/macOS](./infrastructure/monitoring/prometheus_curl.md)
|
||||
- [Windows](./infrastructure/monitoring/prometheus_curl_win.md)
|
||||
|
||||
## Key Metrics
|
||||
|
||||
|
||||
+1
-2
@@ -12,8 +12,7 @@ This folder contains the documentation for the Linear Void project. The document
|
||||
- [Cache Rules](./infrastructure/cloudflare/cache-rules.md) - Cache rules for Cloudflare Workers and Pages.
|
||||
- [Account Troubleshooting](./infrastructure/cloudflare/cloudflare-account-troubleshooting.md) - Troubleshooting Cloudflare account issues.
|
||||
- Monitoring
|
||||
- [Prometheus CURL](./infrastructure/monitoring/prometheus_curl.md) - Prometheus metrics endpoint testing with curl.
|
||||
- [Prometheus CURL Windows](./infrastructure/monitoring/prometheus_curl_win.md) - Prometheus metrics endpoint testing with curl on Windows.
|
||||
- [Prometheus CURL](./infrastructure/monitoring/prometheus_curl.md) - Prometheus metrics endpoint testing with curl and jq. Doocumentation of available metrics and their meanings.
|
||||
|
||||
## API
|
||||
|
||||
|
||||
@@ -3,69 +3,60 @@
|
||||
Here are ready‑to‑run **cURL** commands you can use to query each Prometheus scrape target and list the metric names they expose.
|
||||
The Prometheus HTTP API endpoint `http://localhost:9090/api/v1/label/__name__/values` returns all metric names currently known for the target’s job/instance.
|
||||
|
||||
---
|
||||
|
||||
## 1. Chat (`http://chat:3000/metrics`)
|
||||
|
||||
[Browse `chat`](http://192.168.88.40:9090/api/v1/label/__name__/values?match[]={job=%22chat%22})
|
||||
|
||||
```bash
|
||||
curl -s 'http://localhost:9090/api/v1/label/__name__/values?match[]={job="chat",instance="chat:3000"}' | jq .
|
||||
```
|
||||
|
||||
see [Prometheus Results](./prometheus_results.md#chat-metrics) for full output of all 20+ metric names exposed by the Chat service.
|
||||
|
||||
## 2. NGINX exporter (`http://nginx-exporter:9113/metrics`)
|
||||
|
||||
```bash
|
||||
curl -s 'http://localhost:9090/api/v1/label/__name__/values?match[]={job="nginx",instance="nginx-exporter:9113"}' | jq .
|
||||
```
|
||||
|
||||
see [Prometheus Results](./prometheus_results.md#nginx-metrics) for full output of all 40+ metric names exposed by the NGINX exporter.
|
||||
|
||||
## 3. Node exporter (`http://node-exporter:9100/metrics`)
|
||||
|
||||
```bash
|
||||
curl -s 'http://localhost:9090/api/v1/label/__name__/values?match[]={job="node",instance="node-exporter:9100"}' | jq .
|
||||
```
|
||||
|
||||
see [Prometheus Results](./prometheus_results.md#node-metrics) for full output of all 200+ metric names exposed by the Node exporter.
|
||||
|
||||
## 4. Playout (`http://playout:8081/metrics`)
|
||||
|
||||
```bash
|
||||
curl -s 'http://localhost:9090/api/v1/label/__name__/values?match[]={job="playout",instance="playout:8081"}' | jq .
|
||||
```
|
||||
|
||||
see [Prometheus Results](./prometheus_results.md#playout-metrics) for full output of all 10+ metric names exposed by the Playout service.
|
||||
|
||||
## 5. Postgres exporter (`http://postgres-exporter:9187/metrics`)
|
||||
|
||||
```bash
|
||||
curl -s 'http://localhost:9090/api/v1/label/__name__/values?match[]={job="postgres",instance="postgres-exporter:9187"}' | jq .
|
||||
```
|
||||
|
||||
see [Prometheus Results](./prometheus_results.md#postgres-metrics) for full output of all 100+ metric names exposed by the Postgres exporter.
|
||||
|
||||
## 6. Prometheus itself (`http://prometheus:9090/metrics`)
|
||||
|
||||
```bash
|
||||
curl -s 'http://localhost:9090/api/v1/label/__name__/values?match[]={job="prometheus",instance="prometheus:9090"}' | jq .
|
||||
```
|
||||
|
||||
see [Prometheus Results](./prometheus_results.md#prometheus-metrics) for full output of all 100+ metric names exposed by Prometheus itself.
|
||||
|
||||
## 7. Redis exporter (`http://redis-exporter:9121/metrics`)
|
||||
|
||||
```bash
|
||||
curl -s 'http://localhost:9090/api/v1/label/__name__/values?match[]={job="redis",instance="redis-exporter:9121"}' | jq .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### How it works
|
||||
|
||||
- `http://localhost:9090` – address of your Prometheus server.
|
||||
- `/api/v1/label/__name__/values` – API call that lists _all_ metric names (`__name__` label) known to Prometheus.
|
||||
- `match[]` – a selector that restricts the result to the specific `job` and `instance` you care about.
|
||||
- `-s` – silent mode (no progress meter).
|
||||
- `| jq .` – pretty‑prints the JSON response (optional but handy).
|
||||
|
||||
Running any of these commands will output a JSON array like:
|
||||
|
||||
```json
|
||||
[
|
||||
"up",
|
||||
"process_cpu_seconds_total",
|
||||
"http_requests_total",
|
||||
...
|
||||
]
|
||||
```
|
||||
|
||||
which is the full set of metrics currently being scraped from that target. Adjust the Prometheus host/port if it differs from `http://localhost:9090`.
|
||||
see [Prometheus Results](./prometheus_results.md#redis-metrics) for full output of all 100+ metric names exposed by the Redis exporter.
|
||||
@@ -1,96 +0,0 @@
|
||||
# Prometheus scrape target metric names (Windows)
|
||||
|
||||
Below are **cURL** one‑liners you can run from a Windows command prompt (or PowerShell) that query the Prometheus HTTP API and print the raw JSON list of metric names for each target.
|
||||
Since **jq** isn’t installed, the commands simply echo the JSON response; you can pipe it to `findstr` or PowerShell’s `ConvertFrom‑Json` if you want a prettier view.
|
||||
|
||||
---
|
||||
|
||||
## 1. Chat (`http://chat:3000/metrics`)
|
||||
|
||||
```cmd
|
||||
curl "http://localhost:9090/api/v1/label/__name__/values?match[]={job=\"chat\",instance=\"chat:3000\"}"
|
||||
```
|
||||
|
||||
## 2. NGINX exporter (`http://nginx-exporter:9113/metrics`)
|
||||
|
||||
```cmd
|
||||
curl "http://localhost:9090/api/v1/label/__name__/values?match[]={job=\"nginx\",instance=\"nginx-exporter:9113\"}"
|
||||
```
|
||||
|
||||
## 3. Node exporter (`http://node-exporter:9100/metrics`)
|
||||
|
||||
```cmd
|
||||
curl "http://localhost:9090/api/v1/label/__name__/values?match[]={job=\"node\",instance=\"node-exporter:9100\"}"
|
||||
```
|
||||
|
||||
## 4. Playout (`http://playout:8081/metrics`)
|
||||
|
||||
```cmd
|
||||
curl "http://localhost:9090/api/v1/label/__name__/values?match[]={job=\"playout\",instance=\"playout:8081\"}"
|
||||
```
|
||||
|
||||
## 5. Postgres exporter (`http://postgres-exporter:9187/metrics`)
|
||||
|
||||
```cmd
|
||||
curl "http://localhost:9090/api/v1/label/__name__/values?match[]={job=\"postgres\",instance=\"postgres-exporter:9187\"}"
|
||||
```
|
||||
|
||||
## 6. Prometheus itself (`http://prometheus:9090/metrics`)
|
||||
|
||||
```cmd
|
||||
curl "http://localhost:9090/api/v1/label/__name__/values?match[]={job=\"prometheus\",instance=\"prometheus:9090\"}"
|
||||
```
|
||||
|
||||
## 7. Redis exporter (`http://redis-exporter:9121/metrics`)
|
||||
|
||||
```cmd
|
||||
curl "http://localhost:9090/api/v1/label/__name__/values?match[]={job=\"redis\",instance=\"redis-exporter:9121\"}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Making the output a bit more readable (optional)
|
||||
|
||||
If you have **PowerShell** (available on Windows 10/11), you can format the JSON nicely:
|
||||
|
||||
```powershell
|
||||
curl "http://localhost:9090/api/v1/label/__name__/values?match[]={job=`"chat`",instance=`"chat:3000`"}" |
|
||||
ConvertFrom-Json |
|
||||
Select-Object -ExpandProperty data |
|
||||
Sort-Object
|
||||
```
|
||||
|
||||
Replace the URL part with the one for the target you’re interested in. The pipeline:
|
||||
|
||||
1. Calls `curl` (the built‑in alias for `Invoke-WebRequest` in PowerShell).
|
||||
2. `ConvertFrom-Json` parses the JSON.
|
||||
3. `Select-Object -ExpandProperty data` extracts the array of metric names.
|
||||
4. `Sort-Object` alphabetizes them.
|
||||
|
||||
You can similarly pipe the output to `findstr /i "http_requests_total"` to search for a particular metric name without any extra tools.
|
||||
|
||||
---
|
||||
|
||||
### Quick checklist
|
||||
|
||||
| Item | What to adjust |
|
||||
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Prometheus address** | If your Prometheus container isn’t bound to `localhost:9090`, replace `localhost:9090` with the appropriate host/port (e.g., `127.0.0.1:9090`). |
|
||||
| **Target job/instance** | The `job` and `instance` values must match the labels shown in your UI (they’re case‑sensitive). |
|
||||
| **PowerShell vs CMD** | The raw `curl` commands work in both CMD and PowerShell; the PowerShell‑only pretty‑print uses `ConvertFrom‑Json`. |
|
||||
|
||||
Run the command for any target and you’ll receive a JSON payload like:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "success",
|
||||
"data": [
|
||||
"up",
|
||||
"process_cpu_seconds_total",
|
||||
"http_requests_total",
|
||||
"... more metrics ..."
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
That array (`data`) is the complete list of metric names currently scraped from the specified endpoint.
|
||||
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user