linearvoid/docs/infrastructure/monitoring/prometheus_results.md

# Prometheus Metrics — Linear Void

Raw scrape-target metric inventories captured for the monitoring setup. This file
lists which metric names are exposed by each target. No threshold values, alert
rules, or query results are present beyond the metric inventory itself.

Standard boilerplate metrics are exposed by every Go-based target and are omitted
below where noted: `go_*` (gc/memstats/sched/goroutines), `process_*` (cpu, fds,
memory, network), `promhttp_metric_handler_*`, and Prometheus scrape internals
(`scrape_duration_seconds`, `scrape_samples_scraped`, `scrape_samples_post_metric_relabeling`,
`scrape_series_added`, `up`).

## Chat Target (`linearvoid_chat_*`)
Application metrics for the chat service:
- `linearvoid_chat_admin_sessions_active`
- `linearvoid_chat_build_info`
- `linearvoid_chat_connected_clients`
- `linearvoid_chat_dms_total`
- `linearvoid_chat_messages_total`
- `linearvoid_chat_moderation_actions_total`
- `linearvoid_chat_redis_connected_clients`
- `linearvoid_chat_redis_enabled`
- `linearvoid_chat_redis_memory_max_bytes`
- `linearvoid_chat_redis_memory_usage_ratio`
- `linearvoid_chat_redis_memory_used_bytes`
- `linearvoid_chat_redis_up`
- `linearvoid_chat_request_duration_seconds_bucket/_count/_sum` (histogram)
- Plus `ALERTS`, `ALERTS_FOR_STATE` (alert state series).

## Playout Target (`linearvoid_playout_*`)
- `linearvoid_playout_mode`
- `linearvoid_playout_restarts_total`
- Plus `ALERTS`, `ALERTS_FOR_STATE`.

## Nginx Target (`nginx_*`)
Nginx (Go-based nginx exporter) metrics. Key operational series:
- `nginx_connections_accepted`, `nginx_connections_active`, `nginx_connections_handled`
- `nginx_connections_reading`, `nginx_connections_waiting`, `nginx_connections_writing`
- `nginx_http_requests_total`
- `nginx_exporter_build_info`, `nginx_up`
- Standard Go/process/promhttp/scrape boilerplate.

## Node Target (node_exporter, ZFS host)
Exposes the full standard node_exporter metric set plus ZFS (`node_zfs_*`) and
hwmon (`node_hwmon_*`) collectors. Operationally meaningful series include:
- **CPU/load:** `node_load1/5/15`, `node_cpu_seconds_total`, `node_cpu_scaling_frequency_hertz`,
  `node_procs_running`, `node_procs_blocked`, `node_context_switches_total`
- **Memory:** `node_memory_MemTotal_bytes`, `node_memory_MemAvailable_bytes`,
  `node_memory_MemFree_bytes`, `node_memory_SwapTotal_bytes`, `node_memory_SwapFree_bytes`,
  `node_vmstat_oom_kill`
- **Filesystem/disk:** `node_filesystem_avail_bytes`, `node_filesystem_free_bytes`,
  `node_filesystem_size_bytes`, `node_filesystem_readonly`, `node_disk_read_bytes_total`,
  `node_disk_written_bytes_total`, `node_disk_io_time_seconds_total`
- **Network:** `node_network_receive_bytes_total`, `node_network_transmit_bytes_total`,
  `node_network_up`, plus full `node_netstat_*` TCP/UDP/ICMP series
- **Files/fds:** `node_filefd_allocated`, `node_filefd_maximum`
- **Time:** `node_time_seconds`, `node_time_zone_offset_seconds`, `node_timex_sync_status`
- **ZFS ARC/L2ARC:** extensive `node_zfs_arc_*`/`node_zfs_l2_*` hit/miss/size series,
  `node_zfs_zil_*`, `node_zfs_dbuf_*`, `node_zfs_dnode_*`, `node_zfs_zfetch_*`
- **Thermals:** `node_hwmon_temp_celsius`, `node_hwmon_temp_crit_celsius`
- `node_exporter_build_info`, `node_uname_info`, `node_os_info`, `node_boot_time_seconds`.

## Postgres Target (`pg_*` / postgres_exporter)
Go-based postgres exporter. Key operational series:
- **Health/replication:** `pg_up`, `pg_database_size_bytes`, `pg_database_connection_limit`,
  `pg_replication_is_replica`, `pg_replication_lag_seconds`, `pg_replication_last_replay_seconds`,
  `pg_locks_count`, `pg_wal_segments`, `pg_wal_size_bytes`
- **Settings (config-as-metric):** `pg_settings_*` (autovacuum, bgwriter, checkpoints,
  max_connections, shared_buffers, work_mem, wal_*, etc.)
- **Activity/statistics:** `pg_stat_activity_count`, `pg_stat_activity_max_tx_duration`,
  `pg_stat_bgwriter_*` (checkpoints, buffers, fsync), `pg_stat_database_*` (commits,
  rollbacks, deadlocks, temp files, cache hit/read, tuple ops), `pg_stat_archiver_*`
- **Tables:** `pg_stat_user_tables_*` (live/dead tuples, seq/index scans, vacuum/analyze
  counts), `pg_statio_user_tables_*` (heap/index block hits & reads, incl. TOAST)
- `pg_exporter_*` scrape health series, `postgres_exporter_build_info`, and Go/process boilerplate.

## Prometheus Target (self-scrape)
Prometheus' own metrics. Operationally meaningful series include:
- **Health/status:** `prometheus_build_info`, `prometheus_ready`,
  `prometheus_config_last_reload_successful` (+ timestamp), `prometheus_tsdb_clean_start`
- **TSDB:** `prometheus_tsdb_head_series`, `prometheus_tsdb_head_samples_appended_total`,
  `prometheus_tsdb_blocks_loaded`, `prometheus_tsdb_retention_limit_seconds`,
  `prometheus_tsdb_storage_blocks_bytes`, `prometheus_tsdb_wal_storage_size_bytes`,
  `prometheus_tsdb_lowest_timestamp`/`_seconds`, `prometheus_tsdb_head_max_time`
- **Scraping:** `prometheus_sd_discovered_targets`, `prometheus_sd_failed_configs`,
  `prometheus_sd_last_update_timestamp_seconds`, `prometheus_target_scrape_pool_targets`,
  `prometheus_target_scrape_pool_target_limit`, `prometheus_target_interval_length_seconds`
- **Rules/alerting:** `prometheus_rule_group_*` (evaluation duration/failures/interval/
  rules), `prometheus_rule_evaluation_failures_total`, `prometheus_evaluations_total`,
  `prometheus_notifications_alertmanagers_discovered`, `prometheus_notifications_queue_capacity`
- **Engine/HTTP:** `prometheus_engine_queries`, `prometheus_http_requests_total`,
  `prometheus_http_request_duration_seconds_*`, `prometheus_remote_storage_*` (if remote write enabled)
- Standard Go/process/promhttp/scrape boilerplate.

## Redis Target (redis_exporter)
Go-based redis exporter. Key operational series:
- **Health:** `redis_up`, `redis_instance_info`, `redis_uptime_in_seconds`,
  `redis_start_time_seconds`, `redis_exporter_last_scrape_error`
- **Memory:** `redis_memory_used_bytes`, `redis_memory_used_peak_bytes`,
  `redis_memory_max_bytes`, `redis_memory_used_rss_bytes`, `redis_mem_fragmentation_ratio`,
  `redis_mem_fragmentation_bytes`
- **Clients/connections:** `redis_connected_clients`, `redis_max_clients`,
  `redis_connections_received_total`, `redis_rejected_connections_total`,
  `redis_blocked_clients`
- **Commands/latency:** `redis_commands_processed_total`, `redis_commands_failed_calls_total`,
  `redis_commands_duration_seconds_total`, `redis_commands_latencies_usec_*`,
  `redis_latency_percentiles_usec`, `redis_errors_total`
- **Keys/eviction/expiry:** `redis_db_keys`, `redis_db_keys_expiring`, `redis_db_avg_ttl_seconds`,
  `redis_keyspace_hits_total`, `redis_keyspace_misses_total`, `redis_evicted_keys_total`,
  `redis_expired_keys_total`
- **Persistence:** `redis_rdb_last_bgsave_status`, `redis_rdb_last_bgsave_duration_sec`,
  `redis_aof_enabled`, `redis_aof_last_write_status`, `redis_aof_current_size_bytes`
- **Replication/pubsub:** `redis_connected_slaves`, `redis_master_repl_offset`,
  `redis_replication_backlog_bytes`, `redis_pubsub_channels`, `redis_pubsub_clients`
- **Config:** `redis_config_maxmemory`, `redis_config_maxclients`
- Standard process/promhttp/scrape boilerplate.

## Notes
- The only alert-related series present are Prometheus' standard `ALERTS` and
  `ALERTS_FOR_STATE` on the Chat and Playout targets; no concrete rule thresholds
  are recorded in this file.
- No dashboard definitions, alert thresholds, curl outputs, or numeric sample
  values are captured here — this document is purely the exposed-metric inventory
  per target.