zwitschi/linearvoid
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 9 Wiki Activity

docs: complete API error codes, deploy troubleshooting, monitoring/security rewrite, L2+L3 building blocks

Browse Source
This commit is contained in:
zwitschi committed 2026-08-06 20:00:45 +02:00
1 parent 0f81a81029
commit db3bf96847
5 files changed
+267 -205
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

API.md
+31 -6
View File
@@ -272,11 +272,36 @@ Prometheus exposition includes connected clients, stored messages, moderation ac
## HTTP Status Conventions ## HTTP Status Conventions
- 200/201: successful request | Code | Meaning | When |
- 400: malformed payload, invalid input, missing required fields | ---- | --------------------- | ----------------------------------------------------------- |
- 401: unauthorized admin request | 200 | OK | Successful GET, PUT, or action |
- 404: not found | 201 | Created | Successful POST (upload init, override created) |
- 405: method not allowed | 204 | No Content | Successful DELETE, CORS preflight |
- 409: logical conflict (for example, mismatched resumed upload) | 400 | Bad Request | Malformed JSON, missing required fields, invalid input |
| 401 | Unauthorized | Missing or expired admin session |
| 403 | Forbidden | Valid session but insufficient permissions |
| 404 | Not Found | Resource doesn't exist |
| 405 | Method Not Allowed | Wrong HTTP method for endpoint |
| 409 | Conflict | Schedule override overlap, upload ID mismatch |
| 413 | Payload Too Large | Upload exceeds 2 GB limit |
| 422 | Unprocessable | File fails normalization check (codec, fps, resolution) |
| 429 | Too Many Requests | Rate limit exceeded (chat messages, WS connections, login) |
| 500 | Internal Server Error | Unexpected server failure |
| 503 | Service Unavailable | Redis unavailable for user actions, playout health degraded |
## Rate Limiting
All limits are per-IP (from `X-Forwarded-For` or `RemoteAddr`) unless noted.
| Resource | Limit | Window | Response |
| --------------------- | ---------------------------------- | ---------------------------------------- | ------------------------------------------------------ |
| Chat messages | `CHAT_RATE_LIMIT_MAX` (default 10) | `CHAT_RATE_LIMIT_WINDOW_S` (default 10s) | 429 + `{"type":"system","text":"Rate limit exceeded"}` |
| WebSocket connections | 5 | 60s | 429 + `"too many connections from this IP"` |
| Admin login | 5 | 5min | 429 + `"too many login attempts, try again later"` |
Rate limit headers on applicable responses:
- `X-RateLimit-Limit`: maximum requests per window
- `Retry-After`: seconds until window resets
- 413: payload too large - 413: payload too large
- 500: internal server error - 500: internal server error
DEPLOYMENT.md
+14
View File
@@ -228,5 +228,19 @@ docker compose -f docker-compose.prod.yml down -v
- Missing frontend build artifacts in local-only compose workflows. - Missing frontend build artifacts in local-only compose workflows.
- Wrong registry credentials or missing login. - Wrong registry credentials or missing login.
- Misaligned domain env vars (app/ws/hls) causing mixed-origin errors. - Misaligned domain env vars (app/ws/hls) causing mixed-origin errors.
## Troubleshooting
| Symptom | Check | Fix |
| -------------------------- | --------------------------------------------- | ------------------------------------------------------------- |
| Service not starting | `docker logs linearvoid-<svc> --tail 50` | Check `.env` vars, volume mounts, port conflicts |
| Playout stall / slate mode | `curl http://<host>:8081/health` — mode=slate | Check playlist file, media paths, disk space |
| Chat not connecting | `curl http://<host>:3000/health` | Check Redis: `docker exec linearvoid-redis redis-cli ping` |
| Admin 401 loop | Cookie not sent | Verify `X-Forwarded-Proto: https`, nginx auth_request headers |
| Upload fails (413) | File too large | Check nginx `client_max_body_size` (should be 64m+) |
| HLS not playing | `curl http://<host>/hls/playlist.m3u8` | Check playout health, HLS dir permissions |
| Prometheus no targets | `curl http://<host>:9090/targets` | Verify container network, scrape config |
| Grafana no data | Check datasource | Verify Prometheus URL `http://prometheus:9090` provisioned |
- Proxy-layer body size limits returning 413 on upload. - Proxy-layer body size limits returning 413 on upload.
- Health checks failing before service warmup is complete. - Health checks failing before service warmup is complete.
MONITORING.md
+53 -99
View File
@@ -1,122 +1,76 @@
# Monitoring Guide # Monitoring Guide
This guide covers Prometheus and Grafana setup for Linear Void. Prometheus + Grafana monitoring for Linear Void.
## Current Monitoring Components ## Architecture
- Prometheus container: linearvoid-prometheus ```
- Grafana container: linearvoid-grafana ┌─────────────┐ ┌──────────────┐ ┌───────────┐
- Metrics target: chat service /metrics │ Exporters │───▶│ Prometheus │───▶│ Grafana │
- Base scrape config: monitoring/prometheus/prometheus.yml │ (7 targets) │ │ :9090 │ │ :3001 │
└─────────────┘ └──────┬───────┘ └───────────┘
## Prometheus Configuration │
alerts.yml (8 rules)
Current scrape config:
- Job chat -> chat:3000
- Job prometheus -> prometheus:9090
- Global scrape interval: 15s
File:
- monitoring/prometheus/prometheus.yml
## Start Monitoring Stack
With compose stack up:
- Prometheus UI: <http://localhost:9090>
- Grafana UI: <http://localhost:3001>
Compose services include prometheus and grafana in both docker-compose.yml and docker-compose.prod.yml.
## Grafana Provisioning
Current mount paths:
- Dashboards directory: monitoring/grafana/dashboards
- Datasources directory: monitoring/grafana/datasources
Notes:
- The dashboards directory includes a Prometheus overview dashboard file.
- Datasource provisioning can be done with a YAML file in monitoring/grafana/datasources, or by configuring a datasource in Grafana UI.
Example datasource provisioning file (create if needed):
```yaml
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
``` ```
## Dashboard References ## Scrape Targets
- Provisioned dashboard: [monitoring/grafana/dashboards/dashboard.json](monitoring/grafana/dashboards/dashboard.json) | Job | Target | Port | Metrics |
- Root dashboard snapshot: [monitoring/grafana/dashboard.json](monitoring/grafana/dashboard.json) | ---------- | ----------------- | ---- | ------------------------------------------------------------- |
| chat | chat | 3000 | WebSocket clients, messages, DMs, moderation, Redis, sessions |
| playout | playout | 8081 | Restarts, mode (live/slate) |
| nginx | nginx-exporter | 9113 | Requests, connections |
| redis | redis-exporter | 9121 | Memory, clients, commands |
| postgres | postgres-exporter | 9187 | Connections, queries, locks |
| node | node-exporter | 9100 | CPU, memory, disk, network |
| prometheus | prometheus | 9090 | Self-monitoring |
Grafana screenshot: Config: `monitoring/prometheus/prometheus.yml` (15s scrape interval)
![Grafana dashboard screenshot](docs/img/grafana-screenshot.png) ## Alerting Rules
## Key Metrics to Track 8 rules in `monitoring/prometheus/alerts.yml`:
### Chat/Application | Alert | Severity | Expression |
| -------------------- | -------- | -------------------------------------- |
| ChatDown | critical | `up{job="chat"} == 0` for 2m |
| PlayoutDown | critical | `up{job="playout"} == 0` for 2m |
| RedisDown | warning | `linearvoid_chat_redis_up == 0` for 2m |
| PlayoutSlateActive | warning | `linearvoid_playout_mode == 1` for 1m |
| HighChatLatency | warning | message rate <0.01/s for 5m |
| PrometheusWALFailing | warning | WAL write failures >0 for 5m |
| HighCPU | warning | CPU >80% for 10m |
| LowDiskSpace | warning | Disk <10% free for 5m |
- linearvoid_chat_connected_clients ## Access
- linearvoid_chat_stored_messages_total
- linearvoid_chat_moderation_actions_total
- linearvoid_chat_redis_enabled
- linearvoid_chat_redis_up
- linearvoid_chat_active_admin_sessions
### Prometheus Runtime - Prometheus: `http://<host>:9090` (targets, graph, alerts)
- Grafana: `http://<host>:3001` (default: admin/admin, change via `GRAFANA_PASSWORD`)
- up ## Dashboards
- process_cpu_seconds_total
- process_resident_memory_bytes
- prometheus_tsdb_head_series
- prometheus_tsdb_wal_writes_failed_total
- prometheus_tsdb_compactions_failed_total
## Suggested Alerts - **Linear Void app**: `monitoring/grafana/dashboard.json` — clients, messages, moderation, Redis, sessions, scrape health
- **Prometheus ops**: `monitoring/grafana/dashboards/dashboard.json` — TSDB, CPU/memory, WAL, compactions, query latency
Create a Prometheus rules file (for example monitoring/prometheus/alerts.yml) and wire it into Prometheus. ## Key Metrics
Recommended baseline alerts: - `linearvoid_chat_connected_clients` — active WebSocket connections
- `linearvoid_chat_messages_total` — total stored messages
- Chat target down for 2m - `linearvoid_chat_dms_total` — private messages sent
- Prometheus target down for 2m - `linearvoid_chat_redis_up` — Redis reachable (0/1)
- Redis indicator down for 2m - `linearvoid_playout_restarts_total` — FFmpeg restart counter
- TSDB WAL/compaction failure rate above zero - `linearvoid_playout_mode` — 0=live, 1=slate, 2=degraded
- Elevated handler/query latency p99 - `linearvoid_chat_admin_sessions_active` — active admin logins
## Validate Monitoring End-to-End
1. Verify /metrics through nginx:
- <https://linearvoid.tv/metrics>
1. Confirm targets in Prometheus:
- Status -> Targets page in Prometheus UI
1. Confirm dashboard data in Grafana:
- Import or open provisioned dashboards
## Troubleshooting ## Troubleshooting
### No chat metrics in Prometheus | Symptom | Check |
| ----------------------- | -------------------------------------------------------------------- |
- Check chat container health and logs. | No chat metrics | `curl http://chat:3000/metrics` from Prometheus container |
- Confirm /metrics is reachable from Prometheus container. | No exporter metrics | Verify exporter containers running: `docker ps \| grep exporter` |
- Verify scrape target is chat:3000 in prometheus.yml. | Grafana no data | Datasource URL must be `http://prometheus:9090` |
| Prometheus targets down | Check container network: all must be on `linearvoid-net` |
| Alert not firing | Check Prometheus → Alerts tab, verify `rule_files` in prometheus.yml |
### Grafana shows no data ### Grafana shows no data
SECURITY.md
+64 -98
View File
@@ -1,127 +1,93 @@
# Security Guide # Security Guide
This document summarizes security controls and best practices for Linear Void. Layered security model for Linear Void.
## Security Model Overview ## Threat Model
Security is layered across: ```
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Client │───▶│Cloudflare│───▶│ nginx │───▶│ Backend │
│ (public) │ │ WAF+CDN │ │ proxy │ │ chat │
└──────────┘ └──────────┘ └────┬─────┘ └────┬─────┘
│ │
┌──────┴──────┐ ┌─────┴──────┐
│ /health │ │ /api/admin │
│ /metrics │ │ /ws (chat) │
│ /hls/* │ │ /api/chat/* │
│ (public) │ │ (auth req) │
└─────────────┘ └────────────┘
```
- Cloudflare edge ## Authentication & Sessions
- nginx reverse proxy
- chat/admin backend
- container and host operations
## Authentication and Sessions - Login: `POST /api/admin/login` → cookie `linearvoid_admin_session`
- Cookie flags: `HttpOnly`, `SameSite=Strict`, `Secure` (over HTTPS)
- Server-side expiry via `ADMIN_SESSION_TTL_HOURS`
- Rate limit: 5 attempts/IP/5min via Redis
Admin access is session-cookie based. ## Transport Security (TLS/HSTS)
- Login endpoint: /api/admin/login - Cloudflare TLS mode: **Full (strict)** — valid origin cert required
- Session endpoint: /api/admin/session - nginx listens on 80 internally; Cloudflare handles TLS termination
- Logout endpoint: /api/admin/logout - HSTS header set by Cloudflare: `max-age=31536000; includeSubDomains; preload`
- Cookie: linearvoid_admin_session - WSS upgrade path: `/ws` through nginx → chat:3000
- Cookie flags: HttpOnly, SameSite=Strict, Secure over HTTPS
- Session expiry enforced server-side
Best practices: ## Network Isolation
- Set strong ADMIN_USERNAME and ADMIN_PASSWORD values. - All backend services on internal Docker network (`linearvoid-net`)
- Rotate admin credentials periodically. - Only nginx port exposed to host (443/80)
- Keep ADMIN_SESSION_TTL_HOURS aligned with operational policy. - Database, Redis, exporters inaccessible from outside
- `docker-compose.prod.yml` keeps ports internal via `expose:` not `ports:`
## Transport Security
- Enforce HTTPS for app/admin endpoints.
- Use WSS for chat WebSocket.
- Keep Cloudflare TLS mode strict and valid at origin.
## Reverse Proxy Controls ## Reverse Proxy Controls
nginx controls include: - `auth_request /api/admin/auth` gate on all `/admin` and `/api/admin` paths
- Explicit `client_max_body_size 64m` for resumable upload chunks
- WebSocket upgrade headers: `Upgrade`, `Connection` preserved
- CORS headers on HLS: `Access-Control-Allow-Origin *`
- XSS protection: CSP headers served by Cloudflare
- auth_request guard for /admin and /api/admin paths ## Rate Limiting
- dedicated unauthorized behavior for admin API responses
- explicit websocket upgrade headers for /ws
- upload body-size limits for resumable media chunk uploads
Best practices: | Resource | Limit | Window |
| ----------------- | ----- | ------ |
| WebSocket connect | 5/IP | 60s |
| Chat messages | 10/IP | 10s |
| Admin login | 5/IP | 5min |
- Keep proxy body-size and timeout settings aligned across all proxy layers. Implemented via Redis `INCR` + `EXPIRE`.
- Avoid exposing backend service ports publicly unless required.
## Input Validation and Upload Safety ## Secrets Management
Server-side protections include: - `.env` — local dev only, **gitignored**
- Gitea Actions secrets (`CI_*` prefix) for CI/CD
- Never commit: passwords, tokens, API keys, R2 credentials
- Production `.env` provisioned manually on host, never checked in
- Rotate admin password quarterly
- extension allow-list for media uploads ## Data Protection
- MIME checks
- file-size ceiling (2GB)
- resumable chunk bounds and chunk-index validation
- media path safety checks on delete
Best practices: - PostgreSQL: chat history, moderation logs, media metadata
- Redis: append-only persistence for presence, rate limits, sessions
- Keep upload chunk size bounded. - HLS segments: ephemeral, regenerated by playout
- Reject unsupported content types consistently. - Backups: `scripts/backup-db.sh` (daily), `scripts/backup-weekly.sh`, `scripts/verify-backups.sh`
- Monitor 4xx/5xx rates on upload routes.
## Rate Limiting and Abuse Controls
- Chat rate limiting uses Redis INCR + EXPIRE windows.
- Current baseline: more than 10 messages in 10 seconds is throttled.
- Moderation actions are logged for auditability.
Best practices:
- Tune thresholds by observed abuse patterns.
- Keep moderation logs reviewed and retained per policy.
## Infrastructure and Secrets
- Store secrets in .env (not in committed source).
- Use Gitea Actions secrets for CI/CD credentials and admin credentials.
- Restrict Docker host control plane exposure.
- Limit who can read deployment environment files.
## Cloudflare and Edge Security
Recommended baseline:
- WAF enabled
- Rate limits on sensitive paths
- Bot protection where appropriate
- Cache rules scoped carefully to avoid caching private/control-plane content
## Data Protection and Retention
- Persistent data: postgres, redis append-only, chat state, playlist/media metadata
- Backup and restore drills should be periodic
- Restore procedures should be tested in isolated environments
## Logging and Monitoring for Security
Track and alert on:
- repeated failed admin login attempts
- spikes in 401 and 403 responses
- sudden upload errors (413/400/500 patterns)
- unusual websocket disconnect or message-rate anomalies
## Hardening Checklist ## Hardening Checklist
- [ ] HTTPS/WSS enforced end-to-end - [ ] HTTPS/WSS enforced end-to-end (Cloudflare Full strict)
- [ ] Strong admin credentials configured - [ ] Strong admin credentials configured
- [ ] Secrets only in env/secrets stores - [ ] Secrets only in env/secrets stores (never committed)
- [ ] Proxy auth and websocket headers validated - [ ] Proxy auth and websocket headers validated
- [ ] Upload limits and validation active - [ ] Upload limits and MIME validation active
- [ ] Rate limiting active and tested - [ ] Rate limiting active (WS, messages, login)
- [ ] Cloudflare WAF/rate-limit rules active - [ ] Cloudflare WAF/rate-limit rules active
- [ ] Backup/restore drills completed - [ ] Backup/restore drills completed
- [ ] Security-related monitoring alerts routed - [ ] Security alerts routed (Prometheus alertmanager)
## Incident Response Baseline ## Incident Response
1. Detect issue (alerts/logs/user reports). 1. Detect — alerts (ChatDown, RedisDown, HighChatLatency) or user reports
2. Triage scope (edge, proxy, app, data plane). 2. Triage — edge (Cloudflare), proxy (nginx), app (chat/playout), data (Redis/Postgres)
3. Contain impact (rate limit, credential rotate, route disable). 3. Contain — rate limit, credential rotate, service restart, slate mode activation
4. Recover services and verify health. 4. Recover — verify health endpoints, check Prometheus targets, validate HLS output
5. Record timeline and corrective actions. 5. Record — timeline + corrective actions in `docs/incident-runbook.md`
docs/architecture/README.md
+105 -2
View File
@@ -259,11 +259,114 @@ Detailed black-box descriptions, level-2/3 views, and code locations are in `adr
### Level 2 ### Level 2
Deferred to implementation (see ADR for component responsibilities, interfaces, quality attributes, and risks). Playout enforces normalization; Delivery layer focuses on cache rules and immutable naming. Detailed component diagram with code locations and persistence.
```mermaid
graph TB
subgraph "Cloudflare Edge"
CF[CDN + WAF<br/>Cache Rules]
R2[R2 Bucket<br/>media + HLS]
PAGES[Pages<br/>SPA hosting]
end
subgraph "Docker Host (Hetzner CX22)"
NGINX[nginx<br/>:80/:443<br/>proxy + auth]
CHAT[chat service<br/>:3000<br/>Go + gorilla/websocket]
PLAYOUT[playout service<br/>:8081<br/>Go + FFmpeg]
REDIS[Redis 7<br/>:6379<br/>presence/rate/pubsub]
DB[PostgreSQL 15<br/>:5432<br/>chat history/moderation]
end
subgraph "External"
VIEWER[Viewer Browser<br/>hls.js + WebSocket]
ADMIN[Admin Browser<br/>SPA + uploads]
end
VIEWER -->|HTTPS| CF
VIEWER -->|WSS| NGINX
ADMIN -->|HTTPS| CF
CF --> NGINX
NGINX --> CHAT
NGINX --> PLAYOUT
NGINX -->|/hls/*| R2
CHAT --> REDIS
CHAT --> DB
PLAYOUT --> R2
PLAYOUT -->|reads| R2
```
#### Playout Service (`backend/playout/src/playlist.go`)
- **Role**: Continuous HLS encoding loop using FFmpeg concat demuxer
- **Input**: `data/playlist/playlist.txt` — ordered list of MP4 file paths
- **Output**: HLS segments (`.ts`) + live manifest (`.m3u8`) written to `data/hls/`, mirrored to R2
- **Resilience**: Restart counter (>5/60s → activate slate), Prometheus metrics on port 8081
- **Validation**: ffprobe gate on all ingested files (codec, fps, resolution, keyframe interval)
#### Chat Service (`backend/chat/src/main.go`)
- **Role**: WebSocket chat with rooms, DMs, moderation, admin API
- **Persistence**: PostgreSQL 15 via GORM (messages, moderation logs, media metadata). SQLite for dev fallback
- **Redis**: Presence tracking, rate limiting, pub/sub for multi-instance fan-out
- **Auth**: Session-cookie admin auth with Redis-backed expiry
- **Metrics**: Connected clients, stored messages, DMs, moderation actions on `/metrics`
#### Frontend SPA (`frontend/`)
- **Viewer**: `src/main.ts` — hls.js player, WebSocket chat, presence indicators, schedule overlay
- **Admin**: `src/admin.ts` — 9 dynamic tabs (dashboard, upload, playlist, schedule, moderation, chat-log, users, analytics, system)
- **Build**: Vite → static output in `frontend/dist/`
#### Reverse Proxy (`infrastructure/nginx/nginx.conf`)
- auth_request gate on `/admin` and `/api/admin`
- WebSocket upgrade for `/ws`
- HLS CORS headers + OPTIONS preflight
- `/health` JSON endpoint, `/nginx-status` stub_status
### Level 3 ### Level 3
Deferred to implementation (see ADR for component responsibilities, interfaces, quality attributes, and risks). Playout enforces normalization; Delivery layer focuses on cache rules and immutable naming. Module-level breakdown per service.
#### Chat Service Modules
| Module | File/Lines | Responsibility |
| ---------------------- | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| **Connection Manager** | `main.go` (hub, clients map) | Track active WebSocket connections per room, fan-out messages |
| **Message Parser** | `main.go` (readPump, handleMessage) | Parse JSON envelope `{type, room?, ...}`, validate fields, route to handler |
| **Room Registry** | `main.go` (rooms map, room broadcasts) | Map room name → client set, handle join/leave/close |
| **DM Router** | `main.go` (handleDirectMessage) | Resolve target user in clients map, deliver private message |
| **Persistence Layer** | `main.go` (GORM models: Message, DirectMessage, ModerationLog) | Store/retrieve chat messages, DMs, moderation events |
| **Redis Subsystem** | `main.go` (redisClient, rate limiting, pub/sub, presence) | INCR+EXPIRE rate limit, SETEX presence, PSUBSCRIBE room fan-out |
| **Auth Handler** | `main.go` (handleAdminLogin, adminAuthMiddleware) | Session cookie creation/validation, login rate limiting |
| **Admin API Router** | `main.go` (handleAdmin\*) | Upload, playlist CRUD, schedule overrides, moderation actions, analytics, system summary |
| **Metrics Exporter** | `main.go` (prometheus.MustRegister) | Expose connected_clients, messages_total, dms_total, moderation_actions, redis_up |
#### Playout Service Modules
| Module | File/Lines | Responsibility |
| -------------------- | ---------------------------------------------- | ------------------------------------------------------------------ |
| **Playlist Parser** | `playlist.go` (parsePlaylist) | Read `playlist.txt`, skip comments/blank lines, resolve file paths |
| **FFmpeg Manager** | `playlist.go` (runPlayout, restart logic) | Spawn/monitor FFmpeg concat process, restart on crash |
| **Slate Controller** | `playlist.go` (activateSlate, deactivateSlate) | Switch output to slate loop on excessive failures |
| **R2 Sync** | `playlist.go` (syncToR2) | Upload HLS segments + manifest to R2 bucket |
| **Health Endpoint** | `playlist.go` (healthHandler) | `/health` → JSON with mode, segment count, uptime |
| **Metrics Exporter** | `playlist.go` (prometheus) | Expose restarts_total, mode (live/slate) |
#### Frontend Modules
| Module | File | Responsibility |
| ----------------------- | ----------------------------------------------- | ----------------------------------------------------------------- |
| **HLS Player** | `src/main.ts` | hls.js initialization, attach to video element, error recovery |
| **WebSocket Client** | `src/main.ts` | Connect/disconnect, parse envelope, dispatch to UI handlers |
| **Chat UI** | `src/main.ts` | Message list rendering, room tabs, DM panel, presence dots |
| **Schedule Overlay** | `src/main.ts` | Poll `/api/schedule`, render current/next program |
| **Admin Router** | `src/admin.ts` | Tab-based navigation, dynamic page loading |
| **Upload Manager** | `src/admin.ts` | Multipart upload with chunk progress, ffprobe validation feedback |
| **Playlist Editor** | `src/admin.ts` | Drag-reorder, add/remove items, save to `/api/admin/playlist` |
| **Moderation Panel** | `src/admin.ts` + `public/admin/moderation.html` | Inline mute/ban, action log viewer |
| **Analytics Dashboard** | `src/admin.ts` + `public/admin/analytics.html` | Message counts, top rooms, moderation stats |
| **System Config** | `src/admin.ts` + `public/admin/system.html` | Service health, config metadata, restart buttons |
## Runtime View ## Runtime View