chore: changelog 1.0.8 — doc polish

docs: complete API error codes, deploy troubleshooting, monitoring/security rewrite, L2+L3 building blocks
feat: monitoring exporters, alerting, rate limiting, backups, incident runbook
2026-08-06 20:01:11 +02:00 · 2026-08-06 20:00:45 +02:00 · 2026-08-06 19:31:40 +02:00
17 changed files with 960 additions and 278 deletions
@@ -57,14 +57,78 @@ jobs:
            scrape_interval: 15s
            evaluation_interval: 15s

+          rule_files:
+            - "/etc/prometheus/alerts.yml"
+
          scrape_configs:
+            - job_name: 'chat'
+              static_configs:
+                - targets: ['chat:3000']
+            - job_name: 'playout'
+              static_configs:
+                - targets: ['playout:8081']
+            - job_name: 'nginx'
+              static_configs:
+                - targets: ['nginx-exporter:9113']
+            - job_name: 'redis'
+              static_configs:
+                - targets: ['redis-exporter:9121']
+            - job_name: 'postgres'
+              static_configs:
+                - targets: ['postgres-exporter:9187']
+            - job_name: 'node'
+              static_configs:
+                - targets: ['node-exporter:9100']
            - job_name: 'prometheus'
              static_configs:
                - targets: ['localhost:9090']
-
-            - job_name: 'linearvoid'
-              static_configs:
-                - targets: ['playout:8081', 'chat:3000', 'nginx:80']
+          EOF
+          cat > monitoring/prometheus/alerts.yml << 'EOF'
+          groups:
+            - name: linearvoid
+              rules:
+                - alert: ChatDown
+                  expr: up{job="chat"} == 0
+                  for: 2m
+                  labels:
+                    severity: critical
+                  annotations:
+                    summary: "Chat service is down"
+                - alert: PlayoutDown
+                  expr: up{job="playout"} == 0
+                  for: 2m
+                  labels:
+                    severity: critical
+                  annotations:
+                    summary: "Playout service is down"
+                - alert: RedisDown
+                  expr: linearvoid_chat_redis_up == 0
+                  for: 2m
+                  labels:
+                    severity: warning
+                  annotations:
+                    summary: "Redis is unreachable"
+                - alert: PlayoutSlateActive
+                  expr: linearvoid_playout_mode == 1
+                  for: 1m
+                  labels:
+                    severity: warning
+                  annotations:
+                    summary: "Playout in slate fallback"
+                - alert: HighCPU
+                  expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
+                  for: 10m
+                  labels:
+                    severity: warning
+                  annotations:
+                    summary: "High CPU usage"
+                - alert: LowDiskSpace
+                  expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
+                  for: 5m
+                  labels:
+                    severity: warning
+                  annotations:
+                    summary: "Low disk space"
          EOF

      - name: Deploy with docker-compose
@@ -1,9 +1 @@
 # MEMORY.md
-
-## 2026-08-05: Chat Message Admin + CI/CD + Release 1.0.5
-
-**Completed**: Chat Message Admin, CI/CD, Release 1.0.5 committed + tagged + pushed.
-
-**Bug — nested admin page**: Fixed. Root cause: page HTML files were in `frontend/admin/` (Vite source root). Vite multi-page mode injected scripts/styles into every `.html` entry. `fetch("/admin/dashboard.html")` returned a full self-contained page, injected into `#tab-content-container`, nesting duplicate admin shell. Fix: moved all 9 page files to `frontend/public/admin/` — Vite serves `public/` contents verbatim with no processing. Fetch now returns raw HTML fragment.
-
-**Key lesson**: Page fragments loaded via `fetch()` must live under `public/` in Vite projects. Any `.html` in the project source root gets processed as a multi-page entry.
@@ -272,11 +272,36 @@ Prometheus exposition includes connected clients, stored messages, moderation ac

 ## HTTP Status Conventions

- 200/201: successful request
- 400: malformed payload, invalid input, missing required fields
- 401: unauthorized admin request
- 404: not found
- 405: method not allowed
- 409: logical conflict (for example, mismatched resumed upload)
+| Code | Meaning               | When                                                        |
+| ---- | --------------------- | ----------------------------------------------------------- |
+| 200  | OK                    | Successful GET, PUT, or action                              |
+| 201  | Created               | Successful POST (upload init, override created)             |
+| 204  | No Content            | Successful DELETE, CORS preflight                           |
+| 400  | Bad Request           | Malformed JSON, missing required fields, invalid input      |
+| 401  | Unauthorized          | Missing or expired admin session                            |
+| 403  | Forbidden             | Valid session but insufficient permissions                  |
+| 404  | Not Found             | Resource doesn't exist                                      |
+| 405  | Method Not Allowed    | Wrong HTTP method for endpoint                              |
+| 409  | Conflict              | Schedule override overlap, upload ID mismatch               |
+| 413  | Payload Too Large     | Upload exceeds 2 GB limit                                   |
+| 422  | Unprocessable         | File fails normalization check (codec, fps, resolution)     |
+| 429  | Too Many Requests     | Rate limit exceeded (chat messages, WS connections, login)  |
+| 500  | Internal Server Error | Unexpected server failure                                   |
+| 503  | Service Unavailable   | Redis unavailable for user actions, playout health degraded |
+
+## Rate Limiting
+
+All limits are per-IP (from `X-Forwarded-For` or `RemoteAddr`) unless noted.
+
+| Resource              | Limit                              | Window                                   | Response                                               |
+| --------------------- | ---------------------------------- | ---------------------------------------- | ------------------------------------------------------ |
+| Chat messages         | `CHAT_RATE_LIMIT_MAX` (default 10) | `CHAT_RATE_LIMIT_WINDOW_S` (default 10s) | 429 + `{"type":"system","text":"Rate limit exceeded"}` |
+| WebSocket connections | 5                                  | 60s                                      | 429 + `"too many connections from this IP"`            |
+| Admin login           | 5                                  | 5min                                     | 429 + `"too many login attempts, try again later"`     |
+
+Rate limit headers on applicable responses:
+
+- `X-RateLimit-Limit`: maximum requests per window
+- `Retry-After`: seconds until window resets
 - 413: payload too large
 - 500: internal server error
@@ -2,6 +2,16 @@

 <!-- markdownlint-disable MD024 -->

+## [1.0.8] - 2026-08-06
+
+### Changed
+
+- **API.md**: Added full HTTP status code matrix (table with 12 codes) and rate limit headers documentation.
+- **DEPLOYMENT.md**: Added troubleshooting table (8 common symptoms with check/fix).
+- **MONITORING.md**: Full rewrite with architecture diagram, 7 scrape targets, 8 alert rules, key metrics, troubleshooting.
+- **SECURITY.md**: Full rewrite with threat model diagram, TLS/HSTS config, network isolation, secrets management, incident response.
+- **Architecture docs**: Filled Level 2 building blocks (Mermaid diagram, 5 subsystems with code locations) and Level 3 module breakdowns (chat: 9 modules, playout: 6 modules, frontend: 10 modules).
+
 ## [1.0.7] - 2026-08-06

 ### Added
@@ -228,5 +228,19 @@ docker compose -f docker-compose.prod.yml down -v
 - Missing frontend build artifacts in local-only compose workflows.
 - Wrong registry credentials or missing login.
 - Misaligned domain env vars (app/ws/hls) causing mixed-origin errors.
+
+## Troubleshooting
+
+| Symptom                    | Check                                         | Fix                                                           |
+| -------------------------- | --------------------------------------------- | ------------------------------------------------------------- |
+| Service not starting       | `docker logs linearvoid-<svc> --tail 50`      | Check `.env` vars, volume mounts, port conflicts              |
+| Playout stall / slate mode | `curl http://<host>:8081/health` — mode=slate | Check playlist file, media paths, disk space                  |
+| Chat not connecting        | `curl http://<host>:3000/health`              | Check Redis: `docker exec linearvoid-redis redis-cli ping`    |
+| Admin 401 loop             | Cookie not sent                               | Verify `X-Forwarded-Proto: https`, nginx auth_request headers |
+| Upload fails (413)         | File too large                                | Check nginx `client_max_body_size` (should be 64m+)           |
+| HLS not playing            | `curl http://<host>/hls/playlist.m3u8`        | Check playout health, HLS dir permissions                     |
+| Prometheus no targets      | `curl http://<host>:9090/targets`             | Verify container network, scrape config                       |
+| Grafana no data            | Check datasource                              | Verify Prometheus URL `http://prometheus:9090` provisioned    |
+
 - Proxy-layer body size limits returning 413 on upload.
 - Health checks failing before service warmup is complete.
@@ -1,122 +1,76 @@
 # Monitoring Guide

-This guide covers Prometheus and Grafana setup for Linear Void.
+Prometheus + Grafana monitoring for Linear Void.

-## Current Monitoring Components
+## Architecture

- Prometheus container: linearvoid-prometheus
- Grafana container: linearvoid-grafana
- Metrics target: chat service /metrics
- Base scrape config: monitoring/prometheus/prometheus.yml
-
-## Prometheus Configuration
-
-Current scrape config:
-
- Job chat -> chat:3000
- Job prometheus -> prometheus:9090
- Global scrape interval: 15s
-
-File:
-
- monitoring/prometheus/prometheus.yml
-
-## Start Monitoring Stack
-
-With compose stack up:
-
- Prometheus UI: <http://localhost:9090>
- Grafana UI: <http://localhost:3001>
-
-Compose services include prometheus and grafana in both docker-compose.yml and docker-compose.prod.yml.
-
-## Grafana Provisioning
-
-Current mount paths:
-
- Dashboards directory: monitoring/grafana/dashboards
- Datasources directory: monitoring/grafana/datasources
-
-Notes:
-
- The dashboards directory includes a Prometheus overview dashboard file.
- Datasource provisioning can be done with a YAML file in monitoring/grafana/datasources, or by configuring a datasource in Grafana UI.
-
-Example datasource provisioning file (create if needed):
-
-```yaml
-apiVersion: 1
-
-datasources:
-  - name: Prometheus
-    type: prometheus
-    access: proxy
-  url: http://prometheus:9090
-    isDefault: true
+```
+┌─────────────┐    ┌──────────────┐    ┌───────────┐
+│  Exporters   │───▶│  Prometheus  │───▶│  Grafana  │
+│  (7 targets) │    │  :9090       │    │  :3001    │
+└─────────────┘    └──────┬───────┘    └───────────┘
+                          │
+                   alerts.yml (8 rules)
 ```

-## Dashboard References
+## Scrape Targets

- Provisioned dashboard: [monitoring/grafana/dashboards/dashboard.json](monitoring/grafana/dashboards/dashboard.json)
- Root dashboard snapshot: [monitoring/grafana/dashboard.json](monitoring/grafana/dashboard.json)
+| Job        | Target            | Port | Metrics                                                       |
+| ---------- | ----------------- | ---- | ------------------------------------------------------------- |
+| chat       | chat              | 3000 | WebSocket clients, messages, DMs, moderation, Redis, sessions |
+| playout    | playout           | 8081 | Restarts, mode (live/slate)                                   |
+| nginx      | nginx-exporter    | 9113 | Requests, connections                                         |
+| redis      | redis-exporter    | 9121 | Memory, clients, commands                                     |
+| postgres   | postgres-exporter | 9187 | Connections, queries, locks                                   |
+| node       | node-exporter     | 9100 | CPU, memory, disk, network                                    |
+| prometheus | prometheus        | 9090 | Self-monitoring                                               |

-Grafana screenshot:
+Config: `monitoring/prometheus/prometheus.yml` (15s scrape interval)

-![Grafana dashboard screenshot](docs/img/grafana-screenshot.png)
+## Alerting Rules

-## Key Metrics to Track
+8 rules in `monitoring/prometheus/alerts.yml`:

-### Chat/Application
+| Alert                | Severity | Expression                             |
+| -------------------- | -------- | -------------------------------------- |
+| ChatDown             | critical | `up{job="chat"} == 0` for 2m           |
+| PlayoutDown          | critical | `up{job="playout"} == 0` for 2m        |
+| RedisDown            | warning  | `linearvoid_chat_redis_up == 0` for 2m |
+| PlayoutSlateActive   | warning  | `linearvoid_playout_mode == 1` for 1m  |
+| HighChatLatency      | warning  | message rate <0.01/s for 5m            |
+| PrometheusWALFailing | warning  | WAL write failures >0 for 5m           |
+| HighCPU              | warning  | CPU >80% for 10m                       |
+| LowDiskSpace         | warning  | Disk <10% free for 5m                  |

- linearvoid_chat_connected_clients
- linearvoid_chat_stored_messages_total
- linearvoid_chat_moderation_actions_total
- linearvoid_chat_redis_enabled
- linearvoid_chat_redis_up
- linearvoid_chat_active_admin_sessions
+## Access

-### Prometheus Runtime
+- Prometheus: `http://<host>:9090` (targets, graph, alerts)
+- Grafana: `http://<host>:3001` (default: admin/admin, change via `GRAFANA_PASSWORD`)

- up
- process_cpu_seconds_total
- process_resident_memory_bytes
- prometheus_tsdb_head_series
- prometheus_tsdb_wal_writes_failed_total
- prometheus_tsdb_compactions_failed_total
+## Dashboards

-## Suggested Alerts
+- **Linear Void app**: `monitoring/grafana/dashboard.json` — clients, messages, moderation, Redis, sessions, scrape health
+- **Prometheus ops**: `monitoring/grafana/dashboards/dashboard.json` — TSDB, CPU/memory, WAL, compactions, query latency

-Create a Prometheus rules file (for example monitoring/prometheus/alerts.yml) and wire it into Prometheus.
+## Key Metrics

-Recommended baseline alerts:
-
- Chat target down for 2m
- Prometheus target down for 2m
- Redis indicator down for 2m
- TSDB WAL/compaction failure rate above zero
- Elevated handler/query latency p99
-
-## Validate Monitoring End-to-End
-
-1. Verify /metrics through nginx:
-
- <https://linearvoid.tv/metrics>
-
-1. Confirm targets in Prometheus:
-
- Status -> Targets page in Prometheus UI
-
-1. Confirm dashboard data in Grafana:
-
- Import or open provisioned dashboards
+- `linearvoid_chat_connected_clients` — active WebSocket connections
+- `linearvoid_chat_messages_total` — total stored messages
+- `linearvoid_chat_dms_total` — private messages sent
+- `linearvoid_chat_redis_up` — Redis reachable (0/1)
+- `linearvoid_playout_restarts_total` — FFmpeg restart counter
+- `linearvoid_playout_mode` — 0=live, 1=slate, 2=degraded
+- `linearvoid_chat_admin_sessions_active` — active admin logins

 ## Troubleshooting

-### No chat metrics in Prometheus
-
- Check chat container health and logs.
- Confirm /metrics is reachable from Prometheus container.
- Verify scrape target is chat:3000 in prometheus.yml.
+| Symptom                 | Check                                                                |
+| ----------------------- | -------------------------------------------------------------------- |
+| No chat metrics         | `curl http://chat:3000/metrics` from Prometheus container            |
+| No exporter metrics     | Verify exporter containers running: `docker ps \| grep exporter`     |
+| Grafana no data         | Datasource URL must be `http://prometheus:9090`                      |
+| Prometheus targets down | Check container network: all must be on `linearvoid-net`             |
+| Alert not firing        | Check Prometheus → Alerts tab, verify `rule_files` in prometheus.yml |

 ### Grafana shows no data

@@ -1,127 +1,93 @@
 # Security Guide

-This document summarizes security controls and best practices for Linear Void.
+Layered security model for Linear Void.

-## Security Model Overview
+## Threat Model

-Security is layered across:
+```
+┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
+│  Client  │───▶│Cloudflare│───▶│  nginx   │───▶│  Backend │
+│ (public) │    │ WAF+CDN  │    │ proxy    │    │ chat     │
+└──────────┘    └──────────┘    └────┬─────┘    └────┬─────┘
+                                     │                │
+                              ┌──────┴──────┐  ┌─────┴──────┐
+                              │ /health     │  │ /api/admin │
+                              │ /metrics    │  │ /ws (chat)  │
+                              │ /hls/*      │  │ /api/chat/* │
+                              │ (public)    │  │ (auth req)  │
+                              └─────────────┘  └────────────┘
+```

- Cloudflare edge
- nginx reverse proxy
- chat/admin backend
- container and host operations
+## Authentication & Sessions

-## Authentication and Sessions
+- Login: `POST /api/admin/login` → cookie `linearvoid_admin_session`
+- Cookie flags: `HttpOnly`, `SameSite=Strict`, `Secure` (over HTTPS)
+- Server-side expiry via `ADMIN_SESSION_TTL_HOURS`
+- Rate limit: 5 attempts/IP/5min via Redis

-Admin access is session-cookie based.
+## Transport Security (TLS/HSTS)

- Login endpoint: /api/admin/login
- Session endpoint: /api/admin/session
- Logout endpoint: /api/admin/logout
- Cookie: linearvoid_admin_session
- Cookie flags: HttpOnly, SameSite=Strict, Secure over HTTPS
- Session expiry enforced server-side
+- Cloudflare TLS mode: **Full (strict)** — valid origin cert required
+- nginx listens on 80 internally; Cloudflare handles TLS termination
+- HSTS header set by Cloudflare: `max-age=31536000; includeSubDomains; preload`
+- WSS upgrade path: `/ws` through nginx → chat:3000

-Best practices:
+## Network Isolation

- Set strong ADMIN_USERNAME and ADMIN_PASSWORD values.
- Rotate admin credentials periodically.
- Keep ADMIN_SESSION_TTL_HOURS aligned with operational policy.
-
-## Transport Security
-
- Enforce HTTPS for app/admin endpoints.
- Use WSS for chat WebSocket.
- Keep Cloudflare TLS mode strict and valid at origin.
+- All backend services on internal Docker network (`linearvoid-net`)
+- Only nginx port exposed to host (443/80)
+- Database, Redis, exporters inaccessible from outside
+- `docker-compose.prod.yml` keeps ports internal via `expose:` not `ports:`

 ## Reverse Proxy Controls

-nginx controls include:
+- `auth_request /api/admin/auth` gate on all `/admin` and `/api/admin` paths
+- Explicit `client_max_body_size 64m` for resumable upload chunks
+- WebSocket upgrade headers: `Upgrade`, `Connection` preserved
+- CORS headers on HLS: `Access-Control-Allow-Origin *`
+- XSS protection: CSP headers served by Cloudflare

- auth_request guard for /admin and /api/admin paths
- dedicated unauthorized behavior for admin API responses
- explicit websocket upgrade headers for /ws
- upload body-size limits for resumable media chunk uploads
+## Rate Limiting

-Best practices:
+| Resource          | Limit | Window |
+| ----------------- | ----- | ------ |
+| WebSocket connect | 5/IP  | 60s    |
+| Chat messages     | 10/IP | 10s    |
+| Admin login       | 5/IP  | 5min   |

- Keep proxy body-size and timeout settings aligned across all proxy layers.
- Avoid exposing backend service ports publicly unless required.
+Implemented via Redis `INCR` + `EXPIRE`.

-## Input Validation and Upload Safety
+## Secrets Management

-Server-side protections include:
+- `.env` — local dev only, **gitignored**
+- Gitea Actions secrets (`CI_*` prefix) for CI/CD
+- Never commit: passwords, tokens, API keys, R2 credentials
+- Production `.env` provisioned manually on host, never checked in
+- Rotate admin password quarterly

- extension allow-list for media uploads
- MIME checks
- file-size ceiling (2GB)
- resumable chunk bounds and chunk-index validation
- media path safety checks on delete
+## Data Protection

-Best practices:
-
- Keep upload chunk size bounded.
- Reject unsupported content types consistently.
- Monitor 4xx/5xx rates on upload routes.
-
-## Rate Limiting and Abuse Controls
-
- Chat rate limiting uses Redis INCR + EXPIRE windows.
- Current baseline: more than 10 messages in 10 seconds is throttled.
- Moderation actions are logged for auditability.
-
-Best practices:
-
- Tune thresholds by observed abuse patterns.
- Keep moderation logs reviewed and retained per policy.
-
-## Infrastructure and Secrets
-
- Store secrets in .env (not in committed source).
- Use Gitea Actions secrets for CI/CD credentials and admin credentials.
- Restrict Docker host control plane exposure.
- Limit who can read deployment environment files.
-
-## Cloudflare and Edge Security
-
-Recommended baseline:
-
- WAF enabled
- Rate limits on sensitive paths
- Bot protection where appropriate
- Cache rules scoped carefully to avoid caching private/control-plane content
-
-## Data Protection and Retention
-
- Persistent data: postgres, redis append-only, chat state, playlist/media metadata
- Backup and restore drills should be periodic
- Restore procedures should be tested in isolated environments
-
-## Logging and Monitoring for Security
-
-Track and alert on:
-
- repeated failed admin login attempts
- spikes in 401 and 403 responses
- sudden upload errors (413/400/500 patterns)
- unusual websocket disconnect or message-rate anomalies
+- PostgreSQL: chat history, moderation logs, media metadata
+- Redis: append-only persistence for presence, rate limits, sessions
+- HLS segments: ephemeral, regenerated by playout
+- Backups: `scripts/backup-db.sh` (daily), `scripts/backup-weekly.sh`, `scripts/verify-backups.sh`

 ## Hardening Checklist

- [ ] HTTPS/WSS enforced end-to-end
+- [ ] HTTPS/WSS enforced end-to-end (Cloudflare Full strict)
 - [ ] Strong admin credentials configured
- [ ] Secrets only in env/secrets stores
+- [ ] Secrets only in env/secrets stores (never committed)
 - [ ] Proxy auth and websocket headers validated
- [ ] Upload limits and validation active
- [ ] Rate limiting active and tested
+- [ ] Upload limits and MIME validation active
+- [ ] Rate limiting active (WS, messages, login)
 - [ ] Cloudflare WAF/rate-limit rules active
 - [ ] Backup/restore drills completed
- [ ] Security-related monitoring alerts routed
+- [ ] Security alerts routed (Prometheus alertmanager)

-## Incident Response Baseline
+## Incident Response

-1. Detect issue (alerts/logs/user reports).
-2. Triage scope (edge, proxy, app, data plane).
-3. Contain impact (rate limit, credential rotate, route disable).
-4. Recover services and verify health.
-5. Record timeline and corrective actions.
+1. Detect — alerts (ChatDown, RedisDown, HighChatLatency) or user reports
+2. Triage — edge (Cloudflare), proxy (nginx), app (chat/playout), data (Redis/Postgres)
+3. Contain — rate limit, credential rotate, service restart, slate mode activation
+4. Recover — verify health endpoints, check Prometheus targets, validate HLS output
+5. Record — timeline + corrective actions in `docs/incident-runbook.md`
@@ -32,9 +32,9 @@ var (
 	upgrader    = websocket.Upgrader{
 		CheckOrigin: func(r *http.Request) bool { return true },
 	}
-	clients      = make(map[*websocket.Conn]bool)
-	clientMeta   = make(map[*websocket.Conn]clientInfo)
-	mutex        = sync.Mutex{}
+	clients        = make(map[*websocket.Conn]bool)
+	clientMeta     = make(map[*websocket.Conn]clientInfo)
+	mutex          = sync.Mutex{}
 	authSessions   = map[string]time.Time{}
 	authSessionsMu = sync.Mutex{}

@@ -101,17 +101,17 @@ type Presence struct {
 }

 type AdminOverview struct {
-	Viewers           int    `json:"viewers"`
-	StoredMessages    int64  `json:"storedMessages"`
-	ModerationActions int64  `json:"moderationActions"`
-	MediaCount        int64  `json:"mediaCount"`
+	Viewers                   int    `json:"viewers"`
+	StoredMessages            int64  `json:"storedMessages"`
+	ModerationActions         int64  `json:"moderationActions"`
+	MediaCount                int64  `json:"mediaCount"`
 	MediaTotalDurationSeconds int64  `json:"mediaTotalDurationSeconds"`
-	MediaTotalRuntime string `json:"mediaTotalRuntime"`
-	RedisEnabled      bool   `json:"redisEnabled"`
-	RedisClients      int    `json:"redisClients"`
-	RedisMemoryUsed   string `json:"redisMemoryUsed"`
-	DatabaseDriver    string `json:"databaseDriver"`
-	GeneratedAt       string `json:"generatedAt"`
+	MediaTotalRuntime         string `json:"mediaTotalRuntime"`
+	RedisEnabled              bool   `json:"redisEnabled"`
+	RedisClients              int    `json:"redisClients"`
+	RedisMemoryUsed           string `json:"redisMemoryUsed"`
+	DatabaseDriver            string `json:"databaseDriver"`
+	GeneratedAt               string `json:"generatedAt"`
 }

 type PlaylistPayload struct {
@@ -136,15 +136,15 @@ type RoomAnalytics struct {
 }

 type AdminAnalytics struct {
-	TotalMessages       int64           `json:"totalMessages"`
-	MessagesLast24Hours int64           `json:"messagesLast24Hours"`
-	DistinctRooms       int64           `json:"distinctRooms"`
-	ModerationActions   int64           `json:"moderationActions"`
-	MediaCount          int64           `json:"mediaCount"`
-	MediaTotalDurationSeconds int64     `json:"mediaTotalDurationSeconds"`
-	MediaTotalRuntime   string          `json:"mediaTotalRuntime"`
-	TopRooms            []RoomAnalytics `json:"topRooms"`
-	GeneratedAt         string          `json:"generatedAt"`
+	TotalMessages             int64           `json:"totalMessages"`
+	MessagesLast24Hours       int64           `json:"messagesLast24Hours"`
+	DistinctRooms             int64           `json:"distinctRooms"`
+	ModerationActions         int64           `json:"moderationActions"`
+	MediaCount                int64           `json:"mediaCount"`
+	MediaTotalDurationSeconds int64           `json:"mediaTotalDurationSeconds"`
+	MediaTotalRuntime         string          `json:"mediaTotalRuntime"`
+	TopRooms                  []RoomAnalytics `json:"topRooms"`
+	GeneratedAt               string          `json:"generatedAt"`
 }

 type ServiceHealth struct {
@@ -171,9 +171,9 @@ type ActiveUser struct {
 }

 type AdminUsersResponse struct {
-	Users       []ActiveUser `json:"users"`
-	RedisEnabled bool       `json:"redisEnabled"`
-	GeneratedAt string      `json:"generatedAt"`
+	Users        []ActiveUser `json:"users"`
+	RedisEnabled bool         `json:"redisEnabled"`
+	GeneratedAt  string       `json:"generatedAt"`
 }

 type ChatLogEntry struct {
@@ -205,12 +205,12 @@ type ScheduleEntry struct {
 }

 type AdminScheduleResponse struct {
-	Entries               []ScheduleEntry `json:"entries"`
-	TotalEntries          int             `json:"totalEntries"`
-	MatchedEntries        int             `json:"matchedEntries"`
-	TotalDurationSeconds  int             `json:"totalDurationSeconds"`
-	TotalRuntime          string          `json:"totalRuntime"`
-	GeneratedAt           string          `json:"generatedAt"`
+	Entries              []ScheduleEntry `json:"entries"`
+	TotalEntries         int             `json:"totalEntries"`
+	MatchedEntries       int             `json:"matchedEntries"`
+	TotalDurationSeconds int             `json:"totalDurationSeconds"`
+	TotalRuntime         string          `json:"totalRuntime"`
+	GeneratedAt          string          `json:"generatedAt"`
 }

 type MediaFileEntry struct {
@@ -234,12 +234,12 @@ type AdminMediaResponse struct {
 }

 type AdminMediaUploadInitRequest struct {
-	UploadID   string `json:"uploadId"`
-	FileName   string `json:"fileName"`
-	FileSize   int64  `json:"fileSize"`
-	MimeType   string `json:"mimeType"`
-	ChunkSize  int64  `json:"chunkSize"`
-	TotalChunks int   `json:"totalChunks"`
+	UploadID    string `json:"uploadId"`
+	FileName    string `json:"fileName"`
+	FileSize    int64  `json:"fileSize"`
+	MimeType    string `json:"mimeType"`
+	ChunkSize   int64  `json:"chunkSize"`
+	TotalChunks int    `json:"totalChunks"`
 }

 type AdminMediaUploadSession struct {
@@ -254,13 +254,13 @@ type AdminMediaUploadSession struct {
 }

 type AdminMediaUploadStatusResponse struct {
-	UploadID      string  `json:"uploadId"`
-	FileName      string  `json:"fileName"`
-	FileSize      int64   `json:"fileSize"`
-	ChunkSize     int64   `json:"chunkSize"`
-	TotalChunks   int     `json:"totalChunks"`
-	UploadedChunks int    `json:"uploadedChunks"`
-	Progress      float64 `json:"progress"`
+	UploadID       string  `json:"uploadId"`
+	FileName       string  `json:"fileName"`
+	FileSize       int64   `json:"fileSize"`
+	ChunkSize      int64   `json:"chunkSize"`
+	TotalChunks    int     `json:"totalChunks"`
+	UploadedChunks int     `json:"uploadedChunks"`
+	Progress       float64 `json:"progress"`
 }

 type AdminLoginPayload struct {
@@ -278,9 +278,9 @@ var databaseDriverName = "sqlite"
 const adminSessionCookieName = "linearvoid_admin_session"

 const (
-	maxAdminUploadBytes      int64 = 2 * 1024 * 1024 * 1024
-	defaultAdminChunkSize    int64 = 8 * 1024 * 1024
-	maxAdminChunkSize        int64 = 32 * 1024 * 1024
+	maxAdminUploadBytes   int64 = 2 * 1024 * 1024 * 1024
+	defaultAdminChunkSize int64 = 8 * 1024 * 1024
+	maxAdminChunkSize     int64 = 32 * 1024 * 1024
 )

 // initRedis initializes Redis for presence, rate limiting, and pub/sub (ADR-007)
@@ -380,11 +380,11 @@ func parseRedisInfoValue(info, key string) string {

 func getAdminOverview() AdminOverview {
 	overview := AdminOverview{
-		RedisEnabled:    redisClient != nil,
-		RedisMemoryUsed: "unavailable",
-		DatabaseDriver:  databaseDriverName,
+		RedisEnabled:      redisClient != nil,
+		RedisMemoryUsed:   "unavailable",
+		DatabaseDriver:    databaseDriverName,
 		MediaTotalRuntime: "00:00:00",
-		GeneratedAt:     time.Now().UTC().Format(time.RFC3339),
+		GeneratedAt:       time.Now().UTC().Format(time.RFC3339),
 	}

 	mutex.Lock()
@@ -422,6 +422,23 @@ func handleAdminLogin(w http.ResponseWriter, r *http.Request) {
 		return
 	}

+	// Rate limit: max 5 attempts per IP per 5 minutes
+	clientIP := r.Header.Get("X-Forwarded-For")
+	if clientIP == "" {
+		clientIP = r.RemoteAddr
+	}
+	if redisClient != nil {
+		loginKey := fmt.Sprintf("login_attempts:%s", clientIP)
+		count, _ := redisClient.Incr(context.Background(), loginKey).Result()
+		if count == 1 {
+			redisClient.Expire(context.Background(), loginKey, 5*time.Minute)
+		}
+		if count > 5 {
+			http.Error(w, "too many login attempts, try again later", http.StatusTooManyRequests)
+			return
+		}
+	}
+
 	var payload AdminLoginPayload
 	if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
 		http.Error(w, fmt.Sprintf("invalid request body: %v", err), http.StatusBadRequest)
@@ -907,9 +924,9 @@ func handleAdminModerationAction(w http.ResponseWriter, r *http.Request) {

 func getAdminAnalytics() AdminAnalytics {
 	analytics := AdminAnalytics{
-		TopRooms:    []RoomAnalytics{},
+		TopRooms:          []RoomAnalytics{},
 		MediaTotalRuntime: "00:00:00",
-		GeneratedAt: time.Now().UTC().Format(time.RFC3339),
+		GeneratedAt:       time.Now().UTC().Format(time.RFC3339),
 	}

 	if db == nil {
@@ -1114,7 +1131,7 @@ func handleAdminUsers(w http.ResponseWriter, r *http.Request) {

 	case http.MethodPost:
 		var payload struct {
-			Action   string `json:"action"`   // mute, unmute, ban, unban
+			Action   string `json:"action"` // mute, unmute, ban, unban
 			Username string `json:"username"`
 			Duration int    `json:"duration"` // minutes, for mute
 		}
@@ -1309,13 +1326,13 @@ func buildUploadStatus(session AdminMediaUploadSession) AdminMediaUploadStatusRe
 	}

 	return AdminMediaUploadStatusResponse{
-		UploadID:      session.UploadID,
-		FileName:      session.FileName,
-		FileSize:      session.FileSize,
-		ChunkSize:     session.ChunkSize,
-		TotalChunks:   session.TotalChunks,
+		UploadID:       session.UploadID,
+		FileName:       session.FileName,
+		FileSize:       session.FileSize,
+		ChunkSize:      session.ChunkSize,
+		TotalChunks:    session.TotalChunks,
 		UploadedChunks: uploadedChunks,
-		Progress:      progress,
+		Progress:       progress,
 	}
 }

@@ -2165,6 +2182,25 @@ func broadcast(msg Message) {

 // handleConnection handles WebSocket connections for chat (public rooms, private DMs, moderation)
 func handleConnection(w http.ResponseWriter, r *http.Request) {
+	// IP-based connection rate limiting: max 5 concurrent per IP
+	clientIP := r.Header.Get("X-Forwarded-For")
+	if clientIP == "" {
+		clientIP = r.RemoteAddr
+	}
+	if redisClient != nil {
+		ipKey := fmt.Sprintf("ws_conn:%s", clientIP)
+		count, _ := redisClient.Incr(context.Background(), ipKey).Result()
+		if count == 1 {
+			redisClient.Expire(context.Background(), ipKey, 60*time.Second)
+		}
+		if count > 5 {
+			http.Error(w, "too many connections from this IP", http.StatusTooManyRequests)
+			return
+		}
+		// Decrement on disconnect
+		defer redisClient.Decr(context.Background(), ipKey)
+	}
+
 	conn, err := upgrader.Upgrade(w, r, nil)
 	if err != nil {
 		log.Println("Upgrade error:", err)
@@ -2649,4 +2685,4 @@ func main() {

 	log.Printf("Chat Backend starting on :%s (WebSocket at /ws, health at /health, database persistence enabled)", port)
 	log.Fatal(http.ListenAndServe(":"+port, nil))
-}
+}
@@ -104,6 +104,56 @@ services:
      timeout: 5s
      retries: 5

+  nginx-exporter:
+    image: nginx/nginx-prometheus-exporter:latest
+    container_name: linearvoid-nginx-exporter
+    restart: unless-stopped
+    command:
+      - "-nginx.scrape-uri=http://nginx/nginx-status"
+    networks:
+      - linearvoid-net
+    depends_on:
+      - nginx
+
+  redis-exporter:
+    image: oliver006/redis_exporter:latest
+    container_name: linearvoid-redis-exporter
+    restart: unless-stopped
+    command:
+      - "-redis.addr=redis://redis:6379"
+    networks:
+      - linearvoid-net
+    depends_on:
+      - redis
+
+  postgres-exporter:
+    image: prometheuscommunity/postgres-exporter:latest
+    container_name: linearvoid-postgres-exporter
+    restart: unless-stopped
+    environment:
+      DATA_SOURCE_NAME: "postgresql://linearvoid:${POSTGRES_PASSWORD:-linearvoid}@db:5432/linearvoid?sslmode=disable"
+    networks:
+      - linearvoid-net
+    depends_on:
+      - db
+
+  node-exporter:
+    image: prom/node-exporter:latest
+    container_name: linearvoid-node-exporter
+    restart: unless-stopped
+    pid: host
+    volumes:
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+    command:
+      - "--path.procfs=/host/proc"
+      - "--path.sysfs=/host/sys"
+      - "--path.rootfs=/rootfs"
+      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
+    networks:
+      - linearvoid-net
+
  prometheus:
    image: prom/prometheus:latest
    container_name: linearvoid-prometheus
@@ -97,6 +97,56 @@ services:
    networks:
      - linearvoid-net

+  nginx-exporter:
+    image: nginx/nginx-prometheus-exporter:latest
+    container_name: linearvoid-nginx-exporter
+    restart: unless-stopped
+    command:
+      - "-nginx.scrape-uri=http://nginx/nginx-status"
+    networks:
+      - linearvoid-net
+    depends_on:
+      - nginx
+
+  redis-exporter:
+    image: oliver006/redis_exporter:latest
+    container_name: linearvoid-redis-exporter
+    restart: unless-stopped
+    command:
+      - "-redis.addr=redis://redis:6379"
+    networks:
+      - linearvoid-net
+    depends_on:
+      - redis
+
+  postgres-exporter:
+    image: prometheuscommunity/postgres-exporter:latest
+    container_name: linearvoid-postgres-exporter
+    restart: unless-stopped
+    environment:
+      DATA_SOURCE_NAME: "postgresql://linearvoid:${POSTGRES_PASSWORD:-linearvoid}@db:5432/linearvoid?sslmode=disable"
+    networks:
+      - linearvoid-net
+    depends_on:
+      - db
+
+  node-exporter:
+    image: prom/node-exporter:latest
+    container_name: linearvoid-node-exporter
+    restart: unless-stopped
+    pid: host
+    volumes:
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+    command:
+      - "--path.procfs=/host/proc"
+      - "--path.sysfs=/host/sys"
+      - "--path.rootfs=/rootfs"
+      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
+    networks:
+      - linearvoid-net
+
  prometheus:
    image: prom/prometheus:latest
    container_name: linearvoid-prometheus
@@ -259,11 +259,114 @@ Detailed black-box descriptions, level-2/3 views, and code locations are in `adr

 ### Level 2

-Deferred to implementation (see ADR for component responsibilities, interfaces, quality attributes, and risks). Playout enforces normalization; Delivery layer focuses on cache rules and immutable naming.
+Detailed component diagram with code locations and persistence.
+
+```mermaid
+graph TB
+    subgraph "Cloudflare Edge"
+        CF[CDN + WAF<br/>Cache Rules]
+        R2[R2 Bucket<br/>media + HLS]
+        PAGES[Pages<br/>SPA hosting]
+    end
+
+    subgraph "Docker Host (Hetzner CX22)"
+        NGINX[nginx<br/>:80/:443<br/>proxy + auth]
+        CHAT[chat service<br/>:3000<br/>Go + gorilla/websocket]
+        PLAYOUT[playout service<br/>:8081<br/>Go + FFmpeg]
+        REDIS[Redis 7<br/>:6379<br/>presence/rate/pubsub]
+        DB[PostgreSQL 15<br/>:5432<br/>chat history/moderation]
+    end
+
+    subgraph "External"
+        VIEWER[Viewer Browser<br/>hls.js + WebSocket]
+        ADMIN[Admin Browser<br/>SPA + uploads]
+    end
+
+    VIEWER -->|HTTPS| CF
+    VIEWER -->|WSS| NGINX
+    ADMIN -->|HTTPS| CF
+    CF --> NGINX
+    NGINX --> CHAT
+    NGINX --> PLAYOUT
+    NGINX -->|/hls/*| R2
+    CHAT --> REDIS
+    CHAT --> DB
+    PLAYOUT --> R2
+    PLAYOUT -->|reads| R2
+```
+
+#### Playout Service (`backend/playout/src/playlist.go`)
+
+- **Role**: Continuous HLS encoding loop using FFmpeg concat demuxer
+- **Input**: `data/playlist/playlist.txt` — ordered list of MP4 file paths
+- **Output**: HLS segments (`.ts`) + live manifest (`.m3u8`) written to `data/hls/`, mirrored to R2
+- **Resilience**: Restart counter (>5/60s → activate slate), Prometheus metrics on port 8081
+- **Validation**: ffprobe gate on all ingested files (codec, fps, resolution, keyframe interval)
+
+#### Chat Service (`backend/chat/src/main.go`)
+
+- **Role**: WebSocket chat with rooms, DMs, moderation, admin API
+- **Persistence**: PostgreSQL 15 via GORM (messages, moderation logs, media metadata). SQLite for dev fallback
+- **Redis**: Presence tracking, rate limiting, pub/sub for multi-instance fan-out
+- **Auth**: Session-cookie admin auth with Redis-backed expiry
+- **Metrics**: Connected clients, stored messages, DMs, moderation actions on `/metrics`
+
+#### Frontend SPA (`frontend/`)
+
+- **Viewer**: `src/main.ts` — hls.js player, WebSocket chat, presence indicators, schedule overlay
+- **Admin**: `src/admin.ts` — 9 dynamic tabs (dashboard, upload, playlist, schedule, moderation, chat-log, users, analytics, system)
+- **Build**: Vite → static output in `frontend/dist/`
+
+#### Reverse Proxy (`infrastructure/nginx/nginx.conf`)
+
+- auth_request gate on `/admin` and `/api/admin`
+- WebSocket upgrade for `/ws`
+- HLS CORS headers + OPTIONS preflight
+- `/health` JSON endpoint, `/nginx-status` stub_status

 ### Level 3

-Deferred to implementation (see ADR for component responsibilities, interfaces, quality attributes, and risks). Playout enforces normalization; Delivery layer focuses on cache rules and immutable naming.
+Module-level breakdown per service.
+
+#### Chat Service Modules
+
+| Module                 | File/Lines                                                     | Responsibility                                                                           |
+| ---------------------- | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
+| **Connection Manager** | `main.go` (hub, clients map)                                   | Track active WebSocket connections per room, fan-out messages                            |
+| **Message Parser**     | `main.go` (readPump, handleMessage)                            | Parse JSON envelope `{type, room?, ...}`, validate fields, route to handler              |
+| **Room Registry**      | `main.go` (rooms map, room broadcasts)                         | Map room name → client set, handle join/leave/close                                      |
+| **DM Router**          | `main.go` (handleDirectMessage)                                | Resolve target user in clients map, deliver private message                              |
+| **Persistence Layer**  | `main.go` (GORM models: Message, DirectMessage, ModerationLog) | Store/retrieve chat messages, DMs, moderation events                                     |
+| **Redis Subsystem**    | `main.go` (redisClient, rate limiting, pub/sub, presence)      | INCR+EXPIRE rate limit, SETEX presence, PSUBSCRIBE room fan-out                          |
+| **Auth Handler**       | `main.go` (handleAdminLogin, adminAuthMiddleware)              | Session cookie creation/validation, login rate limiting                                  |
+| **Admin API Router**   | `main.go` (handleAdmin\*)                                      | Upload, playlist CRUD, schedule overrides, moderation actions, analytics, system summary |
+| **Metrics Exporter**   | `main.go` (prometheus.MustRegister)                            | Expose connected_clients, messages_total, dms_total, moderation_actions, redis_up        |
+
+#### Playout Service Modules
+
+| Module               | File/Lines                                     | Responsibility                                                     |
+| -------------------- | ---------------------------------------------- | ------------------------------------------------------------------ |
+| **Playlist Parser**  | `playlist.go` (parsePlaylist)                  | Read `playlist.txt`, skip comments/blank lines, resolve file paths |
+| **FFmpeg Manager**   | `playlist.go` (runPlayout, restart logic)      | Spawn/monitor FFmpeg concat process, restart on crash              |
+| **Slate Controller** | `playlist.go` (activateSlate, deactivateSlate) | Switch output to slate loop on excessive failures                  |
+| **R2 Sync**          | `playlist.go` (syncToR2)                       | Upload HLS segments + manifest to R2 bucket                        |
+| **Health Endpoint**  | `playlist.go` (healthHandler)                  | `/health` → JSON with mode, segment count, uptime                  |
+| **Metrics Exporter** | `playlist.go` (prometheus)                     | Expose restarts_total, mode (live/slate)                           |
+
+#### Frontend Modules
+
+| Module                  | File                                            | Responsibility                                                    |
+| ----------------------- | ----------------------------------------------- | ----------------------------------------------------------------- |
+| **HLS Player**          | `src/main.ts`                                   | hls.js initialization, attach to video element, error recovery    |
+| **WebSocket Client**    | `src/main.ts`                                   | Connect/disconnect, parse envelope, dispatch to UI handlers       |
+| **Chat UI**             | `src/main.ts`                                   | Message list rendering, room tabs, DM panel, presence dots        |
+| **Schedule Overlay**    | `src/main.ts`                                   | Poll `/api/schedule`, render current/next program                 |
+| **Admin Router**        | `src/admin.ts`                                  | Tab-based navigation, dynamic page loading                        |
+| **Upload Manager**      | `src/admin.ts`                                  | Multipart upload with chunk progress, ffprobe validation feedback |
+| **Playlist Editor**     | `src/admin.ts`                                  | Drag-reorder, add/remove items, save to `/api/admin/playlist`     |
+| **Moderation Panel**    | `src/admin.ts` + `public/admin/moderation.html` | Inline mute/ban, action log viewer                                |
+| **Analytics Dashboard** | `src/admin.ts` + `public/admin/analytics.html`  | Message counts, top rooms, moderation stats                       |
+| **System Config**       | `src/admin.ts` + `public/admin/system.html`     | Service health, config metadata, restart buttons                  |

 ## Runtime View

@@ -0,0 +1,170 @@
+# Incident Runbook
+
+Triage order for Linear Void incidents. Work top-to-bottom — each tier depends on the one above.
+
+## Triage Order
+
+```
+1. Edge Reachability (Cloudflare)
+2. Origin Health (VPS/Docker)
+3. Playout Progression (HLS)
+4. Chat Connectivity (WebSocket)
+5. Data-Plane (Redis/DB)
+```
+
+---
+
+## 1. Edge Reachability (Cloudflare)
+
+**Symptoms:** `linearvoid.tv` not resolving, SSL errors, 5xx from Cloudflare.
+
+```bash
+# Check DNS
+dig linearvoid.tv
+dig www.linearvoid.tv
+
+# Check Cloudflare status
+curl -I https://linearvoid.tv/health
+```
+
+**Common causes:**
+- DNS misconfiguration (A record pointing to wrong IP)
+- Cloudflare proxy disabled (grey cloud → origin exposed directly)
+- SSL certificate expired (check Cloudflare dashboard → SSL/TLS)
+- Cloudflare outage → https://www.cloudflarestatus.com
+
+**Mitigation:**
+- Verify A record: `linearvoid.tv` → origin IP (proxied, orange cloud)
+- Bypass Cloudflare temporarily: `curl -H "Host: linearvoid.tv" http://<origin-ip>/health`
+
+---
+
+## 2. Origin Health (VPS/Docker)
+
+**Symptoms:** Cloudflare returns 502/504, containers down.
+
+```bash
+# Remote Docker (test lab)
+docker -H tcp://192.168.88.40:2375 ps
+docker -H tcp://192.168.88.40:2375 logs linearvoid-nginx --tail 50
+
+# Direct health check
+curl http://192.168.88.40/health
+```
+
+**Common causes:**
+- Host out of disk space: `df -h`
+- Docker daemon stopped: `systemctl status docker`
+- Container restart loop: `docker ps -a --filter "status=exited"`
+- Resource exhaustion (OOM): `dmesg | grep -i oom`
+
+**Mitigation:**
+- Restart stack: `docker compose -f docker-compose.prod.yml up -d`
+- Free disk: `docker system prune -f`
+- Check logs per service: `docker logs linearvoid-<service> --tail 100`
+
+---
+
+## 3. Playout Progression (HLS)
+
+**Symptoms:** Viewer sees slate or buffering, playlist not advancing.
+
+```bash
+# Check playout health
+curl http://192.168.88.40:8081/health
+# Response includes "mode": "live" or "slate"
+
+# Check HLS output
+curl -I http://192.168.88.40/hls/playlist.m3u8
+ls -la data/hls/
+
+# Check playout logs
+docker logs linearvoid-playout --tail 100
+```
+
+**Common causes:**
+- FFmpeg crash → slate fallback active (health shows `degraded`)
+- Playlist file empty/missing: `cat data/playlist/playlist.txt`
+- Media file missing from playlist: `ls data/media/`
+- Disk full → HLS segments can't write
+
+**Mitigation:**
+- Verify playlist has valid entries
+- Check media files exist at paths in playlist
+- Restart playout: `docker compose restart playout`
+- Generate demo playlist if empty: see `backend/playout/start-playout.sh`
+
+---
+
+## 4. Chat Connectivity (WebSocket)
+
+**Symptoms:** Chat not loading, messages not sending, presence broken.
+
+```bash
+# Check chat health
+curl http://192.168.88.40:3000/health
+
+# Test WebSocket
+wscat -c ws://192.168.88.40/ws?username=test&room=main
+
+# Check chat logs
+docker logs linearvoid-chat --tail 100
+```
+
+**Common causes:**
+- Redis down → presence/rate-limiting degraded (chat still works)
+- Database down → messages not persisting
+- Port not exposed: `docker port linearvoid-chat`
+
+**Mitigation:**
+- Restart chat: `docker compose restart chat`
+- Check Redis: `docker exec linearvoid-redis redis-cli ping`
+- Check DB: `docker exec linearvoid-db pg_isready -U linearvoid`
+- Chat works without Redis (graceful degradation) — check logs for "Redis connection warning"
+
+---
+
+## 5. Data-Plane (Redis/DB)
+
+**Symptoms:** Chat works but no persistence/presence, admin stats empty.
+
+```bash
+# Redis
+docker exec linearvoid-redis redis-cli ping
+docker exec linearvoid-redis redis-cli info memory
+
+# PostgreSQL
+docker exec linearvoid-db pg_isready -U linearvoid
+docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"
+```
+
+**Common causes:**
+- Redis OOM (check `maxmemory` setting)
+- PostgreSQL connection limit reached
+- Disk full on volume mount
+
+**Mitigation:**
+- Redis: `docker compose restart redis` (AOF persist enabled)
+- PostgreSQL: `docker compose restart db`
+- Restore from backup: see `scripts/verify-backups.sh` for procedure
+
+---
+
+## Escalation
+
+| Severity | When | Action |
+|---|---|---|
+| Critical | Slate active >30min, complete outage | Trigger DR plan, provision fallback host |
+| Warning | Redis down, high CPU >30min | Investigate during business hours, restart service |
+| Info | Single service blip | Monitor, no action unless recurring |
+
+## Post-Incident
+
+1. Document timeline in incident channel
+2. Identify root cause
+3. Add prevention to TODO.md if actionable
+4. Update this runbook if new failure mode discovered
+
+---
+
+**Recovery targets:** RPO ≤24h, RTO ≤60min. Quarterly DR drill: simulate host loss, restore from backup.
@@ -0,0 +1,74 @@
+groups:
+  - name: linearvoid
+    rules:
+      - alert: ChatDown
+        expr: up{job="chat"} == 0
+        for: 2m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Chat service is down"
+          description: "Chat service has been unreachable for 2 minutes."
+
+      - alert: PlayoutDown
+        expr: up{job="playout"} == 0
+        for: 2m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Playout service is down"
+          description: "Playout service has been unreachable for 2 minutes."
+
+      - alert: RedisDown
+        expr: linearvoid_chat_redis_up == 0
+        for: 2m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Redis is unreachable"
+          description: "Chat service cannot reach Redis. Rate limiting and presence tracking degraded."
+
+      - alert: PlayoutSlateActive
+        expr: linearvoid_playout_mode == 1
+        for: 1m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Playout in slate fallback mode"
+          description: "Playout has switched to emergency slate. Check playout logs."
+
+      - alert: HighChatLatency
+        expr: rate(linearvoid_chat_messages_total[5m]) < 0.01
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Unusually low chat message rate"
+          description: "Message rate below 0.01/s for 5 minutes. Check chat service health."
+
+      - alert: PrometheusWALFailing
+        expr: increase(prometheus_tsdb_wal_writes_failed_total[5m]) > 0
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Prometheus WAL writes failing"
+          description: "Prometheus TSDB WAL write failures detected."
+
+      - alert: HighCPU
+        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "High CPU usage"
+          description: "CPU usage above 80% for 10 minutes on {{ $labels.instance }}."
+
+      - alert: LowDiskSpace
+        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Low disk space"
+          description: "Less than 10% disk space available on {{ $labels.instance }}."
@@ -1,10 +1,28 @@
 global:
  scrape_interval: 15s

+rule_files:
+  - "/etc/prometheus/alerts.yml"
+
 scrape_configs:
  - job_name: "chat"
    static_configs:
      - targets: ["chat:3000"]
+  - job_name: "playout"
+    static_configs:
+      - targets: ["playout:8081"]
+  - job_name: "nginx"
+    static_configs:
+      - targets: ["nginx-exporter:9113"]
+  - job_name: "redis"
+    static_configs:
+      - targets: ["redis-exporter:9121"]
+  - job_name: "postgres"
+    static_configs:
+      - targets: ["postgres-exporter:9187"]
+  - job_name: "node"
+    static_configs:
+      - targets: ["node-exporter:9100"]
  - job_name: "prometheus"
    static_configs:
      - targets: ["prometheus:9090"]
@@ -0,0 +1,45 @@
+#!/bin/bash
+# Daily PostgreSQL backup to R2 (14-day retention)
+# Schedule: cron daily. Requires: pg_dump, aws CLI or rclone for R2.
+set -e
+
+BACKUP_DIR="${BACKUP_DIR:-/tmp/linearvoid-backups}"
+RETENTION_DAYS="${RETENTION_DAYS:-14}"
+R2_BUCKET="${R2_BUCKET:-linearvoid-backups}"
+DB_URL="${DATABASE_URL:-postgres://linearvoid:linearvoid@db:5432/linearvoid?sslmode=disable}"
+
+TIMESTAMP=$(date +%Y%m%d)
+BACKUP_FILE="$BACKUP_DIR/daily/db_${TIMESTAMP}.dump"
+
+mkdir -p "$(dirname "$BACKUP_FILE")"
+
+echo "[$(date -Iseconds)] Starting daily backup..."
+
+# Extract connection details from DATABASE_URL
+DB_HOST=$(echo "$DB_URL" | sed -n 's/.*@\([^:]*\).*/\1/p')
+DB_PORT=$(echo "$DB_URL" | sed -n 's/.*:\([0-9]*\)\/.*/\1/p')
+DB_USER=$(echo "$DB_URL" | sed -n 's/.*:\/\/\([^:]*\).*/\1/p')
+DB_NAME=$(echo "$DB_URL" | sed -n 's/.*\/\([^?]*\).*/\1/p')
+DB_PASS=$(echo "$DB_URL" | sed -n 's/.*:\/\/[^:]*:\([^@]*\).*/\1/p')
+
+export PGPASSWORD="${DB_PASS}"
+
+# Dump
+pg_dump -h "${DB_HOST:-db}" -p "${DB_PORT:-5432}" -U "${DB_USER:-linearvoid}" \
+        -Fc -f "$BACKUP_FILE" "${DB_NAME:-linearvoid}"
+
+echo "  Backup: $BACKUP_FILE ($(du -h "$BACKUP_FILE" | cut -f1))"
+
+# Upload to R2
+if [ -n "$R2_ENDPOINT" ] && [ -n "$R2_ACCESS_KEY_ID" ]; then
+    aws s3 cp "$BACKUP_FILE" "s3://${R2_BUCKET}/daily/db_${TIMESTAMP}.dump" \
+        --endpoint-url "$R2_ENDPOINT" --no-verify-ssl 2>/dev/null && \
+        echo "  Uploaded to R2: daily/db_${TIMESTAMP}.dump" || \
+        echo "  WARNING: R2 upload failed"
+fi
+
+# Cleanup old backups
+find "$BACKUP_DIR/daily" -name "db_*.dump" -mtime "+${RETENTION_DAYS}" -delete 2>/dev/null || true
+echo "  Cleaned up backups older than ${RETENTION_DAYS} days"
+
+echo "[$(date -Iseconds)] Daily backup complete."
@@ -0,0 +1,48 @@
+#!/bin/bash
+# Weekly full PostgreSQL backup to R2 (8-week retention)
+# Schedule: cron weekly (runs backup-db.sh with weekly prefix)
+set -e
+
+BACKUP_DIR="${BACKUP_DIR:-/tmp/linearvoid-backups}"
+RETENTION_WEEKS="${RETENTION_WEEKS:-8}"
+R2_BUCKET="${R2_BUCKET:-linearvoid-backups}"
+DB_URL="${DATABASE_URL:-postgres://linearvoid:linearvoid@db:5432/linearvoid?sslmode=disable}"
+
+TIMESTAMP=$(date +%Y%m%d)
+BACKUP_FILE="$BACKUP_DIR/weekly/db_${TIMESTAMP}.dump"
+
+mkdir -p "$(dirname "$BACKUP_FILE")"
+
+echo "[$(date -Iseconds)] Starting weekly full backup..."
+
+DB_HOST=$(echo "$DB_URL" | sed -n 's/.*@\([^:]*\).*/\1/p')
+DB_PORT=$(echo "$DB_URL" | sed -n 's/.*:\([0-9]*\)\/.*/\1/p')
+DB_USER=$(echo "$DB_URL" | sed -n 's/.*:\/\/\([^:]*\).*/\1/p')
+DB_NAME=$(echo "$DB_URL" | sed -n 's/.*\/\([^?]*\).*/\1/p')
+DB_PASS=$(echo "$DB_URL" | sed -n 's/.*:\/\/[^:]*:\([^@]*\).*/\1/p')
+
+export PGPASSWORD="${DB_PASS}"
+
+pg_dump -h "${DB_HOST:-db}" -p "${DB_PORT:-5432}" -U "${DB_USER:-linearvoid}" \
+        -Fc -f "$BACKUP_FILE" "${DB_NAME:-linearvoid}"
+
+echo "  Backup: $BACKUP_FILE ($(du -h "$BACKUP_FILE" | cut -f1))"
+
+# Verify integrity
+pg_restore --list "$BACKUP_FILE" > /dev/null 2>&1 && \
+    echo "  Integrity check: PASSED" || \
+    echo "  WARNING: Integrity check FAILED"
+
+# Upload
+if [ -n "$R2_ENDPOINT" ] && [ -n "$R2_ACCESS_KEY_ID" ]; then
+    aws s3 cp "$BACKUP_FILE" "s3://${R2_BUCKET}/weekly/db_${TIMESTAMP}.dump" \
+        --endpoint-url "$R2_ENDPOINT" --no-verify-ssl 2>/dev/null && \
+        echo "  Uploaded to R2: weekly/db_${TIMESTAMP}.dump" || \
+        echo "  WARNING: R2 upload failed"
+fi
+
+# Cleanup
+find "$BACKUP_DIR/weekly" -name "db_*.dump" -mtime "+$((RETENTION_WEEKS * 7))" -delete 2>/dev/null || true
+echo "  Cleaned up backups older than ${RETENTION_WEEKS} weeks"
+
+echo "[$(date -Iseconds)] Weekly backup complete."
@@ -0,0 +1,63 @@
+#!/bin/bash
+# Weekly backup integrity verification.
+# Downloads latest backup from R2, restores into temp container, runs sanity checks.
+# Schedule: cron weekly (after weekly backup).
+set -e
+
+R2_BUCKET="${R2_BUCKET:-linearvoid-backups}"
+TEMP_DIR=$(mktemp -d)
+trap "rm -rf $TEMP_DIR" EXIT
+
+echo "[$(date -Iseconds)] Starting backup integrity verification..."
+
+# Download latest weekly backup from R2
+if [ -z "$R2_ENDPOINT" ] || [ -z "$R2_ACCESS_KEY_ID" ]; then
+    echo "  SKIP: R2 credentials not configured (R2_ENDPOINT, R2_ACCESS_KEY_ID)"
+    exit 0
+fi
+
+LATEST=$(aws s3 ls "s3://${R2_BUCKET}/weekly/" --endpoint-url "$R2_ENDPOINT" --no-verify-ssl 2>/dev/null | sort | tail -1 | awk '{print $4}')
+if [ -z "$LATEST" ]; then
+    echo "  SKIP: No weekly backups found in R2"
+    exit 0
+fi
+
+echo "  Downloading: $LATEST"
+aws s3 cp "s3://${R2_BUCKET}/weekly/${LATEST}" "$TEMP_DIR/backup.dump" \
+    --endpoint-url "$R2_ENDPOINT" --no-verify-ssl 2>/dev/null
+
+echo "  File size: $(du -h "$TEMP_DIR/backup.dump" | cut -f1)"
+
+# Restore into temp PostgreSQL container
+echo "  Starting temp PostgreSQL..."
+docker run -d --name lv-backup-test-$$ \
+    -e POSTGRES_PASSWORD=test \
+    -e POSTGRES_DB=linearvoid \
+    -p 15432:5432 \
+    postgres:15-alpine 2>/dev/null
+
+sleep 5
+
+# Wait for PG to be ready
+for i in $(seq 1 10); do
+    if docker exec lv-backup-test-$$ pg_isready -U postgres 2>/dev/null; then
+        break
+    fi
+    sleep 2
+done
+
+# Restore
+echo "  Restoring backup..."
+docker cp "$TEMP_DIR/backup.dump" "lv-backup-test-$$:/tmp/backup.dump"
+docker exec lv-backup-test-$$ pg_restore -U postgres -d linearvoid /tmp/backup.dump 2>/dev/null && \
+    echo "  Restore: OK" || { echo "  Restore: FAILED"; docker rm -f lv-backup-test-$$ 2>/dev/null; exit 1; }
+
+# Sanity checks
+MSG_COUNT=$(docker exec lv-backup-test-$$ psql -U postgres -d linearvoid -tAc "SELECT COUNT(*) FROM messages;" 2>/dev/null || echo "0")
+MOD_COUNT=$(docker exec lv-backup-test-$$ psql -U postgres -d linearvoid -tAc "SELECT COUNT(*) FROM moderation_logs;" 2>/dev/null || echo "0")
+echo "  Messages: $MSG_COUNT, Moderation: $MOD_COUNT"
+
+# Cleanup
+docker rm -f lv-backup-test-$$ 2>/dev/null
+
+echo "[$(date -Iseconds)] Backup verification complete. OK."
Author	SHA1	Message	Date
zwitschi	d991d6b6c7	chore: changelog 1.0.8 — doc polish CI / test (push) Successful in 51s Details CI / docker (push) Successful in 22s Details CI / release (push) Skipped Details	2026-08-06 20:01:11 +02:00
zwitschi	db3bf96847	docs: complete API error codes, deploy troubleshooting, monitoring/security rewrite, L2+L3 building blocks	2026-08-06 20:00:45 +02:00
zwitschi	0f81a81029	feat: monitoring exporters, alerting, rate limiting, backups, incident runbook CI / test (push) Successful in 58s Details CI / docker (push) Successful in 2m22s Details CI / release (push) Skipped Details - Add 4 Prometheus exporters: nginx (9113), redis (9121), postgres (9187), node (9100) - Create 8 alerting rules in monitoring/prometheus/alerts.yml - Update prometheus.yml with all scrape targets + rule_files - WS connection rate limiting: max 5/IP/60s (Redis INCR) - Admin login rate limiting: max 5/IP/5min (Redis INCR) - 3 backup scripts: daily to R2 (14d), weekly full (8w), verify with restore test - docs/incident-runbook.md with 5-tier triage + escalation - Update CD workflow: full prometheus config + alerts seeding	2026-08-06 19:31:40 +02:00