chore: release v1.0.9 - monitoring provisioning + cleanup tracking files

docs: update backlog
feat: enhance chat service with Redis pub/sub for multi-instance message handling, add instance ID for deduplication
2026-08-06 21:28:53 +02:00 · 2026-08-06 20:46:54 +02:00 · 2026-08-06 20:44:29 +02:00 · 2026-08-06 20:01:11 +02:00 · 2026-08-06 20:00:45 +02:00 · 2026-08-06 19:31:40 +02:00
23 changed files with 1088 additions and 378 deletions
@@ -57,14 +57,78 @@ jobs:
            scrape_interval: 15s
            evaluation_interval: 15s
          rule_files:
            - "/etc/prometheus/alerts.yml"
          scrape_configs:
            - job_name: 'chat'
              static_configs:
                - targets: ['chat:3000']
            - job_name: 'playout'
              static_configs:
                - targets: ['playout:8081']
            - job_name: 'nginx'
              static_configs:
                - targets: ['nginx-exporter:9113']
            - job_name: 'redis'
              static_configs:
                - targets: ['redis-exporter:9121']
            - job_name: 'postgres'
              static_configs:
                - targets: ['postgres-exporter:9187']
            - job_name: 'node'
              static_configs:
                - targets: ['node-exporter:9100']
            - job_name: 'prometheus'
              static_configs:
                - targets: ['localhost:9090']
-
+          EOF
-            - job_name: 'linearvoid'
+          cat > monitoring/prometheus/alerts.yml << 'EOF'
-              static_configs:
+          groups:
-                - targets: ['playout:8081', 'chat:3000', 'nginx:80']
+            - name: linearvoid
              rules:
                - alert: ChatDown
                  expr: up{job="chat"} == 0
                  for: 2m
                  labels:
                    severity: critical
                  annotations:
                    summary: "Chat service is down"
                - alert: PlayoutDown
                  expr: up{job="playout"} == 0
                  for: 2m
                  labels:
                    severity: critical
                  annotations:
                    summary: "Playout service is down"
                - alert: RedisDown
                  expr: linearvoid_chat_redis_up == 0
                  for: 2m
                  labels:
                    severity: warning
                  annotations:
                    summary: "Redis is unreachable"
                - alert: PlayoutSlateActive
                  expr: linearvoid_playout_mode == 1
                  for: 1m
                  labels:
                    severity: warning
                  annotations:
                    summary: "Playout in slate fallback"
                - alert: HighCPU
                  expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
                  for: 10m
                  labels:
                    severity: warning
                  annotations:
                    summary: "High CPU usage"
                - alert: LowDiskSpace
                  expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
                  for: 5m
                  labels:
                    severity: warning
                  annotations:
                    summary: "Low disk space"
          EOF
      - name: Deploy with docker-compose
@@ -1,9 +0,0 @@
 # MEMORY.md
 ## 2026-08-05: Chat Message Admin + CI/CD + Release 1.0.5
 **Completed**: Chat Message Admin, CI/CD, Release 1.0.5 committed + tagged + pushed.
 **Bug — nested admin page**: Fixed. Root cause: page HTML files were in `frontend/admin/` (Vite source root). Vite multi-page mode injected scripts/styles into every `.html` entry. `fetch("/admin/dashboard.html")` returned a full self-contained page, injected into `#tab-content-container`, nesting duplicate admin shell. Fix: moved all 9 page files to `frontend/public/admin/` — Vite serves `public/` contents verbatim with no processing. Fetch now returns raw HTML fragment.
 **Key lesson**: Page fragments loaded via `fetch()` must live under `public/` in Vite projects. Any `.html` in the project source root gets processed as a multi-page entry.
@@ -1,79 +0,0 @@
 ---
 description: "Caveman-style agent for linearvoid project. Terse output, plan-first workflow, confirmation gating. Use for: implementing TODO.md tasks, editing project files, docs updates, release prep."
 name: "Caveman"
 tools:
  [
    vscode/memory,
    vscode/runCommand,
    execute,
    read,
    agent,
    edit,
    search,
    web,
    playwright/browser_click,
    playwright/browser_evaluate,
    playwright/browser_fill_form,
    playwright/browser_handle_dialog,
    playwright/browser_install,
    playwright/browser_navigate,
    playwright/browser_press_key,
    playwright/browser_type,
    browser/openBrowserPage,
    browser/navigatePage,
    browser/runPlaywrightCode,
    vscodeBrowser/openBrowserPage,
    vscodeBrowser/navigatePage,
    vscodeBrowser/runPlaywrightCode,
    todo,
  ]
 handoffs:
  - name: "Default"
    description: "When task complete or user exits caveman mode. Also handoff for security warnings or irreversible actions."
    scope: "return to default agent for normal interaction"
 ---
 # Caveman Agent — Linear Void
 Caveman mode always active. Terse, no fluff, no filler, no tool-call narration. Technical substance preserved exactly.
 ## Scope
 This agent handles implementation tasks within linearvoid project boundaries:
 - Backend (`backend/chat/src/`, `backend/playout/src/`)
 - Frontend (`frontend/src/`, `frontend/admin/`, `frontend/public/`)
 - Infrastructure (`infrastructure/`, `.gitea/workflows/`)
 - Documentation (`docs/`, `*.md` at project root)
 - Config (`docker-compose*.yml`, `.env.example`, `package.json`, `go.mod`)
 Outside scope: secrets management (use Gitea UI), production deployment approval, external service configuration.
 ## Output Rules
 Drop articles (a/an/the), filler words, pleasantries, hedging. Fragments OK. Short synonyms. Standard acronyms OK (API, DB, CI). Never invent abbreviations. Code blocks, error strings, technical terms verbatim. Never drop negations (not/never/no/only/except). Numbers and units exact.
 ## Project Workflow
 **Documentation first. Always.**
 1. **Read docs before code** — Check `docs/architecture/README.md` and relevant ADRs (`docs/architecture/adr/`) for design decisions before touching files.
 2. **Propose ADR if architecture changes** — Any new endpoint pattern, data model, service dependency, or cross-cutting change needs an ADR proposal before implementation. Write to `docs/architecture/adr/ADR-XXX.md` following existing template.
 3. **Plan** — Write tasks to `.github/instructions/TODO.md` with checkboxes.
 4. **Confirm before acting** — Ask user. Accept `go`, `ok`, `yes`, `next`, `continue`. Abort on `no`, `stop`, `cancel`.
 5. **Do the work** — Edit files, run commands.
 6. **Track completion** — Mark `[x]` in TODO.md. When section done, move to `.github/instructions/DONE.md`.
 7. **Log changes** — Write entry to `CHANGELOG.md` for significant work.
 8. **Sync memory** — Update `.github/MEMORY.md` between major steps and at end of session. Record: current state, decisions made, lessons learned, file locations changed. This is the agent's persistent working memory across sessions.
 ## Architecture
 Source of truth: `docs/architecture/README.md` and ADRs (`docs/architecture/adr/`). Follow project structure in root `README.md`.
 ## Security
 Never commit raw secrets. Never print or log secrets. Use `CI_` prefix for Gitea secrets (never `GITEA_`). `.env` always gitignored.
 ## Task Tracking
 Sync with `manage_todo_list` tool. Mirror state to `.github/instructions/TODO.md` checkboxes.
@@ -1,5 +1,7 @@
 # BACKLOG
 Status reflects 2026-08-06 code scan + implementation batches.
 ## Playout & Delivery
 - [x] Implement FFmpeg + MediaMTX/SRS continuous HLS playout from playlist
@@ -19,7 +21,7 @@
 - [x] Implement WebSocket chat backend (Go) with room fan-out
 - [x] Implement private DM channel routing
 - [x] Implement Redis presence tracking for chat
- [ ] Implement Redis pub/sub for multi-instance chat
+- [x] Implement Redis pub/sub for multi-instance chat
 - [x] Implement Redis rate limiting (INCR + EXPIRE windowing, >10/10s throttle)
 - [x] Implement chat message persistence (SQLite v1, PostgreSQL migration path)
 - [x] Implement moderation commands (/mod, /mute, /ban) in backend
@@ -55,15 +57,15 @@
 ## Infrastructure — Docker
 - [x] Create Dockerfiles for playout, chat, nginx services
- [x] Create docker-compose.yml for all services
+- [x] Create docker-compose.yml for all services (11 containers)
 - [x] Create docker-compose.prod.yml override for production
 ## Infrastructure — CI/CD (Gitea Actions)
- [x] Implement CI workflow (lint, build, test, push images)
+- [x] Implement CI workflow (lint, build, vet, push images)
 - [x] Implement CD workflow (pull, deploy, health checks — IP-agnostic)
 - [x] Generate .env from Gitea secrets in CD workflow
- [x] Seed Prometheus config into named volume during deploy
+- [x] Seed Prometheus config + alerts into named volume during deploy
 - [x] Implement health check validation with retry in CD pipeline
 ## Cloudflare Configuration
@@ -79,10 +81,10 @@
 ## Monitoring & Observability
- [x] Set up Prometheus with chat scrape target
+- [x] Set up Prometheus with chat/playout/nginx/redis/postgres/node scrape targets
- [ ] Add nginx/redis/postgres/node exporters
+- [x] Add nginx-exporter, redis-exporter, postgres-exporter, node-exporter
 - [x] Import Grafana app + operational dashboards
- [ ] Define Prometheus alerting rules
+- [x] Define Prometheus alerting rules (8 rules: ChatDown, PlayoutDown, RedisDown, etc.)
 - [ ] Set up UptimeRobot external monitoring
 ## Security
@@ -90,15 +92,18 @@
 - [x] Implement admin cookie-session auth (HttpOnly, SameSite=Strict, Secure when TLS)
 - [x] Implement server-side session expiration
 - [ ] Enforce HTTPS/WSS at edge and origin
- [x] Implement rate limiting for chat messages with Redis windowing
+- [x] Implement rate limiting for chat messages (Redis windowing)
 - [x] Implement WebSocket connection rate limiting (5/IP/60s)
 - [x] Implement admin login rate limiting (5/IP/5min)
 ## Data & Storage
 - [x] Set up PostgreSQL/SQLite for chat history and moderation data
 - [x] Set up Redis with append-only persistence + maxmemory policy
- [ ] Implement daily/weekly/monthly Postgres backups
+- [x] Implement daily Postgres backups to R2 (14-day retention)
- [ ] Implement backup integrity verification
+- [x] Implement weekly full backup snapshots (8-week retention)
- [ ] Document and rehearse recovery runbook
+- [x] Implement backup integrity verification (restore test)
 - [ ] Document and rehearse recovery runbook (quarterly drill)
 ## Scaling & Performance
@@ -114,8 +119,8 @@
 ## Documentation
 - [x] Document API contracts in API.md
- [ ] Document deployment runbook in DEPLOYMENT.md
+- [x] Document deployment runbook in DEPLOYMENT.md
- [ ] Document monitoring setup in MONITORING.md
+- [x] Document monitoring setup in MONITORING.md
- [ ] Document security model in SECURITY.md
+- [x] Document security model in SECURITY.md
- [ ] Create incident runbook
+- [x] Create incident runbook (docs/incident-runbook.md)
- [ ] Write Level 2/3 building block views
+- [x] Write Level 2/3 building block views
@@ -272,11 +272,36 @@ Prometheus exposition includes connected clients, stored messages, moderation ac
 ## HTTP Status Conventions
- 200/201: successful request
+| Code | Meaning               | When                                                        |
- 400: malformed payload, invalid input, missing required fields
+| ---- | --------------------- | ----------------------------------------------------------- |
- 401: unauthorized admin request
+| 200  | OK                    | Successful GET, PUT, or action                              |
- 404: not found
+| 201  | Created               | Successful POST (upload init, override created)             |
- 405: method not allowed
+| 204  | No Content            | Successful DELETE, CORS preflight                           |
- 409: logical conflict (for example, mismatched resumed upload)
+| 400  | Bad Request           | Malformed JSON, missing required fields, invalid input      |
 | 401  | Unauthorized          | Missing or expired admin session                            |
 | 403  | Forbidden             | Valid session but insufficient permissions                  |
 | 404  | Not Found             | Resource doesn't exist                                      |
 | 405  | Method Not Allowed    | Wrong HTTP method for endpoint                              |
 | 409  | Conflict              | Schedule override overlap, upload ID mismatch               |
 | 413  | Payload Too Large     | Upload exceeds 2 GB limit                                   |
 | 422  | Unprocessable         | File fails normalization check (codec, fps, resolution)     |
 | 429  | Too Many Requests     | Rate limit exceeded (chat messages, WS connections, login)  |
 | 500  | Internal Server Error | Unexpected server failure                                   |
 | 503  | Service Unavailable   | Redis unavailable for user actions, playout health degraded |
 ## Rate Limiting
 All limits are per-IP (from `X-Forwarded-For` or `RemoteAddr`) unless noted.
 | Resource              | Limit                              | Window                                   | Response                                               |
 | --------------------- | ---------------------------------- | ---------------------------------------- | ------------------------------------------------------ |
 | Chat messages         | `CHAT_RATE_LIMIT_MAX` (default 10) | `CHAT_RATE_LIMIT_WINDOW_S` (default 10s) | 429 + `{"type":"system","text":"Rate limit exceeded"}` |
 | WebSocket connections | 5                                  | 60s                                      | 429 + `"too many connections from this IP"`            |
 | Admin login           | 5                                  | 5min                                     | 429 + `"too many login attempts, try again later"`     |
 Rate limit headers on applicable responses:
 - `X-RateLimit-Limit`: maximum requests per window
 - `Retry-After`: seconds until window resets
 - 413: payload too large
 - 500: internal server error
@@ -2,6 +2,31 @@
 <!-- markdownlint-disable MD024 -->
 ## [1.0.9] - 2026-08-06
 ### Added
 - **Prometheus alerts mount**: `alerts.yml` mounted into prometheus container in both dev and prod compose files.
 - **Grafana provisioning**: Prometheus datasource + dashboard provider YAML configs in `monitoring/grafana/`.
 ### Changed
 - **Prod prometheus**: Switched from empty named volume to bind mounts for config files (consistent with dev).
 ### Removed
 - `.github/MEMORY.md`, `.github/agents/`, `.github/instructions/` — tracking files migrated out of repo.
 ## [1.0.8] - 2026-08-06
 ### Changed
 - **API.md**: Added full HTTP status code matrix (table with 12 codes) and rate limit headers documentation.
 - **DEPLOYMENT.md**: Added troubleshooting table (8 common symptoms with check/fix).
 - **MONITORING.md**: Full rewrite with architecture diagram, 7 scrape targets, 8 alert rules, key metrics, troubleshooting.
 - **SECURITY.md**: Full rewrite with threat model diagram, TLS/HSTS config, network isolation, secrets management, incident response.
 - **Architecture docs**: Filled Level 2 building blocks (Mermaid diagram, 5 subsystems with code locations) and Level 3 module breakdowns (chat: 9 modules, playout: 6 modules, frontend: 10 modules).
 ## [1.0.7] - 2026-08-06
 ### Added
@@ -228,5 +228,19 @@ docker compose -f docker-compose.prod.yml down -v
 - Missing frontend build artifacts in local-only compose workflows.
 - Wrong registry credentials or missing login.
 - Misaligned domain env vars (app/ws/hls) causing mixed-origin errors.
 ## Troubleshooting
 | Symptom                    | Check                                         | Fix                                                           |
 | -------------------------- | --------------------------------------------- | ------------------------------------------------------------- |
 | Service not starting       | `docker logs linearvoid-<svc> --tail 50`      | Check `.env` vars, volume mounts, port conflicts              |
 | Playout stall / slate mode | `curl http://<host>:8081/health` — mode=slate | Check playlist file, media paths, disk space                  |
 | Chat not connecting        | `curl http://<host>:3000/health`              | Check Redis: `docker exec linearvoid-redis redis-cli ping`    |
 | Admin 401 loop             | Cookie not sent                               | Verify `X-Forwarded-Proto: https`, nginx auth_request headers |
 | Upload fails (413)         | File too large                                | Check nginx `client_max_body_size` (should be 64m+)           |
 | HLS not playing            | `curl http://<host>/hls/playlist.m3u8`        | Check playout health, HLS dir permissions                     |
 | Prometheus no targets      | `curl http://<host>:9090/targets`             | Verify container network, scrape config                       |
 | Grafana no data            | Check datasource                              | Verify Prometheus URL `http://prometheus:9090` provisioned    |
 - Proxy-layer body size limits returning 413 on upload.
 - Health checks failing before service warmup is complete.
@@ -1,122 +1,76 @@
 # Monitoring Guide
-This guide covers Prometheus and Grafana setup for Linear Void.
+Prometheus + Grafana monitoring for Linear Void.
-## Current Monitoring Components
+## Architecture
- Prometheus container: linearvoid-prometheus
+```text
- Grafana container: linearvoid-grafana
+┌──────────────┐     ┌──────────────┐    ┌───────────┐
- Metrics target: chat service /metrics
+│  Exporters   │───> │  Prometheus  │───>│  Grafana  │
- Base scrape config: monitoring/prometheus/prometheus.yml
+│  (7 targets) │     │  :9090       │    │  :3001    │
-
+└──────────────┘     └───────┬──────┘    └───────────┘
-## Prometheus Configuration
+                             │
-
+                   alerts.yml (8 rules)
 Current scrape config:
 - Job chat -> chat:3000
 - Job prometheus -> prometheus:9090
 - Global scrape interval: 15s
 File:
 - monitoring/prometheus/prometheus.yml
 ## Start Monitoring Stack
 With compose stack up:
 - Prometheus UI: <http://localhost:9090>
 - Grafana UI: <http://localhost:3001>
 Compose services include prometheus and grafana in both docker-compose.yml and docker-compose.prod.yml.
 ## Grafana Provisioning
 Current mount paths:
 - Dashboards directory: monitoring/grafana/dashboards
 - Datasources directory: monitoring/grafana/datasources
 Notes:
 - The dashboards directory includes a Prometheus overview dashboard file.
 - Datasource provisioning can be done with a YAML file in monitoring/grafana/datasources, or by configuring a datasource in Grafana UI.
 Example datasource provisioning file (create if needed):
 ```yaml
 apiVersion: 1
 datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
  url: http://prometheus:9090
    isDefault: true
 ```
-## Dashboard References
+## Scrape Targets
- Provisioned dashboard: [monitoring/grafana/dashboards/dashboard.json](monitoring/grafana/dashboards/dashboard.json)
+| Job        | Target            | Port | Metrics                                                       |
- Root dashboard snapshot: [monitoring/grafana/dashboard.json](monitoring/grafana/dashboard.json)
+| ---------- | ----------------- | ---- | ------------------------------------------------------------- |
 | chat       | chat              | 3000 | WebSocket clients, messages, DMs, moderation, Redis, sessions |
 | playout    | playout           | 8081 | Restarts, mode (live/slate)                                   |
 | nginx      | nginx-exporter    | 9113 | Requests, connections                                         |
 | redis      | redis-exporter    | 9121 | Memory, clients, commands                                     |
 | postgres   | postgres-exporter | 9187 | Connections, queries, locks                                   |
 | node       | node-exporter     | 9100 | CPU, memory, disk, network                                    |
 | prometheus | prometheus        | 9090 | Self-monitoring                                               |
-Grafana screenshot:
+Config: `monitoring/prometheus/prometheus.yml` (15s scrape interval)
-![Grafana dashboard screenshot](docs/img/grafana-screenshot.png)
+## Alerting Rules
-## Key Metrics to Track
+8 rules in `monitoring/prometheus/alerts.yml`:
-### Chat/Application
+| Alert                | Severity | Expression                             |
 | -------------------- | -------- | -------------------------------------- |
 | ChatDown             | critical | `up{job="chat"} == 0` for 2m           |
 | PlayoutDown          | critical | `up{job="playout"} == 0` for 2m        |
 | RedisDown            | warning  | `linearvoid_chat_redis_up == 0` for 2m |
 | PlayoutSlateActive   | warning  | `linearvoid_playout_mode == 1` for 1m  |
 | HighChatLatency      | warning  | message rate <0.01/s for 5m            |
 | PrometheusWALFailing | warning  | WAL write failures >0 for 5m           |
 | HighCPU              | warning  | CPU >80% for 10m                       |
 | LowDiskSpace         | warning  | Disk <10% free for 5m                  |
- linearvoid_chat_connected_clients
+## Access
 - linearvoid_chat_stored_messages_total
 - linearvoid_chat_moderation_actions_total
 - linearvoid_chat_redis_enabled
 - linearvoid_chat_redis_up
 - linearvoid_chat_active_admin_sessions
-### Prometheus Runtime
+- Prometheus: `http://<host>:9090` (targets, graph, alerts)
 - Grafana: `http://<host>:3001` (default: admin/admin, change via `GRAFANA_PASSWORD`)
- up
+## Dashboards
 - process_cpu_seconds_total
 - process_resident_memory_bytes
 - prometheus_tsdb_head_series
 - prometheus_tsdb_wal_writes_failed_total
 - prometheus_tsdb_compactions_failed_total
-## Suggested Alerts
+- **Linear Void app**: `monitoring/grafana/dashboard.json` — clients, messages, moderation, Redis, sessions, scrape health
 - **Prometheus ops**: `monitoring/grafana/dashboards/dashboard.json` — TSDB, CPU/memory, WAL, compactions, query latency
-Create a Prometheus rules file (for example monitoring/prometheus/alerts.yml) and wire it into Prometheus.
+## Key Metrics
-Recommended baseline alerts:
+- `linearvoid_chat_connected_clients` — active WebSocket connections
-
+- `linearvoid_chat_messages_total` — total stored messages
- Chat target down for 2m
+- `linearvoid_chat_dms_total` — private messages sent
- Prometheus target down for 2m
+- `linearvoid_chat_redis_up` — Redis reachable (0/1)
- Redis indicator down for 2m
+- `linearvoid_playout_restarts_total` — FFmpeg restart counter
- TSDB WAL/compaction failure rate above zero
+- `linearvoid_playout_mode` — 0=live, 1=slate, 2=degraded
- Elevated handler/query latency p99
+- `linearvoid_chat_admin_sessions_active` — active admin logins
 ## Validate Monitoring End-to-End
 1. Verify /metrics through nginx:
 - <https://linearvoid.tv/metrics>
 1. Confirm targets in Prometheus:
 - Status -> Targets page in Prometheus UI
 1. Confirm dashboard data in Grafana:
 - Import or open provisioned dashboards
 ## Troubleshooting
-### No chat metrics in Prometheus
+| Symptom                 | Check                                                                |
-
+| ----------------------- | -------------------------------------------------------------------- |
- Check chat container health and logs.
+| No chat metrics         | `curl http://chat:3000/metrics` from Prometheus container            |
- Confirm /metrics is reachable from Prometheus container.
+| No exporter metrics     | Verify exporter containers running: `docker ps \| grep exporter`     |
- Verify scrape target is chat:3000 in prometheus.yml.
+| Grafana no data         | Datasource URL must be `http://prometheus:9090`                      |
 | Prometheus targets down | Check container network: all must be on `linearvoid-net`             |
 | Alert not firing        | Check Prometheus → Alerts tab, verify `rule_files` in prometheus.yml |
 ### Grafana shows no data
@@ -14,7 +14,7 @@
 The architecture (detailed in [`docs/architecture/README.md`](docs/architecture/README.md)) uses:
- Hetzner Ashburn VPS for playout (FFmpeg + MediaMTX/SRS) and chat *(planned production; currently test lab at 192.168.88.40)*.
+- Hetzner Ashburn VPS for playout (FFmpeg + MediaMTX/SRS) and chat _(planned production; currently test lab at 192.168.88.40)_.
 - Cloudflare R2 + CDN with aggressive immutable caching of HLS `.ts` segments ($0 egress).
 - Lightweight static frontend and WebSocket chat backend.
 - Gitea self-hosted CI/CD (Actions) and Container Registry.
@@ -1,127 +1,93 @@
 # Security Guide
-This document summarizes security controls and best practices for Linear Void.
+Layered security model for Linear Void.
-## Security Model Overview
+## Threat Model
-Security is layered across:
+```
 ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
 │  Client  │───▶│Cloudflare│───▶│  nginx   │───▶│  Backend │
 │ (public) │    │ WAF+CDN  │    │ proxy    │    │ chat     │
 └──────────┘    └──────────┘    └────┬─────┘    └────┬─────┘
                                     │                │
                              ┌──────┴──────┐  ┌─────┴──────┐
                              │ /health     │  │ /api/admin │
                              │ /metrics    │  │ /ws (chat)  │
                              │ /hls/*      │  │ /api/chat/* │
                              │ (public)    │  │ (auth req)  │
                              └─────────────┘  └────────────┘
 ```
- Cloudflare edge
+## Authentication & Sessions
 - nginx reverse proxy
 - chat/admin backend
 - container and host operations
-## Authentication and Sessions
+- Login: `POST /api/admin/login` → cookie `linearvoid_admin_session`
 - Cookie flags: `HttpOnly`, `SameSite=Strict`, `Secure` (over HTTPS)
 - Server-side expiry via `ADMIN_SESSION_TTL_HOURS`
 - Rate limit: 5 attempts/IP/5min via Redis
-Admin access is session-cookie based.
+## Transport Security (TLS/HSTS)
- Login endpoint: /api/admin/login
+- Cloudflare TLS mode: **Full (strict)** — valid origin cert required
- Session endpoint: /api/admin/session
+- nginx listens on 80 internally; Cloudflare handles TLS termination
- Logout endpoint: /api/admin/logout
+- HSTS header set by Cloudflare: `max-age=31536000; includeSubDomains; preload`
- Cookie: linearvoid_admin_session
+- WSS upgrade path: `/ws` through nginx → chat:3000
 - Cookie flags: HttpOnly, SameSite=Strict, Secure over HTTPS
 - Session expiry enforced server-side
-Best practices:
+## Network Isolation
- Set strong ADMIN_USERNAME and ADMIN_PASSWORD values.
+- All backend services on internal Docker network (`linearvoid-net`)
- Rotate admin credentials periodically.
+- Only nginx port exposed to host (443/80)
- Keep ADMIN_SESSION_TTL_HOURS aligned with operational policy.
+- Database, Redis, exporters inaccessible from outside
-
+- `docker-compose.prod.yml` keeps ports internal via `expose:` not `ports:`
 ## Transport Security
 - Enforce HTTPS for app/admin endpoints.
 - Use WSS for chat WebSocket.
 - Keep Cloudflare TLS mode strict and valid at origin.
 ## Reverse Proxy Controls
-nginx controls include:
+- `auth_request /api/admin/auth` gate on all `/admin` and `/api/admin` paths
 - Explicit `client_max_body_size 64m` for resumable upload chunks
 - WebSocket upgrade headers: `Upgrade`, `Connection` preserved
 - CORS headers on HLS: `Access-Control-Allow-Origin *`
 - XSS protection: CSP headers served by Cloudflare
- auth_request guard for /admin and /api/admin paths
+## Rate Limiting
 - dedicated unauthorized behavior for admin API responses
 - explicit websocket upgrade headers for /ws
 - upload body-size limits for resumable media chunk uploads
-Best practices:
+| Resource          | Limit | Window |
 | ----------------- | ----- | ------ |
 | WebSocket connect | 5/IP  | 60s    |
 | Chat messages     | 10/IP | 10s    |
 | Admin login       | 5/IP  | 5min   |
- Keep proxy body-size and timeout settings aligned across all proxy layers.
+Implemented via Redis `INCR` + `EXPIRE`.
 - Avoid exposing backend service ports publicly unless required.
-## Input Validation and Upload Safety
+## Secrets Management
-Server-side protections include:
+- `.env` — local dev only, **gitignored**
 - Gitea Actions secrets (`CI_*` prefix) for CI/CD
 - Never commit: passwords, tokens, API keys, R2 credentials
 - Production `.env` provisioned manually on host, never checked in
 - Rotate admin password quarterly
- extension allow-list for media uploads
+## Data Protection
 - MIME checks
 - file-size ceiling (2GB)
 - resumable chunk bounds and chunk-index validation
 - media path safety checks on delete
-Best practices:
+- PostgreSQL: chat history, moderation logs, media metadata
-
+- Redis: append-only persistence for presence, rate limits, sessions
- Keep upload chunk size bounded.
+- HLS segments: ephemeral, regenerated by playout
- Reject unsupported content types consistently.
+- Backups: `scripts/backup-db.sh` (daily), `scripts/backup-weekly.sh`, `scripts/verify-backups.sh`
 - Monitor 4xx/5xx rates on upload routes.
 ## Rate Limiting and Abuse Controls
 - Chat rate limiting uses Redis INCR + EXPIRE windows.
 - Current baseline: more than 10 messages in 10 seconds is throttled.
 - Moderation actions are logged for auditability.
 Best practices:
 - Tune thresholds by observed abuse patterns.
 - Keep moderation logs reviewed and retained per policy.
 ## Infrastructure and Secrets
 - Store secrets in .env (not in committed source).
 - Use Gitea Actions secrets for CI/CD credentials and admin credentials.
 - Restrict Docker host control plane exposure.
 - Limit who can read deployment environment files.
 ## Cloudflare and Edge Security
 Recommended baseline:
 - WAF enabled
 - Rate limits on sensitive paths
 - Bot protection where appropriate
 - Cache rules scoped carefully to avoid caching private/control-plane content
 ## Data Protection and Retention
 - Persistent data: postgres, redis append-only, chat state, playlist/media metadata
 - Backup and restore drills should be periodic
 - Restore procedures should be tested in isolated environments
 ## Logging and Monitoring for Security
 Track and alert on:
 - repeated failed admin login attempts
 - spikes in 401 and 403 responses
 - sudden upload errors (413/400/500 patterns)
 - unusual websocket disconnect or message-rate anomalies
 ## Hardening Checklist
- [ ] HTTPS/WSS enforced end-to-end
+- [ ] HTTPS/WSS enforced end-to-end (Cloudflare Full strict)
 - [ ] Strong admin credentials configured
- [ ] Secrets only in env/secrets stores
+- [ ] Secrets only in env/secrets stores (never committed)
 - [ ] Proxy auth and websocket headers validated
- [ ] Upload limits and validation active
+- [ ] Upload limits and MIME validation active
- [ ] Rate limiting active and tested
+- [ ] Rate limiting active (WS, messages, login)
 - [ ] Cloudflare WAF/rate-limit rules active
 - [ ] Backup/restore drills completed
- [ ] Security-related monitoring alerts routed
+- [ ] Security alerts routed (Prometheus alertmanager)
-## Incident Response Baseline
+## Incident Response
-1. Detect issue (alerts/logs/user reports).
+1. Detect — alerts (ChatDown, RedisDown, HighChatLatency) or user reports
-2. Triage scope (edge, proxy, app, data plane).
+2. Triage — edge (Cloudflare), proxy (nginx), app (chat/playout), data (Redis/Postgres)
-3. Contain impact (rate limit, credential rotate, route disable).
+3. Contain — rate limit, credential rotate, service restart, slate mode activation
-4. Recover services and verify health.
+4. Recover — verify health endpoints, check Prometheus targets, validate HLS output
-5. Record timeline and corrective actions.
+5. Record — timeline + corrective actions in `docs/incident-runbook.md`
@@ -32,17 +32,26 @@ var (
 	upgrader    = websocket.Upgrader{
 		CheckOrigin: func(r *http.Request) bool { return true },
 	}
-	clients      = make(map[*websocket.Conn]bool)
+	clients        = make(map[*websocket.Conn]bool)
-	clientMeta   = make(map[*websocket.Conn]clientInfo)
+	clientMeta     = make(map[*websocket.Conn]clientInfo)
-	mutex        = sync.Mutex{}
+	mutex          = sync.Mutex{}
 	authSessions   = map[string]time.Time{}
 	authSessionsMu = sync.Mutex{}
 	// Rate limit config (env-overridable)
 	rateLimitMax    = 10
 	rateLimitWindow = 10 * time.Second
 	// Instance ID for multi-instance pub/sub deduplication
 	instanceID = fmt.Sprintf("chat-%s", generateShortID())
 )
 func generateShortID() string {
 	b := make([]byte, 6)
 	rand.Read(b)
 	return hex.EncodeToString(b)
 }
 type clientInfo struct {
 	Username string
 	Room     string
@@ -101,17 +110,17 @@ type Presence struct {
 }
 type AdminOverview struct {
-	Viewers           int    `json:"viewers"`
+	Viewers                   int    `json:"viewers"`
-	StoredMessages    int64  `json:"storedMessages"`
+	StoredMessages            int64  `json:"storedMessages"`
-	ModerationActions int64  `json:"moderationActions"`
+	ModerationActions         int64  `json:"moderationActions"`
-	MediaCount        int64  `json:"mediaCount"`
+	MediaCount                int64  `json:"mediaCount"`
 	MediaTotalDurationSeconds int64  `json:"mediaTotalDurationSeconds"`
-	MediaTotalRuntime string `json:"mediaTotalRuntime"`
+	MediaTotalRuntime         string `json:"mediaTotalRuntime"`
-	RedisEnabled      bool   `json:"redisEnabled"`
+	RedisEnabled              bool   `json:"redisEnabled"`
-	RedisClients      int    `json:"redisClients"`
+	RedisClients              int    `json:"redisClients"`
-	RedisMemoryUsed   string `json:"redisMemoryUsed"`
+	RedisMemoryUsed           string `json:"redisMemoryUsed"`
-	DatabaseDriver    string `json:"databaseDriver"`
+	DatabaseDriver            string `json:"databaseDriver"`
-	GeneratedAt       string `json:"generatedAt"`
+	GeneratedAt               string `json:"generatedAt"`
 }
 type PlaylistPayload struct {
@@ -136,15 +145,15 @@ type RoomAnalytics struct {
 }
 type AdminAnalytics struct {
-	TotalMessages       int64           `json:"totalMessages"`
+	TotalMessages             int64           `json:"totalMessages"`
-	MessagesLast24Hours int64           `json:"messagesLast24Hours"`
+	MessagesLast24Hours       int64           `json:"messagesLast24Hours"`
-	DistinctRooms       int64           `json:"distinctRooms"`
+	DistinctRooms             int64           `json:"distinctRooms"`
-	ModerationActions   int64           `json:"moderationActions"`
+	ModerationActions         int64           `json:"moderationActions"`
-	MediaCount          int64           `json:"mediaCount"`
+	MediaCount                int64           `json:"mediaCount"`
-	MediaTotalDurationSeconds int64     `json:"mediaTotalDurationSeconds"`
+	MediaTotalDurationSeconds int64           `json:"mediaTotalDurationSeconds"`
-	MediaTotalRuntime   string          `json:"mediaTotalRuntime"`
+	MediaTotalRuntime         string          `json:"mediaTotalRuntime"`
-	TopRooms            []RoomAnalytics `json:"topRooms"`
+	TopRooms                  []RoomAnalytics `json:"topRooms"`
-	GeneratedAt         string          `json:"generatedAt"`
+	GeneratedAt               string          `json:"generatedAt"`
 }
 type ServiceHealth struct {
@@ -171,9 +180,9 @@ type ActiveUser struct {
 }
 type AdminUsersResponse struct {
-	Users       []ActiveUser `json:"users"`
+	Users        []ActiveUser `json:"users"`
-	RedisEnabled bool       `json:"redisEnabled"`
+	RedisEnabled bool         `json:"redisEnabled"`
-	GeneratedAt string      `json:"generatedAt"`
+	GeneratedAt  string       `json:"generatedAt"`
 }
 type ChatLogEntry struct {
@@ -205,12 +214,12 @@ type ScheduleEntry struct {
 }
 type AdminScheduleResponse struct {
-	Entries               []ScheduleEntry `json:"entries"`
+	Entries              []ScheduleEntry `json:"entries"`
-	TotalEntries          int             `json:"totalEntries"`
+	TotalEntries         int             `json:"totalEntries"`
-	MatchedEntries        int             `json:"matchedEntries"`
+	MatchedEntries       int             `json:"matchedEntries"`
-	TotalDurationSeconds  int             `json:"totalDurationSeconds"`
+	TotalDurationSeconds int             `json:"totalDurationSeconds"`
-	TotalRuntime          string          `json:"totalRuntime"`
+	TotalRuntime         string          `json:"totalRuntime"`
-	GeneratedAt           string          `json:"generatedAt"`
+	GeneratedAt          string          `json:"generatedAt"`
 }
 type MediaFileEntry struct {
@@ -234,12 +243,12 @@ type AdminMediaResponse struct {
 }
 type AdminMediaUploadInitRequest struct {
-	UploadID   string `json:"uploadId"`
+	UploadID    string `json:"uploadId"`
-	FileName   string `json:"fileName"`
+	FileName    string `json:"fileName"`
-	FileSize   int64  `json:"fileSize"`
+	FileSize    int64  `json:"fileSize"`
-	MimeType   string `json:"mimeType"`
+	MimeType    string `json:"mimeType"`
-	ChunkSize  int64  `json:"chunkSize"`
+	ChunkSize   int64  `json:"chunkSize"`
-	TotalChunks int   `json:"totalChunks"`
+	TotalChunks int    `json:"totalChunks"`
 }
 type AdminMediaUploadSession struct {
@@ -254,13 +263,13 @@ type AdminMediaUploadSession struct {
 }
 type AdminMediaUploadStatusResponse struct {
-	UploadID      string  `json:"uploadId"`
+	UploadID       string  `json:"uploadId"`
-	FileName      string  `json:"fileName"`
+	FileName       string  `json:"fileName"`
-	FileSize      int64   `json:"fileSize"`
+	FileSize       int64   `json:"fileSize"`
-	ChunkSize     int64   `json:"chunkSize"`
+	ChunkSize      int64   `json:"chunkSize"`
-	TotalChunks   int     `json:"totalChunks"`
+	TotalChunks    int     `json:"totalChunks"`
-	UploadedChunks int    `json:"uploadedChunks"`
+	UploadedChunks int     `json:"uploadedChunks"`
-	Progress      float64 `json:"progress"`
+	Progress       float64 `json:"progress"`
 }
 type AdminLoginPayload struct {
@@ -278,9 +287,9 @@ var databaseDriverName = "sqlite"
 const adminSessionCookieName = "linearvoid_admin_session"
 const (
-	maxAdminUploadBytes      int64 = 2 * 1024 * 1024 * 1024
+	maxAdminUploadBytes   int64 = 2 * 1024 * 1024 * 1024
-	defaultAdminChunkSize    int64 = 8 * 1024 * 1024
+	defaultAdminChunkSize int64 = 8 * 1024 * 1024
-	maxAdminChunkSize        int64 = 32 * 1024 * 1024
+	maxAdminChunkSize     int64 = 32 * 1024 * 1024
 )
 // initRedis initializes Redis for presence, rate limiting, and pub/sub (ADR-007)
@@ -380,11 +389,11 @@ func parseRedisInfoValue(info, key string) string {
 func getAdminOverview() AdminOverview {
 	overview := AdminOverview{
-		RedisEnabled:    redisClient != nil,
+		RedisEnabled:      redisClient != nil,
-		RedisMemoryUsed: "unavailable",
+		RedisMemoryUsed:   "unavailable",
-		DatabaseDriver:  databaseDriverName,
+		DatabaseDriver:    databaseDriverName,
 		MediaTotalRuntime: "00:00:00",
-		GeneratedAt:     time.Now().UTC().Format(time.RFC3339),
+		GeneratedAt:       time.Now().UTC().Format(time.RFC3339),
 	}
 	mutex.Lock()
@@ -422,6 +431,23 @@ func handleAdminLogin(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 	// Rate limit: max 5 attempts per IP per 5 minutes
 	clientIP := r.Header.Get("X-Forwarded-For")
 	if clientIP == "" {
 		clientIP = r.RemoteAddr
 	}
 	if redisClient != nil {
 		loginKey := fmt.Sprintf("login_attempts:%s", clientIP)
 		count, _ := redisClient.Incr(context.Background(), loginKey).Result()
 		if count == 1 {
 			redisClient.Expire(context.Background(), loginKey, 5*time.Minute)
 		}
 		if count > 5 {
 			http.Error(w, "too many login attempts, try again later", http.StatusTooManyRequests)
 			return
 		}
 	}
 	var payload AdminLoginPayload
 	if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
 		http.Error(w, fmt.Sprintf("invalid request body: %v", err), http.StatusBadRequest)
@@ -907,9 +933,9 @@ func handleAdminModerationAction(w http.ResponseWriter, r *http.Request) {
 func getAdminAnalytics() AdminAnalytics {
 	analytics := AdminAnalytics{
-		TopRooms:    []RoomAnalytics{},
+		TopRooms:          []RoomAnalytics{},
 		MediaTotalRuntime: "00:00:00",
-		GeneratedAt: time.Now().UTC().Format(time.RFC3339),
+		GeneratedAt:       time.Now().UTC().Format(time.RFC3339),
 	}
 	if db == nil {
@@ -1114,7 +1140,7 @@ func handleAdminUsers(w http.ResponseWriter, r *http.Request) {
 	case http.MethodPost:
 		var payload struct {
-			Action   string `json:"action"`   // mute, unmute, ban, unban
+			Action   string `json:"action"` // mute, unmute, ban, unban
 			Username string `json:"username"`
 			Duration int    `json:"duration"` // minutes, for mute
 		}
@@ -1309,13 +1335,13 @@ func buildUploadStatus(session AdminMediaUploadSession) AdminMediaUploadStatusRe
 	}
 	return AdminMediaUploadStatusResponse{
-		UploadID:      session.UploadID,
+		UploadID:       session.UploadID,
-		FileName:      session.FileName,
+		FileName:       session.FileName,
-		FileSize:      session.FileSize,
+		FileSize:       session.FileSize,
-		ChunkSize:     session.ChunkSize,
+		ChunkSize:      session.ChunkSize,
-		TotalChunks:   session.TotalChunks,
+		TotalChunks:    session.TotalChunks,
 		UploadedChunks: uploadedChunks,
-		Progress:      progress,
+		Progress:       progress,
 	}
 }
@@ -2144,7 +2170,7 @@ func handleMetrics(w http.ResponseWriter, r *http.Request) {
 	fmt.Fprintln(w, "# HELP linearvoid_chat_build_info Static build info for chat service.")
 	fmt.Fprintln(w, "# TYPE linearvoid_chat_build_info gauge")
-	fmt.Fprintln(w, "linearvoid_chat_build_info{service=\"chat\",database_driver=\""+databaseDriverName+"\"} 1")
+	fmt.Fprintln(w, "linearvoid_chat_build_info{service=\"chat\",database_driver=\""+databaseDriverName+"\",instance_id=\""+instanceID+"\"} 1")
 }
 // broadcast sends a message to all connected clients in a room and saves to DB
@@ -2155,16 +2181,46 @@ func broadcast(msg Message) {
 	// Persist to configured SQL backend.
 	db.Create(&msg)
 	// Local fan-out
 	for client := range clients {
 		if err := client.WriteJSON(msg); err != nil {
 			client.Close()
 			delete(clients, client)
 			delete(clientMeta, client)
 		}
 	}
 	// Redis pub/sub for multi-instance fan-out
 	if redisClient != nil && msg.Room != "" {
 		payload, _ := json.Marshal(map[string]any{
 			"instance": instanceID,
 			"msg":      msg,
 		})
 		redisClient.Publish(context.Background(), fmt.Sprintf("room:%s:messages", msg.Room), payload)
 	}
 }
 // handleConnection handles WebSocket connections for chat (public rooms, private DMs, moderation)
 func handleConnection(w http.ResponseWriter, r *http.Request) {
 	// IP-based connection rate limiting: max 5 concurrent per IP
 	clientIP := r.Header.Get("X-Forwarded-For")
 	if clientIP == "" {
 		clientIP = r.RemoteAddr
 	}
 	if redisClient != nil {
 		ipKey := fmt.Sprintf("ws_conn:%s", clientIP)
 		count, _ := redisClient.Incr(context.Background(), ipKey).Result()
 		if count == 1 {
 			redisClient.Expire(context.Background(), ipKey, 60*time.Second)
 		}
 		if count > 5 {
 			http.Error(w, "too many connections from this IP", http.StatusTooManyRequests)
 			return
 		}
 		// Decrement on disconnect
 		defer redisClient.Decr(context.Background(), ipKey)
 	}
 	conn, err := upgrader.Upgrade(w, r, nil)
 	if err != nil {
 		log.Println("Upgrade error:", err)
@@ -2297,6 +2353,47 @@ func pushChatHistory(conn *websocket.Conn, room string, limit int) {
 // dmsSent tracks total private messages for Prometheus metrics (set from handleMetrics).
 var dmsSent int64
 // subscribeRoomMessages listens for Redis pub/sub messages and fans out locally.
 func subscribeRoomMessages() {
 	if redisClient == nil {
 		return
 	}
 	pubsub := redisClient.PSubscribe(context.Background(), "room:*:messages")
 	defer pubsub.Close()
 	log.Printf("Redis pub/sub subscriber started (instance %s)", instanceID)
 	for {
 		msg, err := pubsub.ReceiveMessage(context.Background())
 		if err != nil {
 			log.Printf("Redis pub/sub error: %v — retrying in 5s", err)
 			time.Sleep(5 * time.Second)
 			pubsub = redisClient.PSubscribe(context.Background(), "room:*:messages")
 			continue
 		}
 		var envelope struct {
 			Instance string  `json:"instance"`
 			Msg      Message `json:"msg"`
 		}
 		if err := json.Unmarshal([]byte(msg.Payload), &envelope); err != nil {
 			continue
 		}
 		// Skip messages originating from this instance
 		if envelope.Instance == instanceID {
 			continue
 		}
 		// Fan out to local clients
 		mutex.Lock()
 		for client := range clients {
 			client.WriteJSON(envelope.Msg)
 		}
 		mutex.Unlock()
 	}
 }
 // handlePrivateDM routes private messages to the specific user pair
 func handlePrivateDM(msg Message) {
 	msg.Type = "private"
@@ -2604,6 +2701,11 @@ func main() {
 	initRedis()
 	initDB()
 	// Subscribe to Redis pub/sub for multi-instance message fan-out
 	go subscribeRoomMessages()
 	log.Printf("Instance ID: %s", instanceID)
 	http.HandleFunc("/ws", handleConnection)
 	http.HandleFunc("/api/admin/login", handleAdminLogin)
 	http.HandleFunc("/api/admin/logout", handleAdminLogout)
@@ -104,12 +104,63 @@ services:
      timeout: 5s
      retries: 5
  nginx-exporter:
    image: nginx/nginx-prometheus-exporter:latest
    container_name: linearvoid-nginx-exporter
    restart: unless-stopped
    command:
      - "-nginx.scrape-uri=http://nginx/nginx-status"
    networks:
      - linearvoid-net
    depends_on:
      - nginx
  redis-exporter:
    image: oliver006/redis_exporter:latest
    container_name: linearvoid-redis-exporter
    restart: unless-stopped
    command:
      - "-redis.addr=redis://redis:6379"
    networks:
      - linearvoid-net
    depends_on:
      - redis
  postgres-exporter:
    image: prometheuscommunity/postgres-exporter:latest
    container_name: linearvoid-postgres-exporter
    restart: unless-stopped
    environment:
      DATA_SOURCE_NAME: "postgresql://linearvoid:${POSTGRES_PASSWORD:-linearvoid}@db:5432/linearvoid?sslmode=disable"
    networks:
      - linearvoid-net
    depends_on:
      - db
  node-exporter:
    image: prom/node-exporter:latest
    container_name: linearvoid-node-exporter
    restart: unless-stopped
    pid: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - "--path.procfs=/host/proc"
      - "--path.sysfs=/host/sys"
      - "--path.rootfs=/rootfs"
      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
    networks:
      - linearvoid-net
  prometheus:
    image: prom/prometheus:latest
    container_name: linearvoid-prometheus
    restart: unless-stopped
    volumes:
-      - prometheus-config:/etc/prometheus
+      - ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./monitoring/prometheus/alerts.yml:/etc/prometheus/alerts.yml
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"
@@ -149,5 +200,4 @@ volumes:
  redis-data:
  postgres-data:
  prometheus-data:
  prometheus-config:
  grafana-data:
@@ -97,12 +97,63 @@ services:
    networks:
      - linearvoid-net
  nginx-exporter:
    image: nginx/nginx-prometheus-exporter:latest
    container_name: linearvoid-nginx-exporter
    restart: unless-stopped
    command:
      - "-nginx.scrape-uri=http://nginx/nginx-status"
    networks:
      - linearvoid-net
    depends_on:
      - nginx
  redis-exporter:
    image: oliver006/redis_exporter:latest
    container_name: linearvoid-redis-exporter
    restart: unless-stopped
    command:
      - "-redis.addr=redis://redis:6379"
    networks:
      - linearvoid-net
    depends_on:
      - redis
  postgres-exporter:
    image: prometheuscommunity/postgres-exporter:latest
    container_name: linearvoid-postgres-exporter
    restart: unless-stopped
    environment:
      DATA_SOURCE_NAME: "postgresql://linearvoid:${POSTGRES_PASSWORD:-linearvoid}@db:5432/linearvoid?sslmode=disable"
    networks:
      - linearvoid-net
    depends_on:
      - db
  node-exporter:
    image: prom/node-exporter:latest
    container_name: linearvoid-node-exporter
    restart: unless-stopped
    pid: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - "--path.procfs=/host/proc"
      - "--path.sysfs=/host/sys"
      - "--path.rootfs=/rootfs"
      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
    networks:
      - linearvoid-net
  prometheus:
    image: prom/prometheus:latest
    container_name: linearvoid-prometheus
    restart: unless-stopped
    volumes:
      - ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./monitoring/prometheus/alerts.yml:/etc/prometheus/alerts.yml
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"
@@ -259,11 +259,114 @@ Detailed black-box descriptions, level-2/3 views, and code locations are in `adr
 ### Level 2
-Deferred to implementation (see ADR for component responsibilities, interfaces, quality attributes, and risks). Playout enforces normalization; Delivery layer focuses on cache rules and immutable naming.
+Detailed component diagram with code locations and persistence.
 ```mermaid
 graph TB
    subgraph "Cloudflare Edge"
        CF[CDN + WAF<br/>Cache Rules]
        R2[R2 Bucket<br/>media + HLS]
        PAGES[Pages<br/>SPA hosting]
    end
    subgraph "Docker Host (Hetzner CX22)"
        NGINX[nginx<br/>:80/:443<br/>proxy + auth]
        CHAT[chat service<br/>:3000<br/>Go + gorilla/websocket]
        PLAYOUT[playout service<br/>:8081<br/>Go + FFmpeg]
        REDIS[Redis 7<br/>:6379<br/>presence/rate/pubsub]
        DB[PostgreSQL 15<br/>:5432<br/>chat history/moderation]
    end
    subgraph "External"
        VIEWER[Viewer Browser<br/>hls.js + WebSocket]
        ADMIN[Admin Browser<br/>SPA + uploads]
    end
    VIEWER -->|HTTPS| CF
    VIEWER -->|WSS| NGINX
    ADMIN -->|HTTPS| CF
    CF --> NGINX
    NGINX --> CHAT
    NGINX --> PLAYOUT
    NGINX -->|/hls/*| R2
    CHAT --> REDIS
    CHAT --> DB
    PLAYOUT --> R2
    PLAYOUT -->|reads| R2
 ```
 #### Playout Service (`backend/playout/src/playlist.go`)
 - **Role**: Continuous HLS encoding loop using FFmpeg concat demuxer
 - **Input**: `data/playlist/playlist.txt` — ordered list of MP4 file paths
 - **Output**: HLS segments (`.ts`) + live manifest (`.m3u8`) written to `data/hls/`, mirrored to R2
 - **Resilience**: Restart counter (>5/60s → activate slate), Prometheus metrics on port 8081
 - **Validation**: ffprobe gate on all ingested files (codec, fps, resolution, keyframe interval)
 #### Chat Service (`backend/chat/src/main.go`)
 - **Role**: WebSocket chat with rooms, DMs, moderation, admin API
 - **Persistence**: PostgreSQL 15 via GORM (messages, moderation logs, media metadata). SQLite for dev fallback
 - **Redis**: Presence tracking, rate limiting, pub/sub for multi-instance fan-out
 - **Auth**: Session-cookie admin auth with Redis-backed expiry
 - **Metrics**: Connected clients, stored messages, DMs, moderation actions on `/metrics`
 #### Frontend SPA (`frontend/`)
 - **Viewer**: `src/main.ts` — hls.js player, WebSocket chat, presence indicators, schedule overlay
 - **Admin**: `src/admin.ts` — 9 dynamic tabs (dashboard, upload, playlist, schedule, moderation, chat-log, users, analytics, system)
 - **Build**: Vite → static output in `frontend/dist/`
 #### Reverse Proxy (`infrastructure/nginx/nginx.conf`)
 - auth_request gate on `/admin` and `/api/admin`
 - WebSocket upgrade for `/ws`
 - HLS CORS headers + OPTIONS preflight
 - `/health` JSON endpoint, `/nginx-status` stub_status
 ### Level 3
-Deferred to implementation (see ADR for component responsibilities, interfaces, quality attributes, and risks). Playout enforces normalization; Delivery layer focuses on cache rules and immutable naming.
+Module-level breakdown per service.
 #### Chat Service Modules
 | Module                 | File/Lines                                                     | Responsibility                                                                           |
 | ---------------------- | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
 | **Connection Manager** | `main.go` (hub, clients map)                                   | Track active WebSocket connections per room, fan-out messages                            |
 | **Message Parser**     | `main.go` (readPump, handleMessage)                            | Parse JSON envelope `{type, room?, ...}`, validate fields, route to handler              |
 | **Room Registry**      | `main.go` (rooms map, room broadcasts)                         | Map room name → client set, handle join/leave/close                                      |
 | **DM Router**          | `main.go` (handleDirectMessage)                                | Resolve target user in clients map, deliver private message                              |
 | **Persistence Layer**  | `main.go` (GORM models: Message, DirectMessage, ModerationLog) | Store/retrieve chat messages, DMs, moderation events                                     |
 | **Redis Subsystem**    | `main.go` (redisClient, rate limiting, pub/sub, presence)      | INCR+EXPIRE rate limit, SETEX presence, PSUBSCRIBE room fan-out                          |
 | **Auth Handler**       | `main.go` (handleAdminLogin, adminAuthMiddleware)              | Session cookie creation/validation, login rate limiting                                  |
 | **Admin API Router**   | `main.go` (handleAdmin\*)                                      | Upload, playlist CRUD, schedule overrides, moderation actions, analytics, system summary |
 | **Metrics Exporter**   | `main.go` (prometheus.MustRegister)                            | Expose connected_clients, messages_total, dms_total, moderation_actions, redis_up        |
 #### Playout Service Modules
 | Module               | File/Lines                                     | Responsibility                                                     |
 | -------------------- | ---------------------------------------------- | ------------------------------------------------------------------ |
 | **Playlist Parser**  | `playlist.go` (parsePlaylist)                  | Read `playlist.txt`, skip comments/blank lines, resolve file paths |
 | **FFmpeg Manager**   | `playlist.go` (runPlayout, restart logic)      | Spawn/monitor FFmpeg concat process, restart on crash              |
 | **Slate Controller** | `playlist.go` (activateSlate, deactivateSlate) | Switch output to slate loop on excessive failures                  |
 | **R2 Sync**          | `playlist.go` (syncToR2)                       | Upload HLS segments + manifest to R2 bucket                        |
 | **Health Endpoint**  | `playlist.go` (healthHandler)                  | `/health` → JSON with mode, segment count, uptime                  |
 | **Metrics Exporter** | `playlist.go` (prometheus)                     | Expose restarts_total, mode (live/slate)                           |
 #### Frontend Modules
 | Module                  | File                                            | Responsibility                                                    |
 | ----------------------- | ----------------------------------------------- | ----------------------------------------------------------------- |
 | **HLS Player**          | `src/main.ts`                                   | hls.js initialization, attach to video element, error recovery    |
 | **WebSocket Client**    | `src/main.ts`                                   | Connect/disconnect, parse envelope, dispatch to UI handlers       |
 | **Chat UI**             | `src/main.ts`                                   | Message list rendering, room tabs, DM panel, presence dots        |
 | **Schedule Overlay**    | `src/main.ts`                                   | Poll `/api/schedule`, render current/next program                 |
 | **Admin Router**        | `src/admin.ts`                                  | Tab-based navigation, dynamic page loading                        |
 | **Upload Manager**      | `src/admin.ts`                                  | Multipart upload with chunk progress, ffprobe validation feedback |
 | **Playlist Editor**     | `src/admin.ts`                                  | Drag-reorder, add/remove items, save to `/api/admin/playlist`     |
 | **Moderation Panel**    | `src/admin.ts` + `public/admin/moderation.html` | Inline mute/ban, action log viewer                                |
 | **Analytics Dashboard** | `src/admin.ts` + `public/admin/analytics.html`  | Message counts, top rooms, moderation stats                       |
 | **System Config**       | `src/admin.ts` + `public/admin/system.html`     | Service health, config metadata, restart buttons                  |
 ## Runtime View
@@ -0,0 +1,170 @@
 # Incident Runbook
 Triage order for Linear Void incidents. Work top-to-bottom — each tier depends on the one above.
 ## Triage Order
 ```
 1. Edge Reachability (Cloudflare)
 2. Origin Health (VPS/Docker)
 3. Playout Progression (HLS)
 4. Chat Connectivity (WebSocket)
 5. Data-Plane (Redis/DB)
 ```
 ---
 ## 1. Edge Reachability (Cloudflare)
 **Symptoms:** `linearvoid.tv` not resolving, SSL errors, 5xx from Cloudflare.
 ```bash
 # Check DNS
 dig linearvoid.tv
 dig www.linearvoid.tv
 # Check Cloudflare status
 curl -I https://linearvoid.tv/health
 ```
 **Common causes:**
 - DNS misconfiguration (A record pointing to wrong IP)
 - Cloudflare proxy disabled (grey cloud → origin exposed directly)
 - SSL certificate expired (check Cloudflare dashboard → SSL/TLS)
 - Cloudflare outage → https://www.cloudflarestatus.com
 **Mitigation:**
 - Verify A record: `linearvoid.tv` → origin IP (proxied, orange cloud)
 - Bypass Cloudflare temporarily: `curl -H "Host: linearvoid.tv" http://<origin-ip>/health`
 ---
 ## 2. Origin Health (VPS/Docker)
 **Symptoms:** Cloudflare returns 502/504, containers down.
 ```bash
 # Remote Docker (test lab)
 docker -H tcp://192.168.88.40:2375 ps
 docker -H tcp://192.168.88.40:2375 logs linearvoid-nginx --tail 50
 # Direct health check
 curl http://192.168.88.40/health
 ```
 **Common causes:**
 - Host out of disk space: `df -h`
 - Docker daemon stopped: `systemctl status docker`
 - Container restart loop: `docker ps -a --filter "status=exited"`
 - Resource exhaustion (OOM): `dmesg | grep -i oom`
 **Mitigation:**
 - Restart stack: `docker compose -f docker-compose.prod.yml up -d`
 - Free disk: `docker system prune -f`
 - Check logs per service: `docker logs linearvoid-<service> --tail 100`
 ---
 ## 3. Playout Progression (HLS)
 **Symptoms:** Viewer sees slate or buffering, playlist not advancing.
 ```bash
 # Check playout health
 curl http://192.168.88.40:8081/health
 # Response includes "mode": "live" or "slate"
 # Check HLS output
 curl -I http://192.168.88.40/hls/playlist.m3u8
 ls -la data/hls/
 # Check playout logs
 docker logs linearvoid-playout --tail 100
 ```
 **Common causes:**
 - FFmpeg crash → slate fallback active (health shows `degraded`)
 - Playlist file empty/missing: `cat data/playlist/playlist.txt`
 - Media file missing from playlist: `ls data/media/`
 - Disk full → HLS segments can't write
 **Mitigation:**
 - Verify playlist has valid entries
 - Check media files exist at paths in playlist
 - Restart playout: `docker compose restart playout`
 - Generate demo playlist if empty: see `backend/playout/start-playout.sh`
 ---
 ## 4. Chat Connectivity (WebSocket)
 **Symptoms:** Chat not loading, messages not sending, presence broken.
 ```bash
 # Check chat health
 curl http://192.168.88.40:3000/health
 # Test WebSocket
 wscat -c ws://192.168.88.40/ws?username=test&room=main
 # Check chat logs
 docker logs linearvoid-chat --tail 100
 ```
 **Common causes:**
 - Redis down → presence/rate-limiting degraded (chat still works)
 - Database down → messages not persisting
 - Port not exposed: `docker port linearvoid-chat`
 **Mitigation:**
 - Restart chat: `docker compose restart chat`
 - Check Redis: `docker exec linearvoid-redis redis-cli ping`
 - Check DB: `docker exec linearvoid-db pg_isready -U linearvoid`
 - Chat works without Redis (graceful degradation) — check logs for "Redis connection warning"
 ---
 ## 5. Data-Plane (Redis/DB)
 **Symptoms:** Chat works but no persistence/presence, admin stats empty.
 ```bash
 # Redis
 docker exec linearvoid-redis redis-cli ping
 docker exec linearvoid-redis redis-cli info memory
 # PostgreSQL
 docker exec linearvoid-db pg_isready -U linearvoid
 docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"
 ```
 **Common causes:**
 - Redis OOM (check `maxmemory` setting)
 - PostgreSQL connection limit reached
 - Disk full on volume mount
 **Mitigation:**
 - Redis: `docker compose restart redis` (AOF persist enabled)
 - PostgreSQL: `docker compose restart db`
 - Restore from backup: see `scripts/verify-backups.sh` for procedure
 ---
 ## Escalation
 | Severity | When | Action |
 |---|---|---|
 | Critical | Slate active >30min, complete outage | Trigger DR plan, provision fallback host |
 | Warning | Redis down, high CPU >30min | Investigate during business hours, restart service |
 | Info | Single service blip | Monitor, no action unless recurring |
 ## Post-Incident
 1. Document timeline in incident channel
 2. Identify root cause
 3. Add prevention to TODO.md if actionable
 4. Update this runbook if new failure mode discovered
 ---
 **Recovery targets:** RPO ≤24h, RTO ≤60min. Quarterly DR drill: simulate host loss, restore from backup.
@@ -1,7 +1,7 @@
 {
  "name": "frontend",
  "private": true,
-  "version": "1.0.7",
+  "version": "1.0.9",
  "type": "module",
  "scripts": {
    "dev": "vite",
@@ -0,0 +1,12 @@
 apiVersion: 1
 providers:
  - name: "default"
    orgId: 1
    folder: ""
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards
@@ -0,0 +1,9 @@
 apiVersion: 1
 datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true
@@ -0,0 +1,74 @@
 groups:
  - name: linearvoid
    rules:
      - alert: ChatDown
        expr: up{job="chat"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Chat service is down"
          description: "Chat service has been unreachable for 2 minutes."
      - alert: PlayoutDown
        expr: up{job="playout"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Playout service is down"
          description: "Playout service has been unreachable for 2 minutes."
      - alert: RedisDown
        expr: linearvoid_chat_redis_up == 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Redis is unreachable"
          description: "Chat service cannot reach Redis. Rate limiting and presence tracking degraded."
      - alert: PlayoutSlateActive
        expr: linearvoid_playout_mode == 1
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Playout in slate fallback mode"
          description: "Playout has switched to emergency slate. Check playout logs."
      - alert: HighChatLatency
        expr: rate(linearvoid_chat_messages_total[5m]) < 0.01
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Unusually low chat message rate"
          description: "Message rate below 0.01/s for 5 minutes. Check chat service health."
      - alert: PrometheusWALFailing
        expr: increase(prometheus_tsdb_wal_writes_failed_total[5m]) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus WAL writes failing"
          description: "Prometheus TSDB WAL write failures detected."
      - alert: HighCPU
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage"
          description: "CPU usage above 80% for 10 minutes on {{ $labels.instance }}."
      - alert: LowDiskSpace
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low disk space"
          description: "Less than 10% disk space available on {{ $labels.instance }}."
@@ -1,10 +1,28 @@
 global:
  scrape_interval: 15s
 rule_files:
  - "/etc/prometheus/alerts.yml"
 scrape_configs:
  - job_name: "chat"
    static_configs:
      - targets: ["chat:3000"]
  - job_name: "playout"
    static_configs:
      - targets: ["playout:8081"]
  - job_name: "nginx"
    static_configs:
      - targets: ["nginx-exporter:9113"]
  - job_name: "redis"
    static_configs:
      - targets: ["redis-exporter:9121"]
  - job_name: "postgres"
    static_configs:
      - targets: ["postgres-exporter:9187"]
  - job_name: "node"
    static_configs:
      - targets: ["node-exporter:9100"]
  - job_name: "prometheus"
    static_configs:
      - targets: ["prometheus:9090"]
@@ -0,0 +1,45 @@
 #!/bin/bash
 # Daily PostgreSQL backup to R2 (14-day retention)
 # Schedule: cron daily. Requires: pg_dump, aws CLI or rclone for R2.
 set -e
 BACKUP_DIR="${BACKUP_DIR:-/tmp/linearvoid-backups}"
 RETENTION_DAYS="${RETENTION_DAYS:-14}"
 R2_BUCKET="${R2_BUCKET:-linearvoid-backups}"
 DB_URL="${DATABASE_URL:-postgres://linearvoid:linearvoid@db:5432/linearvoid?sslmode=disable}"
 TIMESTAMP=$(date +%Y%m%d)
 BACKUP_FILE="$BACKUP_DIR/daily/db_${TIMESTAMP}.dump"
 mkdir -p "$(dirname "$BACKUP_FILE")"
 echo "[$(date -Iseconds)] Starting daily backup..."
 # Extract connection details from DATABASE_URL
 DB_HOST=$(echo "$DB_URL" | sed -n 's/.*@\([^:]*\).*/\1/p')
 DB_PORT=$(echo "$DB_URL" | sed -n 's/.*:\([0-9]*\)\/.*/\1/p')
 DB_USER=$(echo "$DB_URL" | sed -n 's/.*:\/\/\([^:]*\).*/\1/p')
 DB_NAME=$(echo "$DB_URL" | sed -n 's/.*\/\([^?]*\).*/\1/p')
 DB_PASS=$(echo "$DB_URL" | sed -n 's/.*:\/\/[^:]*:\([^@]*\).*/\1/p')
 export PGPASSWORD="${DB_PASS}"
 # Dump
 pg_dump -h "${DB_HOST:-db}" -p "${DB_PORT:-5432}" -U "${DB_USER:-linearvoid}" \
        -Fc -f "$BACKUP_FILE" "${DB_NAME:-linearvoid}"
 echo "  Backup: $BACKUP_FILE ($(du -h "$BACKUP_FILE" | cut -f1))"
 # Upload to R2
 if [ -n "$R2_ENDPOINT" ] && [ -n "$R2_ACCESS_KEY_ID" ]; then
    aws s3 cp "$BACKUP_FILE" "s3://${R2_BUCKET}/daily/db_${TIMESTAMP}.dump" \
        --endpoint-url "$R2_ENDPOINT" --no-verify-ssl 2>/dev/null && \
        echo "  Uploaded to R2: daily/db_${TIMESTAMP}.dump" || \
        echo "  WARNING: R2 upload failed"
 fi
 # Cleanup old backups
 find "$BACKUP_DIR/daily" -name "db_*.dump" -mtime "+${RETENTION_DAYS}" -delete 2>/dev/null || true
 echo "  Cleaned up backups older than ${RETENTION_DAYS} days"
 echo "[$(date -Iseconds)] Daily backup complete."
@@ -0,0 +1,48 @@
 #!/bin/bash
 # Weekly full PostgreSQL backup to R2 (8-week retention)
 # Schedule: cron weekly (runs backup-db.sh with weekly prefix)
 set -e
 BACKUP_DIR="${BACKUP_DIR:-/tmp/linearvoid-backups}"
 RETENTION_WEEKS="${RETENTION_WEEKS:-8}"
 R2_BUCKET="${R2_BUCKET:-linearvoid-backups}"
 DB_URL="${DATABASE_URL:-postgres://linearvoid:linearvoid@db:5432/linearvoid?sslmode=disable}"
 TIMESTAMP=$(date +%Y%m%d)
 BACKUP_FILE="$BACKUP_DIR/weekly/db_${TIMESTAMP}.dump"
 mkdir -p "$(dirname "$BACKUP_FILE")"
 echo "[$(date -Iseconds)] Starting weekly full backup..."
 DB_HOST=$(echo "$DB_URL" | sed -n 's/.*@\([^:]*\).*/\1/p')
 DB_PORT=$(echo "$DB_URL" | sed -n 's/.*:\([0-9]*\)\/.*/\1/p')
 DB_USER=$(echo "$DB_URL" | sed -n 's/.*:\/\/\([^:]*\).*/\1/p')
 DB_NAME=$(echo "$DB_URL" | sed -n 's/.*\/\([^?]*\).*/\1/p')
 DB_PASS=$(echo "$DB_URL" | sed -n 's/.*:\/\/[^:]*:\([^@]*\).*/\1/p')
 export PGPASSWORD="${DB_PASS}"
 pg_dump -h "${DB_HOST:-db}" -p "${DB_PORT:-5432}" -U "${DB_USER:-linearvoid}" \
        -Fc -f "$BACKUP_FILE" "${DB_NAME:-linearvoid}"
 echo "  Backup: $BACKUP_FILE ($(du -h "$BACKUP_FILE" | cut -f1))"
 # Verify integrity
 pg_restore --list "$BACKUP_FILE" > /dev/null 2>&1 && \
    echo "  Integrity check: PASSED" || \
    echo "  WARNING: Integrity check FAILED"
 # Upload
 if [ -n "$R2_ENDPOINT" ] && [ -n "$R2_ACCESS_KEY_ID" ]; then
    aws s3 cp "$BACKUP_FILE" "s3://${R2_BUCKET}/weekly/db_${TIMESTAMP}.dump" \
        --endpoint-url "$R2_ENDPOINT" --no-verify-ssl 2>/dev/null && \
        echo "  Uploaded to R2: weekly/db_${TIMESTAMP}.dump" || \
        echo "  WARNING: R2 upload failed"
 fi
 # Cleanup
 find "$BACKUP_DIR/weekly" -name "db_*.dump" -mtime "+$((RETENTION_WEEKS * 7))" -delete 2>/dev/null || true
 echo "  Cleaned up backups older than ${RETENTION_WEEKS} weeks"
 echo "[$(date -Iseconds)] Weekly backup complete."
@@ -0,0 +1,63 @@
 #!/bin/bash
 # Weekly backup integrity verification.
 # Downloads latest backup from R2, restores into temp container, runs sanity checks.
 # Schedule: cron weekly (after weekly backup).
 set -e
 R2_BUCKET="${R2_BUCKET:-linearvoid-backups}"
 TEMP_DIR=$(mktemp -d)
 trap "rm -rf $TEMP_DIR" EXIT
 echo "[$(date -Iseconds)] Starting backup integrity verification..."
 # Download latest weekly backup from R2
 if [ -z "$R2_ENDPOINT" ] || [ -z "$R2_ACCESS_KEY_ID" ]; then
    echo "  SKIP: R2 credentials not configured (R2_ENDPOINT, R2_ACCESS_KEY_ID)"
    exit 0
 fi
 LATEST=$(aws s3 ls "s3://${R2_BUCKET}/weekly/" --endpoint-url "$R2_ENDPOINT" --no-verify-ssl 2>/dev/null | sort | tail -1 | awk '{print $4}')
 if [ -z "$LATEST" ]; then
    echo "  SKIP: No weekly backups found in R2"
    exit 0
 fi
 echo "  Downloading: $LATEST"
 aws s3 cp "s3://${R2_BUCKET}/weekly/${LATEST}" "$TEMP_DIR/backup.dump" \
    --endpoint-url "$R2_ENDPOINT" --no-verify-ssl 2>/dev/null
 echo "  File size: $(du -h "$TEMP_DIR/backup.dump" | cut -f1)"
 # Restore into temp PostgreSQL container
 echo "  Starting temp PostgreSQL..."
 docker run -d --name lv-backup-test-$$ \
    -e POSTGRES_PASSWORD=test \
    -e POSTGRES_DB=linearvoid \
    -p 15432:5432 \
    postgres:15-alpine 2>/dev/null
 sleep 5
 # Wait for PG to be ready
 for i in $(seq 1 10); do
    if docker exec lv-backup-test-$$ pg_isready -U postgres 2>/dev/null; then
        break
    fi
    sleep 2
 done
 # Restore
 echo "  Restoring backup..."
 docker cp "$TEMP_DIR/backup.dump" "lv-backup-test-$$:/tmp/backup.dump"
 docker exec lv-backup-test-$$ pg_restore -U postgres -d linearvoid /tmp/backup.dump 2>/dev/null && \
    echo "  Restore: OK" || { echo "  Restore: FAILED"; docker rm -f lv-backup-test-$$ 2>/dev/null; exit 1; }
 # Sanity checks
 MSG_COUNT=$(docker exec lv-backup-test-$$ psql -U postgres -d linearvoid -tAc "SELECT COUNT(*) FROM messages;" 2>/dev/null || echo "0")
 MOD_COUNT=$(docker exec lv-backup-test-$$ psql -U postgres -d linearvoid -tAc "SELECT COUNT(*) FROM moderation_logs;" 2>/dev/null || echo "0")
 echo "  Messages: $MSG_COUNT, Moderation: $MOD_COUNT"
 # Cleanup
 docker rm -f lv-backup-test-$$ 2>/dev/null
 echo "[$(date -Iseconds)] Backup verification complete. OK."
Author	SHA1	Message	Date
zwitschi	a70bad1528	chore: release v1.0.9 - monitoring provisioning + cleanup tracking files CI / test (push) Successful in 56s Details Release / release (push) Successful in 3m20s Details CI / docker (push) Successful in 2m52s Details CI / release (push) Skipped Details	2026-08-06 21:28:53 +02:00
zwitschi	b47f2621e1	docs: update backlog	2026-08-06 20:46:54 +02:00
zwitschi	7244c1d658	feat: enhance chat service with Redis pub/sub for multi-instance message handling, add instance ID for deduplication	2026-08-06 20:44:29 +02:00
zwitschi	d991d6b6c7	chore: changelog 1.0.8 — doc polish CI / test (push) Successful in 51s Details CI / docker (push) Successful in 22s Details CI / release (push) Skipped Details	2026-08-06 20:01:11 +02:00
zwitschi	db3bf96847	docs: complete API error codes, deploy troubleshooting, monitoring/security rewrite, L2+L3 building blocks	2026-08-06 20:00:45 +02:00
zwitschi	0f81a81029	feat: monitoring exporters, alerting, rate limiting, backups, incident runbook CI / test (push) Successful in 58s Details CI / docker (push) Successful in 2m22s Details CI / release (push) Skipped Details - Add 4 Prometheus exporters: nginx (9113), redis (9121), postgres (9187), node (9100) - Create 8 alerting rules in monitoring/prometheus/alerts.yml - Update prometheus.yml with all scrape targets + rule_files - WS connection rate limiting: max 5/IP/60s (Redis INCR) - Admin login rate limiting: max 5/IP/5min (Redis INCR) - 3 backup scripts: daily to R2 (14d), weekly full (8w), verify with restore test - docs/incident-runbook.md with 5-tier triage + escalation - Update CD workflow: full prometheus config + alerts seeding	2026-08-06 19:31:40 +02:00