chore: release v1.0.9 - monitoring provisioning + cleanup tracking files

.github/MEMORY.md

@@ -1 +0,0 @@
-# MEMORY.md

.github/agents/caveman.agent.md

@@ -1,79 +0,0 @@
----
-description: "Caveman-style agent for linearvoid project. Terse output, plan-first workflow, confirmation gating. Use for: implementing TODO.md tasks, editing project files, docs updates, release prep."
-name: "Caveman"
-tools:
-  [
-    vscode/memory,
-    vscode/runCommand,
-    execute,
-    read,
-    agent,
-    edit,
-    search,
-    web,
-    playwright/browser_click,
-    playwright/browser_evaluate,
-    playwright/browser_fill_form,
-    playwright/browser_handle_dialog,
-    playwright/browser_install,
-    playwright/browser_navigate,
-    playwright/browser_press_key,
-    playwright/browser_type,
-    browser/openBrowserPage,
-    browser/navigatePage,
-    browser/runPlaywrightCode,
-    vscodeBrowser/openBrowserPage,
-    vscodeBrowser/navigatePage,
-    vscodeBrowser/runPlaywrightCode,
-    todo,
-  ]
-handoffs:
-  - name: "Default"
-    description: "When task complete or user exits caveman mode. Also handoff for security warnings or irreversible actions."
-    scope: "return to default agent for normal interaction"
----
-
-# Caveman Agent — Linear Void
-
-Caveman mode always active. Terse, no fluff, no filler, no tool-call narration. Technical substance preserved exactly.
-
-## Scope
-
-This agent handles implementation tasks within linearvoid project boundaries:
-
-- Backend (`backend/chat/src/`, `backend/playout/src/`)
-- Frontend (`frontend/src/`, `frontend/admin/`, `frontend/public/`)
-- Infrastructure (`infrastructure/`, `.gitea/workflows/`)
-- Documentation (`docs/`, `*.md` at project root)
-- Config (`docker-compose*.yml`, `.env.example`, `package.json`, `go.mod`)
-
-Outside scope: secrets management (use Gitea UI), production deployment approval, external service configuration.
-
-## Output Rules
-
-Drop articles (a/an/the), filler words, pleasantries, hedging. Fragments OK. Short synonyms. Standard acronyms OK (API, DB, CI). Never invent abbreviations. Code blocks, error strings, technical terms verbatim. Never drop negations (not/never/no/only/except). Numbers and units exact.
-
-## Project Workflow
-
-**Documentation first. Always.**
-
-1. **Read docs before code** — Check `docs/architecture/README.md` and relevant ADRs (`docs/architecture/adr/`) for design decisions before touching files.
-2. **Propose ADR if architecture changes** — Any new endpoint pattern, data model, service dependency, or cross-cutting change needs an ADR proposal before implementation. Write to `docs/architecture/adr/ADR-XXX.md` following existing template.
-3. **Plan** — Write tasks to `.github/instructions/TODO.md` with checkboxes.
-4. **Confirm before acting** — Ask user. Accept `go`, `ok`, `yes`, `next`, `continue`. Abort on `no`, `stop`, `cancel`.
-5. **Do the work** — Edit files, run commands.
-6. **Track completion** — Mark `[x]` in TODO.md. When section done, move to `.github/instructions/DONE.md`.
-7. **Log changes** — Write entry to `CHANGELOG.md` for significant work.
-8. **Sync memory** — Update `.github/MEMORY.md` between major steps and at end of session. Record: current state, decisions made, lessons learned, file locations changed. This is the agent's persistent working memory across sessions.
-
-## Architecture
-
-Source of truth: `docs/architecture/README.md` and ADRs (`docs/architecture/adr/`). Follow project structure in root `README.md`.
-
-## Security
-
-Never commit raw secrets. Never print or log secrets. Use `CI_` prefix for Gitea secrets (never `GITEA_`). `.env` always gitignored.
-
-## Task Tracking
-
-Sync with `manage_todo_list` tool. Mirror state to `.github/instructions/TODO.md` checkboxes.

.github/instructions/BACKLOG.md

@@ -1,5 +1,7 @@
 # BACKLOG
 
+Status reflects 2026-08-06 code scan + implementation batches.
+
 ## Playout & Delivery
 
 - [x] Implement FFmpeg + MediaMTX/SRS continuous HLS playout from playlist
@@ -19,7 +21,7 @@
 - [x] Implement WebSocket chat backend (Go) with room fan-out
 - [x] Implement private DM channel routing
 - [x] Implement Redis presence tracking for chat
-- [ ] Implement Redis pub/sub for multi-instance chat
+- [x] Implement Redis pub/sub for multi-instance chat
 - [x] Implement Redis rate limiting (INCR + EXPIRE windowing, >10/10s throttle)
 - [x] Implement chat message persistence (SQLite v1, PostgreSQL migration path)
 - [x] Implement moderation commands (/mod, /mute, /ban) in backend
@@ -55,15 +57,15 @@
 ## Infrastructure — Docker
 
 - [x] Create Dockerfiles for playout, chat, nginx services
-- [x] Create docker-compose.yml for all services
+- [x] Create docker-compose.yml for all services (11 containers)
 - [x] Create docker-compose.prod.yml override for production
 
 ## Infrastructure — CI/CD (Gitea Actions)
 
-- [x] Implement CI workflow (lint, build, test, push images)
+- [x] Implement CI workflow (lint, build, vet, push images)
 - [x] Implement CD workflow (pull, deploy, health checks — IP-agnostic)
 - [x] Generate .env from Gitea secrets in CD workflow
-- [x] Seed Prometheus config into named volume during deploy
+- [x] Seed Prometheus config + alerts into named volume during deploy
 - [x] Implement health check validation with retry in CD pipeline
 
 ## Cloudflare Configuration
@@ -79,10 +81,10 @@
 
 ## Monitoring & Observability
 
-- [x] Set up Prometheus with chat scrape target
-- [ ] Add nginx/redis/postgres/node exporters
+- [x] Set up Prometheus with chat/playout/nginx/redis/postgres/node scrape targets
+- [x] Add nginx-exporter, redis-exporter, postgres-exporter, node-exporter
 - [x] Import Grafana app + operational dashboards
-- [ ] Define Prometheus alerting rules
+- [x] Define Prometheus alerting rules (8 rules: ChatDown, PlayoutDown, RedisDown, etc.)
 - [ ] Set up UptimeRobot external monitoring
 
 ## Security
@@ -90,15 +92,18 @@
 - [x] Implement admin cookie-session auth (HttpOnly, SameSite=Strict, Secure when TLS)
 - [x] Implement server-side session expiration
 - [ ] Enforce HTTPS/WSS at edge and origin
-- [x] Implement rate limiting for chat messages with Redis windowing
+- [x] Implement rate limiting for chat messages (Redis windowing)
+- [x] Implement WebSocket connection rate limiting (5/IP/60s)
+- [x] Implement admin login rate limiting (5/IP/5min)
 
 ## Data & Storage
 
 - [x] Set up PostgreSQL/SQLite for chat history and moderation data
 - [x] Set up Redis with append-only persistence + maxmemory policy
-- [ ] Implement daily/weekly/monthly Postgres backups
-- [ ] Implement backup integrity verification
-- [ ] Document and rehearse recovery runbook
+- [x] Implement daily Postgres backups to R2 (14-day retention)
+- [x] Implement weekly full backup snapshots (8-week retention)
+- [x] Implement backup integrity verification (restore test)
+- [ ] Document and rehearse recovery runbook (quarterly drill)
 
 ## Scaling & Performance
 
@@ -114,8 +119,8 @@
 ## Documentation
 
 - [x] Document API contracts in API.md
-- [ ] Document deployment runbook in DEPLOYMENT.md
-- [ ] Document monitoring setup in MONITORING.md
-- [ ] Document security model in SECURITY.md
-- [ ] Create incident runbook
-- [ ] Write Level 2/3 building block views
+- [x] Document deployment runbook in DEPLOYMENT.md
+- [x] Document monitoring setup in MONITORING.md
+- [x] Document security model in SECURITY.md
+- [x] Create incident runbook (docs/incident-runbook.md)
+- [x] Write Level 2/3 building block views

CHANGELOG.md

@@ -2,6 +2,21 @@
 
 <!-- markdownlint-disable MD024 -->
 
+## [1.0.9] - 2026-08-06
+
+### Added
+
+- **Prometheus alerts mount**: `alerts.yml` mounted into prometheus container in both dev and prod compose files.
+- **Grafana provisioning**: Prometheus datasource + dashboard provider YAML configs in `monitoring/grafana/`.
+
+### Changed
+
+- **Prod prometheus**: Switched from empty named volume to bind mounts for config files (consistent with dev).
+
+### Removed
+
+- `.github/MEMORY.md`, `.github/agents/`, `.github/instructions/` — tracking files migrated out of repo.
+
 ## [1.0.8] - 2026-08-06
 
 ### Changed

MONITORING.md

@@ -4,12 +4,12 @@ Prometheus + Grafana monitoring for Linear Void.
 
 ## Architecture
 
-```
-┌─────────────┐    ┌──────────────┐    ┌───────────┐
-│  Exporters   │───▶│  Prometheus  │───▶│  Grafana  │
-│  (7 targets) │    │  :9090       │    │  :3001    │
-└─────────────┘    └──────┬───────┘    └───────────┘
-                          │
+```text
+┌──────────────┐     ┌──────────────┐    ┌───────────┐
+│  Exporters   │───> │  Prometheus  │───>│  Grafana  │
+│  (7 targets) │     │  :9090       │    │  :3001    │
+└──────────────┘     └───────┬──────┘    └───────────┘
+                             │
                    alerts.yml (8 rules)
 ```
 

README.md

@@ -14,7 +14,7 @@
 
 The architecture (detailed in [`docs/architecture/README.md`](docs/architecture/README.md)) uses:
 
-- Hetzner Ashburn VPS for playout (FFmpeg + MediaMTX/SRS) and chat *(planned production; currently test lab at 192.168.88.40)*.
+- Hetzner Ashburn VPS for playout (FFmpeg + MediaMTX/SRS) and chat _(planned production; currently test lab at 192.168.88.40)_.
 - Cloudflare R2 + CDN with aggressive immutable caching of HLS `.ts` segments ($0 egress).
 - Lightweight static frontend and WebSocket chat backend.
 - Gitea self-hosted CI/CD (Actions) and Container Registry.

backend/chat/src/main.go

@@ -41,8 +41,17 @@ var (
 	// Rate limit config (env-overridable)
 	rateLimitMax    = 10
 	rateLimitWindow = 10 * time.Second
+
+	// Instance ID for multi-instance pub/sub deduplication
+	instanceID = fmt.Sprintf("chat-%s", generateShortID())
 )
 
+func generateShortID() string {
+	b := make([]byte, 6)
+	rand.Read(b)
+	return hex.EncodeToString(b)
+}
+
 type clientInfo struct {
 	Username string
 	Room     string
@@ -2161,7 +2170,7 @@ func handleMetrics(w http.ResponseWriter, r *http.Request) {
 
 	fmt.Fprintln(w, "# HELP linearvoid_chat_build_info Static build info for chat service.")
 	fmt.Fprintln(w, "# TYPE linearvoid_chat_build_info gauge")
-	fmt.Fprintln(w, "linearvoid_chat_build_info{service=\"chat\",database_driver=\""+databaseDriverName+"\"} 1")
+	fmt.Fprintln(w, "linearvoid_chat_build_info{service=\"chat\",database_driver=\""+databaseDriverName+"\",instance_id=\""+instanceID+"\"} 1")
 }
 
 // broadcast sends a message to all connected clients in a room and saves to DB
@@ -2172,12 +2181,23 @@ func broadcast(msg Message) {
 	// Persist to configured SQL backend.
 	db.Create(&msg)
 
+	// Local fan-out
 	for client := range clients {
 		if err := client.WriteJSON(msg); err != nil {
 			client.Close()
 			delete(clients, client)
+			delete(clientMeta, client)
 		}
 	}
+
+	// Redis pub/sub for multi-instance fan-out
+	if redisClient != nil && msg.Room != "" {
+		payload, _ := json.Marshal(map[string]any{
+			"instance": instanceID,
+			"msg":      msg,
+		})
+		redisClient.Publish(context.Background(), fmt.Sprintf("room:%s:messages", msg.Room), payload)
+	}
 }
 
 // handleConnection handles WebSocket connections for chat (public rooms, private DMs, moderation)
@@ -2333,6 +2353,47 @@ func pushChatHistory(conn *websocket.Conn, room string, limit int) {
 // dmsSent tracks total private messages for Prometheus metrics (set from handleMetrics).
 var dmsSent int64
 
+// subscribeRoomMessages listens for Redis pub/sub messages and fans out locally.
+func subscribeRoomMessages() {
+	if redisClient == nil {
+		return
+	}
+	pubsub := redisClient.PSubscribe(context.Background(), "room:*:messages")
+	defer pubsub.Close()
+
+	log.Printf("Redis pub/sub subscriber started (instance %s)", instanceID)
+
+	for {
+		msg, err := pubsub.ReceiveMessage(context.Background())
+		if err != nil {
+			log.Printf("Redis pub/sub error: %v — retrying in 5s", err)
+			time.Sleep(5 * time.Second)
+			pubsub = redisClient.PSubscribe(context.Background(), "room:*:messages")
+			continue
+		}
+
+		var envelope struct {
+			Instance string  `json:"instance"`
+			Msg      Message `json:"msg"`
+		}
+		if err := json.Unmarshal([]byte(msg.Payload), &envelope); err != nil {
+			continue
+		}
+
+		// Skip messages originating from this instance
+		if envelope.Instance == instanceID {
+			continue
+		}
+
+		// Fan out to local clients
+		mutex.Lock()
+		for client := range clients {
+			client.WriteJSON(envelope.Msg)
+		}
+		mutex.Unlock()
+	}
+}
+
 // handlePrivateDM routes private messages to the specific user pair
 func handlePrivateDM(msg Message) {
 	msg.Type = "private"
@@ -2640,6 +2701,11 @@ func main() {
 	initRedis()
 	initDB()
 
+	// Subscribe to Redis pub/sub for multi-instance message fan-out
+	go subscribeRoomMessages()
+
+	log.Printf("Instance ID: %s", instanceID)
+
 	http.HandleFunc("/ws", handleConnection)
 	http.HandleFunc("/api/admin/login", handleAdminLogin)
 	http.HandleFunc("/api/admin/logout", handleAdminLogout)

docker-compose.prod.yml

@@ -159,7 +159,8 @@ services:
     container_name: linearvoid-prometheus
     restart: unless-stopped
     volumes:
-      - prometheus-config:/etc/prometheus
+      - ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
+      - ./monitoring/prometheus/alerts.yml:/etc/prometheus/alerts.yml
       - prometheus-data:/prometheus
     ports:
       - "9090:9090"
@@ -199,5 +200,4 @@ volumes:
   redis-data:
   postgres-data:
   prometheus-data:
-  prometheus-config:
   grafana-data:

docker-compose.yml

@@ -153,6 +153,7 @@ services:
     restart: unless-stopped
     volumes:
       - ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
+      - ./monitoring/prometheus/alerts.yml:/etc/prometheus/alerts.yml
       - prometheus-data:/prometheus
     ports:
       - "9090:9090"

frontend/package.json

@@ -1,7 +1,7 @@
 {
   "name": "frontend",
   "private": true,
-  "version": "1.0.7",
+  "version": "1.0.9",
   "type": "module",
   "scripts": {
     "dev": "vite",

monitoring/grafana/dashboards/dashboard.yml

@@ -0,0 +1,12 @@
+apiVersion: 1
+
+providers:
+  - name: "default"
+    orgId: 1
+    folder: ""
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 10
+    allowUiUpdates: true
+    options:
+      path: /etc/grafana/provisioning/dashboards

monitoring/grafana/datasources/datasource.yml

@@ -0,0 +1,9 @@
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    editable: true