zwitschi/linearvoid
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 9 Wiki Activity

Refactor load testing scripts to support Bash and PowerShell
CI / test (push) Successful in 2m50s
Details
CI / docker (push) Successful in 5m12s
Details

Browse Source 
- Updated README.md to reflect changes in load testing commands and added usage examples for Bash scripts.
- Converted PowerShell load testing scripts to Bash equivalents for better cross-platform compatibility.
- Added new Bash scripts: run-loadtests.sh, summarize-artillery.sh, and summarize-k6.sh to mirror existing PowerShell functionality.
- Updated admin-api.js and ws-chat.yml for consistency and improved performance metrics reporting.
- Enhanced error handling and output formatting in summary scripts for better readability.
This commit is contained in:
zwitschi committed 2026-09-02 19:47:34 +02:00
1 parent 9f60087a4c
commit 33b6296c5e
22 files changed
+2434 -3885
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CHANGELOG.md
+2
View File
@@ -6,10 +6,12 @@
### Added
- **Bash equivalents of Windows scripts**: Added `scripts/cache-hit-ratio.sh`, `scripts/check-env.sh`, `scripts/check-tokens.sh`, and `scripts/loadtest/run-loadtests.sh`, `scripts/loadtest/summarize-artillery.sh`, `scripts/loadtest/summarize-k6.sh` — Linux/macOS ports of the existing PowerShell scripts (the `.ps1` files are retained).
- **Transient presence flash**: Join/leave announcements now appear as a 3-second flash at the top of the chat panel (`#chat-flash`) with a fade-out, instead of permanent chat rows. Presence indicators still update from the same events.
### Changed
- **Docs condensed and de-Windows'd**: Removed Windows/PowerShell-specific instructions from the docs and condensed the longest documents into tighter, bash-first guides — `SETUP.md`, `DEPLOYMENT.md`, `API.md`, `MONITORING.md`, `STREAMING.md`, `COOLIFY.md`, `incident-runbook.md`, `operations/scaling-performance-playbook.md`, `architecture/README.md`, and the Prometheus metrics inventory. `scripts/loadtest/README.md` now documents the bash load-test helpers.
- **System messages no longer persisted**: The chat backend no longer stores `system`-type messages; chat history and the admin chat-log table only contain real chat messages (moderation actions remain in the audit log).
- **Reconnect grace period**: Disconnects schedule a delayed "left" announcement (5s); a reconnect within 5s cancels it and skips "joined", eliminating join/leave spam on brief reconnects.
- **HLS segment duration 2s**: Default `HLS_SEGMENT_DURATION` changed from 6 to 2 and `HLS_WINDOW_SIZE` from 24 to 12 (2s × 12 = 24s live window), reducing end-to-end latency; the 2s GOP was already in place.
README.md
+5 -3
View File
@@ -213,11 +213,13 @@ One-off and CI-friendly scripts for video preparation, R2 sync, backups, and pla
- loadtest/
- [admin-api.js](scripts/loadtest/admin-api.js) — admin API load test script
- [run-loadtests.ps1](scripts/loadtest/run-loadtests.ps1) — PowerShell script to run load tests
- [summarize-artillery.ps1](scripts/loadtest/summarize-artillery.ps1) — PowerShell script to summarize Artillery load test results
- [summarize-k6.ps1](scripts/loadtest/summarize-k6.ps1) — PowerShell script to summarize k6 load test results
- [run-loadtests.sh](scripts/loadtest/run-loadtests.sh) — bash script to run ws and/or admin load tests
- [summarize-artillery.sh](scripts/loadtest/summarize-artillery.sh) — summarize Artillery JSON results
- [summarize-k6.sh](scripts/loadtest/summarize-k6.sh) — summarize k6 JSON results
- [ws-chat.yml](scripts/loadtest/ws-chat.yml) — WebSocket chat load test configuration
PowerShell equivalents (`*.ps1`) of the three load-test scripts are kept alongside them.
## Getting Started
> **Note:** The `docker compose up` below targets local development. The test lab (192.168.88.40) receives automated CD deploys. Production Hetzner VPS is not yet provisioned — see `DEPLOYMENT.md`.
docs/API.md
+68 -216
View File
@@ -1,216 +1,101 @@
# API Reference
# Linear Void API Reference
This document describes the currently implemented API surfaces in Linear Void.
Base URLs: Public/admin app `https://linearvoid.tv` · WebSocket `wss://linearvoid.tv/ws` · Local fallback `http://localhost` / `ws://localhost/ws`
## Base URLs
## Authentication
- Public app and admin: <https://linearvoid.tv>
- WebSocket endpoint: <wss://linearvoid.tv/ws>
- Local fallback: <http://localhost> and <ws://localhost/ws>
Admin REST endpoints use a cookie session:
## Authentication Model
Admin REST endpoints use a cookie session.
- Cookie name: linearvoid_admin_session
- Cookie attributes: HttpOnly, SameSite=Strict, Secure when request is HTTPS (or X-Forwarded-Proto=https)
- Session TTL: controlled by ADMIN_SESSION_TTL_HOURS (default 12h)
- Cookie: `linearvoid_admin_session` — `HttpOnly`, `SameSite=Strict`, `Secure` when request is HTTPS (or `X-Forwarded-Proto=https`)
- Session TTL: `ADMIN_SESSION_TTL_HOURS` (default 12h)
## Chat WebSocket API
### Connect
Connect: `GET /ws?username={name}&room={room}` — client/server exchange JSON messages.
Endpoint:
| Message (client→server) | Field | Notes |
| ----------------------- | -------------------------------------------------------------------------------- | ----- |
| `type` | `public` for room chat, `private` for DM routing; `system` for moderation/events | |
| `room` | `main`; private DMs use `dm:userA-userB` (usernames sorted) | |
| `username` | sender | |
| `text` | message body | |
- GET /ws?username={name}&room={room}
Client sends JSON messages over the socket.
### Message Envelope (client -> server)
Server→client envelope adds `"timestamp"` (ISO-8601, e.g. `2026-08-04T12:34:56Z`).
```json
{
"room": "main",
"username": "viewer123",
"text": "hello world",
"type": "public"
}
```
Notes:
- type is usually public for room chat and private for DM room routing.
- Private room names are formatted like dm:userA-userB (sorted usernames).
- Moderation/system events are sent as type=system messages.
### Message Envelope (server -> client)
```json
{
"room": "main",
"username": "viewer123",
"text": "hello world",
"type": "public",
"timestamp": "2026-08-04T12:34:56Z"
}
{ "room": "main", "username": "viewer123", "text": "hello world", "type": "public" }
```
## Admin Session API
### Login
| Method & Path | Auth | Description |
| ------------------------ | ----------------------------------- | ----------------------------- |
| `POST /api/admin/login` | none | Authenticate |
| `POST /api/admin/logout` | optional (clears cookie either way) | Logout |
| `GET /api/admin/session` | cookie required | Session check; 401 if invalid |
- POST /api/admin/login
- Auth: none
Request:
Login request:
```json
{
"username": "admin",
"password": "your-password"
}
{ "username": "admin", "password": "your-password" }
```
Success response:
Login/logout/session success response (session returns 401 on failure):
```json
{
"authenticated": true,
"username": "admin"
}
{ "authenticated": true, "username": "admin" }
```
### Logout
Logout response: `{ "authenticated": false }`
- POST /api/admin/logout
- Auth: optional (clears cookie either way)
## Admin Data API (auth required)
Response:
| Method & Path | Returns |
| ----------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `GET /api/admin/overview` | Aggregate counters + runtime/system snapshot |
| `GET /api/admin/playlist` | Playlist |
| `PUT /api/admin/playlist` | Set playlist (body: `{ "content": "file '/app/media/video1.mp4'\nfile '/app/media/video2.mp4'\n" }`) |
| `GET /api/admin/schedule` | Parsed playlist entries + matched media metadata/runtime |
| `GET /api/admin/moderation/logs?limit=50` | Moderation logs |
| `POST /api/admin/moderation/actions` | Perform moderation action |
| `GET /api/admin/analytics` | Totals (messages, moderation actions, rooms) + top room summary |
| `GET /api/admin/system` | Health snapshot (chat/playout/redis/database) + config metadata |
| `GET /api/admin/users` | Presence users (room + username) when Redis available |
Moderation action request:
```json
{
"authenticated": false
}
{ "action": "mute", "target": "username-or-id", "moderator": "admin", "reason": "spam", "room": "main" }
```
### Session Check
- GET /api/admin/session
- Auth: cookie required
Success response:
```json
{
"authenticated": true,
"username": "admin"
}
```
Failure:
- 401 unauthorized
## Admin Data API
All endpoints below require admin authentication.
### Overview
- GET /api/admin/overview
Returns aggregate counters and runtime/system snapshot fields.
### Playlist
- GET /api/admin/playlist
- PUT /api/admin/playlist
PUT request:
```json
{
"content": "file '/app/media/video1.mp4'\nfile '/app/media/video2.mp4'\n"
}
```
### Schedule
- GET /api/admin/schedule
Returns parsed playlist entries plus matched media metadata/runtime.
### Moderation
- GET /api/admin/moderation/logs?limit=50
- POST /api/admin/moderation/actions
Action request:
```json
{
"action": "mute",
"target": "username-or-id",
"moderator": "admin",
"reason": "spam",
"room": "main"
}
```
### Analytics
- GET /api/admin/analytics
Returns totals (messages, moderation actions, rooms) and top room summary.
### System
- GET /api/admin/system
Returns service health snapshot for chat/playout/redis/database and config metadata.
### Active Users
- GET /api/admin/users
Returns currently discovered presence users (room + username) when Redis is available.
## Media Library API
### Media Collection
| Method & Path | Description |
| ------------------------------ | ------------------------------------------------- |
| `GET /api/admin/media` | List media |
| `POST /api/admin/media` | Multipart upload (legacy), form field name `file` |
| `GET /api/admin/media/{id}` | Get item |
| `PUT /api/admin/media/{id}` | Update item |
| `DELETE /api/admin/media/{id}` | Delete item |
- GET /api/admin/media
- POST /api/admin/media (multipart upload, legacy path)
Multipart form field name: file
### Media Item
- GET /api/admin/media/{id}
- PUT /api/admin/media/{id}
- DELETE /api/admin/media/{id}
PUT request:
PUT media body:
```json
{
"status": "ready",
"title": "My Clip",
"tags": "music,live"
}
{ "status": "ready", "title": "My Clip", "tags": "music,live" }
```
## Resumable Upload API
Maximum file size: 2GB
Max file size: **2GB** · Default chunk size: **8MB**
Default chunk size: 8MB
| Method & Path | Description |
| ------------------------------------------------------------------ | ----------------------------------------------------------------- |
| `POST /api/admin/media/uploads` | Initialize or resume session |
| `GET /api/admin/media/uploads/{uploadId}` | Check upload status |
| `PUT /api/admin/media/uploads/{uploadId}/chunk?index={chunkIndex}` | Upload chunk (`Content-Type: application/octet-stream`) |
| `POST /api/admin/media/uploads/{uploadId}/complete` | Complete upload → returns media listing (`files` + `generatedAt`) |
### Initialize or Resume Session
- POST /api/admin/media/uploads
Request:
Initialize request:
```json
{
@@ -223,7 +108,7 @@ Request:
}
```
Response:
Status envelope (returned by init, status check, and chunk upload):
```json
{
@@ -237,61 +122,33 @@ Response:
}
```
### Check Upload Status
## Metrics & Health
- GET /api/admin/media/uploads/{uploadId}
Returns same status envelope as initialize.
### Upload Chunk
- PUT /api/admin/media/uploads/{uploadId}/chunk?index={chunkIndex}
- Content-Type: application/octet-stream
Response returns updated upload status envelope.
### Complete Upload
- POST /api/admin/media/uploads/{uploadId}/complete
On success returns admin media listing payload (files + generatedAt).
## Metrics and Health
### Chat metrics
- GET /metrics
Prometheus exposition includes connected clients, stored messages, moderation actions, Redis indicators, active admin sessions, and build info.
### Health
- GET /health on chat service
- GET /health on playout service
- GET /health on nginx reverse proxy
- `GET /metrics` (chat) — Prometheus: connected clients, stored messages, moderation actions, Redis indicators, active admin sessions, build info
- `GET /health` — chat service, playout service, and nginx reverse proxy each expose it
## HTTP Status Conventions
| Code | Meaning | When |
| ---- | --------------------- | ----------------------------------------------------------- |
| 200 | OK | Successful GET, PUT, or action |
| 200 | OK | Successful GET, PUT, action |
| 201 | Created | Successful POST (upload init, override created) |
| 204 | No Content | Successful DELETE, CORS preflight |
| 400 | Bad Request | Malformed JSON, missing required fields, invalid input |
| 401 | Unauthorized | Missing or expired admin session |
| 403 | Forbidden | Valid session but insufficient permissions |
| 401 | Unauthorized | Missing/expired admin session |
| 403 | Forbidden | Valid session, insufficient permissions |
| 404 | Not Found | Resource doesn't exist |
| 405 | Method Not Allowed | Wrong HTTP method for endpoint |
| 405 | Method Not Allowed | Wrong method for endpoint |
| 409 | Conflict | Schedule override overlap, upload ID mismatch |
| 413 | Payload Too Large | Upload exceeds 2 GB limit |
| 422 | Unprocessable | File fails normalization check (codec, fps, resolution) |
| 429 | Too Many Requests | Rate limit exceeded (chat messages, WS connections, login) |
| 429 | Too Many Requests | Rate limit exceeded (chat, WS, login) |
| 500 | Internal Server Error | Unexpected server failure |
| 503 | Service Unavailable | Redis unavailable for user actions, playout health degraded |
## Rate Limiting
All limits are per-IP (from `X-Forwarded-For` or `RemoteAddr`) unless noted.
Per-IP (from `X-Forwarded-For` or `RemoteAddr`) unless noted.
| Resource | Limit | Window | Response |
| --------------------- | ---------------------------------- | ---------------------------------------- | ------------------------------------------------------ |
@@ -299,9 +156,4 @@ All limits are per-IP (from `X-Forwarded-For` or `RemoteAddr`) unless noted.
| WebSocket connections | 5 | 60s | 429 + `"too many connections from this IP"` |
| Admin login | 5 | 5min | 429 + `"too many login attempts, try again later"` |
Rate limit headers on applicable responses:
- `X-RateLimit-Limit`: maximum requests per window
- `Retry-After`: seconds until window resets
- 413: payload too large
- 500: internal server error
Rate limit headers (applicable responses): `X-RateLimit-Limit` (max requests/window), `Retry-After` (seconds until reset). 413 = payload too large; 500 = internal server error.
docs/COOLIFY.md
+61 -142
View File
@@ -1,138 +1,80 @@
# Coolify Deployment
**Last Updated**: 2026-08-28
*Last Updated: 2026-08-28*
This guide describes an alternative deployment path for Linear Void using
[Coolify](https://github.com/coollabsio/coolify/tree/v4.3.14) (v4.3.14) — an
open-source, self-hostable PaaS that lets you manage servers, applications, and
databases over SSH without vendor lock-in.
Alternative deployment path for Linear Void using [Coolify](https://github.com/coollabsio/coolify/tree/v4.3.14) (v4.3.14), an open-source self-hostable PaaS that manages servers, apps, and databases over SSH. Drop-in alternative to the flows in [SETUP.md](./SETUP.md): local dev (`docker-compose.yml`) and test-lab/prod (`docker-compose.prod.yml` via Gitea Actions CD). Coolify deploys the same Compose stack through a web UI and its own Traefik reverse proxy for automatic HTTPS, domain routing, and one-click redeploys (no manual `DOCKER_HOST`).
Coolify is a drop-in alternative to the two existing deployment flows described
in [SETUP.md](./SETUP.md):
## Why Coolify
1. **Local development** (`docker-compose.yml`) — builds from source on your
workstation.
2. **Test-lab / production** (`docker-compose.prod.yml`) — pulls prebuilt images
from the Gitea registry and is driven by the Gitea Actions CD workflow.
- **Single control plane**: manage playout, chat, nginx, redis, db (plus exporters/monitoring) from one dashboard.
- **Automatic TLS**: Let's Encrypt provisioning/renewal per app/domain, replacing the manual Cloudflare + nginx origin TLS setup for the app hostname.
- **No vendor lock-in**: config lives on your server; containers keep running if you stop using Coolify — you only lose automation.
- **Simpler cutover**: on Hetzner VPS provisioning, point Coolify at the new server and redeploy instead of editing CD `DOCKER_HOST`.
With Coolify, the same Docker Compose stack is deployed through a web UI and
Coolify's own reverse proxy (Traefik), giving you automatic HTTPS, domain
routing, and one-click redeploys without managing `DOCKER_HOST` manually.
## Why use Coolify?
- **Single control plane**: manage the playout, chat, nginx, redis, and db
services (plus exporters and monitoring) from one dashboard.
- **Automatic TLS**: Coolify provisions and renews Let's Encrypt certificates
per application/domain, replacing the manual Cloudflare + nginx origin TLS
dance for the app hostname.
- **No vendor lock-in**: all configuration lives on your server. If you stop
using Coolify, the running containers keep working; you only lose the
automation.
- **Simpler cutover**: when the Hetzner VPS is provisioned, point Coolify at the
new server and redeploy instead of editing CD `DOCKER_HOST` values.
> **Note**: Coolify replaces the _deployment orchestration_ layer only. Cloudflare
> DNS, the R2 media bucket, cache rules, and the edge Worker described in
> [SETUP.md](./SETUP.md) still apply and are unchanged.
> Replaces the *deployment orchestration* layer only. Cloudflare DNS, R2 media bucket, cache rules, and the edge Worker from [SETUP.md](./SETUP.md) are unchanged.
## Prerequisites
- A Linux server running **Docker Engine** (any distro Coolify supports).
- **SSH access** (root or a user with passwordless `sudo`) from the machine
where you run the Coolify install — Coolify uses SSH to provision the target
server.
- At least **2 GB RAM / 20 GB disk** for the Coolify instance itself (plus
additional resources for the Linear Void stack).
- A domain name pointed at the server for HTTPS (e.g. `linearvoid.tv`).
- Ports `80` and `443` reachable on the server for Traefik.
- The Gitea registry credentials (`CI_REGISTRY_USER` / `CI_REGISTRY_TOKEN`) if
you intend to pull the prebuilt `playout`, `chat`, and `nginx` images.
- Linux server with **Docker Engine** (any supported distro).
- **SSH access** (root or passwordless `sudo`) from the Coolify install machine — Coolify provisions the target over SSH.
- **2 GB RAM / 20 GB disk** minimum for Coolify itself (plus resources for the stack).
- A domain pointed at the server for HTTPS (e.g. `linearvoid.tv`).
- Ports `80` and `443` reachable for Traefik.
- Gitea registry credentials (`CI_REGISTRY_USER` / `CI_REGISTRY_TOKEN`) if pulling prebuilt `playout`, `chat`, `nginx` images.
## Installation
Install Coolify on a dedicated server (or alongside your stack — but a dedicated
Coolify server is the recommended layout):
Recommended on a dedicated server:
```bash
curl -fsSL https://cdn.coollabs.io/coolify/install.sh | bash
```
The installer boots a Docker-based Coolify instance and prints the URL and
default credentials. Open the dashboard in your browser and complete first-run
setup.
For non-interactive or offline installs, see the
[installation docs](https://coolify.io/docs/installation) and the
[install script source](https://github.com/coollabsio/coolify/blob/v4.3.14/scripts/install.sh).
The installer boots a Docker-based Coolify instance and prints the URL and default credentials. Complete first-run setup in the dashboard. For non-interactive/offline installs, see the [installation docs](https://coolify.io/docs/installation) and [install script source](https://github.com/coollabsio/coolify/blob/v4.3.14/scripts/install.sh).
## Add the Linear Void server
1. In the Coolify dashboard, go to **Servers → Add**.
2. Choose your provider (or **Localhost** if Coolify runs on the same machine
as the stack).
3. Provide the SSH details. For a remote host, Coolify validates the connection
and installs its agent.
4. Verify the server shows as **Reachable** before creating resources.
1. **Servers → Add**.
2. Choose your provider (or **Localhost** if Coolify runs on the same host as the stack).
3. Provide SSH details; for a remote host Coolify validates and installs its agent.
4. Confirm the server shows **Reachable** before creating resources.
### Current topology (verified 2026-08-30)
The active test-lab setup is **two-tier**:
Active test-lab setup is **two-tier**:
| Component | Host | Environment | Exposes | Docker TCP | Notes |
| --------------------- | --------------- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Coolify dashboard** | `192.168.88.18` | Coolify; docker container; coolify dashboard at `8000` | `https://coolify.allucanget.biz` via Cloudflare tunnel (`coolify-cloudflared`) | No | Coolify manages target servers over SSH |
| **Deployment target** | `192.168.88.40` | Only Linear Void deploys here | `coolify-proxy` = Traefik on 80/443, Port 8000 on this host is bound by Portainer, not Coolify. | Yes | Contains Coolify's per-server helpers (`coolify-proxy`, `coolify-sentinel`, `coolify-cloudflared`) plus the stack via `docker-compose.prod.yml` (the `linearvoid` compose project, 11 containers). |
| Component | Host | Exposes | Docker TCP | Notes |
|---|---|---|---|---|
| **Coolify dashboard** | `192.168.88.18` | `https://coolify.allucanget.biz` via Cloudflare tunnel (`coolify-cloudflared`); dashboard at `8000` | No | Manages target servers over SSH |
| **Deployment target** | `192.168.88.40` | `coolify-proxy` = Traefik on 80/443; port `8000` bound by Portainer, not Coolify | Yes | Per-server helpers (`coolify-proxy`, `coolify-sentinel`, `coolify-cloudflared`) + the stack via `docker-compose.prod.yml` (`linearvoid` compose project, 11 containers) |
The main `coolify` app container does **not** run on `192.168.88.40`; it lives on
the dashboard host `192.168.88.18`. When a Coolify deployment creates a Compose
resource, the app services land on the target server `192.168.88.40`.
The main `coolify` app container runs only on the dashboard host `192.168.88.18`; Compose-resource services land on the target `192.168.88.40`. The dashboard host exposes no Docker TCP socket, so run the checks below against the **target** `192.168.88.40`:
Useful checks:
```powershell
# Environment / deployment health on the target 192.168.88.40 (containers, helpers, health)
./scripts/check-env.ps1
# Credential / token access (Gitea registry, Cloudflare, R2, admin login)
./scripts/check-tokens.ps1
```bash
./scripts/check-env.sh # env/deployment health on target (containers, helpers, health)
./scripts/check-tokens.sh # Gitea registry, Cloudflare, R2, admin credential access
```
The Coolify dashboard runs on the `192.168.88.18` container host (port `8000`,
publicly `https://coolify.allucanget.biz`) and exposes no Docker TCP socket, so
the environment/token scripts are intended to run against the deployment target
`192.168.88.40`, not `192.168.88.18`.
## Deploy the stack
Linear Void is a multi-service Docker Compose stack, so deploy it as a
**Compose** (Docker Compose) resource rather than individual applications.
Deploy as a **Docker Compose** resource (not individual apps):
1. **Projects → Add → Docker Compose**.
2. Select the target server.
3. In **Build Settings**, set the compose file source:
- **Git Repository** — point Coolify at
`https://git.allucanget.biz/zwitschi/linearvoid.git` and use
`docker-compose.prod.yml` as the compose file. Coolify pulls on each
redeploy.
- **Raw Compose** — paste the contents of `docker-compose.prod.yml` directly
if you prefer to manage it manually.
4. If using the Git source, add the Gitea registry login as a **Deployment Key /
environment secret** so the `playout`, `chat`, and `nginx` images can be
pulled. Alternatively, switch those services to `build:` blocks (as in
`docker-compose.yml`) so Coolify builds from source on the server.
- **Git Repository** → point at `https://git.allucanget.biz/zwitschi/linearvoid.git`, compose file `docker-compose.prod.yml` (Coolify pulls on each redeploy).
- **Raw Compose** → paste `docker-compose.prod.yml` directly to manage manually.
4. For the Git source, add the Gitea registry login as a **Deployment Key / environment secret** so `playout`, `chat`, `nginx` images can be pulled. Alternatively switch those services to `build:` blocks (as in `docker-compose.yml`) to build from source on the server.
### Environment variables
Coolify injects a secret store into the compose resource. Map the Linear Void
environment to Coolify's **Environment Variables** tab (these correspond to the
values normally placed in `.env`):
Map Linear Void's `.env` values into Coolify's **Environment Variables** tab:
| Variable | Purpose |
| ---------------------------------------------- | --------------------------------------------------------------------- |
|---|---|
| `DATABASE_URL` | `postgres://linearvoid:<password>@db:5432/linearvoid?sslmode=disable` |
| `REDIS_URL` | `redis://redis:6379/0` |
| `POSTGRES_PASSWORD` | Database password (used by the `db` service) |
| `POSTGRES_PASSWORD` | DB password (used by `db` service) |
| `ADMIN_USERNAME` / `ADMIN_PASSWORD` | Admin panel credentials |
| `ADMIN_SESSION_TTL_HOURS` | Admin session lifetime |
| `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` | Cloudflare R2 credentials |
@@ -142,88 +84,65 @@ values normally placed in `.env`):
| `VITE_PUBLIC_WS_URL` | `wss://linearvoid.tv/ws` |
| `VITE_PUBLIC_HLS_URL` | `https://media.linearvoid.tv/hls/playlist.m3u8` |
Keep internal service URLs on compose hostnames (`redis`, `db`) exactly as in
`docker-compose.prod.yml`.
Keep internal service URLs on compose hostnames (`redis`, `db`) exactly as in `docker-compose.prod.yml`.
### Persistent storage
The compose file uses named volumes (`redis-data`, `postgres-data`) and bind
mounts under `./data` (playlist, media, hls, chat, slate). Configure these under
**Persistent Storage** so data survives redeploys:
Named volumes (`redis-data`, `postgres-data`) and bind mounts under `./data` must be configured under **Persistent Storage** to survive redeploys:
| Mount | Purpose |
| --------------- | ----------------------------------------------- |
|---|---|
| `data/playlist` | Playout playlist file |
| `data/media` | Normalized source videos (read-only to playout) |
| `data/hls` | Generated HLS manifests and segments |
| `data/hls` | Generated HLS manifests/segments |
| `data/chat` | Chat service working data |
| `data/slate` | Slate media |
| `redis-data` | Redis append-only persistence |
| `postgres-data` | PostgreSQL data directory |
Create the `data` subdirectories before the first deploy if Coolify does not
auto-create them (`mkdir -p data/playlist data/media data/hls data/chat`).
If Coolify doesn't auto-create the dirs, create them before first deploy:
```bash
mkdir -p data/playlist data/media data/hls data/chat
```
### Domains and proxy
1. Open the compose resource and assign the **nginx** service a public domain
(e.g. `linearvoid.tv`). Coolify's Traefik reverse proxy then terminates TLS
and forwards to the `nginx` container.
2. Set the **Ports Exposes** mapping so Traefik targets the nginx service port
(`80` inside the container, mapped from `${NGINX_PORT:-8080}:80`).
3. Do **not** also expose the playout (`8081`) or chat (`3000`) health ports
publicly — keep those internal to the compose network and let Traefik route
through nginx.
4. For the media domain (`media.linearvoid.tv`), keep serving HLS from the R2
custom domain (per SETUP.md) rather than routing it through Coolify, so
segments stay on Cloudflare's edge cache.
1. Assign the **nginx** service a public domain (e.g. `linearvoid.tv`); Traefik terminates TLS and forwards to `nginx`.
2. Set **Ports Exposes** so Traefik targets the nginx port (`80` inside container, mapped from `${NGINX_PORT:-8080}:80`).
3. Do **not** publicly expose playout (`8081`) or chat (`3000`) health ports — keep them internal and route through nginx.
4. For `media.linearvoid.tv`, keep serving HLS from the R2 custom domain (per SETUP.md) rather than routing through Coolify, so segments stay on Cloudflare's edge cache.
## Post-deploy verification
After the first deploy, confirm:
- `docker compose ps` (on the server) shows all services healthy.
- `docker compose ps` on the server shows all services healthy.
- Playout health: `https://<host>/health` via nginx (internal `:8081/health`).
- Chat health: internal `:3000/health`.
- HLS is written to `data/hls` and served from `media.linearvoid.tv`.
- HLS written to `data/hls` and served from `media.linearvoid.tv`.
- Chat WebSocket connects over `wss://linearvoid.tv/ws`.
- Admin login works at `/admin` with the configured credentials.
- Admin login works at `/admin` with configured credentials.
## Monitoring
Coolify exposes container logs and basic metrics per resource. The stack also
ships Prometheus + Grafana (`monitoring/`) and nginx/redis exporters. If you
want those reachable through Coolify, expose Grafana on a restricted subdomain
(e.g. `grafana.linearvoid.tv`) and keep Prometheus internal, mirroring the
access guidance in SETUP.md.
Coolify exposes container logs/basic metrics per resource. The stack ships Prometheus + Grafana (`monitoring/`) plus nginx/redis exporters. To reach them via Coolify, expose Grafana on a restricted subdomain (e.g. `grafana.linearvoid.tv`) and keep Prometheus internal, mirroring SETUP.md access guidance.
## Cutover from Gitea CD
Coolify can fully replace the Gitea Actions CD step. To switch:
1. Deploy the stack as a Coolify compose resource (above).
2. Point Cloudflare DNS for `linearvoid.tv` / `www` at the server IP (proxied).
3. Disable (or keep as a fallback) the `cd` job in the Gitea workflow so the two
orchestrators don't fight over the same `DOCKER_HOST`.
4. Keep the Gitea **CI** (build + push images) job so Coolify continues pulling
`:latest` images from the registry.
3. Disable (or keep as fallback) the `cd` job in the Gitea workflow so the two orchestrators don't fight over the same `DOCKER_HOST`.
4. Keep the Gitea **CI** (build + push images) job so Coolify keeps pulling `:latest` from the registry.
## Troubleshooting
- **Image pull fails**: ensure the Gitea registry credentials are set in the
compose resource's secret store, or switch the services to `build:` blocks.
- **Healthchecks fail**: confirm `curl` is available in the built images (the
compose healthchecks rely on `curl`).
- **TLS errors**: check the domain's DNS points at the Coolify server and ports
`80`/`443` are open.
- **HLS not updating**: verify `data/hls` is writable and the playout service
restarts cleanly; see the normalization troubleshooting in SETUP.md.
- **Ports conflict**: Coolify's Traefik binds `80`/`443`; do not also map those
to the host in the compose file.
- **Image pull fails**: set Gitea registry credentials in the compose secret store, or switch services to `build:` blocks.
- **Healthchecks fail**: confirm `curl` exists in the built images (healthchecks rely on it).
- **TLS errors**: check the domain's DNS points at the Coolify server and ports `80`/`443` are open.
- **HLS not updating**: verify `data/hls` is writable and playout restarts cleanly; see SETUP.md normalization troubleshooting.
- **Ports conflict**: Coolify's Traefik binds `80`/`443`; do not also map those to the host in the compose file.
## Further reading
- Coolify repo (v4.3.14): <https://github.com/coollabsio/coolify/tree/v4.3.14>
- Coolify docs: <https://coolify.io/docs>
- Linear Void deployment: [DEPLOYMENT.md](./DEPLOYMENT.md)
- Linear Void setup: [SETUP.md](./SETUP.md)
- [DEPLOYMENT.md](./DEPLOYMENT.md) · [SETUP.md](./SETUP.md)
docs/DEPLOYMENT.md
+53 -175
View File
@@ -1,64 +1,26 @@
# Deployment Guide
## Table of Contents
- [Deployment Guide](#deployment-guide)
- [Table of Contents](#table-of-contents)
- [Environments](#environments)
- [Production Target (Hetzner)](#production-target-hetzner)
- [Planned Infrastructure](#planned-infrastructure)
- [Cutover Plan](#cutover-plan)
- [Production Preflight](#production-preflight)
- [Firewall / Security Group](#firewall--security-group)
- [Pipeline Readiness](#pipeline-readiness)
- [Target Topology](#target-topology)
- [Prerequisites](#prerequisites)
- [Environment File](#environment-file)
- [Gitea Secrets](#gitea-secrets)
- [Pull-and-Run Deployment](#pull-and-run-deployment)
- [First-Time or Recovery Provisioning](#first-time-or-recovery-provisioning)
- [Health Verification](#health-verification)
- [Zero/Low-Downtime Update Pattern](#zerolow-downtime-update-pattern)
- [Rollback](#rollback)
- [Operational Tasks](#operational-tasks)
- [View logs](#view-logs)
- [Restart one service](#restart-one-service)
- [Stop stack](#stop-stack)
- [Keep volumes (recommended) vs remove volumes (destructive)](#keep-volumes-recommended-vs-remove-volumes-destructive)
- [Nginx and Proxy Considerations](#nginx-and-proxy-considerations)
- [Security Baseline](#security-baseline)
- [Common Deployment Pitfalls](#common-deployment-pitfalls)
- [Troubleshooting](#troubleshooting)
## Environments
Linear Void runs in two separate environments:
| Environment | Host | Docker Target | Purpose |
| -------------- | -------------------------- | -------------------------- | ---------------------------------------------- |
|---|---|---|---|
| **Test Lab** | `192.168.88.40` (internal) | `tcp://192.168.88.40:2375` | Integration testing, staging, CI/CD validation |
| **Production** | Hetzner VPS (TBD) | `tcp://<hetzner-ip>:2375` | Public linearvoid.tv serving real viewers |
| **Production** | Hetzner VPS (TBD) | `tcp://<hetzner-ip>:2375` | Public linearvoid.tv |
**Current state (2026-08-06):** Test lab is active and receives automated CD deploys. Production Hetzner VPS is **not yet provisioned**. The CD workflow (`cd.yml`) and `docker-compose.prod.yml` currently target the test lab. Switching to production requires only changing `DOCKER_HOST` — see [Production Deployment](#production-target-hetzner) below.
**Current state (2026-08-06):** Test lab is active and receives automated CD deploys. Production Hetzner VPS is **not yet provisioned**. `cd.yml` and `docker-compose.prod.yml` target the test lab; switching to production only requires changing `DOCKER_HOST`.
## Production Target (Hetzner)
**Status: Not provisioned.** The production deployment plan is documented and the pipeline is ready — only the VPS needs to be created.
**Status: Not provisioned.** Plan documented and pipeline ready; only the VPS needs creation.
### Planned Infrastructure
| Resource | Spec |
| -------------- | ------------------------------------------------ |
| Provider | Hetzner Cloud |
| Region | Ashburn, VA (us-east) |
| Instance | CX22 (2 vCPU, 4 GB RAM, 40 GB SSD) or equivalent |
| OS | Ubuntu 24.04 LTS or Debian 12 |
| Bandwidth | 20 TB included traffic/month |
| Estimated cost | ~€7–10/month base |
- Provider: Hetzner Cloud, Region: Ashburn, VA (us-east)
- Instance: CX22 (2 vCPU, 4 GB RAM, 40 GB SSD) or equivalent
- OS: Ubuntu 24.04 LTS or Debian 12
- Bandwidth: 20 TB included traffic/month; Estimated cost: ~€7–10/month base
### Cutover Plan
1. Provision Hetzner VPS (see [Production Deployment](#production-target-hetzner) section)
1. Provision Hetzner VPS
2. Add production secrets to Gitea (`CI_PROD_*` prefix)
3. Update `DOCKER_HOST` in CD workflow to Hetzner public IP
4. Trigger deploy → validate health checks
@@ -66,23 +28,17 @@ Linear Void runs in two separate environments:
6. Enable HTTPS, WAF, cache rules on production zone
7. Decommission test lab as public target
### Production Preflight
Before the cutover in the plan above, treat the following as the minimum readiness gate:
1. Verify the Hetzner host is reachable over SSH and Docker remote API TLS is working for the Gitea runner.
2. Confirm all required production secrets are present under the `CI_PROD_*` prefix.
3. Confirm `linearvoid.tv`, `www.linearvoid.tv`, and `media.linearvoid.tv` DNS targets are correct for the intended phase.
4. Complete the manual Cloudflare/UptimeRobot checklist and attach evidence in [Cloudflare Uptimerobot checklist](../docs/operations/cloudflare-uptimerobot-checklist.md).
5. Run the production smoke path end to end: viewer SPA, HLS playback, chat connect, admin login, upload, playlist save, and playout health.
6. Verify rollback is still possible before the final DNS switch.
Readiness is not complete until the checklist above and the manual evidence tracker are both satisfied.
### Production Preflight (minimum readiness gate)
1. Verify Hetzner host reachable over SSH and Docker remote API TLS works for the Gitea runner
2. Confirm all production secrets exist under `CI_PROD_*` prefix
3. Confirm `linearvoid.tv`, `www.linearvoid.tv`, `media.linearvoid.tv` DNS targets match intended phase
4. Complete manual Cloudflare/UptimeRobot checklist (see `docs/operations/cloudflare-uptimerobot-checklist.md`)
5. Run production smoke path end-to-end: viewer SPA, HLS playback, chat connect, admin login, upload, playlist save, playout health
6. Verify rollback possible before final DNS switch
### Firewall / Security Group
| Port | Source | Purpose |
| ---- | --------------------- | ------------------------------ |
|---|---|---|
| 22 | Admin IPs | SSH management |
| 80 | 0.0.0.0/0 | HTTP (redirect to HTTPS) |
| 443 | 0.0.0.0/0 | HTTPS (nginx) |
@@ -91,19 +47,14 @@ Readiness is not complete until the checklist above and the manual evidence trac
| 3001 | Admin IPs or internal | Grafana (or tunnel via SSH) |
### Pipeline Readiness
Three Gitea Actions workflows drive the full pipeline:
| Workflow | Trigger | Purpose |
| -------------------- | ----------------- | ----------------------------------------------------------------------------------------- |
| `ci.yml` | Push to `main` | Lint, build frontend, vet Go, build & push Docker images (`:latest`, `:main`) |
|---|---|---|
| `ci.yml` | Push to `main` | Lint, build frontend, vet Go, build & push images (`:latest`, `:main`) |
| `cd.yml` | Push to `main` | Deploy to test lab (`192.168.88.40:2375`) — runs **in parallel** with CI |
| `release.yml` | Tag push `*.*.*` | Build & push versioned Docker images (`:X.Y.Z`, `:X.Y`, `:latest`), create Gitea Release |
| `workers-deploy.yml` | After CI succeeds | Deploy frontend to Cloudflare Workers (see `docs/operations/gitea-cloudflare-workers.md`) |
| `release.yml` | Tag push `*.*.*` | Build & push versioned images (`:X.Y.Z`, `:X.Y`, `:latest`), create Gitea Release |
| `workers-deploy.yml` | After CI succeeds | Deploy frontend to Cloudflare Workers |
CI and CD run simultaneously on every push to `main`. The deploy pulls whatever `:latest` images are available in the registry at that moment.
The CD workflow (`cd.yml`) uses `DOCKER_HOST` to target the deployment host. To switch targets:
CI and CD run simultaneously on every push to `main`. Deploy pulls whatever `:latest` images are available at that moment. `cd.yml` uses `DOCKER_HOST` to target the host:
```bash
# Test lab (current)
@@ -113,10 +64,9 @@ DOCKER_HOST=tcp://192.168.88.40:2375
DOCKER_HOST=tcp://<hetzner-public-ip>:2375
```
No code changes needed in the pipeline. See `DEPLOYMENT.md` Production Deployment section for full checklist.
No pipeline code changes needed to switch targets.
## Target Topology
- Host: Linux server with Docker Engine
- Reverse proxy: nginx container (linearvoid-nginx)
- Core services: playout, chat, redis, postgres
@@ -124,16 +74,14 @@ No code changes needed in the pipeline. See `DEPLOYMENT.md` Production Deploymen
- CDN/edge: Cloudflare
## Prerequisites
- Docker Engine + Docker Compose plugin
- Network access to git.allucanget.biz
- A valid .env file with production values
- DNS configured for linearvoid.tv (and media.linearvoid.tv if split media domain is used)
- Valid `.env` file with production values
- DNS configured for `linearvoid.tv` (and `media.linearvoid.tv` if split media domain used)
## Environment File
Use `.env` with at least:
`.env` requires at minimum:
- `DATABASE_URL` — PostgreSQL connection string
- `REDIS_URL` — Redis connection string
- `ADMIN_USERNAME` / `ADMIN_PASSWORD` — admin dashboard credentials
@@ -141,178 +89,109 @@ Use `.env` with at least:
- `VITE_PUBLIC_APP_URL` / `VITE_PUBLIC_WS_URL` / `VITE_PUBLIC_HLS_URL` — frontend URLs
**Cloudflare R2 (required for HLS segment sync to edge):**
- `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` — R2 write credentials
- `R2_BUCKET_NAME` — bucket name (e.g. `linearvoid-media`)
- `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY`
- `R2_BUCKET_NAME` (e.g. `linearvoid-media`)
- `R2_ENDPOINT` — S3-compatible endpoint URL
- `R2_REGION` — set to `auto`
**Cloudflare (optional, for Workers):**
- `CLOUDFLARE_API_TOKEN` / `CLOUDFLARE_ACCOUNT_ID`
- `MEDIA_DOMAIN` — split media domain (e.g. `media.linearvoid.tv`)
See `.env.example` and `SETUP.md` for full variable reference.
### Gitea Secrets
`cd.yml` injects sensitive values from Gitea secrets into `.env` at deploy time. Configure in the Gitea repository:
The CD workflow (`cd.yml`) injects sensitive values from Gitea secrets into the `.env` file at deploy time. The following secrets must be configured in the Gitea repository:
| Secret Name | Description |
| ---------------------------- | --------------------------------- |
| `DATABASE_URL` | PostgreSQL connection string |
| `ADMIN_USERNAME` | Admin dashboard username |
| `ADMIN_PASSWORD` | Admin dashboard password |
| `R2_ACCESS_KEY_ID` | Cloudflare R2 access key |
| `R2_SECRET_ACCESS_KEY` | Cloudflare R2 secret key |
| `R2_BUCKET_NAME` | R2 bucket name |
| `R2_ENDPOINT` | R2 S3-compatible endpoint URL |
| `R2_ADMIN_ACCESS_KEY_ID` | R2 admin access key (optional) |
| `R2_ADMIN_SECRET_ACCESS_KEY` | R2 admin secret key (optional) |
| `R2_ADMIN_ENDPOINT` | R2 admin endpoint (optional) |
| `CLOUDFLARE_API_TOKEN` | Cloudflare API token (optional) |
| `CLOUDFLARE_ACCOUNT_ID` | Cloudflare account ID (optional) |
| `CI_REGISTRY_USER` | Gitea container registry username |
| `CI_REGISTRY_TOKEN` | Gitea container registry token |
`DATABASE_URL`, `ADMIN_USERNAME`, `ADMIN_PASSWORD`, `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_BUCKET_NAME`, `R2_ENDPOINT`, `R2_ADMIN_ACCESS_KEY_ID` (optional), `R2_ADMIN_SECRET_ACCESS_KEY` (optional), `R2_ADMIN_ENDPOINT` (optional), `CLOUDFLARE_API_TOKEN` (optional), `CLOUDFLARE_ACCOUNT_ID` (optional), `CI_REGISTRY_USER`, `CI_REGISTRY_TOKEN`
## Pull-and-Run Deployment
Images are built and pushed automatically by the CI and release workflows (see [CONTRIBUTING.md](../CONTRIBUTING.md#versioning-and-releases) for the release procedure).
Every push to `main` triggers CI (builds `:main` and `:latest` tags) and CD (deploys `:latest` to test lab) in parallel. Semantic version tags (`*.*.*`, e.g. `1.0.9`) trigger the release workflow which builds `:X.Y.Z`, `:X.Y`, and `:latest` tags and creates a Gitea Release.
From repository root:
Images are built/pushed automatically by CI and release workflows. Every push to `main` triggers CI (`:main`, `:latest`) and CD (deploys `:latest` to test lab) in parallel. Semantic version tags (`*.*.*`, e.g. `1.0.9`) trigger the release workflow (`:X.Y.Z`, `:X.Y`, `:latest` + Gitea Release).
```bash
docker login git.allucanget.biz
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d --remove-orphans
```
## First-Time or Recovery Provisioning
1. Ensure host paths exist:
```bash
mkdir -p data/chat data/playlist data/media data/hls
```
1. Confirm volumes and networks:
```bash
docker volume ls
docker network ls
```
1. Start stack:
```bash
docker compose -f docker-compose.prod.yml up -d
```
## Health Verification
After startup, check service status:
```bash
docker compose -f docker-compose.prod.yml ps
```
Endpoint checks:
- App: <http://{host}:8080/>
- Nginx health: <http://{host}:8080/health>
- Chat health: <http://{host}:3000/health>
- Playout health: <http://{host}:8081/health>
- Prometheus: <http://{host}:9090>
- Grafana: <http://{host}:3001>
- App: `http://{host}:8080/`
- Nginx health: `http://{host}:8080/health`
- Chat health: `http://{host}:3000/health`
- Playout health: `http://{host}:8081/health`
- Prometheus: `http://{host}:9090`
- Grafana: `http://{host}:3001`
Domain checks:
- <https://linearvoid.tv>
- <https://linearvoid.tv/admin>
- <https://linearvoid.tv/metrics>
- `https://linearvoid.tv`
- `https://linearvoid.tv/admin`
- `https://linearvoid.tv/metrics`
## Zero/Low-Downtime Update Pattern
1. Pull newest images.
1. Recreate changed services only:
2. Recreate changed services only:
```bash
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d nginx chat playout
```
1. Re-run health checks.
3. Re-run health checks.
## Rollback
If new images are unhealthy:
1. Pin previous tags in docker-compose.prod.yml.
1. Pin previous tags in `docker-compose.prod.yml`.
2. Pull pinned tags.
3. Restart impacted services.
3. Restart impacted services:
```bash
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d nginx chat playout
```
## Operational Tasks
### View logs
```bash
# View logs
docker compose -f docker-compose.prod.yml logs -f nginx chat playout
```
### Restart one service
```bash
# Restart one service
docker compose -f docker-compose.prod.yml restart chat
```
### Stop stack
```bash
docker compose -f docker-compose.prod.yml down
```
### Keep volumes (recommended) vs remove volumes (destructive)
```bash
# keep data
# Stop stack (keep data)
docker compose -f docker-compose.prod.yml down
# destructive (remove data)
# Stop stack — destructive (remove data)
docker compose -f docker-compose.prod.yml down -v
```
## Nginx and Proxy Considerations
- If an external proxy (for example Nginx Proxy Manager) fronts the stack, align request body limits with internal nginx, see [nginx.conf](../infrastructure/nginx/nginx.conf).
- For resumable admin uploads, ensure client_max_body_size supports chunk requests in both proxy layers.
- If an external proxy (e.g. Nginx Proxy Manager) fronts the stack, align request body limits with internal nginx (see `infrastructure/nginx/nginx.conf`).
- For resumable admin uploads, ensure `client_max_body_size` supports chunk requests in both proxy layers.
## Security Baseline
- Use strong ADMIN_PASSWORD and rotate regularly.
- Use a strong `ADMIN_PASSWORD` and rotate regularly.
- Keep Docker host access restricted.
- Prefer HTTPS/WSS through Cloudflare.
- Restrict exposed ports if only proxy access is needed.
## Common Deployment Pitfalls
- Missing frontend build artifacts in local-only compose workflows.
- Wrong registry credentials or missing login.
- Misaligned domain env vars (app/ws/hls) causing mixed-origin errors.
## Troubleshooting
| Symptom | Check | Fix |
| -------------------------- | --------------------------------------------- | ------------------------------------------------------------- |
|---|---|---|
| Service not starting | `docker logs linearvoid-<svc> --tail 50` | Check `.env` vars, volume mounts, port conflicts |
| Playout stall / slate mode | `curl http://<host>:8081/health` — mode=slate | Check playlist file, media paths, disk space |
| Chat not connecting | `curl http://<host>:3000/health` | Check Redis: `docker exec linearvoid-redis redis-cli ping` |
@@ -322,5 +201,4 @@ docker compose -f docker-compose.prod.yml down -v
| Prometheus no targets | `curl http://<host>:9090/targets` | Verify container network, scrape config |
| Grafana no data | Check datasource | Verify Prometheus URL `http://prometheus:9090` provisioned |
- Proxy-layer body size limits returning 413 on upload.
- Health checks failing before service warmup is complete.
Additional notes: proxy-layer body size limits can return 413 on upload; health checks may fail before service warmup completes.
docs/MONITORING.md
+4 -12
View File
@@ -52,7 +52,7 @@ Config: `monitoring/prometheus/prometheus.yml` (15s scrape interval)
- **Linear Void app**: `monitoring/grafana/dashboard.json` — clients, messages, moderation, Redis, sessions, scrape health
- **Prometheus ops**: `monitoring/grafana/dashboards/dashboard.json` — TSDB, CPU/memory, WAL, compactions, query latency
## Metrics checking
## Metrics Checking
See detailed scripts for querying Prometheus scrape targets:
@@ -78,21 +78,13 @@ See detailed scripts for querying Prometheus scrape targets:
| Prometheus targets down | Check container network: all must be on `linearvoid-net` |
| Alert not firing | Check Prometheus → Alerts tab, verify `rule_files` in prometheus.yml |
### Grafana shows no data
**Grafana no data**: ensure datasource points to `http://prometheus:9090`; confirm Prometheus is healthy; check dashboard query labels match active jobs.
- Ensure datasource points to <http://prometheus:9090>.
- Confirm Prometheus is healthy.
- Check dashboard query labels match active jobs.
### High cardinality or query slowness
- Reduce unneeded labels.
- Limit expensive dashboard panels.
- Tune retention and query windows.
**High cardinality / slow queries**: reduce unneeded labels; limit expensive panels; tune retention and query windows.
## Operational Recommendations
- Keep alert thresholds tied to actual user impact.
- Review dashboards after each release that changes endpoints or metrics names.
- Periodically test alert routing and incident response flow.
- Metrics retention is set to **30 days** time-based plus a **15GB** size cap (`--storage.tsdb.retention.time=30d --storage.tsdb.retention.size=15GB` on the `prometheus` service in `docker-compose.yml` and `docker-compose.prod.yml`). Increase the size cap before extending the time window so the TSDB does not hit the disk limit first.
- Retention: **30 days** time-based plus **15GB** size cap (`--storage.tsdb.retention.time=30d --storage.tsdb.retention.size=15GB` on the `prometheus` service in `docker-compose.yml` and `docker-compose.prod.yml`). Increase the size cap before extending the time window so the TSDB does not hit the disk limit first.
docs/SETUP.md
+71 -494
View File
@@ -2,572 +2,149 @@
**Last Updated**: 2026-08-03
This guide covers prerequisites, environment setup, running the application locally and in production, and troubleshooting. It references the [architecture documentation](./docs/architecture/README.md)(`docs/architecture/README.md`).
Prerequisites, environment setup, and running the app locally and in production. Architecture: [`docs/architecture/README.md`](./architecture/README.md).
## Prerequisites
- **Docker & Docker Compose**: Required for running the full stack (playout, chat, nginx, redis, db, prometheus, grafana). Install from [docker.com](https://www.docker.com).
- **Node.js 22+**: Required for frontend development and production builds (`cd frontend && npm install && npm run build`).
- **npm**: Installed with Node.js and used for the frontend dependency installation and Vite workflow.
- **Go 1.25+**: Recommended when working on the Go services outside Docker (`backend/chat`, `backend/playout`).
- **FFmpeg**: Used by the playout service for HLS generation. Install via `apt install ffmpeg` (Linux), Homebrew (macOS), or Winget (Windows).
- **Git**: For cloning and version control.
- **Cloudflare Account**: For R2 storage and CDN (free tier sufficient for most use cases).
- **Gitea Account**: Required if you want to push images to the self-hosted registry or use the repository's CI/CD flow.
- Set up the required Actions secrets in Gitea at: [https://git.allucanget.biz/zwitschi/linearvoid/settings/actions/secrets](https://git.allucanget.biz/zwitschi/linearvoid/settings/actions/secrets)
- Add the following secrets:
- `CI_REGISTRY_USER`: Your Gitea username (e.g., `zwitschi`)
- `CI_REGISTRY_TOKEN`: A Gitea Personal Access Token with `write:packages` scope
- `ADMIN_USERNAME`: The administrator username for the production admin panel (e.g., `admin`)
- `ADMIN_PASSWORD`: A strong password for the production admin panel
- `DATABASE_URL`: The production database connection string (e.g., `postgres://linearvoid:<password>@db:5432/linearvoid?sslmode=disable`)
- **Hetzner Cloud Account** (optional for production): For the US-East VPS (Ashburn, VA recommended per ADR-001).
- **Docker + Docker Compose** — runs the full stack (playout, chat, nginx, redis, db, prometheus, grafana).
- **Node.js 22+ / npm** — frontend dev and builds (`cd frontend && npm install && npm run build`).
- **Go 1.25+** — working on the Go services outside Docker.
- **FFmpeg** — playout HLS generation (`apt install ffmpeg`).
- **Git**.
- **Cloudflare account** — R2 storage + CDN.
- **Gitea account** — image registry + CI/CD. Set Actions secrets at `https://git.allucanget.biz/zwitschi/linearvoid/settings/actions/secrets`: `CI_REGISTRY_USER`, `CI_REGISTRY_TOKEN` (scope `write:packages`), `ADMIN_USERNAME`, `ADMIN_PASSWORD`, `DATABASE_URL`.
- **Hetzner Cloud (optional, prod)** — US-East VPS (Ashburn, VA per ADR-001).
## Network Environments
Linear Void operates in two distinct environments. Know which one you're targeting.
Two distinct targets; know which one you're using.
```mermaid
flowchart LR
subgraph Test_Lab["Test Lab (192.168.88.0/24)"]
TEST_HOST["192.168.88.40<br/>Docker Engine :2375"]
end
subgraph Prod["Production (Hetzner — NOT provisioned)"]
PROD_HOST["Hetzner CX22 Ashburn, VA<br/>Public IP TBD"]
end
subgraph Gitea["Gitea CI/CD"]
CI["CI: build + push images"]
CD["CD: deploy via DOCKER_HOST"]
end
subgraph CF["Cloudflare"]
DNS["linearvoid.tv"]
R2["R2 Bucket"]
end
CD -->|"DOCKER_HOST=tcp://192.168.88.40:2375"| TEST_HOST
CD -.->|"DOCKER_HOST=tcp://&lt;hetzner-ip&gt;:2375<br/>(when provisioned)"| PROD_HOST
CD -.->|"DOCKER_HOST=tcp://&lt;hetzner-ip&gt;:2375 (when provisioned)"| PROD_HOST
TEST_HOST -->|"test traffic"| DNS
PROD_HOST -.->|"production traffic"| DNS
TEST_HOST --> R2
PROD_HOST -.-> R2
```
**Test Lab (active):** Internal network `192.168.88.0/24`, host `192.168.88.40`. Receives automated CD deploys from Gitea Actions. Used for integration testing and staging validation.
**Production (pending):** Hetzner Cloud VPS, Ashburn VA. Not yet provisioned. CD workflow and compose files are ready — only `DOCKER_HOST` needs changing. See Production Deployment section in [DEPLOYMENT.md](./DEPLOYMENT.md) for cutover plan.
## Development Prerequisites
- **Frontend dependencies**: Run `cd frontend && npm install` before `npm run dev` or `npm run build`.
- **Docker build support**: Local development expects Docker to be able to build the `chat`, `playout`, and `nginx` images referenced in `docker-compose.yml`.
- **Data directories**: Create `data/playlist`, `data/media`, `data/hls`, and `data/chat` before the first run so bind mounts resolve cleanly.
- **Database choice**: The stack defaults to PostgreSQL in `docker-compose.yml`. If you want SQLite for local-only development, update `DATABASE_URL` in `.env` accordingly.
- **Redis availability**: Chat presence, rate limiting, and pub/sub require Redis. The compose stack provides it automatically.
- **Video inputs**: Source videos should be normalized before playout. If the playlist contains invalid files, the playout container will fail its validation flow.
- **Test Lab (active):** internal `192.168.88.0/24`, host `192.168.88.40`; receives automated CD deploys from Gitea Actions. Used for staging validation.
- **Production (pending):** Hetzner CX22, Ashburn VA — not yet provisioned. Compose/CD are ready; only `DOCKER_HOST` needs changing. See [DEPLOYMENT.md](./DEPLOYMENT.md).
## Environment Setup
1. **Clone the Repository** (if not already done):
```bash
git clone https://git.allucanget.biz/zwitschi/linearvoid.git
cd linearvoid
```
2. **Copy Environment File**:
```bash
cp .env.example .env
```
3. **Configure `.env`** (copy from `.env.example` and replace placeholders)
4. **Check local value overrides for Docker Compose**:
- `docker-compose.yml` builds images locally and maps the frontend build output from `frontend/dist` into nginx.
- `docker-compose.prod.yml` pulls prebuilt images from the Gitea registry.
- Internal container-to-container URLs should use compose hostnames, for example `REDIS_URL=redis://redis:6379/0` and `DATABASE_URL=postgres://linearvoid:<password>@db:5432/linearvoid?sslmode=disable`.
- For production public routing, set `VITE_PUBLIC_APP_URL=https://linearvoid.tv`, `VITE_PUBLIC_WS_URL=wss://linearvoid.tv/ws`, and `VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8`.
5. **Create Data Directories**:
```bash
mkdir -p data/playlist data/media data/hls data/chat
```
6. **Prepare Sample Videos** (optional):
- Place normalized MP4 files in `data/media/`.
- Create a `data/playlist/playlist.txt` with paths (see ADR-001 for normalization requirements).
1. `git clone https://git.allucanget.biz/zwitschi/linearvoid.git && cd linearvoid`
2. `cp .env.example .env` then replace placeholders.
3. Compose overrides: `docker-compose.yml` builds images locally and mounts `frontend/dist` into nginx; `docker-compose.prod.yml` pulls prebuilt images from Gitea.
- Internal URLs use compose hostnames: `REDIS_URL=redis://redis:6379/0`, `DATABASE_URL=postgres://linearvoid:<pw>@db:5432/linearvoid?sslmode=disable`.
- Public routing: `VITE_PUBLIC_APP_URL=https://linearvoid.tv`, `VITE_PUBLIC_WS_URL=wss://linearvoid.tv/ws`, `VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8`.
4. Create data dirs: `mkdir -p data/playlist data/media data/hls data/chat`
5. Optional: place normalized MP4s in `data/media/` and create `data/playlist/playlist.txt` (see ADR-001 for normalization).
## Gitea Setup
1. **Create or verify a Gitea user**:
- The user should have access to the repository and permission to read from the container registry.
2. **Create an access token**:
- Generate a token with the scopes needed for package/registry access and, if applicable, repository automation.
- Store it as `CI_REGISTRY_TOKEN` in `.env`.
3. **Configure Gitea Actions Secrets**:
- Go to the repository settings page: [https://git.allucanget.biz/zwitschi/linearvoid/settings/actions/secrets](https://git.allucanget.biz/zwitschi/linearvoid/settings/actions/secrets)
- Add the following secrets to enable automated CI/CD and production deployment:
- `CI_REGISTRY_USER`: Your Gitea username (e.g., `zwitschi`).
- `CI_REGISTRY_TOKEN`: Your Gitea Personal Access Token with `write:packages` scope.
- `ADMIN_USERNAME`: The administrator username for the production admin panel (e.g., `admin`).
- `ADMIN_PASSWORD`: A strong password for the production admin panel.
- `DATABASE_URL`: The production database connection string (e.g., `postgres://linearvoid:<password>@db:5432/linearvoid?sslmode=disable`).
4. **Configure registry variables**:
- Set `CI_REGISTRY_URL` to the Gitea host.
- Set `CI_REGISTRY_NAMESPACE` to the image namespace, for example `zwitschi/linearvoid`.
- Set `CI_REGISTRY_USER` to the account that will push or pull images.
5. **Authenticate Docker to the registry**:
```bash
docker login git.allucanget.biz
```
6. **Build and publish images when needed**:
- Local development uses `docker-compose.yml`, which builds from source.
- Production uses `docker-compose.prod.yml`, which expects the `playout`, `chat`, and `nginx` images to exist in the configured Gitea registry namespace.
7. **Validate image availability**:
- Before a production rollout, verify that the expected `:latest` tags can be pulled from the registry on the target server.
1. User with repo + registry access; generate a token (scope `write:packages`) stored as `CI_REGISTRY_TOKEN`.
2. Set Actions secrets (see Prerequisites) to enable automated CI/CD and deploys.
3. Set registry vars: `CI_REGISTRY_URL`, `CI_REGISTRY_NAMESPACE` (e.g. `zwitschi/linearvoid`), `CI_REGISTRY_USER`.
4. Auth: `docker login git.allucanget.biz`.
5. Local dev builds from source; prod pulls `playout`, `chat`, `nginx` images from the registry namespace. Verify `:latest` tags exist before rollout.
## Cloudflare R2 Setup
1. **Create an R2 bucket**:
- Create a bucket such as `linearvoid-media` in Cloudflare R2.
- Keep the bucket name aligned with `R2_BUCKET_NAME` in `.env`.
2. **Generate R2 API credentials**:
- Create an access key pair with read/write permissions for the bucket.
- Set `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, and `R2_REGION=auto` in `.env`.
3. **Find the S3 endpoint**:
- Use the account-scoped endpoint format `https://<account-id>.r2.cloudflarestorage.com`.
- Store the account ID in both `R2_ENDPOINT` and `CLOUDFLARE_ACCOUNT_ID` where applicable.
4. **Bind a custom media domain**:
- Attach a domain such as `media.linearvoid.tv` to the bucket.
- Set `MEDIA_DOMAIN` to that hostname.
5. **Configure app-domain DNS and TLS**:
- Point `linearvoid.tv` to the public entrypoint serving nginx.
- Point `www.linearvoid.tv` to the same target and redirect it to `https://linearvoid.tv`.
- Keep `media.linearvoid.tv` on the R2 custom domain path so HLS delivery stays split from the app origin.
- Use Cloudflare proxied DNS with Full (strict) TLS so browser traffic uses `https://linearvoid.tv`, `wss://linearvoid.tv/ws`, and `https://media.linearvoid.tv`.
6. **Apply cache behavior**:
- Configure the `.m4s` and `.m3u8` cache rules from [cache-rules](../docs/infrastructure/cloudflare/cache-rules.md).
- Enable Smart Tiered Cache for the R2-backed hostname.
- If you need header normalization for immutable segment caching, deploy `infrastructure/cloudflare/worker.js` to the same zone.
7. **Configure R2 CORS**:
- Add CORS rules to allow the app domain to access the R2 bucket for HLS delivery.
- Example CORS configuration:
```json
[
{
"AllowedOrigins": ["https://linearvoid.tv"],
"AllowedMethods": ["GET"],
"AllowedHeaders": ["*"],
"ExposeHeaders": ["ETag", "Cache-Control"],
"MaxAgeSeconds": 3600
}
]
```
8. **Verify access**:
- Confirm uploads with an S3-compatible client such as `aws s3 ls --endpoint-url <R2_ENDPOINT>`.
- After deployment, check Cloudflare analytics for cache hit ratio and origin request volume.
## Cloudflare
Use this section for full edge setup (DNS, Workers, cache, security, and monitoring).
Execution tracker: `docs/operations/cloudflare-uptimerobot-checklist.md`.
### 1. DNS Setup
1. Add apex record for `linearvoid.tv` pointing to the Cloudflare Worker.
2. Add `www` CNAME to the Cloudflare Worker and configure redirect to `https://linearvoid.tv`.
3. Add `media.linearvoid.tv` for split-domain HLS delivery.
4. Keep all records proxied (orange cloud) unless troubleshooting requires DNS-only mode.
### 2. R2 Bucket
1. Create bucket `linearvoid-media`.
2. Set location hint to `enam` (US-East).
3. Attach custom domain `media.linearvoid.tv` to the bucket.
4. Configure CORS rules to allow `https://linearvoid.tv` for HLS delivery.
5. Verify object read works from the custom domain.
### 3. Cache Rules
Configure Cloudflare Cache Rules:
1. HLS segment files (`*.ts`, `*.m4s`):
- Cache: eligible
- Edge TTL: `31536000`
- Browser TTL: long-lived
- Cache key: include full path
2. HLS manifests (`*.m3u8`):
- Cache: eligible
- Edge TTL: `2s`
- Revalidation: enabled
3. Static assets (`*.js`, `*.css`):
- Cache: eligible
- Edge TTL: `1y`
- Browser TTL: long-lived
- Cache key: include full path
4. Validate behavior using response headers:
- `Cf-Cache-Status` should move from `MISS` to `HIT` for hot objects
Reference: [cache-rules](../docs/infrastructure/cloudflare/cache-rules.md).
### 4. Cloudflare Workers
Also see [the automated Gitea Actions deployment guide](./operations/gitea-cloudflare-workers.md).
Manual setup:
1. Create a Cloudflare Worker for the static SPA (Workers & Pages → Create → Worker).
2. Build the frontend:
```bash
cd frontend && npm run build
```
3. Bind the build output (`frontend/dist`) as Worker static assets.
4. Configure production domain `linearvoid.tv` on the Worker.
### 5. Tiered Cache
1. Enable Smart Tiered Cache in Cloudflare dashboard.
2. Warm a segment and manifest path.
3. Confirm tiered behavior through repeated requests and `Cf-Cache-Status: HIT`.
### 6. WAF and Bot Protection
1. Enable OWASP Managed Ruleset.
2. Start in `Simulate` mode.
3. Review events for 48 hours.
4. Switch to `Block` once false positives are addressed.
5. Enable Bot Fight Mode.
### 7. Rate Limiting
Create rules:
1. Path: `/api/admin/login`
2. Path: `/api/admin/*`
3. Threshold: `10 requests / 10 seconds / IP`
4. Action: `Managed Challenge`
### 8. HTTPS and WSS
1. Enable `Always Use HTTPS`.
2. Enable HSTS with:
- `max-age=31536000`
- `includeSubDomains`
- `preload`
3. Minimum TLS version: `1.2`.
4. Keep SSL mode at `Full (strict)` when origin certs are valid.
5. Enforce HTTPS redirect at both layers:
- Cloudflare edge
- nginx origin
6. Validate websocket endpoint uses `wss://linearvoid.tv/ws`.
### 9. UptimeRobot Monitors
Create external monitors:
1. HTTP monitor:
- URL: `https://linearvoid.tv/health`
- Interval: 5 minutes
2. Keyword monitor:
- URL: `https://linearvoid.tv/hls/playlist.m3u8`
- Keyword: `#EXTM3U`
- Interval: 5 minutes
3. Alert policy:
- Trigger on 2 consecutive failures
- Notify via email and Discord/Slack webhook
### 10. Verification Checklist
1. DNS resolves for `linearvoid.tv`, `www.linearvoid.tv`, and `media.linearvoid.tv`.
2. Workers serve frontend for apex and `www` redirects correctly.
3. HLS segments are cacheable long-term, manifests refresh every ~2s.
4. Admin paths are rate-limited and WAF events are visible.
5. HTTPS and WSS both succeed with valid certificates.
6. UptimeRobot alerts fire as expected when endpoints are intentionally made unavailable.
1. Create R2 bucket (e.g. `linearvoid-media`, matching `R2_BUCKET_NAME`); generate an access key pair → `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_REGION=auto`.
2. S3 endpoint: `https://<account-id>.r2.cloudflarestorage.com` → `R2_ENDPOINT`.
3. Attach `media.linearvoid.tv` as custom media domain → `MEDIA_DOMAIN`; add CORS allowing `https://linearvoid.tv` GET for HLS delivery.
4. Point `linearvoid.tv` (+ `www` redirect) at the app entrypoint; keep `media.*` on R2. Proxied DNS, TLS **Full (strict)**.
5. Apply the `.m4s`/`.m3u8` cache rules from [`cache-rules.md`](./infrastructure/cloudflare/cache-rules.md); enable Smart Tiered Cache; optionally deploy `infrastructure/cloudflare/worker.js` for header normalization.
6. Verify: `aws s3 ls --endpoint-url $R2_ENDPOINT`; check cache hit ratio after serving traffic.
## Running the Application
### Local Development
1. **Start All Services**:
### Local development
```bash
docker compose up --build
docker compose up --build # start all services
docker compose down # stop
```
2. **Access the Application**:
- Preferred local access with the documented proxy setup: <https://linearvoid.tv>
- Admin Interface: <https://linearvoid.tv/admin> (redirects to `/admin/login/` until authenticated)
- Chat WebSocket: `wss://linearvoid.tv/ws`
- Direct-local fallback when bypassing Cloudflare/Nginx Proxy Manager: <http://localhost>, <http://localhost/admin>, and `ws://localhost/ws`
- Health Checks:
- Playout: <http://localhost:8081/health>
- Chat: <http://localhost:3000/health>
- Monitoring:
- Prometheus: <http://localhost:9090>
- Grafana: <http://localhost:3001> (default login: admin/admin)
Access: app/admin `https://linearvoid.tv` (via proxy) or `http://localhost`; WS `wss://linearvoid.tv/ws` / `ws://localhost/ws`. Health: playout `:8081/health`, chat `:3000/health`. Monitoring: Prometheus `:9090`, Grafana `:3001` (default `admin/admin`).
3. **Frontend Development** (optional, for UI changes):
Frontend dev (optional): `cd frontend && npm install && npm run dev` (http://localhost:5173, proxies API). For nginx-backed local runs: `npm run build` (nginx serves `frontend/dist`).
```bash
cd frontend
npm install
npm run dev
```
### Test lab (remote Docker host)
The dev server runs on <http://localhost:5173>. It proxies API calls to the backend services.
Deploy to `192.168.88.40` over Docker's TCP socket (port `2375`).
4. **Build frontend assets for nginx-backed local runs**:
```bash
cd frontend
npm install
npm run build
```
The local nginx container serves `frontend/dist`, so rebuilding the frontend is required before validating production-like static output.
5. **Stop Services**:
```bash
docker compose down
```
### Local Test Environment Deployment (Remote Docker Host)
For testing deployments on a local network server without setting up a full production VPS, you can deploy directly to a remote Docker host (e.g., a home lab server or test machine) using Docker's TCP socket.
#### Test Environment Prerequisites
- A remote server running Docker at IP `192.168.88.40` (running Alpine Linux).
- Docker TCP socket enabled and listening on port `2375` on the remote server.
##### Enabling Docker TCP Socket on Alpine Linux
To allow the Gitea Actions runner (or your local machine) to deploy to the remote Alpine host, you must configure the Docker daemon to listen on the TCP socket:
1. Edit `/etc/conf.d/docker` on the Alpine host:
```bash
vi /etc/conf.d/docker
```
2. Update the `DOCKER_OPTS` line to include both the unix socket and the TCP socket:
Enable TCP on Alpine host — edit `/etc/conf.d/docker`:
```bash
DOCKER_OPTS="-H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375"
```
3. Restart the Docker service:
Then `rc-service docker restart`; verify with `netstat -tlnp | grep 2375`; allow port `2375` from `192.168.88.0/24` in the firewall.
```bash
rc-service docker restart
```
4. Verify that Docker is listening on port `2375`:
```bash
netstat -tlnp | grep 2375
```
5. Ensure any firewall (like `awall` or `iptables`) on the Alpine host allows incoming traffic on port `2375` from your local network subnet (e.g., `192.168.88.0/24`).
#### Deployment Steps
1. **Configure Docker CLI to use the Remote Host**:
You can point your local Docker CLI to the remote server by setting the `DOCKER_HOST` environment variable:
- **PowerShell (Windows)**:
```powershell
$env:DOCKER_HOST="tcp://192.168.88.40:2375"
```
- **Bash (Linux/macOS)**:
Deploy from a machine with the Docker CLI:
```bash
export DOCKER_HOST="tcp://192.168.88.40:2375"
```
2. **Verify Connection**:
Run `docker info` to confirm your local CLI is communicating with the remote server at `192.168.88.40`:
```bash
docker info
```
The output should show the remote server's hostname and system details.
3. **Deploy the Stack**:
With `DOCKER_HOST` set, any `docker` or `docker compose` commands you run locally will execute directly on the remote server.
- **Deploy from Registry (Production-like)**:
```bash
docker info # confirm remote host
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d --remove-orphans
docker compose -f docker-compose.prod.yml up -d --remove-orphans # registry images
# or build from source: docker compose up -d --build
unset DOCKER_HOST # back to local
```
- **Build and Deploy from Source**:
Health: `http://192.168.88.40:8081/health` (playout), `:3000/health` (chat), nginx entrypoint `:8080`.
```bash
docker compose up -d --build
```
### Coolify deployment
4. **Access the Test Environment**:
- Playout Health: `http://192.168.88.40:8081/health`
- Chat Health: `http://192.168.88.40:3000/health`
- Nginx Entrypoint: `http://192.168.88.40:8080` (or the port mapped to `NGINX_PORT` in `.env`)
Coolify replaces the Gitea CD `DOCKER_HOST` orchestration (Cloudflare DNS/R2/cache unchanged). Install (`curl -fsSL https://cdn.coollabs.io/coolify/install.sh | bash`), add the target server via SSH, create a Docker Compose resource from `docker-compose.prod.yml` (or the repo), configure env vars in Coolify's store, map persistent `data/` and named volumes, and assign `linearvoid.tv` to nginx for Traefik TLS. Full walkthrough: [`COOLIFY.md`](./COOLIFY.md).
5. **Reset to Local Docker Host**:
Once testing is complete, clear the `DOCKER_HOST` variable to point back to your local machine:
- **PowerShell**:
### Production setup
```powershell
Remove-Item env:DOCKER_HOST
```
1. Provision a Linux VPS (Docker, Compose, Git); open HTTP/HTTPS ports.
2. `git clone … && cd linearvoid && cp .env.example .env`.
3. Set production `.env`: strong `POSTGRES_PASSWORD`/`GRAFANA_PASSWORD`, compose-hostname internal URLs, `MEDIA_DOMAIN=media.linearvoid.tv`, public `VITE_PUBLIC_*` URLs. For local-only testing bypassing the proxy, use `http://localhost`/`ws://localhost/ws`/`http://localhost/hls/playlist.m3u8`.
4. `mkdir -p data/playlist data/media data/hls data/chat`.
5. Ensure `playout`, `chat`, `nginx` images exist in the registry; `docker login` from the VPS.
6. Deploy: `docker compose -f docker-compose.prod.yml pull && docker compose -f docker-compose.prod.yml up -d`.
7. Verify: `docker compose -f docker-compose.prod.yml ps`, health endpoints, HLS written to `data/hls`, media domain serves via Cloudflare.
8. Harden: HTTPS in front of nginx (Cloudflare, `www` → apex), restrict Grafana/Prometheus to trusted IPs, run under systemd for auto-start.
- **Bash**:
### Production (Hetzner VPS + Cloudflare)
```bash
unset DOCKER_HOST
```
1. Provision CX22 in Ashburn, VA (ADR-001).
2. Cloudflare: R2 bucket (`enam` hint), `media.linearvoid.tv` custom domain + cache rules, `linearvoid.tv`/`www` at the app entrypoint with `www`→apex redirect, Full (strict) TLS, deploy `infrastructure/cloudflare/worker.js`.
3. Deploy from the project: `docker compose -f docker-compose.prod.yml up -d` (or pull from Gitea registry); manage under systemd.
4. Monitoring: Grafana dashboards from `monitoring/grafana/`.
### Coolify Deployment
Coolify is an open-source, self-hostable PaaS (Heroku/Netlify/Vercel
alternative) that can deploy the Linear Void stack through a web UI with
automatic HTTPS and one-click redeploys. It replaces the deployment
orchestration layer (the Gitea CD `DOCKER_HOST` flow) while leaving Cloudflare
DNS, R2 media, and cache rules unchanged.
To deploy via Coolify:
1. **Install Coolify** on a dedicated server:
```bash
curl -fsSL https://cdn.coollabs.io/coolify/install.sh | bash
```
2. **Add the target server** in the dashboard via SSH (or use Localhost if
Coolify runs on the same machine as the stack).
3. **Create a Docker Compose resource** pointing at either the Git repository
(`docker-compose.prod.yml`) or the compose file contents directly.
4. **Configure environment variables** in Coolify's secret store (the values
normally held in `.env`), keeping internal URLs on compose hostnames
(`redis`, `db`).
5. **Map persistent storage** for `data/` mounts and the `redis-data` /
`postgres-data` named volumes so data survives redeploys.
6. **Assign a domain** to the nginx service so Coolify's Traefik proxy
terminates TLS for `linearvoid.tv`, while `media.linearvoid.tv` continues to
serve HLS from Cloudflare R2.
For the full walkthrough, environment-variable table, verification steps, and
cutover notes, see [COOLIFY.md](./COOLIFY.md).
### Production Setup Guide
1. **Prepare the target host**:
- Provision a Linux VPS with Docker Engine, Docker Compose, Git, and enough disk for media, HLS output, PostgreSQL, Redis persistence, and monitoring data.
- Open the ports required for HTTP/HTTPS and any restricted monitoring access you intend to expose.
2. **Clone the repository and create the environment file**:
```bash
git clone https://git.allucanget.biz/zwitschi/linearvoid.git
cd linearvoid
cp .env.example .env
```
3. **Set production-safe `.env` values**:
- Replace placeholder secrets.
- Use compose-network hostnames for internal services, for example `redis` and `db`.
- Set strong values for `POSTGRES_PASSWORD` and `GRAFANA_PASSWORD`.
- Keep `MEDIA_DOMAIN`, R2 credentials, and Gitea registry values consistent with the external infrastructure.
- Set `MEDIA_DOMAIN=media.linearvoid.tv`.
- Set `VITE_PUBLIC_APP_URL=https://linearvoid.tv`, `VITE_PUBLIC_WS_URL=wss://linearvoid.tv/ws`, and `VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8`.
- If you intentionally bypass the public proxy for local-only testing, switch those values to `http://localhost`, `ws://localhost/ws`, and `http://localhost/hls/playlist.m3u8`.
4. **Create runtime directories on the server**:
```bash
mkdir -p data/playlist data/media data/hls data/chat
```
5. **Publish or verify required images in Gitea**:
- Ensure the `playout`, `chat`, and `nginx` images exist under `${CI_REGISTRY_URL}/${CI_REGISTRY_NAMESPACE}`.
- Log in to the registry from the VPS before pulling.
6. **Deploy the production stack**:
```bash
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d
```
7. **Run post-deploy verification**:
- Check `docker compose -f docker-compose.prod.yml ps`.
- Confirm nginx, chat, and playout health endpoints respond.
- Verify that HLS output is being written to `data/hls` and that the media domain serves manifests and segments through Cloudflare.
8. **Harden and operate the deployment**:
- Put HTTPS in front of nginx using Cloudflare or another reverse proxy, with `linearvoid.tv` as canonical app host and `www.linearvoid.tv` redirected.
- Restrict Grafana and Prometheus exposure to trusted IPs or a private network.
- Run the stack under systemd or another service manager if you need auto-start behavior outside Docker restart policies.
### Production Deployment (Hetzner VPS + Cloudflare)
1. **Provision VPS**: Use Hetzner Cloud CX22 in Ashburn, VA (per ADR-001).
2. **Configure Cloudflare**:
- Create R2 bucket with `enam` location hint.
- Set up custom media domain (`media.linearvoid.tv`) with the cache rules from [cache-rules](../docs/infrastructure/cloudflare/cache-rules.md).
- Point `linearvoid.tv` and `www.linearvoid.tv` at the app entrypoint, then redirect `www` to the apex domain.
- Use Full (strict) TLS for both app and media hostnames.
- Apply the Worker from `infrastructure/cloudflare/worker.js` for advanced header management.
3. **Deploy**:
- Copy the project to the VPS.
- Use production compose: `docker compose -f docker-compose.prod.yml up -d`
- Or pull from Gitea registry: `docker compose -f docker-compose.prod.yml pull && docker compose -f docker-compose.prod.yml up -d`
- Configure systemd to manage the Docker Compose stack.
4. **Monitoring**: Access Grafana at the exposed port for dashboards (included in `monitoring/grafana/dashboard.json`).
See `docs/architecture/README.md` for the full arc42 documentation and the ADRs for architectural decisions.
See `docs/architecture/README.md` (arc42) and the ADRs for decisions and cost/scaling guidance (ADR-001).
## Troubleshooting
- **R2 Upload Fails**: Check credentials in `.env` and ensure the bucket exists. Verify with `aws s3 ls --endpoint-url=$R2_ENDPOINT`.
- **Normalization Errors**: Run `backend/playout/start-playout.sh` manually to see ffprobe output. All videos must be H.264/AAC, 30 fps constant, 2s keyframes (ADR-001).
- **Chat Not Connecting**: Check Redis connection and WebSocket URL. The health check at `/health` should return 200.
- **Cache Not Working**: Verify Cloudflare cache rules (`.m4s` = long TTL immutable, `.m3u8` = short TTL). Check cache hit ratio in Cloudflare dashboard.
- **Docker Issues**: Run `docker compose logs` for service-specific errors. Ensure ports are not in use.
- **Admin Interface**: Ensure `ADMIN_USERNAME`, `ADMIN_PASSWORD`, and `ADMIN_SESSION_TTL_HOURS` are set. `/admin/` and `/api/admin/*` are protected by a session-based login flow.
- **Production Pull Fails**: Confirm `docker login` to Gitea succeeded on the server and the image tags referenced by `docker-compose.prod.yml` exist.
- **R2 upload fails** — check `.env` creds/bucket; `aws s3 ls --endpoint-url=$R2_ENDPOINT`.
- **Normalization errors** — run `backend/playout/start-playout.sh` to see ffprobe output. Video must be H.264/AAC, 30 fps constant, 2s keyframes (ADR-001).
- **Chat won't connect** — check Redis and WS URL; `/health` should return 200.
- **Cache not working** — verify `.m4s` long-TTL / `.m3u8` short-TTL cache rules and hit ratio.
- **Docker issues** — `docker compose logs`; check port conflicts.
- **Admin interface** — `ADMIN_USERNAME`, `ADMIN_PASSWORD`, `ADMIN_SESSION_TTL_HOURS` set; session-based login protects `/admin/` and `/api/admin/*`.
- **Prod pull fails** — `docker login` to Gitea succeeded and image tags exist.
## Further Reading
- Architecture: `docs/architecture/README.md` (arc42 template with diagrams).
- Decisions: `docs/architecture/adr/` (ADR-001 to ADR-007).
- Contributing: `CONTRIBUTING.md`.
- Project Structure: Root `README.md`.
The system is ready for use. For production, review ADR-001 for cost estimates and scaling guidance.
Architecture: `docs/architecture/README.md` · Decisions: `docs/architecture/adr/` (ADR-001…007) · Contributing: `CONTRIBUTING.md` · Overview: root `README.md`.
docs/STREAMING.md
+59 -72
View File
@@ -1,6 +1,6 @@
# Streaming Pipeline
This document is the operational contract for Linear Void video ingest, playout, publication, caching, and browser playback. General deployment, chat, and admin details remain in `DEPLOYMENT.md`, `SETUP.md`, and `API.md`.
Operational contract for Linear Void video ingest, playout, publication, caching, and browser playback. Deployment, chat, admin details live in `DEPLOYMENT.md`, `SETUP.md`, `API.md`.
## Production flow
@@ -20,7 +20,7 @@ The playout service is the timeline authority. It reads the ordered playlist, ru
https://media.linearvoid.tv/hls/playlist.m3u8
```
Nginx `/hls/` is a local/test fallback, not the production media origin. The SPA, admin API, and chat use `linearvoid.tv`; media uses `media.linearvoid.tv`.
Nginx `/hls/` is a local/test fallback, not the production origin. The SPA, admin API, and chat use `linearvoid.tv`; media uses `media.linearvoid.tv`.
## Source-media contract
@@ -30,12 +30,12 @@ Only validated files may enter the production playlist.
| ---------- | ------------------------------------------------------------------------ |
| Container | MP4 |
| Video | H.264/AVC, `yuv420p` |
| Resolution | 1280x720 or 1920x1080; avoid changing resolution within a playlist block |
| Resolution | 1280x720 or 1920x1080; avoid resolution changes within a playlist block |
| Frame rate | Constant 30 fps |
| GOP | Closed, fixed 2 seconds: `-g 60 -keyint_min 60 -sc_threshold 0` |
| Audio | AAC-LC, stereo, 48 kHz, 128–192 kbit/s |
The ingest gate uses `ffprobe` to reject incompatible codecs, variable frame rate, missing audio, invalid dimensions, or incorrect keyframe cadence. Assets are immutable after acceptance; corrections create a new asset ID and object key.
The ingest gate uses `ffprobe` to reject incompatible codecs, variable frame rate, missing audio, invalid dimensions, or wrong keyframe cadence. Assets are immutable after acceptance; corrections create a new asset ID/object key.
Reference normalization:
@@ -51,124 +51,111 @@ ffmpeg -i input.mov \
## Playlist and timeline
- `data/playlist/playlist.txt` contains one local media path per line; blank lines and `#` comments are ignored.
- The service validates the complete playlist before activating a revision.
- Playlist edits are written atomically and receive a revision ID. The active FFmpeg run keeps its current item and adopts the new revision at the next item boundary.
- All viewers follow the same live manifest; clients do not calculate the schedule independently.
- Schedule metadata must be derived from the same active playlist revision and timeline so “now/next” agrees with the video.
- When the last item ends, playout returns to the first item unless an active schedule override says otherwise.
- `data/playlist/playlist.txt`: one local media path per line; blank lines and `#` comments ignored.
- Service validates the complete playlist before activating a revision.
- Playlist edits are atomic and get a revision ID. The active FFmpeg run keeps its current item and adopts the new revision at the next item boundary.
- All viewers follow the same live manifest; clients never calculate the schedule independently.
- Schedule metadata derives from the same active revision/timeline so "now/next" agrees with the video.
- When the last item ends, playout returns to the first unless an active schedule override says otherwise.
## Playout and HLS packaging
The playout service wraps FFmpeg, tracks the active item and playlist revision, restarts failed processes, and exposes `/health` plus Prometheus metrics.
Baseline packaging:
The playout service wraps FFmpeg, tracks the active item/revision, restarts failed processes, and exposes `/health` plus Prometheus metrics.
| Setting | Value |
| ---------------- | ------------------------------------------------------------------------------------- |
| -------------- | ---------------------------------------------------------------------------------- |
| Protocol | Standard live HLS, not LL-HLS |
| Segment format | fMP4 (`.m4s`) with an initialization segment; `.ts` may be supported during migration |
| Segment format | fMP4 (`.m4s`) with init segment; `.ts` may be supported during migration |
| Segment target | 2 seconds (`HLS_SEGMENT_DURATION=2`) |
| Live window | 12 segments (`HLS_WINDOW_SIZE=12`), about 24 seconds |
| Live window | 12 segments (`HLS_WINDOW_SIZE=12`), ~24 seconds |
| Codec path | Stream copy for conforming assets; no per-viewer transcoding |
| Expected latency | 15–35 seconds |
Segment boundaries must align with keyframes. Timestamps must remain monotonic across item boundaries. If FFmpeg cannot preserve a seamless timeline, insert `#EXT-X-DISCONTINUITY` and publish the corresponding new initialization segment.
Segment boundaries must align with keyframes; timestamps must stay monotonic across item boundaries. If FFmpeg cannot preserve a seamless timeline, insert `#EXT-X-DISCONTINUITY` and publish the corresponding new init segment.
**Segment duration rationale**: 2s segments were chosen to lower end-to-end latency and keep the live window small (12 × 2s = 24s) while staying well above the manifest TTL (1–2s). The 2s GOP (30fps, `-g 60 -keyint_min 60`) already matches the target, so segments cut cleanly on keyframes. Longer durations (4–6s) increase latency and window depth; shorter than 2s inflates segment/request overhead and R2 Class B operations with little latency gain.
**Segment duration rationale**: 2s lowers latency and keeps the window small (12 × 2s = 24s) while staying above the manifest TTL (1–2s). The 2s GOP (30fps, `-g 60`) already matches the target, so segments cut cleanly on keyframes. Longer (4–6s) raises latency and window depth; shorter than 2s inflates segment/request overhead and R2 Class B ops with little gain.
Every segment name is globally unique and never reused, including after a restart or playlist loop. Use a run/session ID plus a monotonically increasing media sequence, for example:
Every segment name is globally unique and never reused, including after restart or playlist loop — use a run/session ID plus a monotonically increasing media sequence:
```text
hls/r-20260807T170412Z/init-000001.mp4
hls/r-20260807T170412Z/seg-000001234.m4s
```
This naming rule is essential: a URL advertised as immutable must never later contain different bytes.
A URL advertised as immutable must never later contain different bytes.
## Safe publication to R2
Publication order prevents a manifest from referencing an unavailable segment:
Order prevents a manifest referencing an unavailable segment:
1. FFmpeg writes a segment to a temporary local name and closes it.
2. The publisher uploads the initialization segment, if new, and media segment to their final immutable R2 keys.
3. It verifies successful upload and records object size/checksum.
4. It renders the next media manifest referencing only confirmed objects.
5. It uploads the manifest to a temporary key, then replaces `hls/playlist.m3u8` as the final commit step.
1. FFmpeg writes a segment to a temp local name and closes it.
2. Publisher uploads the init segment (if new) and media segment to final immutable R2 keys.
3. Verify upload success; record object size/checksum.
4. Render the next manifest referencing only confirmed objects.
5. Upload the manifest to a temp key, then replace `hls/playlist.m3u8` as the final commit step.
Never upload an actively written segment or publish the manifest first. Use a single publisher instance, or fencing/leader election, so two playout workers cannot update the live manifest concurrently.
Never upload an actively written segment or publish the manifest first. Use a single publisher (or fencing/leader election) so two workers can't update the live manifest concurrently.
The local `data/hls/` directory is staging data and may be regenerated. Source media, playlist revisions, and schedule state are authoritative; generated HLS is not backed up as application state.
`data/hls/` is regenerable staging. Source media, playlist revisions, and schedule state are authoritative; generated HLS is not backed up as application state.
## Cache contract
| Objects | Edge policy | Browser policy | Required headers/behavior |
| -------------------------- | ---------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------- |
| -------------------------- | --------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------- |
| `.m4s`, `.ts`, init `.mp4` | Cache eligible; TTL 1 year | TTL 1 year | `Cache-Control: public, max-age=31536000, immutable`; full path in cache key |
| Live `.m3u8` | Cache eligible; edge TTL 2 seconds | Do not retain beyond freshness interval | `Cache-Control: public, max-age=1, s-maxage=2, must-revalidate` |
| Live `.m3u8` | Cache eligible; edge TTL 2s | Do not retain beyond freshness interval | `Cache-Control: public, max-age=1, s-maxage=2, must-revalidate` |
| Versioned JS/CSS | Cache eligible; TTL 1 year | TTL 1 year | Content-hashed filenames and `immutable` |
Further rules:
- Do not add query-string cache busters to segment URLs; uniqueness belongs in the path.
- Cache keys must preserve hostname and full path. Ignore only explicitly approved tracking parameters.
- HLS responses require `Access-Control-Allow-Origin: https://linearvoid.tv` (or the reviewed origin allowlist), `GET`, `HEAD`, and `OPTIONS` support, and correct MIME types.
- Enable Smart Tiered Cache for `media.linearvoid.tv` to reduce R2 reads on regional misses.
- Purge manifests when necessary, but never purge immutable segments during normal operation.
- Retain expired-window segments in R2 for at least the maximum manifest TTL plus client retry/grace time. A practical minimum is one hour; lifecycle deletion may run later according to storage policy.
- No query-string cache busters on segment URLs; uniqueness belongs in the path.
- Cache keys preserve hostname and full path; ignore only explicitly approved tracking params.
- HLS responses need `Access-Control-Allow-Origin: https://linearvoid.tv` (or reviewed allowlist), `GET`/`HEAD`/`OPTIONS`, and correct MIME types.
- Enable Smart Tiered Cache for `media.linearvoid.tv` to cut R2 reads on regional misses.
- Purge manifests when necessary, never immutable segments during normal operation.
- Retain expired-window segments at least max manifest TTL + retry/grace time; practical minimum 1 hour; lifecycle deletion may run later per storage policy.
Expected validation:
Validation:
```bash
curl -I https://media.linearvoid.tv/hls/playlist.m3u8
curl -I https://media.linearvoid.tv/hls/<segment>.m4s
```
The manifest must refresh within a few seconds. Repeated segment requests should progress from `CF-Cache-Status: MISS` to `HIT`, retain the same ETag/content length, and carry the immutable cache header.
The manifest must refresh within seconds; repeated segment requests should progress `CF-Cache-Status: MISS` → `HIT`, keep the same ETag/content length, and carry the immutable cache header.
## Player behavior
The SPA uses native HLS where supported and hls.js elsewhere.
- Firefox has no native HLS, so it uses the hls.js/MSE path; fMP4 (`.m4s` + init segment) is supported. Autoplay requires `muted` + `playsinline`, which are already set on the `<video>` element.
SPA uses native HLS where supported, hls.js elsewhere.
- Firefox has no native HLS → hls.js/MSE path; fMP4 (`.m4s` + init) is supported. Autoplay needs `muted` + `playsinline`, already set on the `<video>` element.
- Start at the live edge with a conservative buffer; do not seek to playlist start.
- Treat the manifest as the synchronization source and target roughly three segments behind the newest segment unless testing supports a smaller margin.
- Recover transient network/media errors with bounded retries and exponential backoff.
- On a fatal hls.js network error, reload the manifest; on a fatal media error, attempt media recovery once before recreating the player.
- When tab suspension or drift leaves playback materially behind the target live position, seek forward to the target rather than playing the backlog.
- Surface a clear reconnecting state. Never silently loop stale media.
- Schedule/chat timestamps should account for the measured player latency rather than assuming wall-clock playout is already visible.
- Treat the manifest as the sync source; target ~3 segments behind the newest unless testing supports a smaller margin.
- Recover transient network/media errors with bounded retries + exponential backoff.
- On fatal hls.js network error, reload the manifest; on fatal media error, attempt media recovery once before recreating the player.
- If tab suspension/drift leaves playback materially behind the target, seek forward rather than play the backlog.
- Surface a clear reconnecting state; never silently loop stale media.
- Schedule/chat timestamps should account for measured player latency, not wall-clock playout.
## Failure and recovery
The service retries an item failure with a bounded backoff. After the configured failure threshold it switches to a pre-normalized looping slate and reports `mode=slate`. It periodically retries normal playout and returns only after a segment and committed manifest have been published successfully.
The service retries an item failure with bounded backoff. Past the failure threshold it switches to a pre-normalized looping slate and reports `mode=slate`, periodically retrying normal playout, returning only after a segment and committed manifest publish successfully.
On process or host restart:
On process/host restart:
- create a new run/session ID and continue with globally unique segment paths;
- resume from the schedule-derived current position when reliable state exists, otherwise start the current playlist item;
- create a new run/session ID and continue with unique segment paths;
- resume from schedule-derived current position when reliable state exists, else start the current playlist item;
- never overwrite old segment objects;
- keep the last valid manifest available until the replacement stream is playable;
- emit a discontinuity when timeline continuity cannot be guaranteed.
- keep the last valid manifest available until the replacement is playable;
- emit a discontinuity when timeline continuity isn't guaranteed.
Health is not merely “process running.” `/health` should report at least mode (`live`, `slate`, `starting`, `degraded`), active item, playlist revision, run ID, last segment sequence/time, last successful R2 publication, consecutive failures, and uptime. Mark the service unhealthy if the manifest or media sequence has not advanced for more than two segment durations.
`/health` must report more than "process running": at least mode (`live`/`slate`/`starting`/`degraded`), active item, playlist revision, run ID, last segment sequence/time, last successful R2 publication, consecutive failures, uptime. Mark unhealthy if the manifest or media sequence hasn't advanced for more than two segment durations.
## Monitoring and acceptance checks
## Monitoring and acceptance
Monitor and alert on:
Monitor/alert on: manifest age and media-sequence advancement; time since last published segment; FFmpeg restarts and consecutive failures; live/slate mode; R2 upload latency/errors; segment 404/5xx and manifest/segment latency; Cloudflare cache-hit ratio, origin requests, R2 Class B ops; player startup, fatal errors, rebuffer ratio, live-edge drift; local staging disk usage.
- manifest age and media-sequence advancement;
- time since last successfully published segment;
- FFmpeg restarts and consecutive item failures;
- live/slate mode;
- R2 upload latency and errors;
- segment 404/5xx rate and manifest/segment request latency;
- Cloudflare cache-hit ratio, origin requests, and R2 Class B operations;
- player startup time, fatal errors, rebuffer ratio, and live-edge drift;
- local staging disk usage.
A release is streaming-ready only when one full playlist boundary has played without timestamp errors, the manifest never references a missing object, a second viewer joins the same timeline, cache headers match the table above, a forced FFmpeg failure activates the slate, and normal playout recovers without reusing an object URL.
A release is streaming-ready only when: one full playlist boundary plays without timestamp errors; the manifest never references a missing object; a second viewer joins the same timeline; cache headers match the table; a forced FFmpeg failure activates slate; and normal playout recovers without reusing an object URL.
## Required configuration
@@ -179,14 +166,14 @@ A release is streaming-ready only when one full playlist boundary has played wit
| `HLS_WINDOW_SIZE` | Manifest window; baseline `12` |
| `R2_ENDPOINT`, `R2_REGION` | S3-compatible publication endpoint |
| `R2_BUCKET_NAME` | Media bucket |
| `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY` | Write credentials, available only to the publisher |
| `R2_KEEP_RUNS` | Number of run folders to retain (default `2`) |
| `R2_MAX_RETENTION_BYTES` | Max bytes retained across run folders (10 GiB) |
| `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY` | Write credentials, publisher only |
| `R2_KEEP_RUNS` | Run folders to retain (default `2`) |
| `R2_MAX_RETENTION_BYTES` | Max bytes across run folders (10 GiB) |
| `MEDIA_DOMAIN` | Public R2 custom domain |
| `VITE_PUBLIC_HLS_URL` | Public live manifest URL |
Production values are `MEDIA_DOMAIN=media.linearvoid.tv` and `VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8`. Local fallback may use `http://localhost/hls/playlist.m3u8`.
Production values: `MEDIA_DOMAIN=media.linearvoid.tv`, `VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8`. Local fallback may use `http://localhost/hls/playlist.m3u8`.
## Explicitly deferred
Adaptive-bitrate ladders, LL-HLS, DVR/catch-up, multi-origin playout, and regional chat affinity are outside the v1 contract. Adding any of them requires an ADR and an update to this document.
ABR ladders, LL-HLS, DVR/catch-up, multi-origin playout, and regional chat affinity are outside the v1 contract. Adding any requires an ADR and an update to this document.
docs/architecture/README.md
+216 -685
View File
File diff suppressed because it is too large. Load diff
docs/incident-runbook.md
+55 -124
View File
@@ -4,155 +4,101 @@ Triage order for Linear Void incidents. Work top-to-bottom — each tier depends
## Triage Order
1. Edge Reachability (Cloudflare)
2. Origin Health (VPS/Docker)
3. Playout Progression (HLS)
4. Chat Connectivity (WebSocket)
5. Data-Plane (Redis/DB)
1. **Edge Reachability** (Cloudflare)
2. **Origin Health** (VPS/Docker)
3. **Playout Progression** (HLS)
4. **Chat Connectivity** (WebSocket)
5. **Data-Plane** (Redis/DB)
## 1. Edge Reachability (Cloudflare)
**Symptoms:** `linearvoid.tv` not resolving, SSL errors, 5xx from Cloudflare.
```bash
# Check DNS
dig linearvoid.tv
dig www.linearvoid.tv
# Check Cloudflare status
curl -I https://linearvoid.tv/health
```
**Common causes:**
**Common causes:** DNS misconfiguration (A record → wrong IP); Cloudflare proxy disabled (grey cloud exposes origin); SSL certificate expired (Cloudflare → SSL/TLS); Cloudflare outage ([cloudflarestatus.com](https://www.cloudflarestatus.com)).
- DNS misconfiguration (A record pointing to wrong IP)
- Cloudflare proxy disabled (grey cloud → origin exposed directly)
- SSL certificate expired (check Cloudflare dashboard → SSL/TLS)
- Cloudflare outage → check [https://www.cloudflarestatus.com](https://www.cloudflarestatus.com)
**Mitigation:** Verify A record points `linearvoid.tv` → origin IP (proxied, orange cloud). Bypass Cloudflare temporarily:
**Mitigation:**
- Verify A record: `linearvoid.tv` → origin IP (proxied, orange cloud)
- Bypass Cloudflare temporarily: `curl -H "Host: linearvoid.tv" http://<origin-ip>/health`
```bash
curl -H "Host: linearvoid.tv" http://<origin-ip>/health
```
## 2. Origin Health (VPS/Docker)
**Symptoms:** Cloudflare returns 502/504, containers down.
**Symptoms:** Cloudflare 502/504, containers down.
```bash
# Remote Docker (test lab)
docker -H tcp://192.168.88.40:2375 ps
docker -H tcp://192.168.88.40:2375 logs linearvoid-nginx --tail 50
# Direct health check
curl http://192.168.88.40/health
```
**Common causes:**
- Host out of disk space: `df -h`
- Docker daemon stopped: `systemctl status docker`
- Container restart loop: `docker ps -a --filter "status=exited"`
- Resource exhaustion (OOM): `dmesg | grep -i oom`
**Common causes:** Host disk full (`df -h`); Docker daemon stopped (`systemctl status docker`); container restart loop (`docker ps -a --filter "status=exited"`); OOM (`dmesg | grep -i oom`).
**Mitigation:**
- Restart stack: `docker compose -f docker-compose.prod.yml up -d`
- Free disk: `docker system prune -f`
- Check logs per service: `docker logs linearvoid-<service> --tail 100`
```bash
docker compose -f docker-compose.prod.yml up -d # restart stack
docker system prune -f # free disk
docker logs linearvoid-<service> --tail 100 # per-service logs
```
## 3. Playout Progression (HLS)
**Symptoms:** Viewer sees slate or buffering, playlist not advancing.
```bash
# Check playout health
curl http://192.168.88.40:8081/health
# Response includes "mode": "live" or "slate"
# Check HLS output
curl http://192.168.88.40:8081/health # "mode": "live" or "slate"
curl -I http://192.168.88.40/hls/playlist.m3u8
ls -la data/hls/
# Check playout logs
docker logs linearvoid-playout --tail 100
```
**Common causes:**
**Common causes:** FFmpeg crash → slate fallback (health `degraded`); playlist empty/missing (`cat data/playlist/playlist.txt`); media file missing (`ls data/media/`); disk full → HLS segments can't write.
- FFmpeg crash → slate fallback active (health shows `degraded`)
- Playlist file empty/missing: `cat data/playlist/playlist.txt`
- Media file missing from playlist: `ls data/media/`
- Disk full → HLS segments can't write
**Mitigation:**
- Verify playlist has valid entries
- Check media files exist at paths in playlist
- Restart playout: `docker compose restart playout`
- Generate demo playlist if empty: see `backend/playout/start-playout.sh`
**Mitigation:** Validate playlist entries and media paths; restart playout (`docker compose restart playout`); regenerate demo playlist if empty (see `backend/playout/start-playout.sh`).
## 4. Chat Connectivity (WebSocket)
**Symptoms:** Chat not loading, messages not sending, presence broken.
```bash
# Check chat health
curl http://192.168.88.40:3000/health
# Test WebSocket
wscat -c ws://192.168.88.40/ws?username=test&room=main
# Check chat logs
docker logs linearvoid-chat --tail 100
```
**Common causes:**
**Common causes:** Redis down → presence/rate-limiting degraded (chat still works); DB down → messages not persisting; port not exposed (`docker port linearvoid-chat`).
- Redis down → presence/rate-limiting degraded (chat still works)
- Database down → messages not persisting
- Port not exposed: `docker port linearvoid-chat`
**Mitigation:**
- Restart chat: `docker compose restart chat`
- Check Redis: `docker exec linearvoid-redis redis-cli ping`
- Check DB: `docker exec linearvoid-db pg_isready -U linearvoid`
- Chat works without Redis (graceful degradation) — check logs for "Redis connection warning"
**Mitigation:** Restart chat (`docker compose restart chat`); check Redis (`docker exec linearvoid-redis redis-cli ping`); check DB (`docker exec linearvoid-db pg_isready -U linearvoid`). Chat degrades gracefully without Redis — look for "Redis connection warning" in logs.
## 5. Data-Plane (Redis/DB)
**Symptoms:** Chat works but no persistence/presence, admin stats empty.
```bash
# Redis
docker exec linearvoid-redis redis-cli ping
docker exec linearvoid-redis redis-cli info memory
# PostgreSQL
docker exec linearvoid-db pg_isready -U linearvoid
docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"
```
**Common causes:**
**Common causes:** Redis OOM (check `maxmemory`); PostgreSQL connection limit; disk full on volume mount.
- Redis OOM (check `maxmemory` setting)
- PostgreSQL connection limit reached
- Disk full on volume mount
**Mitigation:**
- Redis: `docker compose restart redis` (AOF persist enabled)
- PostgreSQL: `docker compose restart db`
- Restore from backup: see `scripts/verify-backups.sh` for procedure
**Mitigation:** Restart redis (`docker compose restart redis`, AOF persist enabled); restart db (`docker compose restart db`); restore from backup (see `scripts/verify-backups.sh`).
## Escalation
| Severity | When | Action |
| -------- | ------------------------------------ | -------------------------------------------------- |
| -------- | ------------------------------------ | ------------------------------------------- |
| Critical | Slate active >30min, complete outage | Trigger DR plan, provision fallback host |
| Warning | Redis down, high CPU >30min | Investigate during business hours, restart service |
| Info | Single service blip | Monitor, no action unless recurring |
| Warning | Redis down, high CPU >30min | Investigate business hours, restart service |
| Info | Single service blip | Monitor; no action unless recurring |
## Post-Incident
@@ -163,67 +109,52 @@ docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"
## Quarterly Disaster Recovery Drill
Goal: validate recovery targets under controlled host-loss simulation.
**Recovery targets:** RPO <= 24h, RTO <= 60min.
Validate recovery targets under controlled host-loss simulation: **RPO ≤ 24h, RTO ≤ 60min**.
### Prerequisites
- Latest backup artifacts available from `scripts/backup-db.sh` and `scripts/backup-weekly.sh`
- Latest backups from `scripts/backup-db.sh` and `scripts/backup-weekly.sh`
- Operator access to Docker host and DNS controls
- Current deployment variables available (`.env`, compose file, image tags)
- Current deployment variables (`.env`, compose file, image tags)
### Drill Scope
### Scope
- Simulate complete origin loss (all app containers unavailable)
- Rebuild stack on clean host
- Restore DB and media/playlist mounts
- Verify viewer playback, chat, admin auth, monitoring targets
### Drill Steps
### Steps
1. Record drill start timestamp and on-call participants.
2. Capture pre-drill baseline:
- `curl -fsS http://<origin>/health`
- `curl -fsS http://<origin>:3000/health`
- `curl -fsS http://<origin>:8081/health`
3. Simulate origin outage:
- `docker compose -f docker-compose.prod.yml down`
4. Provision clean recovery host (or clean test-lab VM) and copy deployment bundle.
5. Restore required data paths:
- `data/playlist/`
- `data/media/`
- `data/chat/`
- `data/hls/` (optional warm cache)
```bash
curl -fsS http://<origin>/health
curl -fsS http://<origin>:3000/health
curl -fsS http://<origin>:8081/health
```
3. Simulate outage: `docker compose -f docker-compose.prod.yml down`
4. Provision clean recovery host (or clean test-lab VM); copy deployment bundle.
5. Restore data paths: `data/playlist/`, `data/media/`, `data/chat/`, `data/hls/` (optional warm cache).
6. Restore database from latest backup dump.
7. Start stack:
- `docker compose -f docker-compose.prod.yml up -d`
8. Validate service health and runtime:
- `docker ps --format "table {{.Names}}\t{{.Status}}"`
- `curl -fsS http://<origin>/health`
- `curl -fsS http://<origin>/hls/playlist.m3u8 | head -n 5`
- `curl -fsS http://<origin>:9090/-/ready`
9. Validate app behavior:
- Viewer SPA loads and HLS plays
- WebSocket chat connects and message round-trip works
- Admin login/session endpoint returns success
10. Run backup integrity check:
- `scripts/verify-backups.sh`
11. Record drill end timestamp and compute measured RTO/RPO.
7. Start stack: `docker compose -f docker-compose.prod.yml up -d`
8. Validate health and runtime:
```bash
docker ps --format "table {{.Names}}\t{{.Status}}"
curl -fsS http://<origin>/health
curl -fsS http://<origin>/hls/playlist.m3u8 | head -n 5
curl -fsS http://<origin>:9090/-/ready
```
9. Validate app behavior: viewer SPA loads and HLS plays; WebSocket chat connects with message round-trip; admin login/session endpoint succeeds.
10. Run backup integrity check: `scripts/verify-backups.sh`
11. Record drill end timestamp; compute measured RTO/RPO.
### Pass/Fail Criteria
- Pass if all core services healthy and functional in <= 60 minutes.
- Pass if latest recoverable data point is <= 24 hours old.
- Fail if any critical path remains degraded (playout, chat, admin auth, monitoring).
- **Pass:** all core services healthy and functional in ≤ 60 min.
- **Pass:** latest recoverable data point ≤ 24 h old.
- **Fail:** any critical path remains degraded (playout, chat, admin auth, monitoring).
### Evidence Template
Document each drill in incident tracker with:
- Date/time window
- Environment (test-lab, staging, production)
- Backup artifact IDs and timestamps
- Measured RTO and RPO
- Issues found and remediation owner
- Follow-up ticket links
Record in incident tracker: date/time window; environment (test-lab, staging, production); backup artifact IDs and timestamps; measured RTO and RPO; issues found and remediation owner; follow-up ticket links.
docs/infrastructure/cloudflare/cloudflare-account-troubleshooting.md
+7 -7
View File
@@ -6,19 +6,19 @@ Use `wrangler whoami` to verify the account ID and API token are valid, or query
## Validate Cloudflare account and API token
```powershell
```bash
# Validate Cloudflare account and API token in .env
Set-Location -Path "E:\Documents\02-Projects\07-Mentoring\Corey\linearvoid"
Get-Content .env | Select-String 'CLOUDFLARE_API_TOKEN'
Get-Content .env | Select-String 'CLOUDFLARE_ACCOUNT_ID'
cd /path/to/linearvoid
grep -E 'CLOUDFLARE_API_TOKEN|CLOUDFLARE_ACCOUNT_ID' .env
# Test API token with wrangler
npx wrangler whoami
# Or verify with the Workers API directly
$env:CLOUDFLARE_API_TOKEN = (Get-Content .env | Select-String 'CLOUDFLARE_API_TOKEN' | ForEach-Object { $_.Line.Split('=')[1].Trim() })
$env:CLOUDFLARE_ACCOUNT_ID = (Get-Content .env | Select-String 'CLOUDFLARE_ACCOUNT_ID' | ForEach-Object { $_.Line.Split('=')[1].Trim() })
export CLOUDFLARE_API_TOKEN="$(grep '^CLOUDFLARE_API_TOKEN=' .env | cut -d= -f2-)"
export CLOUDFLARE_ACCOUNT_ID="$(grep '^CLOUDFLARE_ACCOUNT_ID=' .env | cut -d= -f2-)"
# List all Workers for the account
curl -X GET "https://api.cloudflare.com/client/v4/accounts/$env:CLOUDFLARE_ACCOUNT_ID/workers/scripts" -H "Authorization: Bearer $env:CLOUDFLARE_API_TOKEN" -H "Content-Type: application/json"
curl -X GET "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/workers/scripts" \
-H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" -H "Content-Type: application/json"
```
docs/infrastructure/monitoring/prometheus_results.md
+114 -1627
View File
File diff suppressed because it is too large. Load diff
docs/operations/scaling-performance-playbook.md
+62 -279
View File
@@ -1,345 +1,156 @@
# Scaling and Performance Playbook
Purpose: define repeatable method to measure performance, capture baseline data, and decide scaling actions.
Scope: planning and execution guide for TODO items under "Scaling & Performance (planning only)".
Purpose: define a repeatable method to measure performance, capture baselines, and decide scaling actions for the "Linear Void" streaming app. Scope: planning/execution for the "Scaling & Performance (planning only)" TODO items.
## 1) Manifest TTL Tuning Plan
Goal: tune `.m3u8` TTL to about `0.5x` segment duration while keeping playback stable.
Goal: tune the `.m3u8` TTL to ~`0.5x` segment duration while keeping playback stable.
Inputs:
- Segment duration (`HLS_SEGMENT_DURATION`)
- Current manifest TTL (Cloudflare cache rule)
- Cache outcome distribution (`Cf-Cache-Status`)
Inputs: segment duration (`HLS_SEGMENT_DURATION`), current manifest TTL (Cloudflare cache rule), cache outcome distribution (`Cf-Cache-Status`).
Procedure:
1. Start from current manifest TTL (2s).
2. Capture 24h baseline headers from repeated `playlist.m3u8` requests.
3. Test candidate TTLs: 1s, 2s, 3s.
4. Compare:
- rebuffer events from player logs
- origin request increase/decrease
- HIT ratio changes
5. Select lowest TTL that keeps playback stable and origin load acceptable.
4. Compare rebuffer events (player logs), origin request increase/decrease, and HIT ratio changes.
5. Select the lowest TTL that keeps playback stable and origin load acceptable.
Acceptance:
- No material increase in rebuffer rate.
- Manifest staleness not observable by users.
Acceptance: no material increase in rebuffer rate; manifest staleness not observable by users.
## 2) TTFB Baseline Plan
Goal: measure warm-cache TTFB from 3+ US regions with target `< 4s`.
Suggested regions:
- US East (Virginia)
- US Central
- US West (California/Oregon)
Regions: US East (Virginia), US Central, US West (California/Oregon).
Method:
1. Use WebPageTest with repeat view enabled.
2. Run at least 5 samples per region.
3. Record median and p95 TTFB.
4. Store results in table (date, region, median, p95, test URL).
4. Store results in a table (date, region, median, p95, test URL).
Acceptance:
- Warm-cache median < 4s in all tested regions.
- Outliers documented with likely cause.
Acceptance: warm-cache median < 4s in all regions; outliers documented with likely cause.
### TTFB Baseline Worksheet
Use one row per region; keep the raw WebPageTest result URL with the summary for comparison.
Use one row per region and keep the raw WebPageTest result URL with the summary so later runs can be compared against the same evidence.
- US East (Virginia) / US Central / US West (California/Oregon): Date/time, Test URL `https://linearvoid.tv`, WebPageTest result URL, Median TTFB, p95 TTFB, Notes.
- US East (Virginia)
- Date/time:
- Test URL: <https://linearvoid.tv>
- WebPageTest result URL:
- Median TTFB:
- p95 TTFB:
- Notes:
- US Central
- Date/time:
- Test URL: <https://linearvoid.tv>
- WebPageTest result URL:
- Median TTFB:
- p95 TTFB:
- Notes:
- US West (California/Oregon)
- Date/time:
- Test URL: <https://linearvoid.tv>
- WebPageTest result URL:
- Median TTFB:
- p95 TTFB:
- Notes:
Suggested capture notes:
- Use repeat view for the warm-cache measurement.
- Keep the same browser profile and test URL across regions.
- Note any CDN cache-rule changes or deploys that occurred between samples.
Capture notes: use repeat view for warm-cache; keep the same browser profile and test URL across regions; note any CDN cache-rule changes or deploys between samples.
## 3) WebSocket Load-Test Plan
Goal: characterize chat service behavior across 10 -> 100 -> 500 -> 1000 concurrent clients.
Goal: characterize chat behavior across 10 → 100 → 500 → 1000 concurrent clients.
Tooling option A (Artillery):
```bash
npx artillery run scripts/loadtest/ws-chat.yml
```
Tooling option B (custom Go harness):
Tooling option B (custom Go harness): open N concurrent WS clients, send a message every X seconds, measure round-trip latency and failures.
- open N concurrent WebSocket clients
- send message every X seconds
- measure round-trip latency and failures
Metrics: p50/p95 message latency; error rate (connect/send/receive); dropped connections; CPU/memory for `chat`, `redis`, `nginx`.
Metrics to capture:
- p50/p95 message latency
- error rate (connect, send, receive)
- dropped connections
- CPU/memory for `chat`, `redis`, `nginx`
Acceptance:
- p95 chat latency <= 1s at target concurrency tier.
- error rate <= 1% during steady state.
Acceptance: p95 chat latency ≤ 1s at target concurrency tier; error rate ≤ 1% during steady state.
### WebSocket Load-Test Worksheet
- Date/time:
- Environment:
- Commit SHA / image tag:
- Test URL:
- Tool used:
- Scenario duration:
- Concurrency tiers reached:
- p50 message latency:
- p95 message latency:
- Error rate:
- Dropped connections:
- CPU/memory notes for chat:
- CPU/memory notes for redis:
- CPU/memory notes for nginx:
- Raw results location:
- Conclusion:
- Next action:
Date/time, Environment, Commit SHA/image tag, Test URL, Tool used, Scenario duration, Concurrency tiers reached, p50 message latency, p95 message latency, Error rate, Dropped connections, CPU/memory notes for chat/redis/nginx, Raw results location, Conclusion, Next action.
## 4) Admin API Benchmark Plan
Goal: measure admin API p95 and identify slow endpoints.
Tooling option A (`k6`):
Endpoints: `/api/admin/overview`, `/api/admin/analytics`, `/api/admin/chat-log`, `/api/admin/users`.
Tooling option A (`k6`):
```bash
k6 run scripts/loadtest/admin-api.js
```
Tooling option B (`wrk`):
```bash
wrk -t4 -c100 -d60s http://<host>/api/admin/overview
```
Endpoints:
- `/api/admin/overview`
- `/api/admin/analytics`
- `/api/admin/chat-log`
- `/api/admin/users`
Acceptance:
- p95 < 1s for primary admin endpoints.
- slow endpoint list with remediation notes.
Acceptance: p95 < 1s for primary admin endpoints; slow endpoint list with remediation notes.
### Admin API Benchmark Worksheet
- Date/time:
- Environment:
- Commit SHA / image tag:
- Test URL:
- Tool used:
- Scenario duration:
- Concurrency / virtual users:
- `/api/admin/overview` p95:
- `/api/admin/analytics` p95:
- `/api/admin/chat-log` p95:
- `/api/admin/users` p95:
- Error rate:
- Slowest endpoint:
- Cache or DB notes:
- Raw results location:
- Conclusion:
- Next action:
Date/time, Environment, Commit SHA/image tag, Test URL, Tool used, Scenario duration, Concurrency/virtual users, `/api/admin/overview|analytics|chat-log|users` p95, Error rate, Slowest endpoint, Cache or DB notes, Raw results location, Conclusion, Next action.
## 5) Cache Hit Ratio Tracking Plan
Goal: track Cloudflare cache hit ratio and alert when below threshold.
Implementation outline:
Implementation:
1. Pull Cloudflare GraphQL analytics on schedule.
2. Export to Prometheus-compatible metric path.
3. Add Grafana panel with 5m/1h/24h windows.
2. Export to a Prometheus-compatible metric path.
3. Add a Grafana panel with 5m/1h/24h windows.
4. Alert when hit ratio < 90% for 15m.
Reference exporter:
- `scripts/cache-hit-ratio.js`
- `scripts/cache-hit-ratio.ps1` for Windows scheduled tasks / manual runs
- Example run: `CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_TAG=... CLOUDFLARE_HOSTNAME=media.linearvoid.tv node scripts/cache-hit-ratio.js`
- Windows example: `pwsh -File scripts/cache-hit-ratio.ps1`
- `scripts/cache-hit-ratio.sh` for scheduled tasks / manual runs
- Example run: `export CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_TAG=... CLOUDFLARE_HOSTNAME=media.linearvoid.tv; node scripts/cache-hit-ratio.js`
- Intended output: Prometheus text format on stdout, suitable for cron, a Node wrapper, or a small scrape endpoint.
Metric target:
- `cache_hit_ratio >= 0.90`
Metric target: `cache_hit_ratio >= 0.90`.
### Cache Hit Ratio Worksheet
- Date/time:
- Environment:
- Commit SHA / image tag:
- Hostname:
- Window minutes:
- Cloudflare zone tag:
- Export command:
- Raw exporter output location:
- Hit requests:
- Total requests:
- Cache hit ratio:
- Threshold met:
- Notes:
Date/time, Environment, Commit SHA/image tag, Hostname, Window minutes, Cloudflare zone tag, Export command, Raw exporter output location, Hit requests, Total requests, Cache hit ratio, Threshold met, Notes.
## 6) Capacity Trigger Rules
Define operational triggers:
Operational triggers:
- Cache miss ratio > 10% for 15m
- Chat p95 latency > 1s for 5m
- Host CPU > 80% for 10m
- Redis memory usage ratio (`linearvoid_chat_redis_memory_usage_ratio`) > 0.85 for 10m
Action mapping:
- Level 1: investigate + restart non-critical components
- Level 2: scale service resources (CPU/RAM)
- Level 3: activate standby planning path
### Capacity Trigger Review
- Date/time:
- Environment:
- Commit SHA / image tag:
- Trigger hit:
- Threshold value:
- Observed value:
- Duration above threshold:
- Level selected:
- Immediate action taken:
- Services involved:
- Follow-up owner:
- Raw metrics / alert link:
- Notes:
Date/time, Environment, Commit SHA/image tag, Trigger hit, Threshold value, Observed value, Duration above threshold, Level selected, Immediate action taken, Services involved, Follow-up owner, Raw metrics/alert link, Notes.
## 7) Phase 2 Service Separation Plan
Objective: separate bottlenecked services while keeping simple ops.
Objective: separate bottlenecked services while keeping ops simple.
Boundaries:
- `playout` isolated first (CPU-heavy)
- `chat` + `redis` grouped for latency
- `db` isolated for IO stability
- monitoring stack optionally separated after sustained load
Initial sizing guidance:
Initial sizing:
- playout host: 4 vCPU / 8 GB
- chat host: 2-4 vCPU / 4-8 GB
- chat host: 2–4 vCPU / 4–8 GB
- db host: 2 vCPU / 4 GB + fast disk
Exit criteria for Phase 2:
- repeated trigger events over 7 days
- capacity saturation despite tuning
Exit criteria: repeated trigger events over 7 days; capacity saturation despite tuning.
### Phase 2 Boundary Worksheet
- Date/time:
- Environment:
- Commit SHA / image tag:
- Trigger history summary:
- Repeated trigger count over 7 days:
- Primary bottleneck:
- Proposed split:
- `playout` target host size:
- `chat` target host size:
- `db` target host size:
- Monitoring separation needed:
- Data/state migration notes:
- Implementation owner:
- Review decision:
- Next action:
Date/time, Environment, Commit SHA/image tag, Trigger history summary, Repeated trigger count over 7 days, Primary bottleneck, Proposed split, `playout`/`chat`/`db` target host size, Monitoring separation needed, Data/state migration notes, Implementation owner, Review decision, Next action.
## 8) Phase 3 Warm Standby Plan
Objective: second-region resilience with DNS failover.
Candidate layout:
Layout:
- Primary: Hetzner Ashburn (US-East)
- Standby: Hetzner Hillsboro (US-West)
Design:
- Warm standby stack deployed and health-checked
- Redis replication or fallback mode documented
- Postgres replication strategy selected (streaming replica or periodic snapshot restore)
- Cloudflare DNS failover policy defined (health-based or manual emergency switch)
Failover drill requirements:
Design: warm standby stack deployed and health-checked; Redis replication or fallback mode documented; Postgres replication strategy selected (streaming replica or periodic snapshot restore); Cloudflare DNS failover policy defined (health-based or manual emergency switch).
### Phase 3 Warm Standby Worksheet
- Date/time:
- Environment:
- Commit SHA / image tag:
- Primary region:
- Standby region:
- DNS failover mode:
- Redis strategy:
- Postgres strategy:
- Health check endpoints:
- Simulated cutover performed:
- Measured RTO:
- Measured RPO:
- Open risks:
- Implementation owner:
- Review decision:
- Notes:
Date/time, Environment, Commit SHA/image tag, Primary region, Standby region, DNS failover mode, Redis strategy, Postgres strategy, Health check endpoints, Simulated cutover performed, Measured RTO, Measured RPO, Open risks, Implementation owner, Review decision, Notes.
## 9) Data Capture Template
For each benchmark run, capture:
- Date/time
- Environment (test-lab/staging/production)
- Commit SHA / image tag
- Test scenario and duration
- Raw metrics (latency, errors, resource usage)
- Conclusion and next action
For each benchmark run capture: Date/time, Environment (test-lab/staging/production), Commit SHA/image tag, Test scenario and duration, Raw metrics (latency, errors, resource usage), Conclusion and next action.
## 10) Execution Order
1. TTFB baseline
2. WebSocket load profile
3. Admin API benchmark
@@ -349,14 +160,9 @@ For each benchmark run, capture:
7. Phase 3 standby design review
## 11) Benchmark Run Sheet
Use the same worksheet for every baseline/re-test so results stay comparable.
Use the same worksheet for every baseline or re-test so results stay comparable.
Preconditions:
- Confirm the target environment and commit SHA/image tag.
- Confirm Cloudflare cache rules are active when testing TTFB or manifest TTL.
- Confirm the chat, admin, and playout services are healthy before load testing.
Preconditions: confirm target environment and commit SHA/image tag; confirm Cloudflare cache rules are active when testing TTFB or manifest TTL; confirm chat, admin, and playout services are healthy before load testing.
Run template:
@@ -373,70 +179,47 @@ Run template:
| Next action | |
Suggested command mapping:
- TTFB baseline: WebPageTest repeat-view run against `https://linearvoid.tv`.
- WebSocket profile: `npx artillery run scripts/loadtest/ws-chat.yml` or the Go harness referenced above.
- Admin API benchmark: `k6 run scripts/loadtest/admin-api.js` or `wrk -t4 -c100 -d60s http://<host>/api/admin/overview`.
## 12) Scaling Decision Gate
Treat the following as the default review gate after any benchmark batch:
Default review gate after any benchmark batch:
1. If warm-cache TTFB meets target but cache hit ratio falls below 90%, tune cache behavior before increasing capacity.
2. If chat or admin latency exceeds target while CPU and memory remain low, prioritize query or handler optimization.
3. If CPU, memory, or Redis saturation appears first, treat the service boundary guidance in Phase 2 as the next action.
3. If CPU, memory, or Redis saturation appears first, treat the Phase 2 service boundary guidance as the next action.
4. If the same threshold trips on multiple runs across 7 days, open the Phase 2 or Phase 3 planning path instead of tuning in place.
Decision note:
- Keep the smallest change that restores the target metric and record the reason in the run sheet.
Decision note: keep the smallest change that restores the target metric and record the reason in the run sheet.
## 13) Artifact-to-Worksheet Field Mapping
Use this section when filling the WebSocket and Admin API worksheets from helper-generated artifacts.
Use when filling the WebSocket and Admin API worksheets from helper-generated artifacts.
### WebSocket worksheet mapping
Artifact sources:
- `artifacts/<timestamp>/ws-chat-report.json`
- `artifacts/<timestamp>/ws-chat-summary.txt`
Artifact sources: `artifacts/<timestamp>/ws-chat-report.json`, `artifacts/<timestamp>/ws-chat-summary.txt`.
Field mapping:
- `Tool used` → `artillery` (or `scripts/loadtest/run-loadtests.sh -s ws`)
- `Raw results location` → artifact folder path for this run
- `Concurrency tiers reached` → `ws-chat.yml` phases (`warmup-10`, `sustain-100`, `sustain-500`, `sustain-1000`)
- `Error rate` → `vusers.failure_rate_percent` from `ws-chat-summary.txt`
- `Dropped connections` → derive from `vusers.created - vusers.completed` (use `vusers.failed` as failure signal)
- `p95 message latency` → first latency line under `summary.*` in `ws-chat-summary.txt` containing `p95=...`
- `p50 message latency` → first latency line under `summary.*` in `ws-chat-summary.txt` containing `median=...`
- Worksheet `Tool used` -> `artillery` (or `run-loadtests.ps1 -Scenario ws`)
- Worksheet `Raw results location` -> artifact folder path for this run
- Worksheet `Concurrency tiers reached` -> `ws-chat.yml` phases (`warmup-10`, `sustain-100`, `sustain-500`, `sustain-1000`)
- Worksheet `Error rate` -> `vusers.failure_rate_percent` from `ws-chat-summary.txt`
- Worksheet `Dropped connections` -> derive from `vusers.created - vusers.completed` (use `vusers.failed` as failure signal)
- Worksheet `p95 message latency` -> first available latency line under `summary.*` in `ws-chat-summary.txt` that includes `p95=...`
- Worksheet `p50 message latency` -> first available latency line under `summary.*` in `ws-chat-summary.txt` that includes `median=...`
Notes:
- The exact `summary.*` metric names in Artillery output can vary by engine/plugins.
- If no latency summary field is emitted, record this explicitly and use server-side latency metrics as fallback evidence.
Notes: the exact `summary.*` metric names in Artillery output can vary by engine/plugins; if no latency summary field is emitted, record this explicitly and use server-side latency metrics as fallback evidence.
### Admin API worksheet mapping
Artifact sources:
- `artifacts/<timestamp>/admin-api-summary.json`
- `artifacts/<timestamp>/admin-api-output.txt`
- `artifacts/<timestamp>/admin-api-endpoint-summary.txt`
Artifact sources: `artifacts/<timestamp>/admin-api-summary.json`, `artifacts/<timestamp>/admin-api-output.txt`, `artifacts/<timestamp>/admin-api-endpoint-summary.txt`.
Field mapping:
- `Tool used` → `k6` (or `scripts/loadtest/run-loadtests.sh -s admin`)
- `Raw results location` → artifact folder path for this run
- `/api/admin/overview|analytics|chat-log|users` p95 → `overview|analytics|chat_log|users` row `P95Ms` in `admin-api-endpoint-summary.txt`
- `Error rate` → `Global failure rate` in `admin-api-endpoint-summary.txt`
- `Slowest endpoint` → endpoint row with highest `P95Ms`
- Worksheet `Tool used` -> `k6` (or `run-loadtests.ps1 -Scenario admin`)
- Worksheet `Raw results location` -> artifact folder path for this run
- Worksheet `/api/admin/overview` p95 -> `overview` row `P95Ms` in `admin-api-endpoint-summary.txt`
- Worksheet `/api/admin/analytics` p95 -> `analytics` row `P95Ms` in `admin-api-endpoint-summary.txt`
- Worksheet `/api/admin/chat-log` p95 -> `chat_log` row `P95Ms` in `admin-api-endpoint-summary.txt`
- Worksheet `/api/admin/users` p95 -> `users` row `P95Ms` in `admin-api-endpoint-summary.txt`
- Worksheet `Error rate` -> `Global failure rate` in `admin-api-endpoint-summary.txt`
- Worksheet `Slowest endpoint` -> endpoint row with highest `P95Ms`
Notes: keep the raw `admin-api-summary.json` file for audits and re-parsing; if an endpoint row is missing, treat it as a test/data issue and re-run with full endpoint coverage.
Notes:
- Keep the raw `admin-api-summary.json` file for audits and re-parsing.
- If an endpoint row is missing, treat as test/data issue and re-run with full endpoint coverage.
Summary helper (bash): run `scripts/loadtest/summarize-artillery.sh <report.json>` after an Artillery WS run, and `scripts/loadtest/summarize-k6.sh <summary.json>` after a k6 admin run, to produce the artifact files used in the mappings above.
package-lock.json
Generated
+1148
View File
File diff suppressed because it is too large. Load diff
package.json
+5
View File
@@ -0,0 +1,5 @@
{
"dependencies": {
"vitest": "^4.1.11"
}
}
scripts/cache-hit-ratio.sh
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# cache-hit-ratio.sh — Thin wrapper that runs the Node.js cache-hit-ratio tool.
# Mirrors scripts/cache-hit-ratio.ps1.
set -euo pipefail
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
exec node "$script_dir/cache-hit-ratio.js" "$@"
scripts/check-env.sh
+132
View File
@@ -0,0 +1,132 @@
#!/usr/bin/env bash
# check-env.sh — Verify the deployment environment for Linear Void.
# Mirrors scripts/check-env.ps1.
#
# Checks, against the target Docker host (default test lab 192.168.88.40):
# 1. Docker Engine reachability over the TCP socket (port 2375).
# 2. Coolify helper stack (proxy/Traefik, sentinel, cloudflared).
# 3. Linear Void containers (playout, chat, nginx, redis, db, prometheus, grafana).
# 4. Public endpoints (app, HLS manifest, admin).
# 5. Health endpoints exposed by the services.
#
# Usage:
# ./scripts/check-env.sh
# ./scripts/check-env.sh -t 192.168.88.40
# ./scripts/check-env.sh -t 192.168.88.40 -p 2375 --only-containers
#
# Exit code is 0 when all required checks pass, 1 otherwise.
set -uo pipefail
TARGET=192.168.88.40
DOCKER_PORT=2375
ONLY_CONTAINERS=0
ONLY_ENDPOINTS=0
TIMEOUT=10
while [[ $# -gt 0 ]]; do
case "$1" in
-t|--target) TARGET="$2"; shift 2;;
-p|--docker-port) DOCKER_PORT="$2"; shift 2;;
--only-containers) ONLY_CONTAINERS=1; shift;;
--only-endpoints) ONLY_ENDPOINTS=1; shift;;
--timeout) TIMEOUT="$2"; shift 2;;
*) echo "Unknown option: $1" >&2; exit 2;;
esac
done
export DOCKER_HOST="tcp://${TARGET}:${DOCKER_PORT}"
FAILED=0
fail() { printf ' [FAIL] %s\n' "$1"; FAILED=1; }
ok() { printf ' [OK] %s\n' "$1"; }
warn() { printf ' [WARN] %s\n' "$1"; }
echo "== Linear Void environment check =="
echo "Target Docker host : $DOCKER_HOST"
echo "Date : $(date -Iseconds)"
# ---------------------------------------------------------------------------
# 1. Docker Engine reachability
# ---------------------------------------------------------------------------
if [[ "$ONLY_ENDPOINTS" -eq 0 ]]; then
echo ""
echo "-- Docker Engine (${TARGET}:${DOCKER_PORT}) --"
if ver=$(docker version --format '{{.Server.Version}}' 2>&1) && [[ "$ver" =~ [0-9] ]]; then
ok "Docker Engine reachable (server version $ver)"
else
fail "Docker Engine not reachable: $ver"
fi
# -------------------------------------------------------------------------
# 2. Coolify helper stack
# -------------------------------------------------------------------------
echo ""
echo "-- Coolify managed-server helpers --"
for c in coolify-proxy coolify-sentinel coolify-cloudflared; do
if state=$(docker inspect "$c" --format '{{.State.Status}} ({{.State.Health.Status}})' 2>/dev/null); then
if [[ "$state" =~ running.*(healthy|running) ]]; then ok "coolify helper: $c -> $state"; else warn "coolify helper: $c -> $state"; fi
else
warn "coolify helper: $c -> NOT FOUND (is this a Coolify-managed server?)"
fi
done
# -------------------------------------------------------------------------
# 3. Linear Void containers
# -------------------------------------------------------------------------
echo ""
echo "-- Linear Void containers --"
required=(linearvoid-playout linearvoid-chat linearvoid-nginx linearvoid-redis linearvoid-db linearvoid-prometheus linearvoid-grafana)
running="$(docker ps --format '{{.Names}}' 2>/dev/null)"
all="$(docker ps -a --format '{{.Names}}\t{{.Status}}' 2>/dev/null)"
for name in "${required[@]}"; do
if grep -qx "$name" <<<"$running"; then
ok "container: $name is running"
elif grep -q "^$name[[:space:]]" <<<"$all"; then
status="$(grep "^$name[[:space:]]" <<<"$all" | head -1)"
fail "container: $name EXISTS but not running -> $status"
else
fail "container: $name MISSING"
fi
done
fi
# ---------------------------------------------------------------------------
# 4. Health endpoints (via Docker host)
# ---------------------------------------------------------------------------
if [[ "$ONLY_CONTAINERS" -eq 0 ]]; then
echo ""
echo "-- Health endpoints --"
checks=(
"chat /health|http://${TARGET}:3000/health"
"nginx /health|http://${TARGET}:8081/health"
"prometheus|http://${TARGET}:9090/-/healthy"
"grafana /api/health|http://${TARGET}:3001/api/health"
)
for entry in "${checks[@]}"; do
name="${entry%%|*}"; url="${entry#*|}"
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time "$TIMEOUT" "$url" 2>/dev/null)
if [[ "$code" == "200" ]]; then ok "$name -> 200"; elif [[ "$code" =~ ^[0-9]{3}$ ]]; then warn "$name -> $code"; else fail "$name -> unreachable"; fi
done
# -------------------------------------------------------------------------
# 5. Public endpoints (routed through the Docker host for reliable TLS)
# -------------------------------------------------------------------------
echo ""
echo "-- Public endpoints (via Docker host) --"
pub=(
"linearvoid.tv app|https://linearvoid.tv/"
"media HLS manifest|https://media.linearvoid.tv/hls/playlist.m3u8"
"coolify dashboard|https://coolify.allucanget.biz/"
)
for entry in "${pub[@]}"; do
name="${entry%%|*}"; url="${entry#*|}"
code=$(docker run --rm curlimages/curl:latest -sS -o /dev/null -w '%{http_code}' --max-time "$TIMEOUT" -L "$url" 2>/dev/null | tr -d '[:space:]')
if [[ "$code" =~ ^[0-9]{3}$ ]]; then ok "$name -> HTTP $code"; else warn "$name -> $code"; fi
done
fi
# ---------------------------------------------------------------------------
echo ""
if [[ "$FAILED" -ne 0 ]]; then echo "RESULT: FAILED"; exit 1; fi
echo "RESULT: OK"
exit 0
scripts/check-tokens.sh
+177
View File
@@ -0,0 +1,177 @@
#!/usr/bin/env bash
# check-tokens.sh — Verify that the credentials in the environment actually work.
# Mirrors scripts/check-tokens.ps1.
#
# Reads .env (or a supplied file) and validates access against live services:
# 1. Gitea container registry (CI_REGISTRY_TOKEN / CI_REGISTRY_USER).
# 2. Cloudflare API token(s) (CLOUDFLARE_API_TOKEN, and any R2 admin token).
# 3. R2/S3 bucket access (R2_ACCESS_KEY_ID + R2_SECRET_ACCESS_KEY) — lists buckets.
# 4. Linear Void admin login (ADMIN_USERNAME / ADMIN_PASSWORD) against the app origin.
#
# NOTE: Runs the checks through the target Docker host (default 192.168.88.40) so
# network + TLS work regardless of this machine's proxy/firewall. Uses the
# curlimages/curl and amazon/aws-cli images, which are pulled on demand.
#
# Usage:
# ./scripts/check-tokens.sh
# ./scripts/check-tokens.sh -e .env
# ./scripts/check-tokens.sh -t 192.168.88.40
#
# Exit code 0 = all credentials valid; 1 = at least one check failed.
set -uo pipefail
ENV_FILE=""
TARGET=192.168.88.40
DOCKER_PORT=2375
TIMEOUT=20
while [[ $# -gt 0 ]]; do
case "$1" in
-e|--env-file) ENV_FILE="$2"; shift 2;;
-t|--target) TARGET="$2"; shift 2;;
-p|--docker-port) DOCKER_PORT="$2"; shift 2;;
--timeout) TIMEOUT="$2"; shift 2;;
*) echo "Unknown option: $1" >&2; exit 2;;
esac
done
export DOCKER_HOST="tcp://${TARGET}:${DOCKER_PORT}"
FAILED=0
fail() { printf ' [FAIL] %s\n' "$1"; FAILED=1; }
ok() { printf ' [OK] %s\n' "$1"; }
warn() { printf ' [WARN] %s\n' "$1"; }
# Resolve env file: explicit, or $PWD/.env, or repo root.
if [[ -z "$ENV_FILE" ]]; then
for cand in "$PWD/.env" "$(dirname "$PWD")/.env"; do
if [[ -f "$cand" ]]; then ENV_FILE="$cand"; break; fi
done
fi
if [[ -z "$ENV_FILE" || ! -f "$ENV_FILE" ]]; then
echo "No .env file found. Pass one with -e." >&2
exit 1
fi
# Parse KEY=VALUE lines into an associative array.
declare -A envs
while IFS= read -r line || [[ -n "$line" ]]; do
if [[ "$line" =~ ^[[:space:]]*([A-Za-z0-9_]+)[[:space:]]*=[[:space:]]*(.*)[[:space:]]*$ ]]; then
envs["${BASH_REMATCH[1]}"]="${BASH_REMATCH[2]}"
fi
done < "$ENV_FILE"
get_val() { [[ -n "${envs[$1]+x}" ]] && echo "${envs[$1]}"; }
# Pull helper images once.
echo "Preparing helper images (curl, aws-cli) on $TARGET..."
docker pull curlimages/curl:latest >/dev/null 2>&1 || true
docker pull amazon/aws-cli:latest >/dev/null 2>&1 || true
# dc_curl URL [bearer] [token] — run curl via the curlimages/curl container.
dc_curl() {
local url="$1" bearer="${2:-}" token="${3:-}"
local -a args=(run --rm curlimages/curl:latest -sS --max-time "$TIMEOUT")
[[ -n "$bearer" ]] && args+=( -H "Authorization: Bearer $bearer" )
[[ -n "$token" ]] && args+=( -H "Authorization: token $token" )
args+=( "$url" )
docker "${args[@]}" 2>&1 | tr -d '\r' | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'
}
# ---------------------------------------------------------------------------
echo "== Linear Void credential / token check =="
echo "Env file : $ENV_FILE"
echo "Docker host: $DOCKER_HOST"
# --- 1. Gitea registry token ---
echo ""
echo "-- Gitea container registry --"
reg_user="$(get_val CI_REGISTRY_USER)"
reg_token="$(get_val CI_REGISTRY_TOKEN)"
if [[ -n "$reg_user" && -n "$reg_token" ]]; then
# Non-sensitive probe: /api/v1/user returns a scope message for token-based creds.
out="$(dc_curl 'https://git.allucanget.biz/api/v1/user' "" "$reg_token")"
if [[ "$out" =~ token.does.not.have.at.least.one.of.required.scope || "$out" =~ '"login"' || "$out" =~ invalid_grant ]]; then
if [[ "$out" =~ invalid_grant|Unauthorized|invalid ]]; then
fail "Gitea registry token rejected ($reg_user): $out"
else
ok "Gitea registry token valid for '$reg_user' (scope: write:package/write:repository)"
fi
else
warn "Gitea registry token response unexpected: $out"
fi
else
warn "CI_REGISTRY_USER / CI_REGISTRY_TOKEN not set; skipping Gitea registry check"
fi
# --- 2. Cloudflare API token ---
echo ""
echo "-- Cloudflare API tokens --"
cf_token="$(get_val CLOUDFLARE_API_TOKEN)"
if [[ -n "$cf_token" ]]; then
out="$(dc_curl 'https://api.cloudflare.com/client/v4/user/tokens/verify' "$cf_token")"
if [[ "$out" =~ '"success":true' ]]; then
if [[ "$out" =~ '"status":"active"' ]]; then ok 'Cloudflare API token valid and ACTIVE'; else warn "Cloudflare API token valid but status: $out"; fi
else
fail "Cloudflare API token rejected: $out"
fi
else
warn "CLOUDFLARE_API_TOKEN not set; skipping Cloudflare check"
fi
# --- 3. Cloudflare R2 / S3 bucket access ---
echo ""
echo "-- Cloudflare R2 (S3-compatible) --"
r2_key="$(get_val R2_ACCESS_KEY_ID)"
r2_secret="$(get_val R2_SECRET_ACCESS_KEY)"
r2_ep="$(get_val R2_ENDPOINT)"
bucket="$(get_val R2_BUCKET_NAME)"
# Use the admin key pair (which has ListBuckets) when present, else the regular key.
list_k="$(get_val R2_ADMIN_ACCESS_KEY_ID)"; list_s="$(get_val R2_ADMIN_SECRET_ACCESS_KEY)"
if [[ -z "$list_k" || -z "$list_s" ]]; then list_k="$r2_key"; list_s="$r2_secret"; fi
if [[ -n "$list_k" && -n "$list_s" && -n "$r2_ep" ]]; then
out="$(docker run --rm --entrypoint sh amazon/aws-cli:latest -c "export AWS_ACCESS_KEY_ID='$list_k'; export AWS_SECRET_ACCESS_KEY='$list_s'; aws s3 ls --endpoint-url '$r2_ep' --region auto 2>&1" 2>&1)"
if grep -q "$bucket" <<<"$out"; then ok "R2 bucket listing OK, '$bucket' present"
elif grep -q 'AccessDenied' <<<"$out"; then fail "R2 listing denied (no ListBuckets permission): $out"
elif grep -qE '^[^[:space:]]+[[:space:]]+' <<<"$out"; then warn "R2 listing returned (unexpected): $out"
else fail "R2 access failed: $out"; fi
else
warn "R2 credentials / endpoint not fully set; skipping R2 check"
fi
# Verify the specific application bucket is reachable with the primary key (read).
if [[ -n "$r2_key" && -n "$r2_secret" && -n "$r2_ep" && -n "$bucket" ]]; then
out="$(docker run --rm --entrypoint sh amazon/aws-cli:latest -c "export AWS_ACCESS_KEY_ID='$r2_key'; export AWS_SECRET_ACCESS_KEY='$r2_secret'; aws s3 ls s3://$bucket/ --endpoint-url '$r2_ep' --region auto --max-keys 5 2>&1 | head -5" 2>&1)"
if grep -q 'AccessDenied' <<<"$out"; then fail "Bucket '$bucket' read denied with primary key"
elif [[ -z "$out" ]]; then ok "Bucket '$bucket' reachable (empty listing, expected)"
elif grep -q 'NoSuchBucket' <<<"$out"; then fail "Bucket '$bucket' NOT FOUND"
else ok "Bucket '$bucket' reachable"; fi
elif [[ -z "$bucket" ]]; then
warn "R2_BUCKET_NAME not set; skipping bucket-level check"
fi
# --- 4. Linear Void admin login ---
echo ""
echo "-- Linear Void admin login --"
admin_user="$(get_val ADMIN_USERNAME)"
admin_pass="$(get_val ADMIN_PASSWORD)"
if [[ -n "$admin_user" && -n "$admin_pass" ]]; then
# Hit the chat backend directly (public origin is caught by nginx SPA fallback).
url="http://${TARGET}:3000/api/admin/login"
body="{\"username\":\"$admin_user\",\"password\":\"$admin_pass\"}"
wrong_body="{\"username\":\"$admin_user\",\"password\":\"__definitely_wrong__\"}"
ok_code="$(docker run --rm curlimages/curl:latest -sS --max-time "$TIMEOUT" -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/json' -d "$body" "$url" 2>/dev/null | tr -d '[:space:]')"
wrong_code="$(docker run --rm curlimages/curl:latest -sS --max-time "$TIMEOUT" -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/json' -d "$wrong_body" "$url" 2>/dev/null | tr -d '[:space:]')"
if [[ "$ok_code" == "200" && "$wrong_code" == "401" ]]; then ok "Admin credentials valid (correct=200, wrong=401)"
else fail "Admin login unexpected (correct=$ok_code, wrong=$wrong_code)"; fi
else
warn "ADMIN_USERNAME / ADMIN_PASSWORD not set; skipping admin check"
fi
# ---------------------------------------------------------------------------
echo ""
if [[ "$FAILED" -ne 0 ]]; then echo "RESULT: FAILED"; exit 1; fi
echo "RESULT: OK (no failing checks)"
exit 0
scripts/loadtest/README.md
+20 -49
View File
@@ -2,66 +2,42 @@
These files back the [scaling/performance playbook](../../docs/operations/scaling-performance-playbook.md).
## WebSocket chat
## WebSocket chat (artillery)
Run with an explicit WebSocket URL, for example:
Run with an explicit WebSocket URL, capturing output to an artifact file:
```bash
WS_URL=wss://linearvoid.tv/ws?username=loadtest&room=main npx artillery run scripts/loadtest/ws-chat.yml
```
Capture output to an artifact file for the worksheet:
```bash
WS_URL=wss://linearvoid.tv/ws?username=loadtest&room=main npx artillery run scripts/loadtest/ws-chat.yml --output artifacts/ws-chat-report.json
```
PowerShell equivalent:
```powershell
$env:WS_URL = 'wss://linearvoid.tv/ws?username=loadtest&room=main'
New-Item -ItemType Directory -Force artifacts | Out-Null
export WS_URL='wss://linearvoid.tv/ws?username=loadtest&room=main'
mkdir -p artifacts
npx artillery run scripts/loadtest/ws-chat.yml --output artifacts/ws-chat-report.json
```
Summarize key counters, rates, and available percentile stats from a saved Artillery JSON report:
```powershell
pwsh -File scripts/loadtest/summarize-artillery.ps1 -ReportFile artifacts/ws-chat-report.json
```bash
scripts/loadtest/summarize-artillery.sh artifacts/ws-chat-report.json
```
## Admin API
## Admin API (k6)
Run with a cookie or session header that can access admin endpoints:
Run with a cookie or session header that can access admin endpoints, capturing machine-readable metrics and a text summary:
```bash
ADMIN_BASE_URL=https://linearvoid.tv ADMIN_SESSION_COOKIE="session=..." k6 run scripts/loadtest/admin-api.js
```
Capture machine-readable metrics and a text summary:
```bash
ADMIN_BASE_URL=https://linearvoid.tv ADMIN_SESSION_COOKIE="session=..." k6 run --summary-export artifacts/admin-api-summary.json scripts/loadtest/admin-api.js > artifacts/admin-api-output.txt
```
PowerShell equivalent:
```powershell
$env:ADMIN_BASE_URL = 'https://linearvoid.tv'
$env:ADMIN_SESSION_COOKIE = 'session=...'
New-Item -ItemType Directory -Force artifacts | Out-Null
k6 run --summary-export artifacts/admin-api-summary.json scripts/loadtest/admin-api.js | Tee-Object -FilePath artifacts/admin-api-output.txt
export ADMIN_BASE_URL='https://linearvoid.tv'
export ADMIN_SESSION_COOKIE='session=...'
mkdir -p artifacts
k6 run --summary-export artifacts/admin-api-summary.json scripts/loadtest/admin-api.js | tee artifacts/admin-api-output.txt
```
The script emits global and per-endpoint thresholds (overview, analytics, chat-log, users) so p95 and error-rate regressions are visible per route.
Summarize endpoint p95 and failure rates from a saved k6 JSON summary:
```powershell
pwsh -File scripts/loadtest/summarize-k6.ps1 -SummaryFile artifacts/admin-api-summary.json
```bash
scripts/loadtest/summarize-k6.sh artifacts/admin-api-summary.json
```
## PowerShell helper
## Runner helper
Use the helper to run one scenario or both and automatically write outputs into a dated artifact folder:
@@ -69,16 +45,11 @@ Use the helper to run one scenario or both and automatically write outputs into
- WebSocket runs also emit `ws-chat-summary.txt` with connection/error counters and available summary percentiles.
- Admin runs also emit `admin-api-endpoint-summary.txt` with per-endpoint p95/error values.
```powershell
pwsh -File scripts/loadtest/run-loadtests.ps1 -Scenario ws -WsUrl 'wss://linearvoid.tv/ws?username=loadtest&room=main'
```
```powershell
pwsh -File scripts/loadtest/run-loadtests.ps1 -Scenario admin -AdminBaseUrl 'https://linearvoid.tv' -AdminSessionCookie 'session=...'
```
```powershell
pwsh -File scripts/loadtest/run-loadtests.ps1 -Scenario all -WsUrl 'wss://linearvoid.tv/ws?username=loadtest&room=main' -AdminBaseUrl 'https://linearvoid.tv' -AdminSessionCookie 'session=...'
```bash
scripts/loadtest/run-loadtests.sh -s ws -w 'wss://linearvoid.tv/ws?username=loadtest&room=main'
scripts/loadtest/run-loadtests.sh -s admin -a 'https://linearvoid.tv' -c 'session=...'
scripts/loadtest/run-loadtests.sh -s all -w 'wss://linearvoid.tv/ws?username=loadtest&room=main' \
-a 'https://linearvoid.tv' -c 'session=...'
```
## Notes
scripts/loadtest/run-loadtests.sh
+72
View File
@@ -0,0 +1,72 @@
#!/usr/bin/env bash
# run-loadtests.sh — Run an artillery (ws) and/or k6 (admin) load-test scenario
# and write outputs into a dated artifacts folder.
# Mirrors scripts/loadtest/run-loadtests.ps1.
#
# Usage:
# ./scripts/loadtest/run-loadtests.sh -s ws -w 'wss://...'
# ./scripts/loadtest/run-loadtests.sh -s admin -a 'https://...' -c 'session=...'
# ./scripts/loadtest/run-loadtests.sh -s all -w 'wss://...' -a 'https://...' -c 'session=...'
set -euo pipefail
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
repo_root="$(cd "$script_dir/../.." && pwd)"
SCENARIO="all"
ARTIFACT_ROOT="artifacts"
WS_URL="${WS_URL:-}"
ADMIN_BASE_URL="${ADMIN_BASE_URL:-}"
ADMIN_SESSION_COOKIE="${ADMIN_SESSION_COOKIE:-}"
usage() {
echo "Usage: $0 [-s ws|admin|all] [-w ws_url] [-a admin_base_url] [-c admin_session_cookie] [-r artifact_root]" >&2
exit 2
}
while [[ $# -gt 0 ]]; do
case "$1" in
-s) SCENARIO="$2"; shift 2;;
-w) WS_URL="$2"; shift 2;;
-a) ADMIN_BASE_URL="$2"; shift 2;;
-c) ADMIN_SESSION_COOKIE="$2"; shift 2;;
-r) ARTIFACT_ROOT="$2"; shift 2;;
-h|--help) usage;;
*) echo "Unknown option: $1" >&2; usage;;
esac
done
case "$SCENARIO" in ws|admin|all) ;; *) echo "Scenario must be ws|admin|all" >&2; exit 2;; esac
if [[ "$SCENARIO" == ws || "$SCENARIO" == all ]] && ! command -v npx >/dev/null 2>&1; then
echo "npx is required for ws scenario but was not found in PATH." >&2; exit 1
fi
if [[ "$SCENARIO" == admin || "$SCENARIO" == all ]] && ! command -v k6 >/dev/null 2>&1; then
echo "k6 is required for admin scenario but was not found in PATH." >&2; exit 1
fi
if [[ "$ARTIFACT_ROOT" = /* ]]; then artifact_base="$ARTIFACT_ROOT"; else artifact_base="$repo_root/$ARTIFACT_ROOT"; fi
artifact_dir="$artifact_base/$(date +%Y-%m-%d_%H%M%S)"
mkdir -p "$artifact_dir"
if [[ "$SCENARIO" == ws || "$SCENARIO" == all ]]; then
[[ -n "$WS_URL" ]] || { echo "WS_URL is required for ws scenario. Pass -w or set \$WS_URL." >&2; exit 1; }
ws_out="$artifact_dir/ws-chat-report.json"
ws_sum="$artifact_dir/ws-chat-summary.txt"
echo "== Running artillery ws scenario =="
WS_URL="$WS_URL" npx artillery run "$script_dir/ws-chat.yml" --output "$ws_out"
"$script_dir/summarize-artillery.sh" "$ws_out" | tee "$ws_sum"
fi
if [[ "$SCENARIO" == admin || "$SCENARIO" == all ]]; then
[[ -n "$ADMIN_BASE_URL" ]] || { echo "ADMIN_BASE_URL is required for admin scenario. Pass -a or set \$ADMIN_BASE_URL." >&2; exit 1; }
[[ -n "$ADMIN_SESSION_COOKIE" ]] || { echo "ADMIN_SESSION_COOKIE is required for admin scenario. Pass -c or set \$ADMIN_SESSION_COOKIE." >&2; exit 1; }
admin_sum="$artifact_dir/admin-api-summary.json"
admin_out="$artifact_dir/admin-api-output.txt"
admin_ep_sum="$artifact_dir/admin-api-endpoint-summary.txt"
echo "== Running k6 admin scenario =="
ADMIN_BASE_URL="$ADMIN_BASE_URL" ADMIN_SESSION_COOKIE="$ADMIN_SESSION_COOKIE" \
k6 run --summary-export "$admin_sum" "$script_dir/admin-api.js" | tee "$admin_out"
"$script_dir/summarize-k6.sh" "$admin_sum" | tee "$admin_ep_sum"
fi
echo "Artifacts written to $artifact_dir"
scripts/loadtest/summarize-artillery.sh
+54
View File
@@ -0,0 +1,54 @@
#!/usr/bin/env bash
# summarize-artillery.sh — Extract counters, rates and percentile stats from an
# Artillery JSON report. Mirrors scripts/loadtest/summarize-artillery.ps1.
# Usage: ./scripts/loadtest/summarize-artillery.sh <artillery-report.json>
set -euo pipefail
if [[ $# -lt 1 ]]; then
echo "Usage: $0 <artillery-report.json>" >&2
exit 2
fi
report_file="$1"
[[ -f "$report_file" ]] || { echo "Report file not found: $report_file" >&2; exit 1; }
REPORT_FILE="$report_file" node <<'EOF'
const fs = require('fs');
const report = JSON.parse(fs.readFileSync(process.env.REPORT_FILE, 'utf8'));
const aggregate = report.aggregate;
if (!aggregate) throw new Error('Artillery report is missing aggregate data.');
const get = (obj, name) => (obj && obj[name] !== undefined) ? obj[name] : undefined;
const counters = aggregate.counters || {};
const rates = aggregate.rates || {};
const summaries = aggregate.summaries || {};
const created = Number(get(counters, 'vusers.created') || 0);
const failed = Number(get(counters, 'vusers.failed') || 0);
const completed = Number(get(counters, 'vusers.completed') || 0);
const errorRate = created > 0 ? +((failed / created) * 100).toFixed(2) : 0;
console.log('=== Artillery WebSocket Summary ===');
console.log(`source: ${process.env.REPORT_FILE}`);
if (report.timestamp) console.log(`timestamp: ${report.timestamp}`);
console.log(`vusers.created: ${created}`);
console.log(`vusers.completed: ${completed}`);
console.log(`vusers.failed: ${failed}`);
console.log(`vusers.failure_rate_percent: ${errorRate}`);
for (const metric of ['ws.messages_sent', 'ws.messages_received', 'ws.ping', 'plugins.metrics-by-endpoint.response_time']) {
const v = get(counters, metric);
if (v !== undefined) console.log(`${metric}: ${v}`);
}
for (const [k, v] of Object.entries(rates || {})) console.log(`rate.${k}: ${v}`);
const fmt = (prefix, stats) => {
if (!stats) return;
const parts = [];
for (const name of ['count', 'min', 'max', 'mean', 'median', 'p95', 'p99']) {
const v = get(stats, name);
if (v !== undefined) parts.push(`${name}=${v}`);
}
if (parts.length) console.log(`${prefix}: ${parts.join(', ')}`);
};
for (const [k, v] of Object.entries(summaries)) fmt(`summary.${k}`, v);
EOF
scripts/loadtest/summarize-k6.sh
+43
View File
@@ -0,0 +1,43 @@
#!/usr/bin/env bash
# summarize-k6.sh — Extract per-endpoint p95 and failure rates from a k6 JSON
# summary export. Mirrors scripts/loadtest/summarize-k6.ps1.
# Usage: ./scripts/loadtest/summarize-k6.sh <k6-summary.json>
set -euo pipefail
if [[ $# -lt 1 ]]; then
echo "Usage: $0 <k6-summary.json>" >&2
exit 2
fi
summary_file="$1"
[[ -f "$summary_file" ]] || { echo "Summary file not found: $summary_file" >&2; exit 1; }
SUMMARY_FILE="$summary_file" node <<'EOF'
const fs = require('fs');
const summary = JSON.parse(fs.readFileSync(process.env.SUMMARY_FILE, 'utf8'));
const metrics = summary.metrics;
if (!metrics) throw new Error('Invalid k6 summary file: missing metrics object.');
const val = (m, key) => (m && m.values && m.values[key] !== undefined) ? Number(m.values[key]) : undefined;
const globalDuration = val(metrics.http_req_duration, 'p(95)');
const globalFailRate = val(metrics.http_req_failed, 'rate');
console.log(`k6 Summary: ${process.env.SUMMARY_FILE}`);
if (globalDuration !== undefined) console.log(`Global p95 duration (ms): ${globalDuration.toFixed(2)}`);
if (globalFailRate !== undefined) console.log(`Global failure rate: ${(globalFailRate * 100).toFixed(2)}%`);
console.log('');
const endpoints = ['overview', 'analytics', 'chat_log', 'users'];
const rows = endpoints.map((ep) => {
const dP95 = val(metrics[`http_req_duration{endpoint:${ep}}`], 'p(95)');
const fRate = val(metrics[`http_req_failed{endpoint:${ep}}`], 'rate');
return {
Endpoint: ep,
P95Ms: dP95 !== undefined ? dP95.toFixed(2) : 'n/a',
FailurePct: fRate !== undefined ? (fRate * 100).toFixed(4) : 'n/a',
};
});
const pad = (s, n) => String(s).padEnd(n);
console.log(pad('ENDPOINT', 12) + pad('P95_MS', 10) + 'FAILURE_PCT');
for (const r of rows) console.log(pad(r.Endpoint, 12) + pad(r.P95Ms, 10) + r.FailurePct);
EOF