Refactor load testing scripts to support Bash and PowerShell
- Updated README.md to reflect changes in load testing commands and added usage examples for Bash scripts. - Converted PowerShell load testing scripts to Bash equivalents for better cross-platform compatibility. - Added new Bash scripts: run-loadtests.sh, summarize-artillery.sh, and summarize-k6.sh to mirror existing PowerShell functionality. - Updated admin-api.js and ws-chat.yml for consistency and improved performance metrics reporting. - Enhanced error handling and output formatting in summary scripts for better readability.
This commit is contained in:
1 parent
9f60087a4c
commit
33b6296c5e
22 files changed
+2498
-3949
No files matched your search
@@ -6,10 +6,12 @@
|
|||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
||||||
|
- **Bash equivalents of Windows scripts**: Added `scripts/cache-hit-ratio.sh`, `scripts/check-env.sh`, `scripts/check-tokens.sh`, and `scripts/loadtest/run-loadtests.sh`, `scripts/loadtest/summarize-artillery.sh`, `scripts/loadtest/summarize-k6.sh` — Linux/macOS ports of the existing PowerShell scripts (the `.ps1` files are retained).
|
||||||
- **Transient presence flash**: Join/leave announcements now appear as a 3-second flash at the top of the chat panel (`#chat-flash`) with a fade-out, instead of permanent chat rows. Presence indicators still update from the same events.
|
- **Transient presence flash**: Join/leave announcements now appear as a 3-second flash at the top of the chat panel (`#chat-flash`) with a fade-out, instead of permanent chat rows. Presence indicators still update from the same events.
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|
||||||
|
- **Docs condensed and de-Windows'd**: Removed Windows/PowerShell-specific instructions from the docs and condensed the longest documents into tighter, bash-first guides — `SETUP.md`, `DEPLOYMENT.md`, `API.md`, `MONITORING.md`, `STREAMING.md`, `COOLIFY.md`, `incident-runbook.md`, `operations/scaling-performance-playbook.md`, `architecture/README.md`, and the Prometheus metrics inventory. `scripts/loadtest/README.md` now documents the bash load-test helpers.
|
||||||
- **System messages no longer persisted**: The chat backend no longer stores `system`-type messages; chat history and the admin chat-log table only contain real chat messages (moderation actions remain in the audit log).
|
- **System messages no longer persisted**: The chat backend no longer stores `system`-type messages; chat history and the admin chat-log table only contain real chat messages (moderation actions remain in the audit log).
|
||||||
- **Reconnect grace period**: Disconnects schedule a delayed "left" announcement (5s); a reconnect within 5s cancels it and skips "joined", eliminating join/leave spam on brief reconnects.
|
- **Reconnect grace period**: Disconnects schedule a delayed "left" announcement (5s); a reconnect within 5s cancels it and skips "joined", eliminating join/leave spam on brief reconnects.
|
||||||
- **HLS segment duration 2s**: Default `HLS_SEGMENT_DURATION` changed from 6 to 2 and `HLS_WINDOW_SIZE` from 24 to 12 (2s × 12 = 24s live window), reducing end-to-end latency; the 2s GOP was already in place.
|
- **HLS segment duration 2s**: Default `HLS_SEGMENT_DURATION` changed from 6 to 2 and `HLS_WINDOW_SIZE` from 24 to 12 (2s × 12 = 24s live window), reducing end-to-end latency; the 2s GOP was already in place.
|
||||||
|
|||||||
@@ -213,11 +213,13 @@ One-off and CI-friendly scripts for video preparation, R2 sync, backups, and pla
|
|||||||
|
|
||||||
- loadtest/
|
- loadtest/
|
||||||
- [admin-api.js](scripts/loadtest/admin-api.js) — admin API load test script
|
- [admin-api.js](scripts/loadtest/admin-api.js) — admin API load test script
|
||||||
- [run-loadtests.ps1](scripts/loadtest/run-loadtests.ps1) — PowerShell script to run load tests
|
- [run-loadtests.sh](scripts/loadtest/run-loadtests.sh) — bash script to run ws and/or admin load tests
|
||||||
- [summarize-artillery.ps1](scripts/loadtest/summarize-artillery.ps1) — PowerShell script to summarize Artillery load test results
|
- [summarize-artillery.sh](scripts/loadtest/summarize-artillery.sh) — summarize Artillery JSON results
|
||||||
- [summarize-k6.ps1](scripts/loadtest/summarize-k6.ps1) — PowerShell script to summarize k6 load test results
|
- [summarize-k6.sh](scripts/loadtest/summarize-k6.sh) — summarize k6 JSON results
|
||||||
- [ws-chat.yml](scripts/loadtest/ws-chat.yml) — WebSocket chat load test configuration
|
- [ws-chat.yml](scripts/loadtest/ws-chat.yml) — WebSocket chat load test configuration
|
||||||
|
|
||||||
|
PowerShell equivalents (`*.ps1`) of the three load-test scripts are kept alongside them.
|
||||||
|
|
||||||
## Getting Started
|
## Getting Started
|
||||||
|
|
||||||
> **Note:** The `docker compose up` below targets local development. The test lab (192.168.88.40) receives automated CD deploys. Production Hetzner VPS is not yet provisioned — see `DEPLOYMENT.md`.
|
> **Note:** The `docker compose up` below targets local development. The test lab (192.168.88.40) receives automated CD deploys. Production Hetzner VPS is not yet provisioned — see `DEPLOYMENT.md`.
|
||||||
|
|||||||
+68
-216
@@ -1,216 +1,101 @@
|
|||||||
# API Reference
|
# Linear Void API Reference
|
||||||
|
|
||||||
This document describes the currently implemented API surfaces in Linear Void.
|
Base URLs: Public/admin app `https://linearvoid.tv` · WebSocket `wss://linearvoid.tv/ws` · Local fallback `http://localhost` / `ws://localhost/ws`
|
||||||
|
|
||||||
## Base URLs
|
## Authentication
|
||||||
|
|
||||||
- Public app and admin: <https://linearvoid.tv>
|
Admin REST endpoints use a cookie session:
|
||||||
- WebSocket endpoint: <wss://linearvoid.tv/ws>
|
|
||||||
- Local fallback: <http://localhost> and <ws://localhost/ws>
|
|
||||||
|
|
||||||
## Authentication Model
|
- Cookie: `linearvoid_admin_session` — `HttpOnly`, `SameSite=Strict`, `Secure` when request is HTTPS (or `X-Forwarded-Proto=https`)
|
||||||
|
- Session TTL: `ADMIN_SESSION_TTL_HOURS` (default 12h)
|
||||||
Admin REST endpoints use a cookie session.
|
|
||||||
|
|
||||||
- Cookie name: linearvoid_admin_session
|
|
||||||
- Cookie attributes: HttpOnly, SameSite=Strict, Secure when request is HTTPS (or X-Forwarded-Proto=https)
|
|
||||||
- Session TTL: controlled by ADMIN_SESSION_TTL_HOURS (default 12h)
|
|
||||||
|
|
||||||
## Chat WebSocket API
|
## Chat WebSocket API
|
||||||
|
|
||||||
### Connect
|
Connect: `GET /ws?username={name}&room={room}` — client/server exchange JSON messages.
|
||||||
|
|
||||||
Endpoint:
|
| Message (client→server) | Field | Notes |
|
||||||
|
| ----------------------- | -------------------------------------------------------------------------------- | ----- |
|
||||||
|
| `type` | `public` for room chat, `private` for DM routing; `system` for moderation/events | |
|
||||||
|
| `room` | `main`; private DMs use `dm:userA-userB` (usernames sorted) | |
|
||||||
|
| `username` | sender | |
|
||||||
|
| `text` | message body | |
|
||||||
|
|
||||||
- GET /ws?username={name}&room={room}
|
Server→client envelope adds `"timestamp"` (ISO-8601, e.g. `2026-08-04T12:34:56Z`).
|
||||||
|
|
||||||
Client sends JSON messages over the socket.
|
|
||||||
|
|
||||||
### Message Envelope (client -> server)
|
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{ "room": "main", "username": "viewer123", "text": "hello world", "type": "public" }
|
||||||
"room": "main",
|
|
||||||
"username": "viewer123",
|
|
||||||
"text": "hello world",
|
|
||||||
"type": "public"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Notes:
|
|
||||||
|
|
||||||
- type is usually public for room chat and private for DM room routing.
|
|
||||||
- Private room names are formatted like dm:userA-userB (sorted usernames).
|
|
||||||
- Moderation/system events are sent as type=system messages.
|
|
||||||
|
|
||||||
### Message Envelope (server -> client)
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"room": "main",
|
|
||||||
"username": "viewer123",
|
|
||||||
"text": "hello world",
|
|
||||||
"type": "public",
|
|
||||||
"timestamp": "2026-08-04T12:34:56Z"
|
|
||||||
}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Admin Session API
|
## Admin Session API
|
||||||
|
|
||||||
### Login
|
| Method & Path | Auth | Description |
|
||||||
|
| ------------------------ | ----------------------------------- | ----------------------------- |
|
||||||
|
| `POST /api/admin/login` | none | Authenticate |
|
||||||
|
| `POST /api/admin/logout` | optional (clears cookie either way) | Logout |
|
||||||
|
| `GET /api/admin/session` | cookie required | Session check; 401 if invalid |
|
||||||
|
|
||||||
- POST /api/admin/login
|
Login request:
|
||||||
- Auth: none
|
|
||||||
|
|
||||||
Request:
|
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{ "username": "admin", "password": "your-password" }
|
||||||
"username": "admin",
|
|
||||||
"password": "your-password"
|
|
||||||
}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Success response:
|
Login/logout/session success response (session returns 401 on failure):
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{ "authenticated": true, "username": "admin" }
|
||||||
"authenticated": true,
|
|
||||||
"username": "admin"
|
|
||||||
}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Logout
|
Logout response: `{ "authenticated": false }`
|
||||||
|
|
||||||
- POST /api/admin/logout
|
## Admin Data API (auth required)
|
||||||
- Auth: optional (clears cookie either way)
|
|
||||||
|
|
||||||
Response:
|
| Method & Path | Returns |
|
||||||
|
| ----------------------------------------- | ---------------------------------------------------------------------------------------------------- |
|
||||||
|
| `GET /api/admin/overview` | Aggregate counters + runtime/system snapshot |
|
||||||
|
| `GET /api/admin/playlist` | Playlist |
|
||||||
|
| `PUT /api/admin/playlist` | Set playlist (body: `{ "content": "file '/app/media/video1.mp4'\nfile '/app/media/video2.mp4'\n" }`) |
|
||||||
|
| `GET /api/admin/schedule` | Parsed playlist entries + matched media metadata/runtime |
|
||||||
|
| `GET /api/admin/moderation/logs?limit=50` | Moderation logs |
|
||||||
|
| `POST /api/admin/moderation/actions` | Perform moderation action |
|
||||||
|
| `GET /api/admin/analytics` | Totals (messages, moderation actions, rooms) + top room summary |
|
||||||
|
| `GET /api/admin/system` | Health snapshot (chat/playout/redis/database) + config metadata |
|
||||||
|
| `GET /api/admin/users` | Presence users (room + username) when Redis available |
|
||||||
|
|
||||||
|
Moderation action request:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{ "action": "mute", "target": "username-or-id", "moderator": "admin", "reason": "spam", "room": "main" }
|
||||||
"authenticated": false
|
|
||||||
}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Session Check
|
|
||||||
|
|
||||||
- GET /api/admin/session
|
|
||||||
- Auth: cookie required
|
|
||||||
|
|
||||||
Success response:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"authenticated": true,
|
|
||||||
"username": "admin"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Failure:
|
|
||||||
|
|
||||||
- 401 unauthorized
|
|
||||||
|
|
||||||
## Admin Data API
|
|
||||||
|
|
||||||
All endpoints below require admin authentication.
|
|
||||||
|
|
||||||
### Overview
|
|
||||||
|
|
||||||
- GET /api/admin/overview
|
|
||||||
|
|
||||||
Returns aggregate counters and runtime/system snapshot fields.
|
|
||||||
|
|
||||||
### Playlist
|
|
||||||
|
|
||||||
- GET /api/admin/playlist
|
|
||||||
- PUT /api/admin/playlist
|
|
||||||
|
|
||||||
PUT request:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"content": "file '/app/media/video1.mp4'\nfile '/app/media/video2.mp4'\n"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Schedule
|
|
||||||
|
|
||||||
- GET /api/admin/schedule
|
|
||||||
|
|
||||||
Returns parsed playlist entries plus matched media metadata/runtime.
|
|
||||||
|
|
||||||
### Moderation
|
|
||||||
|
|
||||||
- GET /api/admin/moderation/logs?limit=50
|
|
||||||
- POST /api/admin/moderation/actions
|
|
||||||
|
|
||||||
Action request:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"action": "mute",
|
|
||||||
"target": "username-or-id",
|
|
||||||
"moderator": "admin",
|
|
||||||
"reason": "spam",
|
|
||||||
"room": "main"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Analytics
|
|
||||||
|
|
||||||
- GET /api/admin/analytics
|
|
||||||
|
|
||||||
Returns totals (messages, moderation actions, rooms) and top room summary.
|
|
||||||
|
|
||||||
### System
|
|
||||||
|
|
||||||
- GET /api/admin/system
|
|
||||||
|
|
||||||
Returns service health snapshot for chat/playout/redis/database and config metadata.
|
|
||||||
|
|
||||||
### Active Users
|
|
||||||
|
|
||||||
- GET /api/admin/users
|
|
||||||
|
|
||||||
Returns currently discovered presence users (room + username) when Redis is available.
|
|
||||||
|
|
||||||
## Media Library API
|
## Media Library API
|
||||||
|
|
||||||
### Media Collection
|
| Method & Path | Description |
|
||||||
|
| ------------------------------ | ------------------------------------------------- |
|
||||||
|
| `GET /api/admin/media` | List media |
|
||||||
|
| `POST /api/admin/media` | Multipart upload (legacy), form field name `file` |
|
||||||
|
| `GET /api/admin/media/{id}` | Get item |
|
||||||
|
| `PUT /api/admin/media/{id}` | Update item |
|
||||||
|
| `DELETE /api/admin/media/{id}` | Delete item |
|
||||||
|
|
||||||
- GET /api/admin/media
|
PUT media body:
|
||||||
- POST /api/admin/media (multipart upload, legacy path)
|
|
||||||
|
|
||||||
Multipart form field name: file
|
|
||||||
|
|
||||||
### Media Item
|
|
||||||
|
|
||||||
- GET /api/admin/media/{id}
|
|
||||||
- PUT /api/admin/media/{id}
|
|
||||||
- DELETE /api/admin/media/{id}
|
|
||||||
|
|
||||||
PUT request:
|
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{ "status": "ready", "title": "My Clip", "tags": "music,live" }
|
||||||
"status": "ready",
|
|
||||||
"title": "My Clip",
|
|
||||||
"tags": "music,live"
|
|
||||||
}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Resumable Upload API
|
## Resumable Upload API
|
||||||
|
|
||||||
Maximum file size: 2GB
|
Max file size: **2GB** · Default chunk size: **8MB**
|
||||||
|
|
||||||
Default chunk size: 8MB
|
| Method & Path | Description |
|
||||||
|
| ------------------------------------------------------------------ | ----------------------------------------------------------------- |
|
||||||
|
| `POST /api/admin/media/uploads` | Initialize or resume session |
|
||||||
|
| `GET /api/admin/media/uploads/{uploadId}` | Check upload status |
|
||||||
|
| `PUT /api/admin/media/uploads/{uploadId}/chunk?index={chunkIndex}` | Upload chunk (`Content-Type: application/octet-stream`) |
|
||||||
|
| `POST /api/admin/media/uploads/{uploadId}/complete` | Complete upload → returns media listing (`files` + `generatedAt`) |
|
||||||
|
|
||||||
### Initialize or Resume Session
|
Initialize request:
|
||||||
|
|
||||||
- POST /api/admin/media/uploads
|
|
||||||
|
|
||||||
Request:
|
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
@@ -223,7 +108,7 @@ Request:
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
Response:
|
Status envelope (returned by init, status check, and chunk upload):
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
@@ -237,61 +122,33 @@ Response:
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
### Check Upload Status
|
## Metrics & Health
|
||||||
|
|
||||||
- GET /api/admin/media/uploads/{uploadId}
|
- `GET /metrics` (chat) — Prometheus: connected clients, stored messages, moderation actions, Redis indicators, active admin sessions, build info
|
||||||
|
- `GET /health` — chat service, playout service, and nginx reverse proxy each expose it
|
||||||
Returns same status envelope as initialize.
|
|
||||||
|
|
||||||
### Upload Chunk
|
|
||||||
|
|
||||||
- PUT /api/admin/media/uploads/{uploadId}/chunk?index={chunkIndex}
|
|
||||||
- Content-Type: application/octet-stream
|
|
||||||
|
|
||||||
Response returns updated upload status envelope.
|
|
||||||
|
|
||||||
### Complete Upload
|
|
||||||
|
|
||||||
- POST /api/admin/media/uploads/{uploadId}/complete
|
|
||||||
|
|
||||||
On success returns admin media listing payload (files + generatedAt).
|
|
||||||
|
|
||||||
## Metrics and Health
|
|
||||||
|
|
||||||
### Chat metrics
|
|
||||||
|
|
||||||
- GET /metrics
|
|
||||||
|
|
||||||
Prometheus exposition includes connected clients, stored messages, moderation actions, Redis indicators, active admin sessions, and build info.
|
|
||||||
|
|
||||||
### Health
|
|
||||||
|
|
||||||
- GET /health on chat service
|
|
||||||
- GET /health on playout service
|
|
||||||
- GET /health on nginx reverse proxy
|
|
||||||
|
|
||||||
## HTTP Status Conventions
|
## HTTP Status Conventions
|
||||||
|
|
||||||
| Code | Meaning | When |
|
| Code | Meaning | When |
|
||||||
| ---- | --------------------- | ----------------------------------------------------------- |
|
| ---- | --------------------- | ----------------------------------------------------------- |
|
||||||
| 200 | OK | Successful GET, PUT, or action |
|
| 200 | OK | Successful GET, PUT, action |
|
||||||
| 201 | Created | Successful POST (upload init, override created) |
|
| 201 | Created | Successful POST (upload init, override created) |
|
||||||
| 204 | No Content | Successful DELETE, CORS preflight |
|
| 204 | No Content | Successful DELETE, CORS preflight |
|
||||||
| 400 | Bad Request | Malformed JSON, missing required fields, invalid input |
|
| 400 | Bad Request | Malformed JSON, missing required fields, invalid input |
|
||||||
| 401 | Unauthorized | Missing or expired admin session |
|
| 401 | Unauthorized | Missing/expired admin session |
|
||||||
| 403 | Forbidden | Valid session but insufficient permissions |
|
| 403 | Forbidden | Valid session, insufficient permissions |
|
||||||
| 404 | Not Found | Resource doesn't exist |
|
| 404 | Not Found | Resource doesn't exist |
|
||||||
| 405 | Method Not Allowed | Wrong HTTP method for endpoint |
|
| 405 | Method Not Allowed | Wrong method for endpoint |
|
||||||
| 409 | Conflict | Schedule override overlap, upload ID mismatch |
|
| 409 | Conflict | Schedule override overlap, upload ID mismatch |
|
||||||
| 413 | Payload Too Large | Upload exceeds 2 GB limit |
|
| 413 | Payload Too Large | Upload exceeds 2 GB limit |
|
||||||
| 422 | Unprocessable | File fails normalization check (codec, fps, resolution) |
|
| 422 | Unprocessable | File fails normalization check (codec, fps, resolution) |
|
||||||
| 429 | Too Many Requests | Rate limit exceeded (chat messages, WS connections, login) |
|
| 429 | Too Many Requests | Rate limit exceeded (chat, WS, login) |
|
||||||
| 500 | Internal Server Error | Unexpected server failure |
|
| 500 | Internal Server Error | Unexpected server failure |
|
||||||
| 503 | Service Unavailable | Redis unavailable for user actions, playout health degraded |
|
| 503 | Service Unavailable | Redis unavailable for user actions, playout health degraded |
|
||||||
|
|
||||||
## Rate Limiting
|
## Rate Limiting
|
||||||
|
|
||||||
All limits are per-IP (from `X-Forwarded-For` or `RemoteAddr`) unless noted.
|
Per-IP (from `X-Forwarded-For` or `RemoteAddr`) unless noted.
|
||||||
|
|
||||||
| Resource | Limit | Window | Response |
|
| Resource | Limit | Window | Response |
|
||||||
| --------------------- | ---------------------------------- | ---------------------------------------- | ------------------------------------------------------ |
|
| --------------------- | ---------------------------------- | ---------------------------------------- | ------------------------------------------------------ |
|
||||||
@@ -299,9 +156,4 @@ All limits are per-IP (from `X-Forwarded-For` or `RemoteAddr`) unless noted.
|
|||||||
| WebSocket connections | 5 | 60s | 429 + `"too many connections from this IP"` |
|
| WebSocket connections | 5 | 60s | 429 + `"too many connections from this IP"` |
|
||||||
| Admin login | 5 | 5min | 429 + `"too many login attempts, try again later"` |
|
| Admin login | 5 | 5min | 429 + `"too many login attempts, try again later"` |
|
||||||
|
|
||||||
Rate limit headers on applicable responses:
|
Rate limit headers (applicable responses): `X-RateLimit-Limit` (max requests/window), `Retry-After` (seconds until reset). 413 = payload too large; 500 = internal server error.
|
||||||
|
|
||||||
- `X-RateLimit-Limit`: maximum requests per window
|
|
||||||
- `Retry-After`: seconds until window resets
|
|
||||||
- 413: payload too large
|
|
||||||
- 500: internal server error
|
|
||||||
+79
-160
@@ -1,229 +1,148 @@
|
|||||||
# Coolify Deployment
|
# Coolify Deployment
|
||||||
|
|
||||||
**Last Updated**: 2026-08-28
|
*Last Updated: 2026-08-28*
|
||||||
|
|
||||||
This guide describes an alternative deployment path for Linear Void using
|
Alternative deployment path for Linear Void using [Coolify](https://github.com/coollabsio/coolify/tree/v4.3.14) (v4.3.14), an open-source self-hostable PaaS that manages servers, apps, and databases over SSH. Drop-in alternative to the flows in [SETUP.md](./SETUP.md): local dev (`docker-compose.yml`) and test-lab/prod (`docker-compose.prod.yml` via Gitea Actions CD). Coolify deploys the same Compose stack through a web UI and its own Traefik reverse proxy for automatic HTTPS, domain routing, and one-click redeploys (no manual `DOCKER_HOST`).
|
||||||
[Coolify](https://github.com/coollabsio/coolify/tree/v4.3.14) (v4.3.14) — an
|
|
||||||
open-source, self-hostable PaaS that lets you manage servers, applications, and
|
|
||||||
databases over SSH without vendor lock-in.
|
|
||||||
|
|
||||||
Coolify is a drop-in alternative to the two existing deployment flows described
|
## Why Coolify
|
||||||
in [SETUP.md](./SETUP.md):
|
|
||||||
|
|
||||||
1. **Local development** (`docker-compose.yml`) — builds from source on your
|
- **Single control plane**: manage playout, chat, nginx, redis, db (plus exporters/monitoring) from one dashboard.
|
||||||
workstation.
|
- **Automatic TLS**: Let's Encrypt provisioning/renewal per app/domain, replacing the manual Cloudflare + nginx origin TLS setup for the app hostname.
|
||||||
2. **Test-lab / production** (`docker-compose.prod.yml`) — pulls prebuilt images
|
- **No vendor lock-in**: config lives on your server; containers keep running if you stop using Coolify — you only lose automation.
|
||||||
from the Gitea registry and is driven by the Gitea Actions CD workflow.
|
- **Simpler cutover**: on Hetzner VPS provisioning, point Coolify at the new server and redeploy instead of editing CD `DOCKER_HOST`.
|
||||||
|
|
||||||
With Coolify, the same Docker Compose stack is deployed through a web UI and
|
> Replaces the *deployment orchestration* layer only. Cloudflare DNS, R2 media bucket, cache rules, and the edge Worker from [SETUP.md](./SETUP.md) are unchanged.
|
||||||
Coolify's own reverse proxy (Traefik), giving you automatic HTTPS, domain
|
|
||||||
routing, and one-click redeploys without managing `DOCKER_HOST` manually.
|
|
||||||
|
|
||||||
## Why use Coolify?
|
|
||||||
|
|
||||||
- **Single control plane**: manage the playout, chat, nginx, redis, and db
|
|
||||||
services (plus exporters and monitoring) from one dashboard.
|
|
||||||
- **Automatic TLS**: Coolify provisions and renews Let's Encrypt certificates
|
|
||||||
per application/domain, replacing the manual Cloudflare + nginx origin TLS
|
|
||||||
dance for the app hostname.
|
|
||||||
- **No vendor lock-in**: all configuration lives on your server. If you stop
|
|
||||||
using Coolify, the running containers keep working; you only lose the
|
|
||||||
automation.
|
|
||||||
- **Simpler cutover**: when the Hetzner VPS is provisioned, point Coolify at the
|
|
||||||
new server and redeploy instead of editing CD `DOCKER_HOST` values.
|
|
||||||
|
|
||||||
> **Note**: Coolify replaces the _deployment orchestration_ layer only. Cloudflare
|
|
||||||
> DNS, the R2 media bucket, cache rules, and the edge Worker described in
|
|
||||||
> [SETUP.md](./SETUP.md) still apply and are unchanged.
|
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
- A Linux server running **Docker Engine** (any distro Coolify supports).
|
- Linux server with **Docker Engine** (any supported distro).
|
||||||
- **SSH access** (root or a user with passwordless `sudo`) from the machine
|
- **SSH access** (root or passwordless `sudo`) from the Coolify install machine — Coolify provisions the target over SSH.
|
||||||
where you run the Coolify install — Coolify uses SSH to provision the target
|
- **2 GB RAM / 20 GB disk** minimum for Coolify itself (plus resources for the stack).
|
||||||
server.
|
- A domain pointed at the server for HTTPS (e.g. `linearvoid.tv`).
|
||||||
- At least **2 GB RAM / 20 GB disk** for the Coolify instance itself (plus
|
- Ports `80` and `443` reachable for Traefik.
|
||||||
additional resources for the Linear Void stack).
|
- Gitea registry credentials (`CI_REGISTRY_USER` / `CI_REGISTRY_TOKEN`) if pulling prebuilt `playout`, `chat`, `nginx` images.
|
||||||
- A domain name pointed at the server for HTTPS (e.g. `linearvoid.tv`).
|
|
||||||
- Ports `80` and `443` reachable on the server for Traefik.
|
|
||||||
- The Gitea registry credentials (`CI_REGISTRY_USER` / `CI_REGISTRY_TOKEN`) if
|
|
||||||
you intend to pull the prebuilt `playout`, `chat`, and `nginx` images.
|
|
||||||
|
|
||||||
## Installation
|
## Installation
|
||||||
|
|
||||||
Install Coolify on a dedicated server (or alongside your stack — but a dedicated
|
Recommended on a dedicated server:
|
||||||
Coolify server is the recommended layout):
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -fsSL https://cdn.coollabs.io/coolify/install.sh | bash
|
curl -fsSL https://cdn.coollabs.io/coolify/install.sh | bash
|
||||||
```
|
```
|
||||||
|
|
||||||
The installer boots a Docker-based Coolify instance and prints the URL and
|
The installer boots a Docker-based Coolify instance and prints the URL and default credentials. Complete first-run setup in the dashboard. For non-interactive/offline installs, see the [installation docs](https://coolify.io/docs/installation) and [install script source](https://github.com/coollabsio/coolify/blob/v4.3.14/scripts/install.sh).
|
||||||
default credentials. Open the dashboard in your browser and complete first-run
|
|
||||||
setup.
|
|
||||||
|
|
||||||
For non-interactive or offline installs, see the
|
|
||||||
[installation docs](https://coolify.io/docs/installation) and the
|
|
||||||
[install script source](https://github.com/coollabsio/coolify/blob/v4.3.14/scripts/install.sh).
|
|
||||||
|
|
||||||
## Add the Linear Void server
|
## Add the Linear Void server
|
||||||
|
|
||||||
1. In the Coolify dashboard, go to **Servers → Add**.
|
1. **Servers → Add**.
|
||||||
2. Choose your provider (or **Localhost** if Coolify runs on the same machine
|
2. Choose your provider (or **Localhost** if Coolify runs on the same host as the stack).
|
||||||
as the stack).
|
3. Provide SSH details; for a remote host Coolify validates and installs its agent.
|
||||||
3. Provide the SSH details. For a remote host, Coolify validates the connection
|
4. Confirm the server shows **Reachable** before creating resources.
|
||||||
and installs its agent.
|
|
||||||
4. Verify the server shows as **Reachable** before creating resources.
|
|
||||||
|
|
||||||
### Current topology (verified 2026-08-30)
|
### Current topology (verified 2026-08-30)
|
||||||
|
|
||||||
The active test-lab setup is **two-tier**:
|
Active test-lab setup is **two-tier**:
|
||||||
|
|
||||||
| Component | Host | Environment | Exposes | Docker TCP | Notes |
|
| Component | Host | Exposes | Docker TCP | Notes |
|
||||||
| --------------------- | --------------- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|---|---|---|---|---|
|
||||||
| **Coolify dashboard** | `192.168.88.18` | Coolify; docker container; coolify dashboard at `8000` | `https://coolify.allucanget.biz` via Cloudflare tunnel (`coolify-cloudflared`) | No | Coolify manages target servers over SSH |
|
| **Coolify dashboard** | `192.168.88.18` | `https://coolify.allucanget.biz` via Cloudflare tunnel (`coolify-cloudflared`); dashboard at `8000` | No | Manages target servers over SSH |
|
||||||
| **Deployment target** | `192.168.88.40` | Only Linear Void deploys here | `coolify-proxy` = Traefik on 80/443, Port 8000 on this host is bound by Portainer, not Coolify. | Yes | Contains Coolify's per-server helpers (`coolify-proxy`, `coolify-sentinel`, `coolify-cloudflared`) plus the stack via `docker-compose.prod.yml` (the `linearvoid` compose project, 11 containers). |
|
| **Deployment target** | `192.168.88.40` | `coolify-proxy` = Traefik on 80/443; port `8000` bound by Portainer, not Coolify | Yes | Per-server helpers (`coolify-proxy`, `coolify-sentinel`, `coolify-cloudflared`) + the stack via `docker-compose.prod.yml` (`linearvoid` compose project, 11 containers) |
|
||||||
|
|
||||||
The main `coolify` app container does **not** run on `192.168.88.40`; it lives on
|
The main `coolify` app container runs only on the dashboard host `192.168.88.18`; Compose-resource services land on the target `192.168.88.40`. The dashboard host exposes no Docker TCP socket, so run the checks below against the **target** `192.168.88.40`:
|
||||||
the dashboard host `192.168.88.18`. When a Coolify deployment creates a Compose
|
|
||||||
resource, the app services land on the target server `192.168.88.40`.
|
|
||||||
|
|
||||||
Useful checks:
|
```bash
|
||||||
|
./scripts/check-env.sh # env/deployment health on target (containers, helpers, health)
|
||||||
```powershell
|
./scripts/check-tokens.sh # Gitea registry, Cloudflare, R2, admin credential access
|
||||||
# Environment / deployment health on the target 192.168.88.40 (containers, helpers, health)
|
|
||||||
./scripts/check-env.ps1
|
|
||||||
|
|
||||||
# Credential / token access (Gitea registry, Cloudflare, R2, admin login)
|
|
||||||
./scripts/check-tokens.ps1
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The Coolify dashboard runs on the `192.168.88.18` container host (port `8000`,
|
|
||||||
publicly `https://coolify.allucanget.biz`) and exposes no Docker TCP socket, so
|
|
||||||
the environment/token scripts are intended to run against the deployment target
|
|
||||||
`192.168.88.40`, not `192.168.88.18`.
|
|
||||||
|
|
||||||
## Deploy the stack
|
## Deploy the stack
|
||||||
|
|
||||||
Linear Void is a multi-service Docker Compose stack, so deploy it as a
|
Deploy as a **Docker Compose** resource (not individual apps):
|
||||||
**Compose** (Docker Compose) resource rather than individual applications.
|
|
||||||
|
|
||||||
1. **Projects → Add → Docker Compose**.
|
1. **Projects → Add → Docker Compose**.
|
||||||
2. Select the target server.
|
2. Select the target server.
|
||||||
3. In **Build Settings**, set the compose file source:
|
3. In **Build Settings**, set the compose file source:
|
||||||
- **Git Repository** — point Coolify at
|
- **Git Repository** → point at `https://git.allucanget.biz/zwitschi/linearvoid.git`, compose file `docker-compose.prod.yml` (Coolify pulls on each redeploy).
|
||||||
`https://git.allucanget.biz/zwitschi/linearvoid.git` and use
|
- **Raw Compose** → paste `docker-compose.prod.yml` directly to manage manually.
|
||||||
`docker-compose.prod.yml` as the compose file. Coolify pulls on each
|
4. For the Git source, add the Gitea registry login as a **Deployment Key / environment secret** so `playout`, `chat`, `nginx` images can be pulled. Alternatively switch those services to `build:` blocks (as in `docker-compose.yml`) to build from source on the server.
|
||||||
redeploy.
|
|
||||||
- **Raw Compose** — paste the contents of `docker-compose.prod.yml` directly
|
|
||||||
if you prefer to manage it manually.
|
|
||||||
4. If using the Git source, add the Gitea registry login as a **Deployment Key /
|
|
||||||
environment secret** so the `playout`, `chat`, and `nginx` images can be
|
|
||||||
pulled. Alternatively, switch those services to `build:` blocks (as in
|
|
||||||
`docker-compose.yml`) so Coolify builds from source on the server.
|
|
||||||
|
|
||||||
### Environment variables
|
### Environment variables
|
||||||
|
|
||||||
Coolify injects a secret store into the compose resource. Map the Linear Void
|
Map Linear Void's `.env` values into Coolify's **Environment Variables** tab:
|
||||||
environment to Coolify's **Environment Variables** tab (these correspond to the
|
|
||||||
values normally placed in `.env`):
|
|
||||||
|
|
||||||
| Variable | Purpose |
|
| Variable | Purpose |
|
||||||
| ---------------------------------------------- | --------------------------------------------------------------------- |
|
|---|---|
|
||||||
| `DATABASE_URL` | `postgres://linearvoid:<password>@db:5432/linearvoid?sslmode=disable` |
|
| `DATABASE_URL` | `postgres://linearvoid:<password>@db:5432/linearvoid?sslmode=disable` |
|
||||||
| `REDIS_URL` | `redis://redis:6379/0` |
|
| `REDIS_URL` | `redis://redis:6379/0` |
|
||||||
| `POSTGRES_PASSWORD` | Database password (used by the `db` service) |
|
| `POSTGRES_PASSWORD` | DB password (used by `db` service) |
|
||||||
| `ADMIN_USERNAME` / `ADMIN_PASSWORD` | Admin panel credentials |
|
| `ADMIN_USERNAME` / `ADMIN_PASSWORD` | Admin panel credentials |
|
||||||
| `ADMIN_SESSION_TTL_HOURS` | Admin session lifetime |
|
| `ADMIN_SESSION_TTL_HOURS` | Admin session lifetime |
|
||||||
| `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` | Cloudflare R2 credentials |
|
| `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` | Cloudflare R2 credentials |
|
||||||
| `R2_BUCKET_NAME` / `R2_ENDPOINT` / `R2_REGION` | R2 bucket config (`R2_REGION=auto`) |
|
| `R2_BUCKET_NAME` / `R2_ENDPOINT` / `R2_REGION` | R2 bucket config (`R2_REGION=auto`) |
|
||||||
| `MEDIA_DOMAIN` | `media.linearvoid.tv` |
|
| `MEDIA_DOMAIN` | `media.linearvoid.tv` |
|
||||||
| `VITE_PUBLIC_APP_URL` | `https://linearvoid.tv` |
|
| `VITE_PUBLIC_APP_URL` | `https://linearvoid.tv` |
|
||||||
| `VITE_PUBLIC_WS_URL` | `wss://linearvoid.tv/ws` |
|
| `VITE_PUBLIC_WS_URL` | `wss://linearvoid.tv/ws` |
|
||||||
| `VITE_PUBLIC_HLS_URL` | `https://media.linearvoid.tv/hls/playlist.m3u8` |
|
| `VITE_PUBLIC_HLS_URL` | `https://media.linearvoid.tv/hls/playlist.m3u8` |
|
||||||
|
|
||||||
Keep internal service URLs on compose hostnames (`redis`, `db`) exactly as in
|
Keep internal service URLs on compose hostnames (`redis`, `db`) exactly as in `docker-compose.prod.yml`.
|
||||||
`docker-compose.prod.yml`.
|
|
||||||
|
|
||||||
### Persistent storage
|
### Persistent storage
|
||||||
|
|
||||||
The compose file uses named volumes (`redis-data`, `postgres-data`) and bind
|
Named volumes (`redis-data`, `postgres-data`) and bind mounts under `./data` must be configured under **Persistent Storage** to survive redeploys:
|
||||||
mounts under `./data` (playlist, media, hls, chat, slate). Configure these under
|
|
||||||
**Persistent Storage** so data survives redeploys:
|
|
||||||
|
|
||||||
| Mount | Purpose |
|
| Mount | Purpose |
|
||||||
| --------------- | ----------------------------------------------- |
|
|---|---|
|
||||||
| `data/playlist` | Playout playlist file |
|
| `data/playlist` | Playout playlist file |
|
||||||
| `data/media` | Normalized source videos (read-only to playout) |
|
| `data/media` | Normalized source videos (read-only to playout) |
|
||||||
| `data/hls` | Generated HLS manifests and segments |
|
| `data/hls` | Generated HLS manifests/segments |
|
||||||
| `data/chat` | Chat service working data |
|
| `data/chat` | Chat service working data |
|
||||||
| `data/slate` | Slate media |
|
| `data/slate` | Slate media |
|
||||||
| `redis-data` | Redis append-only persistence |
|
| `redis-data` | Redis append-only persistence |
|
||||||
| `postgres-data` | PostgreSQL data directory |
|
| `postgres-data` | PostgreSQL data directory |
|
||||||
|
|
||||||
Create the `data` subdirectories before the first deploy if Coolify does not
|
If Coolify doesn't auto-create the dirs, create them before first deploy:
|
||||||
auto-create them (`mkdir -p data/playlist data/media data/hls data/chat`).
|
|
||||||
|
```bash
|
||||||
|
mkdir -p data/playlist data/media data/hls data/chat
|
||||||
|
```
|
||||||
|
|
||||||
### Domains and proxy
|
### Domains and proxy
|
||||||
|
|
||||||
1. Open the compose resource and assign the **nginx** service a public domain
|
1. Assign the **nginx** service a public domain (e.g. `linearvoid.tv`); Traefik terminates TLS and forwards to `nginx`.
|
||||||
(e.g. `linearvoid.tv`). Coolify's Traefik reverse proxy then terminates TLS
|
2. Set **Ports Exposes** so Traefik targets the nginx port (`80` inside container, mapped from `${NGINX_PORT:-8080}:80`).
|
||||||
and forwards to the `nginx` container.
|
3. Do **not** publicly expose playout (`8081`) or chat (`3000`) health ports — keep them internal and route through nginx.
|
||||||
2. Set the **Ports Exposes** mapping so Traefik targets the nginx service port
|
4. For `media.linearvoid.tv`, keep serving HLS from the R2 custom domain (per SETUP.md) rather than routing through Coolify, so segments stay on Cloudflare's edge cache.
|
||||||
(`80` inside the container, mapped from `${NGINX_PORT:-8080}:80`).
|
|
||||||
3. Do **not** also expose the playout (`8081`) or chat (`3000`) health ports
|
|
||||||
publicly — keep those internal to the compose network and let Traefik route
|
|
||||||
through nginx.
|
|
||||||
4. For the media domain (`media.linearvoid.tv`), keep serving HLS from the R2
|
|
||||||
custom domain (per SETUP.md) rather than routing it through Coolify, so
|
|
||||||
segments stay on Cloudflare's edge cache.
|
|
||||||
|
|
||||||
## Post-deploy verification
|
## Post-deploy verification
|
||||||
|
|
||||||
After the first deploy, confirm:
|
- `docker compose ps` on the server shows all services healthy.
|
||||||
|
|
||||||
- `docker compose ps` (on the server) shows all services healthy.
|
|
||||||
- Playout health: `https://<host>/health` via nginx (internal `:8081/health`).
|
- Playout health: `https://<host>/health` via nginx (internal `:8081/health`).
|
||||||
- Chat health: internal `:3000/health`.
|
- Chat health: internal `:3000/health`.
|
||||||
- HLS is written to `data/hls` and served from `media.linearvoid.tv`.
|
- HLS written to `data/hls` and served from `media.linearvoid.tv`.
|
||||||
- Chat WebSocket connects over `wss://linearvoid.tv/ws`.
|
- Chat WebSocket connects over `wss://linearvoid.tv/ws`.
|
||||||
- Admin login works at `/admin` with the configured credentials.
|
- Admin login works at `/admin` with configured credentials.
|
||||||
|
|
||||||
## Monitoring
|
## Monitoring
|
||||||
|
|
||||||
Coolify exposes container logs and basic metrics per resource. The stack also
|
Coolify exposes container logs/basic metrics per resource. The stack ships Prometheus + Grafana (`monitoring/`) plus nginx/redis exporters. To reach them via Coolify, expose Grafana on a restricted subdomain (e.g. `grafana.linearvoid.tv`) and keep Prometheus internal, mirroring SETUP.md access guidance.
|
||||||
ships Prometheus + Grafana (`monitoring/`) and nginx/redis exporters. If you
|
|
||||||
want those reachable through Coolify, expose Grafana on a restricted subdomain
|
|
||||||
(e.g. `grafana.linearvoid.tv`) and keep Prometheus internal, mirroring the
|
|
||||||
access guidance in SETUP.md.
|
|
||||||
|
|
||||||
## Cutover from Gitea CD
|
## Cutover from Gitea CD
|
||||||
|
|
||||||
Coolify can fully replace the Gitea Actions CD step. To switch:
|
|
||||||
|
|
||||||
1. Deploy the stack as a Coolify compose resource (above).
|
1. Deploy the stack as a Coolify compose resource (above).
|
||||||
2. Point Cloudflare DNS for `linearvoid.tv` / `www` at the server IP (proxied).
|
2. Point Cloudflare DNS for `linearvoid.tv` / `www` at the server IP (proxied).
|
||||||
3. Disable (or keep as a fallback) the `cd` job in the Gitea workflow so the two
|
3. Disable (or keep as fallback) the `cd` job in the Gitea workflow so the two orchestrators don't fight over the same `DOCKER_HOST`.
|
||||||
orchestrators don't fight over the same `DOCKER_HOST`.
|
4. Keep the Gitea **CI** (build + push images) job so Coolify keeps pulling `:latest` from the registry.
|
||||||
4. Keep the Gitea **CI** (build + push images) job so Coolify continues pulling
|
|
||||||
`:latest` images from the registry.
|
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
- **Image pull fails**: ensure the Gitea registry credentials are set in the
|
- **Image pull fails**: set Gitea registry credentials in the compose secret store, or switch services to `build:` blocks.
|
||||||
compose resource's secret store, or switch the services to `build:` blocks.
|
- **Healthchecks fail**: confirm `curl` exists in the built images (healthchecks rely on it).
|
||||||
- **Healthchecks fail**: confirm `curl` is available in the built images (the
|
- **TLS errors**: check the domain's DNS points at the Coolify server and ports `80`/`443` are open.
|
||||||
compose healthchecks rely on `curl`).
|
- **HLS not updating**: verify `data/hls` is writable and playout restarts cleanly; see SETUP.md normalization troubleshooting.
|
||||||
- **TLS errors**: check the domain's DNS points at the Coolify server and ports
|
- **Ports conflict**: Coolify's Traefik binds `80`/`443`; do not also map those to the host in the compose file.
|
||||||
`80`/`443` are open.
|
|
||||||
- **HLS not updating**: verify `data/hls` is writable and the playout service
|
|
||||||
restarts cleanly; see the normalization troubleshooting in SETUP.md.
|
|
||||||
- **Ports conflict**: Coolify's Traefik binds `80`/`443`; do not also map those
|
|
||||||
to the host in the compose file.
|
|
||||||
|
|
||||||
## Further reading
|
## Further reading
|
||||||
|
|
||||||
- Coolify repo (v4.3.14): <https://github.com/coollabsio/coolify/tree/v4.3.14>
|
- Coolify repo (v4.3.14): <https://github.com/coollabsio/coolify/tree/v4.3.14>
|
||||||
- Coolify docs: <https://coolify.io/docs>
|
- Coolify docs: <https://coolify.io/docs>
|
||||||
- Linear Void deployment: [DEPLOYMENT.md](./DEPLOYMENT.md)
|
- [DEPLOYMENT.md](./DEPLOYMENT.md) · [SETUP.md](./SETUP.md)
|
||||||
- Linear Void setup: [SETUP.md](./SETUP.md)
|
|
||||||
+72
-194
@@ -1,64 +1,26 @@
|
|||||||
# Deployment Guide
|
# Deployment Guide
|
||||||
|
|
||||||
## Table of Contents
|
|
||||||
|
|
||||||
- [Deployment Guide](#deployment-guide)
|
|
||||||
- [Table of Contents](#table-of-contents)
|
|
||||||
- [Environments](#environments)
|
|
||||||
- [Production Target (Hetzner)](#production-target-hetzner)
|
|
||||||
- [Planned Infrastructure](#planned-infrastructure)
|
|
||||||
- [Cutover Plan](#cutover-plan)
|
|
||||||
- [Production Preflight](#production-preflight)
|
|
||||||
- [Firewall / Security Group](#firewall--security-group)
|
|
||||||
- [Pipeline Readiness](#pipeline-readiness)
|
|
||||||
- [Target Topology](#target-topology)
|
|
||||||
- [Prerequisites](#prerequisites)
|
|
||||||
- [Environment File](#environment-file)
|
|
||||||
- [Gitea Secrets](#gitea-secrets)
|
|
||||||
- [Pull-and-Run Deployment](#pull-and-run-deployment)
|
|
||||||
- [First-Time or Recovery Provisioning](#first-time-or-recovery-provisioning)
|
|
||||||
- [Health Verification](#health-verification)
|
|
||||||
- [Zero/Low-Downtime Update Pattern](#zerolow-downtime-update-pattern)
|
|
||||||
- [Rollback](#rollback)
|
|
||||||
- [Operational Tasks](#operational-tasks)
|
|
||||||
- [View logs](#view-logs)
|
|
||||||
- [Restart one service](#restart-one-service)
|
|
||||||
- [Stop stack](#stop-stack)
|
|
||||||
- [Keep volumes (recommended) vs remove volumes (destructive)](#keep-volumes-recommended-vs-remove-volumes-destructive)
|
|
||||||
- [Nginx and Proxy Considerations](#nginx-and-proxy-considerations)
|
|
||||||
- [Security Baseline](#security-baseline)
|
|
||||||
- [Common Deployment Pitfalls](#common-deployment-pitfalls)
|
|
||||||
- [Troubleshooting](#troubleshooting)
|
|
||||||
|
|
||||||
## Environments
|
## Environments
|
||||||
|
|
||||||
Linear Void runs in two separate environments:
|
| Environment | Host | Docker Target | Purpose |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **Test Lab** | `192.168.88.40` (internal) | `tcp://192.168.88.40:2375` | Integration testing, staging, CI/CD validation |
|
||||||
|
| **Production** | Hetzner VPS (TBD) | `tcp://<hetzner-ip>:2375` | Public linearvoid.tv |
|
||||||
|
|
||||||
| Environment | Host | Docker Target | Purpose |
|
**Current state (2026-08-06):** Test lab is active and receives automated CD deploys. Production Hetzner VPS is **not yet provisioned**. `cd.yml` and `docker-compose.prod.yml` target the test lab; switching to production only requires changing `DOCKER_HOST`.
|
||||||
| -------------- | -------------------------- | -------------------------- | ---------------------------------------------- |
|
|
||||||
| **Test Lab** | `192.168.88.40` (internal) | `tcp://192.168.88.40:2375` | Integration testing, staging, CI/CD validation |
|
|
||||||
| **Production** | Hetzner VPS (TBD) | `tcp://<hetzner-ip>:2375` | Public linearvoid.tv serving real viewers |
|
|
||||||
|
|
||||||
**Current state (2026-08-06):** Test lab is active and receives automated CD deploys. Production Hetzner VPS is **not yet provisioned**. The CD workflow (`cd.yml`) and `docker-compose.prod.yml` currently target the test lab. Switching to production requires only changing `DOCKER_HOST` — see [Production Deployment](#production-target-hetzner) below.
|
|
||||||
|
|
||||||
## Production Target (Hetzner)
|
## Production Target (Hetzner)
|
||||||
|
|
||||||
**Status: Not provisioned.** The production deployment plan is documented and the pipeline is ready — only the VPS needs to be created.
|
**Status: Not provisioned.** Plan documented and pipeline ready; only the VPS needs creation.
|
||||||
|
|
||||||
### Planned Infrastructure
|
### Planned Infrastructure
|
||||||
|
- Provider: Hetzner Cloud, Region: Ashburn, VA (us-east)
|
||||||
| Resource | Spec |
|
- Instance: CX22 (2 vCPU, 4 GB RAM, 40 GB SSD) or equivalent
|
||||||
| -------------- | ------------------------------------------------ |
|
- OS: Ubuntu 24.04 LTS or Debian 12
|
||||||
| Provider | Hetzner Cloud |
|
- Bandwidth: 20 TB included traffic/month; Estimated cost: ~€7–10/month base
|
||||||
| Region | Ashburn, VA (us-east) |
|
|
||||||
| Instance | CX22 (2 vCPU, 4 GB RAM, 40 GB SSD) or equivalent |
|
|
||||||
| OS | Ubuntu 24.04 LTS or Debian 12 |
|
|
||||||
| Bandwidth | 20 TB included traffic/month |
|
|
||||||
| Estimated cost | ~€7–10/month base |
|
|
||||||
|
|
||||||
### Cutover Plan
|
### Cutover Plan
|
||||||
|
1. Provision Hetzner VPS
|
||||||
1. Provision Hetzner VPS (see [Production Deployment](#production-target-hetzner) section)
|
|
||||||
2. Add production secrets to Gitea (`CI_PROD_*` prefix)
|
2. Add production secrets to Gitea (`CI_PROD_*` prefix)
|
||||||
3. Update `DOCKER_HOST` in CD workflow to Hetzner public IP
|
3. Update `DOCKER_HOST` in CD workflow to Hetzner public IP
|
||||||
4. Trigger deploy → validate health checks
|
4. Trigger deploy → validate health checks
|
||||||
@@ -66,44 +28,33 @@ Linear Void runs in two separate environments:
|
|||||||
6. Enable HTTPS, WAF, cache rules on production zone
|
6. Enable HTTPS, WAF, cache rules on production zone
|
||||||
7. Decommission test lab as public target
|
7. Decommission test lab as public target
|
||||||
|
|
||||||
### Production Preflight
|
### Production Preflight (minimum readiness gate)
|
||||||
|
1. Verify Hetzner host reachable over SSH and Docker remote API TLS works for the Gitea runner
|
||||||
Before the cutover in the plan above, treat the following as the minimum readiness gate:
|
2. Confirm all production secrets exist under `CI_PROD_*` prefix
|
||||||
|
3. Confirm `linearvoid.tv`, `www.linearvoid.tv`, `media.linearvoid.tv` DNS targets match intended phase
|
||||||
1. Verify the Hetzner host is reachable over SSH and Docker remote API TLS is working for the Gitea runner.
|
4. Complete manual Cloudflare/UptimeRobot checklist (see `docs/operations/cloudflare-uptimerobot-checklist.md`)
|
||||||
2. Confirm all required production secrets are present under the `CI_PROD_*` prefix.
|
5. Run production smoke path end-to-end: viewer SPA, HLS playback, chat connect, admin login, upload, playlist save, playout health
|
||||||
3. Confirm `linearvoid.tv`, `www.linearvoid.tv`, and `media.linearvoid.tv` DNS targets are correct for the intended phase.
|
6. Verify rollback possible before final DNS switch
|
||||||
4. Complete the manual Cloudflare/UptimeRobot checklist and attach evidence in [Cloudflare Uptimerobot checklist](../docs/operations/cloudflare-uptimerobot-checklist.md).
|
|
||||||
5. Run the production smoke path end to end: viewer SPA, HLS playback, chat connect, admin login, upload, playlist save, and playout health.
|
|
||||||
6. Verify rollback is still possible before the final DNS switch.
|
|
||||||
|
|
||||||
Readiness is not complete until the checklist above and the manual evidence tracker are both satisfied.
|
|
||||||
|
|
||||||
### Firewall / Security Group
|
### Firewall / Security Group
|
||||||
|
| Port | Source | Purpose |
|
||||||
| Port | Source | Purpose |
|
|---|---|---|
|
||||||
| ---- | --------------------- | ------------------------------ |
|
| 22 | Admin IPs | SSH management |
|
||||||
| 22 | Admin IPs | SSH management |
|
| 80 | 0.0.0.0/0 | HTTP (redirect to HTTPS) |
|
||||||
| 80 | 0.0.0.0/0 | HTTP (redirect to HTTPS) |
|
| 443 | 0.0.0.0/0 | HTTPS (nginx) |
|
||||||
| 443 | 0.0.0.0/0 | HTTPS (nginx) |
|
| 2375 | Gitea runner IP only | Docker remote API (TLS) |
|
||||||
| 2375 | Gitea runner IP only | Docker remote API (TLS) |
|
|
||||||
| 9090 | Admin IPs or internal | Prometheus (or tunnel via SSH) |
|
| 9090 | Admin IPs or internal | Prometheus (or tunnel via SSH) |
|
||||||
| 3001 | Admin IPs or internal | Grafana (or tunnel via SSH) |
|
| 3001 | Admin IPs or internal | Grafana (or tunnel via SSH) |
|
||||||
|
|
||||||
### Pipeline Readiness
|
### Pipeline Readiness
|
||||||
|
| Workflow | Trigger | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| `ci.yml` | Push to `main` | Lint, build frontend, vet Go, build & push images (`:latest`, `:main`) |
|
||||||
|
| `cd.yml` | Push to `main` | Deploy to test lab (`192.168.88.40:2375`) — runs **in parallel** with CI |
|
||||||
|
| `release.yml` | Tag push `*.*.*` | Build & push versioned images (`:X.Y.Z`, `:X.Y`, `:latest`), create Gitea Release |
|
||||||
|
| `workers-deploy.yml` | After CI succeeds | Deploy frontend to Cloudflare Workers |
|
||||||
|
|
||||||
Three Gitea Actions workflows drive the full pipeline:
|
CI and CD run simultaneously on every push to `main`. Deploy pulls whatever `:latest` images are available at that moment. `cd.yml` uses `DOCKER_HOST` to target the host:
|
||||||
|
|
||||||
| Workflow | Trigger | Purpose |
|
|
||||||
| -------------------- | ----------------- | ----------------------------------------------------------------------------------------- |
|
|
||||||
| `ci.yml` | Push to `main` | Lint, build frontend, vet Go, build & push Docker images (`:latest`, `:main`) |
|
|
||||||
| `cd.yml` | Push to `main` | Deploy to test lab (`192.168.88.40:2375`) — runs **in parallel** with CI |
|
|
||||||
| `release.yml` | Tag push `*.*.*` | Build & push versioned Docker images (`:X.Y.Z`, `:X.Y`, `:latest`), create Gitea Release |
|
|
||||||
| `workers-deploy.yml` | After CI succeeds | Deploy frontend to Cloudflare Workers (see `docs/operations/gitea-cloudflare-workers.md`) |
|
|
||||||
|
|
||||||
CI and CD run simultaneously on every push to `main`. The deploy pulls whatever `:latest` images are available in the registry at that moment.
|
|
||||||
|
|
||||||
The CD workflow (`cd.yml`) uses `DOCKER_HOST` to target the deployment host. To switch targets:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Test lab (current)
|
# Test lab (current)
|
||||||
@@ -113,10 +64,9 @@ DOCKER_HOST=tcp://192.168.88.40:2375
|
|||||||
DOCKER_HOST=tcp://<hetzner-public-ip>:2375
|
DOCKER_HOST=tcp://<hetzner-public-ip>:2375
|
||||||
```
|
```
|
||||||
|
|
||||||
No code changes needed in the pipeline. See `DEPLOYMENT.md` Production Deployment section for full checklist.
|
No pipeline code changes needed to switch targets.
|
||||||
|
|
||||||
## Target Topology
|
## Target Topology
|
||||||
|
|
||||||
- Host: Linux server with Docker Engine
|
- Host: Linux server with Docker Engine
|
||||||
- Reverse proxy: nginx container (linearvoid-nginx)
|
- Reverse proxy: nginx container (linearvoid-nginx)
|
||||||
- Core services: playout, chat, redis, postgres
|
- Core services: playout, chat, redis, postgres
|
||||||
@@ -124,16 +74,14 @@ No code changes needed in the pipeline. See `DEPLOYMENT.md` Production Deploymen
|
|||||||
- CDN/edge: Cloudflare
|
- CDN/edge: Cloudflare
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
- Docker Engine + Docker Compose plugin
|
- Docker Engine + Docker Compose plugin
|
||||||
- Network access to git.allucanget.biz
|
- Network access to git.allucanget.biz
|
||||||
- A valid .env file with production values
|
- Valid `.env` file with production values
|
||||||
- DNS configured for linearvoid.tv (and media.linearvoid.tv if split media domain is used)
|
- DNS configured for `linearvoid.tv` (and `media.linearvoid.tv` if split media domain used)
|
||||||
|
|
||||||
## Environment File
|
## Environment File
|
||||||
|
|
||||||
Use `.env` with at least:
|
`.env` requires at minimum:
|
||||||
|
|
||||||
- `DATABASE_URL` — PostgreSQL connection string
|
- `DATABASE_URL` — PostgreSQL connection string
|
||||||
- `REDIS_URL` — Redis connection string
|
- `REDIS_URL` — Redis connection string
|
||||||
- `ADMIN_USERNAME` / `ADMIN_PASSWORD` — admin dashboard credentials
|
- `ADMIN_USERNAME` / `ADMIN_PASSWORD` — admin dashboard credentials
|
||||||
@@ -141,186 +89,116 @@ Use `.env` with at least:
|
|||||||
- `VITE_PUBLIC_APP_URL` / `VITE_PUBLIC_WS_URL` / `VITE_PUBLIC_HLS_URL` — frontend URLs
|
- `VITE_PUBLIC_APP_URL` / `VITE_PUBLIC_WS_URL` / `VITE_PUBLIC_HLS_URL` — frontend URLs
|
||||||
|
|
||||||
**Cloudflare R2 (required for HLS segment sync to edge):**
|
**Cloudflare R2 (required for HLS segment sync to edge):**
|
||||||
|
- `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY`
|
||||||
- `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` — R2 write credentials
|
- `R2_BUCKET_NAME` (e.g. `linearvoid-media`)
|
||||||
- `R2_BUCKET_NAME` — bucket name (e.g. `linearvoid-media`)
|
|
||||||
- `R2_ENDPOINT` — S3-compatible endpoint URL
|
- `R2_ENDPOINT` — S3-compatible endpoint URL
|
||||||
- `R2_REGION` — set to `auto`
|
- `R2_REGION` — set to `auto`
|
||||||
|
|
||||||
**Cloudflare (optional, for Workers):**
|
**Cloudflare (optional, for Workers):**
|
||||||
|
|
||||||
- `CLOUDFLARE_API_TOKEN` / `CLOUDFLARE_ACCOUNT_ID`
|
- `CLOUDFLARE_API_TOKEN` / `CLOUDFLARE_ACCOUNT_ID`
|
||||||
- `MEDIA_DOMAIN` — split media domain (e.g. `media.linearvoid.tv`)
|
- `MEDIA_DOMAIN` — split media domain (e.g. `media.linearvoid.tv`)
|
||||||
|
|
||||||
See `.env.example` and `SETUP.md` for full variable reference.
|
See `.env.example` and `SETUP.md` for full variable reference.
|
||||||
|
|
||||||
### Gitea Secrets
|
### Gitea Secrets
|
||||||
|
`cd.yml` injects sensitive values from Gitea secrets into `.env` at deploy time. Configure in the Gitea repository:
|
||||||
|
|
||||||
The CD workflow (`cd.yml`) injects sensitive values from Gitea secrets into the `.env` file at deploy time. The following secrets must be configured in the Gitea repository:
|
`DATABASE_URL`, `ADMIN_USERNAME`, `ADMIN_PASSWORD`, `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_BUCKET_NAME`, `R2_ENDPOINT`, `R2_ADMIN_ACCESS_KEY_ID` (optional), `R2_ADMIN_SECRET_ACCESS_KEY` (optional), `R2_ADMIN_ENDPOINT` (optional), `CLOUDFLARE_API_TOKEN` (optional), `CLOUDFLARE_ACCOUNT_ID` (optional), `CI_REGISTRY_USER`, `CI_REGISTRY_TOKEN`
|
||||||
|
|
||||||
| Secret Name | Description |
|
|
||||||
| ---------------------------- | --------------------------------- |
|
|
||||||
| `DATABASE_URL` | PostgreSQL connection string |
|
|
||||||
| `ADMIN_USERNAME` | Admin dashboard username |
|
|
||||||
| `ADMIN_PASSWORD` | Admin dashboard password |
|
|
||||||
| `R2_ACCESS_KEY_ID` | Cloudflare R2 access key |
|
|
||||||
| `R2_SECRET_ACCESS_KEY` | Cloudflare R2 secret key |
|
|
||||||
| `R2_BUCKET_NAME` | R2 bucket name |
|
|
||||||
| `R2_ENDPOINT` | R2 S3-compatible endpoint URL |
|
|
||||||
| `R2_ADMIN_ACCESS_KEY_ID` | R2 admin access key (optional) |
|
|
||||||
| `R2_ADMIN_SECRET_ACCESS_KEY` | R2 admin secret key (optional) |
|
|
||||||
| `R2_ADMIN_ENDPOINT` | R2 admin endpoint (optional) |
|
|
||||||
| `CLOUDFLARE_API_TOKEN` | Cloudflare API token (optional) |
|
|
||||||
| `CLOUDFLARE_ACCOUNT_ID` | Cloudflare account ID (optional) |
|
|
||||||
| `CI_REGISTRY_USER` | Gitea container registry username |
|
|
||||||
| `CI_REGISTRY_TOKEN` | Gitea container registry token |
|
|
||||||
|
|
||||||
## Pull-and-Run Deployment
|
## Pull-and-Run Deployment
|
||||||
|
|
||||||
Images are built and pushed automatically by the CI and release workflows (see [CONTRIBUTING.md](../CONTRIBUTING.md#versioning-and-releases) for the release procedure).
|
Images are built/pushed automatically by CI and release workflows. Every push to `main` triggers CI (`:main`, `:latest`) and CD (deploys `:latest` to test lab) in parallel. Semantic version tags (`*.*.*`, e.g. `1.0.9`) trigger the release workflow (`:X.Y.Z`, `:X.Y`, `:latest` + Gitea Release).
|
||||||
|
|
||||||
Every push to `main` triggers CI (builds `:main` and `:latest` tags) and CD (deploys `:latest` to test lab) in parallel. Semantic version tags (`*.*.*`, e.g. `1.0.9`) trigger the release workflow which builds `:X.Y.Z`, `:X.Y`, and `:latest` tags and creates a Gitea Release.
|
|
||||||
|
|
||||||
From repository root:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker login git.allucanget.biz
|
docker login git.allucanget.biz
|
||||||
|
|
||||||
docker compose -f docker-compose.prod.yml pull
|
docker compose -f docker-compose.prod.yml pull
|
||||||
docker compose -f docker-compose.prod.yml up -d --remove-orphans
|
docker compose -f docker-compose.prod.yml up -d --remove-orphans
|
||||||
```
|
```
|
||||||
|
|
||||||
## First-Time or Recovery Provisioning
|
## First-Time or Recovery Provisioning
|
||||||
|
|
||||||
1. Ensure host paths exist:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
mkdir -p data/chat data/playlist data/media data/hls
|
mkdir -p data/chat data/playlist data/media data/hls
|
||||||
```
|
|
||||||
|
|
||||||
1. Confirm volumes and networks:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker volume ls
|
docker volume ls
|
||||||
docker network ls
|
docker network ls
|
||||||
```
|
|
||||||
|
|
||||||
1. Start stack:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker compose -f docker-compose.prod.yml up -d
|
docker compose -f docker-compose.prod.yml up -d
|
||||||
```
|
```
|
||||||
|
|
||||||
## Health Verification
|
## Health Verification
|
||||||
|
|
||||||
After startup, check service status:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose -f docker-compose.prod.yml ps
|
docker compose -f docker-compose.prod.yml ps
|
||||||
```
|
```
|
||||||
|
|
||||||
Endpoint checks:
|
Endpoint checks:
|
||||||
|
- App: `http://{host}:8080/`
|
||||||
- App: <http://{host}:8080/>
|
- Nginx health: `http://{host}:8080/health`
|
||||||
- Nginx health: <http://{host}:8080/health>
|
- Chat health: `http://{host}:3000/health`
|
||||||
- Chat health: <http://{host}:3000/health>
|
- Playout health: `http://{host}:8081/health`
|
||||||
- Playout health: <http://{host}:8081/health>
|
- Prometheus: `http://{host}:9090`
|
||||||
- Prometheus: <http://{host}:9090>
|
- Grafana: `http://{host}:3001`
|
||||||
- Grafana: <http://{host}:3001>
|
|
||||||
|
|
||||||
Domain checks:
|
Domain checks:
|
||||||
|
- `https://linearvoid.tv`
|
||||||
- <https://linearvoid.tv>
|
- `https://linearvoid.tv/admin`
|
||||||
- <https://linearvoid.tv/admin>
|
- `https://linearvoid.tv/metrics`
|
||||||
- <https://linearvoid.tv/metrics>
|
|
||||||
|
|
||||||
## Zero/Low-Downtime Update Pattern
|
## Zero/Low-Downtime Update Pattern
|
||||||
|
|
||||||
1. Pull newest images.
|
1. Pull newest images.
|
||||||
1. Recreate changed services only:
|
2. Recreate changed services only:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose -f docker-compose.prod.yml pull
|
docker compose -f docker-compose.prod.yml pull
|
||||||
|
|
||||||
docker compose -f docker-compose.prod.yml up -d nginx chat playout
|
docker compose -f docker-compose.prod.yml up -d nginx chat playout
|
||||||
```
|
```
|
||||||
|
3. Re-run health checks.
|
||||||
1. Re-run health checks.
|
|
||||||
|
|
||||||
## Rollback
|
## Rollback
|
||||||
|
|
||||||
If new images are unhealthy:
|
If new images are unhealthy:
|
||||||
|
1. Pin previous tags in `docker-compose.prod.yml`.
|
||||||
1. Pin previous tags in docker-compose.prod.yml.
|
|
||||||
2. Pull pinned tags.
|
2. Pull pinned tags.
|
||||||
3. Restart impacted services.
|
3. Restart impacted services:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose -f docker-compose.prod.yml pull
|
docker compose -f docker-compose.prod.yml pull
|
||||||
|
|
||||||
docker compose -f docker-compose.prod.yml up -d nginx chat playout
|
docker compose -f docker-compose.prod.yml up -d nginx chat playout
|
||||||
```
|
```
|
||||||
|
|
||||||
## Operational Tasks
|
## Operational Tasks
|
||||||
|
|
||||||
### View logs
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
# View logs
|
||||||
docker compose -f docker-compose.prod.yml logs -f nginx chat playout
|
docker compose -f docker-compose.prod.yml logs -f nginx chat playout
|
||||||
```
|
|
||||||
|
|
||||||
### Restart one service
|
# Restart one service
|
||||||
|
|
||||||
```bash
|
|
||||||
docker compose -f docker-compose.prod.yml restart chat
|
docker compose -f docker-compose.prod.yml restart chat
|
||||||
```
|
|
||||||
|
|
||||||
### Stop stack
|
# Stop stack (keep data)
|
||||||
|
|
||||||
```bash
|
|
||||||
docker compose -f docker-compose.prod.yml down
|
|
||||||
```
|
|
||||||
|
|
||||||
### Keep volumes (recommended) vs remove volumes (destructive)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# keep data
|
|
||||||
docker compose -f docker-compose.prod.yml down
|
docker compose -f docker-compose.prod.yml down
|
||||||
|
|
||||||
# destructive (remove data)
|
# Stop stack — destructive (remove data)
|
||||||
docker compose -f docker-compose.prod.yml down -v
|
docker compose -f docker-compose.prod.yml down -v
|
||||||
```
|
```
|
||||||
|
|
||||||
## Nginx and Proxy Considerations
|
## Nginx and Proxy Considerations
|
||||||
|
- If an external proxy (e.g. Nginx Proxy Manager) fronts the stack, align request body limits with internal nginx (see `infrastructure/nginx/nginx.conf`).
|
||||||
- If an external proxy (for example Nginx Proxy Manager) fronts the stack, align request body limits with internal nginx, see [nginx.conf](../infrastructure/nginx/nginx.conf).
|
- For resumable admin uploads, ensure `client_max_body_size` supports chunk requests in both proxy layers.
|
||||||
- For resumable admin uploads, ensure client_max_body_size supports chunk requests in both proxy layers.
|
|
||||||
|
|
||||||
## Security Baseline
|
## Security Baseline
|
||||||
|
- Use a strong `ADMIN_PASSWORD` and rotate regularly.
|
||||||
- Use strong ADMIN_PASSWORD and rotate regularly.
|
|
||||||
- Keep Docker host access restricted.
|
- Keep Docker host access restricted.
|
||||||
- Prefer HTTPS/WSS through Cloudflare.
|
- Prefer HTTPS/WSS through Cloudflare.
|
||||||
- Restrict exposed ports if only proxy access is needed.
|
- Restrict exposed ports if only proxy access is needed.
|
||||||
|
|
||||||
## Common Deployment Pitfalls
|
## Common Deployment Pitfalls
|
||||||
|
|
||||||
- Missing frontend build artifacts in local-only compose workflows.
|
- Missing frontend build artifacts in local-only compose workflows.
|
||||||
- Wrong registry credentials or missing login.
|
- Wrong registry credentials or missing login.
|
||||||
- Misaligned domain env vars (app/ws/hls) causing mixed-origin errors.
|
- Misaligned domain env vars (app/ws/hls) causing mixed-origin errors.
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
| Symptom | Check | Fix |
|
||||||
|
|---|---|---|
|
||||||
|
| Service not starting | `docker logs linearvoid-<svc> --tail 50` | Check `.env` vars, volume mounts, port conflicts |
|
||||||
|
| Playout stall / slate mode | `curl http://<host>:8081/health` — mode=slate | Check playlist file, media paths, disk space |
|
||||||
|
| Chat not connecting | `curl http://<host>:3000/health` | Check Redis: `docker exec linearvoid-redis redis-cli ping` |
|
||||||
|
| Admin 401 loop | Cookie not sent | Verify `X-Forwarded-Proto: https`, nginx auth_request headers |
|
||||||
|
| Upload fails (413) | File too large | Check nginx `client_max_body_size` (should be 64m+) |
|
||||||
|
| HLS not playing | `curl http://<host>/hls/playlist.m3u8` | Check playout health, HLS dir permissions |
|
||||||
|
| Prometheus no targets | `curl http://<host>:9090/targets` | Verify container network, scrape config |
|
||||||
|
| Grafana no data | Check datasource | Verify Prometheus URL `http://prometheus:9090` provisioned |
|
||||||
|
|
||||||
| Symptom | Check | Fix |
|
Additional notes: proxy-layer body size limits can return 413 on upload; health checks may fail before service warmup completes.
|
||||||
| -------------------------- | --------------------------------------------- | ------------------------------------------------------------- |
|
|
||||||
| Service not starting | `docker logs linearvoid-<svc> --tail 50` | Check `.env` vars, volume mounts, port conflicts |
|
|
||||||
| Playout stall / slate mode | `curl http://<host>:8081/health` — mode=slate | Check playlist file, media paths, disk space |
|
|
||||||
| Chat not connecting | `curl http://<host>:3000/health` | Check Redis: `docker exec linearvoid-redis redis-cli ping` |
|
|
||||||
| Admin 401 loop | Cookie not sent | Verify `X-Forwarded-Proto: https`, nginx auth_request headers |
|
|
||||||
| Upload fails (413) | File too large | Check nginx `client_max_body_size` (should be 64m+) |
|
|
||||||
| HLS not playing | `curl http://<host>/hls/playlist.m3u8` | Check playout health, HLS dir permissions |
|
|
||||||
| Prometheus no targets | `curl http://<host>:9090/targets` | Verify container network, scrape config |
|
|
||||||
| Grafana no data | Check datasource | Verify Prometheus URL `http://prometheus:9090` provisioned |
|
|
||||||
|
|
||||||
- Proxy-layer body size limits returning 413 on upload.
|
|
||||||
- Health checks failing before service warmup is complete.
|
|
||||||
+4
-12
@@ -52,7 +52,7 @@ Config: `monitoring/prometheus/prometheus.yml` (15s scrape interval)
|
|||||||
- **Linear Void app**: `monitoring/grafana/dashboard.json` — clients, messages, moderation, Redis, sessions, scrape health
|
- **Linear Void app**: `monitoring/grafana/dashboard.json` — clients, messages, moderation, Redis, sessions, scrape health
|
||||||
- **Prometheus ops**: `monitoring/grafana/dashboards/dashboard.json` — TSDB, CPU/memory, WAL, compactions, query latency
|
- **Prometheus ops**: `monitoring/grafana/dashboards/dashboard.json` — TSDB, CPU/memory, WAL, compactions, query latency
|
||||||
|
|
||||||
## Metrics checking
|
## Metrics Checking
|
||||||
|
|
||||||
See detailed scripts for querying Prometheus scrape targets:
|
See detailed scripts for querying Prometheus scrape targets:
|
||||||
|
|
||||||
@@ -78,21 +78,13 @@ See detailed scripts for querying Prometheus scrape targets:
|
|||||||
| Prometheus targets down | Check container network: all must be on `linearvoid-net` |
|
| Prometheus targets down | Check container network: all must be on `linearvoid-net` |
|
||||||
| Alert not firing | Check Prometheus → Alerts tab, verify `rule_files` in prometheus.yml |
|
| Alert not firing | Check Prometheus → Alerts tab, verify `rule_files` in prometheus.yml |
|
||||||
|
|
||||||
### Grafana shows no data
|
**Grafana no data**: ensure datasource points to `http://prometheus:9090`; confirm Prometheus is healthy; check dashboard query labels match active jobs.
|
||||||
|
|
||||||
- Ensure datasource points to <http://prometheus:9090>.
|
**High cardinality / slow queries**: reduce unneeded labels; limit expensive panels; tune retention and query windows.
|
||||||
- Confirm Prometheus is healthy.
|
|
||||||
- Check dashboard query labels match active jobs.
|
|
||||||
|
|
||||||
### High cardinality or query slowness
|
|
||||||
|
|
||||||
- Reduce unneeded labels.
|
|
||||||
- Limit expensive dashboard panels.
|
|
||||||
- Tune retention and query windows.
|
|
||||||
|
|
||||||
## Operational Recommendations
|
## Operational Recommendations
|
||||||
|
|
||||||
- Keep alert thresholds tied to actual user impact.
|
- Keep alert thresholds tied to actual user impact.
|
||||||
- Review dashboards after each release that changes endpoints or metrics names.
|
- Review dashboards after each release that changes endpoints or metrics names.
|
||||||
- Periodically test alert routing and incident response flow.
|
- Periodically test alert routing and incident response flow.
|
||||||
- Metrics retention is set to **30 days** time-based plus a **15GB** size cap (`--storage.tsdb.retention.time=30d --storage.tsdb.retention.size=15GB` on the `prometheus` service in `docker-compose.yml` and `docker-compose.prod.yml`). Increase the size cap before extending the time window so the TSDB does not hit the disk limit first.
|
- Retention: **30 days** time-based plus **15GB** size cap (`--storage.tsdb.retention.time=30d --storage.tsdb.retention.size=15GB` on the `prometheus` service in `docker-compose.yml` and `docker-compose.prod.yml`). Increase the size cap before extending the time window so the TSDB does not hit the disk limit first.
|
||||||
+80
-503
@@ -2,572 +2,149 @@
|
|||||||
|
|
||||||
**Last Updated**: 2026-08-03
|
**Last Updated**: 2026-08-03
|
||||||
|
|
||||||
This guide covers prerequisites, environment setup, running the application locally and in production, and troubleshooting. It references the [architecture documentation](./docs/architecture/README.md)(`docs/architecture/README.md`).
|
Prerequisites, environment setup, and running the app locally and in production. Architecture: [`docs/architecture/README.md`](./architecture/README.md).
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
- **Docker & Docker Compose**: Required for running the full stack (playout, chat, nginx, redis, db, prometheus, grafana). Install from [docker.com](https://www.docker.com).
|
- **Docker + Docker Compose** — runs the full stack (playout, chat, nginx, redis, db, prometheus, grafana).
|
||||||
- **Node.js 22+**: Required for frontend development and production builds (`cd frontend && npm install && npm run build`).
|
- **Node.js 22+ / npm** — frontend dev and builds (`cd frontend && npm install && npm run build`).
|
||||||
- **npm**: Installed with Node.js and used for the frontend dependency installation and Vite workflow.
|
- **Go 1.25+** — working on the Go services outside Docker.
|
||||||
- **Go 1.25+**: Recommended when working on the Go services outside Docker (`backend/chat`, `backend/playout`).
|
- **FFmpeg** — playout HLS generation (`apt install ffmpeg`).
|
||||||
- **FFmpeg**: Used by the playout service for HLS generation. Install via `apt install ffmpeg` (Linux), Homebrew (macOS), or Winget (Windows).
|
- **Git**.
|
||||||
- **Git**: For cloning and version control.
|
- **Cloudflare account** — R2 storage + CDN.
|
||||||
- **Cloudflare Account**: For R2 storage and CDN (free tier sufficient for most use cases).
|
- **Gitea account** — image registry + CI/CD. Set Actions secrets at `https://git.allucanget.biz/zwitschi/linearvoid/settings/actions/secrets`: `CI_REGISTRY_USER`, `CI_REGISTRY_TOKEN` (scope `write:packages`), `ADMIN_USERNAME`, `ADMIN_PASSWORD`, `DATABASE_URL`.
|
||||||
- **Gitea Account**: Required if you want to push images to the self-hosted registry or use the repository's CI/CD flow.
|
- **Hetzner Cloud (optional, prod)** — US-East VPS (Ashburn, VA per ADR-001).
|
||||||
- Set up the required Actions secrets in Gitea at: [https://git.allucanget.biz/zwitschi/linearvoid/settings/actions/secrets](https://git.allucanget.biz/zwitschi/linearvoid/settings/actions/secrets)
|
|
||||||
- Add the following secrets:
|
|
||||||
- `CI_REGISTRY_USER`: Your Gitea username (e.g., `zwitschi`)
|
|
||||||
- `CI_REGISTRY_TOKEN`: A Gitea Personal Access Token with `write:packages` scope
|
|
||||||
- `ADMIN_USERNAME`: The administrator username for the production admin panel (e.g., `admin`)
|
|
||||||
- `ADMIN_PASSWORD`: A strong password for the production admin panel
|
|
||||||
- `DATABASE_URL`: The production database connection string (e.g., `postgres://linearvoid:<password>@db:5432/linearvoid?sslmode=disable`)
|
|
||||||
- **Hetzner Cloud Account** (optional for production): For the US-East VPS (Ashburn, VA recommended per ADR-001).
|
|
||||||
|
|
||||||
## Network Environments
|
## Network Environments
|
||||||
|
|
||||||
Linear Void operates in two distinct environments. Know which one you're targeting.
|
Two distinct targets; know which one you're using.
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
flowchart LR
|
flowchart LR
|
||||||
subgraph Test_Lab["Test Lab (192.168.88.0/24)"]
|
subgraph Test_Lab["Test Lab (192.168.88.0/24)"]
|
||||||
TEST_HOST["192.168.88.40<br/>Docker Engine :2375"]
|
TEST_HOST["192.168.88.40<br/>Docker Engine :2375"]
|
||||||
end
|
end
|
||||||
|
|
||||||
subgraph Prod["Production (Hetzner — NOT provisioned)"]
|
subgraph Prod["Production (Hetzner — NOT provisioned)"]
|
||||||
PROD_HOST["Hetzner CX22 Ashburn, VA<br/>Public IP TBD"]
|
PROD_HOST["Hetzner CX22 Ashburn, VA<br/>Public IP TBD"]
|
||||||
end
|
end
|
||||||
|
|
||||||
subgraph Gitea["Gitea CI/CD"]
|
subgraph Gitea["Gitea CI/CD"]
|
||||||
CI["CI: build + push images"]
|
CI["CI: build + push images"]
|
||||||
CD["CD: deploy via DOCKER_HOST"]
|
CD["CD: deploy via DOCKER_HOST"]
|
||||||
end
|
end
|
||||||
|
|
||||||
subgraph CF["Cloudflare"]
|
subgraph CF["Cloudflare"]
|
||||||
DNS["linearvoid.tv"]
|
DNS["linearvoid.tv"]
|
||||||
R2["R2 Bucket"]
|
R2["R2 Bucket"]
|
||||||
end
|
end
|
||||||
|
|
||||||
CD -->|"DOCKER_HOST=tcp://192.168.88.40:2375"| TEST_HOST
|
CD -->|"DOCKER_HOST=tcp://192.168.88.40:2375"| TEST_HOST
|
||||||
CD -.->|"DOCKER_HOST=tcp://<hetzner-ip>:2375<br/>(when provisioned)"| PROD_HOST
|
CD -.->|"DOCKER_HOST=tcp://<hetzner-ip>:2375 (when provisioned)"| PROD_HOST
|
||||||
TEST_HOST -->|"test traffic"| DNS
|
TEST_HOST -->|"test traffic"| DNS
|
||||||
PROD_HOST -.->|"production traffic"| DNS
|
PROD_HOST -.->|"production traffic"| DNS
|
||||||
TEST_HOST --> R2
|
TEST_HOST --> R2
|
||||||
PROD_HOST -.-> R2
|
PROD_HOST -.-> R2
|
||||||
```
|
```
|
||||||
|
|
||||||
**Test Lab (active):** Internal network `192.168.88.0/24`, host `192.168.88.40`. Receives automated CD deploys from Gitea Actions. Used for integration testing and staging validation.
|
- **Test Lab (active):** internal `192.168.88.0/24`, host `192.168.88.40`; receives automated CD deploys from Gitea Actions. Used for staging validation.
|
||||||
|
- **Production (pending):** Hetzner CX22, Ashburn VA — not yet provisioned. Compose/CD are ready; only `DOCKER_HOST` needs changing. See [DEPLOYMENT.md](./DEPLOYMENT.md).
|
||||||
**Production (pending):** Hetzner Cloud VPS, Ashburn VA. Not yet provisioned. CD workflow and compose files are ready — only `DOCKER_HOST` needs changing. See Production Deployment section in [DEPLOYMENT.md](./DEPLOYMENT.md) for cutover plan.
|
|
||||||
|
|
||||||
## Development Prerequisites
|
|
||||||
|
|
||||||
- **Frontend dependencies**: Run `cd frontend && npm install` before `npm run dev` or `npm run build`.
|
|
||||||
- **Docker build support**: Local development expects Docker to be able to build the `chat`, `playout`, and `nginx` images referenced in `docker-compose.yml`.
|
|
||||||
- **Data directories**: Create `data/playlist`, `data/media`, `data/hls`, and `data/chat` before the first run so bind mounts resolve cleanly.
|
|
||||||
- **Database choice**: The stack defaults to PostgreSQL in `docker-compose.yml`. If you want SQLite for local-only development, update `DATABASE_URL` in `.env` accordingly.
|
|
||||||
- **Redis availability**: Chat presence, rate limiting, and pub/sub require Redis. The compose stack provides it automatically.
|
|
||||||
- **Video inputs**: Source videos should be normalized before playout. If the playlist contains invalid files, the playout container will fail its validation flow.
|
|
||||||
|
|
||||||
## Environment Setup
|
## Environment Setup
|
||||||
|
|
||||||
1. **Clone the Repository** (if not already done):
|
1. `git clone https://git.allucanget.biz/zwitschi/linearvoid.git && cd linearvoid`
|
||||||
|
2. `cp .env.example .env` then replace placeholders.
|
||||||
```bash
|
3. Compose overrides: `docker-compose.yml` builds images locally and mounts `frontend/dist` into nginx; `docker-compose.prod.yml` pulls prebuilt images from Gitea.
|
||||||
git clone https://git.allucanget.biz/zwitschi/linearvoid.git
|
- Internal URLs use compose hostnames: `REDIS_URL=redis://redis:6379/0`, `DATABASE_URL=postgres://linearvoid:<pw>@db:5432/linearvoid?sslmode=disable`.
|
||||||
cd linearvoid
|
- Public routing: `VITE_PUBLIC_APP_URL=https://linearvoid.tv`, `VITE_PUBLIC_WS_URL=wss://linearvoid.tv/ws`, `VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8`.
|
||||||
```
|
4. Create data dirs: `mkdir -p data/playlist data/media data/hls data/chat`
|
||||||
|
5. Optional: place normalized MP4s in `data/media/` and create `data/playlist/playlist.txt` (see ADR-001 for normalization).
|
||||||
2. **Copy Environment File**:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cp .env.example .env
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Configure `.env`** (copy from `.env.example` and replace placeholders)
|
|
||||||
|
|
||||||
4. **Check local value overrides for Docker Compose**:
|
|
||||||
- `docker-compose.yml` builds images locally and maps the frontend build output from `frontend/dist` into nginx.
|
|
||||||
- `docker-compose.prod.yml` pulls prebuilt images from the Gitea registry.
|
|
||||||
- Internal container-to-container URLs should use compose hostnames, for example `REDIS_URL=redis://redis:6379/0` and `DATABASE_URL=postgres://linearvoid:<password>@db:5432/linearvoid?sslmode=disable`.
|
|
||||||
- For production public routing, set `VITE_PUBLIC_APP_URL=https://linearvoid.tv`, `VITE_PUBLIC_WS_URL=wss://linearvoid.tv/ws`, and `VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8`.
|
|
||||||
|
|
||||||
5. **Create Data Directories**:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mkdir -p data/playlist data/media data/hls data/chat
|
|
||||||
```
|
|
||||||
|
|
||||||
6. **Prepare Sample Videos** (optional):
|
|
||||||
- Place normalized MP4 files in `data/media/`.
|
|
||||||
- Create a `data/playlist/playlist.txt` with paths (see ADR-001 for normalization requirements).
|
|
||||||
|
|
||||||
## Gitea Setup
|
## Gitea Setup
|
||||||
|
|
||||||
1. **Create or verify a Gitea user**:
|
1. User with repo + registry access; generate a token (scope `write:packages`) stored as `CI_REGISTRY_TOKEN`.
|
||||||
- The user should have access to the repository and permission to read from the container registry.
|
2. Set Actions secrets (see Prerequisites) to enable automated CI/CD and deploys.
|
||||||
|
3. Set registry vars: `CI_REGISTRY_URL`, `CI_REGISTRY_NAMESPACE` (e.g. `zwitschi/linearvoid`), `CI_REGISTRY_USER`.
|
||||||
2. **Create an access token**:
|
4. Auth: `docker login git.allucanget.biz`.
|
||||||
- Generate a token with the scopes needed for package/registry access and, if applicable, repository automation.
|
5. Local dev builds from source; prod pulls `playout`, `chat`, `nginx` images from the registry namespace. Verify `:latest` tags exist before rollout.
|
||||||
- Store it as `CI_REGISTRY_TOKEN` in `.env`.
|
|
||||||
|
|
||||||
3. **Configure Gitea Actions Secrets**:
|
|
||||||
- Go to the repository settings page: [https://git.allucanget.biz/zwitschi/linearvoid/settings/actions/secrets](https://git.allucanget.biz/zwitschi/linearvoid/settings/actions/secrets)
|
|
||||||
- Add the following secrets to enable automated CI/CD and production deployment:
|
|
||||||
- `CI_REGISTRY_USER`: Your Gitea username (e.g., `zwitschi`).
|
|
||||||
- `CI_REGISTRY_TOKEN`: Your Gitea Personal Access Token with `write:packages` scope.
|
|
||||||
- `ADMIN_USERNAME`: The administrator username for the production admin panel (e.g., `admin`).
|
|
||||||
- `ADMIN_PASSWORD`: A strong password for the production admin panel.
|
|
||||||
- `DATABASE_URL`: The production database connection string (e.g., `postgres://linearvoid:<password>@db:5432/linearvoid?sslmode=disable`).
|
|
||||||
|
|
||||||
4. **Configure registry variables**:
|
|
||||||
- Set `CI_REGISTRY_URL` to the Gitea host.
|
|
||||||
- Set `CI_REGISTRY_NAMESPACE` to the image namespace, for example `zwitschi/linearvoid`.
|
|
||||||
- Set `CI_REGISTRY_USER` to the account that will push or pull images.
|
|
||||||
|
|
||||||
5. **Authenticate Docker to the registry**:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker login git.allucanget.biz
|
|
||||||
```
|
|
||||||
|
|
||||||
6. **Build and publish images when needed**:
|
|
||||||
- Local development uses `docker-compose.yml`, which builds from source.
|
|
||||||
- Production uses `docker-compose.prod.yml`, which expects the `playout`, `chat`, and `nginx` images to exist in the configured Gitea registry namespace.
|
|
||||||
|
|
||||||
7. **Validate image availability**:
|
|
||||||
- Before a production rollout, verify that the expected `:latest` tags can be pulled from the registry on the target server.
|
|
||||||
|
|
||||||
## Cloudflare R2 Setup
|
## Cloudflare R2 Setup
|
||||||
|
|
||||||
1. **Create an R2 bucket**:
|
1. Create R2 bucket (e.g. `linearvoid-media`, matching `R2_BUCKET_NAME`); generate an access key pair → `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_REGION=auto`.
|
||||||
- Create a bucket such as `linearvoid-media` in Cloudflare R2.
|
2. S3 endpoint: `https://<account-id>.r2.cloudflarestorage.com` → `R2_ENDPOINT`.
|
||||||
- Keep the bucket name aligned with `R2_BUCKET_NAME` in `.env`.
|
3. Attach `media.linearvoid.tv` as custom media domain → `MEDIA_DOMAIN`; add CORS allowing `https://linearvoid.tv` GET for HLS delivery.
|
||||||
|
4. Point `linearvoid.tv` (+ `www` redirect) at the app entrypoint; keep `media.*` on R2. Proxied DNS, TLS **Full (strict)**.
|
||||||
2. **Generate R2 API credentials**:
|
5. Apply the `.m4s`/`.m3u8` cache rules from [`cache-rules.md`](./infrastructure/cloudflare/cache-rules.md); enable Smart Tiered Cache; optionally deploy `infrastructure/cloudflare/worker.js` for header normalization.
|
||||||
- Create an access key pair with read/write permissions for the bucket.
|
6. Verify: `aws s3 ls --endpoint-url $R2_ENDPOINT`; check cache hit ratio after serving traffic.
|
||||||
- Set `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, and `R2_REGION=auto` in `.env`.
|
|
||||||
|
|
||||||
3. **Find the S3 endpoint**:
|
|
||||||
- Use the account-scoped endpoint format `https://<account-id>.r2.cloudflarestorage.com`.
|
|
||||||
- Store the account ID in both `R2_ENDPOINT` and `CLOUDFLARE_ACCOUNT_ID` where applicable.
|
|
||||||
|
|
||||||
4. **Bind a custom media domain**:
|
|
||||||
- Attach a domain such as `media.linearvoid.tv` to the bucket.
|
|
||||||
- Set `MEDIA_DOMAIN` to that hostname.
|
|
||||||
|
|
||||||
5. **Configure app-domain DNS and TLS**:
|
|
||||||
- Point `linearvoid.tv` to the public entrypoint serving nginx.
|
|
||||||
- Point `www.linearvoid.tv` to the same target and redirect it to `https://linearvoid.tv`.
|
|
||||||
- Keep `media.linearvoid.tv` on the R2 custom domain path so HLS delivery stays split from the app origin.
|
|
||||||
- Use Cloudflare proxied DNS with Full (strict) TLS so browser traffic uses `https://linearvoid.tv`, `wss://linearvoid.tv/ws`, and `https://media.linearvoid.tv`.
|
|
||||||
|
|
||||||
6. **Apply cache behavior**:
|
|
||||||
- Configure the `.m4s` and `.m3u8` cache rules from [cache-rules](../docs/infrastructure/cloudflare/cache-rules.md).
|
|
||||||
- Enable Smart Tiered Cache for the R2-backed hostname.
|
|
||||||
- If you need header normalization for immutable segment caching, deploy `infrastructure/cloudflare/worker.js` to the same zone.
|
|
||||||
|
|
||||||
7. **Configure R2 CORS**:
|
|
||||||
- Add CORS rules to allow the app domain to access the R2 bucket for HLS delivery.
|
|
||||||
- Example CORS configuration:
|
|
||||||
|
|
||||||
```json
|
|
||||||
[
|
|
||||||
{
|
|
||||||
"AllowedOrigins": ["https://linearvoid.tv"],
|
|
||||||
"AllowedMethods": ["GET"],
|
|
||||||
"AllowedHeaders": ["*"],
|
|
||||||
"ExposeHeaders": ["ETag", "Cache-Control"],
|
|
||||||
"MaxAgeSeconds": 3600
|
|
||||||
}
|
|
||||||
]
|
|
||||||
```
|
|
||||||
|
|
||||||
8. **Verify access**:
|
|
||||||
- Confirm uploads with an S3-compatible client such as `aws s3 ls --endpoint-url <R2_ENDPOINT>`.
|
|
||||||
- After deployment, check Cloudflare analytics for cache hit ratio and origin request volume.
|
|
||||||
|
|
||||||
## Cloudflare
|
|
||||||
|
|
||||||
Use this section for full edge setup (DNS, Workers, cache, security, and monitoring).
|
|
||||||
|
|
||||||
Execution tracker: `docs/operations/cloudflare-uptimerobot-checklist.md`.
|
|
||||||
|
|
||||||
### 1. DNS Setup
|
|
||||||
|
|
||||||
1. Add apex record for `linearvoid.tv` pointing to the Cloudflare Worker.
|
|
||||||
2. Add `www` CNAME to the Cloudflare Worker and configure redirect to `https://linearvoid.tv`.
|
|
||||||
3. Add `media.linearvoid.tv` for split-domain HLS delivery.
|
|
||||||
4. Keep all records proxied (orange cloud) unless troubleshooting requires DNS-only mode.
|
|
||||||
|
|
||||||
### 2. R2 Bucket
|
|
||||||
|
|
||||||
1. Create bucket `linearvoid-media`.
|
|
||||||
2. Set location hint to `enam` (US-East).
|
|
||||||
3. Attach custom domain `media.linearvoid.tv` to the bucket.
|
|
||||||
4. Configure CORS rules to allow `https://linearvoid.tv` for HLS delivery.
|
|
||||||
5. Verify object read works from the custom domain.
|
|
||||||
|
|
||||||
### 3. Cache Rules
|
|
||||||
|
|
||||||
Configure Cloudflare Cache Rules:
|
|
||||||
|
|
||||||
1. HLS segment files (`*.ts`, `*.m4s`):
|
|
||||||
- Cache: eligible
|
|
||||||
- Edge TTL: `31536000`
|
|
||||||
- Browser TTL: long-lived
|
|
||||||
- Cache key: include full path
|
|
||||||
2. HLS manifests (`*.m3u8`):
|
|
||||||
- Cache: eligible
|
|
||||||
- Edge TTL: `2s`
|
|
||||||
- Revalidation: enabled
|
|
||||||
3. Static assets (`*.js`, `*.css`):
|
|
||||||
- Cache: eligible
|
|
||||||
- Edge TTL: `1y`
|
|
||||||
- Browser TTL: long-lived
|
|
||||||
- Cache key: include full path
|
|
||||||
4. Validate behavior using response headers:
|
|
||||||
- `Cf-Cache-Status` should move from `MISS` to `HIT` for hot objects
|
|
||||||
|
|
||||||
Reference: [cache-rules](../docs/infrastructure/cloudflare/cache-rules.md).
|
|
||||||
|
|
||||||
### 4. Cloudflare Workers
|
|
||||||
|
|
||||||
Also see [the automated Gitea Actions deployment guide](./operations/gitea-cloudflare-workers.md).
|
|
||||||
|
|
||||||
Manual setup:
|
|
||||||
|
|
||||||
1. Create a Cloudflare Worker for the static SPA (Workers & Pages → Create → Worker).
|
|
||||||
2. Build the frontend:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd frontend && npm run build
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Bind the build output (`frontend/dist`) as Worker static assets.
|
|
||||||
4. Configure production domain `linearvoid.tv` on the Worker.
|
|
||||||
|
|
||||||
### 5. Tiered Cache
|
|
||||||
|
|
||||||
1. Enable Smart Tiered Cache in Cloudflare dashboard.
|
|
||||||
2. Warm a segment and manifest path.
|
|
||||||
3. Confirm tiered behavior through repeated requests and `Cf-Cache-Status: HIT`.
|
|
||||||
|
|
||||||
### 6. WAF and Bot Protection
|
|
||||||
|
|
||||||
1. Enable OWASP Managed Ruleset.
|
|
||||||
2. Start in `Simulate` mode.
|
|
||||||
3. Review events for 48 hours.
|
|
||||||
4. Switch to `Block` once false positives are addressed.
|
|
||||||
5. Enable Bot Fight Mode.
|
|
||||||
|
|
||||||
### 7. Rate Limiting
|
|
||||||
|
|
||||||
Create rules:
|
|
||||||
|
|
||||||
1. Path: `/api/admin/login`
|
|
||||||
2. Path: `/api/admin/*`
|
|
||||||
3. Threshold: `10 requests / 10 seconds / IP`
|
|
||||||
4. Action: `Managed Challenge`
|
|
||||||
|
|
||||||
### 8. HTTPS and WSS
|
|
||||||
|
|
||||||
1. Enable `Always Use HTTPS`.
|
|
||||||
2. Enable HSTS with:
|
|
||||||
- `max-age=31536000`
|
|
||||||
- `includeSubDomains`
|
|
||||||
- `preload`
|
|
||||||
3. Minimum TLS version: `1.2`.
|
|
||||||
4. Keep SSL mode at `Full (strict)` when origin certs are valid.
|
|
||||||
5. Enforce HTTPS redirect at both layers:
|
|
||||||
- Cloudflare edge
|
|
||||||
- nginx origin
|
|
||||||
6. Validate websocket endpoint uses `wss://linearvoid.tv/ws`.
|
|
||||||
|
|
||||||
### 9. UptimeRobot Monitors
|
|
||||||
|
|
||||||
Create external monitors:
|
|
||||||
|
|
||||||
1. HTTP monitor:
|
|
||||||
- URL: `https://linearvoid.tv/health`
|
|
||||||
- Interval: 5 minutes
|
|
||||||
2. Keyword monitor:
|
|
||||||
- URL: `https://linearvoid.tv/hls/playlist.m3u8`
|
|
||||||
- Keyword: `#EXTM3U`
|
|
||||||
- Interval: 5 minutes
|
|
||||||
3. Alert policy:
|
|
||||||
- Trigger on 2 consecutive failures
|
|
||||||
- Notify via email and Discord/Slack webhook
|
|
||||||
|
|
||||||
### 10. Verification Checklist
|
|
||||||
|
|
||||||
1. DNS resolves for `linearvoid.tv`, `www.linearvoid.tv`, and `media.linearvoid.tv`.
|
|
||||||
2. Workers serve frontend for apex and `www` redirects correctly.
|
|
||||||
3. HLS segments are cacheable long-term, manifests refresh every ~2s.
|
|
||||||
4. Admin paths are rate-limited and WAF events are visible.
|
|
||||||
5. HTTPS and WSS both succeed with valid certificates.
|
|
||||||
6. UptimeRobot alerts fire as expected when endpoints are intentionally made unavailable.
|
|
||||||
|
|
||||||
## Running the Application
|
## Running the Application
|
||||||
|
|
||||||
### Local Development
|
### Local development
|
||||||
|
|
||||||
1. **Start All Services**:
|
```bash
|
||||||
|
docker compose up --build # start all services
|
||||||
|
docker compose down # stop
|
||||||
|
```
|
||||||
|
|
||||||
```bash
|
Access: app/admin `https://linearvoid.tv` (via proxy) or `http://localhost`; WS `wss://linearvoid.tv/ws` / `ws://localhost/ws`. Health: playout `:8081/health`, chat `:3000/health`. Monitoring: Prometheus `:9090`, Grafana `:3001` (default `admin/admin`).
|
||||||
docker compose up --build
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Access the Application**:
|
Frontend dev (optional): `cd frontend && npm install && npm run dev` (http://localhost:5173, proxies API). For nginx-backed local runs: `npm run build` (nginx serves `frontend/dist`).
|
||||||
- Preferred local access with the documented proxy setup: <https://linearvoid.tv>
|
|
||||||
- Admin Interface: <https://linearvoid.tv/admin> (redirects to `/admin/login/` until authenticated)
|
|
||||||
- Chat WebSocket: `wss://linearvoid.tv/ws`
|
|
||||||
- Direct-local fallback when bypassing Cloudflare/Nginx Proxy Manager: <http://localhost>, <http://localhost/admin>, and `ws://localhost/ws`
|
|
||||||
- Health Checks:
|
|
||||||
- Playout: <http://localhost:8081/health>
|
|
||||||
- Chat: <http://localhost:3000/health>
|
|
||||||
- Monitoring:
|
|
||||||
- Prometheus: <http://localhost:9090>
|
|
||||||
- Grafana: <http://localhost:3001> (default login: admin/admin)
|
|
||||||
|
|
||||||
3. **Frontend Development** (optional, for UI changes):
|
### Test lab (remote Docker host)
|
||||||
|
|
||||||
```bash
|
Deploy to `192.168.88.40` over Docker's TCP socket (port `2375`).
|
||||||
cd frontend
|
|
||||||
npm install
|
|
||||||
npm run dev
|
|
||||||
```
|
|
||||||
|
|
||||||
The dev server runs on <http://localhost:5173>. It proxies API calls to the backend services.
|
Enable TCP on Alpine host — edit `/etc/conf.d/docker`:
|
||||||
|
|
||||||
4. **Build frontend assets for nginx-backed local runs**:
|
```bash
|
||||||
|
DOCKER_OPTS="-H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375"
|
||||||
|
```
|
||||||
|
|
||||||
```bash
|
Then `rc-service docker restart`; verify with `netstat -tlnp | grep 2375`; allow port `2375` from `192.168.88.0/24` in the firewall.
|
||||||
cd frontend
|
|
||||||
npm install
|
|
||||||
npm run build
|
|
||||||
```
|
|
||||||
|
|
||||||
The local nginx container serves `frontend/dist`, so rebuilding the frontend is required before validating production-like static output.
|
Deploy from a machine with the Docker CLI:
|
||||||
|
|
||||||
5. **Stop Services**:
|
```bash
|
||||||
|
export DOCKER_HOST="tcp://192.168.88.40:2375"
|
||||||
|
docker info # confirm remote host
|
||||||
|
docker compose -f docker-compose.prod.yml pull
|
||||||
|
docker compose -f docker-compose.prod.yml up -d --remove-orphans # registry images
|
||||||
|
# or build from source: docker compose up -d --build
|
||||||
|
unset DOCKER_HOST # back to local
|
||||||
|
```
|
||||||
|
|
||||||
```bash
|
Health: `http://192.168.88.40:8081/health` (playout), `:3000/health` (chat), nginx entrypoint `:8080`.
|
||||||
docker compose down
|
|
||||||
```
|
|
||||||
|
|
||||||
### Local Test Environment Deployment (Remote Docker Host)
|
### Coolify deployment
|
||||||
|
|
||||||
For testing deployments on a local network server without setting up a full production VPS, you can deploy directly to a remote Docker host (e.g., a home lab server or test machine) using Docker's TCP socket.
|
Coolify replaces the Gitea CD `DOCKER_HOST` orchestration (Cloudflare DNS/R2/cache unchanged). Install (`curl -fsSL https://cdn.coollabs.io/coolify/install.sh | bash`), add the target server via SSH, create a Docker Compose resource from `docker-compose.prod.yml` (or the repo), configure env vars in Coolify's store, map persistent `data/` and named volumes, and assign `linearvoid.tv` to nginx for Traefik TLS. Full walkthrough: [`COOLIFY.md`](./COOLIFY.md).
|
||||||
|
|
||||||
#### Test Environment Prerequisites
|
### Production setup
|
||||||
|
|
||||||
- A remote server running Docker at IP `192.168.88.40` (running Alpine Linux).
|
1. Provision a Linux VPS (Docker, Compose, Git); open HTTP/HTTPS ports.
|
||||||
- Docker TCP socket enabled and listening on port `2375` on the remote server.
|
2. `git clone … && cd linearvoid && cp .env.example .env`.
|
||||||
|
3. Set production `.env`: strong `POSTGRES_PASSWORD`/`GRAFANA_PASSWORD`, compose-hostname internal URLs, `MEDIA_DOMAIN=media.linearvoid.tv`, public `VITE_PUBLIC_*` URLs. For local-only testing bypassing the proxy, use `http://localhost`/`ws://localhost/ws`/`http://localhost/hls/playlist.m3u8`.
|
||||||
|
4. `mkdir -p data/playlist data/media data/hls data/chat`.
|
||||||
|
5. Ensure `playout`, `chat`, `nginx` images exist in the registry; `docker login` from the VPS.
|
||||||
|
6. Deploy: `docker compose -f docker-compose.prod.yml pull && docker compose -f docker-compose.prod.yml up -d`.
|
||||||
|
7. Verify: `docker compose -f docker-compose.prod.yml ps`, health endpoints, HLS written to `data/hls`, media domain serves via Cloudflare.
|
||||||
|
8. Harden: HTTPS in front of nginx (Cloudflare, `www` → apex), restrict Grafana/Prometheus to trusted IPs, run under systemd for auto-start.
|
||||||
|
|
||||||
##### Enabling Docker TCP Socket on Alpine Linux
|
### Production (Hetzner VPS + Cloudflare)
|
||||||
|
|
||||||
To allow the Gitea Actions runner (or your local machine) to deploy to the remote Alpine host, you must configure the Docker daemon to listen on the TCP socket:
|
1. Provision CX22 in Ashburn, VA (ADR-001).
|
||||||
|
2. Cloudflare: R2 bucket (`enam` hint), `media.linearvoid.tv` custom domain + cache rules, `linearvoid.tv`/`www` at the app entrypoint with `www`→apex redirect, Full (strict) TLS, deploy `infrastructure/cloudflare/worker.js`.
|
||||||
|
3. Deploy from the project: `docker compose -f docker-compose.prod.yml up -d` (or pull from Gitea registry); manage under systemd.
|
||||||
|
4. Monitoring: Grafana dashboards from `monitoring/grafana/`.
|
||||||
|
|
||||||
1. Edit `/etc/conf.d/docker` on the Alpine host:
|
See `docs/architecture/README.md` (arc42) and the ADRs for decisions and cost/scaling guidance (ADR-001).
|
||||||
|
|
||||||
```bash
|
|
||||||
vi /etc/conf.d/docker
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Update the `DOCKER_OPTS` line to include both the unix socket and the TCP socket:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
DOCKER_OPTS="-H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375"
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Restart the Docker service:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
rc-service docker restart
|
|
||||||
```
|
|
||||||
|
|
||||||
4. Verify that Docker is listening on port `2375`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
netstat -tlnp | grep 2375
|
|
||||||
```
|
|
||||||
|
|
||||||
5. Ensure any firewall (like `awall` or `iptables`) on the Alpine host allows incoming traffic on port `2375` from your local network subnet (e.g., `192.168.88.0/24`).
|
|
||||||
|
|
||||||
#### Deployment Steps
|
|
||||||
|
|
||||||
1. **Configure Docker CLI to use the Remote Host**:
|
|
||||||
You can point your local Docker CLI to the remote server by setting the `DOCKER_HOST` environment variable:
|
|
||||||
- **PowerShell (Windows)**:
|
|
||||||
|
|
||||||
```powershell
|
|
||||||
$env:DOCKER_HOST="tcp://192.168.88.40:2375"
|
|
||||||
```
|
|
||||||
|
|
||||||
- **Bash (Linux/macOS)**:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
export DOCKER_HOST="tcp://192.168.88.40:2375"
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Verify Connection**:
|
|
||||||
Run `docker info` to confirm your local CLI is communicating with the remote server at `192.168.88.40`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker info
|
|
||||||
```
|
|
||||||
|
|
||||||
The output should show the remote server's hostname and system details.
|
|
||||||
|
|
||||||
3. **Deploy the Stack**:
|
|
||||||
With `DOCKER_HOST` set, any `docker` or `docker compose` commands you run locally will execute directly on the remote server.
|
|
||||||
- **Deploy from Registry (Production-like)**:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker compose -f docker-compose.prod.yml pull
|
|
||||||
docker compose -f docker-compose.prod.yml up -d --remove-orphans
|
|
||||||
```
|
|
||||||
|
|
||||||
- **Build and Deploy from Source**:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker compose up -d --build
|
|
||||||
```
|
|
||||||
|
|
||||||
4. **Access the Test Environment**:
|
|
||||||
- Playout Health: `http://192.168.88.40:8081/health`
|
|
||||||
- Chat Health: `http://192.168.88.40:3000/health`
|
|
||||||
- Nginx Entrypoint: `http://192.168.88.40:8080` (or the port mapped to `NGINX_PORT` in `.env`)
|
|
||||||
|
|
||||||
5. **Reset to Local Docker Host**:
|
|
||||||
Once testing is complete, clear the `DOCKER_HOST` variable to point back to your local machine:
|
|
||||||
- **PowerShell**:
|
|
||||||
|
|
||||||
```powershell
|
|
||||||
Remove-Item env:DOCKER_HOST
|
|
||||||
```
|
|
||||||
|
|
||||||
- **Bash**:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
unset DOCKER_HOST
|
|
||||||
```
|
|
||||||
|
|
||||||
### Coolify Deployment
|
|
||||||
|
|
||||||
Coolify is an open-source, self-hostable PaaS (Heroku/Netlify/Vercel
|
|
||||||
alternative) that can deploy the Linear Void stack through a web UI with
|
|
||||||
automatic HTTPS and one-click redeploys. It replaces the deployment
|
|
||||||
orchestration layer (the Gitea CD `DOCKER_HOST` flow) while leaving Cloudflare
|
|
||||||
DNS, R2 media, and cache rules unchanged.
|
|
||||||
|
|
||||||
To deploy via Coolify:
|
|
||||||
|
|
||||||
1. **Install Coolify** on a dedicated server:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl -fsSL https://cdn.coollabs.io/coolify/install.sh | bash
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Add the target server** in the dashboard via SSH (or use Localhost if
|
|
||||||
Coolify runs on the same machine as the stack).
|
|
||||||
3. **Create a Docker Compose resource** pointing at either the Git repository
|
|
||||||
(`docker-compose.prod.yml`) or the compose file contents directly.
|
|
||||||
4. **Configure environment variables** in Coolify's secret store (the values
|
|
||||||
normally held in `.env`), keeping internal URLs on compose hostnames
|
|
||||||
(`redis`, `db`).
|
|
||||||
5. **Map persistent storage** for `data/` mounts and the `redis-data` /
|
|
||||||
`postgres-data` named volumes so data survives redeploys.
|
|
||||||
6. **Assign a domain** to the nginx service so Coolify's Traefik proxy
|
|
||||||
terminates TLS for `linearvoid.tv`, while `media.linearvoid.tv` continues to
|
|
||||||
serve HLS from Cloudflare R2.
|
|
||||||
|
|
||||||
For the full walkthrough, environment-variable table, verification steps, and
|
|
||||||
cutover notes, see [COOLIFY.md](./COOLIFY.md).
|
|
||||||
|
|
||||||
### Production Setup Guide
|
|
||||||
|
|
||||||
1. **Prepare the target host**:
|
|
||||||
- Provision a Linux VPS with Docker Engine, Docker Compose, Git, and enough disk for media, HLS output, PostgreSQL, Redis persistence, and monitoring data.
|
|
||||||
- Open the ports required for HTTP/HTTPS and any restricted monitoring access you intend to expose.
|
|
||||||
|
|
||||||
2. **Clone the repository and create the environment file**:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git clone https://git.allucanget.biz/zwitschi/linearvoid.git
|
|
||||||
cd linearvoid
|
|
||||||
cp .env.example .env
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Set production-safe `.env` values**:
|
|
||||||
- Replace placeholder secrets.
|
|
||||||
- Use compose-network hostnames for internal services, for example `redis` and `db`.
|
|
||||||
- Set strong values for `POSTGRES_PASSWORD` and `GRAFANA_PASSWORD`.
|
|
||||||
- Keep `MEDIA_DOMAIN`, R2 credentials, and Gitea registry values consistent with the external infrastructure.
|
|
||||||
- Set `MEDIA_DOMAIN=media.linearvoid.tv`.
|
|
||||||
- Set `VITE_PUBLIC_APP_URL=https://linearvoid.tv`, `VITE_PUBLIC_WS_URL=wss://linearvoid.tv/ws`, and `VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8`.
|
|
||||||
- If you intentionally bypass the public proxy for local-only testing, switch those values to `http://localhost`, `ws://localhost/ws`, and `http://localhost/hls/playlist.m3u8`.
|
|
||||||
|
|
||||||
4. **Create runtime directories on the server**:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mkdir -p data/playlist data/media data/hls data/chat
|
|
||||||
```
|
|
||||||
|
|
||||||
5. **Publish or verify required images in Gitea**:
|
|
||||||
- Ensure the `playout`, `chat`, and `nginx` images exist under `${CI_REGISTRY_URL}/${CI_REGISTRY_NAMESPACE}`.
|
|
||||||
- Log in to the registry from the VPS before pulling.
|
|
||||||
|
|
||||||
6. **Deploy the production stack**:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker compose -f docker-compose.prod.yml pull
|
|
||||||
docker compose -f docker-compose.prod.yml up -d
|
|
||||||
```
|
|
||||||
|
|
||||||
7. **Run post-deploy verification**:
|
|
||||||
- Check `docker compose -f docker-compose.prod.yml ps`.
|
|
||||||
- Confirm nginx, chat, and playout health endpoints respond.
|
|
||||||
- Verify that HLS output is being written to `data/hls` and that the media domain serves manifests and segments through Cloudflare.
|
|
||||||
|
|
||||||
8. **Harden and operate the deployment**:
|
|
||||||
- Put HTTPS in front of nginx using Cloudflare or another reverse proxy, with `linearvoid.tv` as canonical app host and `www.linearvoid.tv` redirected.
|
|
||||||
- Restrict Grafana and Prometheus exposure to trusted IPs or a private network.
|
|
||||||
- Run the stack under systemd or another service manager if you need auto-start behavior outside Docker restart policies.
|
|
||||||
|
|
||||||
### Production Deployment (Hetzner VPS + Cloudflare)
|
|
||||||
|
|
||||||
1. **Provision VPS**: Use Hetzner Cloud CX22 in Ashburn, VA (per ADR-001).
|
|
||||||
2. **Configure Cloudflare**:
|
|
||||||
- Create R2 bucket with `enam` location hint.
|
|
||||||
- Set up custom media domain (`media.linearvoid.tv`) with the cache rules from [cache-rules](../docs/infrastructure/cloudflare/cache-rules.md).
|
|
||||||
- Point `linearvoid.tv` and `www.linearvoid.tv` at the app entrypoint, then redirect `www` to the apex domain.
|
|
||||||
- Use Full (strict) TLS for both app and media hostnames.
|
|
||||||
- Apply the Worker from `infrastructure/cloudflare/worker.js` for advanced header management.
|
|
||||||
3. **Deploy**:
|
|
||||||
- Copy the project to the VPS.
|
|
||||||
- Use production compose: `docker compose -f docker-compose.prod.yml up -d`
|
|
||||||
- Or pull from Gitea registry: `docker compose -f docker-compose.prod.yml pull && docker compose -f docker-compose.prod.yml up -d`
|
|
||||||
- Configure systemd to manage the Docker Compose stack.
|
|
||||||
4. **Monitoring**: Access Grafana at the exposed port for dashboards (included in `monitoring/grafana/dashboard.json`).
|
|
||||||
|
|
||||||
See `docs/architecture/README.md` for the full arc42 documentation and the ADRs for architectural decisions.
|
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
- **R2 Upload Fails**: Check credentials in `.env` and ensure the bucket exists. Verify with `aws s3 ls --endpoint-url=$R2_ENDPOINT`.
|
- **R2 upload fails** — check `.env` creds/bucket; `aws s3 ls --endpoint-url=$R2_ENDPOINT`.
|
||||||
- **Normalization Errors**: Run `backend/playout/start-playout.sh` manually to see ffprobe output. All videos must be H.264/AAC, 30 fps constant, 2s keyframes (ADR-001).
|
- **Normalization errors** — run `backend/playout/start-playout.sh` to see ffprobe output. Video must be H.264/AAC, 30 fps constant, 2s keyframes (ADR-001).
|
||||||
- **Chat Not Connecting**: Check Redis connection and WebSocket URL. The health check at `/health` should return 200.
|
- **Chat won't connect** — check Redis and WS URL; `/health` should return 200.
|
||||||
- **Cache Not Working**: Verify Cloudflare cache rules (`.m4s` = long TTL immutable, `.m3u8` = short TTL). Check cache hit ratio in Cloudflare dashboard.
|
- **Cache not working** — verify `.m4s` long-TTL / `.m3u8` short-TTL cache rules and hit ratio.
|
||||||
- **Docker Issues**: Run `docker compose logs` for service-specific errors. Ensure ports are not in use.
|
- **Docker issues** — `docker compose logs`; check port conflicts.
|
||||||
- **Admin Interface**: Ensure `ADMIN_USERNAME`, `ADMIN_PASSWORD`, and `ADMIN_SESSION_TTL_HOURS` are set. `/admin/` and `/api/admin/*` are protected by a session-based login flow.
|
- **Admin interface** — `ADMIN_USERNAME`, `ADMIN_PASSWORD`, `ADMIN_SESSION_TTL_HOURS` set; session-based login protects `/admin/` and `/api/admin/*`.
|
||||||
- **Production Pull Fails**: Confirm `docker login` to Gitea succeeded on the server and the image tags referenced by `docker-compose.prod.yml` exist.
|
- **Prod pull fails** — `docker login` to Gitea succeeded and image tags exist.
|
||||||
|
|
||||||
## Further Reading
|
## Further Reading
|
||||||
|
|
||||||
- Architecture: `docs/architecture/README.md` (arc42 template with diagrams).
|
Architecture: `docs/architecture/README.md` · Decisions: `docs/architecture/adr/` (ADR-001…007) · Contributing: `CONTRIBUTING.md` · Overview: root `README.md`.
|
||||||
- Decisions: `docs/architecture/adr/` (ADR-001 to ADR-007).
|
|
||||||
- Contributing: `CONTRIBUTING.md`.
|
|
||||||
- Project Structure: Root `README.md`.
|
|
||||||
|
|
||||||
The system is ready for use. For production, review ADR-001 for cost estimates and scaling guidance.
|
|
||||||
+67
-80
@@ -1,6 +1,6 @@
|
|||||||
# Streaming Pipeline
|
# Streaming Pipeline
|
||||||
|
|
||||||
This document is the operational contract for Linear Void video ingest, playout, publication, caching, and browser playback. General deployment, chat, and admin details remain in `DEPLOYMENT.md`, `SETUP.md`, and `API.md`.
|
Operational contract for Linear Void video ingest, playout, publication, caching, and browser playback. Deployment, chat, admin details live in `DEPLOYMENT.md`, `SETUP.md`, `API.md`.
|
||||||
|
|
||||||
## Production flow
|
## Production flow
|
||||||
|
|
||||||
@@ -20,7 +20,7 @@ The playout service is the timeline authority. It reads the ordered playlist, ru
|
|||||||
https://media.linearvoid.tv/hls/playlist.m3u8
|
https://media.linearvoid.tv/hls/playlist.m3u8
|
||||||
```
|
```
|
||||||
|
|
||||||
Nginx `/hls/` is a local/test fallback, not the production media origin. The SPA, admin API, and chat use `linearvoid.tv`; media uses `media.linearvoid.tv`.
|
Nginx `/hls/` is a local/test fallback, not the production origin. The SPA, admin API, and chat use `linearvoid.tv`; media uses `media.linearvoid.tv`.
|
||||||
|
|
||||||
## Source-media contract
|
## Source-media contract
|
||||||
|
|
||||||
@@ -30,12 +30,12 @@ Only validated files may enter the production playlist.
|
|||||||
| ---------- | ------------------------------------------------------------------------ |
|
| ---------- | ------------------------------------------------------------------------ |
|
||||||
| Container | MP4 |
|
| Container | MP4 |
|
||||||
| Video | H.264/AVC, `yuv420p` |
|
| Video | H.264/AVC, `yuv420p` |
|
||||||
| Resolution | 1280x720 or 1920x1080; avoid changing resolution within a playlist block |
|
| Resolution | 1280x720 or 1920x1080; avoid resolution changes within a playlist block |
|
||||||
| Frame rate | Constant 30 fps |
|
| Frame rate | Constant 30 fps |
|
||||||
| GOP | Closed, fixed 2 seconds: `-g 60 -keyint_min 60 -sc_threshold 0` |
|
| GOP | Closed, fixed 2 seconds: `-g 60 -keyint_min 60 -sc_threshold 0` |
|
||||||
| Audio | AAC-LC, stereo, 48 kHz, 128–192 kbit/s |
|
| Audio | AAC-LC, stereo, 48 kHz, 128–192 kbit/s |
|
||||||
|
|
||||||
The ingest gate uses `ffprobe` to reject incompatible codecs, variable frame rate, missing audio, invalid dimensions, or incorrect keyframe cadence. Assets are immutable after acceptance; corrections create a new asset ID and object key.
|
The ingest gate uses `ffprobe` to reject incompatible codecs, variable frame rate, missing audio, invalid dimensions, or wrong keyframe cadence. Assets are immutable after acceptance; corrections create a new asset ID/object key.
|
||||||
|
|
||||||
Reference normalization:
|
Reference normalization:
|
||||||
|
|
||||||
@@ -51,124 +51,111 @@ ffmpeg -i input.mov \
|
|||||||
|
|
||||||
## Playlist and timeline
|
## Playlist and timeline
|
||||||
|
|
||||||
- `data/playlist/playlist.txt` contains one local media path per line; blank lines and `#` comments are ignored.
|
- `data/playlist/playlist.txt`: one local media path per line; blank lines and `#` comments ignored.
|
||||||
- The service validates the complete playlist before activating a revision.
|
- Service validates the complete playlist before activating a revision.
|
||||||
- Playlist edits are written atomically and receive a revision ID. The active FFmpeg run keeps its current item and adopts the new revision at the next item boundary.
|
- Playlist edits are atomic and get a revision ID. The active FFmpeg run keeps its current item and adopts the new revision at the next item boundary.
|
||||||
- All viewers follow the same live manifest; clients do not calculate the schedule independently.
|
- All viewers follow the same live manifest; clients never calculate the schedule independently.
|
||||||
- Schedule metadata must be derived from the same active playlist revision and timeline so “now/next” agrees with the video.
|
- Schedule metadata derives from the same active revision/timeline so "now/next" agrees with the video.
|
||||||
- When the last item ends, playout returns to the first item unless an active schedule override says otherwise.
|
- When the last item ends, playout returns to the first unless an active schedule override says otherwise.
|
||||||
|
|
||||||
## Playout and HLS packaging
|
## Playout and HLS packaging
|
||||||
|
|
||||||
The playout service wraps FFmpeg, tracks the active item and playlist revision, restarts failed processes, and exposes `/health` plus Prometheus metrics.
|
The playout service wraps FFmpeg, tracks the active item/revision, restarts failed processes, and exposes `/health` plus Prometheus metrics.
|
||||||
|
|
||||||
Baseline packaging:
|
| Setting | Value |
|
||||||
|
| -------------- | ---------------------------------------------------------------------------------- |
|
||||||
|
| Protocol | Standard live HLS, not LL-HLS |
|
||||||
|
| Segment format | fMP4 (`.m4s`) with init segment; `.ts` may be supported during migration |
|
||||||
|
| Segment target | 2 seconds (`HLS_SEGMENT_DURATION=2`) |
|
||||||
|
| Live window | 12 segments (`HLS_WINDOW_SIZE=12`), ~24 seconds |
|
||||||
|
| Codec path | Stream copy for conforming assets; no per-viewer transcoding |
|
||||||
|
| Expected latency | 15–35 seconds |
|
||||||
|
|
||||||
| Setting | Value |
|
Segment boundaries must align with keyframes; timestamps must stay monotonic across item boundaries. If FFmpeg cannot preserve a seamless timeline, insert `#EXT-X-DISCONTINUITY` and publish the corresponding new init segment.
|
||||||
| ---------------- | ------------------------------------------------------------------------------------- |
|
|
||||||
| Protocol | Standard live HLS, not LL-HLS |
|
|
||||||
| Segment format | fMP4 (`.m4s`) with an initialization segment; `.ts` may be supported during migration |
|
|
||||||
| Segment target | 2 seconds (`HLS_SEGMENT_DURATION=2`) |
|
|
||||||
| Live window | 12 segments (`HLS_WINDOW_SIZE=12`), about 24 seconds |
|
|
||||||
| Codec path | Stream copy for conforming assets; no per-viewer transcoding |
|
|
||||||
| Expected latency | 15–35 seconds |
|
|
||||||
|
|
||||||
Segment boundaries must align with keyframes. Timestamps must remain monotonic across item boundaries. If FFmpeg cannot preserve a seamless timeline, insert `#EXT-X-DISCONTINUITY` and publish the corresponding new initialization segment.
|
**Segment duration rationale**: 2s lowers latency and keeps the window small (12 × 2s = 24s) while staying above the manifest TTL (1–2s). The 2s GOP (30fps, `-g 60`) already matches the target, so segments cut cleanly on keyframes. Longer (4–6s) raises latency and window depth; shorter than 2s inflates segment/request overhead and R2 Class B ops with little gain.
|
||||||
|
|
||||||
**Segment duration rationale**: 2s segments were chosen to lower end-to-end latency and keep the live window small (12 × 2s = 24s) while staying well above the manifest TTL (1–2s). The 2s GOP (30fps, `-g 60 -keyint_min 60`) already matches the target, so segments cut cleanly on keyframes. Longer durations (4–6s) increase latency and window depth; shorter than 2s inflates segment/request overhead and R2 Class B operations with little latency gain.
|
Every segment name is globally unique and never reused, including after restart or playlist loop — use a run/session ID plus a monotonically increasing media sequence:
|
||||||
|
|
||||||
Every segment name is globally unique and never reused, including after a restart or playlist loop. Use a run/session ID plus a monotonically increasing media sequence, for example:
|
|
||||||
|
|
||||||
```text
|
```text
|
||||||
hls/r-20260807T170412Z/init-000001.mp4
|
hls/r-20260807T170412Z/init-000001.mp4
|
||||||
hls/r-20260807T170412Z/seg-000001234.m4s
|
hls/r-20260807T170412Z/seg-000001234.m4s
|
||||||
```
|
```
|
||||||
|
|
||||||
This naming rule is essential: a URL advertised as immutable must never later contain different bytes.
|
A URL advertised as immutable must never later contain different bytes.
|
||||||
|
|
||||||
## Safe publication to R2
|
## Safe publication to R2
|
||||||
|
|
||||||
Publication order prevents a manifest from referencing an unavailable segment:
|
Order prevents a manifest referencing an unavailable segment:
|
||||||
|
|
||||||
1. FFmpeg writes a segment to a temporary local name and closes it.
|
1. FFmpeg writes a segment to a temp local name and closes it.
|
||||||
2. The publisher uploads the initialization segment, if new, and media segment to their final immutable R2 keys.
|
2. Publisher uploads the init segment (if new) and media segment to final immutable R2 keys.
|
||||||
3. It verifies successful upload and records object size/checksum.
|
3. Verify upload success; record object size/checksum.
|
||||||
4. It renders the next media manifest referencing only confirmed objects.
|
4. Render the next manifest referencing only confirmed objects.
|
||||||
5. It uploads the manifest to a temporary key, then replaces `hls/playlist.m3u8` as the final commit step.
|
5. Upload the manifest to a temp key, then replace `hls/playlist.m3u8` as the final commit step.
|
||||||
|
|
||||||
Never upload an actively written segment or publish the manifest first. Use a single publisher instance, or fencing/leader election, so two playout workers cannot update the live manifest concurrently.
|
Never upload an actively written segment or publish the manifest first. Use a single publisher (or fencing/leader election) so two workers can't update the live manifest concurrently.
|
||||||
|
|
||||||
The local `data/hls/` directory is staging data and may be regenerated. Source media, playlist revisions, and schedule state are authoritative; generated HLS is not backed up as application state.
|
`data/hls/` is regenerable staging. Source media, playlist revisions, and schedule state are authoritative; generated HLS is not backed up as application state.
|
||||||
|
|
||||||
## Cache contract
|
## Cache contract
|
||||||
|
|
||||||
| Objects | Edge policy | Browser policy | Required headers/behavior |
|
| Objects | Edge policy | Browser policy | Required headers/behavior |
|
||||||
| -------------------------- | ---------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------- |
|
| -------------------------- | --------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------- |
|
||||||
| `.m4s`, `.ts`, init `.mp4` | Cache eligible; TTL 1 year | TTL 1 year | `Cache-Control: public, max-age=31536000, immutable`; full path in cache key |
|
| `.m4s`, `.ts`, init `.mp4` | Cache eligible; TTL 1 year | TTL 1 year | `Cache-Control: public, max-age=31536000, immutable`; full path in cache key |
|
||||||
| Live `.m3u8` | Cache eligible; edge TTL 2 seconds | Do not retain beyond freshness interval | `Cache-Control: public, max-age=1, s-maxage=2, must-revalidate` |
|
| Live `.m3u8` | Cache eligible; edge TTL 2s | Do not retain beyond freshness interval | `Cache-Control: public, max-age=1, s-maxage=2, must-revalidate` |
|
||||||
| Versioned JS/CSS | Cache eligible; TTL 1 year | TTL 1 year | Content-hashed filenames and `immutable` |
|
| Versioned JS/CSS | Cache eligible; TTL 1 year | TTL 1 year | Content-hashed filenames and `immutable` |
|
||||||
|
|
||||||
Further rules:
|
Further rules:
|
||||||
|
|
||||||
- Do not add query-string cache busters to segment URLs; uniqueness belongs in the path.
|
- No query-string cache busters on segment URLs; uniqueness belongs in the path.
|
||||||
- Cache keys must preserve hostname and full path. Ignore only explicitly approved tracking parameters.
|
- Cache keys preserve hostname and full path; ignore only explicitly approved tracking params.
|
||||||
- HLS responses require `Access-Control-Allow-Origin: https://linearvoid.tv` (or the reviewed origin allowlist), `GET`, `HEAD`, and `OPTIONS` support, and correct MIME types.
|
- HLS responses need `Access-Control-Allow-Origin: https://linearvoid.tv` (or reviewed allowlist), `GET`/`HEAD`/`OPTIONS`, and correct MIME types.
|
||||||
- Enable Smart Tiered Cache for `media.linearvoid.tv` to reduce R2 reads on regional misses.
|
- Enable Smart Tiered Cache for `media.linearvoid.tv` to cut R2 reads on regional misses.
|
||||||
- Purge manifests when necessary, but never purge immutable segments during normal operation.
|
- Purge manifests when necessary, never immutable segments during normal operation.
|
||||||
- Retain expired-window segments in R2 for at least the maximum manifest TTL plus client retry/grace time. A practical minimum is one hour; lifecycle deletion may run later according to storage policy.
|
- Retain expired-window segments at least max manifest TTL + retry/grace time; practical minimum 1 hour; lifecycle deletion may run later per storage policy.
|
||||||
|
|
||||||
Expected validation:
|
Validation:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -I https://media.linearvoid.tv/hls/playlist.m3u8
|
curl -I https://media.linearvoid.tv/hls/playlist.m3u8
|
||||||
curl -I https://media.linearvoid.tv/hls/<segment>.m4s
|
curl -I https://media.linearvoid.tv/hls/<segment>.m4s
|
||||||
```
|
```
|
||||||
|
|
||||||
The manifest must refresh within a few seconds. Repeated segment requests should progress from `CF-Cache-Status: MISS` to `HIT`, retain the same ETag/content length, and carry the immutable cache header.
|
The manifest must refresh within seconds; repeated segment requests should progress `CF-Cache-Status: MISS` → `HIT`, keep the same ETag/content length, and carry the immutable cache header.
|
||||||
|
|
||||||
## Player behavior
|
## Player behavior
|
||||||
|
|
||||||
The SPA uses native HLS where supported and hls.js elsewhere.
|
SPA uses native HLS where supported, hls.js elsewhere.
|
||||||
|
|
||||||
- Firefox has no native HLS, so it uses the hls.js/MSE path; fMP4 (`.m4s` + init segment) is supported. Autoplay requires `muted` + `playsinline`, which are already set on the `<video>` element.
|
|
||||||
|
|
||||||
|
- Firefox has no native HLS → hls.js/MSE path; fMP4 (`.m4s` + init) is supported. Autoplay needs `muted` + `playsinline`, already set on the `<video>` element.
|
||||||
- Start at the live edge with a conservative buffer; do not seek to playlist start.
|
- Start at the live edge with a conservative buffer; do not seek to playlist start.
|
||||||
- Treat the manifest as the synchronization source and target roughly three segments behind the newest segment unless testing supports a smaller margin.
|
- Treat the manifest as the sync source; target ~3 segments behind the newest unless testing supports a smaller margin.
|
||||||
- Recover transient network/media errors with bounded retries and exponential backoff.
|
- Recover transient network/media errors with bounded retries + exponential backoff.
|
||||||
- On a fatal hls.js network error, reload the manifest; on a fatal media error, attempt media recovery once before recreating the player.
|
- On fatal hls.js network error, reload the manifest; on fatal media error, attempt media recovery once before recreating the player.
|
||||||
- When tab suspension or drift leaves playback materially behind the target live position, seek forward to the target rather than playing the backlog.
|
- If tab suspension/drift leaves playback materially behind the target, seek forward rather than play the backlog.
|
||||||
- Surface a clear reconnecting state. Never silently loop stale media.
|
- Surface a clear reconnecting state; never silently loop stale media.
|
||||||
- Schedule/chat timestamps should account for the measured player latency rather than assuming wall-clock playout is already visible.
|
- Schedule/chat timestamps should account for measured player latency, not wall-clock playout.
|
||||||
|
|
||||||
## Failure and recovery
|
## Failure and recovery
|
||||||
|
|
||||||
The service retries an item failure with a bounded backoff. After the configured failure threshold it switches to a pre-normalized looping slate and reports `mode=slate`. It periodically retries normal playout and returns only after a segment and committed manifest have been published successfully.
|
The service retries an item failure with bounded backoff. Past the failure threshold it switches to a pre-normalized looping slate and reports `mode=slate`, periodically retrying normal playout, returning only after a segment and committed manifest publish successfully.
|
||||||
|
|
||||||
On process or host restart:
|
On process/host restart:
|
||||||
|
|
||||||
- create a new run/session ID and continue with globally unique segment paths;
|
- create a new run/session ID and continue with unique segment paths;
|
||||||
- resume from the schedule-derived current position when reliable state exists, otherwise start the current playlist item;
|
- resume from schedule-derived current position when reliable state exists, else start the current playlist item;
|
||||||
- never overwrite old segment objects;
|
- never overwrite old segment objects;
|
||||||
- keep the last valid manifest available until the replacement stream is playable;
|
- keep the last valid manifest available until the replacement is playable;
|
||||||
- emit a discontinuity when timeline continuity cannot be guaranteed.
|
- emit a discontinuity when timeline continuity isn't guaranteed.
|
||||||
|
|
||||||
Health is not merely “process running.” `/health` should report at least mode (`live`, `slate`, `starting`, `degraded`), active item, playlist revision, run ID, last segment sequence/time, last successful R2 publication, consecutive failures, and uptime. Mark the service unhealthy if the manifest or media sequence has not advanced for more than two segment durations.
|
`/health` must report more than "process running": at least mode (`live`/`slate`/`starting`/`degraded`), active item, playlist revision, run ID, last segment sequence/time, last successful R2 publication, consecutive failures, uptime. Mark unhealthy if the manifest or media sequence hasn't advanced for more than two segment durations.
|
||||||
|
|
||||||
## Monitoring and acceptance checks
|
## Monitoring and acceptance
|
||||||
|
|
||||||
Monitor and alert on:
|
Monitor/alert on: manifest age and media-sequence advancement; time since last published segment; FFmpeg restarts and consecutive failures; live/slate mode; R2 upload latency/errors; segment 404/5xx and manifest/segment latency; Cloudflare cache-hit ratio, origin requests, R2 Class B ops; player startup, fatal errors, rebuffer ratio, live-edge drift; local staging disk usage.
|
||||||
|
|
||||||
- manifest age and media-sequence advancement;
|
A release is streaming-ready only when: one full playlist boundary plays without timestamp errors; the manifest never references a missing object; a second viewer joins the same timeline; cache headers match the table; a forced FFmpeg failure activates slate; and normal playout recovers without reusing an object URL.
|
||||||
- time since last successfully published segment;
|
|
||||||
- FFmpeg restarts and consecutive item failures;
|
|
||||||
- live/slate mode;
|
|
||||||
- R2 upload latency and errors;
|
|
||||||
- segment 404/5xx rate and manifest/segment request latency;
|
|
||||||
- Cloudflare cache-hit ratio, origin requests, and R2 Class B operations;
|
|
||||||
- player startup time, fatal errors, rebuffer ratio, and live-edge drift;
|
|
||||||
- local staging disk usage.
|
|
||||||
|
|
||||||
A release is streaming-ready only when one full playlist boundary has played without timestamp errors, the manifest never references a missing object, a second viewer joins the same timeline, cache headers match the table above, a forced FFmpeg failure activates the slate, and normal playout recovers without reusing an object URL.
|
|
||||||
|
|
||||||
## Required configuration
|
## Required configuration
|
||||||
|
|
||||||
@@ -179,14 +166,14 @@ A release is streaming-ready only when one full playlist boundary has played wit
|
|||||||
| `HLS_WINDOW_SIZE` | Manifest window; baseline `12` |
|
| `HLS_WINDOW_SIZE` | Manifest window; baseline `12` |
|
||||||
| `R2_ENDPOINT`, `R2_REGION` | S3-compatible publication endpoint |
|
| `R2_ENDPOINT`, `R2_REGION` | S3-compatible publication endpoint |
|
||||||
| `R2_BUCKET_NAME` | Media bucket |
|
| `R2_BUCKET_NAME` | Media bucket |
|
||||||
| `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY` | Write credentials, available only to the publisher |
|
| `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY` | Write credentials, publisher only |
|
||||||
| `R2_KEEP_RUNS` | Number of run folders to retain (default `2`) |
|
| `R2_KEEP_RUNS` | Run folders to retain (default `2`) |
|
||||||
| `R2_MAX_RETENTION_BYTES` | Max bytes retained across run folders (10 GiB) |
|
| `R2_MAX_RETENTION_BYTES` | Max bytes across run folders (10 GiB) |
|
||||||
| `MEDIA_DOMAIN` | Public R2 custom domain |
|
| `MEDIA_DOMAIN` | Public R2 custom domain |
|
||||||
| `VITE_PUBLIC_HLS_URL` | Public live manifest URL |
|
| `VITE_PUBLIC_HLS_URL` | Public live manifest URL |
|
||||||
|
|
||||||
Production values are `MEDIA_DOMAIN=media.linearvoid.tv` and `VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8`. Local fallback may use `http://localhost/hls/playlist.m3u8`.
|
Production values: `MEDIA_DOMAIN=media.linearvoid.tv`, `VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8`. Local fallback may use `http://localhost/hls/playlist.m3u8`.
|
||||||
|
|
||||||
## Explicitly deferred
|
## Explicitly deferred
|
||||||
|
|
||||||
Adaptive-bitrate ladders, LL-HLS, DVR/catch-up, multi-origin playout, and regional chat affinity are outside the v1 contract. Adding any of them requires an ADR and an update to this document.
|
ABR ladders, LL-HLS, DVR/catch-up, multi-origin playout, and regional chat affinity are outside the v1 contract. Adding any requires an ADR and an update to this document.
|
||||||
+224
-693
File diff suppressed because it is too large.
Load diff
+57
-126
@@ -4,155 +4,101 @@ Triage order for Linear Void incidents. Work top-to-bottom — each tier depends
|
|||||||
|
|
||||||
## Triage Order
|
## Triage Order
|
||||||
|
|
||||||
1. Edge Reachability (Cloudflare)
|
1. **Edge Reachability** (Cloudflare)
|
||||||
2. Origin Health (VPS/Docker)
|
2. **Origin Health** (VPS/Docker)
|
||||||
3. Playout Progression (HLS)
|
3. **Playout Progression** (HLS)
|
||||||
4. Chat Connectivity (WebSocket)
|
4. **Chat Connectivity** (WebSocket)
|
||||||
5. Data-Plane (Redis/DB)
|
5. **Data-Plane** (Redis/DB)
|
||||||
|
|
||||||
## 1. Edge Reachability (Cloudflare)
|
## 1. Edge Reachability (Cloudflare)
|
||||||
|
|
||||||
**Symptoms:** `linearvoid.tv` not resolving, SSL errors, 5xx from Cloudflare.
|
**Symptoms:** `linearvoid.tv` not resolving, SSL errors, 5xx from Cloudflare.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Check DNS
|
|
||||||
dig linearvoid.tv
|
dig linearvoid.tv
|
||||||
dig www.linearvoid.tv
|
dig www.linearvoid.tv
|
||||||
|
|
||||||
# Check Cloudflare status
|
|
||||||
curl -I https://linearvoid.tv/health
|
curl -I https://linearvoid.tv/health
|
||||||
```
|
```
|
||||||
|
|
||||||
**Common causes:**
|
**Common causes:** DNS misconfiguration (A record → wrong IP); Cloudflare proxy disabled (grey cloud exposes origin); SSL certificate expired (Cloudflare → SSL/TLS); Cloudflare outage ([cloudflarestatus.com](https://www.cloudflarestatus.com)).
|
||||||
|
|
||||||
- DNS misconfiguration (A record pointing to wrong IP)
|
**Mitigation:** Verify A record points `linearvoid.tv` → origin IP (proxied, orange cloud). Bypass Cloudflare temporarily:
|
||||||
- Cloudflare proxy disabled (grey cloud → origin exposed directly)
|
|
||||||
- SSL certificate expired (check Cloudflare dashboard → SSL/TLS)
|
|
||||||
- Cloudflare outage → check [https://www.cloudflarestatus.com](https://www.cloudflarestatus.com)
|
|
||||||
|
|
||||||
**Mitigation:**
|
```bash
|
||||||
|
curl -H "Host: linearvoid.tv" http://<origin-ip>/health
|
||||||
- Verify A record: `linearvoid.tv` → origin IP (proxied, orange cloud)
|
```
|
||||||
- Bypass Cloudflare temporarily: `curl -H "Host: linearvoid.tv" http://<origin-ip>/health`
|
|
||||||
|
|
||||||
## 2. Origin Health (VPS/Docker)
|
## 2. Origin Health (VPS/Docker)
|
||||||
|
|
||||||
**Symptoms:** Cloudflare returns 502/504, containers down.
|
**Symptoms:** Cloudflare 502/504, containers down.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Remote Docker (test lab)
|
|
||||||
docker -H tcp://192.168.88.40:2375 ps
|
docker -H tcp://192.168.88.40:2375 ps
|
||||||
docker -H tcp://192.168.88.40:2375 logs linearvoid-nginx --tail 50
|
docker -H tcp://192.168.88.40:2375 logs linearvoid-nginx --tail 50
|
||||||
|
|
||||||
# Direct health check
|
|
||||||
curl http://192.168.88.40/health
|
curl http://192.168.88.40/health
|
||||||
```
|
```
|
||||||
|
|
||||||
**Common causes:**
|
**Common causes:** Host disk full (`df -h`); Docker daemon stopped (`systemctl status docker`); container restart loop (`docker ps -a --filter "status=exited"`); OOM (`dmesg | grep -i oom`).
|
||||||
|
|
||||||
- Host out of disk space: `df -h`
|
|
||||||
- Docker daemon stopped: `systemctl status docker`
|
|
||||||
- Container restart loop: `docker ps -a --filter "status=exited"`
|
|
||||||
- Resource exhaustion (OOM): `dmesg | grep -i oom`
|
|
||||||
|
|
||||||
**Mitigation:**
|
**Mitigation:**
|
||||||
|
|
||||||
- Restart stack: `docker compose -f docker-compose.prod.yml up -d`
|
```bash
|
||||||
- Free disk: `docker system prune -f`
|
docker compose -f docker-compose.prod.yml up -d # restart stack
|
||||||
- Check logs per service: `docker logs linearvoid-<service> --tail 100`
|
docker system prune -f # free disk
|
||||||
|
docker logs linearvoid-<service> --tail 100 # per-service logs
|
||||||
|
```
|
||||||
|
|
||||||
## 3. Playout Progression (HLS)
|
## 3. Playout Progression (HLS)
|
||||||
|
|
||||||
**Symptoms:** Viewer sees slate or buffering, playlist not advancing.
|
**Symptoms:** Viewer sees slate or buffering, playlist not advancing.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Check playout health
|
curl http://192.168.88.40:8081/health # "mode": "live" or "slate"
|
||||||
curl http://192.168.88.40:8081/health
|
|
||||||
# Response includes "mode": "live" or "slate"
|
|
||||||
|
|
||||||
# Check HLS output
|
|
||||||
curl -I http://192.168.88.40/hls/playlist.m3u8
|
curl -I http://192.168.88.40/hls/playlist.m3u8
|
||||||
ls -la data/hls/
|
ls -la data/hls/
|
||||||
|
|
||||||
# Check playout logs
|
|
||||||
docker logs linearvoid-playout --tail 100
|
docker logs linearvoid-playout --tail 100
|
||||||
```
|
```
|
||||||
|
|
||||||
**Common causes:**
|
**Common causes:** FFmpeg crash → slate fallback (health `degraded`); playlist empty/missing (`cat data/playlist/playlist.txt`); media file missing (`ls data/media/`); disk full → HLS segments can't write.
|
||||||
|
|
||||||
- FFmpeg crash → slate fallback active (health shows `degraded`)
|
**Mitigation:** Validate playlist entries and media paths; restart playout (`docker compose restart playout`); regenerate demo playlist if empty (see `backend/playout/start-playout.sh`).
|
||||||
- Playlist file empty/missing: `cat data/playlist/playlist.txt`
|
|
||||||
- Media file missing from playlist: `ls data/media/`
|
|
||||||
- Disk full → HLS segments can't write
|
|
||||||
|
|
||||||
**Mitigation:**
|
|
||||||
|
|
||||||
- Verify playlist has valid entries
|
|
||||||
- Check media files exist at paths in playlist
|
|
||||||
- Restart playout: `docker compose restart playout`
|
|
||||||
- Generate demo playlist if empty: see `backend/playout/start-playout.sh`
|
|
||||||
|
|
||||||
## 4. Chat Connectivity (WebSocket)
|
## 4. Chat Connectivity (WebSocket)
|
||||||
|
|
||||||
**Symptoms:** Chat not loading, messages not sending, presence broken.
|
**Symptoms:** Chat not loading, messages not sending, presence broken.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Check chat health
|
|
||||||
curl http://192.168.88.40:3000/health
|
curl http://192.168.88.40:3000/health
|
||||||
|
|
||||||
# Test WebSocket
|
|
||||||
wscat -c ws://192.168.88.40/ws?username=test&room=main
|
wscat -c ws://192.168.88.40/ws?username=test&room=main
|
||||||
|
|
||||||
# Check chat logs
|
|
||||||
docker logs linearvoid-chat --tail 100
|
docker logs linearvoid-chat --tail 100
|
||||||
```
|
```
|
||||||
|
|
||||||
**Common causes:**
|
**Common causes:** Redis down → presence/rate-limiting degraded (chat still works); DB down → messages not persisting; port not exposed (`docker port linearvoid-chat`).
|
||||||
|
|
||||||
- Redis down → presence/rate-limiting degraded (chat still works)
|
**Mitigation:** Restart chat (`docker compose restart chat`); check Redis (`docker exec linearvoid-redis redis-cli ping`); check DB (`docker exec linearvoid-db pg_isready -U linearvoid`). Chat degrades gracefully without Redis — look for "Redis connection warning" in logs.
|
||||||
- Database down → messages not persisting
|
|
||||||
- Port not exposed: `docker port linearvoid-chat`
|
|
||||||
|
|
||||||
**Mitigation:**
|
|
||||||
|
|
||||||
- Restart chat: `docker compose restart chat`
|
|
||||||
- Check Redis: `docker exec linearvoid-redis redis-cli ping`
|
|
||||||
- Check DB: `docker exec linearvoid-db pg_isready -U linearvoid`
|
|
||||||
- Chat works without Redis (graceful degradation) — check logs for "Redis connection warning"
|
|
||||||
|
|
||||||
## 5. Data-Plane (Redis/DB)
|
## 5. Data-Plane (Redis/DB)
|
||||||
|
|
||||||
**Symptoms:** Chat works but no persistence/presence, admin stats empty.
|
**Symptoms:** Chat works but no persistence/presence, admin stats empty.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Redis
|
|
||||||
docker exec linearvoid-redis redis-cli ping
|
docker exec linearvoid-redis redis-cli ping
|
||||||
docker exec linearvoid-redis redis-cli info memory
|
docker exec linearvoid-redis redis-cli info memory
|
||||||
|
|
||||||
# PostgreSQL
|
|
||||||
docker exec linearvoid-db pg_isready -U linearvoid
|
docker exec linearvoid-db pg_isready -U linearvoid
|
||||||
docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"
|
docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Common causes:**
|
**Common causes:** Redis OOM (check `maxmemory`); PostgreSQL connection limit; disk full on volume mount.
|
||||||
|
|
||||||
- Redis OOM (check `maxmemory` setting)
|
**Mitigation:** Restart redis (`docker compose restart redis`, AOF persist enabled); restart db (`docker compose restart db`); restore from backup (see `scripts/verify-backups.sh`).
|
||||||
- PostgreSQL connection limit reached
|
|
||||||
- Disk full on volume mount
|
|
||||||
|
|
||||||
**Mitigation:**
|
|
||||||
|
|
||||||
- Redis: `docker compose restart redis` (AOF persist enabled)
|
|
||||||
- PostgreSQL: `docker compose restart db`
|
|
||||||
- Restore from backup: see `scripts/verify-backups.sh` for procedure
|
|
||||||
|
|
||||||
## Escalation
|
## Escalation
|
||||||
|
|
||||||
| Severity | When | Action |
|
| Severity | When | Action |
|
||||||
| -------- | ------------------------------------ | -------------------------------------------------- |
|
| -------- | ------------------------------------ | ------------------------------------------- |
|
||||||
| Critical | Slate active >30min, complete outage | Trigger DR plan, provision fallback host |
|
| Critical | Slate active >30min, complete outage | Trigger DR plan, provision fallback host |
|
||||||
| Warning | Redis down, high CPU >30min | Investigate during business hours, restart service |
|
| Warning | Redis down, high CPU >30min | Investigate business hours, restart service |
|
||||||
| Info | Single service blip | Monitor, no action unless recurring |
|
| Info | Single service blip | Monitor; no action unless recurring |
|
||||||
|
|
||||||
## Post-Incident
|
## Post-Incident
|
||||||
|
|
||||||
@@ -163,67 +109,52 @@ docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"
|
|||||||
|
|
||||||
## Quarterly Disaster Recovery Drill
|
## Quarterly Disaster Recovery Drill
|
||||||
|
|
||||||
Goal: validate recovery targets under controlled host-loss simulation.
|
Validate recovery targets under controlled host-loss simulation: **RPO ≤ 24h, RTO ≤ 60min**.
|
||||||
|
|
||||||
**Recovery targets:** RPO <= 24h, RTO <= 60min.
|
|
||||||
|
|
||||||
### Prerequisites
|
### Prerequisites
|
||||||
|
|
||||||
- Latest backup artifacts available from `scripts/backup-db.sh` and `scripts/backup-weekly.sh`
|
- Latest backups from `scripts/backup-db.sh` and `scripts/backup-weekly.sh`
|
||||||
- Operator access to Docker host and DNS controls
|
- Operator access to Docker host and DNS controls
|
||||||
- Current deployment variables available (`.env`, compose file, image tags)
|
- Current deployment variables (`.env`, compose file, image tags)
|
||||||
|
|
||||||
### Drill Scope
|
### Scope
|
||||||
|
|
||||||
- Simulate complete origin loss (all app containers unavailable)
|
- Simulate complete origin loss (all app containers unavailable)
|
||||||
- Rebuild stack on clean host
|
- Rebuild stack on clean host
|
||||||
- Restore DB and media/playlist mounts
|
- Restore DB and media/playlist mounts
|
||||||
- Verify viewer playback, chat, admin auth, monitoring targets
|
- Verify viewer playback, chat, admin auth, monitoring targets
|
||||||
|
|
||||||
### Drill Steps
|
### Steps
|
||||||
|
|
||||||
1. Record drill start timestamp and on-call participants.
|
1. Record drill start timestamp and on-call participants.
|
||||||
2. Capture pre-drill baseline:
|
2. Capture pre-drill baseline:
|
||||||
- `curl -fsS http://<origin>/health`
|
```bash
|
||||||
- `curl -fsS http://<origin>:3000/health`
|
curl -fsS http://<origin>/health
|
||||||
- `curl -fsS http://<origin>:8081/health`
|
curl -fsS http://<origin>:3000/health
|
||||||
3. Simulate origin outage:
|
curl -fsS http://<origin>:8081/health
|
||||||
- `docker compose -f docker-compose.prod.yml down`
|
```
|
||||||
4. Provision clean recovery host (or clean test-lab VM) and copy deployment bundle.
|
3. Simulate outage: `docker compose -f docker-compose.prod.yml down`
|
||||||
5. Restore required data paths:
|
4. Provision clean recovery host (or clean test-lab VM); copy deployment bundle.
|
||||||
- `data/playlist/`
|
5. Restore data paths: `data/playlist/`, `data/media/`, `data/chat/`, `data/hls/` (optional warm cache).
|
||||||
- `data/media/`
|
|
||||||
- `data/chat/`
|
|
||||||
- `data/hls/` (optional warm cache)
|
|
||||||
6. Restore database from latest backup dump.
|
6. Restore database from latest backup dump.
|
||||||
7. Start stack:
|
7. Start stack: `docker compose -f docker-compose.prod.yml up -d`
|
||||||
- `docker compose -f docker-compose.prod.yml up -d`
|
8. Validate health and runtime:
|
||||||
8. Validate service health and runtime:
|
```bash
|
||||||
- `docker ps --format "table {{.Names}}\t{{.Status}}"`
|
docker ps --format "table {{.Names}}\t{{.Status}}"
|
||||||
- `curl -fsS http://<origin>/health`
|
curl -fsS http://<origin>/health
|
||||||
- `curl -fsS http://<origin>/hls/playlist.m3u8 | head -n 5`
|
curl -fsS http://<origin>/hls/playlist.m3u8 | head -n 5
|
||||||
- `curl -fsS http://<origin>:9090/-/ready`
|
curl -fsS http://<origin>:9090/-/ready
|
||||||
9. Validate app behavior:
|
```
|
||||||
- Viewer SPA loads and HLS plays
|
9. Validate app behavior: viewer SPA loads and HLS plays; WebSocket chat connects with message round-trip; admin login/session endpoint succeeds.
|
||||||
- WebSocket chat connects and message round-trip works
|
10. Run backup integrity check: `scripts/verify-backups.sh`
|
||||||
- Admin login/session endpoint returns success
|
11. Record drill end timestamp; compute measured RTO/RPO.
|
||||||
10. Run backup integrity check:
|
|
||||||
- `scripts/verify-backups.sh`
|
|
||||||
11. Record drill end timestamp and compute measured RTO/RPO.
|
|
||||||
|
|
||||||
### Pass/Fail Criteria
|
### Pass/Fail Criteria
|
||||||
|
|
||||||
- Pass if all core services healthy and functional in <= 60 minutes.
|
- **Pass:** all core services healthy and functional in ≤ 60 min.
|
||||||
- Pass if latest recoverable data point is <= 24 hours old.
|
- **Pass:** latest recoverable data point ≤ 24 h old.
|
||||||
- Fail if any critical path remains degraded (playout, chat, admin auth, monitoring).
|
- **Fail:** any critical path remains degraded (playout, chat, admin auth, monitoring).
|
||||||
|
|
||||||
### Evidence Template
|
### Evidence Template
|
||||||
|
|
||||||
Document each drill in incident tracker with:
|
Record in incident tracker: date/time window; environment (test-lab, staging, production); backup artifact IDs and timestamps; measured RTO and RPO; issues found and remediation owner; follow-up ticket links.
|
||||||
|
|
||||||
- Date/time window
|
|
||||||
- Environment (test-lab, staging, production)
|
|
||||||
- Backup artifact IDs and timestamps
|
|
||||||
- Measured RTO and RPO
|
|
||||||
- Issues found and remediation owner
|
|
||||||
- Follow-up ticket links
|
|
||||||
@@ -6,19 +6,19 @@ Use `wrangler whoami` to verify the account ID and API token are valid, or query
|
|||||||
|
|
||||||
## Validate Cloudflare account and API token
|
## Validate Cloudflare account and API token
|
||||||
|
|
||||||
```powershell
|
```bash
|
||||||
# Validate Cloudflare account and API token in .env
|
# Validate Cloudflare account and API token in .env
|
||||||
Set-Location -Path "E:\Documents\02-Projects\07-Mentoring\Corey\linearvoid"
|
cd /path/to/linearvoid
|
||||||
Get-Content .env | Select-String 'CLOUDFLARE_API_TOKEN'
|
grep -E 'CLOUDFLARE_API_TOKEN|CLOUDFLARE_ACCOUNT_ID' .env
|
||||||
Get-Content .env | Select-String 'CLOUDFLARE_ACCOUNT_ID'
|
|
||||||
|
|
||||||
# Test API token with wrangler
|
# Test API token with wrangler
|
||||||
npx wrangler whoami
|
npx wrangler whoami
|
||||||
|
|
||||||
# Or verify with the Workers API directly
|
# Or verify with the Workers API directly
|
||||||
$env:CLOUDFLARE_API_TOKEN = (Get-Content .env | Select-String 'CLOUDFLARE_API_TOKEN' | ForEach-Object { $_.Line.Split('=')[1].Trim() })
|
export CLOUDFLARE_API_TOKEN="$(grep '^CLOUDFLARE_API_TOKEN=' .env | cut -d= -f2-)"
|
||||||
$env:CLOUDFLARE_ACCOUNT_ID = (Get-Content .env | Select-String 'CLOUDFLARE_ACCOUNT_ID' | ForEach-Object { $_.Line.Split('=')[1].Trim() })
|
export CLOUDFLARE_ACCOUNT_ID="$(grep '^CLOUDFLARE_ACCOUNT_ID=' .env | cut -d= -f2-)"
|
||||||
|
|
||||||
# List all Workers for the account
|
# List all Workers for the account
|
||||||
curl -X GET "https://api.cloudflare.com/client/v4/accounts/$env:CLOUDFLARE_ACCOUNT_ID/workers/scripts" -H "Authorization: Bearer $env:CLOUDFLARE_API_TOKEN" -H "Content-Type: application/json"
|
curl -X GET "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/workers/scripts" \
|
||||||
|
-H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" -H "Content-Type: application/json"
|
||||||
```
|
```
|
||||||
File diff suppressed because it is too large.
Load diff
@@ -1,345 +1,156 @@
|
|||||||
# Scaling and Performance Playbook
|
# Scaling and Performance Playbook
|
||||||
|
|
||||||
Purpose: define repeatable method to measure performance, capture baseline data, and decide scaling actions.
|
Purpose: define a repeatable method to measure performance, capture baselines, and decide scaling actions for the "Linear Void" streaming app. Scope: planning/execution for the "Scaling & Performance (planning only)" TODO items.
|
||||||
|
|
||||||
Scope: planning and execution guide for TODO items under "Scaling & Performance (planning only)".
|
|
||||||
|
|
||||||
## 1) Manifest TTL Tuning Plan
|
## 1) Manifest TTL Tuning Plan
|
||||||
|
|
||||||
Goal: tune `.m3u8` TTL to about `0.5x` segment duration while keeping playback stable.
|
Goal: tune the `.m3u8` TTL to ~`0.5x` segment duration while keeping playback stable.
|
||||||
|
|
||||||
Inputs:
|
Inputs: segment duration (`HLS_SEGMENT_DURATION`), current manifest TTL (Cloudflare cache rule), cache outcome distribution (`Cf-Cache-Status`).
|
||||||
|
|
||||||
- Segment duration (`HLS_SEGMENT_DURATION`)
|
|
||||||
- Current manifest TTL (Cloudflare cache rule)
|
|
||||||
- Cache outcome distribution (`Cf-Cache-Status`)
|
|
||||||
|
|
||||||
Procedure:
|
Procedure:
|
||||||
|
|
||||||
1. Start from current manifest TTL (2s).
|
1. Start from current manifest TTL (2s).
|
||||||
2. Capture 24h baseline headers from repeated `playlist.m3u8` requests.
|
2. Capture 24h baseline headers from repeated `playlist.m3u8` requests.
|
||||||
3. Test candidate TTLs: 1s, 2s, 3s.
|
3. Test candidate TTLs: 1s, 2s, 3s.
|
||||||
4. Compare:
|
4. Compare rebuffer events (player logs), origin request increase/decrease, and HIT ratio changes.
|
||||||
- rebuffer events from player logs
|
5. Select the lowest TTL that keeps playback stable and origin load acceptable.
|
||||||
- origin request increase/decrease
|
|
||||||
- HIT ratio changes
|
|
||||||
5. Select lowest TTL that keeps playback stable and origin load acceptable.
|
|
||||||
|
|
||||||
Acceptance:
|
Acceptance: no material increase in rebuffer rate; manifest staleness not observable by users.
|
||||||
|
|
||||||
- No material increase in rebuffer rate.
|
|
||||||
- Manifest staleness not observable by users.
|
|
||||||
|
|
||||||
## 2) TTFB Baseline Plan
|
## 2) TTFB Baseline Plan
|
||||||
|
|
||||||
Goal: measure warm-cache TTFB from 3+ US regions with target `< 4s`.
|
Goal: measure warm-cache TTFB from 3+ US regions with target `< 4s`.
|
||||||
|
|
||||||
Suggested regions:
|
Regions: US East (Virginia), US Central, US West (California/Oregon).
|
||||||
|
|
||||||
- US East (Virginia)
|
|
||||||
- US Central
|
|
||||||
- US West (California/Oregon)
|
|
||||||
|
|
||||||
Method:
|
Method:
|
||||||
|
|
||||||
1. Use WebPageTest with repeat view enabled.
|
1. Use WebPageTest with repeat view enabled.
|
||||||
2. Run at least 5 samples per region.
|
2. Run at least 5 samples per region.
|
||||||
3. Record median and p95 TTFB.
|
3. Record median and p95 TTFB.
|
||||||
4. Store results in table (date, region, median, p95, test URL).
|
4. Store results in a table (date, region, median, p95, test URL).
|
||||||
|
|
||||||
Acceptance:
|
Acceptance: warm-cache median < 4s in all regions; outliers documented with likely cause.
|
||||||
|
|
||||||
- Warm-cache median < 4s in all tested regions.
|
|
||||||
- Outliers documented with likely cause.
|
|
||||||
|
|
||||||
### TTFB Baseline Worksheet
|
### TTFB Baseline Worksheet
|
||||||
|
Use one row per region; keep the raw WebPageTest result URL with the summary for comparison.
|
||||||
|
|
||||||
Use one row per region and keep the raw WebPageTest result URL with the summary so later runs can be compared against the same evidence.
|
- US East (Virginia) / US Central / US West (California/Oregon): Date/time, Test URL `https://linearvoid.tv`, WebPageTest result URL, Median TTFB, p95 TTFB, Notes.
|
||||||
|
|
||||||
- US East (Virginia)
|
Capture notes: use repeat view for warm-cache; keep the same browser profile and test URL across regions; note any CDN cache-rule changes or deploys between samples.
|
||||||
- Date/time:
|
|
||||||
- Test URL: <https://linearvoid.tv>
|
|
||||||
- WebPageTest result URL:
|
|
||||||
- Median TTFB:
|
|
||||||
- p95 TTFB:
|
|
||||||
- Notes:
|
|
||||||
- US Central
|
|
||||||
- Date/time:
|
|
||||||
- Test URL: <https://linearvoid.tv>
|
|
||||||
- WebPageTest result URL:
|
|
||||||
- Median TTFB:
|
|
||||||
- p95 TTFB:
|
|
||||||
- Notes:
|
|
||||||
- US West (California/Oregon)
|
|
||||||
- Date/time:
|
|
||||||
- Test URL: <https://linearvoid.tv>
|
|
||||||
- WebPageTest result URL:
|
|
||||||
- Median TTFB:
|
|
||||||
- p95 TTFB:
|
|
||||||
- Notes:
|
|
||||||
|
|
||||||
Suggested capture notes:
|
|
||||||
|
|
||||||
- Use repeat view for the warm-cache measurement.
|
|
||||||
- Keep the same browser profile and test URL across regions.
|
|
||||||
- Note any CDN cache-rule changes or deploys that occurred between samples.
|
|
||||||
|
|
||||||
## 3) WebSocket Load-Test Plan
|
## 3) WebSocket Load-Test Plan
|
||||||
|
|
||||||
Goal: characterize chat service behavior across 10 -> 100 -> 500 -> 1000 concurrent clients.
|
Goal: characterize chat behavior across 10 → 100 → 500 → 1000 concurrent clients.
|
||||||
|
|
||||||
Tooling option A (Artillery):
|
Tooling option A (Artillery):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npx artillery run scripts/loadtest/ws-chat.yml
|
npx artillery run scripts/loadtest/ws-chat.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
Tooling option B (custom Go harness):
|
Tooling option B (custom Go harness): open N concurrent WS clients, send a message every X seconds, measure round-trip latency and failures.
|
||||||
|
|
||||||
- open N concurrent WebSocket clients
|
Metrics: p50/p95 message latency; error rate (connect/send/receive); dropped connections; CPU/memory for `chat`, `redis`, `nginx`.
|
||||||
- send message every X seconds
|
|
||||||
- measure round-trip latency and failures
|
|
||||||
|
|
||||||
Metrics to capture:
|
Acceptance: p95 chat latency ≤ 1s at target concurrency tier; error rate ≤ 1% during steady state.
|
||||||
|
|
||||||
- p50/p95 message latency
|
|
||||||
- error rate (connect, send, receive)
|
|
||||||
- dropped connections
|
|
||||||
- CPU/memory for `chat`, `redis`, `nginx`
|
|
||||||
|
|
||||||
Acceptance:
|
|
||||||
|
|
||||||
- p95 chat latency <= 1s at target concurrency tier.
|
|
||||||
- error rate <= 1% during steady state.
|
|
||||||
|
|
||||||
### WebSocket Load-Test Worksheet
|
### WebSocket Load-Test Worksheet
|
||||||
|
Date/time, Environment, Commit SHA/image tag, Test URL, Tool used, Scenario duration, Concurrency tiers reached, p50 message latency, p95 message latency, Error rate, Dropped connections, CPU/memory notes for chat/redis/nginx, Raw results location, Conclusion, Next action.
|
||||||
- Date/time:
|
|
||||||
- Environment:
|
|
||||||
- Commit SHA / image tag:
|
|
||||||
- Test URL:
|
|
||||||
- Tool used:
|
|
||||||
- Scenario duration:
|
|
||||||
- Concurrency tiers reached:
|
|
||||||
- p50 message latency:
|
|
||||||
- p95 message latency:
|
|
||||||
- Error rate:
|
|
||||||
- Dropped connections:
|
|
||||||
- CPU/memory notes for chat:
|
|
||||||
- CPU/memory notes for redis:
|
|
||||||
- CPU/memory notes for nginx:
|
|
||||||
- Raw results location:
|
|
||||||
- Conclusion:
|
|
||||||
- Next action:
|
|
||||||
|
|
||||||
## 4) Admin API Benchmark Plan
|
## 4) Admin API Benchmark Plan
|
||||||
|
|
||||||
Goal: measure admin API p95 and identify slow endpoints.
|
Goal: measure admin API p95 and identify slow endpoints.
|
||||||
|
|
||||||
Tooling option A (`k6`):
|
Endpoints: `/api/admin/overview`, `/api/admin/analytics`, `/api/admin/chat-log`, `/api/admin/users`.
|
||||||
|
|
||||||
|
Tooling option A (`k6`):
|
||||||
```bash
|
```bash
|
||||||
k6 run scripts/loadtest/admin-api.js
|
k6 run scripts/loadtest/admin-api.js
|
||||||
```
|
```
|
||||||
|
|
||||||
Tooling option B (`wrk`):
|
Tooling option B (`wrk`):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
wrk -t4 -c100 -d60s http://<host>/api/admin/overview
|
wrk -t4 -c100 -d60s http://<host>/api/admin/overview
|
||||||
```
|
```
|
||||||
|
|
||||||
Endpoints:
|
Acceptance: p95 < 1s for primary admin endpoints; slow endpoint list with remediation notes.
|
||||||
|
|
||||||
- `/api/admin/overview`
|
|
||||||
- `/api/admin/analytics`
|
|
||||||
- `/api/admin/chat-log`
|
|
||||||
- `/api/admin/users`
|
|
||||||
|
|
||||||
Acceptance:
|
|
||||||
|
|
||||||
- p95 < 1s for primary admin endpoints.
|
|
||||||
- slow endpoint list with remediation notes.
|
|
||||||
|
|
||||||
### Admin API Benchmark Worksheet
|
### Admin API Benchmark Worksheet
|
||||||
|
Date/time, Environment, Commit SHA/image tag, Test URL, Tool used, Scenario duration, Concurrency/virtual users, `/api/admin/overview|analytics|chat-log|users` p95, Error rate, Slowest endpoint, Cache or DB notes, Raw results location, Conclusion, Next action.
|
||||||
- Date/time:
|
|
||||||
- Environment:
|
|
||||||
- Commit SHA / image tag:
|
|
||||||
- Test URL:
|
|
||||||
- Tool used:
|
|
||||||
- Scenario duration:
|
|
||||||
- Concurrency / virtual users:
|
|
||||||
- `/api/admin/overview` p95:
|
|
||||||
- `/api/admin/analytics` p95:
|
|
||||||
- `/api/admin/chat-log` p95:
|
|
||||||
- `/api/admin/users` p95:
|
|
||||||
- Error rate:
|
|
||||||
- Slowest endpoint:
|
|
||||||
- Cache or DB notes:
|
|
||||||
- Raw results location:
|
|
||||||
- Conclusion:
|
|
||||||
- Next action:
|
|
||||||
|
|
||||||
## 5) Cache Hit Ratio Tracking Plan
|
## 5) Cache Hit Ratio Tracking Plan
|
||||||
|
|
||||||
Goal: track Cloudflare cache hit ratio and alert when below threshold.
|
Goal: track Cloudflare cache hit ratio and alert when below threshold.
|
||||||
|
|
||||||
Implementation outline:
|
Implementation:
|
||||||
|
|
||||||
1. Pull Cloudflare GraphQL analytics on schedule.
|
1. Pull Cloudflare GraphQL analytics on schedule.
|
||||||
2. Export to Prometheus-compatible metric path.
|
2. Export to a Prometheus-compatible metric path.
|
||||||
3. Add Grafana panel with 5m/1h/24h windows.
|
3. Add a Grafana panel with 5m/1h/24h windows.
|
||||||
4. Alert when hit ratio < 90% for 15m.
|
4. Alert when hit ratio < 90% for 15m.
|
||||||
|
|
||||||
Reference exporter:
|
Reference exporter:
|
||||||
|
|
||||||
- `scripts/cache-hit-ratio.js`
|
- `scripts/cache-hit-ratio.js`
|
||||||
- `scripts/cache-hit-ratio.ps1` for Windows scheduled tasks / manual runs
|
- `scripts/cache-hit-ratio.sh` for scheduled tasks / manual runs
|
||||||
- Example run: `CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_TAG=... CLOUDFLARE_HOSTNAME=media.linearvoid.tv node scripts/cache-hit-ratio.js`
|
- Example run: `export CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_TAG=... CLOUDFLARE_HOSTNAME=media.linearvoid.tv; node scripts/cache-hit-ratio.js`
|
||||||
- Windows example: `pwsh -File scripts/cache-hit-ratio.ps1`
|
|
||||||
- Intended output: Prometheus text format on stdout, suitable for cron, a Node wrapper, or a small scrape endpoint.
|
- Intended output: Prometheus text format on stdout, suitable for cron, a Node wrapper, or a small scrape endpoint.
|
||||||
|
|
||||||
Metric target:
|
Metric target: `cache_hit_ratio >= 0.90`.
|
||||||
|
|
||||||
- `cache_hit_ratio >= 0.90`
|
|
||||||
|
|
||||||
### Cache Hit Ratio Worksheet
|
### Cache Hit Ratio Worksheet
|
||||||
|
Date/time, Environment, Commit SHA/image tag, Hostname, Window minutes, Cloudflare zone tag, Export command, Raw exporter output location, Hit requests, Total requests, Cache hit ratio, Threshold met, Notes.
|
||||||
- Date/time:
|
|
||||||
- Environment:
|
|
||||||
- Commit SHA / image tag:
|
|
||||||
- Hostname:
|
|
||||||
- Window minutes:
|
|
||||||
- Cloudflare zone tag:
|
|
||||||
- Export command:
|
|
||||||
- Raw exporter output location:
|
|
||||||
- Hit requests:
|
|
||||||
- Total requests:
|
|
||||||
- Cache hit ratio:
|
|
||||||
- Threshold met:
|
|
||||||
- Notes:
|
|
||||||
|
|
||||||
## 6) Capacity Trigger Rules
|
## 6) Capacity Trigger Rules
|
||||||
|
|
||||||
Define operational triggers:
|
Operational triggers:
|
||||||
|
|
||||||
- Cache miss ratio > 10% for 15m
|
- Cache miss ratio > 10% for 15m
|
||||||
- Chat p95 latency > 1s for 5m
|
- Chat p95 latency > 1s for 5m
|
||||||
- Host CPU > 80% for 10m
|
- Host CPU > 80% for 10m
|
||||||
- Redis memory usage ratio (`linearvoid_chat_redis_memory_usage_ratio`) > 0.85 for 10m
|
- Redis memory usage ratio (`linearvoid_chat_redis_memory_usage_ratio`) > 0.85 for 10m
|
||||||
|
|
||||||
Action mapping:
|
Action mapping:
|
||||||
|
|
||||||
- Level 1: investigate + restart non-critical components
|
- Level 1: investigate + restart non-critical components
|
||||||
- Level 2: scale service resources (CPU/RAM)
|
- Level 2: scale service resources (CPU/RAM)
|
||||||
- Level 3: activate standby planning path
|
- Level 3: activate standby planning path
|
||||||
|
|
||||||
### Capacity Trigger Review
|
### Capacity Trigger Review
|
||||||
|
Date/time, Environment, Commit SHA/image tag, Trigger hit, Threshold value, Observed value, Duration above threshold, Level selected, Immediate action taken, Services involved, Follow-up owner, Raw metrics/alert link, Notes.
|
||||||
- Date/time:
|
|
||||||
- Environment:
|
|
||||||
- Commit SHA / image tag:
|
|
||||||
- Trigger hit:
|
|
||||||
- Threshold value:
|
|
||||||
- Observed value:
|
|
||||||
- Duration above threshold:
|
|
||||||
- Level selected:
|
|
||||||
- Immediate action taken:
|
|
||||||
- Services involved:
|
|
||||||
- Follow-up owner:
|
|
||||||
- Raw metrics / alert link:
|
|
||||||
- Notes:
|
|
||||||
|
|
||||||
## 7) Phase 2 Service Separation Plan
|
## 7) Phase 2 Service Separation Plan
|
||||||
|
|
||||||
Objective: separate bottlenecked services while keeping simple ops.
|
Objective: separate bottlenecked services while keeping ops simple.
|
||||||
|
|
||||||
Boundaries:
|
Boundaries:
|
||||||
|
|
||||||
- `playout` isolated first (CPU-heavy)
|
- `playout` isolated first (CPU-heavy)
|
||||||
- `chat` + `redis` grouped for latency
|
- `chat` + `redis` grouped for latency
|
||||||
- `db` isolated for IO stability
|
- `db` isolated for IO stability
|
||||||
- monitoring stack optionally separated after sustained load
|
- monitoring stack optionally separated after sustained load
|
||||||
|
|
||||||
Initial sizing guidance:
|
Initial sizing:
|
||||||
|
|
||||||
- playout host: 4 vCPU / 8 GB
|
- playout host: 4 vCPU / 8 GB
|
||||||
- chat host: 2-4 vCPU / 4-8 GB
|
- chat host: 2–4 vCPU / 4–8 GB
|
||||||
- db host: 2 vCPU / 4 GB + fast disk
|
- db host: 2 vCPU / 4 GB + fast disk
|
||||||
|
|
||||||
Exit criteria for Phase 2:
|
Exit criteria: repeated trigger events over 7 days; capacity saturation despite tuning.
|
||||||
|
|
||||||
- repeated trigger events over 7 days
|
|
||||||
- capacity saturation despite tuning
|
|
||||||
|
|
||||||
### Phase 2 Boundary Worksheet
|
### Phase 2 Boundary Worksheet
|
||||||
|
Date/time, Environment, Commit SHA/image tag, Trigger history summary, Repeated trigger count over 7 days, Primary bottleneck, Proposed split, `playout`/`chat`/`db` target host size, Monitoring separation needed, Data/state migration notes, Implementation owner, Review decision, Next action.
|
||||||
- Date/time:
|
|
||||||
- Environment:
|
|
||||||
- Commit SHA / image tag:
|
|
||||||
- Trigger history summary:
|
|
||||||
- Repeated trigger count over 7 days:
|
|
||||||
- Primary bottleneck:
|
|
||||||
- Proposed split:
|
|
||||||
- `playout` target host size:
|
|
||||||
- `chat` target host size:
|
|
||||||
- `db` target host size:
|
|
||||||
- Monitoring separation needed:
|
|
||||||
- Data/state migration notes:
|
|
||||||
- Implementation owner:
|
|
||||||
- Review decision:
|
|
||||||
- Next action:
|
|
||||||
|
|
||||||
## 8) Phase 3 Warm Standby Plan
|
## 8) Phase 3 Warm Standby Plan
|
||||||
|
|
||||||
Objective: second-region resilience with DNS failover.
|
Objective: second-region resilience with DNS failover.
|
||||||
|
|
||||||
Candidate layout:
|
Layout:
|
||||||
|
|
||||||
- Primary: Hetzner Ashburn (US-East)
|
- Primary: Hetzner Ashburn (US-East)
|
||||||
- Standby: Hetzner Hillsboro (US-West)
|
- Standby: Hetzner Hillsboro (US-West)
|
||||||
|
|
||||||
Design:
|
Design: warm standby stack deployed and health-checked; Redis replication or fallback mode documented; Postgres replication strategy selected (streaming replica or periodic snapshot restore); Cloudflare DNS failover policy defined (health-based or manual emergency switch).
|
||||||
|
|
||||||
- Warm standby stack deployed and health-checked
|
|
||||||
- Redis replication or fallback mode documented
|
|
||||||
- Postgres replication strategy selected (streaming replica or periodic snapshot restore)
|
|
||||||
- Cloudflare DNS failover policy defined (health-based or manual emergency switch)
|
|
||||||
|
|
||||||
Failover drill requirements:
|
|
||||||
|
|
||||||
### Phase 3 Warm Standby Worksheet
|
### Phase 3 Warm Standby Worksheet
|
||||||
|
Date/time, Environment, Commit SHA/image tag, Primary region, Standby region, DNS failover mode, Redis strategy, Postgres strategy, Health check endpoints, Simulated cutover performed, Measured RTO, Measured RPO, Open risks, Implementation owner, Review decision, Notes.
|
||||||
- Date/time:
|
|
||||||
- Environment:
|
|
||||||
- Commit SHA / image tag:
|
|
||||||
- Primary region:
|
|
||||||
- Standby region:
|
|
||||||
- DNS failover mode:
|
|
||||||
- Redis strategy:
|
|
||||||
- Postgres strategy:
|
|
||||||
- Health check endpoints:
|
|
||||||
- Simulated cutover performed:
|
|
||||||
- Measured RTO:
|
|
||||||
- Measured RPO:
|
|
||||||
- Open risks:
|
|
||||||
- Implementation owner:
|
|
||||||
- Review decision:
|
|
||||||
- Notes:
|
|
||||||
|
|
||||||
## 9) Data Capture Template
|
## 9) Data Capture Template
|
||||||
|
For each benchmark run capture: Date/time, Environment (test-lab/staging/production), Commit SHA/image tag, Test scenario and duration, Raw metrics (latency, errors, resource usage), Conclusion and next action.
|
||||||
For each benchmark run, capture:
|
|
||||||
|
|
||||||
- Date/time
|
|
||||||
- Environment (test-lab/staging/production)
|
|
||||||
- Commit SHA / image tag
|
|
||||||
- Test scenario and duration
|
|
||||||
- Raw metrics (latency, errors, resource usage)
|
|
||||||
- Conclusion and next action
|
|
||||||
|
|
||||||
## 10) Execution Order
|
## 10) Execution Order
|
||||||
|
|
||||||
1. TTFB baseline
|
1. TTFB baseline
|
||||||
2. WebSocket load profile
|
2. WebSocket load profile
|
||||||
3. Admin API benchmark
|
3. Admin API benchmark
|
||||||
@@ -349,14 +160,9 @@ For each benchmark run, capture:
|
|||||||
7. Phase 3 standby design review
|
7. Phase 3 standby design review
|
||||||
|
|
||||||
## 11) Benchmark Run Sheet
|
## 11) Benchmark Run Sheet
|
||||||
|
Use the same worksheet for every baseline/re-test so results stay comparable.
|
||||||
|
|
||||||
Use the same worksheet for every baseline or re-test so results stay comparable.
|
Preconditions: confirm target environment and commit SHA/image tag; confirm Cloudflare cache rules are active when testing TTFB or manifest TTL; confirm chat, admin, and playout services are healthy before load testing.
|
||||||
|
|
||||||
Preconditions:
|
|
||||||
|
|
||||||
- Confirm the target environment and commit SHA/image tag.
|
|
||||||
- Confirm Cloudflare cache rules are active when testing TTFB or manifest TTL.
|
|
||||||
- Confirm the chat, admin, and playout services are healthy before load testing.
|
|
||||||
|
|
||||||
Run template:
|
Run template:
|
||||||
|
|
||||||
@@ -373,70 +179,47 @@ Run template:
|
|||||||
| Next action | |
|
| Next action | |
|
||||||
|
|
||||||
Suggested command mapping:
|
Suggested command mapping:
|
||||||
|
|
||||||
- TTFB baseline: WebPageTest repeat-view run against `https://linearvoid.tv`.
|
- TTFB baseline: WebPageTest repeat-view run against `https://linearvoid.tv`.
|
||||||
- WebSocket profile: `npx artillery run scripts/loadtest/ws-chat.yml` or the Go harness referenced above.
|
- WebSocket profile: `npx artillery run scripts/loadtest/ws-chat.yml` or the Go harness referenced above.
|
||||||
- Admin API benchmark: `k6 run scripts/loadtest/admin-api.js` or `wrk -t4 -c100 -d60s http://<host>/api/admin/overview`.
|
- Admin API benchmark: `k6 run scripts/loadtest/admin-api.js` or `wrk -t4 -c100 -d60s http://<host>/api/admin/overview`.
|
||||||
|
|
||||||
## 12) Scaling Decision Gate
|
## 12) Scaling Decision Gate
|
||||||
|
|
||||||
Treat the following as the default review gate after any benchmark batch:
|
Default review gate after any benchmark batch:
|
||||||
|
|
||||||
1. If warm-cache TTFB meets target but cache hit ratio falls below 90%, tune cache behavior before increasing capacity.
|
1. If warm-cache TTFB meets target but cache hit ratio falls below 90%, tune cache behavior before increasing capacity.
|
||||||
2. If chat or admin latency exceeds target while CPU and memory remain low, prioritize query or handler optimization.
|
2. If chat or admin latency exceeds target while CPU and memory remain low, prioritize query or handler optimization.
|
||||||
3. If CPU, memory, or Redis saturation appears first, treat the service boundary guidance in Phase 2 as the next action.
|
3. If CPU, memory, or Redis saturation appears first, treat the Phase 2 service boundary guidance as the next action.
|
||||||
4. If the same threshold trips on multiple runs across 7 days, open the Phase 2 or Phase 3 planning path instead of tuning in place.
|
4. If the same threshold trips on multiple runs across 7 days, open the Phase 2 or Phase 3 planning path instead of tuning in place.
|
||||||
|
|
||||||
Decision note:
|
Decision note: keep the smallest change that restores the target metric and record the reason in the run sheet.
|
||||||
|
|
||||||
- Keep the smallest change that restores the target metric and record the reason in the run sheet.
|
|
||||||
|
|
||||||
## 13) Artifact-to-Worksheet Field Mapping
|
## 13) Artifact-to-Worksheet Field Mapping
|
||||||
|
Use when filling the WebSocket and Admin API worksheets from helper-generated artifacts.
|
||||||
Use this section when filling the WebSocket and Admin API worksheets from helper-generated artifacts.
|
|
||||||
|
|
||||||
### WebSocket worksheet mapping
|
### WebSocket worksheet mapping
|
||||||
|
Artifact sources: `artifacts/<timestamp>/ws-chat-report.json`, `artifacts/<timestamp>/ws-chat-summary.txt`.
|
||||||
Artifact sources:
|
|
||||||
|
|
||||||
- `artifacts/<timestamp>/ws-chat-report.json`
|
|
||||||
- `artifacts/<timestamp>/ws-chat-summary.txt`
|
|
||||||
|
|
||||||
Field mapping:
|
Field mapping:
|
||||||
|
- `Tool used` → `artillery` (or `scripts/loadtest/run-loadtests.sh -s ws`)
|
||||||
|
- `Raw results location` → artifact folder path for this run
|
||||||
|
- `Concurrency tiers reached` → `ws-chat.yml` phases (`warmup-10`, `sustain-100`, `sustain-500`, `sustain-1000`)
|
||||||
|
- `Error rate` → `vusers.failure_rate_percent` from `ws-chat-summary.txt`
|
||||||
|
- `Dropped connections` → derive from `vusers.created - vusers.completed` (use `vusers.failed` as failure signal)
|
||||||
|
- `p95 message latency` → first latency line under `summary.*` in `ws-chat-summary.txt` containing `p95=...`
|
||||||
|
- `p50 message latency` → first latency line under `summary.*` in `ws-chat-summary.txt` containing `median=...`
|
||||||
|
|
||||||
- Worksheet `Tool used` -> `artillery` (or `run-loadtests.ps1 -Scenario ws`)
|
Notes: the exact `summary.*` metric names in Artillery output can vary by engine/plugins; if no latency summary field is emitted, record this explicitly and use server-side latency metrics as fallback evidence.
|
||||||
- Worksheet `Raw results location` -> artifact folder path for this run
|
|
||||||
- Worksheet `Concurrency tiers reached` -> `ws-chat.yml` phases (`warmup-10`, `sustain-100`, `sustain-500`, `sustain-1000`)
|
|
||||||
- Worksheet `Error rate` -> `vusers.failure_rate_percent` from `ws-chat-summary.txt`
|
|
||||||
- Worksheet `Dropped connections` -> derive from `vusers.created - vusers.completed` (use `vusers.failed` as failure signal)
|
|
||||||
- Worksheet `p95 message latency` -> first available latency line under `summary.*` in `ws-chat-summary.txt` that includes `p95=...`
|
|
||||||
- Worksheet `p50 message latency` -> first available latency line under `summary.*` in `ws-chat-summary.txt` that includes `median=...`
|
|
||||||
|
|
||||||
Notes:
|
|
||||||
|
|
||||||
- The exact `summary.*` metric names in Artillery output can vary by engine/plugins.
|
|
||||||
- If no latency summary field is emitted, record this explicitly and use server-side latency metrics as fallback evidence.
|
|
||||||
|
|
||||||
### Admin API worksheet mapping
|
### Admin API worksheet mapping
|
||||||
|
Artifact sources: `artifacts/<timestamp>/admin-api-summary.json`, `artifacts/<timestamp>/admin-api-output.txt`, `artifacts/<timestamp>/admin-api-endpoint-summary.txt`.
|
||||||
Artifact sources:
|
|
||||||
|
|
||||||
- `artifacts/<timestamp>/admin-api-summary.json`
|
|
||||||
- `artifacts/<timestamp>/admin-api-output.txt`
|
|
||||||
- `artifacts/<timestamp>/admin-api-endpoint-summary.txt`
|
|
||||||
|
|
||||||
Field mapping:
|
Field mapping:
|
||||||
|
- `Tool used` → `k6` (or `scripts/loadtest/run-loadtests.sh -s admin`)
|
||||||
|
- `Raw results location` → artifact folder path for this run
|
||||||
|
- `/api/admin/overview|analytics|chat-log|users` p95 → `overview|analytics|chat_log|users` row `P95Ms` in `admin-api-endpoint-summary.txt`
|
||||||
|
- `Error rate` → `Global failure rate` in `admin-api-endpoint-summary.txt`
|
||||||
|
- `Slowest endpoint` → endpoint row with highest `P95Ms`
|
||||||
|
|
||||||
- Worksheet `Tool used` -> `k6` (or `run-loadtests.ps1 -Scenario admin`)
|
Notes: keep the raw `admin-api-summary.json` file for audits and re-parsing; if an endpoint row is missing, treat it as a test/data issue and re-run with full endpoint coverage.
|
||||||
- Worksheet `Raw results location` -> artifact folder path for this run
|
|
||||||
- Worksheet `/api/admin/overview` p95 -> `overview` row `P95Ms` in `admin-api-endpoint-summary.txt`
|
|
||||||
- Worksheet `/api/admin/analytics` p95 -> `analytics` row `P95Ms` in `admin-api-endpoint-summary.txt`
|
|
||||||
- Worksheet `/api/admin/chat-log` p95 -> `chat_log` row `P95Ms` in `admin-api-endpoint-summary.txt`
|
|
||||||
- Worksheet `/api/admin/users` p95 -> `users` row `P95Ms` in `admin-api-endpoint-summary.txt`
|
|
||||||
- Worksheet `Error rate` -> `Global failure rate` in `admin-api-endpoint-summary.txt`
|
|
||||||
- Worksheet `Slowest endpoint` -> endpoint row with highest `P95Ms`
|
|
||||||
|
|
||||||
Notes:
|
Summary helper (bash): run `scripts/loadtest/summarize-artillery.sh <report.json>` after an Artillery WS run, and `scripts/loadtest/summarize-k6.sh <summary.json>` after a k6 admin run, to produce the artifact files used in the mappings above.
|
||||||
|
|
||||||
- Keep the raw `admin-api-summary.json` file for audits and re-parsing.
|
|
||||||
- If an endpoint row is missing, treat as test/data issue and re-run with full endpoint coverage.
|
|
||||||
Generated
+1148
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,5 @@
|
|||||||
|
{
|
||||||
|
"dependencies": {
|
||||||
|
"vitest": "^4.1.11"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# cache-hit-ratio.sh — Thin wrapper that runs the Node.js cache-hit-ratio tool.
|
||||||
|
# Mirrors scripts/cache-hit-ratio.ps1.
|
||||||
|
set -euo pipefail
|
||||||
|
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
exec node "$script_dir/cache-hit-ratio.js" "$@"
|
||||||
@@ -0,0 +1,132 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# check-env.sh — Verify the deployment environment for Linear Void.
|
||||||
|
# Mirrors scripts/check-env.ps1.
|
||||||
|
#
|
||||||
|
# Checks, against the target Docker host (default test lab 192.168.88.40):
|
||||||
|
# 1. Docker Engine reachability over the TCP socket (port 2375).
|
||||||
|
# 2. Coolify helper stack (proxy/Traefik, sentinel, cloudflared).
|
||||||
|
# 3. Linear Void containers (playout, chat, nginx, redis, db, prometheus, grafana).
|
||||||
|
# 4. Public endpoints (app, HLS manifest, admin).
|
||||||
|
# 5. Health endpoints exposed by the services.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./scripts/check-env.sh
|
||||||
|
# ./scripts/check-env.sh -t 192.168.88.40
|
||||||
|
# ./scripts/check-env.sh -t 192.168.88.40 -p 2375 --only-containers
|
||||||
|
#
|
||||||
|
# Exit code is 0 when all required checks pass, 1 otherwise.
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
TARGET=192.168.88.40
|
||||||
|
DOCKER_PORT=2375
|
||||||
|
ONLY_CONTAINERS=0
|
||||||
|
ONLY_ENDPOINTS=0
|
||||||
|
TIMEOUT=10
|
||||||
|
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
-t|--target) TARGET="$2"; shift 2;;
|
||||||
|
-p|--docker-port) DOCKER_PORT="$2"; shift 2;;
|
||||||
|
--only-containers) ONLY_CONTAINERS=1; shift;;
|
||||||
|
--only-endpoints) ONLY_ENDPOINTS=1; shift;;
|
||||||
|
--timeout) TIMEOUT="$2"; shift 2;;
|
||||||
|
*) echo "Unknown option: $1" >&2; exit 2;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
export DOCKER_HOST="tcp://${TARGET}:${DOCKER_PORT}"
|
||||||
|
|
||||||
|
FAILED=0
|
||||||
|
fail() { printf ' [FAIL] %s\n' "$1"; FAILED=1; }
|
||||||
|
ok() { printf ' [OK] %s\n' "$1"; }
|
||||||
|
warn() { printf ' [WARN] %s\n' "$1"; }
|
||||||
|
|
||||||
|
echo "== Linear Void environment check =="
|
||||||
|
echo "Target Docker host : $DOCKER_HOST"
|
||||||
|
echo "Date : $(date -Iseconds)"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1. Docker Engine reachability
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
if [[ "$ONLY_ENDPOINTS" -eq 0 ]]; then
|
||||||
|
echo ""
|
||||||
|
echo "-- Docker Engine (${TARGET}:${DOCKER_PORT}) --"
|
||||||
|
if ver=$(docker version --format '{{.Server.Version}}' 2>&1) && [[ "$ver" =~ [0-9] ]]; then
|
||||||
|
ok "Docker Engine reachable (server version $ver)"
|
||||||
|
else
|
||||||
|
fail "Docker Engine not reachable: $ver"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
# 2. Coolify helper stack
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "-- Coolify managed-server helpers --"
|
||||||
|
for c in coolify-proxy coolify-sentinel coolify-cloudflared; do
|
||||||
|
if state=$(docker inspect "$c" --format '{{.State.Status}} ({{.State.Health.Status}})' 2>/dev/null); then
|
||||||
|
if [[ "$state" =~ running.*(healthy|running) ]]; then ok "coolify helper: $c -> $state"; else warn "coolify helper: $c -> $state"; fi
|
||||||
|
else
|
||||||
|
warn "coolify helper: $c -> NOT FOUND (is this a Coolify-managed server?)"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
# 3. Linear Void containers
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "-- Linear Void containers --"
|
||||||
|
required=(linearvoid-playout linearvoid-chat linearvoid-nginx linearvoid-redis linearvoid-db linearvoid-prometheus linearvoid-grafana)
|
||||||
|
running="$(docker ps --format '{{.Names}}' 2>/dev/null)"
|
||||||
|
all="$(docker ps -a --format '{{.Names}}\t{{.Status}}' 2>/dev/null)"
|
||||||
|
for name in "${required[@]}"; do
|
||||||
|
if grep -qx "$name" <<<"$running"; then
|
||||||
|
ok "container: $name is running"
|
||||||
|
elif grep -q "^$name[[:space:]]" <<<"$all"; then
|
||||||
|
status="$(grep "^$name[[:space:]]" <<<"$all" | head -1)"
|
||||||
|
fail "container: $name EXISTS but not running -> $status"
|
||||||
|
else
|
||||||
|
fail "container: $name MISSING"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 4. Health endpoints (via Docker host)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
if [[ "$ONLY_CONTAINERS" -eq 0 ]]; then
|
||||||
|
echo ""
|
||||||
|
echo "-- Health endpoints --"
|
||||||
|
checks=(
|
||||||
|
"chat /health|http://${TARGET}:3000/health"
|
||||||
|
"nginx /health|http://${TARGET}:8081/health"
|
||||||
|
"prometheus|http://${TARGET}:9090/-/healthy"
|
||||||
|
"grafana /api/health|http://${TARGET}:3001/api/health"
|
||||||
|
)
|
||||||
|
for entry in "${checks[@]}"; do
|
||||||
|
name="${entry%%|*}"; url="${entry#*|}"
|
||||||
|
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time "$TIMEOUT" "$url" 2>/dev/null)
|
||||||
|
if [[ "$code" == "200" ]]; then ok "$name -> 200"; elif [[ "$code" =~ ^[0-9]{3}$ ]]; then warn "$name -> $code"; else fail "$name -> unreachable"; fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
# 5. Public endpoints (routed through the Docker host for reliable TLS)
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "-- Public endpoints (via Docker host) --"
|
||||||
|
pub=(
|
||||||
|
"linearvoid.tv app|https://linearvoid.tv/"
|
||||||
|
"media HLS manifest|https://media.linearvoid.tv/hls/playlist.m3u8"
|
||||||
|
"coolify dashboard|https://coolify.allucanget.biz/"
|
||||||
|
)
|
||||||
|
for entry in "${pub[@]}"; do
|
||||||
|
name="${entry%%|*}"; url="${entry#*|}"
|
||||||
|
code=$(docker run --rm curlimages/curl:latest -sS -o /dev/null -w '%{http_code}' --max-time "$TIMEOUT" -L "$url" 2>/dev/null | tr -d '[:space:]')
|
||||||
|
if [[ "$code" =~ ^[0-9]{3}$ ]]; then ok "$name -> HTTP $code"; else warn "$name -> $code"; fi
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
if [[ "$FAILED" -ne 0 ]]; then echo "RESULT: FAILED"; exit 1; fi
|
||||||
|
echo "RESULT: OK"
|
||||||
|
exit 0
|
||||||
@@ -0,0 +1,177 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# check-tokens.sh — Verify that the credentials in the environment actually work.
|
||||||
|
# Mirrors scripts/check-tokens.ps1.
|
||||||
|
#
|
||||||
|
# Reads .env (or a supplied file) and validates access against live services:
|
||||||
|
# 1. Gitea container registry (CI_REGISTRY_TOKEN / CI_REGISTRY_USER).
|
||||||
|
# 2. Cloudflare API token(s) (CLOUDFLARE_API_TOKEN, and any R2 admin token).
|
||||||
|
# 3. R2/S3 bucket access (R2_ACCESS_KEY_ID + R2_SECRET_ACCESS_KEY) — lists buckets.
|
||||||
|
# 4. Linear Void admin login (ADMIN_USERNAME / ADMIN_PASSWORD) against the app origin.
|
||||||
|
#
|
||||||
|
# NOTE: Runs the checks through the target Docker host (default 192.168.88.40) so
|
||||||
|
# network + TLS work regardless of this machine's proxy/firewall. Uses the
|
||||||
|
# curlimages/curl and amazon/aws-cli images, which are pulled on demand.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./scripts/check-tokens.sh
|
||||||
|
# ./scripts/check-tokens.sh -e .env
|
||||||
|
# ./scripts/check-tokens.sh -t 192.168.88.40
|
||||||
|
#
|
||||||
|
# Exit code 0 = all credentials valid; 1 = at least one check failed.
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
ENV_FILE=""
|
||||||
|
TARGET=192.168.88.40
|
||||||
|
DOCKER_PORT=2375
|
||||||
|
TIMEOUT=20
|
||||||
|
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
-e|--env-file) ENV_FILE="$2"; shift 2;;
|
||||||
|
-t|--target) TARGET="$2"; shift 2;;
|
||||||
|
-p|--docker-port) DOCKER_PORT="$2"; shift 2;;
|
||||||
|
--timeout) TIMEOUT="$2"; shift 2;;
|
||||||
|
*) echo "Unknown option: $1" >&2; exit 2;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
export DOCKER_HOST="tcp://${TARGET}:${DOCKER_PORT}"
|
||||||
|
|
||||||
|
FAILED=0
|
||||||
|
fail() { printf ' [FAIL] %s\n' "$1"; FAILED=1; }
|
||||||
|
ok() { printf ' [OK] %s\n' "$1"; }
|
||||||
|
warn() { printf ' [WARN] %s\n' "$1"; }
|
||||||
|
|
||||||
|
# Resolve env file: explicit, or $PWD/.env, or repo root.
|
||||||
|
if [[ -z "$ENV_FILE" ]]; then
|
||||||
|
for cand in "$PWD/.env" "$(dirname "$PWD")/.env"; do
|
||||||
|
if [[ -f "$cand" ]]; then ENV_FILE="$cand"; break; fi
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
if [[ -z "$ENV_FILE" || ! -f "$ENV_FILE" ]]; then
|
||||||
|
echo "No .env file found. Pass one with -e." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Parse KEY=VALUE lines into an associative array.
|
||||||
|
declare -A envs
|
||||||
|
while IFS= read -r line || [[ -n "$line" ]]; do
|
||||||
|
if [[ "$line" =~ ^[[:space:]]*([A-Za-z0-9_]+)[[:space:]]*=[[:space:]]*(.*)[[:space:]]*$ ]]; then
|
||||||
|
envs["${BASH_REMATCH[1]}"]="${BASH_REMATCH[2]}"
|
||||||
|
fi
|
||||||
|
done < "$ENV_FILE"
|
||||||
|
|
||||||
|
get_val() { [[ -n "${envs[$1]+x}" ]] && echo "${envs[$1]}"; }
|
||||||
|
|
||||||
|
# Pull helper images once.
|
||||||
|
echo "Preparing helper images (curl, aws-cli) on $TARGET..."
|
||||||
|
docker pull curlimages/curl:latest >/dev/null 2>&1 || true
|
||||||
|
docker pull amazon/aws-cli:latest >/dev/null 2>&1 || true
|
||||||
|
|
||||||
|
# dc_curl URL [bearer] [token] — run curl via the curlimages/curl container.
|
||||||
|
dc_curl() {
|
||||||
|
local url="$1" bearer="${2:-}" token="${3:-}"
|
||||||
|
local -a args=(run --rm curlimages/curl:latest -sS --max-time "$TIMEOUT")
|
||||||
|
[[ -n "$bearer" ]] && args+=( -H "Authorization: Bearer $bearer" )
|
||||||
|
[[ -n "$token" ]] && args+=( -H "Authorization: token $token" )
|
||||||
|
args+=( "$url" )
|
||||||
|
docker "${args[@]}" 2>&1 | tr -d '\r' | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo "== Linear Void credential / token check =="
|
||||||
|
echo "Env file : $ENV_FILE"
|
||||||
|
echo "Docker host: $DOCKER_HOST"
|
||||||
|
|
||||||
|
# --- 1. Gitea registry token ---
|
||||||
|
echo ""
|
||||||
|
echo "-- Gitea container registry --"
|
||||||
|
reg_user="$(get_val CI_REGISTRY_USER)"
|
||||||
|
reg_token="$(get_val CI_REGISTRY_TOKEN)"
|
||||||
|
if [[ -n "$reg_user" && -n "$reg_token" ]]; then
|
||||||
|
# Non-sensitive probe: /api/v1/user returns a scope message for token-based creds.
|
||||||
|
out="$(dc_curl 'https://git.allucanget.biz/api/v1/user' "" "$reg_token")"
|
||||||
|
if [[ "$out" =~ token.does.not.have.at.least.one.of.required.scope || "$out" =~ '"login"' || "$out" =~ invalid_grant ]]; then
|
||||||
|
if [[ "$out" =~ invalid_grant|Unauthorized|invalid ]]; then
|
||||||
|
fail "Gitea registry token rejected ($reg_user): $out"
|
||||||
|
else
|
||||||
|
ok "Gitea registry token valid for '$reg_user' (scope: write:package/write:repository)"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
warn "Gitea registry token response unexpected: $out"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
warn "CI_REGISTRY_USER / CI_REGISTRY_TOKEN not set; skipping Gitea registry check"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- 2. Cloudflare API token ---
|
||||||
|
echo ""
|
||||||
|
echo "-- Cloudflare API tokens --"
|
||||||
|
cf_token="$(get_val CLOUDFLARE_API_TOKEN)"
|
||||||
|
if [[ -n "$cf_token" ]]; then
|
||||||
|
out="$(dc_curl 'https://api.cloudflare.com/client/v4/user/tokens/verify' "$cf_token")"
|
||||||
|
if [[ "$out" =~ '"success":true' ]]; then
|
||||||
|
if [[ "$out" =~ '"status":"active"' ]]; then ok 'Cloudflare API token valid and ACTIVE'; else warn "Cloudflare API token valid but status: $out"; fi
|
||||||
|
else
|
||||||
|
fail "Cloudflare API token rejected: $out"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
warn "CLOUDFLARE_API_TOKEN not set; skipping Cloudflare check"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- 3. Cloudflare R2 / S3 bucket access ---
|
||||||
|
echo ""
|
||||||
|
echo "-- Cloudflare R2 (S3-compatible) --"
|
||||||
|
r2_key="$(get_val R2_ACCESS_KEY_ID)"
|
||||||
|
r2_secret="$(get_val R2_SECRET_ACCESS_KEY)"
|
||||||
|
r2_ep="$(get_val R2_ENDPOINT)"
|
||||||
|
bucket="$(get_val R2_BUCKET_NAME)"
|
||||||
|
|
||||||
|
# Use the admin key pair (which has ListBuckets) when present, else the regular key.
|
||||||
|
list_k="$(get_val R2_ADMIN_ACCESS_KEY_ID)"; list_s="$(get_val R2_ADMIN_SECRET_ACCESS_KEY)"
|
||||||
|
if [[ -z "$list_k" || -z "$list_s" ]]; then list_k="$r2_key"; list_s="$r2_secret"; fi
|
||||||
|
|
||||||
|
if [[ -n "$list_k" && -n "$list_s" && -n "$r2_ep" ]]; then
|
||||||
|
out="$(docker run --rm --entrypoint sh amazon/aws-cli:latest -c "export AWS_ACCESS_KEY_ID='$list_k'; export AWS_SECRET_ACCESS_KEY='$list_s'; aws s3 ls --endpoint-url '$r2_ep' --region auto 2>&1" 2>&1)"
|
||||||
|
if grep -q "$bucket" <<<"$out"; then ok "R2 bucket listing OK, '$bucket' present"
|
||||||
|
elif grep -q 'AccessDenied' <<<"$out"; then fail "R2 listing denied (no ListBuckets permission): $out"
|
||||||
|
elif grep -qE '^[^[:space:]]+[[:space:]]+' <<<"$out"; then warn "R2 listing returned (unexpected): $out"
|
||||||
|
else fail "R2 access failed: $out"; fi
|
||||||
|
else
|
||||||
|
warn "R2 credentials / endpoint not fully set; skipping R2 check"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Verify the specific application bucket is reachable with the primary key (read).
|
||||||
|
if [[ -n "$r2_key" && -n "$r2_secret" && -n "$r2_ep" && -n "$bucket" ]]; then
|
||||||
|
out="$(docker run --rm --entrypoint sh amazon/aws-cli:latest -c "export AWS_ACCESS_KEY_ID='$r2_key'; export AWS_SECRET_ACCESS_KEY='$r2_secret'; aws s3 ls s3://$bucket/ --endpoint-url '$r2_ep' --region auto --max-keys 5 2>&1 | head -5" 2>&1)"
|
||||||
|
if grep -q 'AccessDenied' <<<"$out"; then fail "Bucket '$bucket' read denied with primary key"
|
||||||
|
elif [[ -z "$out" ]]; then ok "Bucket '$bucket' reachable (empty listing, expected)"
|
||||||
|
elif grep -q 'NoSuchBucket' <<<"$out"; then fail "Bucket '$bucket' NOT FOUND"
|
||||||
|
else ok "Bucket '$bucket' reachable"; fi
|
||||||
|
elif [[ -z "$bucket" ]]; then
|
||||||
|
warn "R2_BUCKET_NAME not set; skipping bucket-level check"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- 4. Linear Void admin login ---
|
||||||
|
echo ""
|
||||||
|
echo "-- Linear Void admin login --"
|
||||||
|
admin_user="$(get_val ADMIN_USERNAME)"
|
||||||
|
admin_pass="$(get_val ADMIN_PASSWORD)"
|
||||||
|
if [[ -n "$admin_user" && -n "$admin_pass" ]]; then
|
||||||
|
# Hit the chat backend directly (public origin is caught by nginx SPA fallback).
|
||||||
|
url="http://${TARGET}:3000/api/admin/login"
|
||||||
|
body="{\"username\":\"$admin_user\",\"password\":\"$admin_pass\"}"
|
||||||
|
wrong_body="{\"username\":\"$admin_user\",\"password\":\"__definitely_wrong__\"}"
|
||||||
|
ok_code="$(docker run --rm curlimages/curl:latest -sS --max-time "$TIMEOUT" -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/json' -d "$body" "$url" 2>/dev/null | tr -d '[:space:]')"
|
||||||
|
wrong_code="$(docker run --rm curlimages/curl:latest -sS --max-time "$TIMEOUT" -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/json' -d "$wrong_body" "$url" 2>/dev/null | tr -d '[:space:]')"
|
||||||
|
if [[ "$ok_code" == "200" && "$wrong_code" == "401" ]]; then ok "Admin credentials valid (correct=200, wrong=401)"
|
||||||
|
else fail "Admin login unexpected (correct=$ok_code, wrong=$wrong_code)"; fi
|
||||||
|
else
|
||||||
|
warn "ADMIN_USERNAME / ADMIN_PASSWORD not set; skipping admin check"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
if [[ "$FAILED" -ne 0 ]]; then echo "RESULT: FAILED"; exit 1; fi
|
||||||
|
echo "RESULT: OK (no failing checks)"
|
||||||
|
exit 0
|
||||||
+20
-49
@@ -2,66 +2,42 @@
|
|||||||
|
|
||||||
These files back the [scaling/performance playbook](../../docs/operations/scaling-performance-playbook.md).
|
These files back the [scaling/performance playbook](../../docs/operations/scaling-performance-playbook.md).
|
||||||
|
|
||||||
## WebSocket chat
|
## WebSocket chat (artillery)
|
||||||
|
|
||||||
Run with an explicit WebSocket URL, for example:
|
Run with an explicit WebSocket URL, capturing output to an artifact file:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
WS_URL=wss://linearvoid.tv/ws?username=loadtest&room=main npx artillery run scripts/loadtest/ws-chat.yml
|
export WS_URL='wss://linearvoid.tv/ws?username=loadtest&room=main'
|
||||||
```
|
mkdir -p artifacts
|
||||||
|
|
||||||
Capture output to an artifact file for the worksheet:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
WS_URL=wss://linearvoid.tv/ws?username=loadtest&room=main npx artillery run scripts/loadtest/ws-chat.yml --output artifacts/ws-chat-report.json
|
|
||||||
```
|
|
||||||
|
|
||||||
PowerShell equivalent:
|
|
||||||
|
|
||||||
```powershell
|
|
||||||
$env:WS_URL = 'wss://linearvoid.tv/ws?username=loadtest&room=main'
|
|
||||||
New-Item -ItemType Directory -Force artifacts | Out-Null
|
|
||||||
npx artillery run scripts/loadtest/ws-chat.yml --output artifacts/ws-chat-report.json
|
npx artillery run scripts/loadtest/ws-chat.yml --output artifacts/ws-chat-report.json
|
||||||
```
|
```
|
||||||
|
|
||||||
Summarize key counters, rates, and available percentile stats from a saved Artillery JSON report:
|
Summarize key counters, rates, and available percentile stats from a saved Artillery JSON report:
|
||||||
|
|
||||||
```powershell
|
```bash
|
||||||
pwsh -File scripts/loadtest/summarize-artillery.ps1 -ReportFile artifacts/ws-chat-report.json
|
scripts/loadtest/summarize-artillery.sh artifacts/ws-chat-report.json
|
||||||
```
|
```
|
||||||
|
|
||||||
## Admin API
|
## Admin API (k6)
|
||||||
|
|
||||||
Run with a cookie or session header that can access admin endpoints:
|
Run with a cookie or session header that can access admin endpoints, capturing machine-readable metrics and a text summary:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ADMIN_BASE_URL=https://linearvoid.tv ADMIN_SESSION_COOKIE="session=..." k6 run scripts/loadtest/admin-api.js
|
export ADMIN_BASE_URL='https://linearvoid.tv'
|
||||||
```
|
export ADMIN_SESSION_COOKIE='session=...'
|
||||||
|
mkdir -p artifacts
|
||||||
Capture machine-readable metrics and a text summary:
|
k6 run --summary-export artifacts/admin-api-summary.json scripts/loadtest/admin-api.js | tee artifacts/admin-api-output.txt
|
||||||
|
|
||||||
```bash
|
|
||||||
ADMIN_BASE_URL=https://linearvoid.tv ADMIN_SESSION_COOKIE="session=..." k6 run --summary-export artifacts/admin-api-summary.json scripts/loadtest/admin-api.js > artifacts/admin-api-output.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
PowerShell equivalent:
|
|
||||||
|
|
||||||
```powershell
|
|
||||||
$env:ADMIN_BASE_URL = 'https://linearvoid.tv'
|
|
||||||
$env:ADMIN_SESSION_COOKIE = 'session=...'
|
|
||||||
New-Item -ItemType Directory -Force artifacts | Out-Null
|
|
||||||
k6 run --summary-export artifacts/admin-api-summary.json scripts/loadtest/admin-api.js | Tee-Object -FilePath artifacts/admin-api-output.txt
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The script emits global and per-endpoint thresholds (overview, analytics, chat-log, users) so p95 and error-rate regressions are visible per route.
|
The script emits global and per-endpoint thresholds (overview, analytics, chat-log, users) so p95 and error-rate regressions are visible per route.
|
||||||
|
|
||||||
Summarize endpoint p95 and failure rates from a saved k6 JSON summary:
|
Summarize endpoint p95 and failure rates from a saved k6 JSON summary:
|
||||||
|
|
||||||
```powershell
|
```bash
|
||||||
pwsh -File scripts/loadtest/summarize-k6.ps1 -SummaryFile artifacts/admin-api-summary.json
|
scripts/loadtest/summarize-k6.sh artifacts/admin-api-summary.json
|
||||||
```
|
```
|
||||||
|
|
||||||
## PowerShell helper
|
## Runner helper
|
||||||
|
|
||||||
Use the helper to run one scenario or both and automatically write outputs into a dated artifact folder:
|
Use the helper to run one scenario or both and automatically write outputs into a dated artifact folder:
|
||||||
|
|
||||||
@@ -69,16 +45,11 @@ Use the helper to run one scenario or both and automatically write outputs into
|
|||||||
- WebSocket runs also emit `ws-chat-summary.txt` with connection/error counters and available summary percentiles.
|
- WebSocket runs also emit `ws-chat-summary.txt` with connection/error counters and available summary percentiles.
|
||||||
- Admin runs also emit `admin-api-endpoint-summary.txt` with per-endpoint p95/error values.
|
- Admin runs also emit `admin-api-endpoint-summary.txt` with per-endpoint p95/error values.
|
||||||
|
|
||||||
```powershell
|
```bash
|
||||||
pwsh -File scripts/loadtest/run-loadtests.ps1 -Scenario ws -WsUrl 'wss://linearvoid.tv/ws?username=loadtest&room=main'
|
scripts/loadtest/run-loadtests.sh -s ws -w 'wss://linearvoid.tv/ws?username=loadtest&room=main'
|
||||||
```
|
scripts/loadtest/run-loadtests.sh -s admin -a 'https://linearvoid.tv' -c 'session=...'
|
||||||
|
scripts/loadtest/run-loadtests.sh -s all -w 'wss://linearvoid.tv/ws?username=loadtest&room=main' \
|
||||||
```powershell
|
-a 'https://linearvoid.tv' -c 'session=...'
|
||||||
pwsh -File scripts/loadtest/run-loadtests.ps1 -Scenario admin -AdminBaseUrl 'https://linearvoid.tv' -AdminSessionCookie 'session=...'
|
|
||||||
```
|
|
||||||
|
|
||||||
```powershell
|
|
||||||
pwsh -File scripts/loadtest/run-loadtests.ps1 -Scenario all -WsUrl 'wss://linearvoid.tv/ws?username=loadtest&room=main' -AdminBaseUrl 'https://linearvoid.tv' -AdminSessionCookie 'session=...'
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
|
|||||||
@@ -0,0 +1,72 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# run-loadtests.sh — Run an artillery (ws) and/or k6 (admin) load-test scenario
|
||||||
|
# and write outputs into a dated artifacts folder.
|
||||||
|
# Mirrors scripts/loadtest/run-loadtests.ps1.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./scripts/loadtest/run-loadtests.sh -s ws -w 'wss://...'
|
||||||
|
# ./scripts/loadtest/run-loadtests.sh -s admin -a 'https://...' -c 'session=...'
|
||||||
|
# ./scripts/loadtest/run-loadtests.sh -s all -w 'wss://...' -a 'https://...' -c 'session=...'
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
repo_root="$(cd "$script_dir/../.." && pwd)"
|
||||||
|
|
||||||
|
SCENARIO="all"
|
||||||
|
ARTIFACT_ROOT="artifacts"
|
||||||
|
WS_URL="${WS_URL:-}"
|
||||||
|
ADMIN_BASE_URL="${ADMIN_BASE_URL:-}"
|
||||||
|
ADMIN_SESSION_COOKIE="${ADMIN_SESSION_COOKIE:-}"
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
echo "Usage: $0 [-s ws|admin|all] [-w ws_url] [-a admin_base_url] [-c admin_session_cookie] [-r artifact_root]" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
-s) SCENARIO="$2"; shift 2;;
|
||||||
|
-w) WS_URL="$2"; shift 2;;
|
||||||
|
-a) ADMIN_BASE_URL="$2"; shift 2;;
|
||||||
|
-c) ADMIN_SESSION_COOKIE="$2"; shift 2;;
|
||||||
|
-r) ARTIFACT_ROOT="$2"; shift 2;;
|
||||||
|
-h|--help) usage;;
|
||||||
|
*) echo "Unknown option: $1" >&2; usage;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
case "$SCENARIO" in ws|admin|all) ;; *) echo "Scenario must be ws|admin|all" >&2; exit 2;; esac
|
||||||
|
|
||||||
|
if [[ "$SCENARIO" == ws || "$SCENARIO" == all ]] && ! command -v npx >/dev/null 2>&1; then
|
||||||
|
echo "npx is required for ws scenario but was not found in PATH." >&2; exit 1
|
||||||
|
fi
|
||||||
|
if [[ "$SCENARIO" == admin || "$SCENARIO" == all ]] && ! command -v k6 >/dev/null 2>&1; then
|
||||||
|
echo "k6 is required for admin scenario but was not found in PATH." >&2; exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$ARTIFACT_ROOT" = /* ]]; then artifact_base="$ARTIFACT_ROOT"; else artifact_base="$repo_root/$ARTIFACT_ROOT"; fi
|
||||||
|
artifact_dir="$artifact_base/$(date +%Y-%m-%d_%H%M%S)"
|
||||||
|
mkdir -p "$artifact_dir"
|
||||||
|
|
||||||
|
if [[ "$SCENARIO" == ws || "$SCENARIO" == all ]]; then
|
||||||
|
[[ -n "$WS_URL" ]] || { echo "WS_URL is required for ws scenario. Pass -w or set \$WS_URL." >&2; exit 1; }
|
||||||
|
ws_out="$artifact_dir/ws-chat-report.json"
|
||||||
|
ws_sum="$artifact_dir/ws-chat-summary.txt"
|
||||||
|
echo "== Running artillery ws scenario =="
|
||||||
|
WS_URL="$WS_URL" npx artillery run "$script_dir/ws-chat.yml" --output "$ws_out"
|
||||||
|
"$script_dir/summarize-artillery.sh" "$ws_out" | tee "$ws_sum"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$SCENARIO" == admin || "$SCENARIO" == all ]]; then
|
||||||
|
[[ -n "$ADMIN_BASE_URL" ]] || { echo "ADMIN_BASE_URL is required for admin scenario. Pass -a or set \$ADMIN_BASE_URL." >&2; exit 1; }
|
||||||
|
[[ -n "$ADMIN_SESSION_COOKIE" ]] || { echo "ADMIN_SESSION_COOKIE is required for admin scenario. Pass -c or set \$ADMIN_SESSION_COOKIE." >&2; exit 1; }
|
||||||
|
admin_sum="$artifact_dir/admin-api-summary.json"
|
||||||
|
admin_out="$artifact_dir/admin-api-output.txt"
|
||||||
|
admin_ep_sum="$artifact_dir/admin-api-endpoint-summary.txt"
|
||||||
|
echo "== Running k6 admin scenario =="
|
||||||
|
ADMIN_BASE_URL="$ADMIN_BASE_URL" ADMIN_SESSION_COOKIE="$ADMIN_SESSION_COOKIE" \
|
||||||
|
k6 run --summary-export "$admin_sum" "$script_dir/admin-api.js" | tee "$admin_out"
|
||||||
|
"$script_dir/summarize-k6.sh" "$admin_sum" | tee "$admin_ep_sum"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Artifacts written to $artifact_dir"
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# summarize-artillery.sh — Extract counters, rates and percentile stats from an
|
||||||
|
# Artillery JSON report. Mirrors scripts/loadtest/summarize-artillery.ps1.
|
||||||
|
# Usage: ./scripts/loadtest/summarize-artillery.sh <artillery-report.json>
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
if [[ $# -lt 1 ]]; then
|
||||||
|
echo "Usage: $0 <artillery-report.json>" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
report_file="$1"
|
||||||
|
[[ -f "$report_file" ]] || { echo "Report file not found: $report_file" >&2; exit 1; }
|
||||||
|
|
||||||
|
REPORT_FILE="$report_file" node <<'EOF'
|
||||||
|
const fs = require('fs');
|
||||||
|
const report = JSON.parse(fs.readFileSync(process.env.REPORT_FILE, 'utf8'));
|
||||||
|
const aggregate = report.aggregate;
|
||||||
|
if (!aggregate) throw new Error('Artillery report is missing aggregate data.');
|
||||||
|
|
||||||
|
const get = (obj, name) => (obj && obj[name] !== undefined) ? obj[name] : undefined;
|
||||||
|
const counters = aggregate.counters || {};
|
||||||
|
const rates = aggregate.rates || {};
|
||||||
|
const summaries = aggregate.summaries || {};
|
||||||
|
|
||||||
|
const created = Number(get(counters, 'vusers.created') || 0);
|
||||||
|
const failed = Number(get(counters, 'vusers.failed') || 0);
|
||||||
|
const completed = Number(get(counters, 'vusers.completed') || 0);
|
||||||
|
const errorRate = created > 0 ? +((failed / created) * 100).toFixed(2) : 0;
|
||||||
|
|
||||||
|
console.log('=== Artillery WebSocket Summary ===');
|
||||||
|
console.log(`source: ${process.env.REPORT_FILE}`);
|
||||||
|
if (report.timestamp) console.log(`timestamp: ${report.timestamp}`);
|
||||||
|
console.log(`vusers.created: ${created}`);
|
||||||
|
console.log(`vusers.completed: ${completed}`);
|
||||||
|
console.log(`vusers.failed: ${failed}`);
|
||||||
|
console.log(`vusers.failure_rate_percent: ${errorRate}`);
|
||||||
|
|
||||||
|
for (const metric of ['ws.messages_sent', 'ws.messages_received', 'ws.ping', 'plugins.metrics-by-endpoint.response_time']) {
|
||||||
|
const v = get(counters, metric);
|
||||||
|
if (v !== undefined) console.log(`${metric}: ${v}`);
|
||||||
|
}
|
||||||
|
for (const [k, v] of Object.entries(rates || {})) console.log(`rate.${k}: ${v}`);
|
||||||
|
|
||||||
|
const fmt = (prefix, stats) => {
|
||||||
|
if (!stats) return;
|
||||||
|
const parts = [];
|
||||||
|
for (const name of ['count', 'min', 'max', 'mean', 'median', 'p95', 'p99']) {
|
||||||
|
const v = get(stats, name);
|
||||||
|
if (v !== undefined) parts.push(`${name}=${v}`);
|
||||||
|
}
|
||||||
|
if (parts.length) console.log(`${prefix}: ${parts.join(', ')}`);
|
||||||
|
};
|
||||||
|
for (const [k, v] of Object.entries(summaries)) fmt(`summary.${k}`, v);
|
||||||
|
EOF
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# summarize-k6.sh — Extract per-endpoint p95 and failure rates from a k6 JSON
|
||||||
|
# summary export. Mirrors scripts/loadtest/summarize-k6.ps1.
|
||||||
|
# Usage: ./scripts/loadtest/summarize-k6.sh <k6-summary.json>
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
if [[ $# -lt 1 ]]; then
|
||||||
|
echo "Usage: $0 <k6-summary.json>" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
summary_file="$1"
|
||||||
|
[[ -f "$summary_file" ]] || { echo "Summary file not found: $summary_file" >&2; exit 1; }
|
||||||
|
|
||||||
|
SUMMARY_FILE="$summary_file" node <<'EOF'
|
||||||
|
const fs = require('fs');
|
||||||
|
const summary = JSON.parse(fs.readFileSync(process.env.SUMMARY_FILE, 'utf8'));
|
||||||
|
const metrics = summary.metrics;
|
||||||
|
if (!metrics) throw new Error('Invalid k6 summary file: missing metrics object.');
|
||||||
|
|
||||||
|
const val = (m, key) => (m && m.values && m.values[key] !== undefined) ? Number(m.values[key]) : undefined;
|
||||||
|
const globalDuration = val(metrics.http_req_duration, 'p(95)');
|
||||||
|
const globalFailRate = val(metrics.http_req_failed, 'rate');
|
||||||
|
|
||||||
|
console.log(`k6 Summary: ${process.env.SUMMARY_FILE}`);
|
||||||
|
if (globalDuration !== undefined) console.log(`Global p95 duration (ms): ${globalDuration.toFixed(2)}`);
|
||||||
|
if (globalFailRate !== undefined) console.log(`Global failure rate: ${(globalFailRate * 100).toFixed(2)}%`);
|
||||||
|
console.log('');
|
||||||
|
|
||||||
|
const endpoints = ['overview', 'analytics', 'chat_log', 'users'];
|
||||||
|
const rows = endpoints.map((ep) => {
|
||||||
|
const dP95 = val(metrics[`http_req_duration{endpoint:${ep}}`], 'p(95)');
|
||||||
|
const fRate = val(metrics[`http_req_failed{endpoint:${ep}}`], 'rate');
|
||||||
|
return {
|
||||||
|
Endpoint: ep,
|
||||||
|
P95Ms: dP95 !== undefined ? dP95.toFixed(2) : 'n/a',
|
||||||
|
FailurePct: fRate !== undefined ? (fRate * 100).toFixed(4) : 'n/a',
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
|
const pad = (s, n) => String(s).padEnd(n);
|
||||||
|
console.log(pad('ENDPOINT', 12) + pad('P95_MS', 10) + 'FAILURE_PCT');
|
||||||
|
for (const r of rows) console.log(pad(r.Endpoint, 12) + pad(r.P95Ms, 10) + r.FailurePct);
|
||||||
|
EOF
|
||||||
Reference in new issue
Block a user