299 lines
11 KiB
Markdown
299 lines
11 KiB
Markdown
# Deployment Guide
|
||
|
||
## Table of Contents
|
||
|
||
- [Deployment Guide](#deployment-guide)
|
||
- [Table of Contents](#table-of-contents)
|
||
- [Environments](#environments)
|
||
- [Production Target (Hetzner)](#production-target-hetzner)
|
||
- [Planned Infrastructure](#planned-infrastructure)
|
||
- [Cutover Plan](#cutover-plan)
|
||
- [Production Preflight](#production-preflight)
|
||
- [Firewall / Security Group](#firewall--security-group)
|
||
- [Pipeline Readiness](#pipeline-readiness)
|
||
- [Target Topology](#target-topology)
|
||
- [Prerequisites](#prerequisites)
|
||
- [Environment File](#environment-file)
|
||
- [Pull-and-Run Deployment](#pull-and-run-deployment)
|
||
- [First-Time or Recovery Provisioning](#first-time-or-recovery-provisioning)
|
||
- [Health Verification](#health-verification)
|
||
- [Zero/Low-Downtime Update Pattern](#zerolow-downtime-update-pattern)
|
||
- [Rollback](#rollback)
|
||
- [Operational Tasks](#operational-tasks)
|
||
- [View logs](#view-logs)
|
||
- [Restart one service](#restart-one-service)
|
||
- [Stop stack](#stop-stack)
|
||
- [Keep volumes (recommended) vs remove volumes (destructive)](#keep-volumes-recommended-vs-remove-volumes-destructive)
|
||
- [Nginx and Proxy Considerations](#nginx-and-proxy-considerations)
|
||
- [Security Baseline](#security-baseline)
|
||
- [Common Deployment Pitfalls](#common-deployment-pitfalls)
|
||
- [Troubleshooting](#troubleshooting)
|
||
|
||
## Environments
|
||
|
||
Linear Void runs in two separate environments:
|
||
|
||
| Environment | Host | Docker Target | Purpose |
|
||
| -------------- | -------------------------- | -------------------------- | ---------------------------------------------- |
|
||
| **Test Lab** | `192.168.88.40` (internal) | `tcp://192.168.88.40:2375` | Integration testing, staging, CI/CD validation |
|
||
| **Production** | Hetzner VPS (TBD) | `tcp://<hetzner-ip>:2375` | Public linearvoid.tv serving real viewers |
|
||
|
||
**Current state (2026-08-06):** Test lab is active and receives automated CD deploys. Production Hetzner VPS is **not yet provisioned**. The CD workflow (`cd.yml`) and `docker-compose.prod.yml` currently target the test lab. Switching to production requires only changing `DOCKER_HOST` — see [Production Deployment](#production-target-hetzner) below.
|
||
|
||
## Production Target (Hetzner)
|
||
|
||
**Status: Not provisioned.** The production deployment plan is documented and the pipeline is ready — only the VPS needs to be created.
|
||
|
||
### Planned Infrastructure
|
||
|
||
| Resource | Spec |
|
||
| -------------- | ------------------------------------------------ |
|
||
| Provider | Hetzner Cloud |
|
||
| Region | Ashburn, VA (us-east) |
|
||
| Instance | CX22 (2 vCPU, 4 GB RAM, 40 GB SSD) or equivalent |
|
||
| OS | Ubuntu 24.04 LTS or Debian 12 |
|
||
| Bandwidth | 20 TB included traffic/month |
|
||
| Estimated cost | ~€7–10/month base |
|
||
|
||
### Cutover Plan
|
||
|
||
1. Provision Hetzner VPS (see [Production Deployment](#production-target-hetzner) section)
|
||
2. Add production secrets to Gitea (`CI_PROD_*` prefix)
|
||
3. Update `DOCKER_HOST` in CD workflow to Hetzner public IP
|
||
4. Trigger deploy → validate health checks
|
||
5. Update Cloudflare A record `linearvoid.tv` → Hetzner IP
|
||
6. Enable HTTPS, WAF, cache rules on production zone
|
||
7. Decommission test lab as public target
|
||
|
||
### Production Preflight
|
||
|
||
Before the cutover in the plan above, treat the following as the minimum readiness gate:
|
||
|
||
1. Verify the Hetzner host is reachable over SSH and Docker remote API TLS is working for the Gitea runner.
|
||
2. Confirm all required production secrets are present under the `CI_PROD_*` prefix.
|
||
3. Confirm `linearvoid.tv`, `www.linearvoid.tv`, and `media.linearvoid.tv` DNS targets are correct for the intended phase.
|
||
4. Complete the manual Cloudflare/UptimeRobot checklist and attach evidence in `docs/operations/cloudflare-uptimerobot-checklist.md`.
|
||
5. Run the production smoke path end to end: viewer SPA, HLS playback, chat connect, admin login, upload, playlist save, and playout health.
|
||
6. Verify rollback is still possible before the final DNS switch.
|
||
|
||
Readiness is not complete until the checklist above and the manual evidence tracker are both satisfied.
|
||
|
||
### Firewall / Security Group
|
||
|
||
| Port | Source | Purpose |
|
||
| ---- | --------------------- | ------------------------------ |
|
||
| 22 | Admin IPs | SSH management |
|
||
| 80 | 0.0.0.0/0 | HTTP (redirect to HTTPS) |
|
||
| 443 | 0.0.0.0/0 | HTTPS (nginx) |
|
||
| 2375 | Gitea runner IP only | Docker remote API (TLS) |
|
||
| 9090 | Admin IPs or internal | Prometheus (or tunnel via SSH) |
|
||
| 3001 | Admin IPs or internal | Grafana (or tunnel via SSH) |
|
||
|
||
### Pipeline Readiness
|
||
|
||
Three Gitea Actions workflows drive the full pipeline:
|
||
|
||
| Workflow | Trigger | Purpose |
|
||
| ------------- | ---------------- | ---------------------------------------------------------------------------------------- |
|
||
| `ci.yml` | Push to `main` | Lint, build frontend, vet Go, build & push Docker images (`:latest`, `:main`) |
|
||
| `cd.yml` | Push to `main` | Deploy to test lab (`192.168.88.40:2375`) — runs **in parallel** with CI |
|
||
| `release.yml` | Tag push `*.*.*` | Build & push versioned Docker images (`:X.Y.Z`, `:X.Y`, `:latest`), create Gitea Release |
|
||
|
||
CI and CD run simultaneously on every push to `main`. The deploy pulls whatever `:latest` images are available in the registry at that moment.
|
||
|
||
The CD workflow (`cd.yml`) uses `DOCKER_HOST` to target the deployment host. To switch targets:
|
||
|
||
```bash
|
||
# Test lab (current)
|
||
DOCKER_HOST=tcp://192.168.88.40:2375
|
||
|
||
# Production (when provisioned)
|
||
DOCKER_HOST=tcp://<hetzner-public-ip>:2375
|
||
```
|
||
|
||
No code changes needed in the pipeline. See `DEPLOYMENT.md` Production Deployment section for full checklist.
|
||
|
||
## Target Topology
|
||
|
||
- Host: Linux server with Docker Engine
|
||
- Reverse proxy: nginx container (linearvoid-nginx)
|
||
- Core services: playout, chat, redis, postgres
|
||
- Observability: prometheus, grafana
|
||
- CDN/edge: Cloudflare
|
||
|
||
## Prerequisites
|
||
|
||
- Docker Engine + Docker Compose plugin
|
||
- Network access to git.allucanget.biz
|
||
- A valid .env file with production values
|
||
- DNS configured for linearvoid.tv (and media.linearvoid.tv if split media domain is used)
|
||
|
||
## Environment File
|
||
|
||
Use .env with at least:
|
||
|
||
- CI_REGISTRY_URL
|
||
- CI_REGISTRY_NAMESPACE
|
||
- CHAT_PORT
|
||
- NGINX_PORT
|
||
- DATABASE_URL (if overridden)
|
||
- REDIS_URL (if overridden)
|
||
- ADMIN_USERNAME
|
||
- ADMIN_PASSWORD
|
||
- VITE_PUBLIC_APP_URL
|
||
- VITE_PUBLIC_WS_URL
|
||
- VITE_PUBLIC_HLS_URL
|
||
- Cloudflare/R2 variables when media sync is enabled
|
||
|
||
See .env.example and SETUP.md for full variable reference.
|
||
|
||
## Pull-and-Run Deployment
|
||
|
||
Images are built and pushed automatically by the CI and release workflows (see [CONTRIBUTING.md](../CONTRIBUTING.md#versioning-and-releases) for the release procedure).
|
||
|
||
Every push to `main` triggers CI (builds `:main` and `:latest` tags) and CD (deploys `:latest` to test lab) in parallel. Semantic version tags (`*.*.*`, e.g. `1.0.9`) trigger the release workflow which builds `:X.Y.Z`, `:X.Y`, and `:latest` tags and creates a Gitea Release.
|
||
|
||
From repository root:
|
||
|
||
```bash
|
||
docker login git.allucanget.biz
|
||
|
||
docker compose -f docker-compose.prod.yml pull
|
||
docker compose -f docker-compose.prod.yml up -d --remove-orphans
|
||
```
|
||
|
||
## First-Time or Recovery Provisioning
|
||
|
||
1. Ensure host paths exist:
|
||
|
||
```bash
|
||
mkdir -p data/chat data/playlist data/media data/hls
|
||
```
|
||
|
||
1. Confirm volumes and networks:
|
||
|
||
```bash
|
||
docker volume ls
|
||
docker network ls
|
||
```
|
||
|
||
1. Start stack:
|
||
|
||
```bash
|
||
docker compose -f docker-compose.prod.yml up -d
|
||
```
|
||
|
||
## Health Verification
|
||
|
||
After startup, check service status:
|
||
|
||
```bash
|
||
docker compose -f docker-compose.prod.yml ps
|
||
```
|
||
|
||
Endpoint checks:
|
||
|
||
- App: <http://{host}:8080/>
|
||
- Nginx health: <http://{host}:8080/health>
|
||
- Chat health: <http://{host}:3000/health>
|
||
- Playout health: <http://{host}:8081/health>
|
||
- Prometheus: <http://{host}:9090>
|
||
- Grafana: <http://{host}:3001>
|
||
|
||
Domain checks:
|
||
|
||
- <https://linearvoid.tv>
|
||
- <https://linearvoid.tv/admin>
|
||
- <https://linearvoid.tv/metrics>
|
||
|
||
## Zero/Low-Downtime Update Pattern
|
||
|
||
1. Pull newest images.
|
||
1. Recreate changed services only:
|
||
|
||
```bash
|
||
docker compose -f docker-compose.prod.yml pull
|
||
|
||
docker compose -f docker-compose.prod.yml up -d nginx chat playout
|
||
```
|
||
|
||
1. Re-run health checks.
|
||
|
||
## Rollback
|
||
|
||
If new images are unhealthy:
|
||
|
||
1. Pin previous tags in docker-compose.prod.yml.
|
||
2. Pull pinned tags.
|
||
3. Restart impacted services.
|
||
|
||
```bash
|
||
docker compose -f docker-compose.prod.yml pull
|
||
|
||
docker compose -f docker-compose.prod.yml up -d nginx chat playout
|
||
```
|
||
|
||
## Operational Tasks
|
||
|
||
### View logs
|
||
|
||
```bash
|
||
docker compose -f docker-compose.prod.yml logs -f nginx chat playout
|
||
```
|
||
|
||
### Restart one service
|
||
|
||
```bash
|
||
docker compose -f docker-compose.prod.yml restart chat
|
||
```
|
||
|
||
### Stop stack
|
||
|
||
```bash
|
||
docker compose -f docker-compose.prod.yml down
|
||
```
|
||
|
||
### Keep volumes (recommended) vs remove volumes (destructive)
|
||
|
||
```bash
|
||
# keep data
|
||
docker compose -f docker-compose.prod.yml down
|
||
|
||
# destructive (remove data)
|
||
docker compose -f docker-compose.prod.yml down -v
|
||
```
|
||
|
||
## Nginx and Proxy Considerations
|
||
|
||
- If an external proxy (for example Nginx Proxy Manager) fronts the stack, align request body limits with internal nginx.
|
||
- For resumable admin uploads, ensure client_max_body_size supports chunk requests in both proxy layers.
|
||
|
||
## Security Baseline
|
||
|
||
- Use strong ADMIN_PASSWORD and rotate regularly.
|
||
- Keep Docker host access restricted.
|
||
- Prefer HTTPS/WSS through Cloudflare.
|
||
- Restrict exposed ports if only proxy access is needed.
|
||
|
||
## Common Deployment Pitfalls
|
||
|
||
- Missing frontend build artifacts in local-only compose workflows.
|
||
- Wrong registry credentials or missing login.
|
||
- Misaligned domain env vars (app/ws/hls) causing mixed-origin errors.
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Check | Fix |
|
||
| -------------------------- | --------------------------------------------- | ------------------------------------------------------------- |
|
||
| Service not starting | `docker logs linearvoid-<svc> --tail 50` | Check `.env` vars, volume mounts, port conflicts |
|
||
| Playout stall / slate mode | `curl http://<host>:8081/health` — mode=slate | Check playlist file, media paths, disk space |
|
||
| Chat not connecting | `curl http://<host>:3000/health` | Check Redis: `docker exec linearvoid-redis redis-cli ping` |
|
||
| Admin 401 loop | Cookie not sent | Verify `X-Forwarded-Proto: https`, nginx auth_request headers |
|
||
| Upload fails (413) | File too large | Check nginx `client_max_body_size` (should be 64m+) |
|
||
| HLS not playing | `curl http://<host>/hls/playlist.m3u8` | Check playout health, HLS dir permissions |
|
||
| Prometheus no targets | `curl http://<host>:9090/targets` | Verify container network, scrape config |
|
||
| Grafana no data | Check datasource | Verify Prometheus URL `http://prometheus:9090` provisioned |
|
||
|
||
- Proxy-layer body size limits returning 413 on upload.
|
||
- Health checks failing before service warmup is complete.
|