Files
linearvoid/docs/DEPLOYMENT.md
T
zwitschi f9aba11b90
CD / deploy (push) Successful in 57s
CI / test (push) Failing after 1m29s
CI / docker (push) Skipped
feat: add metrics, observability, test fixtures, and scaling playbook artifacts
2026-08-10 00:23:50 +02:00

11 KiB
Raw Permalink Blame History

Deployment Guide

Table of Contents

Environments

Linear Void runs in two separate environments:

Environment Host Docker Target Purpose
Test Lab 192.168.88.40 (internal) tcp://192.168.88.40:2375 Integration testing, staging, CI/CD validation
Production Hetzner VPS (TBD) tcp://<hetzner-ip>:2375 Public linearvoid.tv serving real viewers

Current state (2026-08-06): Test lab is active and receives automated CD deploys. Production Hetzner VPS is not yet provisioned. The CD workflow (cd.yml) and docker-compose.prod.yml currently target the test lab. Switching to production requires only changing DOCKER_HOST — see Production Deployment below.

Production Target (Hetzner)

Status: Not provisioned. The production deployment plan is documented and the pipeline is ready — only the VPS needs to be created.

Planned Infrastructure

Resource Spec
Provider Hetzner Cloud
Region Ashburn, VA (us-east)
Instance CX22 (2 vCPU, 4 GB RAM, 40 GB SSD) or equivalent
OS Ubuntu 24.04 LTS or Debian 12
Bandwidth 20 TB included traffic/month
Estimated cost ~€7–10/month base

Cutover Plan

  1. Provision Hetzner VPS (see Production Deployment section)
  2. Add production secrets to Gitea (CI_PROD_* prefix)
  3. Update DOCKER_HOST in CD workflow to Hetzner public IP
  4. Trigger deploy → validate health checks
  5. Update Cloudflare A record linearvoid.tv → Hetzner IP
  6. Enable HTTPS, WAF, cache rules on production zone
  7. Decommission test lab as public target

Production Preflight

Before the cutover in the plan above, treat the following as the minimum readiness gate:

  1. Verify the Hetzner host is reachable over SSH and Docker remote API TLS is working for the Gitea runner.
  2. Confirm all required production secrets are present under the CI_PROD_* prefix.
  3. Confirm linearvoid.tv, www.linearvoid.tv, and media.linearvoid.tv DNS targets are correct for the intended phase.
  4. Complete the manual Cloudflare/UptimeRobot checklist and attach evidence in docs/operations/cloudflare-uptimerobot-checklist.md.
  5. Run the production smoke path end to end: viewer SPA, HLS playback, chat connect, admin login, upload, playlist save, and playout health.
  6. Verify rollback is still possible before the final DNS switch.

Readiness is not complete until the checklist above and the manual evidence tracker are both satisfied.

Firewall / Security Group

Port Source Purpose
22 Admin IPs SSH management
80 0.0.0.0/0 HTTP (redirect to HTTPS)
443 0.0.0.0/0 HTTPS (nginx)
2375 Gitea runner IP only Docker remote API (TLS)
9090 Admin IPs or internal Prometheus (or tunnel via SSH)
3001 Admin IPs or internal Grafana (or tunnel via SSH)

Pipeline Readiness

Three Gitea Actions workflows drive the full pipeline:

Workflow Trigger Purpose
ci.yml Push to main Lint, build frontend, vet Go, build & push Docker images (:latest, :main)
cd.yml Push to main Deploy to test lab (192.168.88.40:2375) — runs in parallel with CI
release.yml Tag push *.*.* Build & push versioned Docker images (:X.Y.Z, :X.Y, :latest), create Gitea Release

CI and CD run simultaneously on every push to main. The deploy pulls whatever :latest images are available in the registry at that moment.

The CD workflow (cd.yml) uses DOCKER_HOST to target the deployment host. To switch targets:

# Test lab (current)
DOCKER_HOST=tcp://192.168.88.40:2375

# Production (when provisioned)
DOCKER_HOST=tcp://<hetzner-public-ip>:2375

No code changes needed in the pipeline. See DEPLOYMENT.md Production Deployment section for full checklist.

Target Topology

  • Host: Linux server with Docker Engine
  • Reverse proxy: nginx container (linearvoid-nginx)
  • Core services: playout, chat, redis, postgres
  • Observability: prometheus, grafana
  • CDN/edge: Cloudflare

Prerequisites

  • Docker Engine + Docker Compose plugin
  • Network access to git.allucanget.biz
  • A valid .env file with production values
  • DNS configured for linearvoid.tv (and media.linearvoid.tv if split media domain is used)

Environment File

Use .env with at least:

  • CI_REGISTRY_URL
  • CI_REGISTRY_NAMESPACE
  • CHAT_PORT
  • NGINX_PORT
  • DATABASE_URL (if overridden)
  • REDIS_URL (if overridden)
  • ADMIN_USERNAME
  • ADMIN_PASSWORD
  • VITE_PUBLIC_APP_URL
  • VITE_PUBLIC_WS_URL
  • VITE_PUBLIC_HLS_URL
  • Cloudflare/R2 variables when media sync is enabled

See .env.example and SETUP.md for full variable reference.

Pull-and-Run Deployment

Images are built and pushed automatically by the CI and release workflows (see CONTRIBUTING.md for the release procedure).

Every push to main triggers CI (builds :main and :latest tags) and CD (deploys :latest to test lab) in parallel. Semantic version tags (*.*.*, e.g. 1.0.9) trigger the release workflow which builds :X.Y.Z, :X.Y, and :latest tags and creates a Gitea Release.

From repository root:

docker login git.allucanget.biz

docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d --remove-orphans

First-Time or Recovery Provisioning

  1. Ensure host paths exist:
mkdir -p data/chat data/playlist data/media data/hls
  1. Confirm volumes and networks:
docker volume ls
docker network ls
  1. Start stack:
docker compose -f docker-compose.prod.yml up -d

Health Verification

After startup, check service status:

docker compose -f docker-compose.prod.yml ps

Endpoint checks:

  • App: http://{host}:8080/
  • Nginx health: http://{host}:8080/health
  • Chat health: http://{host}:3000/health
  • Playout health: http://{host}:8081/health
  • Prometheus: http://{host}:9090
  • Grafana: http://{host}:3001

Domain checks:

Zero/Low-Downtime Update Pattern

  1. Pull newest images.
  2. Recreate changed services only:
docker compose -f docker-compose.prod.yml pull

docker compose -f docker-compose.prod.yml up -d nginx chat playout
  1. Re-run health checks.

Rollback

If new images are unhealthy:

  1. Pin previous tags in docker-compose.prod.yml.
  2. Pull pinned tags.
  3. Restart impacted services.
docker compose -f docker-compose.prod.yml pull

docker compose -f docker-compose.prod.yml up -d nginx chat playout

Operational Tasks

View logs

docker compose -f docker-compose.prod.yml logs -f nginx chat playout

Restart one service

docker compose -f docker-compose.prod.yml restart chat

Stop stack

docker compose -f docker-compose.prod.yml down
# keep data
docker compose -f docker-compose.prod.yml down

# destructive (remove data)
docker compose -f docker-compose.prod.yml down -v

Nginx and Proxy Considerations

  • If an external proxy (for example Nginx Proxy Manager) fronts the stack, align request body limits with internal nginx.
  • For resumable admin uploads, ensure client_max_body_size supports chunk requests in both proxy layers.

Security Baseline

  • Use strong ADMIN_PASSWORD and rotate regularly.
  • Keep Docker host access restricted.
  • Prefer HTTPS/WSS through Cloudflare.
  • Restrict exposed ports if only proxy access is needed.

Common Deployment Pitfalls

  • Missing frontend build artifacts in local-only compose workflows.
  • Wrong registry credentials or missing login.
  • Misaligned domain env vars (app/ws/hls) causing mixed-origin errors.

Troubleshooting

Symptom Check Fix
Service not starting docker logs linearvoid-<svc> --tail 50 Check .env vars, volume mounts, port conflicts
Playout stall / slate mode curl http://<host>:8081/health — mode=slate Check playlist file, media paths, disk space
Chat not connecting curl http://<host>:3000/health Check Redis: docker exec linearvoid-redis redis-cli ping
Admin 401 loop Cookie not sent Verify X-Forwarded-Proto: https, nginx auth_request headers
Upload fails (413) File too large Check nginx client_max_body_size (should be 64m+)
HLS not playing curl http://<host>/hls/playlist.m3u8 Check playout health, HLS dir permissions
Prometheus no targets curl http://<host>:9090/targets Verify container network, scrape config
Grafana no data Check datasource Verify Prometheus URL http://prometheus:9090 provisioned
  • Proxy-layer body size limits returning 413 on upload.
  • Health checks failing before service warmup is complete.