linearvoid/docs/DEPLOYMENT.md

# Deployment Guide

## Environments

| Environment | Host | Docker Target | Purpose |
|---|---|---|---|
| **Test Lab** | `192.168.88.40` (internal) | `tcp://192.168.88.40:2375` | Integration testing, staging, CI/CD validation |
| **Production** | Hetzner VPS (TBD) | `tcp://<hetzner-ip>:2375` | Public linearvoid.tv |

**Current state (2026-08-06):** Test lab is active and receives automated CD deploys. Production Hetzner VPS is **not yet provisioned**. `cd.yml` and `docker-compose.prod.yml` target the test lab; switching to production only requires changing `DOCKER_HOST`.

## Production Target (Hetzner)

**Status: Not provisioned.** Plan documented and pipeline ready; only the VPS needs creation.

### Planned Infrastructure
- Provider: Hetzner Cloud, Region: Ashburn, VA (us-east)
- Instance: CX22 (2 vCPU, 4 GB RAM, 40 GB SSD) or equivalent
- OS: Ubuntu 24.04 LTS or Debian 12
- Bandwidth: 20 TB included traffic/month; Estimated cost: ~€7–10/month base

### Cutover Plan
1. Provision Hetzner VPS
2. Add production secrets to Gitea (`CI_PROD_*` prefix)
3. Update `DOCKER_HOST` in CD workflow to Hetzner public IP
4. Trigger deploy → validate health checks
5. Update Cloudflare A record `linearvoid.tv` → Hetzner IP
6. Enable HTTPS, WAF, cache rules on production zone
7. Decommission test lab as public target

### Production Preflight (minimum readiness gate)
1. Verify Hetzner host reachable over SSH and Docker remote API TLS works for the Gitea runner
2. Confirm all production secrets exist under `CI_PROD_*` prefix
3. Confirm `linearvoid.tv`, `www.linearvoid.tv`, `media.linearvoid.tv` DNS targets match intended phase
4. Complete manual Cloudflare/UptimeRobot checklist (see `docs/operations/cloudflare-uptimerobot-checklist.md`)
5. Run production smoke path end-to-end: viewer SPA, HLS playback, chat connect, admin login, upload, playlist save, playout health
6. Verify rollback possible before final DNS switch

### Firewall / Security Group
| Port | Source | Purpose |
|---|---|---|
| 22 | Admin IPs | SSH management |
| 80 | 0.0.0.0/0 | HTTP (redirect to HTTPS) |
| 443 | 0.0.0.0/0 | HTTPS (nginx) |
| 2375 | Gitea runner IP only | Docker remote API (TLS) |
| 9090 | Admin IPs or internal | Prometheus (or tunnel via SSH) |
| 3001 | Admin IPs or internal | Grafana (or tunnel via SSH) |

### Pipeline Readiness
| Workflow | Trigger | Purpose |
|---|---|---|
| `ci.yml` | Push to `main` | Lint, build frontend, vet Go, build & push images (`:latest`, `:main`) |
| `cd.yml` | Push to `main` | Deploy to test lab (`192.168.88.40:2375`) — runs **in parallel** with CI |
| `release.yml` | Tag push `*.*.*` | Build & push versioned images (`:X.Y.Z`, `:X.Y`, `:latest`), create Gitea Release |
| `workers-deploy.yml` | After CI succeeds | Deploy frontend to Cloudflare Workers |

CI and CD run simultaneously on every push to `main`. Deploy pulls whatever `:latest` images are available at that moment. `cd.yml` uses `DOCKER_HOST` to target the host:

```bash
# Test lab (current)
DOCKER_HOST=tcp://192.168.88.40:2375

# Production (when provisioned)
DOCKER_HOST=tcp://<hetzner-public-ip>:2375
```

No pipeline code changes needed to switch targets.

## Target Topology
- Host: Linux server with Docker Engine
- Reverse proxy: nginx container (linearvoid-nginx)
- Core services: playout, chat, redis, postgres
- Observability: prometheus, grafana
- CDN/edge: Cloudflare

## Prerequisites
- Docker Engine + Docker Compose plugin
- Network access to git.allucanget.biz
- Valid `.env` file with production values
- DNS configured for `linearvoid.tv` (and `media.linearvoid.tv` if split media domain used)

## Environment File

`.env` requires at minimum:
- `DATABASE_URL` — PostgreSQL connection string
- `REDIS_URL` — Redis connection string
- `ADMIN_USERNAME` / `ADMIN_PASSWORD` — admin dashboard credentials
- `ADMIN_SESSION_TTL_HOURS` — session expiry (default 12)
- `VITE_PUBLIC_APP_URL` / `VITE_PUBLIC_WS_URL` / `VITE_PUBLIC_HLS_URL` — frontend URLs

**Cloudflare R2 (required for HLS segment sync to edge):**
- `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY`
- `R2_BUCKET_NAME` (e.g. `linearvoid-media`)
- `R2_ENDPOINT` — S3-compatible endpoint URL
- `R2_REGION` — set to `auto`

**Cloudflare (optional, for Workers):**
- `CLOUDFLARE_API_TOKEN` / `CLOUDFLARE_ACCOUNT_ID`
- `MEDIA_DOMAIN` — split media domain (e.g. `media.linearvoid.tv`)

See `.env.example` and `SETUP.md` for full variable reference.

### Gitea Secrets
`cd.yml` injects sensitive values from Gitea secrets into `.env` at deploy time. Configure in the Gitea repository:

`DATABASE_URL`, `ADMIN_USERNAME`, `ADMIN_PASSWORD`, `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_BUCKET_NAME`, `R2_ENDPOINT`, `R2_ADMIN_ACCESS_KEY_ID` (optional), `R2_ADMIN_SECRET_ACCESS_KEY` (optional), `R2_ADMIN_ENDPOINT` (optional), `CLOUDFLARE_API_TOKEN` (optional), `CLOUDFLARE_ACCOUNT_ID` (optional), `CI_REGISTRY_USER`, `CI_REGISTRY_TOKEN`

## Pull-and-Run Deployment

Images are built/pushed automatically by CI and release workflows. Every push to `main` triggers CI (`:main`, `:latest`) and CD (deploys `:latest` to test lab) in parallel. Semantic version tags (`*.*.*`, e.g. `1.0.9`) trigger the release workflow (`:X.Y.Z`, `:X.Y`, `:latest` + Gitea Release).

```bash
docker login git.allucanget.biz
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d --remove-orphans
```

## First-Time or Recovery Provisioning
```bash
mkdir -p data/chat data/playlist data/media data/hls
docker volume ls
docker network ls
docker compose -f docker-compose.prod.yml up -d
```

## Health Verification
```bash
docker compose -f docker-compose.prod.yml ps
```
Endpoint checks:
- App: `http://{host}:8080/`
- Nginx health: `http://{host}:8080/health`
- Chat health: `http://{host}:3000/health`
- Playout health: `http://{host}:8081/health`
- Prometheus: `http://{host}:9090`
- Grafana: `http://{host}:3001`

Domain checks:
- `https://linearvoid.tv`
- `https://linearvoid.tv/admin`
- `https://linearvoid.tv/metrics`

## Zero/Low-Downtime Update Pattern
1. Pull newest images.
2. Recreate changed services only:
```bash
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d nginx chat playout
```
3. Re-run health checks.

## Rollback
If new images are unhealthy:
1. Pin previous tags in `docker-compose.prod.yml`.
2. Pull pinned tags.
3. Restart impacted services:
```bash
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d nginx chat playout
```

## Operational Tasks
```bash
# View logs
docker compose -f docker-compose.prod.yml logs -f nginx chat playout

# Restart one service
docker compose -f docker-compose.prod.yml restart chat

# Stop stack (keep data)
docker compose -f docker-compose.prod.yml down

# Stop stack — destructive (remove data)
docker compose -f docker-compose.prod.yml down -v
```

## Nginx and Proxy Considerations
- If an external proxy (e.g. Nginx Proxy Manager) fronts the stack, align request body limits with internal nginx (see `infrastructure/nginx/nginx.conf`).
- For resumable admin uploads, ensure `client_max_body_size` supports chunk requests in both proxy layers.

## Security Baseline
- Use a strong `ADMIN_PASSWORD` and rotate regularly.
- Keep Docker host access restricted.
- Prefer HTTPS/WSS through Cloudflare.
- Restrict exposed ports if only proxy access is needed.

## Common Deployment Pitfalls
- Missing frontend build artifacts in local-only compose workflows.
- Wrong registry credentials or missing login.
- Misaligned domain env vars (app/ws/hls) causing mixed-origin errors.

## Troubleshooting
| Symptom | Check | Fix |
|---|---|---|
| Service not starting | `docker logs linearvoid-<svc> --tail 50` | Check `.env` vars, volume mounts, port conflicts |
| Playout stall / slate mode | `curl http://<host>:8081/health` — mode=slate | Check playlist file, media paths, disk space |
| Chat not connecting | `curl http://<host>:3000/health` | Check Redis: `docker exec linearvoid-redis redis-cli ping` |
| Admin 401 loop | Cookie not sent | Verify `X-Forwarded-Proto: https`, nginx auth_request headers |
| Upload fails (413) | File too large | Check nginx `client_max_body_size` (should be 64m+) |
| HLS not playing | `curl http://<host>/hls/playlist.m3u8` | Check playout health, HLS dir permissions |
| Prometheus no targets | `curl http://<host>:9090/targets` | Verify container network, scrape config |
| Grafana no data | Check datasource | Verify Prometheus URL `http://prometheus:9090` provisioned |

Additional notes: proxy-layer body size limits can return 413 on upload; health checks may fail before service warmup completes.