linearvoid/docs/DEPLOYMENT.md

# Deployment Guide

## Environments

| Environment    | Host                       | Docker Target              | Purpose                                        |
| -------------- | -------------------------- | -------------------------- | ---------------------------------------------- |
| **Test Lab**   | `192.168.88.40` (internal) | `tcp://192.168.88.40:2375` | Integration testing, staging, CI/CD validation |
| **Production** | Hetzner VPS (TBD)          | `tcp://<hetzner-ip>:2375`  | Public linearvoid.tv                           |

**Current state (2026-08-06):** Test lab is active and receives automated CD deploys. Production Hetzner VPS is **not yet provisioned**. `cd.yml` and `docker-compose.prod.yml` target the test lab; switching to production only requires changing `DOCKER_HOST`.

## Production Target (Hetzner)

**Status: Not provisioned.** Plan documented and pipeline ready; only the VPS needs creation.

### Planned Infrastructure

- Provider: Hetzner Cloud, Region: Ashburn, VA (us-east)
- Instance: CX22 (2 vCPU, 4 GB RAM, 40 GB SSD) or equivalent
- OS: Ubuntu 24.04 LTS or Debian 12
- Bandwidth: 20 TB included traffic/month; Estimated cost: ~€7–10/month base

### Cutover Plan

1. Provision Hetzner VPS
2. Add production secrets to Gitea (`CI_PROD_*` prefix)
3. Update `DOCKER_HOST` in CD workflow to Hetzner public IP
4. Trigger deploy → validate health checks
5. Update Cloudflare A record `linearvoid.tv` → Hetzner IP
6. Enable HTTPS, WAF, cache rules on production zone
7. Decommission test lab as public target

### Production Preflight (minimum readiness gate)

1. Verify Hetzner host reachable over SSH and Docker remote API TLS works for the Gitea runner
2. Confirm all production secrets exist under `CI_PROD_*` prefix
3. Confirm `linearvoid.tv`, `www.linearvoid.tv`, `media.linearvoid.tv` DNS targets match intended phase
4. Complete manual Cloudflare/UptimeRobot checklist (see `docs/operations/cloudflare-uptimerobot-checklist.md`)
5. Run production smoke path end-to-end: viewer SPA, HLS playback, chat connect, admin login, upload, playlist save, playout health
6. Verify rollback possible before final DNS switch

### Firewall / Security Group

| Port | Source                | Purpose                        |
| ---- | --------------------- | ------------------------------ |
| 22   | Admin IPs             | SSH management                 |
| 80   | 0.0.0.0/0             | HTTP (redirect to HTTPS)       |
| 443  | 0.0.0.0/0             | HTTPS (nginx)                  |
| 2375 | Gitea runner IP only  | Docker remote API (TLS)        |
| 9090 | Admin IPs or internal | Prometheus (or tunnel via SSH) |
| 3001 | Admin IPs or internal | Grafana (or tunnel via SSH)    |

### Pipeline Readiness

| Workflow             | Trigger           | Purpose                                                                           |
| -------------------- | ----------------- | --------------------------------------------------------------------------------- |
| `ci.yml`             | Push to `main`    | Lint, build frontend, vet Go, build & push images (`:latest`, `:main`)            |
| `cd.yml`             | Push to `main`    | Deploy to test lab (`192.168.88.40:2375`) — runs **in parallel** with CI          |
| `release.yml`        | Tag push `*.*.*`  | Build & push versioned images (`:X.Y.Z`, `:X.Y`, `:latest`), create Gitea Release |
| `workers-deploy.yml` | After CI succeeds | Deploy frontend to Cloudflare Workers                                             |

CI and CD run simultaneously on every push to `main`. Deploy pulls whatever `:latest` images are available at that moment. `cd.yml` uses `DOCKER_HOST` to target the host:

```bash
# Test lab (current)
DOCKER_HOST=tcp://192.168.88.40:2375

# Production (when provisioned)
DOCKER_HOST=tcp://<hetzner-public-ip>:2375
```

No pipeline code changes needed to switch targets.

## Target Topology

- Host: Linux server with Docker Engine
- Reverse proxy: nginx container (linearvoid-nginx)
- Core services: playout, chat, redis, postgres
- Observability: prometheus, grafana
- CDN/edge: Cloudflare

## Prerequisites

- Docker Engine + Docker Compose plugin
- Network access to git.allucanget.biz
- Valid `.env` file with production values
- DNS configured for `linearvoid.tv` (and `media.linearvoid.tv` if split media domain used)

## Environment File

`.env` requires at minimum:

- `DATABASE_URL` — PostgreSQL connection string
- `REDIS_URL` — Redis connection string
- `ADMIN_USERNAME` / `ADMIN_PASSWORD` — admin dashboard credentials
- `ADMIN_SESSION_TTL_HOURS` — session expiry (default 12)
- `VITE_PUBLIC_APP_URL` / `VITE_PUBLIC_WS_URL` / `VITE_PUBLIC_HLS_URL` — frontend URLs

**Cloudflare R2 (required for HLS segment sync to edge):**

- `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY`
- `R2_BUCKET_NAME` (e.g. `linearvoid-media`)
- `R2_ENDPOINT` — S3-compatible endpoint URL
- `R2_REGION` — set to `auto`

**Cloudflare (optional, for Workers):**

- `CLOUDFLARE_API_TOKEN` / `CLOUDFLARE_ACCOUNT_ID`
- `MEDIA_DOMAIN` — split media domain (e.g. `media.linearvoid.tv`)

See `.env.example` and `SETUP.md` for full variable reference.

### Gitea Secrets

`cd.yml` injects sensitive values from Gitea secrets into `.env` at deploy time. Configure in the Gitea repository:

`DATABASE_URL`, `ADMIN_USERNAME`, `ADMIN_PASSWORD`, `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_BUCKET_NAME`, `R2_ENDPOINT`, `R2_ADMIN_ACCESS_KEY_ID` (optional), `R2_ADMIN_SECRET_ACCESS_KEY` (optional), `R2_ADMIN_ENDPOINT` (optional), `CLOUDFLARE_API_TOKEN` (optional), `CLOUDFLARE_ACCOUNT_ID` (optional), `CI_REGISTRY_USER`, `CI_REGISTRY_TOKEN`

## Pull-and-Run Deployment

Images are built/pushed automatically by CI and release workflows. Every push to `main` triggers CI (`:main`, `:latest`) and CD (deploys `:latest` to test lab) in parallel. Semantic version tags (`*.*.*`, e.g. `1.0.9`) trigger the release workflow (`:X.Y.Z`, `:X.Y`, `:latest` + Gitea Release).

```bash
docker login git.allucanget.biz
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d --remove-orphans
```

## First-Time or Recovery Provisioning

```bash
mkdir -p data/chat data/playlist data/media data/hls
docker volume ls
docker network ls
docker compose -f docker-compose.prod.yml up -d
```

## Health Verification

```bash
docker compose -f docker-compose.prod.yml ps
```

Endpoint checks:

- App: `http://{host}:8080/`
- Nginx health: `http://{host}:8080/health`
- Chat health: `http://{host}:3000/health`
- Playout health: `http://{host}:8081/health`
- Prometheus: `http://{host}:9090`
- Grafana: `http://{host}:3001`

Domain checks:

- `https://linearvoid.tv`
- `https://linearvoid.tv/admin`
- `https://linearvoid.tv/metrics`

## Zero/Low-Downtime Update Pattern

1. Pull newest images.
2. Recreate changed services only:

   ```bash
   docker compose -f docker-compose.prod.yml pull
   docker compose -f docker-compose.prod.yml up -d nginx chat playout
   ```

3. Re-run health checks.

## Rollback

If new images are unhealthy:

1. Pin previous tags in `docker-compose.prod.yml`.
2. Pull pinned tags.
3. Restart impacted services:

```bash
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d nginx chat playout
```

## Operational Tasks

```bash
# View logs
docker compose -f docker-compose.prod.yml logs -f nginx chat playout

# Restart one service
docker compose -f docker-compose.prod.yml restart chat

# Stop stack (keep data)
docker compose -f docker-compose.prod.yml down

# Stop stack — destructive (remove data)
docker compose -f docker-compose.prod.yml down -v
```

## Nginx and Proxy Considerations

- If an external proxy (e.g. Nginx Proxy Manager) fronts the stack, align request body limits with internal nginx (see `infrastructure/nginx/nginx.conf`).
- For resumable admin uploads, ensure `client_max_body_size` supports chunk requests in both proxy layers.

## Security Baseline

- Use a strong `ADMIN_PASSWORD` and rotate regularly.
- Keep Docker host access restricted.
- Prefer HTTPS/WSS through Cloudflare.
- Restrict exposed ports if only proxy access is needed.

## Common Deployment Pitfalls

- Missing frontend build artifacts in local-only compose workflows.
- Wrong registry credentials or missing login.
- Misaligned domain env vars (app/ws/hls) causing mixed-origin errors.

## Troubleshooting

| Symptom                    | Check                                         | Fix                                                           |
| -------------------------- | --------------------------------------------- | ------------------------------------------------------------- |
| Service not starting       | `docker logs linearvoid-<svc> --tail 50`      | Check `.env` vars, volume mounts, port conflicts              |
| Playout stall / slate mode | `curl http://<host>:8081/health` — mode=slate | Check playlist file, media paths, disk space                  |
| Chat not connecting        | `curl http://<host>:3000/health`              | Check Redis: `docker exec linearvoid-redis redis-cli ping`    |
| Admin 401 loop             | Cookie not sent                               | Verify `X-Forwarded-Proto: https`, nginx auth_request headers |
| Upload fails (413)         | File too large                                | Check nginx `client_max_body_size` (should be 64m+)           |
| HLS not playing            | `curl http://<host>/hls/playlist.m3u8`        | Check playout health, HLS dir permissions                     |
| Prometheus no targets      | `curl http://<host>:9090/targets`             | Verify container network, scrape config                       |
| Grafana no data            | Check datasource                              | Verify Prometheus URL `http://prometheus:9090` provisioned    |

Additional notes: proxy-layer body size limits can return 413 on upload; health checks may fail before service warmup completes.