linearvoid/DEPLOYMENT.md

# Deployment Guide

## Environments

Linear Void runs in two separate environments:

| Environment    | Host                       | Docker Target              | Purpose                                        |
| -------------- | -------------------------- | -------------------------- | ---------------------------------------------- |
| **Test Lab**   | `192.168.88.40` (internal) | `tcp://192.168.88.40:2375` | Integration testing, staging, CI/CD validation |
| **Production** | Hetzner VPS (TBD)          | `tcp://<hetzner-ip>:2375`  | Public linearvoid.tv serving real viewers      |

**Current state (2026-08-06):** Test lab is active and receives automated CD deploys. Production Hetzner VPS is **not yet provisioned**. The CD workflow (`cd.yml`) and `docker-compose.prod.yml` currently target the test lab. Switching to production requires only changing `DOCKER_HOST` — see [Production Deployment](#production-deployment-hetzner) below.

## Production Target (Hetzner)

**Status: Not provisioned.** The production deployment plan is documented and the pipeline is ready — only the VPS needs to be created.

### Planned Infrastructure

| Resource       | Spec                                             |
| -------------- | ------------------------------------------------ |
| Provider       | Hetzner Cloud                                    |
| Region         | Ashburn, VA (us-east)                            |
| Instance       | CX22 (2 vCPU, 4 GB RAM, 40 GB SSD) or equivalent |
| OS             | Ubuntu 24.04 LTS or Debian 12                    |
| Bandwidth      | 20 TB included traffic/month                     |
| Estimated cost | ~€7–10/month base                                |

### Cutover Plan

1. Provision Hetzner VPS (see `TODO.md` #92)
2. Add production secrets to Gitea (`CI_PROD_*` prefix)
3. Update `DOCKER_HOST` in CD workflow to Hetzner public IP
4. Trigger deploy → validate health checks
5. Update Cloudflare A record `linearvoid.tv` → Hetzner IP
6. Enable HTTPS, WAF, cache rules on production zone
7. Decommission test lab as public target

### Firewall / Security Group

| Port | Source                | Purpose                        |
| ---- | --------------------- | ------------------------------ |
| 22   | Admin IPs             | SSH management                 |
| 80   | 0.0.0.0/0             | HTTP (redirect to HTTPS)       |
| 443  | 0.0.0.0/0             | HTTPS (nginx)                  |
| 2375 | Gitea runner IP only  | Docker remote API (TLS)        |
| 9090 | Admin IPs or internal | Prometheus (or tunnel via SSH) |
| 3001 | Admin IPs or internal | Grafana (or tunnel via SSH)    |

### Pipeline Readiness

The Gitea Actions CD workflow (`cd.yml`) already supports `DOCKER_HOST` as a configurable variable. To switch targets:

```bash
# Test lab (current)
DOCKER_HOST=tcp://192.168.88.40:2375

# Production (when provisioned)
DOCKER_HOST=tcp://<hetzner-public-ip>:2375
```

No code changes needed in the pipeline. See `TODO.md` #91 and #92–#99 for full production deployment checklist.

## Target Topology

- Host: Linux server with Docker Engine
- Reverse proxy: nginx container (linearvoid-nginx)
- Core services: playout, chat, redis, postgres
- Observability: prometheus, grafana
- CDN/edge: Cloudflare

## Prerequisites

- Docker Engine + Docker Compose plugin
- Network access to git.allucanget.biz
- A valid .env file with production values
- DNS configured for linearvoid.tv (and media.linearvoid.tv if split media domain is used)

## Environment File

Use .env with at least:

- CI_REGISTRY_URL
- CI_REGISTRY_NAMESPACE
- CHAT_PORT
- NGINX_PORT
- DATABASE_URL (if overridden)
- REDIS_URL (if overridden)
- ADMIN_USERNAME
- ADMIN_PASSWORD
- VITE_PUBLIC_APP_URL
- VITE_PUBLIC_WS_URL
- VITE_PUBLIC_HLS_URL
- Cloudflare/R2 variables when media sync is enabled

See .env.example and SETUP.md for full variable reference.

## Pull-and-Run Deployment

Images are built and pushed automatically by the CI and release workflows (see [CONTRIBUTING.md](CONTRIBUTING.md#versioning-and-releases) for the release procedure).

Every push to `main` produces `:main` and `:latest` tags. Semantic version tags (`v*`) produce `:X.Y.Z`, `:X.Y`, and `:latest` tags.

From repository root:

```bash
docker login git.allucanget.biz

docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d --remove-orphans
```

## First-Time or Recovery Provisioning

1. Ensure host paths exist:

```bash
mkdir -p data/chat data/playlist data/media data/hls
```

1. Confirm volumes and networks:

```bash
docker volume ls
docker network ls
```

1. Start stack:

```bash
docker compose -f docker-compose.prod.yml up -d
```

## Health Verification

After startup, check service status:

```bash
docker compose -f docker-compose.prod.yml ps
```

Endpoint checks:

- App: <http://{host}:8080/>
- Nginx health: <http://{host}:8080/health>
- Chat health: <http://{host}:3000/health>
- Playout health: <http://{host}:8081/health>
- Prometheus: <http://{host}:9090>
- Grafana: <http://{host}:3001>

Domain checks:

- <https://linearvoid.tv>
- <https://linearvoid.tv/admin>
- <https://linearvoid.tv/metrics>

## Zero/Low-Downtime Update Pattern

1. Pull newest images.
1. Recreate changed services only:

```bash
docker compose -f docker-compose.prod.yml pull

docker compose -f docker-compose.prod.yml up -d nginx chat playout
```

1. Re-run health checks.

## Rollback

If new images are unhealthy:

1. Pin previous tags in docker-compose.prod.yml.
2. Pull pinned tags.
3. Restart impacted services.

```bash
docker compose -f docker-compose.prod.yml pull

docker compose -f docker-compose.prod.yml up -d nginx chat playout
```

## Operational Tasks

### View logs

```bash
docker compose -f docker-compose.prod.yml logs -f nginx chat playout
```

### Restart one service

```bash
docker compose -f docker-compose.prod.yml restart chat
```

### Stop stack

```bash
docker compose -f docker-compose.prod.yml down
```

### Keep volumes (recommended) vs remove volumes (destructive)

```bash
# keep data
docker compose -f docker-compose.prod.yml down

# destructive (remove data)
docker compose -f docker-compose.prod.yml down -v
```

## Nginx and Proxy Considerations

- If an external proxy (for example Nginx Proxy Manager) fronts the stack, align request body limits with internal nginx.
- For resumable admin uploads, ensure client_max_body_size supports chunk requests in both proxy layers.

## Security Baseline

- Use strong ADMIN_PASSWORD and rotate regularly.
- Keep Docker host access restricted.
- Prefer HTTPS/WSS through Cloudflare.
- Restrict exposed ports if only proxy access is needed.

## Common Deployment Pitfalls

- Missing frontend build artifacts in local-only compose workflows.
- Wrong registry credentials or missing login.
- Misaligned domain env vars (app/ws/hls) causing mixed-origin errors.

## Troubleshooting

| Symptom                    | Check                                         | Fix                                                           |
| -------------------------- | --------------------------------------------- | ------------------------------------------------------------- |
| Service not starting       | `docker logs linearvoid-<svc> --tail 50`      | Check `.env` vars, volume mounts, port conflicts              |
| Playout stall / slate mode | `curl http://<host>:8081/health` — mode=slate | Check playlist file, media paths, disk space                  |
| Chat not connecting        | `curl http://<host>:3000/health`              | Check Redis: `docker exec linearvoid-redis redis-cli ping`    |
| Admin 401 loop             | Cookie not sent                               | Verify `X-Forwarded-Proto: https`, nginx auth_request headers |
| Upload fails (413)         | File too large                                | Check nginx `client_max_body_size` (should be 64m+)           |
| HLS not playing            | `curl http://<host>/hls/playlist.m3u8`        | Check playout health, HLS dir permissions                     |
| Prometheus no targets      | `curl http://<host>:9090/targets`             | Verify container network, scrape config                       |
| Grafana no data            | Check datasource                              | Verify Prometheus URL `http://prometheus:9090` provisioned    |

- Proxy-layer body size limits returning 413 on upload.
- Health checks failing before service warmup is complete.