diff --git a/README.md b/README.md index e1c3068..26627fc 100644 --- a/README.md +++ b/README.md @@ -227,7 +227,7 @@ Contributions are welcome! See [`CONTRIBUTING.md`](CONTRIBUTING.md) for guidelin 3. Open PRs on [Gitea](https://git.allucanget.biz/zwitschi/linearvoid). 4. CI validates builds and tests automatically. -Since this a linear project, we also rely on a linear development and release model: Only main branch is used for releases, and all features are merged into main after review and CI validation. Releases are tagged with semantic versioning (1.0.3, 1.0.4, etc.) and automatically built/pushed to the container registry. Version numbers are declared in `frontend/package.json`, `CHANGELOG.md`, and the version badge in this README. For patches, we use even numbers for stable versions (1.0.2, 1.0.4) and odd numbers for pre-release/unstable versions (1.0.3, 1.0.5). +Since this is a linear project, we also rely on a linear development and release model: only `main` branch is used for releases, and all features are merged into `main` after review and CI validation. Releases use semantic version tags in `*.*.*` format (for example `1.0.9`) and are automatically built/pushed to the container registry. Version numbers are declared in `frontend/package.json`, `CHANGELOG.md`, and the version badge in this README. [MIT License](LICENSE) diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 46291ef..83d6dff 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -394,7 +394,7 @@ Notable aspects: Manifest freshness ensures all viewers stay on same timeline; c ### CI/CD Pipeline Architecture -The deployment follows a two-stage pipeline: **CI builds and pushes images**, then **CD pulls and deploys** them. Currently targeting the **test lab** (192.168.88.40). Production Hetzner VPS is planned but not yet provisioned — the same pipeline will deploy there by changing `DOCKER_HOST`. +The deployment uses parallel workflows on each push to `main`: **CI builds and pushes images** while **CD deploys latest images** to the test lab. A dedicated release workflow handles semantic version tags. Current runtime target is the **test lab** (192.168.88.40). Production Hetzner VPS is planned but not yet provisioned — the same pipeline will deploy there by changing `DOCKER_HOST`. ```mermaid flowchart LR @@ -431,18 +431,22 @@ flowchart LR **Pipeline Sequence:** 1. **CI Workflow** (`.gitea/workflows/ci.yml`): - - Triggers on `push` to `main` or `pull_request` - - Runs `test` job: frontend lint/build, Go validation, `docker compose build` + - Triggers on `push` to `main` and `pull_request` + - Runs `test` job: frontend lint/build and Go validation - Runs `docker` job (after `test` passes, only on `main` push): - Builds `playout`, `chat`, `nginx` images with Buildx - Pushes to Gitea Container Registry with `main` and `latest` tags 2. **CD Workflow** (`.gitea/workflows/cd.yml`): - - Triggers on `workflow_run` when CI completes successfully, or manually via `workflow_dispatch` - - Pulls latest images from registry + - Triggers on `push` to `main` (parallel to CI), or manually via `workflow_dispatch` + - Logs in remote Docker host to private registry - Generates `.env` from secrets - - Seeds Prometheus config into named volume - - Deploys via `docker compose -f docker-compose.prod.yml up -d` to remote host (`192.168.88.40:2375` — test lab; switch to Hetzner IP for production) + - Seeds Prometheus and Grafana provisioning into named volumes + - Deploys via `docker compose -f docker-compose.prod.yml up -d --remove-orphans --force-recreate` to remote host (`192.168.88.40:2375` — test lab; switch to Hetzner IP for production) - Runs health checks with retry logic for playout initialization +3. **Release Workflow** (`.gitea/workflows/release.yml`): + - Triggers on semantic tags in `*.*.*` format (for example `1.0.9`) + - Builds and pushes versioned image tags (`X.Y.Z`, `X.Y`, `latest`) + - Creates release entry in Gitea ### Infrastructure Level 1 @@ -524,14 +528,14 @@ Linear Void uses a two-layer monitoring model: - Config file: `monitoring/prometheus/prometheus.yml` - Default scrape interval: `15s` -- Current scrape jobs: `chat` -> `chat:3000`, `prometheus` -> `prometheus:9090`. - -Recommended next-step scrape targets (if/when exporters are introduced): - -- `nginx` (via nginx-exporter) -- `redis` (via redis-exporter) -- `postgres` (via postgres-exporter) -- `node` (via node-exporter) +- Current scrape jobs (7 total): + - `chat` -> `chat:3000` + - `playout` -> `playout:8081` + - `nginx` -> `nginx-exporter:9113` + - `redis` -> `redis-exporter:9121` + - `postgres` -> `postgres-exporter:9187` + - `node` -> `node-exporter:9100` + - `prometheus` -> `localhost:9090` #### Grafana Dashboards @@ -542,22 +546,20 @@ Use the app dashboard for service-level behavior and the Prometheus overview das #### Alerting Rules (Prometheus Rule Groups) -Define alert rules as grouped policies (for example in `monitoring/prometheus/alerts.yml`) and include them from `prometheus.yml`. +Rules are implemented in `monitoring/prometheus/alerts.yml` and loaded via `rule_files` from `prometheus.yml`. -Minimum recommended alert set: +Current rule set (8): -- **Availability** - `up{job="chat"} == 0` for `2m`; `up{job="prometheus"} == 0` for `2m`. -- **Chat service health** - `linearvoid_chat_redis_up == 0` for `2m`; sudden disconnect spike / connected clients collapse. -- **Playout continuity** - No HLS segment progression for `30-60s` (from playout-specific metric when available). -- **Prometheus capacity and ingestion health** - Sustained growth in `prometheus_tsdb_wal_writes_failed_total` or `prometheus_tsdb_compactions_failed_total`, abnormal jump in `prometheus_tsdb_out_of_order_samples_total`, and high `prometheus_engine_queries` near concurrent limit. -- **Latency and user-impact** - Elevated p99 handler latency from `prometheus_http_request_duration_seconds_bucket` and elevated p99 query execution latency from `prometheus_engine_query_duration_histogram_seconds_bucket`. +- `ChatDown` +- `PlayoutDown` +- `RedisDown` +- `PlayoutSlateActive` +- `HighChatLatency` +- `PrometheusWALFailing` +- `HighCPU` +- `LowDiskSpace` -Route warning/critical severities to the team channel and keep runbook links attached to each alert definition. +Route warning/critical severities to the operations channel and keep runbook links attached to each alert definition. Operational resilience baseline: