zwitschi/linearvoid
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 9 Wiki Activity
Files
1019972e2501e7309ab89b692a56b1c4f62922b9
linearvoid/README.md
T

342 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Linear Void
[Linear Void](https://linearvoid.tv) is a self-managed **24/7 linear video channel** for a primarily US audience. It runs a continuous playlist of self-owned videos with true/near-true synchronization, a custom lightweight viewer SPA (hls.js player + chat), and full independence from third-party platforms (YouTube, Twitch, etc.).
**Repository**: [git.allucanget.biz/zwitschi/linearvoid](https://git.allucanget.biz/zwitschi/linearvoid)
[![Build Status](https://git.allucanget.biz/zwitschi/linearvoid/actions/workflows/ci.yml/badge.svg)](https://git.allucanget.biz/zwitschi/linearvoid/actions)
[![License](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Gitea](https://img.shields.io/badge/Gitea-1.27.1-blue)](https://git.allucanget.biz)
[![Version](https://img.shields.io/badge/version-1.0.11-blue)](https://git.allucanget.biz/zwitschi/linearvoid/releases)
[![Release Notes](https://img.shields.io/badge/release-notes-green)](CHANGELOG.md)
## Architecture
The architecture (detailed in [`docs/architecture/README.md`](docs/architecture/README.md)) uses:
- Hetzner Ashburn VPS for playout (FFmpeg + MediaMTX/SRS) and chat _(planned production; currently test lab at 192.168.88.40)_.
- Cloudflare R2 + CDN with aggressive immutable caching of `.m4s` segments ($0 egress).
- Lightweight static frontend and WebSocket chat backend.
- Gitea self-hosted CI/CD (Actions) and Container Registry.
See [`docs/architecture/adr/`](docs/architecture/adr/) for the full set of decision records (ADR-001 through ADR-007) covering architecture, storage calculations, cache rules, cost model, module structure, tech stack, admin interfaces, data storage, and caching strategy.
## Key Features
- Continuous synchronized playback from a playlist (standard HLS, 15–35s latency).
- Self-hosted custom SPA with embedded chat (public rooms + private 1:1 DMs, moderation).
- Extreme cost efficiency via Cloudflare R2 edge caching.
- Video normalization, monitoring, fallback slate, and easy playlist management.
- US-centric performance (Ashburn origin + Cloudflare POPs).
## Architecture Diagram
```mermaid
flowchart TD
V[US Viewers] -->|HTTPS| CF[Cloudflare CDN]
CF -->|Cache HIT| R2[Cloudflare R2]
CF -->|Cache MISS| N[Nginx :80]
N -->|/ws| CH[Chat :3000]
N -->|/admin| CH
N -->|/hls| P[Playout :8081]
N -->|/*| FE[Frontend SPA]
CH --> R[(Redis)]
CH --> DB[(PostgreSQL)]
P -->|ffmpeg| HLS[/data/hls/]
P -->|sync| R2
```
## Project Structure
This structure directly reflects the building blocks, runtime flows, deployment view, and cross-cutting concepts defined in the arc42 documentation.
```text
linearvoid/
├── docs/ # see Documentation section below
├── backend/ # see Backend section below
├── frontend/ # see Frontend section below
├── infrastructure/ # see Infrastructure section below
├── monitoring/ # see Monitoring section below
├── scripts/ # see Scripts section below
├── .editorconfig # EditorConfig for consistent coding styles
├── .env.example # Example environment variables for local development
├── .gitignore # Ignore node_modules, build artifacts, etc.
├── .prettierignore # Ignore files for Prettier formatting
├── CHANGELOG.md # Release notes and version history
├── CONTRIBUTING.md # Contribution guidelines
├── docker-compose.prod.yml # Production deployment (Hetzner + Cloudflare)
├── docker-compose.yml # Top-level for dev (playout + chat + nginx)
└── README.md # This file
```
### Documentation
- `docs/`: Folder containing all documentation for the project, including architecture, ADRs, operational procedures, and setup guides.
- `docs/architecture/`: Architecture documentation, including high-level system architecture, component interactions, and decision records (ADRs).
see [README.md](docs/README.md) for full architecture, ADRs, and operational documentation.
### Backend
- `backend/playout/`: Core linear engine. FFmpeg consumes a normalized playlist and outputs continuous HLS. Includes normalization validation.
- `backend/chat/`: Independent WebSocket service (Node.js/Socket.io or Go). Handles public rooms, private DMs, presence (Redis), persistence (PostgreSQL/SQLite), rate limiting, and moderation.
- `backend/shared/`: Reusable code (types, utils, video validation).
```text
backend/
├── chat/ # WebSocket chat service (Node.js or Go)
│ ├── src/ # Chat service source code (rooms, DMs, moderation)
| | ├── auth/ # Admin authentication and session management
| | | ├── auth_test.go # Unit tests for authentication
| | | └── auth.go # Authentication logic (login, session validation)
| | ├── config/ # Configuration files for chat service
| | | ├── config_test.go # Unit tests for configuration
| | | └── config.go # Configuration loading and validation
| | ├── db/ # Database access and models
| | | ├── db_test.go # Unit tests for database access
| | | ├── db.go # Database connection and query logic
| | | ├── models.go # Database models (users, messages, rooms)
| | | └── redis.go # Redis connection and presence management
| | ├── handlers/ # WebSocket message handlers
| | | ├── handlers_test.go # Unit tests for WebSocket message handlers
| | | └── handlers.go # WebSocket message handling logic
| | ├── media/ # Media-related utilities (e.g., video validation)
| | | ├── media_test.go # Unit tests for media-related utilities
| | | └── media.go # Media-related utility functions
| | ├── metrics/ # Prometheus metrics endpoints
| | | ├── metrics_test.go # Unit tests for Prometheus metrics
| | | └── metrics.go # Prometheus metrics collection and exposition
| | ├── ratelimit/ # Rate limiting logic (Redis-based)
| | | ├── ratelimit_test.go # Unit tests for rate limiting
| | | └── ratelimit.go # Rate limiting logic using Redis
| | ├── ws/ # WebSocket handlers
| | | ├── ws_test.go # Unit tests for WebSocket handlers
| | | └── ws.go # WebSocket connection and message handling
| | └── main.go # Entry point for chat service
| ├── Dockerfile # Dockerfile for chat service
| ├── go.mod # Go module file for chat service
| └── go.sum # Go checksum file for chat service
└── playout/ # Continuous HLS generation from playlist (Go)
├── demodata/ # Sample media and playlist files for testing
| ├── media/ # Sample media files for testing
| └── playlist/ # Sample playlist files for testing
├── src/ # Playout service source code (FFmpeg orchestration)
| ├── go.mod # Go module file for playout service
| ├── go.sum # Go checksum file for playout service
| ├── playlist_test.go # Unit tests for playlist management
| └── playlist.go # Playlist management logic
└── testdata/ # Test data for playout service (sample playlists, media)
| ├── comments_only.txt # Sample playlist with comments only
| ├── empty_playlist.txt # Sample empty playlist
| ├── invalid_24fps.mp4 # Sample invalid 24fps video for testing
| ├── invalid_h265.mp4 # Sample invalid H.265 video for testing
| ├── mixed_valid_invalid.txt # Sample playlist with mixed valid and invalid entries
| ├── valid_720p30.mp4 # Sample valid 720p30 video for testing
| └── valid_playlist.txt # Sample valid playlist for testing
├── Dockerfile # Dockerfile for the playout service
└── start-playout.sh # Script to start the playout service
```
### Frontend
The viewer experience — minimal, modern SPA with custom CSS + vanilla JS. Uses `hls.js` for playback and a WebSocket client for chat.
```text
frontend/
├── admin/ # Admin SPA (playlist management, moderation, analytics)
│ ├── login/ # Login page for admin authentication
| | └── index.html # Admin login page
| └── index.html # Admin SPA
├── public/ # Static assets (favicon, images, etc.)
│ ├── admin/ # Admin static assets
| | ├── analytics.html # Analytics dashboard page
| | ├── chat-log.html # Chat log viewer page
| | ├── dashboard.html # Admin dashboard page
| | ├── moderation.html # User moderation page
| | ├── playlist.html # Playlist management page
| | ├── schedule.html # Schedule overrides page
| | ├── system.html # System configuration page
| | ├── upload.html # Video upload page
| | └── users.html # User management page
│ ├── favicon.ico # Favicon for the SPA
| └── icons.svg # SVG icons for the SPA
├── src/ # Source code for the SPA
│ ├── assets/ # Images, fonts, and other static assets
| | ├── hero.png # Hero image for the SPA
| | ├── typescript.svg # TypeScript logo for the SPA
| | └── vite.svg # Vite logo for the SPA
│ ├── admin.test.ts # Admin frontend tests (Vitest)
│ ├── admin.ts # Admin SPA logic (playlist, moderation, analytics)
│ ├── counter.ts # Counter component for testing
│ ├── main.test.ts # Main frontend tests (Vitest)
│ ├── main.ts # Main SPA logic (hls.js player, WebSocket chat)
│ └── style.css # Custom CSS for the SPA
├── .gitignore # Ignore node_modules, build artifacts, etc.
├── index.html # Main SPA entry point
├── package-lock.json # Lockfile for npm dependencies
├── package.json # Build tools (Vite recommended)
├── tsconfig.json # TypeScript configuration
├── vite.config.ts # Vite configuration
└── vitest.config.ts # Vitest configuration
```
### Infrastructure
Everything needed to deploy and configure the stack (Docker, Nginx, Cloudflare rules, future Terraform).
- Cloudflare
- [worker.js](infrastructure/cloudflare/worker.js) — adds immutable cache headers to HLS segments
- Nginx
- [Dockerfile](infrastructure/nginx/Dockerfile) — builds Nginx with custom config
- [nginx.conf](infrastructure/nginx/nginx.conf) — reverse proxy, static SPA, HLS delivery
### Monitoring
Grafana dashboards, Prometheus rules, logging configuration.
- Grafana
- dashboards
- [dashboard.json](monitoring//grafana/dashboards/dashboard.json) — operational dashboards for playout, chat, and Redis
- [dashboard.yml](monitoring//grafana/dashboards/dashboard.yml) — operational dashboards for playout, chat, and Redis
- datasources
- [datasource.yml](monitoring/grafana/datasources/datasource.yml) — Prometheus datasource configuration
- [dashboard.json](monitoring/grafana/dashboard.json) — operational dashboards for playout, chat, and Redis
- Prometheus
- [alerts.yml](monitoring/prometheus/alerts.yml) — alerting rules for playout, chat, Redis, and system health
- [prometheus.yml](monitoring/prometheus/prometheus.yml) — scrape targets for playout, chat, Redis, and system metrics
### Scripts
One-off and CI-friendly scripts for video preparation, R2 sync, backups, and playlist management.
- loadtest/
- [admin-api.js](scripts/loadtest/admin-api.js) — admin API load test script
- [run-loadtests.ps1](scripts/loadtest/run-loadtests.ps1) — PowerShell script to run load tests
- [summarize-artillery.ps1](scripts/loadtest/summarize-artillery.ps1) — PowerShell script to summarize Artillery load test results
- [summarize-k6.ps1](scripts/loadtest/summarize-k6.ps1) — PowerShell script to summarize k6 load test results
- [ws-chat.yml](scripts/loadtest/ws-chat.yml) — WebSocket chat load test configuration
## Getting Started
> **Note:** The `docker compose up` below targets local development. The test lab (192.168.88.40) receives automated CD deploys. Production Hetzner VPS is not yet provisioned — see `DEPLOYMENT.md`.
```bash
git clone https://git.allucanget.biz/zwitschi/linearvoid.git
cd linearvoid
cp .env.example .env
# Edit .env with your R2 credentials and domain settings
docker compose up --build
```
Then open:
- **Viewer**: [https://linearvoid.tv](https://linearvoid.tv) (or `http://localhost`)
- **Admin**: [https://linearvoid.tv/admin](https://linearvoid.tv/admin) (or `http://localhost/admin`)
- **Metrics**: [https://linearvoid.tv/metrics](https://linearvoid.tv/metrics)
Full setup guide: [`SETUP.md`](SETUP.md).
See the [architecture docs](docs/architecture/README.md) for detailed architecture and ADRs.
## Development
- Use Docker Compose for local development (playout + chat + frontend).
- Test lab host: `192.168.88.40` (internal network, receives CD deploys). Production Hetzner VPS TBD.
- Public production host model: `https://linearvoid.tv` for viewer/admin/chat and `https://media.linearvoid.tv` for HLS delivery.
- Normalize all source videos before adding to playlist (see ADR-001 for exact codec/fps/keyframe requirements).
- Monitor cache hit ratio and R2 operations closely — they determine real cost.
### Environment Variables
| Variable | Default | Description |
| ---------------------- | ----------------------------------------------- | -------------------------------- |
| `R2_ACCESS_KEY_ID` | — | Cloudflare R2 access key |
| `R2_SECRET_ACCESS_KEY` | — | Cloudflare R2 secret key |
| `R2_BUCKET_NAME` | `linearvoid-media` | R2 bucket for HLS segments |
| `DATABASE_URL` | `postgres://...@db:5432/...` | Chat DB (Postgres or SQLite) |
| `REDIS_URL` | `redis://redis:6379/0` | Redis for presence/rate limiting |
| `VITE_PUBLIC_APP_URL` | `https://linearvoid.tv` | Public viewer URL |
| `VITE_PUBLIC_WS_URL` | `wss://linearvoid.tv/ws` | WebSocket endpoint |
| `VITE_PUBLIC_HLS_URL` | `https://media.linearvoid.tv/hls/playlist.m3u8` | HLS playlist URL |
| `ADMIN_USERNAME` | `admin` | Admin panel username |
| `ADMIN_PASSWORD` | — | Admin panel password |
| `HLS_SEGMENT_DURATION` | `6` | HLS segment duration in seconds |
| `HLS_WINDOW_SIZE` | `24` | HLS playlist window size |
Full reference: [`.env.example`](.env.example).
### Troubleshooting
| Problem | Check |
| ------------------------ | ------------------------------------------------------------------------------------- |
| **Playout not starting** | Verify playlist exists in `data/playlist/playlist.txt` and media files are accessible |
| **HLS not playing** | Check `docker compose logs playout` for ffmpeg errors; verify `data/hls/` permissions |
| **Chat not connecting** | Ensure Redis is running: `docker compose logs redis`; check `DATABASE_URL` |
| **Admin panel 401** | Verify `ADMIN_USERNAME`/`ADMIN_PASSWORD` in `.env` match login credentials |
| **Docker build fails** | Run `docker compose build --no-cache`; check `backend/playout/.dockerignore` |
| **R2 upload failures** | Verify `R2_ACCESS_KEY_ID` and `R2_SECRET_ACCESS_KEY`; check bucket CORS settings |
### API Endpoints
| Service | Endpoint | Description |
| ----------- | ------------------------ | -------------------------- |
| **Chat** | `GET /health` | Health check |
| **Chat** | `GET /metrics` | Prometheus metrics |
| **Chat** | `GET /ws` | WebSocket upgrade for chat |
| **Chat** | `GET /api/admin/*` | Admin API (auth required) |
| **Chat** | `POST /api/admin/login` | Admin login |
| **Playout** | `GET /health` | Health check (port 8081) |
| **Nginx** | `GET /` | Viewer SPA |
| **Nginx** | `GET /admin/` | Admin panel |
| **Nginx** | `GET /hls/playlist.m3u8` | HLS manifest |
## Architecture Documentation
The architecture is fully documented using the [arc42 template](docs/architecture/README.md) (with populated sections, Mermaid context and building-block diagrams, deployment view, cross-cutting concepts, quality goals, risks, and glossary).
**Decision Records** (in `docs/architecture/adr/`):
- **[ADR-001](docs/architecture/adr/ADR-001.md)**: Overall system architecture, HLS + Cloudflare R2 caching strategy, storage calculations, cost model, risks, and implementation plan.
- **[ADR-002](docs/architecture/adr/ADR-002-module-structure.md)**: Chosen project folder/module structure (frontend, backend/playout, backend/chat, infrastructure, scripts, monitoring) and mapping to arc42 building blocks.
- **[ADR-003](docs/architecture/adr/ADR-003-tech-stack.md)**: Technology stack, language choices per module (TypeScript for frontend, Go preferred for chat and playout wrapper, Bash/Python for scripts), Docker-first approach, and all implicit assumptions.
- **[ADR-004](docs/architecture/adr/ADR-004-admin-video.md)**: Video admin web interface (upload, playlist management, schedule overrides, normalization enforcement).
- **[ADR-005](docs/architecture/adr/ADR-005-admin-moderation-analytics.md)**: Web admin interfaces for user management, moderation dashboard, analytics, and system configuration.
- **[ADR-006](docs/architecture/adr/ADR-006-data-storage.md)**: Data storage strategy (SQLite vs PostgreSQL for v1, migration path, admin interface capabilities).
- **[ADR-007](docs/architecture/adr/ADR-007-redis-usage.md)**: Redis usage (exact use cases, TTL policies, persistence, fallback behavior, admin configuration).
See the [architecture README](docs/architecture/README.md) for the big-picture view and the individual ADR files for detailed rationale, alternatives, and consequences.
Contributions, issues, and further evolution (LL-HLS, ABR, multi-channel, E2EE) should reference the architecture documentation.
## Resources
- [CHANGELOG.md](CHANGELOG.md) - Version history and completed features.
- [CONTRIBUTING.md](CONTRIBUTING.md) - Guidelines for contributing to the project.
- [SETUP.md](docs/SETUP.md) - Complete setup guide with prerequisites, environment configuration, running the application, and troubleshooting.
- [API.md](docs/API.md) - Implemented WebSocket and REST API reference.
- [DEPLOYMENT.md](docs/DEPLOYMENT.md) - Production deployment and operations guide.
- [MONITORING.md](docs/MONITORING.md) - Prometheus and Grafana setup and troubleshooting.
- [SECURITY.md](docs/SECURITY.md) - Security controls, hardening checklist, and incident baseline.
- [Scaling Playbook](docs/operations/scaling-performance-playbook.md) - Performance test method, thresholds, and scaling decision flow.
- [Architecture Documentation](docs/architecture/README.md) - Full arc42 template with diagrams and decisions.
- [ADRs](docs/architecture/adr/) - Detailed decision records (ADR-001 to ADR-007).
**CI/CD**: [View workflows](https://git.allucanget.biz/zwitschi/linearvoid/actions) on Gitea Actions.
**Container Registry**: `git.allucanget.biz/zwitschi/linearvoid/<service>:latest`
## Contributing
Contributions are welcome! See [`CONTRIBUTING.md`](CONTRIBUTING.md) for guidelines. Quick overview:
1. Read the [architecture docs](docs/architecture/README.md) and ADRs first.
2. Code against the defined module structure (frontend, backend/playout, backend/chat, infrastructure, scripts, monitoring).
3. Open PRs on [Gitea](https://git.allucanget.biz/zwitschi/linearvoid).
4. CI validates builds and tests automatically.
Since this is a linear project, we also rely on a linear development and release model: only `main` branch is used for releases, and all features are merged into `main` after review and CI validation. Releases use semantic version tags in `*.*.*` format (for example `1.0.9`) and are automatically built/pushed to the container registry. Version numbers are declared in `frontend/package.json`, `CHANGELOG.md`, and the version badge in this README.
[MIT License](LICENSE)
The project is ready for use. See [`SETUP.md`](docs/SETUP.md) for detailed installation instructions.
Reference in new issue View Git Blame Copy Permalink