zwitschi/linearvoid
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 9 Wiki Activity

Refactor load testing scripts to support Bash and PowerShell
CI / test (push) Successful in 2m50s
Details
CI / docker (push) Successful in 5m12s
Details

Browse Source 
- Updated README.md to reflect changes in load testing commands and added usage examples for Bash scripts.
- Converted PowerShell load testing scripts to Bash equivalents for better cross-platform compatibility.
- Added new Bash scripts: run-loadtests.sh, summarize-artillery.sh, and summarize-k6.sh to mirror existing PowerShell functionality.
- Updated admin-api.js and ws-chat.yml for consistency and improved performance metrics reporting.
- Enhanced error handling and output formatting in summary scripts for better readability.
This commit is contained in:
zwitschi committed 2026-09-02 19:47:34 +02:00
1 parent 9f60087a4c
commit 33b6296c5e
82 files changed
+19682 -21133
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/STREAMING.md
+67 -80
View File
@@ -1,6 +1,6 @@
# Streaming Pipeline
This document is the operational contract for Linear Void video ingest, playout, publication, caching, and browser playback. General deployment, chat, and admin details remain in `DEPLOYMENT.md`, `SETUP.md`, and `API.md`.
Operational contract for Linear Void video ingest, playout, publication, caching, and browser playback. Deployment, chat, admin details live in `DEPLOYMENT.md`, `SETUP.md`, `API.md`.
## Production flow
@@ -20,7 +20,7 @@ The playout service is the timeline authority. It reads the ordered playlist, ru
https://media.linearvoid.tv/hls/playlist.m3u8
```
Nginx `/hls/` is a local/test fallback, not the production media origin. The SPA, admin API, and chat use `linearvoid.tv`; media uses `media.linearvoid.tv`.
Nginx `/hls/` is a local/test fallback, not the production origin. The SPA, admin API, and chat use `linearvoid.tv`; media uses `media.linearvoid.tv`.
## Source-media contract
@@ -30,12 +30,12 @@ Only validated files may enter the production playlist.
| ---------- | ------------------------------------------------------------------------ |
| Container | MP4 |
| Video | H.264/AVC, `yuv420p` |
| Resolution | 1280x720 or 1920x1080; avoid changing resolution within a playlist block |
| Resolution | 1280x720 or 1920x1080; avoid resolution changes within a playlist block |
| Frame rate | Constant 30 fps |
| GOP | Closed, fixed 2 seconds: `-g 60 -keyint_min 60 -sc_threshold 0` |
| Audio | AAC-LC, stereo, 48 kHz, 128–192 kbit/s |
The ingest gate uses `ffprobe` to reject incompatible codecs, variable frame rate, missing audio, invalid dimensions, or incorrect keyframe cadence. Assets are immutable after acceptance; corrections create a new asset ID and object key.
The ingest gate uses `ffprobe` to reject incompatible codecs, variable frame rate, missing audio, invalid dimensions, or wrong keyframe cadence. Assets are immutable after acceptance; corrections create a new asset ID/object key.
Reference normalization:
@@ -51,124 +51,111 @@ ffmpeg -i input.mov \
## Playlist and timeline
- `data/playlist/playlist.txt` contains one local media path per line; blank lines and `#` comments are ignored.
- The service validates the complete playlist before activating a revision.
- Playlist edits are written atomically and receive a revision ID. The active FFmpeg run keeps its current item and adopts the new revision at the next item boundary.
- All viewers follow the same live manifest; clients do not calculate the schedule independently.
- Schedule metadata must be derived from the same active playlist revision and timeline so “now/next” agrees with the video.
- When the last item ends, playout returns to the first item unless an active schedule override says otherwise.
- `data/playlist/playlist.txt`: one local media path per line; blank lines and `#` comments ignored.
- Service validates the complete playlist before activating a revision.
- Playlist edits are atomic and get a revision ID. The active FFmpeg run keeps its current item and adopts the new revision at the next item boundary.
- All viewers follow the same live manifest; clients never calculate the schedule independently.
- Schedule metadata derives from the same active revision/timeline so "now/next" agrees with the video.
- When the last item ends, playout returns to the first unless an active schedule override says otherwise.
## Playout and HLS packaging
The playout service wraps FFmpeg, tracks the active item and playlist revision, restarts failed processes, and exposes `/health` plus Prometheus metrics.
The playout service wraps FFmpeg, tracks the active item/revision, restarts failed processes, and exposes `/health` plus Prometheus metrics.
Baseline packaging:
| Setting | Value |
| -------------- | ---------------------------------------------------------------------------------- |
| Protocol | Standard live HLS, not LL-HLS |
| Segment format | fMP4 (`.m4s`) with init segment; `.ts` may be supported during migration |
| Segment target | 2 seconds (`HLS_SEGMENT_DURATION=2`) |
| Live window | 12 segments (`HLS_WINDOW_SIZE=12`), ~24 seconds |
| Codec path | Stream copy for conforming assets; no per-viewer transcoding |
| Expected latency | 15–35 seconds |
| Setting | Value |
| ---------------- | ------------------------------------------------------------------------------------- |
| Protocol | Standard live HLS, not LL-HLS |
| Segment format | fMP4 (`.m4s`) with an initialization segment; `.ts` may be supported during migration |
| Segment target | 2 seconds (`HLS_SEGMENT_DURATION=2`) |
| Live window | 12 segments (`HLS_WINDOW_SIZE=12`), about 24 seconds |
| Codec path | Stream copy for conforming assets; no per-viewer transcoding |
| Expected latency | 15–35 seconds |
Segment boundaries must align with keyframes; timestamps must stay monotonic across item boundaries. If FFmpeg cannot preserve a seamless timeline, insert `#EXT-X-DISCONTINUITY` and publish the corresponding new init segment.
Segment boundaries must align with keyframes. Timestamps must remain monotonic across item boundaries. If FFmpeg cannot preserve a seamless timeline, insert `#EXT-X-DISCONTINUITY` and publish the corresponding new initialization segment.
**Segment duration rationale**: 2s lowers latency and keeps the window small (12 × 2s = 24s) while staying above the manifest TTL (1–2s). The 2s GOP (30fps, `-g 60`) already matches the target, so segments cut cleanly on keyframes. Longer (4–6s) raises latency and window depth; shorter than 2s inflates segment/request overhead and R2 Class B ops with little gain.
**Segment duration rationale**: 2s segments were chosen to lower end-to-end latency and keep the live window small (12 × 2s = 24s) while staying well above the manifest TTL (1–2s). The 2s GOP (30fps, `-g 60 -keyint_min 60`) already matches the target, so segments cut cleanly on keyframes. Longer durations (4–6s) increase latency and window depth; shorter than 2s inflates segment/request overhead and R2 Class B operations with little latency gain.
Every segment name is globally unique and never reused, including after a restart or playlist loop. Use a run/session ID plus a monotonically increasing media sequence, for example:
Every segment name is globally unique and never reused, including after restart or playlist loop — use a run/session ID plus a monotonically increasing media sequence:
```text
hls/r-20260807T170412Z/init-000001.mp4
hls/r-20260807T170412Z/seg-000001234.m4s
```
This naming rule is essential: a URL advertised as immutable must never later contain different bytes.
A URL advertised as immutable must never later contain different bytes.
## Safe publication to R2
Publication order prevents a manifest from referencing an unavailable segment:
Order prevents a manifest referencing an unavailable segment:
1. FFmpeg writes a segment to a temporary local name and closes it.
2. The publisher uploads the initialization segment, if new, and media segment to their final immutable R2 keys.
3. It verifies successful upload and records object size/checksum.
4. It renders the next media manifest referencing only confirmed objects.
5. It uploads the manifest to a temporary key, then replaces `hls/playlist.m3u8` as the final commit step.
1. FFmpeg writes a segment to a temp local name and closes it.
2. Publisher uploads the init segment (if new) and media segment to final immutable R2 keys.
3. Verify upload success; record object size/checksum.
4. Render the next manifest referencing only confirmed objects.
5. Upload the manifest to a temp key, then replace `hls/playlist.m3u8` as the final commit step.
Never upload an actively written segment or publish the manifest first. Use a single publisher instance, or fencing/leader election, so two playout workers cannot update the live manifest concurrently.
Never upload an actively written segment or publish the manifest first. Use a single publisher (or fencing/leader election) so two workers can't update the live manifest concurrently.
The local `data/hls/` directory is staging data and may be regenerated. Source media, playlist revisions, and schedule state are authoritative; generated HLS is not backed up as application state.
`data/hls/` is regenerable staging. Source media, playlist revisions, and schedule state are authoritative; generated HLS is not backed up as application state.
## Cache contract
| Objects | Edge policy | Browser policy | Required headers/behavior |
| -------------------------- | ---------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------- |
| `.m4s`, `.ts`, init `.mp4` | Cache eligible; TTL 1 year | TTL 1 year | `Cache-Control: public, max-age=31536000, immutable`; full path in cache key |
| Live `.m3u8` | Cache eligible; edge TTL 2 seconds | Do not retain beyond freshness interval | `Cache-Control: public, max-age=1, s-maxage=2, must-revalidate` |
| Versioned JS/CSS | Cache eligible; TTL 1 year | TTL 1 year | Content-hashed filenames and `immutable` |
| Objects | Edge policy | Browser policy | Required headers/behavior |
| -------------------------- | --------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------- |
| `.m4s`, `.ts`, init `.mp4` | Cache eligible; TTL 1 year | TTL 1 year | `Cache-Control: public, max-age=31536000, immutable`; full path in cache key |
| Live `.m3u8` | Cache eligible; edge TTL 2s | Do not retain beyond freshness interval | `Cache-Control: public, max-age=1, s-maxage=2, must-revalidate` |
| Versioned JS/CSS | Cache eligible; TTL 1 year | TTL 1 year | Content-hashed filenames and `immutable` |
Further rules:
- Do not add query-string cache busters to segment URLs; uniqueness belongs in the path.
- Cache keys must preserve hostname and full path. Ignore only explicitly approved tracking parameters.
- HLS responses require `Access-Control-Allow-Origin: https://linearvoid.tv` (or the reviewed origin allowlist), `GET`, `HEAD`, and `OPTIONS` support, and correct MIME types.
- Enable Smart Tiered Cache for `media.linearvoid.tv` to reduce R2 reads on regional misses.
- Purge manifests when necessary, but never purge immutable segments during normal operation.
- Retain expired-window segments in R2 for at least the maximum manifest TTL plus client retry/grace time. A practical minimum is one hour; lifecycle deletion may run later according to storage policy.
- No query-string cache busters on segment URLs; uniqueness belongs in the path.
- Cache keys preserve hostname and full path; ignore only explicitly approved tracking params.
- HLS responses need `Access-Control-Allow-Origin: https://linearvoid.tv` (or reviewed allowlist), `GET`/`HEAD`/`OPTIONS`, and correct MIME types.
- Enable Smart Tiered Cache for `media.linearvoid.tv` to cut R2 reads on regional misses.
- Purge manifests when necessary, never immutable segments during normal operation.
- Retain expired-window segments at least max manifest TTL + retry/grace time; practical minimum 1 hour; lifecycle deletion may run later per storage policy.
Expected validation:
Validation:
```bash
curl -I https://media.linearvoid.tv/hls/playlist.m3u8
curl -I https://media.linearvoid.tv/hls/<segment>.m4s
```
The manifest must refresh within a few seconds. Repeated segment requests should progress from `CF-Cache-Status: MISS` to `HIT`, retain the same ETag/content length, and carry the immutable cache header.
The manifest must refresh within seconds; repeated segment requests should progress `CF-Cache-Status: MISS` → `HIT`, keep the same ETag/content length, and carry the immutable cache header.
## Player behavior
The SPA uses native HLS where supported and hls.js elsewhere.
- Firefox has no native HLS, so it uses the hls.js/MSE path; fMP4 (`.m4s` + init segment) is supported. Autoplay requires `muted` + `playsinline`, which are already set on the `<video>` element.
SPA uses native HLS where supported, hls.js elsewhere.
- Firefox has no native HLS → hls.js/MSE path; fMP4 (`.m4s` + init) is supported. Autoplay needs `muted` + `playsinline`, already set on the `<video>` element.
- Start at the live edge with a conservative buffer; do not seek to playlist start.
- Treat the manifest as the synchronization source and target roughly three segments behind the newest segment unless testing supports a smaller margin.
- Recover transient network/media errors with bounded retries and exponential backoff.
- On a fatal hls.js network error, reload the manifest; on a fatal media error, attempt media recovery once before recreating the player.
- When tab suspension or drift leaves playback materially behind the target live position, seek forward to the target rather than playing the backlog.
- Surface a clear reconnecting state. Never silently loop stale media.
- Schedule/chat timestamps should account for the measured player latency rather than assuming wall-clock playout is already visible.
- Treat the manifest as the sync source; target ~3 segments behind the newest unless testing supports a smaller margin.
- Recover transient network/media errors with bounded retries + exponential backoff.
- On fatal hls.js network error, reload the manifest; on fatal media error, attempt media recovery once before recreating the player.
- If tab suspension/drift leaves playback materially behind the target, seek forward rather than play the backlog.
- Surface a clear reconnecting state; never silently loop stale media.
- Schedule/chat timestamps should account for measured player latency, not wall-clock playout.
## Failure and recovery
The service retries an item failure with a bounded backoff. After the configured failure threshold it switches to a pre-normalized looping slate and reports `mode=slate`. It periodically retries normal playout and returns only after a segment and committed manifest have been published successfully.
The service retries an item failure with bounded backoff. Past the failure threshold it switches to a pre-normalized looping slate and reports `mode=slate`, periodically retrying normal playout, returning only after a segment and committed manifest publish successfully.
On process or host restart:
On process/host restart:
- create a new run/session ID and continue with globally unique segment paths;
- resume from the schedule-derived current position when reliable state exists, otherwise start the current playlist item;
- create a new run/session ID and continue with unique segment paths;
- resume from schedule-derived current position when reliable state exists, else start the current playlist item;
- never overwrite old segment objects;
- keep the last valid manifest available until the replacement stream is playable;
- emit a discontinuity when timeline continuity cannot be guaranteed.
- keep the last valid manifest available until the replacement is playable;
- emit a discontinuity when timeline continuity isn't guaranteed.
Health is not merely “process running.” `/health` should report at least mode (`live`, `slate`, `starting`, `degraded`), active item, playlist revision, run ID, last segment sequence/time, last successful R2 publication, consecutive failures, and uptime. Mark the service unhealthy if the manifest or media sequence has not advanced for more than two segment durations.
`/health` must report more than "process running": at least mode (`live`/`slate`/`starting`/`degraded`), active item, playlist revision, run ID, last segment sequence/time, last successful R2 publication, consecutive failures, uptime. Mark unhealthy if the manifest or media sequence hasn't advanced for more than two segment durations.
## Monitoring and acceptance checks
## Monitoring and acceptance
Monitor and alert on:
Monitor/alert on: manifest age and media-sequence advancement; time since last published segment; FFmpeg restarts and consecutive failures; live/slate mode; R2 upload latency/errors; segment 404/5xx and manifest/segment latency; Cloudflare cache-hit ratio, origin requests, R2 Class B ops; player startup, fatal errors, rebuffer ratio, live-edge drift; local staging disk usage.
- manifest age and media-sequence advancement;
- time since last successfully published segment;
- FFmpeg restarts and consecutive item failures;
- live/slate mode;
- R2 upload latency and errors;
- segment 404/5xx rate and manifest/segment request latency;
- Cloudflare cache-hit ratio, origin requests, and R2 Class B operations;
- player startup time, fatal errors, rebuffer ratio, and live-edge drift;
- local staging disk usage.
A release is streaming-ready only when one full playlist boundary has played without timestamp errors, the manifest never references a missing object, a second viewer joins the same timeline, cache headers match the table above, a forced FFmpeg failure activates the slate, and normal playout recovers without reusing an object URL.
A release is streaming-ready only when: one full playlist boundary plays without timestamp errors; the manifest never references a missing object; a second viewer joins the same timeline; cache headers match the table; a forced FFmpeg failure activates slate; and normal playout recovers without reusing an object URL.
## Required configuration
@@ -179,14 +166,14 @@ A release is streaming-ready only when one full playlist boundary has played wit
| `HLS_WINDOW_SIZE` | Manifest window; baseline `12` |
| `R2_ENDPOINT`, `R2_REGION` | S3-compatible publication endpoint |
| `R2_BUCKET_NAME` | Media bucket |
| `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY` | Write credentials, available only to the publisher |
| `R2_KEEP_RUNS` | Number of run folders to retain (default `2`) |
| `R2_MAX_RETENTION_BYTES` | Max bytes retained across run folders (10 GiB) |
| `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY` | Write credentials, publisher only |
| `R2_KEEP_RUNS` | Run folders to retain (default `2`) |
| `R2_MAX_RETENTION_BYTES` | Max bytes across run folders (10 GiB) |
| `MEDIA_DOMAIN` | Public R2 custom domain |
| `VITE_PUBLIC_HLS_URL` | Public live manifest URL |
Production values are `MEDIA_DOMAIN=media.linearvoid.tv` and `VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8`. Local fallback may use `http://localhost/hls/playlist.m3u8`.
Production values: `MEDIA_DOMAIN=media.linearvoid.tv`, `VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8`. Local fallback may use `http://localhost/hls/playlist.m3u8`.
## Explicitly deferred
Adaptive-bitrate ladders, LL-HLS, DVR/catch-up, multi-origin playout, and regional chat affinity are outside the v1 contract. Adding any of them requires an ADR and an update to this document.
ABR ladders, LL-HLS, DVR/catch-up, multi-origin playout, and regional chat affinity are outside the v1 contract. Adding any requires an ADR and an update to this document.