linearvoid/docs/STREAMING.md

Streaming Pipeline

This document is the operational contract for Linear Void video ingest, playout, publication, caching, and browser playback. General deployment, chat, and admin details remain in DEPLOYMENT.md, SETUP.md, and API.md.

Production flow

flowchart TD
    A["Normalized MP4 assets"] --> B["Playlist and schedule"]
    B --> C["Playout service and FFmpeg"]
    C --> D["Local HLS staging"]
    D --> E["R2 object storage"]
    E --> F["Cloudflare edge"]
    F --> G["hls.js or native HLS"]

The playout service is the timeline authority. It reads the ordered playlist, runs FFmpeg continuously, stages a live HLS window under data/hls/, and publishes complete objects to R2. Viewers read only the public R2 custom domain through Cloudflare:

https://media.linearvoid.tv/hls/playlist.m3u8

Nginx /hls/ is a local/test fallback, not the production media origin. The SPA, admin API, and chat use linearvoid.tv; media uses media.linearvoid.tv.

Source-media contract

Only validated files may enter the production playlist.

Property	Required profile
Container	MP4
Video	H.264/AVC, yuv420p
Resolution	1280x720 or 1920x1080; avoid changing resolution within a playlist block
Frame rate	Constant 30 fps
GOP	Closed, fixed 2 seconds: -g 60 -keyint_min 60 -sc_threshold 0
Audio	AAC-LC, stereo, 48 kHz, 128–192 kbit/s

The ingest gate uses ffprobe to reject incompatible codecs, variable frame rate, missing audio, invalid dimensions, or incorrect keyframe cadence. Assets are immutable after acceptance; corrections create a new asset ID and object key.

Reference normalization:

ffmpeg -i input.mov \
  -c:v libx264 -preset medium -crf 21 \
  -r 30 -g 60 -keyint_min 60 -sc_threshold 0 \
  -pix_fmt yuv420p \
  -c:a aac -b:a 160k -ar 48000 -ac 2 \
  -movflags +faststart \
  output-normalized.mp4

Playlist and timeline

• data/playlist/playlist.txt contains one local media path per line; blank lines and # comments are ignored.
• The service validates the complete playlist before activating a revision.
• Playlist edits are written atomically and receive a revision ID. The active FFmpeg run keeps its current item and adopts the new revision at the next item boundary.
• All viewers follow the same live manifest; clients do not calculate the schedule independently.
• Schedule metadata must be derived from the same active playlist revision and timeline so “now/next” agrees with the video.
• When the last item ends, playout returns to the first item unless an active schedule override says otherwise.

Playout and HLS packaging

The playout service wraps FFmpeg, tracks the active item and playlist revision, restarts failed processes, and exposes /health plus Prometheus metrics.

Baseline packaging:

Setting	Value
Protocol	Standard live HLS, not LL-HLS
Segment format	fMP4 (.m4s) with an initialization segment; .ts may be supported during migration
Segment target	2 seconds (HLS_SEGMENT_DURATION=2)
Live window	12 segments (HLS_WINDOW_SIZE=12), about 24 seconds
Codec path	Stream copy for conforming assets; no per-viewer transcoding
Expected latency	15–35 seconds

Segment boundaries must align with keyframes. Timestamps must remain monotonic across item boundaries. If FFmpeg cannot preserve a seamless timeline, insert #EXT-X-DISCONTINUITY and publish the corresponding new initialization segment.

Segment duration rationale: 2s segments were chosen to lower end-to-end latency and keep the live window small (12 × 2s = 24s) while staying well above the manifest TTL (1–2s). The 2s GOP (30fps, -g 60 -keyint_min 60) already matches the target, so segments cut cleanly on keyframes. Longer durations (4–6s) increase latency and window depth; shorter than 2s inflates segment/request overhead and R2 Class B operations with little latency gain.

Every segment name is globally unique and never reused, including after a restart or playlist loop. Use a run/session ID plus a monotonically increasing media sequence, for example:

hls/r-20260807T170412Z/init-000001.mp4
hls/r-20260807T170412Z/seg-000001234.m4s

This naming rule is essential: a URL advertised as immutable must never later contain different bytes.

Safe publication to R2

Publication order prevents a manifest from referencing an unavailable segment:

1. FFmpeg writes a segment to a temporary local name and closes it.
2. The publisher uploads the initialization segment, if new, and media segment to their final immutable R2 keys.
3. It verifies successful upload and records object size/checksum.
4. It renders the next media manifest referencing only confirmed objects.
5. It uploads the manifest to a temporary key, then replaces hls/playlist.m3u8 as the final commit step.

Never upload an actively written segment or publish the manifest first. Use a single publisher instance, or fencing/leader election, so two playout workers cannot update the live manifest concurrently.

The local data/hls/ directory is staging data and may be regenerated. Source media, playlist revisions, and schedule state are authoritative; generated HLS is not backed up as application state.

Cache contract

Objects	Edge policy	Browser policy	Required headers/behavior
.m4s, .ts, init .mp4	Cache eligible; TTL 1 year	TTL 1 year	Cache-Control: public, max-age=31536000, immutable; full path in cache key
Live .m3u8	Cache eligible; edge TTL 2 seconds	Do not retain beyond freshness interval	Cache-Control: public, max-age=1, s-maxage=2, must-revalidate
Versioned JS/CSS	Cache eligible; TTL 1 year	TTL 1 year	Content-hashed filenames and immutable

Further rules:

• Do not add query-string cache busters to segment URLs; uniqueness belongs in the path.
• Cache keys must preserve hostname and full path. Ignore only explicitly approved tracking parameters.
• HLS responses require Access-Control-Allow-Origin: https://linearvoid.tv (or the reviewed origin allowlist), GET, HEAD, and OPTIONS support, and correct MIME types.
• Enable Smart Tiered Cache for media.linearvoid.tv to reduce R2 reads on regional misses.
• Purge manifests when necessary, but never purge immutable segments during normal operation.
• Retain expired-window segments in R2 for at least the maximum manifest TTL plus client retry/grace time. A practical minimum is one hour; lifecycle deletion may run later according to storage policy.

Expected validation:

curl -I https://media.linearvoid.tv/hls/playlist.m3u8
curl -I https://media.linearvoid.tv/hls/<segment>.m4s

The manifest must refresh within a few seconds. Repeated segment requests should progress from CF-Cache-Status: MISS to HIT, retain the same ETag/content length, and carry the immutable cache header.

Player behavior

The SPA uses native HLS where supported and hls.js elsewhere.

• Firefox has no native HLS, so it uses the hls.js/MSE path; fMP4 (.m4s + init segment) is supported. Autoplay requires muted + playsinline, which are already set on the <video> element.

• Start at the live edge with a conservative buffer; do not seek to playlist start.

• Treat the manifest as the synchronization source and target roughly three segments behind the newest segment unless testing supports a smaller margin.

• Recover transient network/media errors with bounded retries and exponential backoff.

• On a fatal hls.js network error, reload the manifest; on a fatal media error, attempt media recovery once before recreating the player.

• When tab suspension or drift leaves playback materially behind the target live position, seek forward to the target rather than playing the backlog.

• Surface a clear reconnecting state. Never silently loop stale media.

• Schedule/chat timestamps should account for the measured player latency rather than assuming wall-clock playout is already visible.

Failure and recovery

The service retries an item failure with a bounded backoff. After the configured failure threshold it switches to a pre-normalized looping slate and reports mode=slate. It periodically retries normal playout and returns only after a segment and committed manifest have been published successfully.

On process or host restart:

• create a new run/session ID and continue with globally unique segment paths;
• resume from the schedule-derived current position when reliable state exists, otherwise start the current playlist item;
• never overwrite old segment objects;
• keep the last valid manifest available until the replacement stream is playable;
• emit a discontinuity when timeline continuity cannot be guaranteed.

Health is not merely “process running.” /health should report at least mode (live, slate, starting, degraded), active item, playlist revision, run ID, last segment sequence/time, last successful R2 publication, consecutive failures, and uptime. Mark the service unhealthy if the manifest or media sequence has not advanced for more than two segment durations.

Monitoring and acceptance checks

Monitor and alert on:

• manifest age and media-sequence advancement;
• time since last successfully published segment;
• FFmpeg restarts and consecutive item failures;
• live/slate mode;
• R2 upload latency and errors;
• segment 404/5xx rate and manifest/segment request latency;
• Cloudflare cache-hit ratio, origin requests, and R2 Class B operations;
• player startup time, fatal errors, rebuffer ratio, and live-edge drift;
• local staging disk usage.

A release is streaming-ready only when one full playlist boundary has played without timestamp errors, the manifest never references a missing object, a second viewer joins the same timeline, cache headers match the table above, a forced FFmpeg failure activates the slate, and normal playout recovers without reusing an object URL.

Required configuration

Variable	Purpose
PLAYLIST_PATH	Active playlist file
HLS_SEGMENT_DURATION	Target segment duration; baseline 2
HLS_WINDOW_SIZE	Manifest window; baseline 12
R2_ENDPOINT, R2_REGION	S3-compatible publication endpoint
R2_BUCKET_NAME	Media bucket
R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY	Write credentials, available only to the publisher
R2_KEEP_RUNS	Number of run folders to retain (default 2)
R2_MAX_RETENTION_BYTES	Max bytes retained across run folders (10 GiB)
MEDIA_DOMAIN	Public R2 custom domain
VITE_PUBLIC_HLS_URL	Public live manifest URL

Production values are MEDIA_DOMAIN=media.linearvoid.tv and VITE_PUBLIC_HLS_URL=https://media.linearvoid.tv/hls/playlist.m3u8. Local fallback may use http://localhost/hls/playlist.m3u8.

Explicitly deferred

Adaptive-bitrate ladders, LL-HLS, DVR/catch-up, multi-origin playout, and regional chat affinity are outside the v1 contract. Adding any of them requires an ADR and an update to this document.