diff --git a/.env.example b/.env.example index 2306595..1158a72 100644 --- a/.env.example +++ b/.env.example @@ -8,7 +8,8 @@ R2_SECRET_ACCESS_KEY=your-r2-secret-key R2_BUCKET_NAME=linearvoid-media R2_ENDPOINT=https://your-account-id.r2.cloudflarestorage.com R2_REGION=auto -R2_SEGMENT_RETENTION_HOURS=24 +R2_KEEP_RUNS=2 +R2_MAX_RETENTION_BYTES=10737418240 # Cloudflare (for custom domain, cache rules, Workers if used) CLOUDFLARE_API_TOKEN=your-cloudflare-api-token diff --git a/.github/MEMORY.md b/.github/MEMORY.md index d2420da..7e2b59c 100644 --- a/.github/MEMORY.md +++ b/.github/MEMORY.md @@ -138,7 +138,7 @@ ## 2026-08-13 - Completed all remaining enhancement tasks (only manual/infrastructure phases left): - - **Playout**: automatic media normalization on upload (`media.NormalizeVideo`, `*.normalized.mp4`, statuses `ready`/`needs_normalization`/`normalizing`/`normalization_failed`, `POST /api/admin/media/{id}/normalize`); R2 segment retention (`cleanupOldSegments`, `R2_SEGMENT_RETENTION_HOURS`); seamless looping (playlist SHA-256 fingerprint + `-start_number`); single `start-playout.sh` entrypoint + `-validate` mode; loop/cleanup metrics. + - **Playout**: automatic media normalization on upload (`media.NormalizeVideo`, `*.normalized.mp4`, statuses `ready`/`needs_normalization`/`normalizing`/`normalization_failed`, `POST /api/admin/media/{id}/normalize`); R2 segment retention (`cleanupOldRuns`, keep last `R2_KEEP_RUNS` runs, `R2_MAX_RETENTION_BYTES` cap); seamless looping (playlist SHA-256 fingerprint + `-start_number`); single `start-playout.sh` entrypoint + `-validate` mode; loop/cleanup metrics. - **Application configuration**: thread-safe JSON runtime config store (`backend/chat/src/config/runtime.go`) shared with playout via `/app/playlist/config.json`; `GET/PUT /api/admin/config`; admin UI config form. - **Matomo analytics**: public `GET /api/matomo` + viewer pageview snippet in `index.html`. - **Chat user accounts**: register/login/logout/session endpoints, bcrypt hashing, `linearvoid_user_session` cookie, viewer sign-in UI. diff --git a/.github/instructions/TODO.md b/.github/instructions/TODO.md index 8fe6395..ea0b9b0 100644 --- a/.github/instructions/TODO.md +++ b/.github/instructions/TODO.md @@ -2,10 +2,6 @@ > Completed tasks moved to [DONE](./DONE.md). Remaining tasks in [BACKLOG](./BACKLOG.md). -## R2 Data Retention - -- [ ] Adjust internal data retention for R2: remove folders /hls/r-{ISOTIME} after next run, keep only the last 2 runs or max. 10GB of data - ## Frontend Tests ### User Frontend diff --git a/CHANGELOG.md b/CHANGELOG.md index 965d4c7..6f6b6b5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,12 +14,14 @@ - **Reconnect grace period**: Disconnects schedule a delayed "left" announcement (5s); a reconnect within 5s cancels it and skips "joined", eliminating join/leave spam on brief reconnects. - **HLS segment duration 2s**: Default `HLS_SEGMENT_DURATION` changed from 6 to 2 and `HLS_WINDOW_SIZE` from 24 to 12 (2s × 12 = 24s live window), reducing end-to-end latency; the 2s GOP was already in place. - **Chat font sizing**: `.message-text` set to `0.95rem` (slightly smaller than the default, larger than the `0.83rem` system/flash text). +- **R2 retention by run count + size**: Playout now keeps the newest `R2_KEEP_RUNS` run folders (default 2) and enforces an `R2_MAX_RETENTION_BYTES` cap (default 10 GiB), replacing the old 24h time-based `R2_SEGMENT_RETENTION_HOURS` cleanup. Cleanup runs after each new run plus an hourly backstop, and never deletes the live manifest. Adds `linearvoid_playout_r2_runs_retained` and `linearvoid_playout_r2_bytes_retained` gauges. ### Tests - WebSocket: `TestHub_Broadcast_SystemNotPersisted`, `TestHub_ReconnectGracePeriod`. - Handlers: `TestHandleAdminChatLog_ExcludesSystem`. - Frontend: `presence flash` test. +- Playout: `TestCleanupOldRuns`, `TestCleanupOldRuns_SizeCap`, `TestR2KeepRuns`, `TestR2MaxRetentionBytes`, `TestR2RunFromKey`. ## [1.0.13] - 2026-08-13 diff --git a/backend/playout/src/playlist.go b/backend/playout/src/playlist.go index 5496de1..2e44d6f 100644 --- a/backend/playout/src/playlist.go +++ b/backend/playout/src/playlist.go @@ -11,6 +11,7 @@ import ( "os" "os/exec" "path/filepath" + "sort" "strconv" "strings" "time" @@ -66,6 +67,10 @@ var ( // R2 retention cleanup counter r2CleanupObjectsTotal int64 + // R2 retention gauges + r2RunsRetained int64 + r2BytesRetained int64 + // Continuous playout loop metrics loopsTotal int64 lastRunDuration time.Duration @@ -78,6 +83,10 @@ const ( // Cache-Control header values for R2 uploads. segmentCacheControl = "public, max-age=31536000, immutable" manifestCacheControl = "public, max-age=1, s-maxage=2, must-revalidate" + + // R2 retention policy. + defaultR2KeepRuns = 2 + defaultR2MaxRetentionBytes = int64(10 * 1024 * 1024 * 1024) // 10 GiB ) // envWithDefault returns the value of the environment variable or the default if not set. @@ -658,6 +667,12 @@ func main() { fmt.Fprintln(w, "# HELP linearvoid_playout_r2_cleanup_objects_total Total R2 objects deleted by retention cleanup.") fmt.Fprintln(w, "# TYPE linearvoid_playout_r2_cleanup_objects_total counter") fmt.Fprintf(w, "linearvoid_playout_r2_cleanup_objects_total %d\n", r2CleanupObjectsTotal) + fmt.Fprintln(w, "# HELP linearvoid_playout_r2_runs_retained Number of R2 run folders retained by cleanup.") + fmt.Fprintln(w, "# TYPE linearvoid_playout_r2_runs_retained gauge") + fmt.Fprintf(w, "linearvoid_playout_r2_runs_retained %d\n", r2RunsRetained) + fmt.Fprintln(w, "# HELP linearvoid_playout_r2_bytes_retained Total bytes retained in R2 run folders.") + fmt.Fprintln(w, "# TYPE linearvoid_playout_r2_bytes_retained gauge") + fmt.Fprintf(w, "linearvoid_playout_r2_bytes_retained %d\n", r2BytesRetained) fmt.Fprintln(w, "# HELP linearvoid_playout_loops_total Total clean playlist completions (loops).") fmt.Fprintln(w, "# TYPE linearvoid_playout_loops_total counter") fmt.Fprintf(w, "linearvoid_playout_loops_total %d\n", loopsTotal) @@ -682,6 +697,15 @@ func main() { // New run ID on each restart for unique R2 segment keys (STREAMING.md) initRunID() + // Remove run folders older than the retention policy (keep the last N runs, + // cap total bytes) now that the next run has started. This never touches the + // current or previous run folders. + if r2Client != nil { + if _, err := cleanupOldRuns(context.Background()); err != nil { + fmt.Println("R2 cleanup error:", err) + } + } + // Reset HLS output only when the playlist changed, we are recovering from a // crash/slate, or this is the first run. Otherwise continue the segment // sequence so the loop is seamless for viewers (no gap on clean loops). @@ -834,68 +858,51 @@ func syncLatestToR2(hlsDir string) error { return nil } -// r2SegmentRetentionHours returns the R2 segment retention window in hours. -func r2SegmentRetentionHours() time.Duration { - h, err := strconv.Atoi(envWithDefault("R2_SEGMENT_RETENTION_HOURS", "24")) - if err != nil || h <= 0 { - h = 24 +// r2KeepRuns returns how many of the most recent run folders to keep in R2. +func r2KeepRuns() int { + n, err := strconv.Atoi(envWithDefault("R2_KEEP_RUNS", "2")) + if err != nil || n < 1 { + return defaultR2KeepRuns } - return time.Duration(h) * time.Hour + return n } -// cleanupOldSegments deletes R2 objects under the hls/ prefix that are older than -// the retention window, excluding the live manifest and any non-run keys. -// It returns the number of objects deleted. -func cleanupOldSegments(ctx context.Context) (int, error) { - if r2Client == nil { - if err := initR2Client(); err != nil { - return 0, err - } +// r2MaxRetentionBytes returns the maximum total bytes to retain across run folders. +func r2MaxRetentionBytes() int64 { + n, err := strconv.ParseInt(envWithDefault("R2_MAX_RETENTION_BYTES", "10737418240"), 10, 64) + if err != nil || n <= 0 { + return defaultR2MaxRetentionBytes } + return n +} - cutoff := time.Now().UTC().Add(-r2SegmentRetentionHours()) - - var stale []string - var token *string - for { - output, err := listObjects(r2Client, &s3.ListObjectsV2Input{ - Bucket: aws.String(bucket), - Prefix: aws.String("hls/"), - ContinuationToken: token, - }) - if err != nil { - return 0, err - } - for _, obj := range output.Contents { - key := aws.ToString(obj.Key) - if key == "hls/playlist.m3u8" || key == "hls/playlist.m3u8.tmp" { - continue - } - // Only consider segment/init objects under a run directory: hls/r-.../... - if !strings.HasPrefix(key, "hls/r-") { - continue - } - if obj.LastModified != nil && obj.LastModified.Before(cutoff) { - stale = append(stale, key) - } - } - if output.IsTruncated == nil || !*output.IsTruncated { - break - } - token = output.NextContinuationToken +// r2RunFromKey extracts the run folder name ("r-...") from an R2 object key. +func r2RunFromKey(key string) (string, bool) { + rest := strings.TrimPrefix(key, "hls/") + if rest == key || !strings.HasPrefix(rest, "r-") { + return "", false } - - if len(stale) == 0 { - return 0, nil + if idx := strings.IndexByte(rest, '/'); idx > 0 { + return rest[:idx], true } + return "", false +} +// runObjects groups the R2 object keys and total size for a single run folder. +type runObjects struct { + keys []string + size int64 +} + +// deleteR2Keys deletes the given R2 object keys in batches of up to 1000. +func deleteR2Keys(keys []string) (int, error) { deleted := 0 - for i := 0; i < len(stale); i += 1000 { + for i := 0; i < len(keys); i += 1000 { end := i + 1000 - if end > len(stale) { - end = len(stale) + if end > len(keys) { + end = len(keys) } - batch := stale[i:end] + batch := keys[i:end] objects := make([]types.ObjectIdentifier, len(batch)) for j, key := range batch { objects[j] = types.ObjectIdentifier{Key: aws.String(key)} @@ -911,6 +918,118 @@ func cleanupOldSegments(ctx context.Context) (int, error) { } deleted += len(batch) } + return deleted, nil +} + +// cleanupOldRuns lists run-scoped objects under hls/, keeps the most recent +// R2_KEEP_RUNS runs, and deletes older runs. It also enforces the +// R2_MAX_RETENTION_BYTES cap by removing oldest retained runs first, always +// keeping at least the newest run. The live manifest keys are never deleted. +func cleanupOldRuns(ctx context.Context) (int, error) { + if r2Client == nil { + if err := initR2Client(); err != nil { + return 0, err + } + } + + runs := map[string]*runObjects{} + var token *string + for { + output, err := listObjects(r2Client, &s3.ListObjectsV2Input{ + Bucket: aws.String(bucket), + Prefix: aws.String("hls/"), + ContinuationToken: token, + }) + if err != nil { + return 0, err + } + for _, obj := range output.Contents { + key := aws.ToString(obj.Key) + if key == "hls/playlist.m3u8" || key == "hls/playlist.m3u8.tmp" { + continue + } + run, ok := r2RunFromKey(key) + if !ok { + continue + } + r, exists := runs[run] + if !exists { + r = &runObjects{} + runs[run] = r + } + r.keys = append(r.keys, key) + if obj.Size != nil { + r.size += *obj.Size + } + } + if output.IsTruncated == nil || !*output.IsTruncated { + break + } + token = output.NextContinuationToken + } + + if len(runs) == 0 { + r2RunsRetained = 0 + r2BytesRetained = 0 + return 0, nil + } + + // Run IDs are ISO timestamps, so lexicographic order equals chronological order. + sortedRuns := make([]string, 0, len(runs)) + for run := range runs { + sortedRuns = append(sortedRuns, run) + } + sort.Strings(sortedRuns) // ascending: oldest -> newest + + deleteRun := map[string]bool{} + keepRuns := r2KeepRuns() + + // Mark all but the newest keepRuns runs for deletion. + if len(sortedRuns) > keepRuns { + for _, run := range sortedRuns[:len(sortedRuns)-keepRuns] { + deleteRun[run] = true + } + } + + // Enforce the total size cap on retained runs (oldest retained first), always + // keeping the newest run even if it alone exceeds the cap. + maxBytes := r2MaxRetentionBytes() + retained := make([]string, 0, keepRuns) + var retainedBytes int64 + for _, run := range sortedRuns { + if !deleteRun[run] { + retained = append(retained, run) + retainedBytes += runs[run].size + } + } + for i := 0; retainedBytes > maxBytes && i < len(retained)-1; i++ { + run := retained[i] + deleteRun[run] = true + retainedBytes -= runs[run].size + } + + var toDelete []string + for run, del := range deleteRun { + if del { + toDelete = append(toDelete, runs[run].keys...) + } + } + + deleted, err := deleteR2Keys(toDelete) + if err != nil { + return deleted, err + } + + keptRuns := int64(0) + keptBytes := int64(0) + for run, r := range runs { + if !deleteRun[run] { + keptRuns++ + keptBytes += r.size + } + } + r2RunsRetained = keptRuns + r2BytesRetained = keptBytes r2CleanupObjectsTotal += int64(deleted) return deleted, nil @@ -918,13 +1037,13 @@ func cleanupOldSegments(ctx context.Context) (int, error) { // runR2CleanupLoop runs retention cleanup once at startup, then hourly. func runR2CleanupLoop() { - if _, err := cleanupOldSegments(context.Background()); err != nil { + if _, err := cleanupOldRuns(context.Background()); err != nil { fmt.Println("R2 cleanup error:", err) } ticker := time.NewTicker(time.Hour) defer ticker.Stop() for range ticker.C { - if _, err := cleanupOldSegments(context.Background()); err != nil { + if _, err := cleanupOldRuns(context.Background()); err != nil { fmt.Println("R2 cleanup error:", err) } } diff --git a/backend/playout/src/playlist_test.go b/backend/playout/src/playlist_test.go index 1851108..bbd5176 100644 --- a/backend/playout/src/playlist_test.go +++ b/backend/playout/src/playlist_test.go @@ -593,54 +593,95 @@ func TestSyncToR2_NoCacheHeadersForUnknownExt(t *testing.T) { } } -func TestR2SegmentRetentionHours(t *testing.T) { - os.Setenv("R2_SEGMENT_RETENTION_HOURS", "") - if got := r2SegmentRetentionHours(); got != 24*time.Hour { - t.Errorf("default = %v, want 24h", got) +func TestR2KeepRuns(t *testing.T) { + os.Setenv("R2_KEEP_RUNS", "") + if got := r2KeepRuns(); got != 2 { + t.Errorf("default = %d, want 2", got) } - os.Setenv("R2_SEGMENT_RETENTION_HOURS", "48") - if got := r2SegmentRetentionHours(); got != 48*time.Hour { - t.Errorf("override = %v, want 48h", got) + os.Setenv("R2_KEEP_RUNS", "5") + if got := r2KeepRuns(); got != 5 { + t.Errorf("override = %d, want 5", got) } - os.Setenv("R2_SEGMENT_RETENTION_HOURS", "invalid") - if got := r2SegmentRetentionHours(); got != 24*time.Hour { - t.Errorf("invalid = %v, want 24h fallback", got) + os.Setenv("R2_KEEP_RUNS", "0") + if got := r2KeepRuns(); got != 2 { + t.Errorf("invalid = %d, want 2 fallback", got) } - os.Unsetenv("R2_SEGMENT_RETENTION_HOURS") + os.Unsetenv("R2_KEEP_RUNS") } -func TestCleanupOldSegments(t *testing.T) { +func TestR2MaxRetentionBytes(t *testing.T) { + os.Setenv("R2_MAX_RETENTION_BYTES", "") + if got := r2MaxRetentionBytes(); got != 10*1024*1024*1024 { + t.Errorf("default = %d, want 10GiB", got) + } + + os.Setenv("R2_MAX_RETENTION_BYTES", "100") + if got := r2MaxRetentionBytes(); got != 100 { + t.Errorf("override = %d, want 100", got) + } + + os.Setenv("R2_MAX_RETENTION_BYTES", "invalid") + if got := r2MaxRetentionBytes(); got != 10*1024*1024*1024 { + t.Errorf("invalid = %d, want 10GiB fallback", got) + } + + os.Unsetenv("R2_MAX_RETENTION_BYTES") +} + +func TestR2RunFromKey(t *testing.T) { + cases := []struct { + key string + run string + ok bool + }{ + {"hls/r-20260815T120000Z/seg-1.m4s", "r-20260815T120000Z", true}, + {"hls/r-20260815T120000Z/init.mp4", "r-20260815T120000Z", true}, + {"hls/playlist.m3u8", "", false}, + {"hls/playlist.m3u8.tmp", "", false}, + {"other/r-x/seg.m4s", "", false}, + {"hls/r-noslash", "", false}, + } + for _, c := range cases { + run, ok := r2RunFromKey(c.key) + if run != c.run || ok != c.ok { + t.Errorf("r2RunFromKey(%q) = (%q, %v), want (%q, %v)", c.key, run, ok, c.run, c.ok) + } + } +} + +func TestCleanupOldRuns(t *testing.T) { prevClient := r2Client prevBucket := bucket prevList := listObjects prevDelete := deleteObjects prevTotal := r2CleanupObjectsTotal + prevRuns := r2RunsRetained + prevBytes := r2BytesRetained defer func() { r2Client = prevClient bucket = prevBucket listObjects = prevList deleteObjects = prevDelete r2CleanupObjectsTotal = prevTotal + r2RunsRetained = prevRuns + r2BytesRetained = prevBytes }() bucket = "linearvoid-media" r2Client = &s3.Client{} - now := time.Now().UTC() - old := now.Add(-25 * time.Hour) - recent := now.Add(-1 * time.Hour) - + size := int64(1024) listObjects = func(_ *s3.Client, input *s3.ListObjectsV2Input) (*s3.ListObjectsV2Output, error) { return &s3.ListObjectsV2Output{ Contents: []types.Object{ - {Key: aws.String("hls/r-old/seg-1.m4s"), LastModified: &old}, - {Key: aws.String("hls/r-old/init.mp4"), LastModified: &old}, - {Key: aws.String("hls/r-new/seg-2.m4s"), LastModified: &recent}, - {Key: aws.String("hls/playlist.m3u8"), LastModified: &old}, - {Key: aws.String("hls/playlist.m3u8.tmp"), LastModified: &old}, + {Key: aws.String("hls/r-20260813T000000Z/seg-1.m4s"), Size: &size}, + {Key: aws.String("hls/r-20260814T000000Z/seg-2.m4s"), Size: &size}, + {Key: aws.String("hls/r-20260815T000000Z/seg-3.m4s"), Size: &size}, + {Key: aws.String("hls/playlist.m3u8"), Size: &size}, + {Key: aws.String("hls/playlist.m3u8.tmp"), Size: &size}, }, IsTruncated: aws.Bool(false), }, nil @@ -654,23 +695,101 @@ func TestCleanupOldSegments(t *testing.T) { return &s3.DeleteObjectsOutput{}, nil } - n, err := cleanupOldSegments(context.Background()) + n, err := cleanupOldRuns(context.Background()) if err != nil { - t.Fatalf("cleanupOldSegments error: %v", err) + t.Fatalf("cleanupOldRuns error: %v", err) } + if n != 1 { + t.Fatalf("deleted = %d, want 1 (oldest run)", n) + } + + joined := strings.Join(deletedKeys, ",") + if !strings.Contains(joined, "hls/r-20260813T000000Z/seg-1.m4s") { + t.Fatalf("expected oldest run deleted, got: %v", deletedKeys) + } + if strings.Contains(joined, "playlist.m3u8") { + t.Fatalf("manifest must not be deleted: %v", deletedKeys) + } + if r2RunsRetained != 2 { + t.Fatalf("r2RunsRetained = %d, want 2", r2RunsRetained) + } + if r2BytesRetained != 2*size { + t.Fatalf("r2BytesRetained = %d, want %d", r2BytesRetained, 2*size) + } + if r2CleanupObjectsTotal != 1 { + t.Fatalf("r2CleanupObjectsTotal = %d, want 1", r2CleanupObjectsTotal) + } +} + +func TestCleanupOldRuns_SizeCap(t *testing.T) { + prevClient := r2Client + prevBucket := bucket + prevList := listObjects + prevDelete := deleteObjects + prevTotal := r2CleanupObjectsTotal + prevRuns := r2RunsRetained + prevBytes := r2BytesRetained + defer func() { + r2Client = prevClient + bucket = prevBucket + listObjects = prevList + deleteObjects = prevDelete + r2CleanupObjectsTotal = prevTotal + r2RunsRetained = prevRuns + r2BytesRetained = prevBytes + }() + + bucket = "linearvoid-media" + r2Client = &s3.Client{} + + // Force a small cap so the size policy trims even the last 2 runs. + os.Setenv("R2_KEEP_RUNS", "2") + os.Setenv("R2_MAX_RETENTION_BYTES", "1024") + defer os.Unsetenv("R2_KEEP_RUNS") + defer os.Unsetenv("R2_MAX_RETENTION_BYTES") + + size := int64(1024) + listObjects = func(_ *s3.Client, input *s3.ListObjectsV2Input) (*s3.ListObjectsV2Output, error) { + return &s3.ListObjectsV2Output{ + Contents: []types.Object{ + {Key: aws.String("hls/r-20260813T000000Z/seg-1.m4s"), Size: &size}, + {Key: aws.String("hls/r-20260814T000000Z/seg-2.m4s"), Size: &size}, + {Key: aws.String("hls/r-20260815T000000Z/seg-3.m4s"), Size: &size}, + }, + IsTruncated: aws.Bool(false), + }, nil + } + + var deletedKeys []string + deleteObjects = func(_ *s3.Client, input *s3.DeleteObjectsInput) (*s3.DeleteObjectsOutput, error) { + for _, obj := range input.Delete.Objects { + deletedKeys = append(deletedKeys, aws.ToString(obj.Key)) + } + return &s3.DeleteObjectsOutput{}, nil + } + + n, err := cleanupOldRuns(context.Background()) + if err != nil { + t.Fatalf("cleanupOldRuns error: %v", err) + } + // 3 runs -> keep 2 -> oldest deleted by count; remaining 2 bytes (2048) exceed + // the 1024 cap -> the older of the kept two is deleted, leaving only the newest. if n != 2 { t.Fatalf("deleted = %d, want 2", n) } joined := strings.Join(deletedKeys, ",") - if !strings.Contains(joined, "hls/r-old/seg-1.m4s") || !strings.Contains(joined, "hls/r-old/init.mp4") { - t.Fatalf("unexpected deleted keys: %v", deletedKeys) + if !strings.Contains(joined, "hls/r-20260813T000000Z/seg-1.m4s") { + t.Fatalf("expected oldest run deleted, got: %v", deletedKeys) } - if strings.Contains(joined, "playlist.m3u8") { - t.Fatalf("manifest must not be deleted: %v", deletedKeys) + if !strings.Contains(joined, "hls/r-20260814T000000Z/seg-2.m4s") { + t.Fatalf("expected second-oldest run deleted by size cap, got: %v", deletedKeys) } - if r2CleanupObjectsTotal != 2 { - t.Fatalf("r2CleanupObjectsTotal = %d, want 2", r2CleanupObjectsTotal) + if strings.Contains(joined, "hls/r-20260815T000000Z/seg-3.m4s") { + t.Fatalf("newest run must be kept, got: %v", deletedKeys) + } + if r2RunsRetained != 1 { + t.Fatalf("r2RunsRetained = %d, want 1", r2RunsRetained) } } diff --git a/docs/STREAMING.md b/docs/STREAMING.md index c2ef344..3e364f8 100644 --- a/docs/STREAMING.md +++ b/docs/STREAMING.md @@ -180,6 +180,8 @@ A release is streaming-ready only when one full playlist boundary has played wit | `R2_ENDPOINT`, `R2_REGION` | S3-compatible publication endpoint | | `R2_BUCKET_NAME` | Media bucket | | `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY` | Write credentials, available only to the publisher | +| `R2_KEEP_RUNS` | Number of run folders to retain (default `2`) | +| `R2_MAX_RETENTION_BYTES` | Max bytes retained across run folders (10 GiB) | | `MEDIA_DOMAIN` | Public R2 custom domain | | `VITE_PUBLIC_HLS_URL` | Public live manifest URL | diff --git a/docs/architecture/README.md b/docs/architecture/README.md index db4ae58..d75ad30 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -311,7 +311,7 @@ graph TB - **Resilience**: Restart counter (>5/60s → activate slate), Prometheus metrics on port 8081 - **Validation**: ffprobe gate on all ingested files (codec, fps, resolution, keyframe interval); `-validate` mode checks the playlist without running the loop - **Seamless looping**: Only resets HLS output when the playlist fingerprint (SHA-256) changes, on crash/slate recovery, or first run; otherwise continues segment numbering via `-start_number` -- **R2 retention**: `cleanupOldSegments` deletes `hls/r-*/*` objects older than `R2_SEGMENT_RETENTION_HOURS` (default 24h) hourly + at startup +- **R2 retention**: `cleanupOldRuns` keeps the newest `R2_KEEP_RUNS` run folders (default 2) and caps retained data at `R2_MAX_RETENTION_BYTES` (default 10 GiB), triggered after each run plus an hourly backstop - **Shared config**: Reads the runtime `config.json` (`loadSharedHLSConfig`) as an env fallback for HLS segment duration/window #### Chat Service (`backend/chat/src/main.go`) @@ -365,7 +365,7 @@ Module-level breakdown per service. | **Loop Controller** | `playlist.go` (playlistFingerprint, hlsNextStartNumber) | Detect playlist changes (SHA-256), continue segment numbering on clean loops | | **Slate Controller** | `playlist.go` (activateSlate, deactivateSlate) | Switch output to slate loop on excessive failures | | **R2 Sync** | `playlist.go` (syncToR2) | Upload HLS segments + manifest to R2 bucket | -| **R2 Cleanup** | `playlist.go` (cleanupOldSegments) | Delete stale `hls/r-*/*` segments older than the retention window | +| **R2 Cleanup** | `playlist.go` (cleanupOldRuns) | Keep the last N run folders under the size cap; delete older `hls/r-*/*` objects | | **Health Endpoint** | `playlist.go` (healthHandler) | `/health` → JSON with mode, segment count, uptime | | **Metrics Exporter** | `playlist.go` (prometheus) | Expose restarts_total, mode (live/slate), loops_total, run_duration_seconds, r2_cleanup_objects_total |