feat(playout): replace time-based R2 retention with run-count and size caps
CI / test (push) Successful in 1m33s
CI / docker (push) Successful in 51s

This commit is contained in:
zwitschi committed 2026-08-15 18:59:26 +02:00
1 parent 06b277e9c1
commit 6950bba85d
8 files changed
+325 -86

No files matched your search

+2 -1
View File
@@ -8,7 +8,8 @@ R2_SECRET_ACCESS_KEY=your-r2-secret-key
R2_BUCKET_NAME=linearvoid-media R2_BUCKET_NAME=linearvoid-media
R2_ENDPOINT=https://your-account-id.r2.cloudflarestorage.com R2_ENDPOINT=https://your-account-id.r2.cloudflarestorage.com
R2_REGION=auto R2_REGION=auto
R2_SEGMENT_RETENTION_HOURS=24 R2_KEEP_RUNS=2
R2_MAX_RETENTION_BYTES=10737418240
# Cloudflare (for custom domain, cache rules, Workers if used) # Cloudflare (for custom domain, cache rules, Workers if used)
CLOUDFLARE_API_TOKEN=your-cloudflare-api-token CLOUDFLARE_API_TOKEN=your-cloudflare-api-token
+1 -1
View File
@@ -138,7 +138,7 @@
## 2026-08-13 ## 2026-08-13
- Completed all remaining enhancement tasks (only manual/infrastructure phases left): - Completed all remaining enhancement tasks (only manual/infrastructure phases left):
- **Playout**: automatic media normalization on upload (`media.NormalizeVideo`, `*.normalized.mp4`, statuses `ready`/`needs_normalization`/`normalizing`/`normalization_failed`, `POST /api/admin/media/{id}/normalize`); R2 segment retention (`cleanupOldSegments`, `R2_SEGMENT_RETENTION_HOURS`); seamless looping (playlist SHA-256 fingerprint + `-start_number`); single `start-playout.sh` entrypoint + `-validate` mode; loop/cleanup metrics. - **Playout**: automatic media normalization on upload (`media.NormalizeVideo`, `*.normalized.mp4`, statuses `ready`/`needs_normalization`/`normalizing`/`normalization_failed`, `POST /api/admin/media/{id}/normalize`); R2 segment retention (`cleanupOldRuns`, keep last `R2_KEEP_RUNS` runs, `R2_MAX_RETENTION_BYTES` cap); seamless looping (playlist SHA-256 fingerprint + `-start_number`); single `start-playout.sh` entrypoint + `-validate` mode; loop/cleanup metrics.
- **Application configuration**: thread-safe JSON runtime config store (`backend/chat/src/config/runtime.go`) shared with playout via `/app/playlist/config.json`; `GET/PUT /api/admin/config`; admin UI config form. - **Application configuration**: thread-safe JSON runtime config store (`backend/chat/src/config/runtime.go`) shared with playout via `/app/playlist/config.json`; `GET/PUT /api/admin/config`; admin UI config form.
- **Matomo analytics**: public `GET /api/matomo` + viewer pageview snippet in `index.html`. - **Matomo analytics**: public `GET /api/matomo` + viewer pageview snippet in `index.html`.
- **Chat user accounts**: register/login/logout/session endpoints, bcrypt hashing, `linearvoid_user_session` cookie, viewer sign-in UI. - **Chat user accounts**: register/login/logout/session endpoints, bcrypt hashing, `linearvoid_user_session` cookie, viewer sign-in UI.
-4
View File
@@ -2,10 +2,6 @@
> Completed tasks moved to [DONE](./DONE.md). Remaining tasks in [BACKLOG](./BACKLOG.md). > Completed tasks moved to [DONE](./DONE.md). Remaining tasks in [BACKLOG](./BACKLOG.md).
## R2 Data Retention
- [ ] Adjust internal data retention for R2: remove folders /hls/r-{ISOTIME} after next run, keep only the last 2 runs or max. 10GB of data
## Frontend Tests ## Frontend Tests
### User Frontend ### User Frontend
+2
View File
@@ -14,12 +14,14 @@
- **Reconnect grace period**: Disconnects schedule a delayed "left" announcement (5s); a reconnect within 5s cancels it and skips "joined", eliminating join/leave spam on brief reconnects. - **Reconnect grace period**: Disconnects schedule a delayed "left" announcement (5s); a reconnect within 5s cancels it and skips "joined", eliminating join/leave spam on brief reconnects.
- **HLS segment duration 2s**: Default `HLS_SEGMENT_DURATION` changed from 6 to 2 and `HLS_WINDOW_SIZE` from 24 to 12 (2s × 12 = 24s live window), reducing end-to-end latency; the 2s GOP was already in place. - **HLS segment duration 2s**: Default `HLS_SEGMENT_DURATION` changed from 6 to 2 and `HLS_WINDOW_SIZE` from 24 to 12 (2s × 12 = 24s live window), reducing end-to-end latency; the 2s GOP was already in place.
- **Chat font sizing**: `.message-text` set to `0.95rem` (slightly smaller than the default, larger than the `0.83rem` system/flash text). - **Chat font sizing**: `.message-text` set to `0.95rem` (slightly smaller than the default, larger than the `0.83rem` system/flash text).
- **R2 retention by run count + size**: Playout now keeps the newest `R2_KEEP_RUNS` run folders (default 2) and enforces an `R2_MAX_RETENTION_BYTES` cap (default 10 GiB), replacing the old 24h time-based `R2_SEGMENT_RETENTION_HOURS` cleanup. Cleanup runs after each new run plus an hourly backstop, and never deletes the live manifest. Adds `linearvoid_playout_r2_runs_retained` and `linearvoid_playout_r2_bytes_retained` gauges.
### Tests ### Tests
- WebSocket: `TestHub_Broadcast_SystemNotPersisted`, `TestHub_ReconnectGracePeriod`. - WebSocket: `TestHub_Broadcast_SystemNotPersisted`, `TestHub_ReconnectGracePeriod`.
- Handlers: `TestHandleAdminChatLog_ExcludesSystem`. - Handlers: `TestHandleAdminChatLog_ExcludesSystem`.
- Frontend: `presence flash` test. - Frontend: `presence flash` test.
- Playout: `TestCleanupOldRuns`, `TestCleanupOldRuns_SizeCap`, `TestR2KeepRuns`, `TestR2MaxRetentionBytes`, `TestR2RunFromKey`.
## [1.0.13] - 2026-08-13 ## [1.0.13] - 2026-08-13
+168 -49
View File
@@ -11,6 +11,7 @@ import (
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"sort"
"strconv" "strconv"
"strings" "strings"
"time" "time"
@@ -66,6 +67,10 @@ var (
// R2 retention cleanup counter // R2 retention cleanup counter
r2CleanupObjectsTotal int64 r2CleanupObjectsTotal int64
// R2 retention gauges
r2RunsRetained int64
r2BytesRetained int64
// Continuous playout loop metrics // Continuous playout loop metrics
loopsTotal int64 loopsTotal int64
lastRunDuration time.Duration lastRunDuration time.Duration
@@ -78,6 +83,10 @@ const (
// Cache-Control header values for R2 uploads. // Cache-Control header values for R2 uploads.
segmentCacheControl = "public, max-age=31536000, immutable" segmentCacheControl = "public, max-age=31536000, immutable"
manifestCacheControl = "public, max-age=1, s-maxage=2, must-revalidate" manifestCacheControl = "public, max-age=1, s-maxage=2, must-revalidate"
// R2 retention policy.
defaultR2KeepRuns = 2
defaultR2MaxRetentionBytes = int64(10 * 1024 * 1024 * 1024) // 10 GiB
) )
// envWithDefault returns the value of the environment variable or the default if not set. // envWithDefault returns the value of the environment variable or the default if not set.
@@ -658,6 +667,12 @@ func main() {
fmt.Fprintln(w, "# HELP linearvoid_playout_r2_cleanup_objects_total Total R2 objects deleted by retention cleanup.") fmt.Fprintln(w, "# HELP linearvoid_playout_r2_cleanup_objects_total Total R2 objects deleted by retention cleanup.")
fmt.Fprintln(w, "# TYPE linearvoid_playout_r2_cleanup_objects_total counter") fmt.Fprintln(w, "# TYPE linearvoid_playout_r2_cleanup_objects_total counter")
fmt.Fprintf(w, "linearvoid_playout_r2_cleanup_objects_total %d\n", r2CleanupObjectsTotal) fmt.Fprintf(w, "linearvoid_playout_r2_cleanup_objects_total %d\n", r2CleanupObjectsTotal)
fmt.Fprintln(w, "# HELP linearvoid_playout_r2_runs_retained Number of R2 run folders retained by cleanup.")
fmt.Fprintln(w, "# TYPE linearvoid_playout_r2_runs_retained gauge")
fmt.Fprintf(w, "linearvoid_playout_r2_runs_retained %d\n", r2RunsRetained)
fmt.Fprintln(w, "# HELP linearvoid_playout_r2_bytes_retained Total bytes retained in R2 run folders.")
fmt.Fprintln(w, "# TYPE linearvoid_playout_r2_bytes_retained gauge")
fmt.Fprintf(w, "linearvoid_playout_r2_bytes_retained %d\n", r2BytesRetained)
fmt.Fprintln(w, "# HELP linearvoid_playout_loops_total Total clean playlist completions (loops).") fmt.Fprintln(w, "# HELP linearvoid_playout_loops_total Total clean playlist completions (loops).")
fmt.Fprintln(w, "# TYPE linearvoid_playout_loops_total counter") fmt.Fprintln(w, "# TYPE linearvoid_playout_loops_total counter")
fmt.Fprintf(w, "linearvoid_playout_loops_total %d\n", loopsTotal) fmt.Fprintf(w, "linearvoid_playout_loops_total %d\n", loopsTotal)
@@ -682,6 +697,15 @@ func main() {
// New run ID on each restart for unique R2 segment keys (STREAMING.md) // New run ID on each restart for unique R2 segment keys (STREAMING.md)
initRunID() initRunID()
// Remove run folders older than the retention policy (keep the last N runs,
// cap total bytes) now that the next run has started. This never touches the
// current or previous run folders.
if r2Client != nil {
if _, err := cleanupOldRuns(context.Background()); err != nil {
fmt.Println("R2 cleanup error:", err)
}
}
// Reset HLS output only when the playlist changed, we are recovering from a // Reset HLS output only when the playlist changed, we are recovering from a
// crash/slate, or this is the first run. Otherwise continue the segment // crash/slate, or this is the first run. Otherwise continue the segment
// sequence so the loop is seamless for viewers (no gap on clean loops). // sequence so the loop is seamless for viewers (no gap on clean loops).
@@ -834,68 +858,51 @@ func syncLatestToR2(hlsDir string) error {
return nil return nil
} }
// r2SegmentRetentionHours returns the R2 segment retention window in hours. // r2KeepRuns returns how many of the most recent run folders to keep in R2.
func r2SegmentRetentionHours() time.Duration { func r2KeepRuns() int {
h, err := strconv.Atoi(envWithDefault("R2_SEGMENT_RETENTION_HOURS", "24")) n, err := strconv.Atoi(envWithDefault("R2_KEEP_RUNS", "2"))
if err != nil || h <= 0 { if err != nil || n < 1 {
h = 24 return defaultR2KeepRuns
} }
return time.Duration(h) * time.Hour return n
} }
// cleanupOldSegments deletes R2 objects under the hls/ prefix that are older than // r2MaxRetentionBytes returns the maximum total bytes to retain across run folders.
// the retention window, excluding the live manifest and any non-run keys. func r2MaxRetentionBytes() int64 {
// It returns the number of objects deleted. n, err := strconv.ParseInt(envWithDefault("R2_MAX_RETENTION_BYTES", "10737418240"), 10, 64)
func cleanupOldSegments(ctx context.Context) (int, error) { if err != nil || n <= 0 {
if r2Client == nil { return defaultR2MaxRetentionBytes
if err := initR2Client(); err != nil {
return 0, err
} }
return n
} }
cutoff := time.Now().UTC().Add(-r2SegmentRetentionHours()) // r2RunFromKey extracts the run folder name ("r-...") from an R2 object key.
func r2RunFromKey(key string) (string, bool) {
var stale []string rest := strings.TrimPrefix(key, "hls/")
var token *string if rest == key || !strings.HasPrefix(rest, "r-") {
for { return "", false
output, err := listObjects(r2Client, &s3.ListObjectsV2Input{
Bucket: aws.String(bucket),
Prefix: aws.String("hls/"),
ContinuationToken: token,
})
if err != nil {
return 0, err
} }
for _, obj := range output.Contents { if idx := strings.IndexByte(rest, '/'); idx > 0 {
key := aws.ToString(obj.Key) return rest[:idx], true
if key == "hls/playlist.m3u8" || key == "hls/playlist.m3u8.tmp" {
continue
} }
// Only consider segment/init objects under a run directory: hls/r-.../... return "", false
if !strings.HasPrefix(key, "hls/r-") {
continue
}
if obj.LastModified != nil && obj.LastModified.Before(cutoff) {
stale = append(stale, key)
}
}
if output.IsTruncated == nil || !*output.IsTruncated {
break
}
token = output.NextContinuationToken
} }
if len(stale) == 0 { // runObjects groups the R2 object keys and total size for a single run folder.
return 0, nil type runObjects struct {
keys []string
size int64
} }
// deleteR2Keys deletes the given R2 object keys in batches of up to 1000.
func deleteR2Keys(keys []string) (int, error) {
deleted := 0 deleted := 0
for i := 0; i < len(stale); i += 1000 { for i := 0; i < len(keys); i += 1000 {
end := i + 1000 end := i + 1000
if end > len(stale) { if end > len(keys) {
end = len(stale) end = len(keys)
} }
batch := stale[i:end] batch := keys[i:end]
objects := make([]types.ObjectIdentifier, len(batch)) objects := make([]types.ObjectIdentifier, len(batch))
for j, key := range batch { for j, key := range batch {
objects[j] = types.ObjectIdentifier{Key: aws.String(key)} objects[j] = types.ObjectIdentifier{Key: aws.String(key)}
@@ -911,6 +918,118 @@ func cleanupOldSegments(ctx context.Context) (int, error) {
} }
deleted += len(batch) deleted += len(batch)
} }
return deleted, nil
}
// cleanupOldRuns lists run-scoped objects under hls/, keeps the most recent
// R2_KEEP_RUNS runs, and deletes older runs. It also enforces the
// R2_MAX_RETENTION_BYTES cap by removing oldest retained runs first, always
// keeping at least the newest run. The live manifest keys are never deleted.
func cleanupOldRuns(ctx context.Context) (int, error) {
if r2Client == nil {
if err := initR2Client(); err != nil {
return 0, err
}
}
runs := map[string]*runObjects{}
var token *string
for {
output, err := listObjects(r2Client, &s3.ListObjectsV2Input{
Bucket: aws.String(bucket),
Prefix: aws.String("hls/"),
ContinuationToken: token,
})
if err != nil {
return 0, err
}
for _, obj := range output.Contents {
key := aws.ToString(obj.Key)
if key == "hls/playlist.m3u8" || key == "hls/playlist.m3u8.tmp" {
continue
}
run, ok := r2RunFromKey(key)
if !ok {
continue
}
r, exists := runs[run]
if !exists {
r = &runObjects{}
runs[run] = r
}
r.keys = append(r.keys, key)
if obj.Size != nil {
r.size += *obj.Size
}
}
if output.IsTruncated == nil || !*output.IsTruncated {
break
}
token = output.NextContinuationToken
}
if len(runs) == 0 {
r2RunsRetained = 0
r2BytesRetained = 0
return 0, nil
}
// Run IDs are ISO timestamps, so lexicographic order equals chronological order.
sortedRuns := make([]string, 0, len(runs))
for run := range runs {
sortedRuns = append(sortedRuns, run)
}
sort.Strings(sortedRuns) // ascending: oldest -> newest
deleteRun := map[string]bool{}
keepRuns := r2KeepRuns()
// Mark all but the newest keepRuns runs for deletion.
if len(sortedRuns) > keepRuns {
for _, run := range sortedRuns[:len(sortedRuns)-keepRuns] {
deleteRun[run] = true
}
}
// Enforce the total size cap on retained runs (oldest retained first), always
// keeping the newest run even if it alone exceeds the cap.
maxBytes := r2MaxRetentionBytes()
retained := make([]string, 0, keepRuns)
var retainedBytes int64
for _, run := range sortedRuns {
if !deleteRun[run] {
retained = append(retained, run)
retainedBytes += runs[run].size
}
}
for i := 0; retainedBytes > maxBytes && i < len(retained)-1; i++ {
run := retained[i]
deleteRun[run] = true
retainedBytes -= runs[run].size
}
var toDelete []string
for run, del := range deleteRun {
if del {
toDelete = append(toDelete, runs[run].keys...)
}
}
deleted, err := deleteR2Keys(toDelete)
if err != nil {
return deleted, err
}
keptRuns := int64(0)
keptBytes := int64(0)
for run, r := range runs {
if !deleteRun[run] {
keptRuns++
keptBytes += r.size
}
}
r2RunsRetained = keptRuns
r2BytesRetained = keptBytes
r2CleanupObjectsTotal += int64(deleted) r2CleanupObjectsTotal += int64(deleted)
return deleted, nil return deleted, nil
@@ -918,13 +1037,13 @@ func cleanupOldSegments(ctx context.Context) (int, error) {
// runR2CleanupLoop runs retention cleanup once at startup, then hourly. // runR2CleanupLoop runs retention cleanup once at startup, then hourly.
func runR2CleanupLoop() { func runR2CleanupLoop() {
if _, err := cleanupOldSegments(context.Background()); err != nil { if _, err := cleanupOldRuns(context.Background()); err != nil {
fmt.Println("R2 cleanup error:", err) fmt.Println("R2 cleanup error:", err)
} }
ticker := time.NewTicker(time.Hour) ticker := time.NewTicker(time.Hour)
defer ticker.Stop() defer ticker.Stop()
for range ticker.C { for range ticker.C {
if _, err := cleanupOldSegments(context.Background()); err != nil { if _, err := cleanupOldRuns(context.Background()); err != nil {
fmt.Println("R2 cleanup error:", err) fmt.Println("R2 cleanup error:", err)
} }
} }
+148 -29
View File
@@ -593,54 +593,95 @@ func TestSyncToR2_NoCacheHeadersForUnknownExt(t *testing.T) {
} }
} }
func TestR2SegmentRetentionHours(t *testing.T) { func TestR2KeepRuns(t *testing.T) {
os.Setenv("R2_SEGMENT_RETENTION_HOURS", "") os.Setenv("R2_KEEP_RUNS", "")
if got := r2SegmentRetentionHours(); got != 24*time.Hour { if got := r2KeepRuns(); got != 2 {
t.Errorf("default = %v, want 24h", got) t.Errorf("default = %d, want 2", got)
} }
os.Setenv("R2_SEGMENT_RETENTION_HOURS", "48") os.Setenv("R2_KEEP_RUNS", "5")
if got := r2SegmentRetentionHours(); got != 48*time.Hour { if got := r2KeepRuns(); got != 5 {
t.Errorf("override = %v, want 48h", got) t.Errorf("override = %d, want 5", got)
} }
os.Setenv("R2_SEGMENT_RETENTION_HOURS", "invalid") os.Setenv("R2_KEEP_RUNS", "0")
if got := r2SegmentRetentionHours(); got != 24*time.Hour { if got := r2KeepRuns(); got != 2 {
t.Errorf("invalid = %v, want 24h fallback", got) t.Errorf("invalid = %d, want 2 fallback", got)
} }
os.Unsetenv("R2_SEGMENT_RETENTION_HOURS") os.Unsetenv("R2_KEEP_RUNS")
} }
func TestCleanupOldSegments(t *testing.T) { func TestR2MaxRetentionBytes(t *testing.T) {
os.Setenv("R2_MAX_RETENTION_BYTES", "")
if got := r2MaxRetentionBytes(); got != 10*1024*1024*1024 {
t.Errorf("default = %d, want 10GiB", got)
}
os.Setenv("R2_MAX_RETENTION_BYTES", "100")
if got := r2MaxRetentionBytes(); got != 100 {
t.Errorf("override = %d, want 100", got)
}
os.Setenv("R2_MAX_RETENTION_BYTES", "invalid")
if got := r2MaxRetentionBytes(); got != 10*1024*1024*1024 {
t.Errorf("invalid = %d, want 10GiB fallback", got)
}
os.Unsetenv("R2_MAX_RETENTION_BYTES")
}
func TestR2RunFromKey(t *testing.T) {
cases := []struct {
key string
run string
ok bool
}{
{"hls/r-20260815T120000Z/seg-1.m4s", "r-20260815T120000Z", true},
{"hls/r-20260815T120000Z/init.mp4", "r-20260815T120000Z", true},
{"hls/playlist.m3u8", "", false},
{"hls/playlist.m3u8.tmp", "", false},
{"other/r-x/seg.m4s", "", false},
{"hls/r-noslash", "", false},
}
for _, c := range cases {
run, ok := r2RunFromKey(c.key)
if run != c.run || ok != c.ok {
t.Errorf("r2RunFromKey(%q) = (%q, %v), want (%q, %v)", c.key, run, ok, c.run, c.ok)
}
}
}
func TestCleanupOldRuns(t *testing.T) {
prevClient := r2Client prevClient := r2Client
prevBucket := bucket prevBucket := bucket
prevList := listObjects prevList := listObjects
prevDelete := deleteObjects prevDelete := deleteObjects
prevTotal := r2CleanupObjectsTotal prevTotal := r2CleanupObjectsTotal
prevRuns := r2RunsRetained
prevBytes := r2BytesRetained
defer func() { defer func() {
r2Client = prevClient r2Client = prevClient
bucket = prevBucket bucket = prevBucket
listObjects = prevList listObjects = prevList
deleteObjects = prevDelete deleteObjects = prevDelete
r2CleanupObjectsTotal = prevTotal r2CleanupObjectsTotal = prevTotal
r2RunsRetained = prevRuns
r2BytesRetained = prevBytes
}() }()
bucket = "linearvoid-media" bucket = "linearvoid-media"
r2Client = &s3.Client{} r2Client = &s3.Client{}
now := time.Now().UTC() size := int64(1024)
old := now.Add(-25 * time.Hour)
recent := now.Add(-1 * time.Hour)
listObjects = func(_ *s3.Client, input *s3.ListObjectsV2Input) (*s3.ListObjectsV2Output, error) { listObjects = func(_ *s3.Client, input *s3.ListObjectsV2Input) (*s3.ListObjectsV2Output, error) {
return &s3.ListObjectsV2Output{ return &s3.ListObjectsV2Output{
Contents: []types.Object{ Contents: []types.Object{
{Key: aws.String("hls/r-old/seg-1.m4s"), LastModified: &old}, {Key: aws.String("hls/r-20260813T000000Z/seg-1.m4s"), Size: &size},
{Key: aws.String("hls/r-old/init.mp4"), LastModified: &old}, {Key: aws.String("hls/r-20260814T000000Z/seg-2.m4s"), Size: &size},
{Key: aws.String("hls/r-new/seg-2.m4s"), LastModified: &recent}, {Key: aws.String("hls/r-20260815T000000Z/seg-3.m4s"), Size: &size},
{Key: aws.String("hls/playlist.m3u8"), LastModified: &old}, {Key: aws.String("hls/playlist.m3u8"), Size: &size},
{Key: aws.String("hls/playlist.m3u8.tmp"), LastModified: &old}, {Key: aws.String("hls/playlist.m3u8.tmp"), Size: &size},
}, },
IsTruncated: aws.Bool(false), IsTruncated: aws.Bool(false),
}, nil }, nil
@@ -654,23 +695,101 @@ func TestCleanupOldSegments(t *testing.T) {
return &s3.DeleteObjectsOutput{}, nil return &s3.DeleteObjectsOutput{}, nil
} }
n, err := cleanupOldSegments(context.Background()) n, err := cleanupOldRuns(context.Background())
if err != nil { if err != nil {
t.Fatalf("cleanupOldSegments error: %v", err) t.Fatalf("cleanupOldRuns error: %v", err)
} }
if n != 1 {
t.Fatalf("deleted = %d, want 1 (oldest run)", n)
}
joined := strings.Join(deletedKeys, ",")
if !strings.Contains(joined, "hls/r-20260813T000000Z/seg-1.m4s") {
t.Fatalf("expected oldest run deleted, got: %v", deletedKeys)
}
if strings.Contains(joined, "playlist.m3u8") {
t.Fatalf("manifest must not be deleted: %v", deletedKeys)
}
if r2RunsRetained != 2 {
t.Fatalf("r2RunsRetained = %d, want 2", r2RunsRetained)
}
if r2BytesRetained != 2*size {
t.Fatalf("r2BytesRetained = %d, want %d", r2BytesRetained, 2*size)
}
if r2CleanupObjectsTotal != 1 {
t.Fatalf("r2CleanupObjectsTotal = %d, want 1", r2CleanupObjectsTotal)
}
}
func TestCleanupOldRuns_SizeCap(t *testing.T) {
prevClient := r2Client
prevBucket := bucket
prevList := listObjects
prevDelete := deleteObjects
prevTotal := r2CleanupObjectsTotal
prevRuns := r2RunsRetained
prevBytes := r2BytesRetained
defer func() {
r2Client = prevClient
bucket = prevBucket
listObjects = prevList
deleteObjects = prevDelete
r2CleanupObjectsTotal = prevTotal
r2RunsRetained = prevRuns
r2BytesRetained = prevBytes
}()
bucket = "linearvoid-media"
r2Client = &s3.Client{}
// Force a small cap so the size policy trims even the last 2 runs.
os.Setenv("R2_KEEP_RUNS", "2")
os.Setenv("R2_MAX_RETENTION_BYTES", "1024")
defer os.Unsetenv("R2_KEEP_RUNS")
defer os.Unsetenv("R2_MAX_RETENTION_BYTES")
size := int64(1024)
listObjects = func(_ *s3.Client, input *s3.ListObjectsV2Input) (*s3.ListObjectsV2Output, error) {
return &s3.ListObjectsV2Output{
Contents: []types.Object{
{Key: aws.String("hls/r-20260813T000000Z/seg-1.m4s"), Size: &size},
{Key: aws.String("hls/r-20260814T000000Z/seg-2.m4s"), Size: &size},
{Key: aws.String("hls/r-20260815T000000Z/seg-3.m4s"), Size: &size},
},
IsTruncated: aws.Bool(false),
}, nil
}
var deletedKeys []string
deleteObjects = func(_ *s3.Client, input *s3.DeleteObjectsInput) (*s3.DeleteObjectsOutput, error) {
for _, obj := range input.Delete.Objects {
deletedKeys = append(deletedKeys, aws.ToString(obj.Key))
}
return &s3.DeleteObjectsOutput{}, nil
}
n, err := cleanupOldRuns(context.Background())
if err != nil {
t.Fatalf("cleanupOldRuns error: %v", err)
}
// 3 runs -> keep 2 -> oldest deleted by count; remaining 2 bytes (2048) exceed
// the 1024 cap -> the older of the kept two is deleted, leaving only the newest.
if n != 2 { if n != 2 {
t.Fatalf("deleted = %d, want 2", n) t.Fatalf("deleted = %d, want 2", n)
} }
joined := strings.Join(deletedKeys, ",") joined := strings.Join(deletedKeys, ",")
if !strings.Contains(joined, "hls/r-old/seg-1.m4s") || !strings.Contains(joined, "hls/r-old/init.mp4") { if !strings.Contains(joined, "hls/r-20260813T000000Z/seg-1.m4s") {
t.Fatalf("unexpected deleted keys: %v", deletedKeys) t.Fatalf("expected oldest run deleted, got: %v", deletedKeys)
} }
if strings.Contains(joined, "playlist.m3u8") { if !strings.Contains(joined, "hls/r-20260814T000000Z/seg-2.m4s") {
t.Fatalf("manifest must not be deleted: %v", deletedKeys) t.Fatalf("expected second-oldest run deleted by size cap, got: %v", deletedKeys)
} }
if r2CleanupObjectsTotal != 2 { if strings.Contains(joined, "hls/r-20260815T000000Z/seg-3.m4s") {
t.Fatalf("r2CleanupObjectsTotal = %d, want 2", r2CleanupObjectsTotal) t.Fatalf("newest run must be kept, got: %v", deletedKeys)
}
if r2RunsRetained != 1 {
t.Fatalf("r2RunsRetained = %d, want 1", r2RunsRetained)
} }
} }
+2
View File
@@ -180,6 +180,8 @@ A release is streaming-ready only when one full playlist boundary has played wit
| `R2_ENDPOINT`, `R2_REGION` | S3-compatible publication endpoint | | `R2_ENDPOINT`, `R2_REGION` | S3-compatible publication endpoint |
| `R2_BUCKET_NAME` | Media bucket | | `R2_BUCKET_NAME` | Media bucket |
| `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY` | Write credentials, available only to the publisher | | `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY` | Write credentials, available only to the publisher |
| `R2_KEEP_RUNS` | Number of run folders to retain (default `2`) |
| `R2_MAX_RETENTION_BYTES` | Max bytes retained across run folders (10 GiB) |
| `MEDIA_DOMAIN` | Public R2 custom domain | | `MEDIA_DOMAIN` | Public R2 custom domain |
| `VITE_PUBLIC_HLS_URL` | Public live manifest URL | | `VITE_PUBLIC_HLS_URL` | Public live manifest URL |
+2 -2
View File
@@ -311,7 +311,7 @@ graph TB
- **Resilience**: Restart counter (>5/60s → activate slate), Prometheus metrics on port 8081 - **Resilience**: Restart counter (>5/60s → activate slate), Prometheus metrics on port 8081
- **Validation**: ffprobe gate on all ingested files (codec, fps, resolution, keyframe interval); `-validate` mode checks the playlist without running the loop - **Validation**: ffprobe gate on all ingested files (codec, fps, resolution, keyframe interval); `-validate` mode checks the playlist without running the loop
- **Seamless looping**: Only resets HLS output when the playlist fingerprint (SHA-256) changes, on crash/slate recovery, or first run; otherwise continues segment numbering via `-start_number` - **Seamless looping**: Only resets HLS output when the playlist fingerprint (SHA-256) changes, on crash/slate recovery, or first run; otherwise continues segment numbering via `-start_number`
- **R2 retention**: `cleanupOldSegments` deletes `hls/r-*/*` objects older than `R2_SEGMENT_RETENTION_HOURS` (default 24h) hourly + at startup - **R2 retention**: `cleanupOldRuns` keeps the newest `R2_KEEP_RUNS` run folders (default 2) and caps retained data at `R2_MAX_RETENTION_BYTES` (default 10 GiB), triggered after each run plus an hourly backstop
- **Shared config**: Reads the runtime `config.json` (`loadSharedHLSConfig`) as an env fallback for HLS segment duration/window - **Shared config**: Reads the runtime `config.json` (`loadSharedHLSConfig`) as an env fallback for HLS segment duration/window
#### Chat Service (`backend/chat/src/main.go`) #### Chat Service (`backend/chat/src/main.go`)
@@ -365,7 +365,7 @@ Module-level breakdown per service.
| **Loop Controller** | `playlist.go` (playlistFingerprint, hlsNextStartNumber) | Detect playlist changes (SHA-256), continue segment numbering on clean loops | | **Loop Controller** | `playlist.go` (playlistFingerprint, hlsNextStartNumber) | Detect playlist changes (SHA-256), continue segment numbering on clean loops |
| **Slate Controller** | `playlist.go` (activateSlate, deactivateSlate) | Switch output to slate loop on excessive failures | | **Slate Controller** | `playlist.go` (activateSlate, deactivateSlate) | Switch output to slate loop on excessive failures |
| **R2 Sync** | `playlist.go` (syncToR2) | Upload HLS segments + manifest to R2 bucket | | **R2 Sync** | `playlist.go` (syncToR2) | Upload HLS segments + manifest to R2 bucket |
| **R2 Cleanup** | `playlist.go` (cleanupOldSegments) | Delete stale `hls/r-*/*` segments older than the retention window | | **R2 Cleanup** | `playlist.go` (cleanupOldRuns) | Keep the last N run folders under the size cap; delete older `hls/r-*/*` objects |
| **Health Endpoint** | `playlist.go` (healthHandler) | `/health` → JSON with mode, segment count, uptime | | **Health Endpoint** | `playlist.go` (healthHandler) | `/health` → JSON with mode, segment count, uptime |
| **Metrics Exporter** | `playlist.go` (prometheus) | Expose restarts_total, mode (live/slate), loops_total, run_duration_seconds, r2_cleanup_objects_total | | **Metrics Exporter** | `playlist.go` (prometheus) | Expose restarts_total, mode (live/slate), loops_total, run_duration_seconds, r2_cleanup_objects_total |