feat(playout): replace time-based R2 retention with run-count and size caps
This commit is contained in:
1 parent
06b277e9c1
commit
6950bba85d
8 files changed
+329
-90
No files matched your search
+2
-1
@@ -8,7 +8,8 @@ R2_SECRET_ACCESS_KEY=your-r2-secret-key
|
|||||||
R2_BUCKET_NAME=linearvoid-media
|
R2_BUCKET_NAME=linearvoid-media
|
||||||
R2_ENDPOINT=https://your-account-id.r2.cloudflarestorage.com
|
R2_ENDPOINT=https://your-account-id.r2.cloudflarestorage.com
|
||||||
R2_REGION=auto
|
R2_REGION=auto
|
||||||
R2_SEGMENT_RETENTION_HOURS=24
|
R2_KEEP_RUNS=2
|
||||||
|
R2_MAX_RETENTION_BYTES=10737418240
|
||||||
|
|
||||||
# Cloudflare (for custom domain, cache rules, Workers if used)
|
# Cloudflare (for custom domain, cache rules, Workers if used)
|
||||||
CLOUDFLARE_API_TOKEN=your-cloudflare-api-token
|
CLOUDFLARE_API_TOKEN=your-cloudflare-api-token
|
||||||
|
|||||||
+1
-1
@@ -138,7 +138,7 @@
|
|||||||
## 2026-08-13
|
## 2026-08-13
|
||||||
|
|
||||||
- Completed all remaining enhancement tasks (only manual/infrastructure phases left):
|
- Completed all remaining enhancement tasks (only manual/infrastructure phases left):
|
||||||
- **Playout**: automatic media normalization on upload (`media.NormalizeVideo`, `*.normalized.mp4`, statuses `ready`/`needs_normalization`/`normalizing`/`normalization_failed`, `POST /api/admin/media/{id}/normalize`); R2 segment retention (`cleanupOldSegments`, `R2_SEGMENT_RETENTION_HOURS`); seamless looping (playlist SHA-256 fingerprint + `-start_number`); single `start-playout.sh` entrypoint + `-validate` mode; loop/cleanup metrics.
|
- **Playout**: automatic media normalization on upload (`media.NormalizeVideo`, `*.normalized.mp4`, statuses `ready`/`needs_normalization`/`normalizing`/`normalization_failed`, `POST /api/admin/media/{id}/normalize`); R2 segment retention (`cleanupOldRuns`, keep last `R2_KEEP_RUNS` runs, `R2_MAX_RETENTION_BYTES` cap); seamless looping (playlist SHA-256 fingerprint + `-start_number`); single `start-playout.sh` entrypoint + `-validate` mode; loop/cleanup metrics.
|
||||||
- **Application configuration**: thread-safe JSON runtime config store (`backend/chat/src/config/runtime.go`) shared with playout via `/app/playlist/config.json`; `GET/PUT /api/admin/config`; admin UI config form.
|
- **Application configuration**: thread-safe JSON runtime config store (`backend/chat/src/config/runtime.go`) shared with playout via `/app/playlist/config.json`; `GET/PUT /api/admin/config`; admin UI config form.
|
||||||
- **Matomo analytics**: public `GET /api/matomo` + viewer pageview snippet in `index.html`.
|
- **Matomo analytics**: public `GET /api/matomo` + viewer pageview snippet in `index.html`.
|
||||||
- **Chat user accounts**: register/login/logout/session endpoints, bcrypt hashing, `linearvoid_user_session` cookie, viewer sign-in UI.
|
- **Chat user accounts**: register/login/logout/session endpoints, bcrypt hashing, `linearvoid_user_session` cookie, viewer sign-in UI.
|
||||||
|
|||||||
@@ -2,10 +2,6 @@
|
|||||||
|
|
||||||
> Completed tasks moved to [DONE](./DONE.md). Remaining tasks in [BACKLOG](./BACKLOG.md).
|
> Completed tasks moved to [DONE](./DONE.md). Remaining tasks in [BACKLOG](./BACKLOG.md).
|
||||||
|
|
||||||
## R2 Data Retention
|
|
||||||
|
|
||||||
- [ ] Adjust internal data retention for R2: remove folders /hls/r-{ISOTIME} after next run, keep only the last 2 runs or max. 10GB of data
|
|
||||||
|
|
||||||
## Frontend Tests
|
## Frontend Tests
|
||||||
|
|
||||||
### User Frontend
|
### User Frontend
|
||||||
|
|||||||
@@ -14,12 +14,14 @@
|
|||||||
- **Reconnect grace period**: Disconnects schedule a delayed "left" announcement (5s); a reconnect within 5s cancels it and skips "joined", eliminating join/leave spam on brief reconnects.
|
- **Reconnect grace period**: Disconnects schedule a delayed "left" announcement (5s); a reconnect within 5s cancels it and skips "joined", eliminating join/leave spam on brief reconnects.
|
||||||
- **HLS segment duration 2s**: Default `HLS_SEGMENT_DURATION` changed from 6 to 2 and `HLS_WINDOW_SIZE` from 24 to 12 (2s × 12 = 24s live window), reducing end-to-end latency; the 2s GOP was already in place.
|
- **HLS segment duration 2s**: Default `HLS_SEGMENT_DURATION` changed from 6 to 2 and `HLS_WINDOW_SIZE` from 24 to 12 (2s × 12 = 24s live window), reducing end-to-end latency; the 2s GOP was already in place.
|
||||||
- **Chat font sizing**: `.message-text` set to `0.95rem` (slightly smaller than the default, larger than the `0.83rem` system/flash text).
|
- **Chat font sizing**: `.message-text` set to `0.95rem` (slightly smaller than the default, larger than the `0.83rem` system/flash text).
|
||||||
|
- **R2 retention by run count + size**: Playout now keeps the newest `R2_KEEP_RUNS` run folders (default 2) and enforces an `R2_MAX_RETENTION_BYTES` cap (default 10 GiB), replacing the old 24h time-based `R2_SEGMENT_RETENTION_HOURS` cleanup. Cleanup runs after each new run plus an hourly backstop, and never deletes the live manifest. Adds `linearvoid_playout_r2_runs_retained` and `linearvoid_playout_r2_bytes_retained` gauges.
|
||||||
|
|
||||||
### Tests
|
### Tests
|
||||||
|
|
||||||
- WebSocket: `TestHub_Broadcast_SystemNotPersisted`, `TestHub_ReconnectGracePeriod`.
|
- WebSocket: `TestHub_Broadcast_SystemNotPersisted`, `TestHub_ReconnectGracePeriod`.
|
||||||
- Handlers: `TestHandleAdminChatLog_ExcludesSystem`.
|
- Handlers: `TestHandleAdminChatLog_ExcludesSystem`.
|
||||||
- Frontend: `presence flash` test.
|
- Frontend: `presence flash` test.
|
||||||
|
- Playout: `TestCleanupOldRuns`, `TestCleanupOldRuns_SizeCap`, `TestR2KeepRuns`, `TestR2MaxRetentionBytes`, `TestR2RunFromKey`.
|
||||||
|
|
||||||
## [1.0.13] - 2026-08-13
|
## [1.0.13] - 2026-08-13
|
||||||
|
|
||||||
|
|||||||
+172
-53
@@ -11,6 +11,7 @@ import (
|
|||||||
"os"
|
"os"
|
||||||
"os/exec"
|
"os/exec"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
@@ -66,6 +67,10 @@ var (
|
|||||||
// R2 retention cleanup counter
|
// R2 retention cleanup counter
|
||||||
r2CleanupObjectsTotal int64
|
r2CleanupObjectsTotal int64
|
||||||
|
|
||||||
|
// R2 retention gauges
|
||||||
|
r2RunsRetained int64
|
||||||
|
r2BytesRetained int64
|
||||||
|
|
||||||
// Continuous playout loop metrics
|
// Continuous playout loop metrics
|
||||||
loopsTotal int64
|
loopsTotal int64
|
||||||
lastRunDuration time.Duration
|
lastRunDuration time.Duration
|
||||||
@@ -78,6 +83,10 @@ const (
|
|||||||
// Cache-Control header values for R2 uploads.
|
// Cache-Control header values for R2 uploads.
|
||||||
segmentCacheControl = "public, max-age=31536000, immutable"
|
segmentCacheControl = "public, max-age=31536000, immutable"
|
||||||
manifestCacheControl = "public, max-age=1, s-maxage=2, must-revalidate"
|
manifestCacheControl = "public, max-age=1, s-maxage=2, must-revalidate"
|
||||||
|
|
||||||
|
// R2 retention policy.
|
||||||
|
defaultR2KeepRuns = 2
|
||||||
|
defaultR2MaxRetentionBytes = int64(10 * 1024 * 1024 * 1024) // 10 GiB
|
||||||
)
|
)
|
||||||
|
|
||||||
// envWithDefault returns the value of the environment variable or the default if not set.
|
// envWithDefault returns the value of the environment variable or the default if not set.
|
||||||
@@ -658,6 +667,12 @@ func main() {
|
|||||||
fmt.Fprintln(w, "# HELP linearvoid_playout_r2_cleanup_objects_total Total R2 objects deleted by retention cleanup.")
|
fmt.Fprintln(w, "# HELP linearvoid_playout_r2_cleanup_objects_total Total R2 objects deleted by retention cleanup.")
|
||||||
fmt.Fprintln(w, "# TYPE linearvoid_playout_r2_cleanup_objects_total counter")
|
fmt.Fprintln(w, "# TYPE linearvoid_playout_r2_cleanup_objects_total counter")
|
||||||
fmt.Fprintf(w, "linearvoid_playout_r2_cleanup_objects_total %d\n", r2CleanupObjectsTotal)
|
fmt.Fprintf(w, "linearvoid_playout_r2_cleanup_objects_total %d\n", r2CleanupObjectsTotal)
|
||||||
|
fmt.Fprintln(w, "# HELP linearvoid_playout_r2_runs_retained Number of R2 run folders retained by cleanup.")
|
||||||
|
fmt.Fprintln(w, "# TYPE linearvoid_playout_r2_runs_retained gauge")
|
||||||
|
fmt.Fprintf(w, "linearvoid_playout_r2_runs_retained %d\n", r2RunsRetained)
|
||||||
|
fmt.Fprintln(w, "# HELP linearvoid_playout_r2_bytes_retained Total bytes retained in R2 run folders.")
|
||||||
|
fmt.Fprintln(w, "# TYPE linearvoid_playout_r2_bytes_retained gauge")
|
||||||
|
fmt.Fprintf(w, "linearvoid_playout_r2_bytes_retained %d\n", r2BytesRetained)
|
||||||
fmt.Fprintln(w, "# HELP linearvoid_playout_loops_total Total clean playlist completions (loops).")
|
fmt.Fprintln(w, "# HELP linearvoid_playout_loops_total Total clean playlist completions (loops).")
|
||||||
fmt.Fprintln(w, "# TYPE linearvoid_playout_loops_total counter")
|
fmt.Fprintln(w, "# TYPE linearvoid_playout_loops_total counter")
|
||||||
fmt.Fprintf(w, "linearvoid_playout_loops_total %d\n", loopsTotal)
|
fmt.Fprintf(w, "linearvoid_playout_loops_total %d\n", loopsTotal)
|
||||||
@@ -682,6 +697,15 @@ func main() {
|
|||||||
// New run ID on each restart for unique R2 segment keys (STREAMING.md)
|
// New run ID on each restart for unique R2 segment keys (STREAMING.md)
|
||||||
initRunID()
|
initRunID()
|
||||||
|
|
||||||
|
// Remove run folders older than the retention policy (keep the last N runs,
|
||||||
|
// cap total bytes) now that the next run has started. This never touches the
|
||||||
|
// current or previous run folders.
|
||||||
|
if r2Client != nil {
|
||||||
|
if _, err := cleanupOldRuns(context.Background()); err != nil {
|
||||||
|
fmt.Println("R2 cleanup error:", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Reset HLS output only when the playlist changed, we are recovering from a
|
// Reset HLS output only when the playlist changed, we are recovering from a
|
||||||
// crash/slate, or this is the first run. Otherwise continue the segment
|
// crash/slate, or this is the first run. Otherwise continue the segment
|
||||||
// sequence so the loop is seamless for viewers (no gap on clean loops).
|
// sequence so the loop is seamless for viewers (no gap on clean loops).
|
||||||
@@ -834,68 +858,51 @@ func syncLatestToR2(hlsDir string) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// r2SegmentRetentionHours returns the R2 segment retention window in hours.
|
// r2KeepRuns returns how many of the most recent run folders to keep in R2.
|
||||||
func r2SegmentRetentionHours() time.Duration {
|
func r2KeepRuns() int {
|
||||||
h, err := strconv.Atoi(envWithDefault("R2_SEGMENT_RETENTION_HOURS", "24"))
|
n, err := strconv.Atoi(envWithDefault("R2_KEEP_RUNS", "2"))
|
||||||
if err != nil || h <= 0 {
|
if err != nil || n < 1 {
|
||||||
h = 24
|
return defaultR2KeepRuns
|
||||||
}
|
}
|
||||||
return time.Duration(h) * time.Hour
|
return n
|
||||||
}
|
}
|
||||||
|
|
||||||
// cleanupOldSegments deletes R2 objects under the hls/ prefix that are older than
|
// r2MaxRetentionBytes returns the maximum total bytes to retain across run folders.
|
||||||
// the retention window, excluding the live manifest and any non-run keys.
|
func r2MaxRetentionBytes() int64 {
|
||||||
// It returns the number of objects deleted.
|
n, err := strconv.ParseInt(envWithDefault("R2_MAX_RETENTION_BYTES", "10737418240"), 10, 64)
|
||||||
func cleanupOldSegments(ctx context.Context) (int, error) {
|
if err != nil || n <= 0 {
|
||||||
if r2Client == nil {
|
return defaultR2MaxRetentionBytes
|
||||||
if err := initR2Client(); err != nil {
|
|
||||||
return 0, err
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
cutoff := time.Now().UTC().Add(-r2SegmentRetentionHours())
|
// r2RunFromKey extracts the run folder name ("r-...") from an R2 object key.
|
||||||
|
func r2RunFromKey(key string) (string, bool) {
|
||||||
var stale []string
|
rest := strings.TrimPrefix(key, "hls/")
|
||||||
var token *string
|
if rest == key || !strings.HasPrefix(rest, "r-") {
|
||||||
for {
|
return "", false
|
||||||
output, err := listObjects(r2Client, &s3.ListObjectsV2Input{
|
|
||||||
Bucket: aws.String(bucket),
|
|
||||||
Prefix: aws.String("hls/"),
|
|
||||||
ContinuationToken: token,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
return 0, err
|
|
||||||
}
|
|
||||||
for _, obj := range output.Contents {
|
|
||||||
key := aws.ToString(obj.Key)
|
|
||||||
if key == "hls/playlist.m3u8" || key == "hls/playlist.m3u8.tmp" {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
// Only consider segment/init objects under a run directory: hls/r-.../...
|
|
||||||
if !strings.HasPrefix(key, "hls/r-") {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if obj.LastModified != nil && obj.LastModified.Before(cutoff) {
|
|
||||||
stale = append(stale, key)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if output.IsTruncated == nil || !*output.IsTruncated {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
token = output.NextContinuationToken
|
|
||||||
}
|
}
|
||||||
|
if idx := strings.IndexByte(rest, '/'); idx > 0 {
|
||||||
if len(stale) == 0 {
|
return rest[:idx], true
|
||||||
return 0, nil
|
|
||||||
}
|
}
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
|
||||||
|
// runObjects groups the R2 object keys and total size for a single run folder.
|
||||||
|
type runObjects struct {
|
||||||
|
keys []string
|
||||||
|
size int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// deleteR2Keys deletes the given R2 object keys in batches of up to 1000.
|
||||||
|
func deleteR2Keys(keys []string) (int, error) {
|
||||||
deleted := 0
|
deleted := 0
|
||||||
for i := 0; i < len(stale); i += 1000 {
|
for i := 0; i < len(keys); i += 1000 {
|
||||||
end := i + 1000
|
end := i + 1000
|
||||||
if end > len(stale) {
|
if end > len(keys) {
|
||||||
end = len(stale)
|
end = len(keys)
|
||||||
}
|
}
|
||||||
batch := stale[i:end]
|
batch := keys[i:end]
|
||||||
objects := make([]types.ObjectIdentifier, len(batch))
|
objects := make([]types.ObjectIdentifier, len(batch))
|
||||||
for j, key := range batch {
|
for j, key := range batch {
|
||||||
objects[j] = types.ObjectIdentifier{Key: aws.String(key)}
|
objects[j] = types.ObjectIdentifier{Key: aws.String(key)}
|
||||||
@@ -911,6 +918,118 @@ func cleanupOldSegments(ctx context.Context) (int, error) {
|
|||||||
}
|
}
|
||||||
deleted += len(batch)
|
deleted += len(batch)
|
||||||
}
|
}
|
||||||
|
return deleted, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// cleanupOldRuns lists run-scoped objects under hls/, keeps the most recent
|
||||||
|
// R2_KEEP_RUNS runs, and deletes older runs. It also enforces the
|
||||||
|
// R2_MAX_RETENTION_BYTES cap by removing oldest retained runs first, always
|
||||||
|
// keeping at least the newest run. The live manifest keys are never deleted.
|
||||||
|
func cleanupOldRuns(ctx context.Context) (int, error) {
|
||||||
|
if r2Client == nil {
|
||||||
|
if err := initR2Client(); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
runs := map[string]*runObjects{}
|
||||||
|
var token *string
|
||||||
|
for {
|
||||||
|
output, err := listObjects(r2Client, &s3.ListObjectsV2Input{
|
||||||
|
Bucket: aws.String(bucket),
|
||||||
|
Prefix: aws.String("hls/"),
|
||||||
|
ContinuationToken: token,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
for _, obj := range output.Contents {
|
||||||
|
key := aws.ToString(obj.Key)
|
||||||
|
if key == "hls/playlist.m3u8" || key == "hls/playlist.m3u8.tmp" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
run, ok := r2RunFromKey(key)
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
r, exists := runs[run]
|
||||||
|
if !exists {
|
||||||
|
r = &runObjects{}
|
||||||
|
runs[run] = r
|
||||||
|
}
|
||||||
|
r.keys = append(r.keys, key)
|
||||||
|
if obj.Size != nil {
|
||||||
|
r.size += *obj.Size
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if output.IsTruncated == nil || !*output.IsTruncated {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
token = output.NextContinuationToken
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(runs) == 0 {
|
||||||
|
r2RunsRetained = 0
|
||||||
|
r2BytesRetained = 0
|
||||||
|
return 0, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run IDs are ISO timestamps, so lexicographic order equals chronological order.
|
||||||
|
sortedRuns := make([]string, 0, len(runs))
|
||||||
|
for run := range runs {
|
||||||
|
sortedRuns = append(sortedRuns, run)
|
||||||
|
}
|
||||||
|
sort.Strings(sortedRuns) // ascending: oldest -> newest
|
||||||
|
|
||||||
|
deleteRun := map[string]bool{}
|
||||||
|
keepRuns := r2KeepRuns()
|
||||||
|
|
||||||
|
// Mark all but the newest keepRuns runs for deletion.
|
||||||
|
if len(sortedRuns) > keepRuns {
|
||||||
|
for _, run := range sortedRuns[:len(sortedRuns)-keepRuns] {
|
||||||
|
deleteRun[run] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Enforce the total size cap on retained runs (oldest retained first), always
|
||||||
|
// keeping the newest run even if it alone exceeds the cap.
|
||||||
|
maxBytes := r2MaxRetentionBytes()
|
||||||
|
retained := make([]string, 0, keepRuns)
|
||||||
|
var retainedBytes int64
|
||||||
|
for _, run := range sortedRuns {
|
||||||
|
if !deleteRun[run] {
|
||||||
|
retained = append(retained, run)
|
||||||
|
retainedBytes += runs[run].size
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for i := 0; retainedBytes > maxBytes && i < len(retained)-1; i++ {
|
||||||
|
run := retained[i]
|
||||||
|
deleteRun[run] = true
|
||||||
|
retainedBytes -= runs[run].size
|
||||||
|
}
|
||||||
|
|
||||||
|
var toDelete []string
|
||||||
|
for run, del := range deleteRun {
|
||||||
|
if del {
|
||||||
|
toDelete = append(toDelete, runs[run].keys...)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
deleted, err := deleteR2Keys(toDelete)
|
||||||
|
if err != nil {
|
||||||
|
return deleted, err
|
||||||
|
}
|
||||||
|
|
||||||
|
keptRuns := int64(0)
|
||||||
|
keptBytes := int64(0)
|
||||||
|
for run, r := range runs {
|
||||||
|
if !deleteRun[run] {
|
||||||
|
keptRuns++
|
||||||
|
keptBytes += r.size
|
||||||
|
}
|
||||||
|
}
|
||||||
|
r2RunsRetained = keptRuns
|
||||||
|
r2BytesRetained = keptBytes
|
||||||
|
|
||||||
r2CleanupObjectsTotal += int64(deleted)
|
r2CleanupObjectsTotal += int64(deleted)
|
||||||
return deleted, nil
|
return deleted, nil
|
||||||
@@ -918,13 +1037,13 @@ func cleanupOldSegments(ctx context.Context) (int, error) {
|
|||||||
|
|
||||||
// runR2CleanupLoop runs retention cleanup once at startup, then hourly.
|
// runR2CleanupLoop runs retention cleanup once at startup, then hourly.
|
||||||
func runR2CleanupLoop() {
|
func runR2CleanupLoop() {
|
||||||
if _, err := cleanupOldSegments(context.Background()); err != nil {
|
if _, err := cleanupOldRuns(context.Background()); err != nil {
|
||||||
fmt.Println("R2 cleanup error:", err)
|
fmt.Println("R2 cleanup error:", err)
|
||||||
}
|
}
|
||||||
ticker := time.NewTicker(time.Hour)
|
ticker := time.NewTicker(time.Hour)
|
||||||
defer ticker.Stop()
|
defer ticker.Stop()
|
||||||
for range ticker.C {
|
for range ticker.C {
|
||||||
if _, err := cleanupOldSegments(context.Background()); err != nil {
|
if _, err := cleanupOldRuns(context.Background()); err != nil {
|
||||||
fmt.Println("R2 cleanup error:", err)
|
fmt.Println("R2 cleanup error:", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -593,54 +593,95 @@ func TestSyncToR2_NoCacheHeadersForUnknownExt(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestR2SegmentRetentionHours(t *testing.T) {
|
func TestR2KeepRuns(t *testing.T) {
|
||||||
os.Setenv("R2_SEGMENT_RETENTION_HOURS", "")
|
os.Setenv("R2_KEEP_RUNS", "")
|
||||||
if got := r2SegmentRetentionHours(); got != 24*time.Hour {
|
if got := r2KeepRuns(); got != 2 {
|
||||||
t.Errorf("default = %v, want 24h", got)
|
t.Errorf("default = %d, want 2", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
os.Setenv("R2_SEGMENT_RETENTION_HOURS", "48")
|
os.Setenv("R2_KEEP_RUNS", "5")
|
||||||
if got := r2SegmentRetentionHours(); got != 48*time.Hour {
|
if got := r2KeepRuns(); got != 5 {
|
||||||
t.Errorf("override = %v, want 48h", got)
|
t.Errorf("override = %d, want 5", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
os.Setenv("R2_SEGMENT_RETENTION_HOURS", "invalid")
|
os.Setenv("R2_KEEP_RUNS", "0")
|
||||||
if got := r2SegmentRetentionHours(); got != 24*time.Hour {
|
if got := r2KeepRuns(); got != 2 {
|
||||||
t.Errorf("invalid = %v, want 24h fallback", got)
|
t.Errorf("invalid = %d, want 2 fallback", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
os.Unsetenv("R2_SEGMENT_RETENTION_HOURS")
|
os.Unsetenv("R2_KEEP_RUNS")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestCleanupOldSegments(t *testing.T) {
|
func TestR2MaxRetentionBytes(t *testing.T) {
|
||||||
|
os.Setenv("R2_MAX_RETENTION_BYTES", "")
|
||||||
|
if got := r2MaxRetentionBytes(); got != 10*1024*1024*1024 {
|
||||||
|
t.Errorf("default = %d, want 10GiB", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
os.Setenv("R2_MAX_RETENTION_BYTES", "100")
|
||||||
|
if got := r2MaxRetentionBytes(); got != 100 {
|
||||||
|
t.Errorf("override = %d, want 100", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
os.Setenv("R2_MAX_RETENTION_BYTES", "invalid")
|
||||||
|
if got := r2MaxRetentionBytes(); got != 10*1024*1024*1024 {
|
||||||
|
t.Errorf("invalid = %d, want 10GiB fallback", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
os.Unsetenv("R2_MAX_RETENTION_BYTES")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestR2RunFromKey(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
key string
|
||||||
|
run string
|
||||||
|
ok bool
|
||||||
|
}{
|
||||||
|
{"hls/r-20260815T120000Z/seg-1.m4s", "r-20260815T120000Z", true},
|
||||||
|
{"hls/r-20260815T120000Z/init.mp4", "r-20260815T120000Z", true},
|
||||||
|
{"hls/playlist.m3u8", "", false},
|
||||||
|
{"hls/playlist.m3u8.tmp", "", false},
|
||||||
|
{"other/r-x/seg.m4s", "", false},
|
||||||
|
{"hls/r-noslash", "", false},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
run, ok := r2RunFromKey(c.key)
|
||||||
|
if run != c.run || ok != c.ok {
|
||||||
|
t.Errorf("r2RunFromKey(%q) = (%q, %v), want (%q, %v)", c.key, run, ok, c.run, c.ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCleanupOldRuns(t *testing.T) {
|
||||||
prevClient := r2Client
|
prevClient := r2Client
|
||||||
prevBucket := bucket
|
prevBucket := bucket
|
||||||
prevList := listObjects
|
prevList := listObjects
|
||||||
prevDelete := deleteObjects
|
prevDelete := deleteObjects
|
||||||
prevTotal := r2CleanupObjectsTotal
|
prevTotal := r2CleanupObjectsTotal
|
||||||
|
prevRuns := r2RunsRetained
|
||||||
|
prevBytes := r2BytesRetained
|
||||||
defer func() {
|
defer func() {
|
||||||
r2Client = prevClient
|
r2Client = prevClient
|
||||||
bucket = prevBucket
|
bucket = prevBucket
|
||||||
listObjects = prevList
|
listObjects = prevList
|
||||||
deleteObjects = prevDelete
|
deleteObjects = prevDelete
|
||||||
r2CleanupObjectsTotal = prevTotal
|
r2CleanupObjectsTotal = prevTotal
|
||||||
|
r2RunsRetained = prevRuns
|
||||||
|
r2BytesRetained = prevBytes
|
||||||
}()
|
}()
|
||||||
|
|
||||||
bucket = "linearvoid-media"
|
bucket = "linearvoid-media"
|
||||||
r2Client = &s3.Client{}
|
r2Client = &s3.Client{}
|
||||||
|
|
||||||
now := time.Now().UTC()
|
size := int64(1024)
|
||||||
old := now.Add(-25 * time.Hour)
|
|
||||||
recent := now.Add(-1 * time.Hour)
|
|
||||||
|
|
||||||
listObjects = func(_ *s3.Client, input *s3.ListObjectsV2Input) (*s3.ListObjectsV2Output, error) {
|
listObjects = func(_ *s3.Client, input *s3.ListObjectsV2Input) (*s3.ListObjectsV2Output, error) {
|
||||||
return &s3.ListObjectsV2Output{
|
return &s3.ListObjectsV2Output{
|
||||||
Contents: []types.Object{
|
Contents: []types.Object{
|
||||||
{Key: aws.String("hls/r-old/seg-1.m4s"), LastModified: &old},
|
{Key: aws.String("hls/r-20260813T000000Z/seg-1.m4s"), Size: &size},
|
||||||
{Key: aws.String("hls/r-old/init.mp4"), LastModified: &old},
|
{Key: aws.String("hls/r-20260814T000000Z/seg-2.m4s"), Size: &size},
|
||||||
{Key: aws.String("hls/r-new/seg-2.m4s"), LastModified: &recent},
|
{Key: aws.String("hls/r-20260815T000000Z/seg-3.m4s"), Size: &size},
|
||||||
{Key: aws.String("hls/playlist.m3u8"), LastModified: &old},
|
{Key: aws.String("hls/playlist.m3u8"), Size: &size},
|
||||||
{Key: aws.String("hls/playlist.m3u8.tmp"), LastModified: &old},
|
{Key: aws.String("hls/playlist.m3u8.tmp"), Size: &size},
|
||||||
},
|
},
|
||||||
IsTruncated: aws.Bool(false),
|
IsTruncated: aws.Bool(false),
|
||||||
}, nil
|
}, nil
|
||||||
@@ -654,23 +695,101 @@ func TestCleanupOldSegments(t *testing.T) {
|
|||||||
return &s3.DeleteObjectsOutput{}, nil
|
return &s3.DeleteObjectsOutput{}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
n, err := cleanupOldSegments(context.Background())
|
n, err := cleanupOldRuns(context.Background())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("cleanupOldSegments error: %v", err)
|
t.Fatalf("cleanupOldRuns error: %v", err)
|
||||||
}
|
}
|
||||||
|
if n != 1 {
|
||||||
|
t.Fatalf("deleted = %d, want 1 (oldest run)", n)
|
||||||
|
}
|
||||||
|
|
||||||
|
joined := strings.Join(deletedKeys, ",")
|
||||||
|
if !strings.Contains(joined, "hls/r-20260813T000000Z/seg-1.m4s") {
|
||||||
|
t.Fatalf("expected oldest run deleted, got: %v", deletedKeys)
|
||||||
|
}
|
||||||
|
if strings.Contains(joined, "playlist.m3u8") {
|
||||||
|
t.Fatalf("manifest must not be deleted: %v", deletedKeys)
|
||||||
|
}
|
||||||
|
if r2RunsRetained != 2 {
|
||||||
|
t.Fatalf("r2RunsRetained = %d, want 2", r2RunsRetained)
|
||||||
|
}
|
||||||
|
if r2BytesRetained != 2*size {
|
||||||
|
t.Fatalf("r2BytesRetained = %d, want %d", r2BytesRetained, 2*size)
|
||||||
|
}
|
||||||
|
if r2CleanupObjectsTotal != 1 {
|
||||||
|
t.Fatalf("r2CleanupObjectsTotal = %d, want 1", r2CleanupObjectsTotal)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCleanupOldRuns_SizeCap(t *testing.T) {
|
||||||
|
prevClient := r2Client
|
||||||
|
prevBucket := bucket
|
||||||
|
prevList := listObjects
|
||||||
|
prevDelete := deleteObjects
|
||||||
|
prevTotal := r2CleanupObjectsTotal
|
||||||
|
prevRuns := r2RunsRetained
|
||||||
|
prevBytes := r2BytesRetained
|
||||||
|
defer func() {
|
||||||
|
r2Client = prevClient
|
||||||
|
bucket = prevBucket
|
||||||
|
listObjects = prevList
|
||||||
|
deleteObjects = prevDelete
|
||||||
|
r2CleanupObjectsTotal = prevTotal
|
||||||
|
r2RunsRetained = prevRuns
|
||||||
|
r2BytesRetained = prevBytes
|
||||||
|
}()
|
||||||
|
|
||||||
|
bucket = "linearvoid-media"
|
||||||
|
r2Client = &s3.Client{}
|
||||||
|
|
||||||
|
// Force a small cap so the size policy trims even the last 2 runs.
|
||||||
|
os.Setenv("R2_KEEP_RUNS", "2")
|
||||||
|
os.Setenv("R2_MAX_RETENTION_BYTES", "1024")
|
||||||
|
defer os.Unsetenv("R2_KEEP_RUNS")
|
||||||
|
defer os.Unsetenv("R2_MAX_RETENTION_BYTES")
|
||||||
|
|
||||||
|
size := int64(1024)
|
||||||
|
listObjects = func(_ *s3.Client, input *s3.ListObjectsV2Input) (*s3.ListObjectsV2Output, error) {
|
||||||
|
return &s3.ListObjectsV2Output{
|
||||||
|
Contents: []types.Object{
|
||||||
|
{Key: aws.String("hls/r-20260813T000000Z/seg-1.m4s"), Size: &size},
|
||||||
|
{Key: aws.String("hls/r-20260814T000000Z/seg-2.m4s"), Size: &size},
|
||||||
|
{Key: aws.String("hls/r-20260815T000000Z/seg-3.m4s"), Size: &size},
|
||||||
|
},
|
||||||
|
IsTruncated: aws.Bool(false),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var deletedKeys []string
|
||||||
|
deleteObjects = func(_ *s3.Client, input *s3.DeleteObjectsInput) (*s3.DeleteObjectsOutput, error) {
|
||||||
|
for _, obj := range input.Delete.Objects {
|
||||||
|
deletedKeys = append(deletedKeys, aws.ToString(obj.Key))
|
||||||
|
}
|
||||||
|
return &s3.DeleteObjectsOutput{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
n, err := cleanupOldRuns(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("cleanupOldRuns error: %v", err)
|
||||||
|
}
|
||||||
|
// 3 runs -> keep 2 -> oldest deleted by count; remaining 2 bytes (2048) exceed
|
||||||
|
// the 1024 cap -> the older of the kept two is deleted, leaving only the newest.
|
||||||
if n != 2 {
|
if n != 2 {
|
||||||
t.Fatalf("deleted = %d, want 2", n)
|
t.Fatalf("deleted = %d, want 2", n)
|
||||||
}
|
}
|
||||||
|
|
||||||
joined := strings.Join(deletedKeys, ",")
|
joined := strings.Join(deletedKeys, ",")
|
||||||
if !strings.Contains(joined, "hls/r-old/seg-1.m4s") || !strings.Contains(joined, "hls/r-old/init.mp4") {
|
if !strings.Contains(joined, "hls/r-20260813T000000Z/seg-1.m4s") {
|
||||||
t.Fatalf("unexpected deleted keys: %v", deletedKeys)
|
t.Fatalf("expected oldest run deleted, got: %v", deletedKeys)
|
||||||
}
|
}
|
||||||
if strings.Contains(joined, "playlist.m3u8") {
|
if !strings.Contains(joined, "hls/r-20260814T000000Z/seg-2.m4s") {
|
||||||
t.Fatalf("manifest must not be deleted: %v", deletedKeys)
|
t.Fatalf("expected second-oldest run deleted by size cap, got: %v", deletedKeys)
|
||||||
}
|
}
|
||||||
if r2CleanupObjectsTotal != 2 {
|
if strings.Contains(joined, "hls/r-20260815T000000Z/seg-3.m4s") {
|
||||||
t.Fatalf("r2CleanupObjectsTotal = %d, want 2", r2CleanupObjectsTotal)
|
t.Fatalf("newest run must be kept, got: %v", deletedKeys)
|
||||||
|
}
|
||||||
|
if r2RunsRetained != 1 {
|
||||||
|
t.Fatalf("r2RunsRetained = %d, want 1", r2RunsRetained)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -180,6 +180,8 @@ A release is streaming-ready only when one full playlist boundary has played wit
|
|||||||
| `R2_ENDPOINT`, `R2_REGION` | S3-compatible publication endpoint |
|
| `R2_ENDPOINT`, `R2_REGION` | S3-compatible publication endpoint |
|
||||||
| `R2_BUCKET_NAME` | Media bucket |
|
| `R2_BUCKET_NAME` | Media bucket |
|
||||||
| `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY` | Write credentials, available only to the publisher |
|
| `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY` | Write credentials, available only to the publisher |
|
||||||
|
| `R2_KEEP_RUNS` | Number of run folders to retain (default `2`) |
|
||||||
|
| `R2_MAX_RETENTION_BYTES` | Max bytes retained across run folders (10 GiB) |
|
||||||
| `MEDIA_DOMAIN` | Public R2 custom domain |
|
| `MEDIA_DOMAIN` | Public R2 custom domain |
|
||||||
| `VITE_PUBLIC_HLS_URL` | Public live manifest URL |
|
| `VITE_PUBLIC_HLS_URL` | Public live manifest URL |
|
||||||
|
|
||||||
|
|||||||
@@ -311,7 +311,7 @@ graph TB
|
|||||||
- **Resilience**: Restart counter (>5/60s → activate slate), Prometheus metrics on port 8081
|
- **Resilience**: Restart counter (>5/60s → activate slate), Prometheus metrics on port 8081
|
||||||
- **Validation**: ffprobe gate on all ingested files (codec, fps, resolution, keyframe interval); `-validate` mode checks the playlist without running the loop
|
- **Validation**: ffprobe gate on all ingested files (codec, fps, resolution, keyframe interval); `-validate` mode checks the playlist without running the loop
|
||||||
- **Seamless looping**: Only resets HLS output when the playlist fingerprint (SHA-256) changes, on crash/slate recovery, or first run; otherwise continues segment numbering via `-start_number`
|
- **Seamless looping**: Only resets HLS output when the playlist fingerprint (SHA-256) changes, on crash/slate recovery, or first run; otherwise continues segment numbering via `-start_number`
|
||||||
- **R2 retention**: `cleanupOldSegments` deletes `hls/r-*/*` objects older than `R2_SEGMENT_RETENTION_HOURS` (default 24h) hourly + at startup
|
- **R2 retention**: `cleanupOldRuns` keeps the newest `R2_KEEP_RUNS` run folders (default 2) and caps retained data at `R2_MAX_RETENTION_BYTES` (default 10 GiB), triggered after each run plus an hourly backstop
|
||||||
- **Shared config**: Reads the runtime `config.json` (`loadSharedHLSConfig`) as an env fallback for HLS segment duration/window
|
- **Shared config**: Reads the runtime `config.json` (`loadSharedHLSConfig`) as an env fallback for HLS segment duration/window
|
||||||
|
|
||||||
#### Chat Service (`backend/chat/src/main.go`)
|
#### Chat Service (`backend/chat/src/main.go`)
|
||||||
@@ -365,7 +365,7 @@ Module-level breakdown per service.
|
|||||||
| **Loop Controller** | `playlist.go` (playlistFingerprint, hlsNextStartNumber) | Detect playlist changes (SHA-256), continue segment numbering on clean loops |
|
| **Loop Controller** | `playlist.go` (playlistFingerprint, hlsNextStartNumber) | Detect playlist changes (SHA-256), continue segment numbering on clean loops |
|
||||||
| **Slate Controller** | `playlist.go` (activateSlate, deactivateSlate) | Switch output to slate loop on excessive failures |
|
| **Slate Controller** | `playlist.go` (activateSlate, deactivateSlate) | Switch output to slate loop on excessive failures |
|
||||||
| **R2 Sync** | `playlist.go` (syncToR2) | Upload HLS segments + manifest to R2 bucket |
|
| **R2 Sync** | `playlist.go` (syncToR2) | Upload HLS segments + manifest to R2 bucket |
|
||||||
| **R2 Cleanup** | `playlist.go` (cleanupOldSegments) | Delete stale `hls/r-*/*` segments older than the retention window |
|
| **R2 Cleanup** | `playlist.go` (cleanupOldRuns) | Keep the last N run folders under the size cap; delete older `hls/r-*/*` objects |
|
||||||
| **Health Endpoint** | `playlist.go` (healthHandler) | `/health` → JSON with mode, segment count, uptime |
|
| **Health Endpoint** | `playlist.go` (healthHandler) | `/health` → JSON with mode, segment count, uptime |
|
||||||
| **Metrics Exporter** | `playlist.go` (prometheus) | Expose restarts_total, mode (live/slate), loops_total, run_duration_seconds, r2_cleanup_objects_total |
|
| **Metrics Exporter** | `playlist.go` (prometheus) | Expose restarts_total, mode (live/slate), loops_total, run_duration_seconds, r2_cleanup_objects_total |
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user