diff --git a/.env.example b/.env.example index ad22a46..2306595 100644 --- a/.env.example +++ b/.env.example @@ -54,8 +54,8 @@ NGINX_PORT=80 # Playout configuration PLAYLIST_PATH=/app/playlist/playlist.txt -HLS_SEGMENT_DURATION=6 -HLS_WINDOW_SIZE=24 +HLS_SEGMENT_DURATION=2 +HLS_WINDOW_SIZE=12 # Shared runtime config (written by chat admin via /api/admin/config, read by playout). # Both resolve to the same host file through the ./data/playlist volume. diff --git a/.github/MEMORY.md b/.github/MEMORY.md index c804ac0..d2420da 100644 --- a/.github/MEMORY.md +++ b/.github/MEMORY.md @@ -134,3 +134,21 @@ - Updated `.gitea/workflows/cd.yml` "Prepare environment" step to include all R2* variables sourced from Gitea secrets (`R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_BUCKET_NAME`, `R2_ENDPOINT`, `R2_ADMIN*\*`), Cloudflare variables (`CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`), and frontend public URL variables. - Updated `docs/DEPLOYMENT.md` "Environment File" section: replaced vague "Cloudflare/R2 variables" bullet with explicit variable lists and added a new "Gitea Secrets" table documenting all 15 required secrets. - All playout tests pass (20 tests), `go vet` clean. + +## 2026-08-13 + +- Completed all remaining enhancement tasks (only manual/infrastructure phases left): + - **Playout**: automatic media normalization on upload (`media.NormalizeVideo`, `*.normalized.mp4`, statuses `ready`/`needs_normalization`/`normalizing`/`normalization_failed`, `POST /api/admin/media/{id}/normalize`); R2 segment retention (`cleanupOldSegments`, `R2_SEGMENT_RETENTION_HOURS`); seamless looping (playlist SHA-256 fingerprint + `-start_number`); single `start-playout.sh` entrypoint + `-validate` mode; loop/cleanup metrics. + - **Application configuration**: thread-safe JSON runtime config store (`backend/chat/src/config/runtime.go`) shared with playout via `/app/playlist/config.json`; `GET/PUT /api/admin/config`; admin UI config form. + - **Matomo analytics**: public `GET /api/matomo` + viewer pageview snippet in `index.html`. + - **Chat user accounts**: register/login/logout/session endpoints, bcrypt hashing, `linearvoid_user_session` cookie, viewer sign-in UI. + - **DM history & search**: `GET /api/dm/history` (paginated) + `GET /api/dm/search` + DM search UI. + - **Chat UI**: typing indicator, user-list toggle, media status badges, "Add to playlist" gated to `ready`. + - **Prometheus TSDB retention**: `--storage.tsdb.retention.time=30d` + `--storage.tsdb.retention.size=15GB` in both compose files; alert rule count corrected 8 → 10. +- Updated `docs/architecture/README.md` to reflect all `[Unreleased]` changes (normalization, looping, retention, runtime config, Matomo, user accounts, DM history/search, new API endpoints, module tables). +- Released **1.0.13**: promoted changelog, bumped version, tagged and pushed. +- Extended frontend test coverage: added `counter.test.ts`, reconnecting/typing/auth/websocket tests, admin failure + status-badge tests → 56 tests; statements 19.89% → 28.58% (CI coverage threshold passing). +- Fixed Go test data race in `TestHandleAdminMediaByID_Normalize` (waits for async normalization goroutine) + shared in-memory SQLite via `SetMaxOpenConns(1)`. +- Fixed viewer crash: `index.html` was missing the `#messages` chat container (stray ``); restored structure + made `addMessage` defensive. +- Set `Doto` as the body font (`--font-body`) and added Google Fonts links to admin + login pages. +- Files changed: across `backend/chat`, `backend/playout`, `frontend`, `docs`, `docker-compose*.yml`, `.github/instructions/*`. diff --git a/.github/instructions/DONE.md b/.github/instructions/DONE.md index f96f9e6..a058700 100644 --- a/.github/instructions/DONE.md +++ b/.github/instructions/DONE.md @@ -1161,3 +1161,24 @@ Completed the three Playout enhancement workstreams from `TODO.md`. Remaining po - Fixed `Uncaught TypeError: Cannot read properties of null (reading 'appendChild')` in the viewer bundle: `index.html` was missing the `#messages` chat container (and had a stray ``), so `addMessage` crashed on WebSocket open/message. - Restored chat panel structure: added `
` and wrapped `#presence-list` in `
`. - Made `addMessage` defensive (returns early when `#messages` is absent), matching the null-guards in `renderPresence`/`showTyping`/`hideTyping`. + +## Session 2026-08-13 — Chat improvements + +### Transient presence flash + system-message cleanup + +- Added `#chat-flash` container (`index.html`) + `.chat-flash` CSS (top of chat panel, `0.83rem`, 3s auto-hide with 0.5s fade-out). +- `main.ts` `ws.onmessage` routes join/leave system messages to `showFlash()` + presence list instead of `addMessage()`. +- Backend: `Broadcast` no longer persists `type: "system"` messages; `PushChatHistory` excludes system rows; `HandleAdminChatLog` filters `type <> 'system'`. +- 5s reconnect grace: disconnect schedules a delayed "left" via `time.AfterFunc` (`Hub.PresenceGracePeriod`); a reconnect within grace cancels it and skips "joined". +- Font sizing: `.message-text` set to `0.95rem` (between default and system `0.83rem`); message alignment already correct. +- Tests: `TestHub_Broadcast_SystemNotPersisted`, `TestHub_ReconnectGracePeriod`, `TestHandleAdminChatLog_ExcludesSystem`, frontend `presence flash`; ws test DB fixed to `SetMaxOpenConns(1)`. + +## Session 2026-08-13 — Playout improvements + +### HLS segment duration 2s + window 12 + +- `backend/playout/src/playlist.go`: `HLS_SEGMENT_DURATION` default `"6"→"2"`, `HLS_WINDOW_SIZE` `"24"→"12"`. +- `backend/chat/src/config/runtime.go`: `defaultHLSSegmentDuration 6→2`, `defaultHLSWindowSize 24→12`. +- ffmpeg GOP already 2s (`-g 60 -keyint_min 60` @30fps), so segments cut cleanly on keyframes — no encoder change needed. +- Updated `runtime_test.go` defaults + env-fallback tests, `.env.example`, `docs/STREAMING.md` (2s rationale + Firefox notes), `docs/operations/cloudflare-pages-setup.md`, `docs/architecture/README.md`. +- Verified: chat + playout `go test ./...` and `go vet ./...` all pass. diff --git a/.github/instructions/TODO.md b/.github/instructions/TODO.md index c101403..23109ce 100644 --- a/.github/instructions/TODO.md +++ b/.github/instructions/TODO.md @@ -2,6 +2,16 @@ > Completed tasks moved to [DONE](./DONE.md). Remaining tasks in [BACKLOG](./BACKLOG.md). +## Chat improvements (remaining) + +- [ ] Manual smoke: viewer join/leave shows a 3s flash (not a chat row); reconnect <5s shows no duplicate; admin chat-log table has no system rows + +## Playout improvements (remaining) + +- [ ] Test in Firefox: live playback, rebuffer/recovery, chat sync; record quirks in `docs/STREAMING.md` +- [ ] Rebuild + run playout; confirm `playlist.m3u8` lists ~12 segments at 2s each +- [ ] Manual: verify viewer latency and cache-hit behavior after the change + ## Setup phases remaining ### Phase 5: complete manual Cloudflare and external-service setup diff --git a/CHANGELOG.md b/CHANGELOG.md index d4cc32d..965d4c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,23 @@ ## [Unreleased] +### Added + +- **Transient presence flash**: Join/leave announcements now appear as a 3-second flash at the top of the chat panel (`#chat-flash`) with a fade-out, instead of permanent chat rows. Presence indicators still update from the same events. + +### Changed + +- **System messages no longer persisted**: The chat backend no longer stores `system`-type messages; chat history and the admin chat-log table only contain real chat messages (moderation actions remain in the audit log). +- **Reconnect grace period**: Disconnects schedule a delayed "left" announcement (5s); a reconnect within 5s cancels it and skips "joined", eliminating join/leave spam on brief reconnects. +- **HLS segment duration 2s**: Default `HLS_SEGMENT_DURATION` changed from 6 to 2 and `HLS_WINDOW_SIZE` from 24 to 12 (2s × 12 = 24s live window), reducing end-to-end latency; the 2s GOP was already in place. +- **Chat font sizing**: `.message-text` set to `0.95rem` (slightly smaller than the default, larger than the `0.83rem` system/flash text). + +### Tests + +- WebSocket: `TestHub_Broadcast_SystemNotPersisted`, `TestHub_ReconnectGracePeriod`. +- Handlers: `TestHandleAdminChatLog_ExcludesSystem`. +- Frontend: `presence flash` test. + ## [1.0.13] - 2026-08-13 ### Added diff --git a/backend/chat/src/config/runtime.go b/backend/chat/src/config/runtime.go index 3d956cf..4f50112 100644 --- a/backend/chat/src/config/runtime.go +++ b/backend/chat/src/config/runtime.go @@ -21,8 +21,8 @@ const ( // Default HLS values used when neither env nor persisted config is set. const ( - defaultHLSSegmentDuration = 6 - defaultHLSWindowSize = 24 + defaultHLSSegmentDuration = 2 + defaultHLSWindowSize = 12 ) var ( diff --git a/backend/chat/src/config/runtime_test.go b/backend/chat/src/config/runtime_test.go index c234ca2..c2b979e 100644 --- a/backend/chat/src/config/runtime_test.go +++ b/backend/chat/src/config/runtime_test.go @@ -51,26 +51,26 @@ func TestGettersDefaults(t *testing.T) { if got := MatomoSiteID(); got != "" { t.Errorf("MatomoSiteID() = %q, want empty", got) } - if got := HLSSegmentDuration(); got != 6 { - t.Errorf("HLSSegmentDuration() = %d, want 6", got) + if got := HLSSegmentDuration(); got != 2 { + t.Errorf("HLSSegmentDuration() = %d, want 2", got) } - if got := HLSWindowSize(); got != 24 { - t.Errorf("HLSWindowSize() = %d, want 24", got) + if got := HLSWindowSize(); got != 12 { + t.Errorf("HLSWindowSize() = %d, want 12", got) } } func TestGettersEnvFallback(t *testing.T) { resetRuntime(t) os.Setenv("HLS_SEGMENT_DURATION", "4") - os.Setenv("HLS_WINDOW_SIZE", "12") + os.Setenv("HLS_WINDOW_SIZE", "30") os.Setenv("MATOMO_URL", "https://analytics.example.com") os.Setenv("MATOMO_SITE_ID", "7") if got := HLSSegmentDuration(); got != 4 { t.Errorf("HLSSegmentDuration() = %d, want 4", got) } - if got := HLSWindowSize(); got != 12 { - t.Errorf("HLSWindowSize() = %d, want 12", got) + if got := HLSWindowSize(); got != 30 { + t.Errorf("HLSWindowSize() = %d, want 30", got) } if got := MatomoURL(); got != "https://analytics.example.com" { t.Errorf("MatomoURL() = %q", got) diff --git a/backend/chat/src/handlers/handlers.go b/backend/chat/src/handlers/handlers.go index 090d514..35a3251 100644 --- a/backend/chat/src/handlers/handlers.go +++ b/backend/chat/src/handlers/handlers.go @@ -1829,7 +1829,7 @@ func HandleAdminChatLog(deps *Dependencies) http.HandlerFunc { limit = parsedLimit } - dbQuery := deps.DB.Model(&db.Message{}) + dbQuery := deps.DB.Model(&db.Message{}).Where("type <> ?", "system") if room != "" { dbQuery = dbQuery.Where("room = ?", room) } diff --git a/backend/chat/src/handlers/handlers_test.go b/backend/chat/src/handlers/handlers_test.go index 1e7a813..43898b5 100644 --- a/backend/chat/src/handlers/handlers_test.go +++ b/backend/chat/src/handlers/handlers_test.go @@ -951,6 +951,32 @@ func TestHandleAdminChatLog_FilterByRoom(t *testing.T) { } } +func TestHandleAdminChatLog_ExcludesSystem(t *testing.T) { + deps, gormDB, _ := setupTestDeps(t) + + gormDB.Create(&db.Message{Room: "main", Username: "system", Text: "bob joined main", Timestamp: time.Now(), Type: "system"}) + gormDB.Create(&db.Message{Room: "main", Username: "u1", Text: "hello", Timestamp: time.Now(), Type: "public"}) + + handler := HandleAdminChatLog(deps) + req := httptest.NewRequest("GET", "/api/admin/chat-log", nil) + w := httptest.NewRecorder() + + handler(w, req) + + if w.Code != http.StatusOK { + t.Errorf("status = %d, want 200", w.Code) + } + + var resp ChatLogListResponse + json.NewDecoder(w.Body).Decode(&resp) + if resp.Total != 1 { + t.Errorf("Total = %d, want 1 (system messages must be excluded)", resp.Total) + } + if len(resp.Entries) != 1 || resp.Entries[0].Text != "hello" { + t.Errorf("entries = %v, want only the public chat message", resp.Entries) + } +} + func TestHandleAdminChatLogByID_Get(t *testing.T) { deps, gormDB, _ := setupTestDeps(t) diff --git a/backend/chat/src/ws/ws.go b/backend/chat/src/ws/ws.go index 8988e37..aca0c6f 100644 --- a/backend/chat/src/ws/ws.go +++ b/backend/chat/src/ws/ws.go @@ -41,19 +41,25 @@ type Hub struct { RateLimitWindow time.Duration AdminUsername string DMsSent int64 + + presenceMu sync.Mutex + presenceTimers map[string]*time.Timer + PresenceGracePeriod time.Duration } // NewHub creates a new WebSocket hub. func NewHub(gormDB *gorm.DB, redis db.RedisClient, instanceID string, rateMax int, rateWindow time.Duration, adminUser string) *Hub { return &Hub{ - clients: make(map[*websocket.Conn]bool), - clientMeta: make(map[*websocket.Conn]ClientInfo), - DB: gormDB, - Redis: redis, - InstanceID: instanceID, - RateLimitMax: rateMax, - RateLimitWindow: rateWindow, - AdminUsername: adminUser, + clients: make(map[*websocket.Conn]bool), + clientMeta: make(map[*websocket.Conn]ClientInfo), + DB: gormDB, + Redis: redis, + InstanceID: instanceID, + RateLimitMax: rateMax, + RateLimitWindow: rateWindow, + AdminUsername: adminUser, + presenceTimers: make(map[string]*time.Timer), + PresenceGracePeriod: 5 * time.Second, } } @@ -65,12 +71,14 @@ func (h *Hub) ConnectedClients() int { } // Broadcast sends a message to all connected clients and persists to DB. +// System messages (join/leave announcements, moderation notices) fan out but +// are not persisted, so the chat-log table only holds real chat messages. func (h *Hub) Broadcast(msg db.Message) { h.mu.Lock() defer h.mu.Unlock() - // Persist to configured SQL backend. - if h.DB != nil { + // Persist to configured SQL backend (chat messages only). + if h.DB != nil && msg.Type != "system" { h.DB.Create(&msg) } @@ -93,6 +101,41 @@ func (h *Hub) Broadcast(msg db.Message) { } } +// scheduleLeaveAnnouncement broadcasts a "left" message after a short grace +// period so quick reconnects don't produce join/leave spam. +func (h *Hub) scheduleLeaveAnnouncement(room, username string) { + key := room + "\x00" + username + h.presenceMu.Lock() + if h.presenceTimers == nil { + h.presenceTimers = make(map[string]*time.Timer) + } + grace := h.PresenceGracePeriod + if grace <= 0 { + grace = 5 * time.Second + } + h.presenceTimers[key] = time.AfterFunc(grace, func() { + h.presenceMu.Lock() + delete(h.presenceTimers, key) + h.presenceMu.Unlock() + h.Broadcast(db.Message{Room: room, Username: "system", Text: username + " left " + room, Type: "system", Timestamp: time.Now()}) + }) + h.presenceMu.Unlock() +} + +// cancelLeaveAnnouncement cancels a pending "left" announcement for a +// room+user and reports whether one existed (i.e. a reconnect within grace). +func (h *Hub) cancelLeaveAnnouncement(room, username string) bool { + key := room + "\x00" + username + h.presenceMu.Lock() + defer h.presenceMu.Unlock() + if timer, ok := h.presenceTimers[key]; ok { + timer.Stop() + delete(h.presenceTimers, key) + return true + } + return false +} + // BroadcastTyping notifies other users in the room that a user is typing. func (h *Hub) BroadcastTyping(username, room string) { h.mu.Lock() @@ -110,7 +153,7 @@ func (h *Hub) PushChatHistory(conn *websocket.Conn, room string, limit int) { return } var messages []db.Message - h.DB.Where("room = ?", room).Order("timestamp desc").Limit(limit).Find(&messages) + h.DB.Where("room = ? AND type <> ?", room, "system").Order("timestamp desc").Limit(limit).Find(&messages) // Reverse to chronological order for i := len(messages)/2 - 1; i >= 0; i-- { opp := len(messages) - 1 - i @@ -158,10 +201,14 @@ func (h *Hub) HandleConnection(w http.ResponseWriter, r *http.Request) { h.clientMeta[conn] = ClientInfo{Username: username, Room: room} h.mu.Unlock() - // Announce presence (ADR-007) + // Announce presence (ADR-007). A reconnect within the grace window cancels + // the pending "left" announcement and skips a redundant "joined". + rejoined := h.cancelLeaveAnnouncement(room, username) if h.Redis != nil && room != "" { _ = h.Redis.Set(context.Background(), fmt.Sprintf("presence:%s:%s", room, username), "online", 60*time.Second) - h.Broadcast(db.Message{Room: room, Username: "system", Text: username + " joined " + room, Type: "system", Timestamp: time.Now()}) + if !rejoined { + h.Broadcast(db.Message{Room: room, Username: "system", Text: username + " joined " + room, Type: "system", Timestamp: time.Now()}) + } } // Push chat history on join (last 50 messages) @@ -196,7 +243,7 @@ func (h *Hub) HandleConnection(w http.ResponseWriter, r *http.Request) { h.mu.Unlock() if h.Redis != nil && room != "" { _ = h.Redis.Del(context.Background(), fmt.Sprintf("presence:%s:%s", room, username)) - h.Broadcast(db.Message{Room: room, Username: "system", Text: username + " left " + room, Type: "system", Timestamp: time.Now()}) + h.scheduleLeaveAnnouncement(room, username) } conn.Close() }() diff --git a/backend/chat/src/ws/ws_test.go b/backend/chat/src/ws/ws_test.go index 42c6730..ee4b297 100644 --- a/backend/chat/src/ws/ws_test.go +++ b/backend/chat/src/ws/ws_test.go @@ -113,6 +113,11 @@ func setupTestHub(t *testing.T) (*Hub, *gorm.DB, *mockRedis) { if err != nil { t.Fatalf("failed to open test DB: %v", err) } + sqlDB, err := gormDB.DB() + if err != nil { + t.Fatalf("failed to get sql.DB: %v", err) + } + sqlDB.SetMaxOpenConns(1) gormDB.AutoMigrate(&db.Message{}, &db.ModerationLog{}) mr := newMockRedis() hub := NewHub(gormDB, mr, "test-instance", 10, 10*time.Second, "admin") @@ -560,4 +565,47 @@ func TestHub_SubscribeRoomMessages(t *testing.T) { }) mr.Publish(ctx, "room:main:messages", string(ownPayload)) time.Sleep(50 * time.Millisecond) +} + +func TestHub_Broadcast_SystemNotPersisted(t *testing.T) { + hub, _, _ := setupTestHub(t) + + hub.Broadcast(db.Message{Room: "main", Username: "system", Text: "bob joined main", Type: "system", Timestamp: time.Now()}) + + var count int64 + hub.DB.Model(&db.Message{}).Where("type = ?", "system").Count(&count) + if count != 0 { + t.Errorf("system message count = %d, want 0 (system messages must not persist)", count) + } + + hub.Broadcast(db.Message{Room: "main", Username: "bob", Text: "hi", Type: "public", Timestamp: time.Now()}) + hub.DB.Model(&db.Message{}).Where("type = ?", "public").Count(&count) + if count != 1 { + t.Errorf("public message count = %d, want 1", count) + } +} + +func TestHub_ReconnectGracePeriod(t *testing.T) { + hub, _, _ := setupTestHub(t) + hub.PresenceGracePeriod = 50 * time.Millisecond + + // Disconnect schedules a delayed "left"; reconnect cancels it. + hub.scheduleLeaveAnnouncement("main", "bob") + if !hub.cancelLeaveAnnouncement("main", "bob") { + t.Error("expected a pending leave announcement to be cancelled on reconnect") + } + if hub.cancelLeaveAnnouncement("main", "bob") { + t.Error("expected no pending leave announcement after cancellation") + } + + // A fresh disconnect that is never reconnected fires after the grace period + // and cleans up its timer. + hub.scheduleLeaveAnnouncement("main", "alice") + time.Sleep(120 * time.Millisecond) + hub.presenceMu.Lock() + _, exists := hub.presenceTimers["main\x00alice"] + hub.presenceMu.Unlock() + if exists { + t.Error("expected the leave announcement timer to clean itself up after firing") + } } \ No newline at end of file diff --git a/backend/playout/src/playlist.go b/backend/playout/src/playlist.go index f4dede2..5496de1 100644 --- a/backend/playout/src/playlist.go +++ b/backend/playout/src/playlist.go @@ -47,8 +47,8 @@ var ( restartWindow = time.Now() slateDir = "/app/slate" hlsDir = "/app/hls" - hlsSegmentDuration = envWithDefault("HLS_SEGMENT_DURATION", "6") - hlsWindowSize = envWithDefault("HLS_WINDOW_SIZE", "24") + hlsSegmentDuration = envWithDefault("HLS_SEGMENT_DURATION", "2") + hlsWindowSize = envWithDefault("HLS_WINDOW_SIZE", "12") // Unique segment naming (STREAMING.md contract: r-{runID}/seg-{seq}.m4s) runID string diff --git a/docs/STREAMING.md b/docs/STREAMING.md index c408ed6..c2ef344 100644 --- a/docs/STREAMING.md +++ b/docs/STREAMING.md @@ -68,13 +68,15 @@ Baseline packaging: | ---------------- | ------------------------------------------------------------------------------------- | | Protocol | Standard live HLS, not LL-HLS | | Segment format | fMP4 (`.m4s`) with an initialization segment; `.ts` may be supported during migration | -| Segment target | 6 seconds (`HLS_SEGMENT_DURATION=6`) | -| Live window | 24 segments (`HLS_WINDOW_SIZE=24`), about 144 seconds | +| Segment target | 2 seconds (`HLS_SEGMENT_DURATION=2`) | +| Live window | 12 segments (`HLS_WINDOW_SIZE=12`), about 24 seconds | | Codec path | Stream copy for conforming assets; no per-viewer transcoding | | Expected latency | 15–35 seconds | Segment boundaries must align with keyframes. Timestamps must remain monotonic across item boundaries. If FFmpeg cannot preserve a seamless timeline, insert `#EXT-X-DISCONTINUITY` and publish the corresponding new initialization segment. +**Segment duration rationale**: 2s segments were chosen to lower end-to-end latency and keep the live window small (12 × 2s = 24s) while staying well above the manifest TTL (1–2s). The 2s GOP (30fps, `-g 60 -keyint_min 60`) already matches the target, so segments cut cleanly on keyframes. Longer durations (4–6s) increase latency and window depth; shorter than 2s inflates segment/request overhead and R2 Class B operations with little latency gain. + Every segment name is globally unique and never reused, including after a restart or playlist loop. Use a run/session ID plus a monotonically increasing media sequence, for example: ```text @@ -128,6 +130,8 @@ The manifest must refresh within a few seconds. Repeated segment requests should The SPA uses native HLS where supported and hls.js elsewhere. +- Firefox has no native HLS, so it uses the hls.js/MSE path; fMP4 (`.m4s` + init segment) is supported. Autoplay requires `muted` + `playsinline`, which are already set on the `