refactor: modularize chat backend, update docs for .m4s segments
This commit is contained in:
zwitschi committed
1 parent
46e878bef6
commit
b3a10ba3ef
24 files changed
+3771
-2781
No files matched your search
+27
-27
@@ -113,14 +113,14 @@ Full decision rationale, alternatives, and consequences in [`adr/ADR-001.md`](ad
|
||||
|
||||
### Quality Goals
|
||||
|
||||
| Priority | Quality Attribute | Goal | Scenario |
|
||||
| -------- | ------------------------- | -------------------------------------------------------- | -------------------------------------------------------------------------------- |
|
||||
| 1 | **Cost Efficiency** | <$25/month for moderate scale (100s concurrent) | R2 $0 egress + immutable `.ts` edge caching eliminates per-viewer bandwidth cost |
|
||||
| 2 | **Performance / Latency** | Low US latency; acceptable HLS delay | Hetzner Ashburn origin + Cloudflare US POPs (15–35s end-to-end) |
|
||||
| 3 | **Availability** | 24/7 with minimal downtime | FFmpeg/MediaMTX auto-restart loop, fallback slate, monitoring |
|
||||
| 4 | **Scalability** | Grow from 10s to 1000s viewers without proportional cost | Edge caching + R2; only request volume limits |
|
||||
| 5 | **Maintainability** | Lightweight, clear configs, low ops overhead | Static SPA, Go/Node chat, SQLite/PostgreSQL+Redis |
|
||||
| 6 | **Security / Usability** | Secure transport, moderation tools, good UX | HTTPS/WSS, Cloudflare WAF, rate limits, intuitive chat/player |
|
||||
| Priority | Quality Attribute | Goal | Scenario |
|
||||
| -------- | ------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
|
||||
| 1 | **Cost Efficiency** | <$25/month for moderate scale (100s concurrent) | R2 $0 egress + immutable `.m4s` and `.ts` edge caching eliminates per-viewer bandwidth cost |
|
||||
| 2 | **Performance / Latency** | Low US latency; acceptable HLS delay | Hetzner Ashburn origin + Cloudflare US POPs (15–35s end-to-end) |
|
||||
| 3 | **Availability** | 24/7 with minimal downtime | FFmpeg/MediaMTX auto-restart loop, fallback slate, monitoring |
|
||||
| 4 | **Scalability** | Grow from 10s to 1000s viewers without proportional cost | Edge caching + R2; only request volume limits |
|
||||
| 5 | **Maintainability** | Lightweight, clear configs, low ops overhead | Static SPA, Go/Node chat, SQLite/PostgreSQL+Redis |
|
||||
| 6 | **Security / Usability** | Secure transport, moderation tools, good UX | HTTPS/WSS, Cloudflare WAF, rate limits, intuitive chat/player |
|
||||
|
||||
### Stakeholders
|
||||
|
||||
@@ -186,7 +186,7 @@ flowchart TD
|
||||
|
||||
**External Interfaces**:
|
||||
|
||||
- **Cloudflare**: CDN (cache rules for `.ts` vs `.m3u8`), R2 (object storage, $0 egress), Pages (static hosting), WAF/TLS.
|
||||
- **Cloudflare**: CDN (cache rules for `.m4s` and `.ts` vs `.m3u8`), R2 (object storage, $0 egress), Pages (static hosting), WAF/TLS.
|
||||
- **Hetzner VPS**: Compute for playout and chat (low-cost US-East with good peering).
|
||||
- **Browsers**: Native HLS (`<video>` + hls.js) and WebSocket APIs.
|
||||
- **Source**: Pre-normalized MP4 files uploaded to R2.
|
||||
@@ -196,7 +196,7 @@ This diagram synthesizes the context views from all original ADRs. Full decision
|
||||
### Technical Context
|
||||
|
||||
- **Playout**: FFmpeg consuming normalized playlist → continuous HLS.
|
||||
- **Packaging/Delivery**: MediaMTX/SRS/Nginx → immutable `.ts` segments to R2; Cloudflare edge cache.
|
||||
- **Packaging/Delivery**: MediaMTX/SRS/Nginx → immutable `.m4s` and `.ts` segments to R2; Cloudflare edge cache.
|
||||
- **Frontend**: Static SPA with hls.js.
|
||||
- **Chat**: Lightweight WebSocket service (Node/Go) + Redis (presence/rate limits) + PostgreSQL/SQLite (history/moderation).
|
||||
- **Normalization**: All source videos must be H.264/AAC, 30 fps constant, 2s keyframes.
|
||||
@@ -212,7 +212,7 @@ This diagram synthesizes the context views from all original ADRs. Full decision
|
||||
|
||||
1. **US-East Playout on Hetzner Ashburn VPS** (`CX22` or equivalent): FFmpeg + MediaMTX/SRS for continuous playlist-to-HLS conversion (`-c copy` where possible).
|
||||
2. **Immutable HLS to Cloudflare R2**: Segments use unique names; R2 acts as origin with $0 egress.
|
||||
3. **Aggressive Edge Caching**: Long TTL for `.ts` (immutable), short TTL (1–10s) for live `.m3u8`. Tiered Cache enabled. This makes viewer delivery essentially free.
|
||||
3. **Aggressive Edge Caching**: Long TTL for `.m4s` and `.ts` (immutable), short TTL (1–10s) for live `.m3u8`. Tiered Cache enabled. This makes viewer delivery essentially free.
|
||||
4. **Lightweight Custom Frontend**: Static HTML/JS SPA + hls.js player + embedded chat sidebar.
|
||||
5. **Separate but co-located Chat Service**: WebSockets with persistence and moderation (low bandwidth).
|
||||
6. **Normalization & Monitoring**: Pre-process all videos; use dashboards + alerts for playout health, cache hit ratio, R2 ops.
|
||||
@@ -230,7 +230,7 @@ Overview (Mermaid diagram):
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Cloudflare
|
||||
CDN[CDN + Cache Rules<br/>.ts immutable long TTL<br/>.m3u8 short TTL]
|
||||
CDN[CDN + Cache Rules<br/>.m4s and .ts immutable long TTL<br/>.m3u8 short TTL]
|
||||
R2[R2 - HLS + Source]
|
||||
Pages[Pages - Static SPA]
|
||||
end
|
||||
@@ -306,7 +306,7 @@ graph TB
|
||||
|
||||
- **Role**: Continuous HLS encoding loop using FFmpeg concat demuxer
|
||||
- **Input**: `data/playlist/playlist.txt` — ordered list of MP4 file paths
|
||||
- **Output**: HLS segments (`.ts`) + live manifest (`.m3u8`) written to `data/hls/`, mirrored to R2
|
||||
- **Output**: HLS segments (`.m4s` and `.ts`) + live manifest (`.m3u8`) written to `data/hls/`, mirrored to R2
|
||||
- **Resilience**: Restart counter (>5/60s → activate slate), Prometheus metrics on port 8081
|
||||
- **Validation**: ffprobe gate on all ingested files (codec, fps, resolution, keyframe interval)
|
||||
|
||||
@@ -383,7 +383,7 @@ Module-level breakdown per service.
|
||||
2. Playout worker (FFmpeg loop with concat demuxer) reads next file, packages as HLS segments (4-6s).
|
||||
3. Segments uploaded to R2 with unique immutable names; live `.m3u8` updated.
|
||||
4. Viewer loads SPA → requests master/live manifest → hls.js player follows current edge.
|
||||
5. Cloudflare serves `.ts` from nearest edge cache (origin/R2 hit only once per POP/region).
|
||||
5. Cloudflare serves `.m4s` and `.ts` from nearest edge cache (origin/R2 hit only once per POP/region).
|
||||
6. Parallel WebSocket connection to chat service for synchronized interaction.
|
||||
|
||||
Notable aspects: Manifest freshness ensures all viewers stay on same timeline; cache hit ratio directly impacts cost and latency. Fallback slate activates on playout failure. Full cache policy, example configs (FFmpeg bash loop, Nginx RTMP HLS), and storage tables in `adr/ADR-001.md`.
|
||||
@@ -469,7 +469,7 @@ flowchart LR
|
||||
**Cloudflare Layer**:
|
||||
|
||||
- R2 bucket for media.
|
||||
- Cache Rules (`.ts`: Cache Everything, immutable, long TTL; `.m3u8`: short TTL/must-revalidate).
|
||||
- Cache Rules (`.m4s` and `.ts`: Cache Everything, immutable, long TTL; `.m3u8`: short TTL/must-revalidate).
|
||||
- WAF, rate limiting, Tiered Cache.
|
||||
- Pages for SPA hosting.
|
||||
|
||||
@@ -479,7 +479,7 @@ Future evolution: Docker Compose, managed DB, secondary regions (detailed in sub
|
||||
|
||||
### Caching & Cost Optimization
|
||||
|
||||
Immutable segment naming + targeted Cloudflare Cache Rules + R2 ($0 egress) = near-zero viewer bandwidth cost. Long TTL on `.ts`, short on live manifests. Monitor hit ratio and R2 Class B ops.
|
||||
Immutable segment naming + targeted Cloudflare Cache Rules + R2 ($0 egress) = near-zero viewer bandwidth cost. Long TTL on `.m4s` and `.ts`, short on live manifests. Monitor hit ratio and R2 Class B ops.
|
||||
|
||||
### Video Normalization & Playout
|
||||
|
||||
@@ -888,16 +888,16 @@ Document and rehearse this runbook quarterly; recovery that is not exercised is
|
||||
|
||||
## Glossary
|
||||
|
||||
| Term | Definition |
|
||||
| ----------------- | ------------------------------------------------------------------------------------------ |
|
||||
| HLS | HTTP Live Streaming – protocol using `.m3u8` manifests and media segments (`.ts` or fMP4). |
|
||||
| Immutable Segment | HLS chunk with unique, unchanging name enabling long-lived caching. |
|
||||
| R2 | Cloudflare object storage ($0 egress, ideal origin for cached HLS). |
|
||||
| Edge Cache / POP | Cloudflare points-of-presence that cache and serve content close to viewers. |
|
||||
| Playout | Continuous generation of a scheduled linear program from a playlist. |
|
||||
| Normalization | Pre-processing videos to consistent codec/fps/keyframe settings for seamless playback. |
|
||||
| Cache Rule | Cloudflare configuration defining caching behavior, TTL, and headers per URL pattern. |
|
||||
| SPOF | Single Point of Failure (current single VPS origin). |
|
||||
| Tiered Cache | Cloudflare feature reducing origin requests by checking upper-tier caches first. |
|
||||
| Term | Definition |
|
||||
| ----------------- | -------------------------------------------------------------------------------------- |
|
||||
| HLS | HTTP Live Streaming – protocol using `.m3u8` manifests and media segments (fMP4). |
|
||||
| Immutable Segment | HLS chunk with unique, unchanging name enabling long-lived caching. |
|
||||
| R2 | Cloudflare object storage ($0 egress, ideal origin for cached HLS). |
|
||||
| Edge Cache / POP | Cloudflare points-of-presence that cache and serve content close to viewers. |
|
||||
| Playout | Continuous generation of a scheduled linear program from a playlist. |
|
||||
| Normalization | Pre-processing videos to consistent codec/fps/keyframe settings for seamless playback. |
|
||||
| Cache Rule | Cloudflare configuration defining caching behavior, TTL, and headers per URL pattern. |
|
||||
| SPOF | Single Point of Failure (current single VPS origin). |
|
||||
| Tiered Cache | Cloudflare feature reducing origin requests by checking upper-tier caches first. |
|
||||
|
||||
**References**: Cloudflare R2/Cache docs, MediaMTX/SRS/FFmpeg HLS guides, Hetzner pricing, hls.js documentation, standard video bitrate calculators. All original root `ADR-001-*.md` files have been consolidated and can be archived/deleted.
|
||||
@@ -19,11 +19,11 @@ We require a completely independent 24/7 linear video channel that broadcasts a
|
||||
**Key distinctions**:
|
||||
|
||||
- **Playout**: Continuous scheduled program from playlist.
|
||||
- **Packaging**: FFmpeg → standard HLS (`.m3u8` + `.ts` segments, 4–6s duration).
|
||||
- **Packaging**: FFmpeg → standard HLS (`.m3u8` + `.m4s` segments, 4–6s duration).
|
||||
- **Delivery**: Immutable segments served via Cloudflare edge caching + R2 (zero egress cost).
|
||||
- **Interaction**: Separate lightweight WebSocket service for chat.
|
||||
|
||||
Bandwidth (not storage or compute) is the dominant cost driver. A single viewer at ~4 Mbps consumes ~1.3–1.8 TB/month; 100 concurrent viewers = ~130–180 TB/month. Direct VPS delivery is only viable for PoC/small audiences.
|
||||
Bandwidth (not storage or compute) is the dominant cost driver. A single viewer at ~4 Mbps consumes ~1.3–1.8 TB/month; 100 concurrent viewers = ~130–180 TB/month. Direct VPS delivery is only viable for PoC/small audiences. HLS segments are typically `.m4s` files, with `.m4s` being the default for fMP4 segments.
|
||||
|
||||
## Decision Drivers
|
||||
|
||||
@@ -31,7 +31,7 @@ Bandwidth (not storage or compute) is the dominant cost driver. A single viewer
|
||||
2. Full independence and ownership (no video platform lock-in).
|
||||
3. Strong US performance and low latency.
|
||||
4. True/near-true synchronization via live-style HLS (not pure VOD).
|
||||
5. Aggressive immutable caching of `.ts` segments to achieve near-zero egress.
|
||||
5. Aggressive immutable caching of `.m4s` segments to achieve near-zero egress.
|
||||
6. Browser-native playback (`hls.js` + native HLS).
|
||||
7. Lightweight, maintainable stack with modest operational complexity.
|
||||
8. Support for custom multi-room + private chat with persistence and moderation.
|
||||
@@ -58,9 +58,9 @@ Adopt a **self-managed US-East playout + Cloudflare R2 + edge-cached HLS** archi
|
||||
|
||||
### 2. Storage & Delivery (Cloudflare R2 + CDN)
|
||||
|
||||
- HLS segments (`.ts`) and manifests uploaded to **Cloudflare R2** (Eastern North America hint `enam`).
|
||||
- HLS segments (`.m4s`) and manifests uploaded to **Cloudflare R2** (Eastern North America hint `enam`).
|
||||
- Custom domain (`media.linearvoid.tv`) fronted by Cloudflare with **Cache Rules**:
|
||||
- `*.ts` (and init `.mp4` if using fMP4 later): `Cache Everything`, immutable, Edge TTL 24h–1yr (`Cache-Control: public, max-age=31536000, immutable`).
|
||||
- `*.m4s` (and init `.mp4` if using fMP4 later): `Cache Everything`, immutable, Edge TTL 24h–1yr (`Cache-Control: public, max-age=31536000, immutable`).
|
||||
- `*.m3u8`: Short TTL (1–10s) or `must-revalidate` (`Cache-Control: public, max-age=1..10, must-revalidate`).
|
||||
- Master playlist: TTL ~30s.
|
||||
- Tiered Cache enabled where possible.
|
||||
@@ -90,7 +90,7 @@ Adopt a **self-managed US-East playout + Cloudflare R2 + edge-cached HLS** archi
|
||||
|
||||
**Initial Targets**:
|
||||
|
||||
- Protocol: Standard HLS (`.ts` segments). Defer LL-HLS/CMAF/WebRTC.
|
||||
- Protocol: Standard HLS (`.m4s` segments). Defer LL-HLS/CMAF/WebRTC.
|
||||
- Max rendition: 720p30 initially (upgrade to 1080p30 or ABR later).
|
||||
- Latency: 15–35s (acceptable for linear programming/chat).
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ We must explicitly document language/runtime choices per module and the assumpti
|
||||
|
||||
1. All video assets **must** be normalized before being added to any playlist (H.264, AAC, 30 fps constant, 2s keyframes, CRF encoding).
|
||||
2. Deployment is **Docker-first**; services will be containerized.
|
||||
3. Cloudflare is the primary CDN and will be configured with specific Cache Rules (`.ts` immutable long TTL, `.m3u8` short TTL).
|
||||
3. Cloudflare is the primary CDN and will be configured with specific [Cache Rules](../../../infrastructure/cloudflare/cache-rules.md).
|
||||
4. Chat does **not** require end-to-end encryption in v1 (separate ADR if needed).
|
||||
5. Single VPS origin initially; multi-region is a later evolution.
|
||||
6. Monitoring focuses on playout health, cache hit ratio, and R2 operation counts.
|
||||
|
||||
@@ -161,4 +161,69 @@ docker exec linearvoid-db psql -U linearvoid -c "SELECT count(*) FROM messages;"
|
||||
3. Add prevention to docs if actionable
|
||||
4. Update this runbook if new failure mode discovered
|
||||
|
||||
**Recovery targets:** RPO ≤24h, RTO ≤60min. Quarterly DR drill: simulate host loss, restore from backup.
|
||||
## Quarterly Disaster Recovery Drill
|
||||
|
||||
Goal: validate recovery targets under controlled host-loss simulation.
|
||||
|
||||
**Recovery targets:** RPO <= 24h, RTO <= 60min.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Latest backup artifacts available from `scripts/backup-db.sh` and `scripts/backup-weekly.sh`
|
||||
- Operator access to Docker host and DNS controls
|
||||
- Current deployment variables available (`.env`, compose file, image tags)
|
||||
|
||||
### Drill Scope
|
||||
|
||||
- Simulate complete origin loss (all app containers unavailable)
|
||||
- Rebuild stack on clean host
|
||||
- Restore DB and media/playlist mounts
|
||||
- Verify viewer playback, chat, admin auth, monitoring targets
|
||||
|
||||
### Drill Steps
|
||||
|
||||
1. Record drill start timestamp and on-call participants.
|
||||
2. Capture pre-drill baseline:
|
||||
- `curl -fsS http://<origin>/health`
|
||||
- `curl -fsS http://<origin>:3000/health`
|
||||
- `curl -fsS http://<origin>:8081/health`
|
||||
3. Simulate origin outage:
|
||||
- `docker compose -f docker-compose.prod.yml down`
|
||||
4. Provision clean recovery host (or clean test-lab VM) and copy deployment bundle.
|
||||
5. Restore required data paths:
|
||||
- `data/playlist/`
|
||||
- `data/media/`
|
||||
- `data/chat/`
|
||||
- `data/hls/` (optional warm cache)
|
||||
6. Restore database from latest backup dump.
|
||||
7. Start stack:
|
||||
- `docker compose -f docker-compose.prod.yml up -d`
|
||||
8. Validate service health and runtime:
|
||||
- `docker ps --format "table {{.Names}}\t{{.Status}}"`
|
||||
- `curl -fsS http://<origin>/health`
|
||||
- `curl -fsS http://<origin>/hls/playlist.m3u8 | head -n 5`
|
||||
- `curl -fsS http://<origin>:9090/-/ready`
|
||||
9. Validate app behavior:
|
||||
- Viewer SPA loads and HLS plays
|
||||
- WebSocket chat connects and message round-trip works
|
||||
- Admin login/session endpoint returns success
|
||||
10. Run backup integrity check:
|
||||
- `scripts/verify-backups.sh`
|
||||
11. Record drill end timestamp and compute measured RTO/RPO.
|
||||
|
||||
### Pass/Fail Criteria
|
||||
|
||||
- Pass if all core services healthy and functional in <= 60 minutes.
|
||||
- Pass if latest recoverable data point is <= 24 hours old.
|
||||
- Fail if any critical path remains degraded (playout, chat, admin auth, monitoring).
|
||||
|
||||
### Evidence Template
|
||||
|
||||
Document each drill in incident tracker with:
|
||||
|
||||
- Date/time window
|
||||
- Environment (test-lab, staging, production)
|
||||
- Backup artifact IDs and timestamps
|
||||
- Measured RTO and RPO
|
||||
- Issues found and remediation owner
|
||||
- Follow-up ticket links
|
||||
@@ -0,0 +1,117 @@
|
||||
# Cloudflare and UptimeRobot Manual Completion Checklist
|
||||
|
||||
Purpose: execute and record all manual dashboard tasks that cannot be automated from repository workflows.
|
||||
|
||||
## Status Legend
|
||||
|
||||
- [ ] Pending
|
||||
- [x] Done
|
||||
|
||||
## 1) Cloudflare Pages Link and Build
|
||||
|
||||
- [ ] Connect repo to Cloudflare Pages project
|
||||
- [ ] Build command set to: `cd frontend && npm run build`
|
||||
- [ ] Output directory set to: `frontend/dist`
|
||||
- [ ] Production domain attached to `linearvoid.tv`
|
||||
|
||||
Evidence to capture:
|
||||
|
||||
- Screenshot of Pages deployment settings
|
||||
- Deployment URL and latest deployment timestamp
|
||||
|
||||
## 2) Tiered Cache Enablement and Validation
|
||||
|
||||
- [ ] Smart Tiered Cache enabled for zone
|
||||
- [ ] Warm request made to hot HLS segment
|
||||
- [ ] Warm request made to live manifest
|
||||
- [ ] `Cf-Cache-Status: HIT` observed on repeated requests
|
||||
|
||||
Validation commands:
|
||||
|
||||
```bash
|
||||
curl -I https://media.linearvoid.tv/hls/playlist.m3u8
|
||||
curl -I https://media.linearvoid.tv/hls/<segment>.m4s
|
||||
```
|
||||
|
||||
Evidence to capture:
|
||||
|
||||
- Headers from first request (expected MISS)
|
||||
- Headers from repeated request (expected HIT)
|
||||
|
||||
## 3) WAF and Bot Protection
|
||||
|
||||
- [ ] OWASP Managed Ruleset enabled
|
||||
- [ ] Mode set to Simulate
|
||||
- [ ] Simulate mode reviewed for 48h
|
||||
- [ ] Mode switched to Block after false-positive review
|
||||
- [ ] Bot Fight Mode enabled
|
||||
|
||||
Evidence to capture:
|
||||
|
||||
- WAF ruleset status screenshot
|
||||
- Count of blocked/challenged events after cutover
|
||||
|
||||
## 4) Rate Limiting Rules
|
||||
|
||||
- [ ] Rule created for `/api/admin/login`
|
||||
- [ ] Rule created for `/api/admin/*`
|
||||
- [ ] Threshold set to `10 requests / 10 seconds / IP`
|
||||
- [ ] Action set to Managed Challenge
|
||||
- [ ] Rule order validated
|
||||
|
||||
Validation approach:
|
||||
|
||||
- Use burst requests from a test client and verify challenge behavior without blocking legitimate admin flow.
|
||||
|
||||
Evidence to capture:
|
||||
|
||||
- Rule definitions screenshot
|
||||
- Test result notes (challenge seen / no lockout)
|
||||
|
||||
## 5) HTTPS and WSS Hardening
|
||||
|
||||
- [ ] Always Use HTTPS enabled
|
||||
- [ ] HSTS enabled with max-age 31536000
|
||||
- [ ] HSTS includeSubDomains enabled
|
||||
- [ ] HSTS preload enabled
|
||||
- [ ] Minimum TLS version set to 1.2+
|
||||
- [ ] SSL mode verified as Full (strict)
|
||||
- [ ] WSS endpoint validated through proxy (`wss://linearvoid.tv/ws`)
|
||||
|
||||
Validation commands:
|
||||
|
||||
```bash
|
||||
curl -I https://linearvoid.tv
|
||||
curl -I https://linearvoid.tv/health
|
||||
```
|
||||
|
||||
Evidence to capture:
|
||||
|
||||
- Response headers showing HSTS
|
||||
- Successful browser WebSocket connect in devtools
|
||||
|
||||
## 6) UptimeRobot Monitors
|
||||
|
||||
- [ ] HTTP monitor added for `https://linearvoid.tv/health` (5m)
|
||||
- [ ] Keyword monitor added for `https://linearvoid.tv/hls/playlist.m3u8` (keyword `#EXTM3U`, 5m)
|
||||
- [ ] Alert trigger set to 2 consecutive failures
|
||||
- [ ] Notification channels configured (email + Discord/Slack webhook)
|
||||
|
||||
Evidence to capture:
|
||||
|
||||
- Monitor dashboard screenshot
|
||||
- Test alert delivery confirmation
|
||||
|
||||
## 7) Sign-Off
|
||||
|
||||
- [ ] Cloudflare manual tasks complete
|
||||
- [ ] UptimeRobot tasks complete
|
||||
- [ ] TODO items updated to done
|
||||
- [ ] Changelog entry added for completion batch
|
||||
|
||||
Sign-off fields:
|
||||
|
||||
- Owner:
|
||||
- Date:
|
||||
- Environment (test-lab or production):
|
||||
- Notes:
|
||||
@@ -0,0 +1,203 @@
|
||||
# Scaling and Performance Playbook
|
||||
|
||||
Purpose: define repeatable method to measure performance, capture baseline data, and decide scaling actions.
|
||||
|
||||
Scope: planning and execution guide for TODO items under "Scaling & Performance (planning only)".
|
||||
|
||||
## 1) Manifest TTL Tuning Plan
|
||||
|
||||
Goal: tune `.m3u8` TTL to about `0.5x` segment duration while keeping playback stable.
|
||||
|
||||
Inputs:
|
||||
|
||||
- Segment duration (`HLS_SEGMENT_DURATION`)
|
||||
- Current manifest TTL (Cloudflare cache rule)
|
||||
- Cache outcome distribution (`Cf-Cache-Status`)
|
||||
|
||||
Procedure:
|
||||
|
||||
1. Start from current manifest TTL (2s).
|
||||
2. Capture 24h baseline headers from repeated `playlist.m3u8` requests.
|
||||
3. Test candidate TTLs: 1s, 2s, 3s.
|
||||
4. Compare:
|
||||
- rebuffer events from player logs
|
||||
- origin request increase/decrease
|
||||
- HIT ratio changes
|
||||
5. Select lowest TTL that keeps playback stable and origin load acceptable.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- No material increase in rebuffer rate.
|
||||
- Manifest staleness not observable by users.
|
||||
|
||||
## 2) TTFB Baseline Plan
|
||||
|
||||
Goal: measure warm-cache TTFB from 3+ US regions with target `< 4s`.
|
||||
|
||||
Suggested regions:
|
||||
|
||||
- US East (Virginia)
|
||||
- US Central
|
||||
- US West (California/Oregon)
|
||||
|
||||
Method:
|
||||
|
||||
1. Use WebPageTest with repeat view enabled.
|
||||
2. Run at least 5 samples per region.
|
||||
3. Record median and p95 TTFB.
|
||||
4. Store results in table (date, region, median, p95, test URL).
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Warm-cache median < 4s in all tested regions.
|
||||
- Outliers documented with likely cause.
|
||||
|
||||
## 3) WebSocket Load-Test Plan
|
||||
|
||||
Goal: characterize chat service behavior across 10 -> 100 -> 500 -> 1000 concurrent clients.
|
||||
|
||||
Tooling option A (Artillery):
|
||||
|
||||
```bash
|
||||
npx artillery run scripts/loadtest/ws-chat.yml
|
||||
```
|
||||
|
||||
Tooling option B (custom Go harness):
|
||||
|
||||
- open N concurrent WebSocket clients
|
||||
- send message every X seconds
|
||||
- measure round-trip latency and failures
|
||||
|
||||
Metrics to capture:
|
||||
|
||||
- p50/p95 message latency
|
||||
- error rate (connect, send, receive)
|
||||
- dropped connections
|
||||
- CPU/memory for `chat`, `redis`, `nginx`
|
||||
|
||||
Acceptance:
|
||||
|
||||
- p95 chat latency <= 1s at target concurrency tier.
|
||||
- error rate <= 1% during steady state.
|
||||
|
||||
## 4) Admin API Benchmark Plan
|
||||
|
||||
Goal: measure admin API p95 and identify slow endpoints.
|
||||
|
||||
Tooling option A (`k6`):
|
||||
|
||||
```bash
|
||||
k6 run scripts/loadtest/admin-api.js
|
||||
```
|
||||
|
||||
Tooling option B (`wrk`):
|
||||
|
||||
```bash
|
||||
wrk -t4 -c100 -d60s http://<host>/api/admin/overview
|
||||
```
|
||||
|
||||
Endpoints:
|
||||
|
||||
- `/api/admin/overview`
|
||||
- `/api/admin/analytics`
|
||||
- `/api/admin/chat-log`
|
||||
- `/api/admin/users`
|
||||
|
||||
Acceptance:
|
||||
|
||||
- p95 < 1s for primary admin endpoints.
|
||||
- slow endpoint list with remediation notes.
|
||||
|
||||
## 5) Cache Hit Ratio Tracking Plan
|
||||
|
||||
Goal: track Cloudflare cache hit ratio and alert when below threshold.
|
||||
|
||||
Implementation outline:
|
||||
|
||||
1. Pull Cloudflare GraphQL analytics on schedule.
|
||||
2. Export to Prometheus-compatible metric path.
|
||||
3. Add Grafana panel with 5m/1h/24h windows.
|
||||
4. Alert when hit ratio < 90% for 15m.
|
||||
|
||||
Metric target:
|
||||
|
||||
- `cache_hit_ratio >= 0.90`
|
||||
|
||||
## 6) Capacity Trigger Rules
|
||||
|
||||
Define operational triggers:
|
||||
|
||||
- Cache miss ratio > 10% for 15m
|
||||
- Chat p95 latency > 1s for 5m
|
||||
- Host CPU > 80% for 10m
|
||||
- Redis memory > 85% sustained
|
||||
|
||||
Action mapping:
|
||||
|
||||
- Level 1: investigate + restart non-critical components
|
||||
- Level 2: scale service resources (CPU/RAM)
|
||||
- Level 3: activate standby planning path
|
||||
|
||||
## 7) Phase 2 Service Separation Plan
|
||||
|
||||
Objective: separate bottlenecked services while keeping simple ops.
|
||||
|
||||
Boundaries:
|
||||
|
||||
- `playout` isolated first (CPU-heavy)
|
||||
- `chat` + `redis` grouped for latency
|
||||
- `db` isolated for IO stability
|
||||
- monitoring stack optionally separated after sustained load
|
||||
|
||||
Initial sizing guidance:
|
||||
|
||||
- playout host: 4 vCPU / 8 GB
|
||||
- chat host: 2-4 vCPU / 4-8 GB
|
||||
- db host: 2 vCPU / 4 GB + fast disk
|
||||
|
||||
Exit criteria for Phase 2:
|
||||
|
||||
- repeated trigger events over 7 days
|
||||
- capacity saturation despite tuning
|
||||
|
||||
## 8) Phase 3 Warm Standby Plan
|
||||
|
||||
Objective: second-region resilience with DNS failover.
|
||||
|
||||
Candidate layout:
|
||||
|
||||
- Primary: Hetzner Ashburn (US-East)
|
||||
- Standby: Hetzner Hillsboro (US-West)
|
||||
|
||||
Design:
|
||||
|
||||
- Warm standby stack deployed and health-checked
|
||||
- Redis replication or fallback mode documented
|
||||
- Postgres replication strategy selected (streaming replica or periodic snapshot restore)
|
||||
- Cloudflare DNS failover policy defined (health-based or manual emergency switch)
|
||||
|
||||
Failover drill requirements:
|
||||
|
||||
- periodic simulated cutover
|
||||
- measured RTO/RPO logged in incident records
|
||||
|
||||
## 9) Data Capture Template
|
||||
|
||||
For each benchmark run, capture:
|
||||
|
||||
- Date/time
|
||||
- Environment (test-lab/staging/production)
|
||||
- Commit SHA / image tag
|
||||
- Test scenario and duration
|
||||
- Raw metrics (latency, errors, resource usage)
|
||||
- Conclusion and next action
|
||||
|
||||
## 10) Execution Order
|
||||
|
||||
1. TTFB baseline
|
||||
2. WebSocket load profile
|
||||
3. Admin API benchmark
|
||||
4. Cache ratio panel + alerting
|
||||
5. Trigger tuning
|
||||
6. Phase 2 boundary decision
|
||||
7. Phase 3 standby design review
|
||||
Reference in new issue
Block a user