refactor: modularize chat backend, update docs for .m4s segments
This commit is contained in:
zwitschi committed
1 parent
46e878bef6
commit
b3a10ba3ef
24 files changed
+3771
-2781
No files matched your search
@@ -0,0 +1,117 @@
|
||||
# Cloudflare and UptimeRobot Manual Completion Checklist
|
||||
|
||||
Purpose: execute and record all manual dashboard tasks that cannot be automated from repository workflows.
|
||||
|
||||
## Status Legend
|
||||
|
||||
- [ ] Pending
|
||||
- [x] Done
|
||||
|
||||
## 1) Cloudflare Pages Link and Build
|
||||
|
||||
- [ ] Connect repo to Cloudflare Pages project
|
||||
- [ ] Build command set to: `cd frontend && npm run build`
|
||||
- [ ] Output directory set to: `frontend/dist`
|
||||
- [ ] Production domain attached to `linearvoid.tv`
|
||||
|
||||
Evidence to capture:
|
||||
|
||||
- Screenshot of Pages deployment settings
|
||||
- Deployment URL and latest deployment timestamp
|
||||
|
||||
## 2) Tiered Cache Enablement and Validation
|
||||
|
||||
- [ ] Smart Tiered Cache enabled for zone
|
||||
- [ ] Warm request made to hot HLS segment
|
||||
- [ ] Warm request made to live manifest
|
||||
- [ ] `Cf-Cache-Status: HIT` observed on repeated requests
|
||||
|
||||
Validation commands:
|
||||
|
||||
```bash
|
||||
curl -I https://media.linearvoid.tv/hls/playlist.m3u8
|
||||
curl -I https://media.linearvoid.tv/hls/<segment>.m4s
|
||||
```
|
||||
|
||||
Evidence to capture:
|
||||
|
||||
- Headers from first request (expected MISS)
|
||||
- Headers from repeated request (expected HIT)
|
||||
|
||||
## 3) WAF and Bot Protection
|
||||
|
||||
- [ ] OWASP Managed Ruleset enabled
|
||||
- [ ] Mode set to Simulate
|
||||
- [ ] Simulate mode reviewed for 48h
|
||||
- [ ] Mode switched to Block after false-positive review
|
||||
- [ ] Bot Fight Mode enabled
|
||||
|
||||
Evidence to capture:
|
||||
|
||||
- WAF ruleset status screenshot
|
||||
- Count of blocked/challenged events after cutover
|
||||
|
||||
## 4) Rate Limiting Rules
|
||||
|
||||
- [ ] Rule created for `/api/admin/login`
|
||||
- [ ] Rule created for `/api/admin/*`
|
||||
- [ ] Threshold set to `10 requests / 10 seconds / IP`
|
||||
- [ ] Action set to Managed Challenge
|
||||
- [ ] Rule order validated
|
||||
|
||||
Validation approach:
|
||||
|
||||
- Use burst requests from a test client and verify challenge behavior without blocking legitimate admin flow.
|
||||
|
||||
Evidence to capture:
|
||||
|
||||
- Rule definitions screenshot
|
||||
- Test result notes (challenge seen / no lockout)
|
||||
|
||||
## 5) HTTPS and WSS Hardening
|
||||
|
||||
- [ ] Always Use HTTPS enabled
|
||||
- [ ] HSTS enabled with max-age 31536000
|
||||
- [ ] HSTS includeSubDomains enabled
|
||||
- [ ] HSTS preload enabled
|
||||
- [ ] Minimum TLS version set to 1.2+
|
||||
- [ ] SSL mode verified as Full (strict)
|
||||
- [ ] WSS endpoint validated through proxy (`wss://linearvoid.tv/ws`)
|
||||
|
||||
Validation commands:
|
||||
|
||||
```bash
|
||||
curl -I https://linearvoid.tv
|
||||
curl -I https://linearvoid.tv/health
|
||||
```
|
||||
|
||||
Evidence to capture:
|
||||
|
||||
- Response headers showing HSTS
|
||||
- Successful browser WebSocket connect in devtools
|
||||
|
||||
## 6) UptimeRobot Monitors
|
||||
|
||||
- [ ] HTTP monitor added for `https://linearvoid.tv/health` (5m)
|
||||
- [ ] Keyword monitor added for `https://linearvoid.tv/hls/playlist.m3u8` (keyword `#EXTM3U`, 5m)
|
||||
- [ ] Alert trigger set to 2 consecutive failures
|
||||
- [ ] Notification channels configured (email + Discord/Slack webhook)
|
||||
|
||||
Evidence to capture:
|
||||
|
||||
- Monitor dashboard screenshot
|
||||
- Test alert delivery confirmation
|
||||
|
||||
## 7) Sign-Off
|
||||
|
||||
- [ ] Cloudflare manual tasks complete
|
||||
- [ ] UptimeRobot tasks complete
|
||||
- [ ] TODO items updated to done
|
||||
- [ ] Changelog entry added for completion batch
|
||||
|
||||
Sign-off fields:
|
||||
|
||||
- Owner:
|
||||
- Date:
|
||||
- Environment (test-lab or production):
|
||||
- Notes:
|
||||
@@ -0,0 +1,203 @@
|
||||
# Scaling and Performance Playbook
|
||||
|
||||
Purpose: define repeatable method to measure performance, capture baseline data, and decide scaling actions.
|
||||
|
||||
Scope: planning and execution guide for TODO items under "Scaling & Performance (planning only)".
|
||||
|
||||
## 1) Manifest TTL Tuning Plan
|
||||
|
||||
Goal: tune `.m3u8` TTL to about `0.5x` segment duration while keeping playback stable.
|
||||
|
||||
Inputs:
|
||||
|
||||
- Segment duration (`HLS_SEGMENT_DURATION`)
|
||||
- Current manifest TTL (Cloudflare cache rule)
|
||||
- Cache outcome distribution (`Cf-Cache-Status`)
|
||||
|
||||
Procedure:
|
||||
|
||||
1. Start from current manifest TTL (2s).
|
||||
2. Capture 24h baseline headers from repeated `playlist.m3u8` requests.
|
||||
3. Test candidate TTLs: 1s, 2s, 3s.
|
||||
4. Compare:
|
||||
- rebuffer events from player logs
|
||||
- origin request increase/decrease
|
||||
- HIT ratio changes
|
||||
5. Select lowest TTL that keeps playback stable and origin load acceptable.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- No material increase in rebuffer rate.
|
||||
- Manifest staleness not observable by users.
|
||||
|
||||
## 2) TTFB Baseline Plan
|
||||
|
||||
Goal: measure warm-cache TTFB from 3+ US regions with target `< 4s`.
|
||||
|
||||
Suggested regions:
|
||||
|
||||
- US East (Virginia)
|
||||
- US Central
|
||||
- US West (California/Oregon)
|
||||
|
||||
Method:
|
||||
|
||||
1. Use WebPageTest with repeat view enabled.
|
||||
2. Run at least 5 samples per region.
|
||||
3. Record median and p95 TTFB.
|
||||
4. Store results in table (date, region, median, p95, test URL).
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Warm-cache median < 4s in all tested regions.
|
||||
- Outliers documented with likely cause.
|
||||
|
||||
## 3) WebSocket Load-Test Plan
|
||||
|
||||
Goal: characterize chat service behavior across 10 -> 100 -> 500 -> 1000 concurrent clients.
|
||||
|
||||
Tooling option A (Artillery):
|
||||
|
||||
```bash
|
||||
npx artillery run scripts/loadtest/ws-chat.yml
|
||||
```
|
||||
|
||||
Tooling option B (custom Go harness):
|
||||
|
||||
- open N concurrent WebSocket clients
|
||||
- send message every X seconds
|
||||
- measure round-trip latency and failures
|
||||
|
||||
Metrics to capture:
|
||||
|
||||
- p50/p95 message latency
|
||||
- error rate (connect, send, receive)
|
||||
- dropped connections
|
||||
- CPU/memory for `chat`, `redis`, `nginx`
|
||||
|
||||
Acceptance:
|
||||
|
||||
- p95 chat latency <= 1s at target concurrency tier.
|
||||
- error rate <= 1% during steady state.
|
||||
|
||||
## 4) Admin API Benchmark Plan
|
||||
|
||||
Goal: measure admin API p95 and identify slow endpoints.
|
||||
|
||||
Tooling option A (`k6`):
|
||||
|
||||
```bash
|
||||
k6 run scripts/loadtest/admin-api.js
|
||||
```
|
||||
|
||||
Tooling option B (`wrk`):
|
||||
|
||||
```bash
|
||||
wrk -t4 -c100 -d60s http://<host>/api/admin/overview
|
||||
```
|
||||
|
||||
Endpoints:
|
||||
|
||||
- `/api/admin/overview`
|
||||
- `/api/admin/analytics`
|
||||
- `/api/admin/chat-log`
|
||||
- `/api/admin/users`
|
||||
|
||||
Acceptance:
|
||||
|
||||
- p95 < 1s for primary admin endpoints.
|
||||
- slow endpoint list with remediation notes.
|
||||
|
||||
## 5) Cache Hit Ratio Tracking Plan
|
||||
|
||||
Goal: track Cloudflare cache hit ratio and alert when below threshold.
|
||||
|
||||
Implementation outline:
|
||||
|
||||
1. Pull Cloudflare GraphQL analytics on schedule.
|
||||
2. Export to Prometheus-compatible metric path.
|
||||
3. Add Grafana panel with 5m/1h/24h windows.
|
||||
4. Alert when hit ratio < 90% for 15m.
|
||||
|
||||
Metric target:
|
||||
|
||||
- `cache_hit_ratio >= 0.90`
|
||||
|
||||
## 6) Capacity Trigger Rules
|
||||
|
||||
Define operational triggers:
|
||||
|
||||
- Cache miss ratio > 10% for 15m
|
||||
- Chat p95 latency > 1s for 5m
|
||||
- Host CPU > 80% for 10m
|
||||
- Redis memory > 85% sustained
|
||||
|
||||
Action mapping:
|
||||
|
||||
- Level 1: investigate + restart non-critical components
|
||||
- Level 2: scale service resources (CPU/RAM)
|
||||
- Level 3: activate standby planning path
|
||||
|
||||
## 7) Phase 2 Service Separation Plan
|
||||
|
||||
Objective: separate bottlenecked services while keeping simple ops.
|
||||
|
||||
Boundaries:
|
||||
|
||||
- `playout` isolated first (CPU-heavy)
|
||||
- `chat` + `redis` grouped for latency
|
||||
- `db` isolated for IO stability
|
||||
- monitoring stack optionally separated after sustained load
|
||||
|
||||
Initial sizing guidance:
|
||||
|
||||
- playout host: 4 vCPU / 8 GB
|
||||
- chat host: 2-4 vCPU / 4-8 GB
|
||||
- db host: 2 vCPU / 4 GB + fast disk
|
||||
|
||||
Exit criteria for Phase 2:
|
||||
|
||||
- repeated trigger events over 7 days
|
||||
- capacity saturation despite tuning
|
||||
|
||||
## 8) Phase 3 Warm Standby Plan
|
||||
|
||||
Objective: second-region resilience with DNS failover.
|
||||
|
||||
Candidate layout:
|
||||
|
||||
- Primary: Hetzner Ashburn (US-East)
|
||||
- Standby: Hetzner Hillsboro (US-West)
|
||||
|
||||
Design:
|
||||
|
||||
- Warm standby stack deployed and health-checked
|
||||
- Redis replication or fallback mode documented
|
||||
- Postgres replication strategy selected (streaming replica or periodic snapshot restore)
|
||||
- Cloudflare DNS failover policy defined (health-based or manual emergency switch)
|
||||
|
||||
Failover drill requirements:
|
||||
|
||||
- periodic simulated cutover
|
||||
- measured RTO/RPO logged in incident records
|
||||
|
||||
## 9) Data Capture Template
|
||||
|
||||
For each benchmark run, capture:
|
||||
|
||||
- Date/time
|
||||
- Environment (test-lab/staging/production)
|
||||
- Commit SHA / image tag
|
||||
- Test scenario and duration
|
||||
- Raw metrics (latency, errors, resource usage)
|
||||
- Conclusion and next action
|
||||
|
||||
## 10) Execution Order
|
||||
|
||||
1. TTFB baseline
|
||||
2. WebSocket load profile
|
||||
3. Admin API benchmark
|
||||
4. Cache ratio panel + alerting
|
||||
5. Trigger tuning
|
||||
6. Phase 2 boundary decision
|
||||
7. Phase 3 standby design review
|
||||
Reference in new issue
Block a user