Harden the DDG collector with token-bucket pacing and prefer the ddgs package
Full build / Import + headless tests (push) Successful in 40s
Full build / Full desktop build + optional release (push) Skipped

This commit is contained in:
zwitschi committed 2026-11-03 17:32:39 +01:00
1 parent 480a1ba05f
commit 87e1c9e7a5
5 files changed
+426 -32

No files matched your search

+6 -3
View File
@@ -5,9 +5,12 @@ Session memory for durable decisions, state, and gotchas. Keep concise, remove o
## Decisions
- **DuckDuckGo collection tool (2026-11-03):** `tools/refcollect.py` wraps the `duckduckgo_search`
package (installed locally; renamed `ddgs` upstream) and retries DDG's HTTP 202 rate limit with
backoff. DDG rate-limits by IP for a while after probing, so the per-venue collections should be
run in small spaced batches. Downloads are `local_only`/`unchecked-local` and never committed.
package (installed locally; renamed `ddgs` upstream). All outbound requests share one
`TokenBucket` (`--rate` 0.5 req/s, `--burst` 3) and a polite retry routine: full-jitter
exponential backoff, `Retry-After` honoured, permanent HTTP failures fail fast, six attempts
maximum. Requests send a complete Firefox header set, not just the UA. DDG rate-limits by IP for
a while after probing, so the per-venue collections should still be run in small spaced batches.
Downloads are `local_only`/`unchecked-local` and never committed.
- **Local-only reference override (2026-11-03, owner decision):** for development-only photo
research the licence gate does **not** apply. Downloads for this track live in `tmp/research/`
(gitignored) and may come from DuckDuckGo image search, street-view frames, neighbouring