Harden the DDG collector with token-bucket pacing and prefer the ddgs package
This commit is contained in:
1 parent
480a1ba05f
commit
87e1c9e7a5
5 files changed
+426
-32
No files matched your search
@@ -5,9 +5,12 @@ Session memory for durable decisions, state, and gotchas. Keep concise, remove o
|
||||
## Decisions
|
||||
|
||||
- **DuckDuckGo collection tool (2026-11-03):** `tools/refcollect.py` wraps the `duckduckgo_search`
|
||||
package (installed locally; renamed `ddgs` upstream) and retries DDG's HTTP 202 rate limit with
|
||||
backoff. DDG rate-limits by IP for a while after probing, so the per-venue collections should be
|
||||
run in small spaced batches. Downloads are `local_only`/`unchecked-local` and never committed.
|
||||
package (installed locally; renamed `ddgs` upstream). All outbound requests share one
|
||||
`TokenBucket` (`--rate` 0.5 req/s, `--burst` 3) and a polite retry routine: full-jitter
|
||||
exponential backoff, `Retry-After` honoured, permanent HTTP failures fail fast, six attempts
|
||||
maximum. Requests send a complete Firefox header set, not just the UA. DDG rate-limits by IP for
|
||||
a while after probing, so the per-venue collections should still be run in small spaced batches.
|
||||
Downloads are `local_only`/`unchecked-local` and never committed.
|
||||
- **Local-only reference override (2026-11-03, owner decision):** for development-only photo
|
||||
research the licence gate does **not** apply. Downloads for this track live in `tmp/research/`
|
||||
(gitignored) and may come from DuckDuckGo image search, street-view frames, neighbouring
|
||||
|
||||
Reference in new issue
Block a user