Harden the DDG collector with token-bucket pacing and prefer the ddgs package
Full build / Import + headless tests (push) Successful in 40s
Full build / Full desktop build + optional release (push) Skipped

This commit is contained in:
zwitschi committed 2026-11-03 17:32:39 +01:00
1 parent 480a1ba05f
commit 87e1c9e7a5
5 files changed
+426 -32

No files matched your search

+2 -1
View File
@@ -44,7 +44,8 @@ licence-free zone only.
- [x] Add a DuckDuckGo image-search fetcher: `tools/refcollect.py search <query> --slug <block>
--limit N` downloads top results into `tmp/research/<block>/` with a source-URL manifest
(no licence filter, marked `local_only`/`unchecked-local`), and `sheet` composes a local
contact sheet. Retries DDG's HTTP 202 rate limit with backoff. Tests in
contact sheet. Requests go through a token bucket plus polite jittered backoff that honours
`Retry-After`, and send a full Firefox header set. Tests in
`tools/test_refcollect.py`. **Run note:** DDG rate-limits aggressively; space batches and
run the per-venue collection below when the limit clears.
- [ ] Collect Hollywood block reference: El Capitan facade (day/night/close-up), the El Capitan