Harden the DDG collector with token-bucket pacing and prefer the ddgs package
This commit is contained in:
1 parent
480a1ba05f
commit
87e1c9e7a5
5 files changed
+426
-32
No files matched your search
@@ -44,7 +44,8 @@ licence-free zone only.
|
||||
- [x] Add a DuckDuckGo image-search fetcher: `tools/refcollect.py search <query> --slug <block>
|
||||
--limit N` downloads top results into `tmp/research/<block>/` with a source-URL manifest
|
||||
(no licence filter, marked `local_only`/`unchecked-local`), and `sheet` composes a local
|
||||
contact sheet. Retries DDG's HTTP 202 rate limit with backoff. Tests in
|
||||
contact sheet. Requests go through a token bucket plus polite jittered backoff that honours
|
||||
`Retry-After`, and send a full Firefox header set. Tests in
|
||||
`tools/test_refcollect.py`. **Run note:** DDG rate-limits aggressively; space batches and
|
||||
run the per-venue collection below when the limit clears.
|
||||
- [ ] Collect Hollywood block reference: El Capitan facade (day/night/close-up), the El Capitan
|
||||
|
||||
Reference in new issue
Block a user