Harden the DDG collector with token-bucket pacing and prefer the ddgs package
This commit is contained in:
1 parent
480a1ba05f
commit
87e1c9e7a5
5 files changed
+426
-32
No files matched your search
@@ -229,7 +229,12 @@ python3 tools/refcollect.py sheet --slug comedy-store # local contact sheet (n
|
||||
```
|
||||
|
||||
Requires `pip install duckduckgo-search` (renamed `ddgs` upstream). DuckDuckGo rate-limits by IP
|
||||
(HTTP 202) and the tool retries with backoff, so run per-venue batches spaced out.
|
||||
(HTTP 202), so every request — the search itself and each image download — passes through a shared
|
||||
token bucket (`--rate`, default 0.5 req/s, and `--burst`, default 3) and a polite retry routine:
|
||||
transient statuses (408/425/429/5xx/52x) and network errors back off exponentially with full
|
||||
jitter, `Retry-After` is honoured, permanent failures fail fast, and the loop gives up after six
|
||||
attempts. Requests carry a full, coherent Firefox header set (`--user-agent` overrides the UA
|
||||
string); the UA alone is no longer enough to look like a browser.
|
||||
|
||||
### Reference-board research (`refboard`)
|
||||
|
||||
|
||||
Reference in new issue
Block a user