Harden the DDG collector with token-bucket pacing and prefer the ddgs package
Full build / Import + headless tests (push) Successful in 40s
Full build / Full desktop build + optional release (push) Skipped

This commit is contained in:
zwitschi committed 2026-11-03 17:32:39 +01:00
1 parent 480a1ba05f
commit 87e1c9e7a5
5 files changed
+426 -32

No files matched your search

+6 -1
View File
@@ -229,7 +229,12 @@ python3 tools/refcollect.py sheet --slug comedy-store # local contact sheet (n
```
Requires `pip install duckduckgo-search` (renamed `ddgs` upstream). DuckDuckGo rate-limits by IP
(HTTP 202) and the tool retries with backoff, so run per-venue batches spaced out.
(HTTP 202), so every request — the search itself and each image download — passes through a shared
token bucket (`--rate`, default 0.5 req/s, and `--burst`, default 3) and a polite retry routine:
transient statuses (408/425/429/5xx/52x) and network errors back off exponentially with full
jitter, `Retry-After` is honoured, permanent failures fail fast, and the loop gives up after six
attempts. Requests carry a full, coherent Firefox header set (`--user-agent` overrides the UA
string); the UA alone is no longer enough to look like a browser.
### Reference-board research (`refboard`)