Make the vision judge discover cost-effective OpenRouter models
Full build / Import + headless tests (push) Successful in 40s
Full build / Full desktop build + optional release (push) Successful in 4m6s

Add openrouter_judge.py to rank live image+JSON models by advertised price,
with ordered fallbacks, auth-error handling and auto selection. Wire both judges
to --model auto, record the model used, and add fake-catalogue/mocked-call
tests. Re-run the judges with the valid key: 13/13 screen designs match, 0 diff;
the Forward+ A/B report records the selected model and ~$0.006 spend. Headless
231/231, Python 94/94.
This commit is contained in:
zwitschi committed 2026-11-01 19:55:43 +01:00
1 parent 28f2a48811
commit 659b0e35b2
15 files changed
+535 -65

No files matched your search

+6
View File
@@ -185,6 +185,12 @@ python3 tools/scene_concepts.py realism --run docs/screens/realism/<run> --judge
--model google/gemini-2.5-flash --repeats 3
```
Judges default to `--model auto`: `tools/openrouter_judge.py` reads the live
OpenRouter catalogue, keeps image-input models that support `response_format`,
and ranks them by advertised token/image price. The run falls through the list
on provider/model errors and aborts on an invalid key. Use
`GIG_LINE_JUDGE_MODEL` or `--model <id>` for a pinned comparison.
`capture_realism.gd` runs six stages (day, portrait, evening, late, Joke Shop, studio);
`--stages=day,late` limits it while tuning; `--flashlight` turns on the guard lamp for
night stages (used by the Desktop HQ A/B). The `portrait` shot freezes one NPC on open sidewalk