Make the vision judge discover cost-effective OpenRouter models
Full build / Import + headless tests (push) Successful in 40s
Full build / Full desktop build + optional release (push) Successful in 4m6s

Add openrouter_judge.py to rank live image+JSON models by advertised price,
with ordered fallbacks, auth-error handling and auto selection. Wire both judges
to --model auto, record the model used, and add fake-catalogue/mocked-call
tests. Re-run the judges with the valid key: 13/13 screen designs match, 0 diff;
the Forward+ A/B report records the selected model and ~$0.006 spend. Headless
231/231, Python 94/94.
This commit is contained in:
zwitschi committed 2026-11-01 19:55:43 +01:00
1 parent 28f2a48811
commit 659b0e35b2
15 files changed
+535 -65

No files matched your search

+25
View File
@@ -1311,3 +1311,28 @@ path.
to Compatibility instead of failing to start). Verified with the preset in the project
root: `godot --headless --path . --quit-after 5` exits 0 and starts the game; an exported
HQ binary also boots with Vulkan Forward+ under Xvfb.
## Vision judge reliability (v0.7.2)
Completed 2026-11-01. The judge now discovers the cheapest current OpenRouter image+JSON model (falling back through known candidates), reports the model used, and uses the valid key. Full screen-design judge: 13/13 match, 0 diff.
The judge failed with an invalid-key HTTP 400 even though the `.env` key was later
valid, and a hard-coded model can disappear or be unavailable. Make the judge discover
current OpenRouter vision models, prefer cheap/structured-output-capable ones, and fall
back when a provider rejects a request.
- [x] Add `tools/openrouter_judge.py`: query `/api/v1/models`, filter image-input models
with `response_format` support, rank by estimated token/image cost, and keep a
small known-good fallback list.
- [x] Let both `screen_designs.py judge` and `scene_concepts.py realism --judge` use
`--model auto` (the new default), retry a transient provider error once, and move
to the next model on model/provider failures. An invalid key fails with exit 2.
- [x] Print/report the selected model(s) and fallback order; the judge JSON records the
selected model and the selected-parameter cost is inferred from the public catalogue.
- [x] Add `tools/test_openrouter_judge.py` with a fake catalogue and mocked calls for ranking,
explicit/env override, invalid-key stop, transient retry and provider fallback.
- [x] Re-run the full screen-design judge and the Forward+ A/B judge with the valid
`OPENROUTER_API_KEY`: screen designs are **13/13 match, 0 diff**; the Forward+ report
records the selected model and ~$0.006 spend.
- [x] Update `docs/asset-tooling.md`, `MEMORY.md`, `README.md` and `CHANGELOG.md`; the
public catalogue picks the cheapest image+JSON model (`z-ai/glm-5.3-flash` at run time).