Add a vision-model screenshot judge to the QA tooling
Full build / Import + headless tests (push) Successful in 32s
Full build / Full desktop build + optional release (push) Successful in 3m30s

`tools/screen_designs.py judge` sends each capture together with its concept
mockup to a vision model and asks whether the capture realises the same screen and
layout intent. This covers what pixel/perceptual metrics cannot judge.

- Prompt explicitly ignores art style, fidelity, fonts, and placeholder text, since
  the references are stylised mockups.
- Verdicts are classed match / review (mismatch below --fail-confidence, default
  0.6) / diff (confident mismatch, exits non-zero) / unknown; --json, --only,
  --dry-run, --model supported.
- Reads the repo .env for OPENROUTER_API_KEY without overriding real env vars.
- Default judge model google/gemini-2.5-flash-lite; openai/gpt-4o-mini verified too.

Tests/docs
- 7 new offline tests in tools/test_screen_designs.py (request payload embeds both
  images, JSON/fenced/parts/non-JSON parsing, threshold classification, .env
  loading); 30 Python tests pass.
- Live run: 3 captures match, 3 review, 0 confident diffs.
- README, TODO section moved to DONE, MEMORY. Version 0.2.15.
This commit is contained in:
zwitschi committed 2026-10-31 22:11:45 +01:00
1 parent 74f14014b2
commit 72b9378e69
7 files changed
+347 -21

No files matched your search

+10 -2
View File
@@ -21,8 +21,8 @@ Session memory for durable decisions, state, and gotchas. Keep concise, remove o
Windowed control smoke: `godot --path . --script res://src/test/smoke_focus_radial.gd` → exit 0.
- `main` is in sync with origin. **v0.2.2** tags the session-shell + world-follow-up commit;
CI builds the portable binaries, NSIS installer, and AppImage on the tag push.
- Next work: the two remaining TODO sections — the vision-model screenshot judge
(`tools/screen_designs.py`, OpenRouter key in `.env`) and editor tooling (GUT / `.tres` venues).
- Next work: only the optional editor-tooling section remains (GUT plugin, `.tres` venues);
everything else in `TODO.md` is complete.
- Do not re-open `override_access`. Do not bind wand to `W`. Do not remove the radio easter egg.
## Current State
@@ -95,6 +95,14 @@ Session memory for durable decisions, state, and gotchas. Keep concise, remove o
`_begin_incident()` now starts one per focused NPC (`has_open_incident()` guard), verbs call
`_ensure_incident()`, and `ShiftCoordinator._finish_shift()` closes leftovers as `unresolved`
before building the review.
- **Vision judge (2026-10-31):** `python3 tools/screen_designs.py judge` sends the capture plus
its concept mockup to a vision model (`DEFAULT_JUDGE_MODEL = google/gemini-2.5-flash-lite`,
override with `--model`; `openai/gpt-4o-mini` also works). Reads the repo `.env` via
`_load_env_file()` (never overrides real env). Verdicts are classed `match` / `review`
(mismatch below `--fail-confidence`, default 0.6) / `diff` (confident mismatch → exit 1) /
`unknown`. `--dry-run`, `--json`, `--only` supported. The prompt explicitly ignores art style,
because the references are stylised mockups; the live run gave 3 match, 3 review, 0 diff.
Python tests: `cd tools && python3 -m unittest test_asset_pipeline test_screen_designs`.
- **String formatting safety (2026-10-31):** `Strings.placeholder_count()` counts `%` specs
(ignoring `%%`), `fmt()` warns and returns the template when the arg count does not match
(instead of raising an engine error), and `validate()` lints the table for trailing `%`,