Add a vision-model screenshot judge to the QA tooling
`tools/screen_designs.py judge` sends each capture together with its concept mockup to a vision model and asks whether the capture realises the same screen and layout intent. This covers what pixel/perceptual metrics cannot judge. - Prompt explicitly ignores art style, fidelity, fonts, and placeholder text, since the references are stylised mockups. - Verdicts are classed match / review (mismatch below --fail-confidence, default 0.6) / diff (confident mismatch, exits non-zero) / unknown; --json, --only, --dry-run, --model supported. - Reads the repo .env for OPENROUTER_API_KEY without overriding real env vars. - Default judge model google/gemini-2.5-flash-lite; openai/gpt-4o-mini verified too. Tests/docs - 7 new offline tests in tools/test_screen_designs.py (request payload embeds both images, JSON/fenced/parts/non-JSON parsing, threshold classification, .env loading); 30 Python tests pass. - Live run: 3 captures match, 3 review, 0 confident diffs. - README, TODO section moved to DONE, MEMORY. Version 0.2.15.
This commit is contained in:
1 parent
74f14014b2
commit
72b9378e69
7 files changed
+347
-21
No files matched your search
@@ -21,8 +21,8 @@ Session memory for durable decisions, state, and gotchas. Keep concise, remove o
|
||||
Windowed control smoke: `godot --path . --script res://src/test/smoke_focus_radial.gd` → exit 0.
|
||||
- `main` is in sync with origin. **v0.2.2** tags the session-shell + world-follow-up commit;
|
||||
CI builds the portable binaries, NSIS installer, and AppImage on the tag push.
|
||||
- Next work: the two remaining TODO sections — the vision-model screenshot judge
|
||||
(`tools/screen_designs.py`, OpenRouter key in `.env`) and editor tooling (GUT / `.tres` venues).
|
||||
- Next work: only the optional editor-tooling section remains (GUT plugin, `.tres` venues);
|
||||
everything else in `TODO.md` is complete.
|
||||
- Do not re-open `override_access`. Do not bind wand to `W`. Do not remove the radio easter egg.
|
||||
|
||||
## Current State
|
||||
@@ -95,6 +95,14 @@ Session memory for durable decisions, state, and gotchas. Keep concise, remove o
|
||||
`_begin_incident()` now starts one per focused NPC (`has_open_incident()` guard), verbs call
|
||||
`_ensure_incident()`, and `ShiftCoordinator._finish_shift()` closes leftovers as `unresolved`
|
||||
before building the review.
|
||||
- **Vision judge (2026-10-31):** `python3 tools/screen_designs.py judge` sends the capture plus
|
||||
its concept mockup to a vision model (`DEFAULT_JUDGE_MODEL = google/gemini-2.5-flash-lite`,
|
||||
override with `--model`; `openai/gpt-4o-mini` also works). Reads the repo `.env` via
|
||||
`_load_env_file()` (never overrides real env). Verdicts are classed `match` / `review`
|
||||
(mismatch below `--fail-confidence`, default 0.6) / `diff` (confident mismatch → exit 1) /
|
||||
`unknown`. `--dry-run`, `--json`, `--only` supported. The prompt explicitly ignores art style,
|
||||
because the references are stylised mockups; the live run gave 3 match, 3 review, 0 diff.
|
||||
Python tests: `cd tools && python3 -m unittest test_asset_pipeline test_screen_designs`.
|
||||
- **String formatting safety (2026-10-31):** `Strings.placeholder_count()` counts `%` specs
|
||||
(ignoring `%%`), `fmt()` warns and returns the template when the arg count does not match
|
||||
(instead of raising an engine error), and `validate()` lints the table for trailing `%`,
|
||||
|
||||
Reference in new issue
Block a user