Add a vision-model screenshot judge to the QA tooling
`tools/screen_designs.py judge` sends each capture together with its concept mockup to a vision model and asks whether the capture realises the same screen and layout intent. This covers what pixel/perceptual metrics cannot judge. - Prompt explicitly ignores art style, fidelity, fonts, and placeholder text, since the references are stylised mockups. - Verdicts are classed match / review (mismatch below --fail-confidence, default 0.6) / diff (confident mismatch, exits non-zero) / unknown; --json, --only, --dry-run, --model supported. - Reads the repo .env for OPENROUTER_API_KEY without overriding real env vars. - Default judge model google/gemini-2.5-flash-lite; openai/gpt-4o-mini verified too. Tests/docs - 7 new offline tests in tools/test_screen_designs.py (request payload embeds both images, JSON/fenced/parts/non-JSON parsing, threshold classification, .env loading); 30 Python tests pass. - Live run: 3 captures match, 3 review, 0 confident diffs. - README, TODO section moved to DONE, MEMORY. Version 0.2.15.
This commit is contained in:
1 parent
74f14014b2
commit
72b9378e69
7 files changed
+347
-21
No files matched your search
@@ -31,7 +31,7 @@ The central design goal is simple:
|
||||
> Downloads are public (no sign-in required) since the host's instance-wide
|
||||
> `REQUIRE_SIGNIN_VIEW` was disabled on 2026-10-31.
|
||||
>
|
||||
> **Latest: v0.2.14** — string-formatting safety, plus public downloads (no sign-in) on the host.
|
||||
> **Latest: v0.2.15** — vision-model screenshot judge for the QA tooling.
|
||||
> Releases are built automatically when a `vX.Y.Z` tag is pushed
|
||||
> (see [`docs/RELEASING.md`](docs/RELEASING.md)); manual builds use
|
||||
> `./tools/build_releases.sh`. Installer/AppImage packaging runs in the same pipeline
|
||||
@@ -205,13 +205,26 @@ python3 tools/screen_designs.py generate
|
||||
# Compare docs/screenshots/*.png against the designs
|
||||
python3 tools/screen_designs.py compare
|
||||
python3 tools/screen_designs.py compare --json
|
||||
|
||||
# Ask a vision model whether each capture realises its design's screen and layout
|
||||
# (requires OPENROUTER_API_KEY; the repo .env is read automatically)
|
||||
python3 tools/screen_designs.py judge
|
||||
python3 tools/screen_designs.py judge --model openai/gpt-4o-mini --json
|
||||
python3 tools/screen_designs.py judge --only 03- # one capture
|
||||
python3 tools/screen_designs.py judge --dry-run # build requests, no network
|
||||
```
|
||||
|
||||
`compare` flags gross drift: wrong aspect ratio, undersized frame, near-blank
|
||||
frame, and large layout/palette/detail distance. It is a regression gate, not a
|
||||
semantic judge — pixel and perceptual metrics cannot decide whether a frame's
|
||||
composition is *right*, so a vision-model pass is the documented follow-up (see
|
||||
`TODO.md`). Regenerate the designs after changing screen layout, then re-run
|
||||
composition is *right*.
|
||||
|
||||
`judge` covers that gap: it sends each capture together with its concept mockup to
|
||||
a vision model and asks whether the capture realises the same screen and layout
|
||||
intent. Art style, fidelity, fonts, and placeholder text are explicitly ignored.
|
||||
Verdicts are classed as `match`, `review` (a mismatch below `--fail-confidence`,
|
||||
default 0.6 — expected when a concept mockup is more stylised than the game), or
|
||||
`diff` (a confident mismatch, which exits non-zero). Regenerate the designs after changing screen layout, then re-run
|
||||
`compare` after regenerating the game screenshots with
|
||||
`src/test/capture_screenshots.gd`.
|
||||
|
||||
|
||||
Reference in new issue
Block a user