Add a vision-model screenshot judge to the QA tooling
`tools/screen_designs.py judge` sends each capture together with its concept mockup to a vision model and asks whether the capture realises the same screen and layout intent. This covers what pixel/perceptual metrics cannot judge. - Prompt explicitly ignores art style, fidelity, fonts, and placeholder text, since the references are stylised mockups. - Verdicts are classed match / review (mismatch below --fail-confidence, default 0.6) / diff (confident mismatch, exits non-zero) / unknown; --json, --only, --dry-run, --model supported. - Reads the repo .env for OPENROUTER_API_KEY without overriding real env vars. - Default judge model google/gemini-2.5-flash-lite; openai/gpt-4o-mini verified too. Tests/docs - 7 new offline tests in tools/test_screen_designs.py (request payload embeds both images, JSON/fenced/parts/non-JSON parsing, threshold classification, .env loading); 30 Python tests pass. - Live run: 3 captures match, 3 review, 0 confident diffs. - README, TODO section moved to DONE, MEMORY. Version 0.2.15.
This commit is contained in:
1 parent
74f14014b2
commit
72b9378e69
7 files changed
+347
-21
No files matched your search
@@ -1,20 +1,8 @@
|
||||
# TODO
|
||||
|
||||
Godot 4.3 prototype. Latest release **v0.2.14**. Headless suite **153/153**, plus the session,
|
||||
PLAN-verb, crowd, emergency, timeline, audio, scenario-content, and windowed focus/radial smokes.
|
||||
Anonymous downloads are enabled on the host.
|
||||
|
||||
## Screen design references (QA tooling)
|
||||
|
||||
- [x] `tools/screen_designs.py generate` writes one reference design per screen
|
||||
(from the `docs/` descriptions) to `docs/screens/designs/`, with an offline
|
||||
`--draft` wireframe mode and an OpenRouter `--ai` mockup mode.
|
||||
- [x] `tools/screen_designs.py compare` checks `docs/screenshots/*.png` against the
|
||||
designs (aspect ratio, size, blank-frame, and layout/palette/detail drift).
|
||||
- [ ] Vision-model judge for semantic screenshot comparison. Pixel and perceptual
|
||||
metrics cannot decide whether a frame's composition is correct; a follow-up
|
||||
should send the capture + reference design to a vision model and ask whether
|
||||
they depict the same screen.
|
||||
Godot 4.3 prototype. Latest release **v0.2.15**. Headless suite **153/153** plus 30 Python
|
||||
tooling tests; the session, PLAN-verb, crowd, emergency, timeline, audio, scenario-content, and
|
||||
windowed focus/radial smokes pass. Anonymous downloads are enabled on the host.
|
||||
|
||||
## Tooling (when an editor workflow exists)
|
||||
|
||||
|
||||
Reference in new issue
Block a user