Add a vision-model screenshot judge to the QA tooling
Full build / Import + headless tests (push) Successful in 32s
Full build / Full desktop build + optional release (push) Successful in 3m30s

`tools/screen_designs.py judge` sends each capture together with its concept
mockup to a vision model and asks whether the capture realises the same screen and
layout intent. This covers what pixel/perceptual metrics cannot judge.

- Prompt explicitly ignores art style, fidelity, fonts, and placeholder text, since
  the references are stylised mockups.
- Verdicts are classed match / review (mismatch below --fail-confidence, default
  0.6) / diff (confident mismatch, exits non-zero) / unknown; --json, --only,
  --dry-run, --model supported.
- Reads the repo .env for OPENROUTER_API_KEY without overriding real env vars.
- Default judge model google/gemini-2.5-flash-lite; openai/gpt-4o-mini verified too.

Tests/docs
- 7 new offline tests in tools/test_screen_designs.py (request payload embeds both
  images, JSON/fenced/parts/non-JSON parsing, threshold classification, .env
  loading); 30 Python tests pass.
- Live run: 3 captures match, 3 review, 0 confident diffs.
- README, TODO section moved to DONE, MEMORY. Version 0.2.15.
This commit is contained in:
zwitschi committed 2026-10-31 22:11:45 +01:00
1 parent 74f14014b2
commit 72b9378e69
7 files changed
+347 -21

No files matched your search

+15
View File
@@ -568,3 +568,18 @@ Completed 2026-10-31 (working tree). Headless **153/153**.
- [x] Keep `%` placeholders stable in `Strings.fmt` when adding keys (positional formatting).
## Screen design references (QA tooling)
Completed 2026-10-31 (working tree). 30 Python tooling tests pass; the live judge run classed 3 captures as match and 3 as review (concept mockups are more stylised), with no confident mismatch.
- [x] `tools/screen_designs.py generate` writes one reference design per screen
(from the `docs/` descriptions) to `docs/screens/designs/`, with an offline
`--draft` wireframe mode and an OpenRouter `--ai` mockup mode.
- [x] `tools/screen_designs.py compare` checks `docs/screenshots/*.png` against the
designs (aspect ratio, size, blank-frame, and layout/palette/detail drift).
- [x] Vision-model judge for semantic screenshot comparison: `judge` sends the
capture + reference design to a vision model, classes verdicts as
match/review/diff with a confidence threshold, supports `--dry-run`, `--json`,
`--only`, and `--model`, and reads the repo `.env` for the API key.