A single vision model has been wrong in both directions on this project - it missed real defects and
disputed frames that were correct - so agreement between different model families is worth more than one
confident opinion.
judge_screenshot.py now runs two models by default (openai/gpt-4o-mini + qwen/qwen3.7-flash, both served
on OpenRouter; claude-3.5-haiku is gone, which the first attempt discovered with a 404). combine()
reconciles them: agreement carries the verdict with the mean confidence, disagreement reports 'uncertain'
and keeps both reasons rather than letting the more confident model win. --model still forces a single
judgement, and every line prints which model said what.
comfyui_generate.py generates the same prompt and seed with two checkpoints, scores both by
high-frequency detail (mean absolute Laplacian, which catches blurry or flat results), keeps the loser as
<out>__<model>.png with its sidecar, and moves the winner onto --out. --alt-model restores the single
run.
Tests cover the reconciliation - agreement, disagreement, unknown verdicts, and no verdicts at all - and the
suite is at 103.
--shots was parsing a table shape the README no longer has, so it silently found nothing.
It now reads Tools/visual_targets.json, the same file the render gate uses, and reports one
line per shot with the model's verdict and reasoning.
The result is not good news and is committed as such: 0 of 9 shots judged 'yes'. Shot 7 shows
the scene itself wrong - a disassembled car floating over a blank surface with no street or
buildings in frame - and the others fail for missing detail rather than for a broken frame.
visual_loop --verify still reports 9/9, because it only measures exposure and size.
A warning now sits at the top of Screenshots/README.md: treat every image there as stale until
the scene is fixed. The cause is not yet identified - the map builder's recent changes are 15
lines of vehicle staging, which does not explain a missing street.
- Tools/judge_screenshot.py: judges a screenshot against an expected-scene
description via OpenRouter multimodal image understanding, reading
OPENROUTER_API_KEY from the ignored .env. --image+--expect for one shot,
--shots to run every review shot in Screenshots/README.md. Returns a
{verdict, confidence, reason, missing} JSON.
- Documented in AGENTS.md (Visual review pre-check step) and Tools/README.md;
recorded the OpenRouter payload format gotcha in MEMORY.md.
zwitschi