feat(tools): every image judged by two models, every generated texture drawn by two checkpoints

A single vision model has been wrong in both directions on this project - it missed real defects and
disputed frames that were correct - so agreement between different model families is worth more than one
confident opinion.

judge_screenshot.py now runs two models by default (openai/gpt-4o-mini + qwen/qwen3.7-flash, both served
on OpenRouter; claude-3.5-haiku is gone, which the first attempt discovered with a 404). combine()
reconciles them: agreement carries the verdict with the mean confidence, disagreement reports 'uncertain'
and keeps both reasons rather than letting the more confident model win. --model still forces a single
judgement, and every line prints which model said what.

comfyui_generate.py generates the same prompt and seed with two checkpoints, scores both by
high-frequency detail (mean absolute Laplacian, which catches blurry or flat results), keeps the loser as
<out>__<model>.png with its sidecar, and moves the winner onto --out. --alt-model  restores the single
run.

Tests cover the reconciliation - agreement, disagreement, unknown verdicts, and no verdicts at all - and the
suite is at 103.
This commit is contained in:
zwitschi committed 2026-11-10 09:12:26 +01:00
1 parent bf7dfe1052
commit abde390045
4 files changed
+158 -10

No files matched your search

+2 -2
View File
@@ -69,7 +69,7 @@ Geographic tools live in [`Geo/tools/`](../Geo/README.md) and the Blender-side b
| Tool | Purpose |
| --- | --- |
| `render_screenshots.py` / `render_screenshots.sh` | Rebuilds the review map, runs the game, captures the shot list into `Screenshots/` |
| `judge_screenshot.py` | Vision-model pre-check: judges a screenshot against an expected-scene description (`--image` + `--expect`, or `--shots` for every shot in the README). Reads `OPENROUTER_API_KEY` from `.env` (ignored). Optional, advisory only — the model is unreliable on this content and never replaces a human verdict. |
| `judge_screenshot.py` | Vision-model pre-check: judges a screenshot against an expected-scene description (`--image` + `--expect`, or `--shots` for every shot in the README). Reads `OPENROUTER_API_KEY` from `.env` (ignored). Judged by **two models by default** (`--models a,b`), reconciled: agreement carries the verdict, disagreement reports `uncertain` with both reasons. Optional, advisory only — never replaces a human verdict. | `--image` + `--expect`; `--model` for a single judgement |
| `visual_loop.py` | The reference-vs-render loop. `--verify` gates each capture against its expectation in `visual_targets.json` (blank/short/wrong-size/tone-less shots fail); `--compare` measures luminance, contrast, edge density, palette and 3×3 composition against the reference photograph and turns the deltas into modelling actions naming the skill to consult (`blender-modeler`, `archviz`, `materials`, `lookdev`, `camera-cinematography`); `--iteration N` writes the artefacts under `Art/reference/iterations/N/`, which is **gitignored until the owner confirms a shot**. |
| `visual_targets.json` | Per-shot expectation and reference photograph for `visual_loop.py`. Every entry must be a demo-map shot. |
@@ -94,7 +94,7 @@ Geographic tools live in [`Geo/tools/`](../Geo/README.md) and the Blender-side b
| --- | --- |
| `input_emulation.py` | Pure-stdlib `ctypes`/`fcntl` driver that creates a virtual keyboard+mouse on `/dev/uinput`. The desktop treats it as real hardware, so it proves whether OS input actually reaches the game. Needs `/dev/uinput` writable (ACL on this host). |
| `input_test.py` | Injects a scripted sequence into the focused window: `--sequence move:2,look:3,jump:1,reset:1,pause:1,click`. |
| `comfyui_generate.py` | Local ComfyUI driver: list checkpoints, run txt2img through the HTTP API, and write a provenance sidecar (model, prompt, seed, full graph). Local, human-triggered. | `models`; `txt2img --prompt ... --out <png>` | `python3 Tools/comfyui_generate.py models` |
| `comfyui_generate.py` | Local ComfyUI driver: list checkpoints, run txt2img through the HTTP API, and write a provenance sidecar (model, prompt, seed, full graph). **Two checkpoints by default**: both are generated from the same prompt and seed, scored by high-frequency detail, and the better one takes `--out` while the loser is kept beside it. Local, human-triggered. | `models`; `txt2img --prompt ... --out <png> [--alt-model ...]` | `python3 Tools/comfyui_generate.py models` |
| `comfyui_character_blueprint.py` | Generates a character blueprint sheet - front, side, back and head views plus a contact sheet - from one description, with fixed seeds and a JSON record. | `--desc`, `--out`, `--model`, `--size` | `python3 Tools/comfyui_character_blueprint.py --desc "lean pedestrian, navy tee, jeans" --out Art/characters/blueprints` |
| `character_match.py` | Compares a rendered character against a blueprint: figure aspect, fill and coverage deltas, mean CIE76 colour distance, dominant-palette distance, and a side-by-side montage. | `--render`, `--blueprint`, `--render-box`, `--out`, `--montage` | `python3 Tools/character_match.py --render Screenshots/demo/review-demo-third-person-front.png --blueprint Art/characters/blueprints/front.png --render-box 0.42,0.46,0.56,0.97 --montage /tmp/gigline/match.png` |
| `generate_character_textures.py` | Procedural skin-pore and hair-strand textures with sidecars; no model involved. | `--out` | `python3 Tools/generate_character_textures.py --out Art/textures/generated` |