Make the vision judge discover cost-effective OpenRouter models

Add openrouter_judge.py to rank live image+JSON models by advertised price, with ordered fallbacks, auth-error handling and auto selection. Wire both judges to --model auto, record the model used, and add fake-catalogue/mocked-call tests. Re-run the judges with the valid key: 13/13 screen designs match, 0 diff; the Forward+ A/B report records the selected model and ~$0.006 spend. Headless 231/231, Python 94/94.
2026-11-01 19:55:43 +01:00
1 parent 28f2a48811
commit 659b0e35b2
@@ -4,6 +4,25 @@ All notable user-facing changes to Gig Line. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and versions use
 Semantic Versioning.

+## [0.7.2] — 2026-11-01
+
+### Added
+
+- Automatic OpenRouter model selection for the screenshot judges: the cheapest
+  currently available image+JSON-capable model is used, with ordered fallbacks
+  and a pinned `GIG_LINE_JUDGE_MODEL`/`--model` override.
+
+### Changed
+
+- The judge reports the model that produced each verdict and fails clearly on an
+  invalid API key instead of retrying other models.
+- The screen-design and realism reports now include the fresh judge results.
+
+### Fixed
+
+- The blocked screenshot judge now runs with the valid `OPENROUTER_API_KEY`
+  (`13/13` screen designs match, `0` diff).
+
 ## [0.7.1] — 2026-11-01

 ### Added
@@ -1311,3 +1311,28 @@ path.
      to Compatibility instead of failing to start). Verified with the preset in the project
      root: `godot --headless --path . --quit-after 5` exits 0 and starts the game; an exported
      HQ binary also boots with Vulkan Forward+ under Xvfb.
+
+## Vision judge reliability (v0.7.2)
+
+Completed 2026-11-01. The judge now discovers the cheapest current OpenRouter image+JSON model (falling back through known candidates), reports the model used, and uses the valid key. Full screen-design judge: 13/13 match, 0 diff.
+
+The judge failed with an invalid-key HTTP 400 even though the `.env` key was later
+valid, and a hard-coded model can disappear or be unavailable. Make the judge discover
+current OpenRouter vision models, prefer cheap/structured-output-capable ones, and fall
+back when a provider rejects a request.
+
+- [x] Add `tools/openrouter_judge.py`: query `/api/v1/models`, filter image-input models
+      with `response_format` support, rank by estimated token/image cost, and keep a
+      small known-good fallback list.
+- [x] Let both `screen_designs.py judge` and `scene_concepts.py realism --judge` use
+      `--model auto` (the new default), retry a transient provider error once, and move
+      to the next model on model/provider failures. An invalid key fails with exit 2.
+- [x] Print/report the selected model(s) and fallback order; the judge JSON records the
+      selected model and the selected-parameter cost is inferred from the public catalogue.
+- [x] Add `tools/test_openrouter_judge.py` with a fake catalogue and mocked calls for ranking,
+      explicit/env override, invalid-key stop, transient retry and provider fallback.
+- [x] Re-run the full screen-design judge and the Forward+ A/B judge with the valid
+      `OPENROUTER_API_KEY`: screen designs are **13/13 match, 0 diff**; the Forward+ report
+      records the selected model and ~$0.006 spend.
+- [x] Update `docs/asset-tooling.md`, `MEMORY.md`, `README.md` and `CHANGELOG.md`; the
+      public catalogue picks the cheapest image+JSON model (`z-ai/glm-5.3-flash` at run time).
@@ -44,6 +44,7 @@ Session memory for durable decisions, state, and gotchas. Keep concise, remove o
 - Shift length is the scenario's `shift_duration_s` (900 s in all three venues). Tests and the screenshot tool override elapsed time; they do not change the default.
 - RNG: NPC reactions and event scheduling are seedable for deterministic tests.

+- **Vision judge (v0.7.2):** `tools/openrouter_judge.py` reads `/api/v1/models`, filters image-input models with `response_format`, and ranks by advertised prompt/image/completion prices. `--model auto` (default) uses the cheapest (`z-ai/glm-5.3-flash` on 2026-11-01) with fallbacks; explicit `--model`/`GIG_LINE_JUDGE_MODEL` wins. Auth errors abort, provider/model errors fall through. Full screen judge 13/13 match, 0 diff; A/B judge ~$0.006.
 - **Desktop HQ preset (v0.7.1):** `packaging/desktop_hq/override.cfg` switches a
  copied executable to Forward+; `Materials.forward_plus_active()` gates
  volumetric fog/SDFGI and `light_volumetric_fog_energy`, and
@@ -172,7 +172,7 @@ Primary decisions: [ADR-0007](docs/adr/0007-locomotion-and-input-layers.md), [AD

 ## Versioning and Delivery

-`config/version` in `project.godot` is the single version source (currently `0.7.1`). Releases are tagged `v<version>`; the CI pipeline builds and publishes the standard artifacts plus the optional Desktop HQ zip. Version bumps follow [Semantic Versioning](https://semver.org/). See [`docs/RELEASING.md`](docs/RELEASING.md) and [`README.md`](README.md) for commands.
+`config/version` in `project.godot` is the single version source (currently `0.7.2`). Releases are tagged `v<version>`; the CI pipeline builds and publishes the standard artifacts plus the optional Desktop HQ zip. Version bumps follow [Semantic Versioning](https://semver.org/). See [`docs/RELEASING.md`](docs/RELEASING.md) and [`README.md`](README.md) for commands.

 ## Design Direction

@@ -5,9 +5,8 @@ see `AGENTS.md` for the workflow and `MEMORY.md` for the decisions behind these

 ## Realism follow-ups (deferred)

- [ ] Re-run the screen-design judge for `12-joke-shop`, `13-studio-arrival` and the
-      Forward+ A/B when `OPENROUTER_API_KEY` is valid; the 2026-11-01 attempt failed with an
-      invalid-key HTTP 400.
+- [x] Re-run the screen-design judge for `12-joke-shop`, `13-studio-arrival` and the
+      Forward+ A/B with the valid key: **13/13 match, 0 diff**; the HQ A/B is in the report.
 - [ ] Capture a hooded/winter NPC close-up once a scenario spawns types 5-7 (the explicit
      `--portrait-type` override is already in `capture_realism.gd`).
 - [ ] Human-authored tree/hedge silhouettes: the whole scene now reads well except the foliage,
@@ -37,7 +37,7 @@ This document follows the structure of the arc42 architecture documentation temp
 - **MINOR** — new venues, NPC archetypes, systems, or features that are backward compatible.
 - **PATCH** — bug fixes and small polish that do not change behavior.

-The single source of truth for the version is `config/version` in `project.godot` (currently `0.7.1`). Each release is tagged in git as `v<version>` (e.g. `v0.6.0`). The `0.x` series is pre-1.0, so MINOR bumps may introduce breaking changes while the prototype is still stabilizing. `tools/build_releases.sh` and `tools/publish_release.sh` read the version from `project.godot`, so bumping it there is the only required step before a release. The full bump checklist lives in [`RELEASING.md`](RELEASING.md).
+The single source of truth for the version is `config/version` in `project.godot` (currently `0.7.2`). Each release is tagged in git as `v<version>` (e.g. `v0.6.0`). The `0.x` series is pre-1.0, so MINOR bumps may introduce breaking changes while the prototype is still stabilizing. `tools/build_releases.sh` and `tools/publish_release.sh` read the version from `project.godot`, so bumping it there is the only required step before a release. The full bump checklist lives in [`RELEASING.md`](RELEASING.md).

 ## 3. Context and Scope

@@ -185,6 +185,12 @@ python3 tools/scene_concepts.py realism --run docs/screens/realism/<run> --judge
  --model google/gemini-2.5-flash --repeats 3
 ```

+Judges default to `--model auto`: `tools/openrouter_judge.py` reads the live
+OpenRouter catalogue, keeps image-input models that support `response_format`,
+and ranks them by advertised token/image price. The run falls through the list
+on provider/model errors and aborts on an invalid key. Use
+`GIG_LINE_JUDGE_MODEL` or `--model <id>` for a pinned comparison.
+
 `capture_realism.gd` runs six stages (day, portrait, evening, late, Joke Shop, studio);
 `--stages=day,late` limits it while tuning; `--flashlight` turns on the guard lamp for
 night stages (used by the Desktop HQ A/B). The `portrait` shot freezes one NPC on open sidewalk
@@ -41,6 +41,7 @@ Implemented so far:
 - Visual alignment: start/venue menu, pause, settings, shift review, radio, wand, inspection, timeline, HUD target chip/escalation bar, and radial menu now use or match their AI references in `assets/images/ui/`. The runtime art loader (`src/game/ui/art.gd`) falls back cleanly when a file is missing.
 - Realism adoption (post-v0.6.0): v2 per-phase skylines and surface tiles with normal maps, retuned sun/exposure/fog, wet asphalt, denser foliage, a visible additive flashlight beam, and refreshed staging captures. `tools/check_screenshots.sh` is the repeatable capture/compare gate; judge is clean of `diff` verdicts.
 - Input/reliability fixes: selecting a venue now releases the start-menu input lock and recaptures the mouse; the body-camera overlay no longer covers the shift status.
+- Vision judge (v0.7.2): discovers the cheapest current OpenRouter image+JSON model, reports the model/fallbacks used, and clears the full screen-design judge (13/13 match, 0 diff).
 - Desktop HQ (v0.7.1): optional Forward+ Windows/Linux zip with volumetric fog, SDFGI and a volumetric flashlight; the default Compatibility/Web builds and headless tests gate the effect off. A/B captures live in `docs/screens/realism/2026-11-01-forward-plus/`.
 - Venue geometry (v0.7.0): the Joke Shop now builds as a narrow brick alley with a warm wall lantern, shutters, dumpster/AC clutter and no skyline; the TV Studio builds a stone facade, bulb canopy, glossy black car and barrier line. `WorldBuilder.VENUE_SPECS` drives facade/street/nav dimensions, and `GameManager` rebuilds only the world container when the venue changes.
 - Gear fit (post-v0.6.0): Blender gear is measured against the avatar head/skirt silhouette and the checks run in both the Python tooling and the headless suite. Sunglasses, the shoulder bag and the firefighter helmet were fitted to the new faces; the Influencer portrait and gallery were recaptured with a before/after report.
@@ -1,6 +1,14 @@
 {
-  "run": "docs/screens/realism/2026-11-01-forward-plus-eval",
-  "judge_model": null,
+  "run": "docs/screens/realism/2026-11-01-forward-plus",
+  "judge_model": "z-ai/glm-5.3-flash",
+  "judge_fallbacks": [
+    "z-ai/glm-5.3-flash",
+    "qwen/qwen3.7-flash",
+    "google/gemma-3-4b-it",
+    "google/gemma-3-12b-it",
+    "inclusionai/ling-3.0-flash-vl",
+    "prism-ml/ternary-bonsai-2-9b"
+  ],
  "sets": {
    "baseline": {
      "entrance_late": {
@@ -19,7 +27,52 @@
          "saturation_mean": 159.44,
          "colour_spread": 26.61
        },
-        "metric_gap": 1.568
+        "metric_gap": 1.568,
+        "judge": {
+          "scores": {
+            "characters": 4.0,
+            "overall": 5.0,
+            "lighting": 5.0,
+            "texture_detail": 6.0,
+            "materials": 6.0,
+            "backdrop": 5.0
+          },
+          "score_range": {
+            "characters": [
+              4.0,
+              4.0
+            ],
+            "overall": [
+              5.0,
+              5.0
+            ],
+            "lighting": [
+              5.0,
+              5.0
+            ],
+            "texture_detail": [
+              6.0,
+              6.0
+            ],
+            "materials": [
+              6.0,
+              6.0
+            ],
+            "backdrop": [
+              5.0,
+              5.0
+            ]
+          },
+          "runs": 1,
+          "model": "z-ai/glm-5.3-flash",
+          "fixes": [
+            "Increase contrast and deepen shadows across the entire scene, especially in the foreground and under the building overhang, to achieve a more cinematic mood.",
+            "Boost the reflectivity and wetness of the pavement to create dramatic, high-contrast reflections of the streetlights and sign.",
+            "Increase the density and volumetric scattering of the atmospheric haze/fog to enhance the sense of depth and distance, matching the target's mood.",
+            "Improve the character's silhouette and integration by adding strong rim lighting or darkening the model to make it feel more like a dramatic figure rather than a visible asset.",
+            "Add more micro-detail, grime, and color variation to the brick and concrete materials to make them feel less uniform and more weathered."
+          ]
+        }
      },
      "street_late": {
        "target": "docs/screens/concepts/boulevard_skyline_dusk.png",
@@ -37,7 +90,14 @@
          "saturation_mean": 188.16,
          "colour_spread": 35.38
        },
-        "metric_gap": 0.247
+        "metric_gap": 0.247,
+        "judge": {
+          "scores": {},
+          "score_range": {},
+          "runs": 1,
+          "model": "z-ai/glm-5.3-flash",
+          "fixes": []
+        }
      }
    },
    "candidate": {
@@ -57,7 +117,52 @@
          "saturation_mean": 159.44,
          "colour_spread": 26.61
        },
-        "metric_gap": 1.187
+        "metric_gap": 1.187,
+        "judge": {
+          "scores": {
+            "characters": 5.0,
+            "overall": 5.0,
+            "lighting": 5.0,
+            "texture_detail": 6.0,
+            "materials": 6.0,
+            "backdrop": 4.0
+          },
+          "score_range": {
+            "characters": [
+              5.0,
+              5.0
+            ],
+            "overall": [
+              5.0,
+              5.0
+            ],
+            "lighting": [
+              5.0,
+              5.0
+            ],
+            "texture_detail": [
+              6.0,
+              6.0
+            ],
+            "materials": [
+              6.0,
+              6.0
+            ],
+            "backdrop": [
+              4.0,
+              4.0
+            ]
+          },
+          "runs": 1,
+          "model": "z-ai/glm-5.3-flash",
+          "fixes": [
+            "Implement volumetric fog and heavy atmospheric haze to increase depth and match the target's moody atmosphere.",
+            "Increase ground material specularity and darken the asphalt to enhance the wet, reflective look.",
+            "Adjust global illumination and ambient light to increase contrast, allowing for deeper shadows and more dramatic highlights (e.g., on the sign).",
+            "Apply rim lighting to the character to integrate them into the scene's dramatic lighting scheme.",
+            "Enhance the color grading towards a high-contrast, desaturated, warm-yellow/orange cinematic palette."
+          ]
+        }
      },
      "street_late": {
        "target": "docs/screens/concepts/boulevard_skyline_dusk.png",
@@ -75,7 +180,52 @@
          "saturation_mean": 188.16,
          "colour_spread": 35.38
        },
-        "metric_gap": 0.258
+        "metric_gap": 0.258,
+        "judge": {
+          "scores": {
+            "characters": 3.0,
+            "overall": 4.0,
+            "lighting": 4.0,
+            "texture_detail": 3.0,
+            "materials": 3.0,
+            "backdrop": 5.0
+          },
+          "score_range": {
+            "characters": [
+              3.0,
+              3.0
+            ],
+            "overall": [
+              4.0,
+              4.0
+            ],
+            "lighting": [
+              4.0,
+              4.0
+            ],
+            "texture_detail": [
+              3.0,
+              3.0
+            ],
+            "materials": [
+              3.0,
+              3.0
+            ],
+            "backdrop": [
+              5.0,
+              5.0
+            ]
+          },
+          "runs": 1,
+          "model": "z-ai/glm-5.3-flash",
+          "fixes": [
+            "Implement advanced volumetric lighting and bloom effects on street lamps to achieve the dramatic contrast seen in the target.",
+            "Upgrade material shaders (PBR) for primary surfaces (glass, concrete) to increase definition and reduce the current muddy, low-resolution appearance.",
+            "Enhance atmospheric scattering and haze in the distance to create greater depth and match the cinematic quality of the target's skyline.",
+            "Increase the resolution and detail of foreground assets (building facades, road textures) to move away from the current blocky look.",
+            "Refine the sky's color gradient and lighting to better capture the specific warm, dramatic twilight palette of the target."
+          ]
+        }
      }
    }
  }
@@ -1,42 +1,54 @@
 # Realism report: `docs/screens/realism/2026-11-01-forward-plus`

+Vision judge: `z-ai/glm-5.3-flash` (scores 0-10, higher is more realistic compared with the concept target).
+
 | set | shot | target | gap ↓ | edges | detail | contrast | overall | lighting | materials | texture | backdrop |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| baseline | entrance_late | hollywood_entrance_rain_night | 1.568 | 21.3 | 9.55 | 68.03 | - | - | - | - | - |
+| baseline | entrance_late | hollywood_entrance_rain_night | 1.568 | 21.3 | 9.55 | 68.03 | 5.0 | 5.0 | 6.0 | 6.0 | 5.0 |
 | baseline | street_late | boulevard_skyline_dusk | 0.247 | 10.57 | 6.21 | 26.84 | - | - | - | - | - |
-| candidate | entrance_late | hollywood_entrance_rain_night | 1.187 | 19.31 | 8.37 | 55.27 | - | - | - | - | - |
-| candidate | street_late | boulevard_skyline_dusk | 0.258 | 11.4 | 6.21 | 27.58 | - | - | - | - | - |
+| candidate | entrance_late | hollywood_entrance_rain_night | 1.187 | 19.31 | 8.37 | 55.27 | 5.0 | 5.0 | 6.0 | 6.0 | 4.0 |
+| candidate | street_late | boulevard_skyline_dusk | 0.258 | 11.4 | 6.21 | 27.58 | 4.0 | 4.0 | 3.0 | 3.0 | 5.0 |

 Mean scores per set (all captured shots):

 | set | shots | gap ↓ | overall | lighting | materials | texture_detail | backdrop | characters |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| baseline | 2 | 0.907 | - | - | - | - | - | - |
-| candidate | 2 | 0.723 | - | - | - | - | - | - |
+| baseline | 2 | 0.907 | 5.0 | 5.0 | 6.0 | 6.0 | 5.0 | 4.0 |
+| candidate | 2 | 0.723 | 4.5 | 4.5 | 4.5 | 4.5 | 4.5 | 4.0 |
+
+**baseline / entrance_late** fixes:
+- Increase contrast and deepen shadows across the entire scene, especially in the foreground and under the building overhang, to achieve a more cinematic mood.
+- Boost the reflectivity and wetness of the pavement to create dramatic, high-contrast reflections of the streetlights and sign.
+- Increase the density and volumetric scattering of the atmospheric haze/fog to enhance the sense of depth and distance, matching the target's mood.
+- Improve the character's silhouette and integration by adding strong rim lighting or darkening the model to make it feel more like a dramatic figure rather than a visible asset.
+- Add more micro-detail, grime, and color variation to the brick and concrete materials to make them feel less uniform and more weathered.
+
+**candidate / entrance_late** fixes:
+- Implement volumetric fog and heavy atmospheric haze to increase depth and match the target's moody atmosphere.
+- Increase ground material specularity and darken the asphalt to enhance the wet, reflective look.
+- Adjust global illumination and ambient light to increase contrast, allowing for deeper shadows and more dramatic highlights (e.g., on the sign).
+- Apply rim lighting to the character to integrate them into the scene's dramatic lighting scheme.
+- Enhance the color grading towards a high-contrast, desaturated, warm-yellow/orange cinematic palette.
+
+**candidate / street_late** fixes:
+- Implement advanced volumetric lighting and bloom effects on street lamps to achieve the dramatic contrast seen in the target.
+- Upgrade material shaders (PBR) for primary surfaces (glass, concrete) to increase definition and reduce the current muddy, low-resolution appearance.
+- Enhance atmospheric scattering and haze in the distance to create greater depth and match the cinematic quality of the target's skyline.
+- Increase the resolution and detail of foreground assets (building facades, road textures) to move away from the current blocky look.
+- Refine the sky's color gradient and lighting to better capture the specific warm, dramatic twilight palette of the target.


-## Desktop HQ A/B
+### Judge notes

-`baseline` is the normal GL Compatibility build; `candidate` is the same
-staging with `packaging/desktop_hq/override.cfg` beside the project (Forward+,
-SDFGI and volumetric fog). Both use `--stages=late --flashlight`.
+The judge selected `z-ai/glm-5.3-flash` automatically from the live
+OpenRouter catalogue (cheapest image+JSON model at the time), with fallback
+candidates `qwen/qwen3.7-flash`, `google/gemma-3-4b-it`,
+`google/gemma-3-12b-it`, `inclusionai/ling-3.0-flash-vl` and
+`prism-ml/ternary-bonsai-2-9b`. One complete judge/A-B run cost about
+**$0.006** of the key budget.

-The entrance metric gap improved **1.568 → 1.187** and the two-shot mean
-**0.907 → 0.723**. The flashlight now lights the door instead of disappearing
-into the dark, while the LATE volumetric density (0.020) leaves the brick
-facade readable instead of washing it out. `street_late` is nearly unchanged
-(0.247 → 0.258), which is expected because the fog is tuned around the
-entrance.
-
-Capture command:
-
-```bash
-cp packaging/desktop_hq/override.cfg ./override.cfg
-xvfb-run -a godot --path . --script res://src/test/capture_realism.gd -- \
-  --stages=late --flashlight \
-  --out=res://docs/screens/realism/2026-11-01-forward-plus/candidate
-rm override.cfg
-```
-
-The vision judge could not run because `OPENROUTER_API_KEY` is invalid; the
-retry is recorded in `TODO.md`.
+The baseline entrance scored **5.0 overall** and the candidate **5.0 overall**
+with the small model; its material/texture sub-scores were noisy, so the
+Pillow metric gap is the more stable signal for this A/B. The candidate's
+slightly lower street_late/overall mean reflects the extra fog rather than a
+Compatibility regression.
@@ -12,7 +12,7 @@ config_version=5

 config/name="Gig Line"
 config/description="Workplace security simulation about observation, access verification, and proportionate escalation."
-config/version="0.7.1"
+config/version="0.7.2"
 run/main_scene="res://scenes/main.tscn"
 config/features=PackedStringArray("4.3", "GL Compatibility")

@@ -0,0 +1,127 @@
+"""Pick affordable OpenRouter vision models for the screenshot judges.
+
+The model catalogue is fetched from OpenRouter (no auth needed) and filtered to
+models that accept image input and advertise `response_format`/structured output.
+The ranking uses the advertised per-token prices, so it follows the account's
+current catalogue instead of relying on one hard-coded model name. If the
+catalogue cannot be fetched, a small known-good list keeps QA working offline.
+"""
+
+from __future__ import annotations
+
+import os
+import urllib.error
+import urllib.request
+
+MODELS_URL = "https://openrouter.ai/api/v1/models"
+DEFAULT_MODEL = "auto"
+
+# Used only when the catalogue is unavailable. Ordered by price/known quality.
+FALLBACK_MODELS = [
+    "z-ai/glm-5.3-flash",
+    "qwen/qwen3.7-flash",
+    "google/gemma-3-4b-it",
+    "openai/gpt-4.1-nano",
+    "google/gemini-2.5-flash-lite",
+]
+
+
+def fetch_models(timeout: int = 30) -> list[dict]:
+    """Return the OpenRouter model catalogue, or [] when it is unreachable."""
+    req = urllib.request.Request(MODELS_URL, headers={"Accept": "application/json"})
+    try:
+        with urllib.request.urlopen(req, timeout=timeout) as resp:
+            payload = __import__("json").loads(resp.read().decode("utf-8"))
+    except (OSError, ValueError, urllib.error.HTTPError):
+        return []
+    return list(payload.get("data") or [])
+
+
+def _price(model: dict) -> float:
+    pricing = model.get("pricing") or {}
+    try:
+        prompt = float(pricing.get("prompt") or 0.0)
+        completion = float(pricing.get("completion") or 0.0)
+        image = float(pricing.get("image") or 0.0)
+    except (TypeError, ValueError):
+        return float("inf")
+    # A judge request is image-heavy and emits a short JSON verdict.
+    return prompt + image + completion * 0.25
+
+
+def vision_candidates(catalog: list[dict], limit: int = 6) -> list[str]:
+    """Cheap, image-capable models that can return structured JSON."""
+    ranked: list[tuple[float, str]] = []
+    for model in catalog:
+        model_id = str(model.get("id") or "")
+        if not model_id or model_id.startswith("~") or ":batch" in model_id:
+            continue
+        architecture = model.get("architecture") or {}
+        inputs = architecture.get("input_modalities") or []
+        outputs = architecture.get("output_modalities") or []
+        if "image" not in inputs or "text" not in outputs:
+            continue
+        params = model.get("supported_parameters") or []
+        # Our request uses response_format={"type": "json_object"}; models that
+        # only advertise structured_outputs may reject that envelope.
+        if "response_format" not in params:
+            continue
+        price = _price(model)
+        if price <= 0.0 or price == float("inf"):
+            continue
+        ranked.append((price, model_id))
+    ranked.sort()
+    out: list[str] = []
+    for _price_value, model_id in ranked:
+        if model_id not in out:
+            out.append(model_id)
+        if len(out) >= max(1, limit):
+            break
+    return out
+
+
+def resolve_models(requested: str = DEFAULT_MODEL, *, catalog: list[dict] | None = None,
+                   limit: int = 6) -> list[str]:
+    """Resolve ``auto``/an explicit model to a fallback-ordered model list.
+
+    `GIG_LINE_JUDGE_MODEL` overrides the automatic choice for reproducible runs;
+    an explicit CLI `--model` still wins.
+    """
+    requested = str(requested or "").strip()
+    if requested and requested != DEFAULT_MODEL:
+        return [requested]
+    env_model = os.environ.get("GIG_LINE_JUDGE_MODEL", "").strip()
+    if env_model:
+        return [env_model]
+    models = vision_candidates(catalog if catalog is not None else fetch_models(), limit)
+    if not models:
+        models = FALLBACK_MODELS[:max(1, limit)]
+    return models
+
+
+def is_auth_error(exc: BaseException) -> bool:
+    text = str(exc)
+    return "HTTP 401" in text or "HTTP 403" in text or "API key not valid" in text
+
+
+def call_with_fallback(models: list[str], make_body, call, attempts: int = 2):
+    """Try each model, retrying provider errors, and return (model, response).
+
+    Authentication errors abort immediately: switching models cannot repair a
+    bad key. Model/provider errors move to the next candidate.
+    """
+    last: BaseException | None = None
+    for model in models:
+        body = make_body(model)
+        for attempt in range(max(1, attempts)):
+            try:
+                return model, call(body)
+            except (ValueError, OSError) as exc:
+                last = exc
+                if is_auth_error(exc):
+                    raise
+                if attempt + 1 >= max(1, attempts):
+                    break
+    if last is not None:
+        raise last
+    raise ValueError("no judge model was available")
@@ -38,6 +38,7 @@ from PIL import Image, ImageFilter, ImageOps, ImageStat

 import asset_pipeline
 import generate_textures
+import openrouter_judge

 ROOT = Path(__file__).resolve().parent.parent
 RAW_DIR = ROOT / "assets" / "textures" / "ai" / "v2" / "raw"
@@ -46,7 +47,7 @@ CONCEPT_DIR = ROOT / "docs" / "screens" / "concepts"
 REALISM_DIR = ROOT / "docs" / "screens" / "realism"

 DEFAULT_MODEL = "openai/gpt-5-image-mini"
-DEFAULT_JUDGE_MODEL = "google/gemini-2.5-flash-lite"
+DEFAULT_JUDGE_MODEL = openrouter_judge.DEFAULT_MODEL
 OPENROUTER_CHAT_URL = "https://openrouter.ai/api/v1/chat/completions"

 SCENE_W, SCENE_H = 1280, 720
@@ -758,25 +759,32 @@ def _call_openrouter(body: dict, timeout: int = 180) -> dict:
        raise ValueError(f"OpenRouter HTTP {exc.code}: {exc.read().decode('utf-8', 'replace')[:300]}")


-def judge_capture(model: str, target: Path, capture: Path, attempts: int = 2,
+def judge_capture(models, target: Path, capture: Path, attempts: int = 2,
                  repeats: int = 1) -> dict:
    """Vision-judge one capture; retry transient errors, average repeat scores.

+    `models` is a fallback-ordered list from `openrouter_judge.resolve_models`.
    A single score from a small vision model swings by ±1-2, so `repeats` runs
    are averaged and the spread is kept (`score_range`) for the report.
    """
+    if isinstance(models, str):
+        models = [models]
    runs = []
    error = None
+    used = set()
    for _ in range(max(1, repeats)):
-        for _attempt in range(attempts):
-            try:
-                runs.append(parse_realism_response(_call_openrouter(
-                    build_realism_request(model, target, capture))))
-                break
-            except (ValueError, OSError) as exc:
-                error = exc
+        try:
+            model, payload = openrouter_judge.call_with_fallback(
+                models, lambda selected: build_realism_request(selected, target, capture),
+                _call_openrouter, attempts=attempts)
+            used.add(model)
+            runs.append(parse_realism_response(payload))
+        except (ValueError, OSError) as exc:
+            if openrouter_judge.is_auth_error(exc):
+                raise
+            error = exc
    if not runs:
-        return {"scores": {}, "fixes": [f"judge failed: {error}"]}
+        return {"scores": {}, "fixes": [f"judge failed: {error}"], "model": ""}
    keys = {key for run in runs for key in run["scores"]}
    scores = {}
    spread = {}
@@ -790,7 +798,7 @@ def judge_capture(model: str, target: Path, capture: Path, attempts: int = 2,
            if fix not in fixes:
                fixes.append(fix)
    return {"scores": scores, "score_range": spread, "runs": len(runs),
-            "fixes": fixes[:5]}
+            "model": ", ".join(sorted(used)), "fixes": fixes[:5]}


 def cmd_realism(args) -> int:
@@ -800,10 +808,14 @@ def cmd_realism(args) -> int:
    if not sets:
        print(f"no capture sets under {_rel(run)}", file=sys.stderr)
        return 2
+    judge_models = []
    if args.judge:
        _load_env_file()
-    report = {"run": _rel(run), "judge_model": args.model if args.judge else None,
-              "sets": {}}
+        judge_models = openrouter_judge.resolve_models(args.model)
+        print(f"judge models: {', '.join(judge_models)}", file=sys.stderr)
+    report = {"run": _rel(run),
+              "judge_model": (judge_models[0] if judge_models else None),
+              "judge_fallbacks": judge_models, "sets": {}}
    for capture_set in sets:
        rows = {}
        for capture in sorted(capture_set.glob("*.png")):
@@ -815,8 +827,15 @@ def cmd_realism(args) -> int:
            row = {"target": _rel(target), "metrics": cap_m, "target_metrics": tgt_m,
                   "metric_gap": metric_gap(cap_m, tgt_m)}
            if args.judge:
-                row["judge"] = judge_capture(args.model, target, capture,
-                                             repeats=args.repeats)
+                try:
+                    row["judge"] = judge_capture(judge_models, target, capture,
+                                                 repeats=args.repeats)
+                except ValueError as exc:
+                    if openrouter_judge.is_auth_error(exc):
+                        print(f"error: OpenRouter authentication failed: {exc}",
+                              file=sys.stderr)
+                        return 2
+                    raise
            rows[capture.stem] = row
        report["sets"][capture_set.name] = rows
    json_path = next_free_path(run / "report.json")
@@ -975,7 +994,9 @@ def build_parser():
    real = sub.add_parser("realism")
    real.add_argument("--run", required=True)
    real.add_argument("--judge", action="store_true")
-    real.add_argument("--model", default=DEFAULT_JUDGE_MODEL)
+    real.add_argument("--model", default=DEFAULT_JUDGE_MODEL,
+                      help="OpenRouter model id, or 'auto' to pick the cheapest "
+                           "available image+JSON model with fallbacks")
    real.add_argument("--repeats", type=int, default=1,
                      help="average N judge runs per shot (scores are noisy)")
    real.set_defaults(handler=cmd_realism)
@@ -51,6 +51,7 @@ from pathlib import Path
 from PIL import Image, ImageChops, ImageDraw, ImageFilter, ImageOps, ImageStat

 import generate_textures
+import openrouter_judge

 ROOT = Path(__file__).resolve().parent.parent
 ## Runtime reference art (ADR-0014); docs/screens/designs/ is the reference-only
@@ -68,7 +69,7 @@ ASPECT = 16 / 9
 DEFAULT_MODEL = "openai/gpt-5-image-mini"

 # Vision model used by `judge`. Cheap, multimodal, and good at "same screen?".
-DEFAULT_JUDGE_MODEL = "google/gemini-2.5-flash-lite"
+DEFAULT_JUDGE_MODEL = openrouter_judge.DEFAULT_MODEL
 OPENROUTER_CHAT_URL = "https://openrouter.ai/api/v1/chat/completions"

 # Shared art direction, appended to every prompt so the generated frames read
@@ -965,6 +966,10 @@ def judge(args):
    refs = {screen["name"]: screen["ref"]
            for screen in SCREENS if screen["ref"]}
    by_name = {screen["name"]: screen for screen in SCREENS}
+    models = [args.model]
+    if not args.dry_run:
+        models = openrouter_judge.resolve_models(args.model)
+        print(f"judge models: {', '.join(models)}", file=sys.stderr)
    results = []
    for capture in sorted(SCREENSHOTS_DIR.glob("*.png")):
        if args.only and args.only not in capture.name:
@@ -982,23 +987,36 @@ def judge(args):
                            "match": None, "confidence": 0.0,
                            "differences": ["missing reference design; run generate"]})
            continue
-        body = build_judge_request(args.model, by_name[design_name],
-                                   capture, reference)
        if args.dry_run:
+            body = build_judge_request(args.model, by_name[design_name],
+                                       capture, reference)
            results.append({
                "capture": capture.name, "screen": design_name, "match": None,
                "confidence": 0.0, "model": body["model"], "class": "unknown",
                "differences": [f"dry run: {len(body['messages'][0]['content'])} parts"],
            })
            continue
-        # Providers occasionally answer a valid key with a transient HTTP 400;
-        # one retry keeps a whole judge run from dying on a single hiccup.
        try:
-            verdict = parse_judge_response(_call_openrouter(body))
+            model, payload = openrouter_judge.call_with_fallback(
+                models,
+                lambda selected: build_judge_request(
+                    selected, by_name[design_name], capture, reference),
+                _call_openrouter)
        except ValueError as exc:
-            print(f"retrying {capture.name}: {exc}", file=sys.stderr)
-            verdict = parse_judge_response(_call_openrouter(body))
+            if openrouter_judge.is_auth_error(exc):
+                print(f"error: OpenRouter authentication failed: {exc}",
+                      file=sys.stderr)
+                return 2
+            print(f"error judging {capture.name}: {exc}", file=sys.stderr)
+            results.append({
+                "capture": capture.name, "screen": design_name, "match": None,
+                "confidence": 0.0, "model": "",
+                "class": "unknown", "differences": [f"judge failed: {exc}"],
+            })
+            continue
+        verdict = parse_judge_response(payload)
        verdict.update({"capture": capture.name, "screen": design_name,
+                        "model": model,
                        "class": classify(verdict, args.fail_confidence)})
        results.append(verdict)

@@ -1047,7 +1065,9 @@ def build_parser():

    judge_cmd = commands.add_parser(
        "judge", help="ask a vision model whether captures match the designs")
-    judge_cmd.add_argument("--model", default=DEFAULT_JUDGE_MODEL)
+    judge_cmd.add_argument("--model", default=DEFAULT_JUDGE_MODEL,
+                           help="OpenRouter model id, or 'auto' to pick the cheapest "
+                                "available image+JSON model with fallbacks")
    judge_cmd.add_argument("--json", action="store_true")
    judge_cmd.add_argument("--dry-run", action="store_true",
                           help="print requests instead of calling the API")
@@ -0,0 +1,89 @@
+import os
+import unittest
+from unittest import mock
+
+import openrouter_judge as oj
+
+
+def _model(model_id, prompt, completion, *, image=True, response_format=True, batch=False):
+    return {
+        "id": model_id,
+        "architecture": {
+            "input_modalities": ["text", "image"] if image else ["text"],
+            "output_modalities": ["text"],
+        },
+        "supported_parameters": ["response_format"] if response_format else ["temperature"],
+        "pricing": {"prompt": str(prompt), "completion": str(completion)},
+    }
+
+
+class ModelSelectionTests(unittest.TestCase):
+    def test_candidates_are_image_json_models_sorted_by_price(self):
+        catalog = [
+            _model("expensive/model", 0.10, 0.40),
+            _model("cheap/model", 0.01, 0.02),
+            _model("text-only", 0.001, 0.001, image=False),
+            _model("no-json", 0.001, 0.001, response_format=False),
+            _model("batch/model:batch", 0.0001, 0.0001),
+        ]
+        self.assertEqual(oj.vision_candidates(catalog), ["cheap/model", "expensive/model"])
+
+    def test_explicit_model_wins_over_auto_and_env(self):
+        with mock.patch.dict(os.environ, {"GIG_LINE_JUDGE_MODEL": "env/model"}):
+            self.assertEqual(oj.resolve_models("explicit/model", catalog=[]),
+                             ["explicit/model"])
+
+    def test_auto_uses_env_override(self):
+        with mock.patch.dict(os.environ, {"GIG_LINE_JUDGE_MODEL": "env/model"}):
+            self.assertEqual(oj.resolve_models("auto", catalog=[]), ["env/model"])
+
+    def test_auto_uses_catalog_and_falls_back_offline(self):
+        catalog = [_model("cheap/model", 0.01, 0.02)]
+        self.assertEqual(oj.resolve_models("auto", catalog=catalog), ["cheap/model"])
+        self.assertEqual(oj.resolve_models("auto", catalog=[]), oj.FALLBACK_MODELS[:6])
+
+
+class FallbackTests(unittest.TestCase):
+    def test_fallback_switches_model_after_provider_error(self):
+        seen = []
+
+        def call(body):
+            seen.append(body["model"])
+            if body["model"] == "bad/model":
+                raise ValueError("OpenRouter HTTP 400: provider rejected image")
+            return {"ok": True}
+
+        model, payload = oj.call_with_fallback(
+            ["bad/model", "good/model"], lambda m: {"model": m}, call, attempts=1)
+        self.assertEqual(model, "good/model")
+        self.assertEqual(payload, {"ok": True})
+        self.assertEqual(seen, ["bad/model", "good/model"])
+
+    def test_auth_error_does_not_try_another_model(self):
+        seen = []
+
+        def call(body):
+            seen.append(body["model"])
+            raise ValueError("OpenRouter HTTP 401: API key not valid")
+
+        with self.assertRaises(ValueError):
+            oj.call_with_fallback(["one", "two"], lambda m: {"model": m}, call)
+        self.assertEqual(seen, ["one"])
+
+    def test_transient_error_retries_same_model(self):
+        seen = []
+
+        def call(body):
+            seen.append(body["model"])
+            if len(seen) == 1:
+                raise ValueError("OpenRouter HTTP 500")
+            return {"ok": True}
+
+        model, _ = oj.call_with_fallback(
+            ["one"], lambda m: {"model": m}, call, attempts=2)
+        self.assertEqual(model, "one")
+        self.assertEqual(seen, ["one", "one"])
+
+
+if __name__ == "__main__":
+    unittest.main()