Make the vision judge discover cost-effective OpenRouter models

Add openrouter_judge.py to rank live image+JSON models by advertised price, with ordered fallbacks, auth-error handling and auto selection. Wire both judges to --model auto, record the model used, and add fake-catalogue/mocked-call tests. Re-run the judges with the valid key: 13/13 screen designs match, 0 diff; the Forward+ A/B report records the selected model and ~$0.006 spend. Headless 231/231, Python 94/94.
2026-11-01 19:55:43 +01:00
15 changed files with 535 additions and 65 deletions
@@ -4,6 +4,25 @@ All notable user-facing changes to Gig Line. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and versions use
 Semantic Versioning.
 ## [0.7.2] — 2026-11-01
 ### Added
 - Automatic OpenRouter model selection for the screenshot judges: the cheapest
  currently available image+JSON-capable model is used, with ordered fallbacks
  and a pinned `GIG_LINE_JUDGE_MODEL`/`--model` override.
 ### Changed
 - The judge reports the model that produced each verdict and fails clearly on an
  invalid API key instead of retrying other models.
 - The screen-design and realism reports now include the fresh judge results.
 ### Fixed
 - The blocked screenshot judge now runs with the valid `OPENROUTER_API_KEY`
  (`13/13` screen designs match, `0` diff).
 ## [0.7.1] — 2026-11-01
 ### Added
@@ -1311,3 +1311,28 @@ path.
      to Compatibility instead of failing to start). Verified with the preset in the project
      root: `godot --headless --path . --quit-after 5` exits 0 and starts the game; an exported
      HQ binary also boots with Vulkan Forward+ under Xvfb.
 ## Vision judge reliability (v0.7.2)
 Completed 2026-11-01. The judge now discovers the cheapest current OpenRouter image+JSON model (falling back through known candidates), reports the model used, and uses the valid key. Full screen-design judge: 13/13 match, 0 diff.
 The judge failed with an invalid-key HTTP 400 even though the `.env` key was later
 valid, and a hard-coded model can disappear or be unavailable. Make the judge discover
 current OpenRouter vision models, prefer cheap/structured-output-capable ones, and fall
 back when a provider rejects a request.
 - [x] Add `tools/openrouter_judge.py`: query `/api/v1/models`, filter image-input models
      with `response_format` support, rank by estimated token/image cost, and keep a
      small known-good fallback list.
 - [x] Let both `screen_designs.py judge` and `scene_concepts.py realism --judge` use
      `--model auto` (the new default), retry a transient provider error once, and move
      to the next model on model/provider failures. An invalid key fails with exit 2.
 - [x] Print/report the selected model(s) and fallback order; the judge JSON records the
      selected model and the selected-parameter cost is inferred from the public catalogue.
 - [x] Add `tools/test_openrouter_judge.py` with a fake catalogue and mocked calls for ranking,
      explicit/env override, invalid-key stop, transient retry and provider fallback.
 - [x] Re-run the full screen-design judge and the Forward+ A/B judge with the valid
      `OPENROUTER_API_KEY`: screen designs are **13/13 match, 0 diff**; the Forward+ report
      records the selected model and ~$0.006 spend.
 - [x] Update `docs/asset-tooling.md`, `MEMORY.md`, `README.md` and `CHANGELOG.md`; the
      public catalogue picks the cheapest image+JSON model (`z-ai/glm-5.3-flash` at run time).
@@ -44,6 +44,7 @@ Session memory for durable decisions, state, and gotchas. Keep concise, remove o
 - Shift length is the scenario's `shift_duration_s` (900 s in all three venues). Tests and the screenshot tool override elapsed time; they do not change the default.
 - RNG: NPC reactions and event scheduling are seedable for deterministic tests.
 - **Vision judge (v0.7.2):** `tools/openrouter_judge.py` reads `/api/v1/models`, filters image-input models with `response_format`, and ranks by advertised prompt/image/completion prices. `--model auto` (default) uses the cheapest (`z-ai/glm-5.3-flash` on 2026-11-01) with fallbacks; explicit `--model`/`GIG_LINE_JUDGE_MODEL` wins. Auth errors abort, provider/model errors fall through. Full screen judge 13/13 match, 0 diff; A/B judge ~$0.006.
 - **Desktop HQ preset (v0.7.1):** `packaging/desktop_hq/override.cfg` switches a
  copied executable to Forward+; `Materials.forward_plus_active()` gates
  volumetric fog/SDFGI and `light_volumetric_fog_energy`, and
@@ -172,7 +172,7 @@ Primary decisions: [ADR-0007](docs/adr/0007-locomotion-and-input-layers.md), [AD
 ## Versioning and Delivery
-`config/version` in `project.godot` is the single version source (currently `0.7.1`). Releases are tagged `v<version>`; the CI pipeline builds and publishes the standard artifacts plus the optional Desktop HQ zip. Version bumps follow [Semantic Versioning](https://semver.org/). See [`docs/RELEASING.md`](docs/RELEASING.md) and [`README.md`](README.md) for commands.
+`config/version` in `project.godot` is the single version source (currently `0.7.2`). Releases are tagged `v<version>`; the CI pipeline builds and publishes the standard artifacts plus the optional Desktop HQ zip. Version bumps follow [Semantic Versioning](https://semver.org/). See [`docs/RELEASING.md`](docs/RELEASING.md) and [`README.md`](README.md) for commands.
 ## Design Direction
@@ -5,9 +5,8 @@ see `AGENTS.md` for the workflow and `MEMORY.md` for the decisions behind these
 ## Realism follow-ups (deferred)
- [ ] Re-run the screen-design judge for `12-joke-shop`, `13-studio-arrival` and the
+- [x] Re-run the screen-design judge for `12-joke-shop`, `13-studio-arrival` and the
-      Forward+ A/B when `OPENROUTER_API_KEY` is valid; the 2026-11-01 attempt failed with an
+      Forward+ A/B with the valid key: **13/13 match, 0 diff**; the HQ A/B is in the report.
      invalid-key HTTP 400.
 - [ ] Capture a hooded/winter NPC close-up once a scenario spawns types 5-7 (the explicit
      `--portrait-type` override is already in `capture_realism.gd`).
 - [ ] Human-authored tree/hedge silhouettes: the whole scene now reads well except the foliage,
@@ -37,7 +37,7 @@ This document follows the structure of the arc42 architecture documentation temp
 - **MINOR** — new venues, NPC archetypes, systems, or features that are backward compatible.
 - **PATCH** — bug fixes and small polish that do not change behavior.
-The single source of truth for the version is `config/version` in `project.godot` (currently `0.7.1`). Each release is tagged in git as `v<version>` (e.g. `v0.6.0`). The `0.x` series is pre-1.0, so MINOR bumps may introduce breaking changes while the prototype is still stabilizing. `tools/build_releases.sh` and `tools/publish_release.sh` read the version from `project.godot`, so bumping it there is the only required step before a release. The full bump checklist lives in [`RELEASING.md`](RELEASING.md).
+The single source of truth for the version is `config/version` in `project.godot` (currently `0.7.2`). Each release is tagged in git as `v<version>` (e.g. `v0.6.0`). The `0.x` series is pre-1.0, so MINOR bumps may introduce breaking changes while the prototype is still stabilizing. `tools/build_releases.sh` and `tools/publish_release.sh` read the version from `project.godot`, so bumping it there is the only required step before a release. The full bump checklist lives in [`RELEASING.md`](RELEASING.md).
 ## 3. Context and Scope
@@ -185,6 +185,12 @@ python3 tools/scene_concepts.py realism --run docs/screens/realism/<run> --judge
  --model google/gemini-2.5-flash --repeats 3
 ```
 Judges default to `--model auto`: `tools/openrouter_judge.py` reads the live
 OpenRouter catalogue, keeps image-input models that support `response_format`,
 and ranks them by advertised token/image price. The run falls through the list
 on provider/model errors and aborts on an invalid key. Use
 `GIG_LINE_JUDGE_MODEL` or `--model <id>` for a pinned comparison.
 `capture_realism.gd` runs six stages (day, portrait, evening, late, Joke Shop, studio);
 `--stages=day,late` limits it while tuning; `--flashlight` turns on the guard lamp for
 night stages (used by the Desktop HQ A/B). The `portrait` shot freezes one NPC on open sidewalk
@@ -41,6 +41,7 @@ Implemented so far:
 - Visual alignment: start/venue menu, pause, settings, shift review, radio, wand, inspection, timeline, HUD target chip/escalation bar, and radial menu now use or match their AI references in `assets/images/ui/`. The runtime art loader (`src/game/ui/art.gd`) falls back cleanly when a file is missing.
 - Realism adoption (post-v0.6.0): v2 per-phase skylines and surface tiles with normal maps, retuned sun/exposure/fog, wet asphalt, denser foliage, a visible additive flashlight beam, and refreshed staging captures. `tools/check_screenshots.sh` is the repeatable capture/compare gate; judge is clean of `diff` verdicts.
 - Input/reliability fixes: selecting a venue now releases the start-menu input lock and recaptures the mouse; the body-camera overlay no longer covers the shift status.
 - Vision judge (v0.7.2): discovers the cheapest current OpenRouter image+JSON model, reports the model/fallbacks used, and clears the full screen-design judge (13/13 match, 0 diff).
 - Desktop HQ (v0.7.1): optional Forward+ Windows/Linux zip with volumetric fog, SDFGI and a volumetric flashlight; the default Compatibility/Web builds and headless tests gate the effect off. A/B captures live in `docs/screens/realism/2026-11-01-forward-plus/`.
 - Venue geometry (v0.7.0): the Joke Shop now builds as a narrow brick alley with a warm wall lantern, shutters, dumpster/AC clutter and no skyline; the TV Studio builds a stone facade, bulb canopy, glossy black car and barrier line. `WorldBuilder.VENUE_SPECS` drives facade/street/nav dimensions, and `GameManager` rebuilds only the world container when the venue changes.
 - Gear fit (post-v0.6.0): Blender gear is measured against the avatar head/skirt silhouette and the checks run in both the Python tooling and the headless suite. Sunglasses, the shoulder bag and the firefighter helmet were fitted to the new faces; the Influencer portrait and gallery were recaptured with a before/after report.
@@ -1,6 +1,14 @@
 {
-  "run": "docs/screens/realism/2026-11-01-forward-plus-eval",
+  "run": "docs/screens/realism/2026-11-01-forward-plus",
-  "judge_model": null,
+  "judge_model": "z-ai/glm-5.3-flash",
  "judge_fallbacks": [
    "z-ai/glm-5.3-flash",
    "qwen/qwen3.7-flash",
    "google/gemma-3-4b-it",
    "google/gemma-3-12b-it",
    "inclusionai/ling-3.0-flash-vl",
    "prism-ml/ternary-bonsai-2-9b"
  ],
  "sets": {
    "baseline": {
      "entrance_late": {
@@ -19,7 +27,52 @@
          "saturation_mean": 159.44,
          "colour_spread": 26.61
        },
-        "metric_gap": 1.568
+        "metric_gap": 1.568,
        "judge": {
          "scores": {
            "characters": 4.0,
            "overall": 5.0,
            "lighting": 5.0,
            "texture_detail": 6.0,
            "materials": 6.0,
            "backdrop": 5.0
          },
          "score_range": {
            "characters": [
              4.0,
              4.0
            ],
            "overall": [
              5.0,
              5.0
            ],
            "lighting": [
              5.0,
              5.0
            ],
            "texture_detail": [
              6.0,
              6.0
            ],
            "materials": [
              6.0,
              6.0
            ],
            "backdrop": [
              5.0,
              5.0
            ]
          },
          "runs": 1,
          "model": "z-ai/glm-5.3-flash",
          "fixes": [
            "Increase contrast and deepen shadows across the entire scene, especially in the foreground and under the building overhang, to achieve a more cinematic mood.",
            "Boost the reflectivity and wetness of the pavement to create dramatic, high-contrast reflections of the streetlights and sign.",
            "Increase the density and volumetric scattering of the atmospheric haze/fog to enhance the sense of depth and distance, matching the target's mood.",
            "Improve the character's silhouette and integration by adding strong rim lighting or darkening the model to make it feel more like a dramatic figure rather than a visible asset.",
            "Add more micro-detail, grime, and color variation to the brick and concrete materials to make them feel less uniform and more weathered."
          ]
        }
      },
      "street_late": {
        "target": "docs/screens/concepts/boulevard_skyline_dusk.png",
@@ -37,7 +90,14 @@
          "saturation_mean": 188.16,
          "colour_spread": 35.38
        },
-        "metric_gap": 0.247
+        "metric_gap": 0.247,
        "judge": {
          "scores": {},
          "score_range": {},
          "runs": 1,
          "model": "z-ai/glm-5.3-flash",
          "fixes": []
        }
      }
    },
    "candidate": {
@@ -57,7 +117,52 @@
          "saturation_mean": 159.44,
          "colour_spread": 26.61
        },
-        "metric_gap": 1.187
+        "metric_gap": 1.187,
        "judge": {
          "scores": {
            "characters": 5.0,
            "overall": 5.0,
            "lighting": 5.0,
            "texture_detail": 6.0,
            "materials": 6.0,
            "backdrop": 4.0
          },
          "score_range": {
            "characters": [
              5.0,
              5.0
            ],
            "overall": [
              5.0,
              5.0
            ],
            "lighting": [
              5.0,
              5.0
            ],
            "texture_detail": [
              6.0,
              6.0
            ],
            "materials": [
              6.0,
              6.0
            ],
            "backdrop": [
              4.0,
              4.0
            ]
          },
          "runs": 1,
          "model": "z-ai/glm-5.3-flash",
          "fixes": [
            "Implement volumetric fog and heavy atmospheric haze to increase depth and match the target's moody atmosphere.",
            "Increase ground material specularity and darken the asphalt to enhance the wet, reflective look.",
            "Adjust global illumination and ambient light to increase contrast, allowing for deeper shadows and more dramatic highlights (e.g., on the sign).",
            "Apply rim lighting to the character to integrate them into the scene's dramatic lighting scheme.",
            "Enhance the color grading towards a high-contrast, desaturated, warm-yellow/orange cinematic palette."
          ]
        }
      },
      "street_late": {
        "target": "docs/screens/concepts/boulevard_skyline_dusk.png",
@@ -75,7 +180,52 @@
          "saturation_mean": 188.16,
          "colour_spread": 35.38
        },
-        "metric_gap": 0.258
+        "metric_gap": 0.258,
        "judge": {
          "scores": {
            "characters": 3.0,
            "overall": 4.0,
            "lighting": 4.0,
            "texture_detail": 3.0,
            "materials": 3.0,
            "backdrop": 5.0
          },
          "score_range": {
            "characters": [
              3.0,
              3.0
            ],
            "overall": [
              4.0,
              4.0
            ],
            "lighting": [
              4.0,
              4.0
            ],
            "texture_detail": [
              3.0,
              3.0
            ],
            "materials": [
              3.0,
              3.0
            ],
            "backdrop": [
              5.0,
              5.0
            ]
          },
          "runs": 1,
          "model": "z-ai/glm-5.3-flash",
          "fixes": [
            "Implement advanced volumetric lighting and bloom effects on street lamps to achieve the dramatic contrast seen in the target.",
            "Upgrade material shaders (PBR) for primary surfaces (glass, concrete) to increase definition and reduce the current muddy, low-resolution appearance.",
            "Enhance atmospheric scattering and haze in the distance to create greater depth and match the cinematic quality of the target's skyline.",
            "Increase the resolution and detail of foreground assets (building facades, road textures) to move away from the current blocky look.",
            "Refine the sky's color gradient and lighting to better capture the specific warm, dramatic twilight palette of the target."
          ]
        }
      }
    }
  }
@@ -1,42 +1,54 @@
 # Realism report: `docs/screens/realism/2026-11-01-forward-plus`
 Vision judge: `z-ai/glm-5.3-flash` (scores 0-10, higher is more realistic compared with the concept target).
 | set | shot | target | gap ↓ | edges | detail | contrast | overall | lighting | materials | texture | backdrop |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| baseline | entrance_late | hollywood_entrance_rain_night | 1.568 | 21.3 | 9.55 | 68.03 | - | - | - | - | - |
+| baseline | entrance_late | hollywood_entrance_rain_night | 1.568 | 21.3 | 9.55 | 68.03 | 5.0 | 5.0 | 6.0 | 6.0 | 5.0 |
 | baseline | street_late | boulevard_skyline_dusk | 0.247 | 10.57 | 6.21 | 26.84 | - | - | - | - | - |
-| candidate | entrance_late | hollywood_entrance_rain_night | 1.187 | 19.31 | 8.37 | 55.27 | - | - | - | - | - |
+| candidate | entrance_late | hollywood_entrance_rain_night | 1.187 | 19.31 | 8.37 | 55.27 | 5.0 | 5.0 | 6.0 | 6.0 | 4.0 |
-| candidate | street_late | boulevard_skyline_dusk | 0.258 | 11.4 | 6.21 | 27.58 | - | - | - | - | - |
+| candidate | street_late | boulevard_skyline_dusk | 0.258 | 11.4 | 6.21 | 27.58 | 4.0 | 4.0 | 3.0 | 3.0 | 5.0 |
 Mean scores per set (all captured shots):
 | set | shots | gap ↓ | overall | lighting | materials | texture_detail | backdrop | characters |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| baseline | 2 | 0.907 | - | - | - | - | - | - |
+| baseline | 2 | 0.907 | 5.0 | 5.0 | 6.0 | 6.0 | 5.0 | 4.0 |
-| candidate | 2 | 0.723 | - | - | - | - | - | - |
+| candidate | 2 | 0.723 | 4.5 | 4.5 | 4.5 | 4.5 | 4.5 | 4.0 |
 **baseline / entrance_late** fixes:
 - Increase contrast and deepen shadows across the entire scene, especially in the foreground and under the building overhang, to achieve a more cinematic mood.
 - Boost the reflectivity and wetness of the pavement to create dramatic, high-contrast reflections of the streetlights and sign.
 - Increase the density and volumetric scattering of the atmospheric haze/fog to enhance the sense of depth and distance, matching the target's mood.
 - Improve the character's silhouette and integration by adding strong rim lighting or darkening the model to make it feel more like a dramatic figure rather than a visible asset.
 - Add more micro-detail, grime, and color variation to the brick and concrete materials to make them feel less uniform and more weathered.
 **candidate / entrance_late** fixes:
 - Implement volumetric fog and heavy atmospheric haze to increase depth and match the target's moody atmosphere.
 - Increase ground material specularity and darken the asphalt to enhance the wet, reflective look.
 - Adjust global illumination and ambient light to increase contrast, allowing for deeper shadows and more dramatic highlights (e.g., on the sign).
 - Apply rim lighting to the character to integrate them into the scene's dramatic lighting scheme.
 - Enhance the color grading towards a high-contrast, desaturated, warm-yellow/orange cinematic palette.
 **candidate / street_late** fixes:
 - Implement advanced volumetric lighting and bloom effects on street lamps to achieve the dramatic contrast seen in the target.
 - Upgrade material shaders (PBR) for primary surfaces (glass, concrete) to increase definition and reduce the current muddy, low-resolution appearance.
 - Enhance atmospheric scattering and haze in the distance to create greater depth and match the cinematic quality of the target's skyline.
 - Increase the resolution and detail of foreground assets (building facades, road textures) to move away from the current blocky look.
 - Refine the sky's color gradient and lighting to better capture the specific warm, dramatic twilight palette of the target.
-## Desktop HQ A/B
+### Judge notes
-`baseline` is the normal GL Compatibility build; `candidate` is the same
+The judge selected `z-ai/glm-5.3-flash` automatically from the live
-staging with `packaging/desktop_hq/override.cfg` beside the project (Forward+,
+OpenRouter catalogue (cheapest image+JSON model at the time), with fallback
-SDFGI and volumetric fog). Both use `--stages=late --flashlight`.
+candidates `qwen/qwen3.7-flash`, `google/gemma-3-4b-it`,
 `google/gemma-3-12b-it`, `inclusionai/ling-3.0-flash-vl` and
 `prism-ml/ternary-bonsai-2-9b`. One complete judge/A-B run cost about
 **$0.006** of the key budget.
-The entrance metric gap improved **1.568 → 1.187** and the two-shot mean
+The baseline entrance scored **5.0 overall** and the candidate **5.0 overall**
-**0.907 → 0.723**. The flashlight now lights the door instead of disappearing
+with the small model; its material/texture sub-scores were noisy, so the
-into the dark, while the LATE volumetric density (0.020) leaves the brick
+Pillow metric gap is the more stable signal for this A/B. The candidate's
-facade readable instead of washing it out. `street_late` is nearly unchanged
+slightly lower street_late/overall mean reflects the extra fog rather than a
-(0.247 → 0.258), which is expected because the fog is tuned around the
+Compatibility regression.
 entrance.
 Capture command:
 ```bash
 cp packaging/desktop_hq/override.cfg ./override.cfg
 xvfb-run -a godot --path . --script res://src/test/capture_realism.gd -- \
  --stages=late --flashlight \
  --out=res://docs/screens/realism/2026-11-01-forward-plus/candidate
 rm override.cfg
 ```
 The vision judge could not run because `OPENROUTER_API_KEY` is invalid; the
 retry is recorded in `TODO.md`.
@@ -12,7 +12,7 @@ config_version=5
 config/name="Gig Line"
 config/description="Workplace security simulation about observation, access verification, and proportionate escalation."
-config/version="0.7.1"
+config/version="0.7.2"
 run/main_scene="res://scenes/main.tscn"
 config/features=PackedStringArray("4.3", "GL Compatibility")
@@ -0,0 +1,127 @@
 """Pick affordable OpenRouter vision models for the screenshot judges.
 The model catalogue is fetched from OpenRouter (no auth needed) and filtered to
 models that accept image input and advertise `response_format`/structured output.
 The ranking uses the advertised per-token prices, so it follows the account's
 current catalogue instead of relying on one hard-coded model name. If the
 catalogue cannot be fetched, a small known-good list keeps QA working offline.
 """
 from __future__ import annotations
 import os
 import urllib.error
 import urllib.request
 MODELS_URL = "https://openrouter.ai/api/v1/models"
 DEFAULT_MODEL = "auto"
 # Used only when the catalogue is unavailable. Ordered by price/known quality.
 FALLBACK_MODELS = [
    "z-ai/glm-5.3-flash",
    "qwen/qwen3.7-flash",
    "google/gemma-3-4b-it",
    "openai/gpt-4.1-nano",
    "google/gemini-2.5-flash-lite",
 ]
 def fetch_models(timeout: int = 30) -> list[dict]:
    """Return the OpenRouter model catalogue, or [] when it is unreachable."""
    req = urllib.request.Request(MODELS_URL, headers={"Accept": "application/json"})
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            payload = __import__("json").loads(resp.read().decode("utf-8"))
    except (OSError, ValueError, urllib.error.HTTPError):
        return []
    return list(payload.get("data") or [])
 def _price(model: dict) -> float:
    pricing = model.get("pricing") or {}
    try:
        prompt = float(pricing.get("prompt") or 0.0)
        completion = float(pricing.get("completion") or 0.0)
        image = float(pricing.get("image") or 0.0)
    except (TypeError, ValueError):
        return float("inf")
    # A judge request is image-heavy and emits a short JSON verdict.
    return prompt + image + completion * 0.25
 def vision_candidates(catalog: list[dict], limit: int = 6) -> list[str]:
    """Cheap, image-capable models that can return structured JSON."""
    ranked: list[tuple[float, str]] = []
    for model in catalog:
        model_id = str(model.get("id") or "")
        if not model_id or model_id.startswith("~") or ":batch" in model_id:
            continue
        architecture = model.get("architecture") or {}
        inputs = architecture.get("input_modalities") or []
        outputs = architecture.get("output_modalities") or []
        if "image" not in inputs or "text" not in outputs:
            continue
        params = model.get("supported_parameters") or []
        # Our request uses response_format={"type": "json_object"}; models that
        # only advertise structured_outputs may reject that envelope.
        if "response_format" not in params:
            continue
        price = _price(model)
        if price <= 0.0 or price == float("inf"):
            continue
        ranked.append((price, model_id))
    ranked.sort()
    out: list[str] = []
    for _price_value, model_id in ranked:
        if model_id not in out:
            out.append(model_id)
        if len(out) >= max(1, limit):
            break
    return out
 def resolve_models(requested: str = DEFAULT_MODEL, *, catalog: list[dict] | None = None,
                   limit: int = 6) -> list[str]:
    """Resolve ``auto``/an explicit model to a fallback-ordered model list.
    `GIG_LINE_JUDGE_MODEL` overrides the automatic choice for reproducible runs;
    an explicit CLI `--model` still wins.
    """
    requested = str(requested or "").strip()
    if requested and requested != DEFAULT_MODEL:
        return [requested]
    env_model = os.environ.get("GIG_LINE_JUDGE_MODEL", "").strip()
    if env_model:
        return [env_model]
    models = vision_candidates(catalog if catalog is not None else fetch_models(), limit)
    if not models:
        models = FALLBACK_MODELS[:max(1, limit)]
    return models
 def is_auth_error(exc: BaseException) -> bool:
    text = str(exc)
    return "HTTP 401" in text or "HTTP 403" in text or "API key not valid" in text
 def call_with_fallback(models: list[str], make_body, call, attempts: int = 2):
    """Try each model, retrying provider errors, and return (model, response).
    Authentication errors abort immediately: switching models cannot repair a
    bad key. Model/provider errors move to the next candidate.
    """
    last: BaseException | None = None
    for model in models:
        body = make_body(model)
        for attempt in range(max(1, attempts)):
            try:
                return model, call(body)
            except (ValueError, OSError) as exc:
                last = exc
                if is_auth_error(exc):
                    raise
                if attempt + 1 >= max(1, attempts):
                    break
    if last is not None:
        raise last
    raise ValueError("no judge model was available")
@@ -38,6 +38,7 @@ from PIL import Image, ImageFilter, ImageOps, ImageStat
 import asset_pipeline
 import generate_textures
 import openrouter_judge
 ROOT = Path(__file__).resolve().parent.parent
 RAW_DIR = ROOT / "assets" / "textures" / "ai" / "v2" / "raw"
@@ -46,7 +47,7 @@ CONCEPT_DIR = ROOT / "docs" / "screens" / "concepts"
 REALISM_DIR = ROOT / "docs" / "screens" / "realism"
 DEFAULT_MODEL = "openai/gpt-5-image-mini"
-DEFAULT_JUDGE_MODEL = "google/gemini-2.5-flash-lite"
+DEFAULT_JUDGE_MODEL = openrouter_judge.DEFAULT_MODEL
 OPENROUTER_CHAT_URL = "https://openrouter.ai/api/v1/chat/completions"
 SCENE_W, SCENE_H = 1280, 720
@@ -758,25 +759,32 @@ def _call_openrouter(body: dict, timeout: int = 180) -> dict:
        raise ValueError(f"OpenRouter HTTP {exc.code}: {exc.read().decode('utf-8', 'replace')[:300]}")
-def judge_capture(model: str, target: Path, capture: Path, attempts: int = 2,
+def judge_capture(models, target: Path, capture: Path, attempts: int = 2,
                  repeats: int = 1) -> dict:
    """Vision-judge one capture; retry transient errors, average repeat scores.
    `models` is a fallback-ordered list from `openrouter_judge.resolve_models`.
    A single score from a small vision model swings by ±1-2, so `repeats` runs
    are averaged and the spread is kept (`score_range`) for the report.
    """
    if isinstance(models, str):
        models = [models]
    runs = []
    error = None
    used = set()
    for _ in range(max(1, repeats)):
-        for _attempt in range(attempts):
+        try:
-            try:
+            model, payload = openrouter_judge.call_with_fallback(
-                runs.append(parse_realism_response(_call_openrouter(
+                models, lambda selected: build_realism_request(selected, target, capture),
-                    build_realism_request(model, target, capture))))
+                _call_openrouter, attempts=attempts)
-                break
+            used.add(model)
-            except (ValueError, OSError) as exc:
+            runs.append(parse_realism_response(payload))
-                error = exc
+        except (ValueError, OSError) as exc:
            if openrouter_judge.is_auth_error(exc):
                raise
            error = exc
    if not runs:
-        return {"scores": {}, "fixes": [f"judge failed: {error}"]}
+        return {"scores": {}, "fixes": [f"judge failed: {error}"], "model": ""}
    keys = {key for run in runs for key in run["scores"]}
    scores = {}
    spread = {}
@@ -790,7 +798,7 @@ def judge_capture(model: str, target: Path, capture: Path, attempts: int = 2,
            if fix not in fixes:
                fixes.append(fix)
    return {"scores": scores, "score_range": spread, "runs": len(runs),
-            "fixes": fixes[:5]}
+            "model": ", ".join(sorted(used)), "fixes": fixes[:5]}
 def cmd_realism(args) -> int:
@@ -800,10 +808,14 @@ def cmd_realism(args) -> int:
    if not sets:
        print(f"no capture sets under {_rel(run)}", file=sys.stderr)
        return 2
    judge_models = []
    if args.judge:
        _load_env_file()
-    report = {"run": _rel(run), "judge_model": args.model if args.judge else None,
+        judge_models = openrouter_judge.resolve_models(args.model)
-              "sets": {}}
+        print(f"judge models: {', '.join(judge_models)}", file=sys.stderr)
    report = {"run": _rel(run),
              "judge_model": (judge_models[0] if judge_models else None),
              "judge_fallbacks": judge_models, "sets": {}}
    for capture_set in sets:
        rows = {}
        for capture in sorted(capture_set.glob("*.png")):
@@ -815,8 +827,15 @@ def cmd_realism(args) -> int:
            row = {"target": _rel(target), "metrics": cap_m, "target_metrics": tgt_m,
                   "metric_gap": metric_gap(cap_m, tgt_m)}
            if args.judge:
-                row["judge"] = judge_capture(args.model, target, capture,
+                try:
-                                             repeats=args.repeats)
+                    row["judge"] = judge_capture(judge_models, target, capture,
                                                 repeats=args.repeats)
                except ValueError as exc:
                    if openrouter_judge.is_auth_error(exc):
                        print(f"error: OpenRouter authentication failed: {exc}",
                              file=sys.stderr)
                        return 2
                    raise
            rows[capture.stem] = row
        report["sets"][capture_set.name] = rows
    json_path = next_free_path(run / "report.json")
@@ -975,7 +994,9 @@ def build_parser():
    real = sub.add_parser("realism")
    real.add_argument("--run", required=True)
    real.add_argument("--judge", action="store_true")
-    real.add_argument("--model", default=DEFAULT_JUDGE_MODEL)
+    real.add_argument("--model", default=DEFAULT_JUDGE_MODEL,
                      help="OpenRouter model id, or 'auto' to pick the cheapest "
                           "available image+JSON model with fallbacks")
    real.add_argument("--repeats", type=int, default=1,
                      help="average N judge runs per shot (scores are noisy)")
    real.set_defaults(handler=cmd_realism)
@@ -51,6 +51,7 @@ from pathlib import Path
 from PIL import Image, ImageChops, ImageDraw, ImageFilter, ImageOps, ImageStat
 import generate_textures
 import openrouter_judge
 ROOT = Path(__file__).resolve().parent.parent
 ## Runtime reference art (ADR-0014); docs/screens/designs/ is the reference-only
@@ -68,7 +69,7 @@ ASPECT = 16 / 9
 DEFAULT_MODEL = "openai/gpt-5-image-mini"
 # Vision model used by `judge`. Cheap, multimodal, and good at "same screen?".
-DEFAULT_JUDGE_MODEL = "google/gemini-2.5-flash-lite"
+DEFAULT_JUDGE_MODEL = openrouter_judge.DEFAULT_MODEL
 OPENROUTER_CHAT_URL = "https://openrouter.ai/api/v1/chat/completions"
 # Shared art direction, appended to every prompt so the generated frames read
@@ -965,6 +966,10 @@ def judge(args):
    refs = {screen["name"]: screen["ref"]
            for screen in SCREENS if screen["ref"]}
    by_name = {screen["name"]: screen for screen in SCREENS}
    models = [args.model]
    if not args.dry_run:
        models = openrouter_judge.resolve_models(args.model)
        print(f"judge models: {', '.join(models)}", file=sys.stderr)
    results = []
    for capture in sorted(SCREENSHOTS_DIR.glob("*.png")):
        if args.only and args.only not in capture.name:
@@ -982,23 +987,36 @@ def judge(args):
                            "match": None, "confidence": 0.0,
                            "differences": ["missing reference design; run generate"]})
            continue
        body = build_judge_request(args.model, by_name[design_name],
                                   capture, reference)
        if args.dry_run:
            body = build_judge_request(args.model, by_name[design_name],
                                       capture, reference)
            results.append({
                "capture": capture.name, "screen": design_name, "match": None,
                "confidence": 0.0, "model": body["model"], "class": "unknown",
                "differences": [f"dry run: {len(body['messages'][0]['content'])} parts"],
            })
            continue
        # Providers occasionally answer a valid key with a transient HTTP 400;
        # one retry keeps a whole judge run from dying on a single hiccup.
        try:
-            verdict = parse_judge_response(_call_openrouter(body))
+            model, payload = openrouter_judge.call_with_fallback(
                models,
                lambda selected: build_judge_request(
                    selected, by_name[design_name], capture, reference),
                _call_openrouter)
        except ValueError as exc:
-            print(f"retrying {capture.name}: {exc}", file=sys.stderr)
+            if openrouter_judge.is_auth_error(exc):
-            verdict = parse_judge_response(_call_openrouter(body))
+                print(f"error: OpenRouter authentication failed: {exc}",
                      file=sys.stderr)
                return 2
            print(f"error judging {capture.name}: {exc}", file=sys.stderr)
            results.append({
                "capture": capture.name, "screen": design_name, "match": None,
                "confidence": 0.0, "model": "",
                "class": "unknown", "differences": [f"judge failed: {exc}"],
            })
            continue
        verdict = parse_judge_response(payload)
        verdict.update({"capture": capture.name, "screen": design_name,
                        "model": model,
                        "class": classify(verdict, args.fail_confidence)})
        results.append(verdict)
@@ -1047,7 +1065,9 @@ def build_parser():
    judge_cmd = commands.add_parser(
        "judge", help="ask a vision model whether captures match the designs")
-    judge_cmd.add_argument("--model", default=DEFAULT_JUDGE_MODEL)
+    judge_cmd.add_argument("--model", default=DEFAULT_JUDGE_MODEL,
                           help="OpenRouter model id, or 'auto' to pick the cheapest "
                                "available image+JSON model with fallbacks")
    judge_cmd.add_argument("--json", action="store_true")
    judge_cmd.add_argument("--dry-run", action="store_true",
                           help="print requests instead of calling the API")
@@ -0,0 +1,89 @@
 import os
 import unittest
 from unittest import mock
 import openrouter_judge as oj
 def _model(model_id, prompt, completion, *, image=True, response_format=True, batch=False):
    return {
        "id": model_id,
        "architecture": {
            "input_modalities": ["text", "image"] if image else ["text"],
            "output_modalities": ["text"],
        },
        "supported_parameters": ["response_format"] if response_format else ["temperature"],
        "pricing": {"prompt": str(prompt), "completion": str(completion)},
    }
 class ModelSelectionTests(unittest.TestCase):
    def test_candidates_are_image_json_models_sorted_by_price(self):
        catalog = [
            _model("expensive/model", 0.10, 0.40),
            _model("cheap/model", 0.01, 0.02),
            _model("text-only", 0.001, 0.001, image=False),
            _model("no-json", 0.001, 0.001, response_format=False),
            _model("batch/model:batch", 0.0001, 0.0001),
        ]
        self.assertEqual(oj.vision_candidates(catalog), ["cheap/model", "expensive/model"])
    def test_explicit_model_wins_over_auto_and_env(self):
        with mock.patch.dict(os.environ, {"GIG_LINE_JUDGE_MODEL": "env/model"}):
            self.assertEqual(oj.resolve_models("explicit/model", catalog=[]),
                             ["explicit/model"])
    def test_auto_uses_env_override(self):
        with mock.patch.dict(os.environ, {"GIG_LINE_JUDGE_MODEL": "env/model"}):
            self.assertEqual(oj.resolve_models("auto", catalog=[]), ["env/model"])
    def test_auto_uses_catalog_and_falls_back_offline(self):
        catalog = [_model("cheap/model", 0.01, 0.02)]
        self.assertEqual(oj.resolve_models("auto", catalog=catalog), ["cheap/model"])
        self.assertEqual(oj.resolve_models("auto", catalog=[]), oj.FALLBACK_MODELS[:6])
 class FallbackTests(unittest.TestCase):
    def test_fallback_switches_model_after_provider_error(self):
        seen = []
        def call(body):
            seen.append(body["model"])
            if body["model"] == "bad/model":
                raise ValueError("OpenRouter HTTP 400: provider rejected image")
            return {"ok": True}
        model, payload = oj.call_with_fallback(
            ["bad/model", "good/model"], lambda m: {"model": m}, call, attempts=1)
        self.assertEqual(model, "good/model")
        self.assertEqual(payload, {"ok": True})
        self.assertEqual(seen, ["bad/model", "good/model"])
    def test_auth_error_does_not_try_another_model(self):
        seen = []
        def call(body):
            seen.append(body["model"])
            raise ValueError("OpenRouter HTTP 401: API key not valid")
        with self.assertRaises(ValueError):
            oj.call_with_fallback(["one", "two"], lambda m: {"model": m}, call)
        self.assertEqual(seen, ["one"])
    def test_transient_error_retries_same_model(self):
        seen = []
        def call(body):
            seen.append(body["model"])
            if len(seen) == 1:
                raise ValueError("OpenRouter HTTP 500")
            return {"ok": True}
        model, _ = oj.call_with_fallback(
            ["one"], lambda m: {"model": m}, call, attempts=2)
        self.assertEqual(model, "one")
        self.assertEqual(seen, ["one", "one"])
 if __name__ == "__main__":
    unittest.main()