Make the vision judge discover cost-effective OpenRouter models
Add openrouter_judge.py to rank live image+JSON models by advertised price, with ordered fallbacks, auth-error handling and auto selection. Wire both judges to --model auto, record the model used, and add fake-catalogue/mocked-call tests. Re-run the judges with the valid key: 13/13 screen designs match, 0 diff; the Forward+ A/B report records the selected model and ~$0.006 spend. Headless 231/231, Python 94/94.
This commit is contained in:
zwitschi committed
1 parent
28f2a48811
commit
659b0e35b2
15 files changed
+535
-65
No files matched your search
@@ -37,7 +37,7 @@ This document follows the structure of the arc42 architecture documentation temp
|
||||
- **MINOR** — new venues, NPC archetypes, systems, or features that are backward compatible.
|
||||
- **PATCH** — bug fixes and small polish that do not change behavior.
|
||||
|
||||
The single source of truth for the version is `config/version` in `project.godot` (currently `0.7.1`). Each release is tagged in git as `v<version>` (e.g. `v0.6.0`). The `0.x` series is pre-1.0, so MINOR bumps may introduce breaking changes while the prototype is still stabilizing. `tools/build_releases.sh` and `tools/publish_release.sh` read the version from `project.godot`, so bumping it there is the only required step before a release. The full bump checklist lives in [`RELEASING.md`](RELEASING.md).
|
||||
The single source of truth for the version is `config/version` in `project.godot` (currently `0.7.2`). Each release is tagged in git as `v<version>` (e.g. `v0.6.0`). The `0.x` series is pre-1.0, so MINOR bumps may introduce breaking changes while the prototype is still stabilizing. `tools/build_releases.sh` and `tools/publish_release.sh` read the version from `project.godot`, so bumping it there is the only required step before a release. The full bump checklist lives in [`RELEASING.md`](RELEASING.md).
|
||||
|
||||
## 3. Context and Scope
|
||||
|
||||
|
||||
@@ -185,6 +185,12 @@ python3 tools/scene_concepts.py realism --run docs/screens/realism/<run> --judge
|
||||
--model google/gemini-2.5-flash --repeats 3
|
||||
```
|
||||
|
||||
Judges default to `--model auto`: `tools/openrouter_judge.py` reads the live
|
||||
OpenRouter catalogue, keeps image-input models that support `response_format`,
|
||||
and ranks them by advertised token/image price. The run falls through the list
|
||||
on provider/model errors and aborts on an invalid key. Use
|
||||
`GIG_LINE_JUDGE_MODEL` or `--model <id>` for a pinned comparison.
|
||||
|
||||
`capture_realism.gd` runs six stages (day, portrait, evening, late, Joke Shop, studio);
|
||||
`--stages=day,late` limits it while tuning; `--flashlight` turns on the guard lamp for
|
||||
night stages (used by the Desktop HQ A/B). The `portrait` shot freezes one NPC on open sidewalk
|
||||
|
||||
@@ -41,6 +41,7 @@ Implemented so far:
|
||||
- Visual alignment: start/venue menu, pause, settings, shift review, radio, wand, inspection, timeline, HUD target chip/escalation bar, and radial menu now use or match their AI references in `assets/images/ui/`. The runtime art loader (`src/game/ui/art.gd`) falls back cleanly when a file is missing.
|
||||
- Realism adoption (post-v0.6.0): v2 per-phase skylines and surface tiles with normal maps, retuned sun/exposure/fog, wet asphalt, denser foliage, a visible additive flashlight beam, and refreshed staging captures. `tools/check_screenshots.sh` is the repeatable capture/compare gate; judge is clean of `diff` verdicts.
|
||||
- Input/reliability fixes: selecting a venue now releases the start-menu input lock and recaptures the mouse; the body-camera overlay no longer covers the shift status.
|
||||
- Vision judge (v0.7.2): discovers the cheapest current OpenRouter image+JSON model, reports the model/fallbacks used, and clears the full screen-design judge (13/13 match, 0 diff).
|
||||
- Desktop HQ (v0.7.1): optional Forward+ Windows/Linux zip with volumetric fog, SDFGI and a volumetric flashlight; the default Compatibility/Web builds and headless tests gate the effect off. A/B captures live in `docs/screens/realism/2026-11-01-forward-plus/`.
|
||||
- Venue geometry (v0.7.0): the Joke Shop now builds as a narrow brick alley with a warm wall lantern, shutters, dumpster/AC clutter and no skyline; the TV Studio builds a stone facade, bulb canopy, glossy black car and barrier line. `WorldBuilder.VENUE_SPECS` drives facade/street/nav dimensions, and `GameManager` rebuilds only the world container when the venue changes.
|
||||
- Gear fit (post-v0.6.0): Blender gear is measured against the avatar head/skirt silhouette and the checks run in both the Python tooling and the headless suite. Sunglasses, the shoulder bag and the firefighter helmet were fitted to the new faces; the Influencer portrait and gallery were recaptured with a before/after report.
|
||||
|
||||
@@ -1,6 +1,14 @@
|
||||
{
|
||||
"run": "docs/screens/realism/2026-11-01-forward-plus-eval",
|
||||
"judge_model": null,
|
||||
"run": "docs/screens/realism/2026-11-01-forward-plus",
|
||||
"judge_model": "z-ai/glm-5.3-flash",
|
||||
"judge_fallbacks": [
|
||||
"z-ai/glm-5.3-flash",
|
||||
"qwen/qwen3.7-flash",
|
||||
"google/gemma-3-4b-it",
|
||||
"google/gemma-3-12b-it",
|
||||
"inclusionai/ling-3.0-flash-vl",
|
||||
"prism-ml/ternary-bonsai-2-9b"
|
||||
],
|
||||
"sets": {
|
||||
"baseline": {
|
||||
"entrance_late": {
|
||||
@@ -19,7 +27,52 @@
|
||||
"saturation_mean": 159.44,
|
||||
"colour_spread": 26.61
|
||||
},
|
||||
"metric_gap": 1.568
|
||||
"metric_gap": 1.568,
|
||||
"judge": {
|
||||
"scores": {
|
||||
"characters": 4.0,
|
||||
"overall": 5.0,
|
||||
"lighting": 5.0,
|
||||
"texture_detail": 6.0,
|
||||
"materials": 6.0,
|
||||
"backdrop": 5.0
|
||||
},
|
||||
"score_range": {
|
||||
"characters": [
|
||||
4.0,
|
||||
4.0
|
||||
],
|
||||
"overall": [
|
||||
5.0,
|
||||
5.0
|
||||
],
|
||||
"lighting": [
|
||||
5.0,
|
||||
5.0
|
||||
],
|
||||
"texture_detail": [
|
||||
6.0,
|
||||
6.0
|
||||
],
|
||||
"materials": [
|
||||
6.0,
|
||||
6.0
|
||||
],
|
||||
"backdrop": [
|
||||
5.0,
|
||||
5.0
|
||||
]
|
||||
},
|
||||
"runs": 1,
|
||||
"model": "z-ai/glm-5.3-flash",
|
||||
"fixes": [
|
||||
"Increase contrast and deepen shadows across the entire scene, especially in the foreground and under the building overhang, to achieve a more cinematic mood.",
|
||||
"Boost the reflectivity and wetness of the pavement to create dramatic, high-contrast reflections of the streetlights and sign.",
|
||||
"Increase the density and volumetric scattering of the atmospheric haze/fog to enhance the sense of depth and distance, matching the target's mood.",
|
||||
"Improve the character's silhouette and integration by adding strong rim lighting or darkening the model to make it feel more like a dramatic figure rather than a visible asset.",
|
||||
"Add more micro-detail, grime, and color variation to the brick and concrete materials to make them feel less uniform and more weathered."
|
||||
]
|
||||
}
|
||||
},
|
||||
"street_late": {
|
||||
"target": "docs/screens/concepts/boulevard_skyline_dusk.png",
|
||||
@@ -37,7 +90,14 @@
|
||||
"saturation_mean": 188.16,
|
||||
"colour_spread": 35.38
|
||||
},
|
||||
"metric_gap": 0.247
|
||||
"metric_gap": 0.247,
|
||||
"judge": {
|
||||
"scores": {},
|
||||
"score_range": {},
|
||||
"runs": 1,
|
||||
"model": "z-ai/glm-5.3-flash",
|
||||
"fixes": []
|
||||
}
|
||||
}
|
||||
},
|
||||
"candidate": {
|
||||
@@ -57,7 +117,52 @@
|
||||
"saturation_mean": 159.44,
|
||||
"colour_spread": 26.61
|
||||
},
|
||||
"metric_gap": 1.187
|
||||
"metric_gap": 1.187,
|
||||
"judge": {
|
||||
"scores": {
|
||||
"characters": 5.0,
|
||||
"overall": 5.0,
|
||||
"lighting": 5.0,
|
||||
"texture_detail": 6.0,
|
||||
"materials": 6.0,
|
||||
"backdrop": 4.0
|
||||
},
|
||||
"score_range": {
|
||||
"characters": [
|
||||
5.0,
|
||||
5.0
|
||||
],
|
||||
"overall": [
|
||||
5.0,
|
||||
5.0
|
||||
],
|
||||
"lighting": [
|
||||
5.0,
|
||||
5.0
|
||||
],
|
||||
"texture_detail": [
|
||||
6.0,
|
||||
6.0
|
||||
],
|
||||
"materials": [
|
||||
6.0,
|
||||
6.0
|
||||
],
|
||||
"backdrop": [
|
||||
4.0,
|
||||
4.0
|
||||
]
|
||||
},
|
||||
"runs": 1,
|
||||
"model": "z-ai/glm-5.3-flash",
|
||||
"fixes": [
|
||||
"Implement volumetric fog and heavy atmospheric haze to increase depth and match the target's moody atmosphere.",
|
||||
"Increase ground material specularity and darken the asphalt to enhance the wet, reflective look.",
|
||||
"Adjust global illumination and ambient light to increase contrast, allowing for deeper shadows and more dramatic highlights (e.g., on the sign).",
|
||||
"Apply rim lighting to the character to integrate them into the scene's dramatic lighting scheme.",
|
||||
"Enhance the color grading towards a high-contrast, desaturated, warm-yellow/orange cinematic palette."
|
||||
]
|
||||
}
|
||||
},
|
||||
"street_late": {
|
||||
"target": "docs/screens/concepts/boulevard_skyline_dusk.png",
|
||||
@@ -75,7 +180,52 @@
|
||||
"saturation_mean": 188.16,
|
||||
"colour_spread": 35.38
|
||||
},
|
||||
"metric_gap": 0.258
|
||||
"metric_gap": 0.258,
|
||||
"judge": {
|
||||
"scores": {
|
||||
"characters": 3.0,
|
||||
"overall": 4.0,
|
||||
"lighting": 4.0,
|
||||
"texture_detail": 3.0,
|
||||
"materials": 3.0,
|
||||
"backdrop": 5.0
|
||||
},
|
||||
"score_range": {
|
||||
"characters": [
|
||||
3.0,
|
||||
3.0
|
||||
],
|
||||
"overall": [
|
||||
4.0,
|
||||
4.0
|
||||
],
|
||||
"lighting": [
|
||||
4.0,
|
||||
4.0
|
||||
],
|
||||
"texture_detail": [
|
||||
3.0,
|
||||
3.0
|
||||
],
|
||||
"materials": [
|
||||
3.0,
|
||||
3.0
|
||||
],
|
||||
"backdrop": [
|
||||
5.0,
|
||||
5.0
|
||||
]
|
||||
},
|
||||
"runs": 1,
|
||||
"model": "z-ai/glm-5.3-flash",
|
||||
"fixes": [
|
||||
"Implement advanced volumetric lighting and bloom effects on street lamps to achieve the dramatic contrast seen in the target.",
|
||||
"Upgrade material shaders (PBR) for primary surfaces (glass, concrete) to increase definition and reduce the current muddy, low-resolution appearance.",
|
||||
"Enhance atmospheric scattering and haze in the distance to create greater depth and match the cinematic quality of the target's skyline.",
|
||||
"Increase the resolution and detail of foreground assets (building facades, road textures) to move away from the current blocky look.",
|
||||
"Refine the sky's color gradient and lighting to better capture the specific warm, dramatic twilight palette of the target."
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,42 +1,54 @@
|
||||
# Realism report: `docs/screens/realism/2026-11-01-forward-plus`
|
||||
|
||||
Vision judge: `z-ai/glm-5.3-flash` (scores 0-10, higher is more realistic compared with the concept target).
|
||||
|
||||
| set | shot | target | gap ↓ | edges | detail | contrast | overall | lighting | materials | texture | backdrop |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| baseline | entrance_late | hollywood_entrance_rain_night | 1.568 | 21.3 | 9.55 | 68.03 | - | - | - | - | - |
|
||||
| baseline | entrance_late | hollywood_entrance_rain_night | 1.568 | 21.3 | 9.55 | 68.03 | 5.0 | 5.0 | 6.0 | 6.0 | 5.0 |
|
||||
| baseline | street_late | boulevard_skyline_dusk | 0.247 | 10.57 | 6.21 | 26.84 | - | - | - | - | - |
|
||||
| candidate | entrance_late | hollywood_entrance_rain_night | 1.187 | 19.31 | 8.37 | 55.27 | - | - | - | - | - |
|
||||
| candidate | street_late | boulevard_skyline_dusk | 0.258 | 11.4 | 6.21 | 27.58 | - | - | - | - | - |
|
||||
| candidate | entrance_late | hollywood_entrance_rain_night | 1.187 | 19.31 | 8.37 | 55.27 | 5.0 | 5.0 | 6.0 | 6.0 | 4.0 |
|
||||
| candidate | street_late | boulevard_skyline_dusk | 0.258 | 11.4 | 6.21 | 27.58 | 4.0 | 4.0 | 3.0 | 3.0 | 5.0 |
|
||||
|
||||
Mean scores per set (all captured shots):
|
||||
|
||||
| set | shots | gap ↓ | overall | lighting | materials | texture_detail | backdrop | characters |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| baseline | 2 | 0.907 | - | - | - | - | - | - |
|
||||
| candidate | 2 | 0.723 | - | - | - | - | - | - |
|
||||
| baseline | 2 | 0.907 | 5.0 | 5.0 | 6.0 | 6.0 | 5.0 | 4.0 |
|
||||
| candidate | 2 | 0.723 | 4.5 | 4.5 | 4.5 | 4.5 | 4.5 | 4.0 |
|
||||
|
||||
**baseline / entrance_late** fixes:
|
||||
- Increase contrast and deepen shadows across the entire scene, especially in the foreground and under the building overhang, to achieve a more cinematic mood.
|
||||
- Boost the reflectivity and wetness of the pavement to create dramatic, high-contrast reflections of the streetlights and sign.
|
||||
- Increase the density and volumetric scattering of the atmospheric haze/fog to enhance the sense of depth and distance, matching the target's mood.
|
||||
- Improve the character's silhouette and integration by adding strong rim lighting or darkening the model to make it feel more like a dramatic figure rather than a visible asset.
|
||||
- Add more micro-detail, grime, and color variation to the brick and concrete materials to make them feel less uniform and more weathered.
|
||||
|
||||
**candidate / entrance_late** fixes:
|
||||
- Implement volumetric fog and heavy atmospheric haze to increase depth and match the target's moody atmosphere.
|
||||
- Increase ground material specularity and darken the asphalt to enhance the wet, reflective look.
|
||||
- Adjust global illumination and ambient light to increase contrast, allowing for deeper shadows and more dramatic highlights (e.g., on the sign).
|
||||
- Apply rim lighting to the character to integrate them into the scene's dramatic lighting scheme.
|
||||
- Enhance the color grading towards a high-contrast, desaturated, warm-yellow/orange cinematic palette.
|
||||
|
||||
**candidate / street_late** fixes:
|
||||
- Implement advanced volumetric lighting and bloom effects on street lamps to achieve the dramatic contrast seen in the target.
|
||||
- Upgrade material shaders (PBR) for primary surfaces (glass, concrete) to increase definition and reduce the current muddy, low-resolution appearance.
|
||||
- Enhance atmospheric scattering and haze in the distance to create greater depth and match the cinematic quality of the target's skyline.
|
||||
- Increase the resolution and detail of foreground assets (building facades, road textures) to move away from the current blocky look.
|
||||
- Refine the sky's color gradient and lighting to better capture the specific warm, dramatic twilight palette of the target.
|
||||
|
||||
|
||||
## Desktop HQ A/B
|
||||
### Judge notes
|
||||
|
||||
`baseline` is the normal GL Compatibility build; `candidate` is the same
|
||||
staging with `packaging/desktop_hq/override.cfg` beside the project (Forward+,
|
||||
SDFGI and volumetric fog). Both use `--stages=late --flashlight`.
|
||||
The judge selected `z-ai/glm-5.3-flash` automatically from the live
|
||||
OpenRouter catalogue (cheapest image+JSON model at the time), with fallback
|
||||
candidates `qwen/qwen3.7-flash`, `google/gemma-3-4b-it`,
|
||||
`google/gemma-3-12b-it`, `inclusionai/ling-3.0-flash-vl` and
|
||||
`prism-ml/ternary-bonsai-2-9b`. One complete judge/A-B run cost about
|
||||
**$0.006** of the key budget.
|
||||
|
||||
The entrance metric gap improved **1.568 → 1.187** and the two-shot mean
|
||||
**0.907 → 0.723**. The flashlight now lights the door instead of disappearing
|
||||
into the dark, while the LATE volumetric density (0.020) leaves the brick
|
||||
facade readable instead of washing it out. `street_late` is nearly unchanged
|
||||
(0.247 → 0.258), which is expected because the fog is tuned around the
|
||||
entrance.
|
||||
|
||||
Capture command:
|
||||
|
||||
```bash
|
||||
cp packaging/desktop_hq/override.cfg ./override.cfg
|
||||
xvfb-run -a godot --path . --script res://src/test/capture_realism.gd -- \
|
||||
--stages=late --flashlight \
|
||||
--out=res://docs/screens/realism/2026-11-01-forward-plus/candidate
|
||||
rm override.cfg
|
||||
```
|
||||
|
||||
The vision judge could not run because `OPENROUTER_API_KEY` is invalid; the
|
||||
retry is recorded in `TODO.md`.
|
||||
The baseline entrance scored **5.0 overall** and the candidate **5.0 overall**
|
||||
with the small model; its material/texture sub-scores were noisy, so the
|
||||
Pillow metric gap is the more stable signal for this A/B. The candidate's
|
||||
slightly lower street_late/overall mean reflects the extra fog rather than a
|
||||
Compatibility regression.
|
||||
Reference in new issue
Block a user