test(review): judge every shot from the targets file, and record that all nine fail

--shots was parsing a table shape the README no longer has, so it silently found nothing.
It now reads Tools/visual_targets.json, the same file the render gate uses, and reports one
line per shot with the model's verdict and reasoning.

The result is not good news and is committed as such: 0 of 9 shots judged 'yes'. Shot 7 shows
the scene itself wrong - a disassembled car floating over a blank surface with no street or
buildings in frame - and the others fail for missing detail rather than for a broken frame.
visual_loop --verify still reports 9/9, because it only measures exposure and size.

A warning now sits at the top of Screenshots/README.md: treat every image there as stale until
the scene is fixed. The cause is not yet identified - the map builder's recent changes are 15
lines of vehicle staging, which does not explain a missing street.
This commit is contained in:
zwitschi committed 2026-11-09 16:55:59 +01:00
1 parent 58095b6a33
commit fd2ad4b391
38 files changed
+70 -69

No files matched your search

Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+2
View File
@@ -12,6 +12,8 @@ Tools/render_screenshots.sh --verify
Stages the demo shot cameras, runs the game on `/Game/Hollywood/DemoMap`, copies the PNGs into `Screenshots/demo/` and then runs [`Tools/visual_loop.py`](../Tools/visual_loop.py) `--verify` to check that each shot still shows the content its expectation describes. About a minute. Stages the demo shot cameras, runs the game on `/Game/Hollywood/DemoMap`, copies the PNGs into `Screenshots/demo/` and then runs [`Tools/visual_loop.py`](../Tools/visual_loop.py) `--verify` to check that each shot still shows the content its expectation describes. About a minute.
> **WARNING — the current capture set is broken (2026-11-09).** All nine shots were judged `no` by the vision model, and shot 7 shows the scene itself wrong: a disassembled car floating over a blank surface with no street or buildings in frame. `--verify` still reports 9/9 because it only measures exposure and size. Treat every image in this file as stale until this is fixed; the cause is not yet identified.
## Status at a glance ## Status at a glance
Every shot is embedded below, in order, so the whole review happens in this file. Every shot is embedded below, in order, so the whole review happens in this file.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+22 -23
View File
@@ -112,26 +112,27 @@ def judge(api_key, image_path, expect, model, timeout=60):
return json.loads(m.group(0)) return json.loads(m.group(0))
def parse_shots_from_readme(): def parse_shots_from_targets():
"""Parse the Screenshots/README.md tables into (path, expected) pairs.""" """Read the review shots and their expectations from Tools/visual_targets.json.
readme = os.path.join(PROJECT_ROOT, "Screenshots", "README.md")
if not os.path.isfile(readme): Parsing the README's tables broke when the README was restructured into per-shot
sys.exit(f"error: no {readme}") sections with the comparison table appended: the old parser looked for a table whose
text = open(readme, encoding="utf-8").read() first cell held a backticked `review-` name. The targets file is the same source the
render gate uses, so it is both stable and the thing the expectations are written to.
"""
targets_path = os.path.join(PROJECT_ROOT, "Tools", "visual_targets.json")
if not os.path.isfile(targets_path):
sys.exit(f"error: no {targets_path}")
import json
with open(targets_path) as handle:
data = json.load(handle)
rows = [] rows = []
# Each table row: | File | ... | ... | What it shows | for target in data.get("targets", []):
for line in text.splitlines(): shot = target.get("shot") or ""
if not line.strip().startswith("|"): expect = target.get("expect") or ""
if not shot or not expect:
continue continue
cells = [c.strip() for c in line.strip().strip("|").split("|")] rows.append((shot, expect))
if not cells or "`review-" not in cells[0]:
continue
fname = re.search(r"`(review-[^`]+\.png)`", cells[0])
if not fname:
continue
# the last cell is the "What it shows" description
expect = cells[-1]
rows.append((fname.group(1), expect))
return rows return rows
@@ -151,16 +152,14 @@ def main():
api_key = load_api_key() api_key = load_api_key()
if args.shots: if args.shots:
shots = parse_shots_from_readme() shots = parse_shots_from_targets()
if not shots: if not shots:
sys.exit("error: no review shots found in Screenshots/README.md") sys.exit("error: no shots found in Tools/visual_targets.json")
# Only review-map shots (skip the demo/ ones which have a preview column too).
shots = [s for s in shots if "/demo/" not in s[0]]
n = len(shots) n = len(shots)
print(f"Judging {n} screenshots with {args.model} ...") print(f"Judging {n} screenshots with {args.model} ...")
ok = 0 ok = 0
for i, (fname, expect) in enumerate(shots, 1): for i, (fname, expect) in enumerate(shots, 1):
path = os.path.join(PROJECT_ROOT, "Screenshots", fname.replace("Screenshots/", "")) path = os.path.join(PROJECT_ROOT, fname)
if not os.path.isfile(path): if not os.path.isfile(path):
print(f"[{i}/{n}] {fname}: MISSING (skip)") print(f"[{i}/{n}] {fname}: MISSING (skip)")
continue continue