feat(qa): reference-vs-render iteration loop; screenshots are demo-only

Tools/visual_loop.py:
- --verify gates every capture against its expectation in Tools/visual_targets.json
  (missing, blank, tone-less or wrong-size shots fail), so 'each screenshot shows
  what was expected' is checked rather than assumed.
- --compare measures luminance, RMS contrast, edge density, palette distance and a
  3x3 luma grid against the reference photograph, with a 0-1 similarity score.
- --iteration N writes Art/reference/iterations/N/{metrics.json,feedback.md}; the
  feedback names the skill to consult (blender-modeler, archviz, materials, lookdev,
  camera-cinematography), so the next modelling pass starts from a measurement.
- Iterations are gitignored until the owner confirms a shot; only confirmed shots
  are promoted into Screenshots/.

Screenshots/ holds demo-map captures only now: render_screenshots.sh renders just
DemoMap, collects the shots and runs the verification gate (--verify); the review
staging map is no longer rendered for review and its PNGs are removed. The README
is rewritten around the four demo shots with per-shot expected content and a
verified column, and no longer claims the review staging set.

Tests: 67 pass (13 new) covering metrics on synthetic images, comparison scoring,
the feedback rules and the target spec (including that every target is a demo shot
and that iterations are gitignored).
This commit is contained in:
zwitschi committed 2026-11-08 19:52:25 +01:00
1 parent ad5bb25009
commit bd1f8a9c2e
25 files changed
+688 -121

No files matched your search

+3 -1
View File
@@ -56,7 +56,9 @@ Geographic tools live in [`Geo/tools/`](../Geo/README.md) and the Blender-side b
| Tool | Purpose |
| --- | --- |
| `render_screenshots.py` / `render_screenshots.sh` | Rebuilds the review map, runs the game, captures the shot list into `Screenshots/` |
| `judge_screenshot.py` | Vision-model pre-check: judges a screenshot against an expected-scene description (`--image` + `--expect`, or `--shots` for every shot in the README). Reads `OPENROUTER_API_KEY` from `.env` (ignored). Optional helper — a human must still approve. |
| `judge_screenshot.py` | Vision-model pre-check: judges a screenshot against an expected-scene description (`--image` + `--expect`, or `--shots` for every shot in the README). Reads `OPENROUTER_API_KEY` from `.env` (ignored). Optional, advisory only — the model is unreliable on this content and never replaces a human verdict. |
| `visual_loop.py` | The reference-vs-render loop. `--verify` gates each capture against its expectation in `visual_targets.json` (blank/short/wrong-size/tone-less shots fail); `--compare` measures luminance, contrast, edge density, palette and 3×3 composition against the reference photograph and turns the deltas into modelling actions naming the skill to consult (`blender-modeler`, `archviz`, `materials`, `lookdev`, `camera-cinematography`); `--iteration N` writes the artefacts under `Art/reference/iterations/N/`, which is **gitignored until the owner confirms a shot**. |
| `visual_targets.json` | Per-shot expectation and reference photograph for `visual_loop.py`. Every entry must be a demo-map shot. |
## Gameplay QA
+32 -34
View File
@@ -1,56 +1,54 @@
#!/usr/bin/env bash
# Render in-game screenshots for visual review into <repo>/Screenshots/ (gitignored).
# Render the demo-map screenshots used for visual review into <repo>/Screenshots/demo/.
#
# Usage: Tools/render_screenshots.sh
# Rebuilds /Game/Test/ReviewMap and /Game/Hollywood/DemoMap (shot lists), runs
# the game on each map (the GigLineShotActor walks the cameras, captures, quits),
# and copies the PNGs into Screenshots/. PNGs are never committed.
# Usage: Tools/render_screenshots.sh [--verify]
#
# Only the **demo map** is captured: the review staging map exists to hold the
# hero/camera list, not to produce review screenshots. Everything here is what a
# player actually sees when the demo runs.
#
# --verify run Tools/visual_loop.py --verify afterwards and fail if a shot
# does not show the content its expectation describes
#
# The PNGs are committed (see Screenshots/README.md): a reviewer must see exactly
# what the build rendered.
set -euo pipefail
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
ENGINE="${UE_ENGINE_DIR:-/run/media/system/DATA/Engines/UE_5.8.4}"
PROJ="$REPO/Game/GigLine/GigLine.uproject"
VERIFY="no"
[ "${1:-}" = "--verify" ] && VERIFY="yes"
echo "== 1/4 build review + demo shot maps =="
echo "== 1/3 stage the demo shot cameras =="
cd "$ENGINE"
./Engine/Binaries/Linux/UnrealEditor-Cmd "$PROJ" \
-run=pythonscript -script="$REPO/Tools/ue_build_review_map.py" \
-nullrhi -unattended -nopause -stdout
cat /tmp/ue-review-map-result.txt | tail -3
tail -3 /tmp/ue-review-map-result.txt 2>/dev/null || true
run_capture() {
local map="$1" label="$2"
echo "== capture: $label =="
./Engine/Binaries/Linux/UnrealEditor-Cmd "$PROJ" "$map" \
-game -windowed -ResX=1920 -ResY=1080 -nopause -nosplash -stdout
}
echo "== 2/3 capture the demo map =="
./Engine/Binaries/Linux/UnrealEditor-Cmd "$PROJ" /Game/Hollywood/DemoMap \
-game -windowed -ResX=1920 -ResY=1080 -nopause -nosplash -stdout
echo "== 2/4 run game on review map (hero/scene shots) =="
run_capture "/Game/Test/ReviewMap" "review"
echo "== 3/4 run game on demo map (walkable-demo shots) =="
run_capture "/Game/Hollywood/DemoMap" "demo"
echo "== 4/4 collect screenshots =="
echo "== 3/3 collect the demo screenshots =="
mkdir -p "$REPO/Screenshots/demo"
rm -f "$REPO"/Screenshots/review-*.png "$REPO"/Screenshots/demo/review-*.png
rm -f "$REPO"/Screenshots/demo/review-*.png
found=0
for src in \
"$REPO/Game/GigLine/Saved/Screenshots"/*/review-*.png \
"$REPO/Game/GigLine/Saved/Screenshots"/*/*/review-*.png \
"$HOME/.config/Epic/GigLine/Saved/Screenshots"/*/review-*.png; do
[ -e "$src" ] || continue
name=$(basename "$src")
if [[ "$name" == review-?("")-demo-* || "$name" == review-*-demo-* || "$name" == review-0?-demo-* ]] \
|| [[ "$name" == *demo-* ]]; then
cp -f "$src" "$REPO/Screenshots/demo/$name"
else
cp -f "$src" "$REPO/Screenshots/$name"
fi
found=1
case "$(basename "$src")" in
*demo-*) cp -f "$src" "$REPO/Screenshots/demo/"; found=1 ;;
esac
done
[ "$found" = "1" ] || { echo "ERROR: no review-*.png found"; exit 1; }
echo "--- Screenshots/ ---"
ls -1 "$REPO/Screenshots"
echo "--- Screenshots/demo/ ---"
ls -1 "$REPO/Screenshots/demo"
[ "$found" = "1" ] || { echo "ERROR: no demo screenshots found"; exit 1; }
ls -1 "$REPO/Screenshots/demo"
if [ "$VERIFY" = "yes" ]; then
echo "== verify: does each shot show what it should? =="
cd "$REPO"
python3 Tools/visual_loop.py --verify
fi
+139
View File
@@ -0,0 +1,139 @@
"""Tests for Tools/visual_loop.py — metrics, comparison and feedback rules.
Synthetic 8-bit images are written to a temp directory and measured, so the
numbers can be reasoned about directly (a mid-grey flat frame has near-zero edge
density and contrast; a checkerboard has a lot of both).
"""
import os
import sys
import tempfile
import unittest
from PIL import Image
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
sys.path.insert(0, os.path.join(ROOT, "Tools"))
import visual_loop as vl # noqa: E402
def write_image(path, kind, size=(64, 64)):
img = Image.new("RGB", size)
if kind == "flat":
img.paste((128, 128, 128), (0, 0, size[0], size[1]))
elif kind == "dark":
img.paste((10, 10, 10), (0, 0, size[0], size[1]))
elif kind == "checker":
px = img.load()
for y in range(size[1]):
for x in range(size[0]):
px[x, y] = (240, 240, 240) if (x // 4 + y // 4) % 2 else (20, 20, 20)
elif kind == "sky":
# bright top third, mid stone bottom: composition like an elevation
px = img.load()
for y in range(size[1]):
for x in range(size[0]):
px[x, y] = (150, 190, 230) if y < size[1] // 3 else (170, 165, 155)
img.save(path)
return path
class MetricsTest(unittest.TestCase):
def test_flat_frame_has_no_edges_and_no_contrast(self):
with tempfile.TemporaryDirectory() as tmp:
m = vl.metrics(write_image(os.path.join(tmp, "a.png"), "flat"))
self.assertEqual(m["size"], [64, 64])
self.assertAlmostEqual(m["mean_luma"], 128.0, delta=1.0)
self.assertLess(m["rms_contrast"], 1.0)
self.assertLess(m["edge_density"], vl.MIN_EDGE_DENSITY)
def test_checker_frame_has_edges_and_contrast(self):
with tempfile.TemporaryDirectory() as tmp:
m = vl.metrics(write_image(os.path.join(tmp, "b.png"), "checker"))
self.assertGreater(m["edge_density"], 0.01)
self.assertGreater(m["rms_contrast"], 50.0)
def test_composition_grid_is_nine_cells(self):
with tempfile.TemporaryDirectory() as tmp:
m = vl.metrics(write_image(os.path.join(tmp, "c.png"), "sky"))
self.assertEqual(len(m["grid_luma"]), 9)
# sky row should be brighter than the stone rows
self.assertGreater(m["grid_luma"][0], m["grid_luma"][8])
def test_dominant_palette_is_reported(self):
with tempfile.TemporaryDirectory() as tmp:
m = vl.metrics(write_image(os.path.join(tmp, "d.png"), "flat"))
self.assertTrue(m["dominant"])
self.assertAlmostEqual(m["dominant"][0]["share"], 1.0, delta=0.05)
class CompareTest(unittest.TestCase):
def _metrics(self, kind):
with tempfile.TemporaryDirectory() as tmp:
return vl.metrics(write_image(os.path.join(tmp, "x.png"), kind))
def test_identical_images_score_high(self):
m = self._metrics("checker")
delta = vl.compare(m, m)
self.assertGreater(delta["score"], 0.9)
self.assertEqual(delta["delta_luma"], 0.0)
def test_flat_against_checker_scores_low_on_edges(self):
flat, checker = self._metrics("flat"), self._metrics("checker")
delta = vl.compare(flat, checker)
self.assertLess(delta["edge_ratio"], 0.5)
self.assertLess(delta["score"], 0.9)
def test_dark_against_bright_reports_a_luma_delta(self):
dark, bright = self._metrics("dark"), self._metrics("checker")
delta = vl.compare(dark, bright)
self.assertLess(delta["delta_luma"], -50)
class FeedbackTest(unittest.TestCase):
def test_low_edge_ratio_names_the_modelling_skills(self):
notes = vl.feedback("x", {}, {}, {
"delta_luma": 0, "contrast_ratio": 1.0, "edge_ratio": 0.3,
"palette_distance": 10, "composition_correlation": 0.8, "score": 0.5})
joined = " ".join(notes)
self.assertIn("Relief", joined)
self.assertIn("blender-modeler", joined)
def test_bright_and_compressed_flags_lighting_and_lookdev(self):
notes = vl.feedback("x", {}, {}, {
"delta_luma": 40, "contrast_ratio": 0.5, "edge_ratio": 1.0,
"palette_distance": 10, "composition_correlation": 0.8, "score": 0.5})
joined = " ".join(notes)
self.assertIn("Too bright", joined)
self.assertIn("lookdev", joined)
def test_framing_mismatch_names_the_camera_skill(self):
notes = vl.feedback("x", {}, {}, {
"delta_luma": 0, "contrast_ratio": 1.0, "edge_ratio": 1.0,
"palette_distance": 10, "composition_correlation": 0.1, "score": 0.5})
self.assertIn("camera-cinematography", " ".join(notes))
def test_no_delta_reports_no_action(self):
notes = vl.feedback("x", {}, {}, {
"delta_luma": 0, "contrast_ratio": 1.0, "edge_ratio": 1.0,
"palette_distance": 5, "composition_correlation": 0.9, "score": 0.95})
self.assertIn("No metric is outside its tolerance", notes[0])
class TargetSpecTest(unittest.TestCase):
def test_every_target_has_expectation_and_reference(self):
for target in vl.load_targets():
self.assertTrue(target["expect"].strip(), target["name"])
self.assertTrue(target["references"], target["name"])
self.assertTrue(target["shot"].startswith("Screenshots/demo/"),
f"{target['name']} must be a demo-map shot")
def test_iteration_output_is_gitignored(self):
# Iterations are deliberately uncommitted until the owner confirms a shot.
ignore = open(os.path.join(ROOT, ".gitignore"), encoding="utf-8").read()
self.assertIn("/Art/reference/iterations/", ignore)
if __name__ == "__main__":
unittest.main()
+389
View File
@@ -0,0 +1,389 @@
#!/usr/bin/env python3
"""Iterate a building against the reference photography and the engine render.
The loop this tool implements:
```
reference photograph ─┐
├─ compare ─► metrics + feedback ─► blender-skill actions
engine screenshot ─┘ │
└─ next iteration (uncommitted)
```
It answers three questions with numbers instead of opinions:
1. **Does the shot show what it was supposed to?** (`--verify`) — every capture
has an expectation in `Tools/visual_targets.json`; a shot that is blank, the
wrong size, or wildly unlike its expectation is reported as a failure.
2. **How far is the render from the reference?** (`--compare`) — luminance,
contrast, edge density, palette and 3×3 composition are measured on both and
reported as deltas with a 0–1 similarity score.
3. **What should be modelled next?** (`--feedback`) — the deltas are translated
into concrete actions phrased for the Blender skill packs (silhouette, relief,
value range, hue, framing).
Iterations are **not committed**: every artefact goes under
`Art/reference/iterations/<iteration>/`, which is gitignored. Promote a shot into
`Screenshots/` only after the owner confirms it.
python3 Tools/visual_loop.py --verify # gate the current set
python3 Tools/visual_loop.py --compare --iteration 3 # measure vs references
python3 Tools/visual_loop.py --compare --iteration 3 --judge # + vision advisory
Metrics are computed with Pillow (already in requirements-tools.txt); no numpy.
"""
from __future__ import annotations
import argparse
import json
import os
import shutil
import sys
from PIL import Image, ImageFilter, ImageStat
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
TARGETS = os.path.join(ROOT, "Tools", "visual_targets.json")
ITER_ROOT = os.path.join(ROOT, "Art", "reference", "iterations")
# A shot must clear these floors or it is treated as broken, not merely ugly.
MIN_EDGE_DENSITY = 0.010
MIN_CONTRAST = 12.0
MIN_MEAN_LUMA = 8.0
# --- metrics ----------------------------------------------------------------
def _luma_gray(image):
return image.convert("L")
def metrics(image_path):
"""Measure one image. Everything is on the 8-bit luma/RGB 0..255 scale."""
with Image.open(image_path) as handle:
image = handle.convert("RGB")
width, height = image.size
gray = _luma_gray(image)
stat = ImageStat.Stat(gray)
# Edge density: fraction of pixels above a low threshold after FIND_EDGES.
# Stands in for "how much relief and silhouette detail is in the frame".
# Filter, then crop: FIND_EDGES samples outside the image as zero, so it leaves
# its own border ring at a high value (236 of 3600 px on a 60x60 frame, 6.5%),
# which would make a perfectly flat picture look detailed. Cropping *before*
# filtering does not help — the new border gets the same treatment.
filtered = gray.filter(ImageFilter.FIND_EDGES)
inner = filtered.crop((2, 2, max(3, width - 2), max(3, height - 2)))
hist = inner.histogram()
area = float(inner.width * inner.height)
edge_pixels = sum(hist[24:])
edge_density = edge_pixels / area if area else 0.0
# Palette: quantise and keep the dominant colours with their share.
total = float(width * height)
quantised = image.quantize(colors=12, method=Image.Quantize.MEDIANCUT)
palette = quantised.getpalette() or []
counts = sorted(quantised.getcolors() or [], reverse=True)
dominant = []
for count, index in counts[:5]:
base = index * 3
if base + 2 < len(palette):
dominant.append({
"rgb": palette[base:base + 3],
"share": round(count / total, 3),
})
# 3x3 composition grid of mean luma, so "too much sky" is measurable.
grid = []
for row in range(3):
for col in range(3):
box = (int(col * width / 3), int(row * height / 3),
int((col + 1) * width / 3), int((row + 1) * height / 3))
grid.append(round(ImageStat.Stat(gray.crop(box)).mean[0], 1))
return {
"path": os.path.relpath(image_path, ROOT),
"size": [width, height],
"mean_luma": round(stat.mean[0], 1),
"rms_contrast": round(stat.stddev[0], 1),
"edge_density": round(edge_density, 4),
"dominant": dominant,
"grid_luma": grid,
}
def _palette_distance(a, b):
"""Mean nearest-colour distance between two dominant palettes, 0..441."""
if not a or not b:
return None
total = 0.0
for entry in a:
pa = entry["rgb"]
best = min(
sum((pa[i] - pb["rgb"][i]) ** 2 for i in range(3)) ** 0.5 for pb in b)
total += best
return round(total / len(a), 1)
def _correlation(values_a, values_b):
n = len(values_a)
mean_a = sum(values_a) / n
mean_b = sum(values_b) / n
var_a = sum((v - mean_a) ** 2 for v in values_a)
var_b = sum((v - mean_b) ** 2 for v in values_b)
if var_a <= 0.0 or var_b <= 0.0:
return 0.0
cov = sum((a - mean_a) * (b - mean_b) for a, b in zip(values_a, values_b))
return round(cov / (var_a * var_b) ** 0.5, 3)
def compare(shot_metrics, reference_metrics):
"""Delta between a render and a reference, plus a 0..1 similarity score."""
luma = shot_metrics["mean_luma"] - reference_metrics["mean_luma"]
contrast_ratio = (shot_metrics["rms_contrast"] /
reference_metrics["rms_contrast"]) if reference_metrics["rms_contrast"] else 1.0
edge_ratio = (shot_metrics["edge_density"] /
reference_metrics["edge_density"]) if reference_metrics["edge_density"] else 1.0
palette = _palette_distance(shot_metrics["dominant"], reference_metrics["dominant"])
composition = _correlation(shot_metrics["grid_luma"], reference_metrics["grid_luma"])
# Score: luminance within 25, contrast and edge ratios near 1, palette near 0,
# composition correlated. Each term is clamped to 0..1 and equally weighted.
def near(value, target, span):
return max(0.0, 1.0 - abs(value - target) / span)
score = (
near(luma, 0.0, 25.0)
+ near(contrast_ratio, 1.0, 1.0)
+ near(edge_ratio, 1.0, 1.5)
+ (1.0 if palette is None else near(palette, 0.0, 120.0))
+ max(0.0, composition)
) / 5.0
return {
"delta_luma": round(luma, 1),
"contrast_ratio": round(contrast_ratio, 2),
"edge_ratio": round(edge_ratio, 2),
"palette_distance": palette,
"composition_correlation": composition,
"score": round(score, 3),
}
# --- feedback ---------------------------------------------------------------
# Each rule maps a measured delta to an action phrased for the Blender skills.
def feedback(name, shot, reference, delta):
notes = []
if delta["edge_ratio"] < 0.6:
notes.append(
f"**Relief and silhouette detail** are {1 - delta['edge_ratio']:.0%} below "
f"the reference (`blender-modeler` / `archviz`): add window rhythm, "
f"pilasters, cornices or quoins that cast a shadow line.")
elif delta["edge_ratio"] > 1.8:
notes.append(
"**Too busy** against the reference: the render has far more edge detail. "
"Check for stray kit pieces, z-fighting or duplicated geometry.")
if delta["delta_luma"] <= -15:
notes.append(
f"**Too dark** by {abs(delta['delta_luma']):.0f} luma (`materials` / "
f"`lighting`): raise the material albedo or the exposure.")
elif delta["delta_luma"] >= 15:
notes.append(
f"**Too bright** by {delta['delta_luma']:.0f} luma (`materials` / "
f"`lighting`): lower the albedo or the exposure.")
if delta["contrast_ratio"] < 0.7:
notes.append(
"**Value range compressed** (`lookdev`): the reference has much stronger "
"light/shadow separation. Deepen recesses or widen the material range.")
elif delta["contrast_ratio"] > 1.5:
notes.append(
"**Value range too harsh** (`lookdev`): soften the key or the shadow "
"contrast so the surface does not clip.")
if delta["palette_distance"] is not None and delta["palette_distance"] > 60:
notes.append(
f"**Hue drift** ({delta['palette_distance']:.0f} palette distance, "
f"`materials`): compare the dominant colours in the iteration report "
f"against the reference and correct the base colours.")
if delta["composition_correlation"] < 0.3:
notes.append(
"**Framing differs** (`camera-cinematography`): the 3x3 luma grid does "
"not match the reference. Check the camera height, lens and how much "
"sky/ground is in frame before changing the model.")
if not notes:
notes.append(
"No metric is outside its tolerance. Differences left are stylistic; "
"confirm visually rather than by number.")
return notes
# --- verification -----------------------------------------------------------
def verify(targets, judge=False):
"""Check that each shot exists, is not broken, and looks like its expectation."""
results = []
for target in targets:
shot = os.path.join(ROOT, target["shot"])
entry = {"name": target["name"], "shot": target["shot"], "expect": target["expect"]}
if not os.path.exists(shot):
entry["status"] = "MISSING"
results.append(entry)
continue
m = metrics(shot)
problems = []
if m["edge_density"] < MIN_EDGE_DENSITY:
problems.append(f"almost flat (edge density {m['edge_density']})")
if m["rms_contrast"] < MIN_CONTRAST:
problems.append(f"no tonal range (contrast {m['rms_contrast']})")
if m["mean_luma"] < MIN_MEAN_LUMA:
problems.append(f"black frame (luma {m['mean_luma']})")
expected_size = target.get("size")
if expected_size and m["size"] != expected_size:
problems.append(f"size {m['size']} != {expected_size}")
entry["metrics"] = {k: m[k] for k in
("size", "mean_luma", "rms_contrast", "edge_density")}
entry["status"] = "FAIL" if problems else "OK"
entry["problems"] = problems
if judge and os.environ.get("OPENROUTER_API_KEY") or (
judge and os.path.exists(os.path.join(ROOT, ".env"))):
entry["judge"] = run_judge(shot, target["expect"])
results.append(entry)
return results
def run_judge(shot, expect):
"""Advisory vision check. Never a gate: the model is unreliable on this content."""
import subprocess
try:
proc = subprocess.run(
[sys.executable, os.path.join(ROOT, "Tools", "judge_screenshot.py"),
"--image", shot, "--expect", expect],
capture_output=True, text=True, timeout=180, cwd=ROOT)
out = (proc.stdout or "") + (proc.stderr or "")
for line in out.splitlines():
if line.startswith("verdict="):
return line.strip()
return "unavailable"
except Exception as exc: # noqa: BLE001 - advisory only
return f"error: {exc}"
# --- reporting --------------------------------------------------------------
def write_iteration(iteration, rows, args):
"""Write the iteration artefacts (gitignored) and return the directory."""
out_dir = os.path.join(ITER_ROOT, str(iteration))
os.makedirs(out_dir, exist_ok=True)
for row in rows:
shot = os.path.join(ROOT, row["shot"])
if os.path.exists(shot):
shutil.copy2(shot, os.path.join(out_dir, os.path.basename(shot)))
with open(os.path.join(out_dir, "metrics.json"), "w", encoding="utf-8") as handle:
json.dump({"iteration": iteration, "targets": rows}, handle, indent=2)
lines = [f"# Iteration {iteration} — reference vs render", ""]
for row in rows:
lines.append(f"## {row['name']}")
lines.append("")
lines.append(f"- shot: `{row['shot']}`")
if row.get("reference"):
lines.append(f"- reference: `{row['reference']}`")
delta = row.get("delta")
if delta:
lines.append(
f"- **score {delta['score']:.2f}** · luma {delta['delta_luma']:+.0f} · "
f"contrast ×{delta['contrast_ratio']} · edges ×{delta['edge_ratio']} · "
f"palette {delta['palette_distance']} · composition r={delta['composition_correlation']}")
lines.append("")
for note in row.get("feedback", []):
lines.append(f"- {note}")
lines.append("")
path = os.path.join(out_dir, "feedback.md")
with open(path, "w", encoding="utf-8") as handle:
handle.write("\n".join(lines) + "\n")
return out_dir, path
def load_targets():
with open(TARGETS, encoding="utf-8") as handle:
return json.load(handle)["targets"]
def main(argv=None) -> int:
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--verify", action="store_true",
help="gate the current shots against their expectations")
ap.add_argument("--compare", action="store_true",
help="measure each shot against its reference(s)")
ap.add_argument("--feedback", action="store_true", help="include action notes (default with --compare)")
ap.add_argument("--judge", action="store_true", help="add the advisory vision verdict")
ap.add_argument("--iteration", default=None, help="iteration label for the output directory")
args = ap.parse_args(argv)
if not (args.verify or args.compare):
args.verify = args.compare = True
targets = load_targets()
if args.verify:
print("== verification ==")
results = verify(targets, judge=args.judge)
failed = [r for r in results if r["status"] != "OK"]
for r in results:
extra = f" ({'; '.join(r['problems'])})" if r.get("problems") else ""
judge = f" [judge: {r['judge']}]" if r.get("judge") else ""
print(f" {r['status']:7} {r['name']:32}{extra}{judge}")
print(f" -> {len(results) - len(failed)}/{len(results)} shots OK")
if args.compare:
print("\n== comparison ==")
rows = []
for target in targets:
shot_path = os.path.join(ROOT, target["shot"])
if not os.path.exists(shot_path):
continue
shot_m = metrics(shot_path)
best = None
for ref_rel in target.get("references", []):
ref_path = os.path.join(ROOT, ref_rel)
if not os.path.exists(ref_path):
continue
ref_m = metrics(ref_path)
delta = compare(shot_m, ref_m)
if best is None or delta["score"] > best[1]["score"]:
best = (ref_rel, delta, ref_m)
if best is None:
continue
ref_rel, delta, ref_m = best
rows.append({
"name": target["name"],
"shot": target["shot"],
"reference": ref_rel,
"delta": delta,
"feedback": feedback(target["name"], shot_m, ref_m, delta),
"expect": target["expect"],
})
print(f" {target['name']:32} score {delta['score']:.2f} "
f"luma {delta['delta_luma']:+.0f} edges ×{delta['edge_ratio']}")
if args.iteration is not None:
out_dir, feedback_path = write_iteration(args.iteration, rows, args)
print(f"\n iteration written to {os.path.relpath(out_dir, ROOT)}")
print(f" feedback: {os.path.relpath(feedback_path, ROOT)}")
print(" (gitignored: promote a shot into Screenshots/ only after owner confirmation)")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+53
View File
@@ -0,0 +1,53 @@
{
"comment": "What each demo-map capture must show, and which reference photograph it is measured against. Used by Tools/visual_loop.py --verify (does the shot show the expected content?) and --compare (how far is the render from the reference?).",
"targets": [
{
"name": "demo-spawn-view",
"shot": "Screenshots/demo/review-demo-spawn-view.png",
"size": [
1920,
1080
],
"references": [
"Art/reference/hollywood/el-capitan-facade-day-wikimedia-2009-10-27.jpg"
],
"expect": "The whole south block from the north sidewalk: the neoclassical hero facade between its neighbouring buildings, both sidewalks and the road in the foreground, daylight"
},
{
"name": "demo-street-level",
"shot": "Screenshots/demo/review-demo-street-level.png",
"size": [
1920,
1080
],
"references": [
"Art/reference/hollywood/el-capitan-theatre-blade-sign-day-wikimedia-2008-10.jpg"
],
"expect": "Street level from the north side: the hero colonnade with the adjacent frontage buildings either side, shopfronts at pavement level, daylight"
},
{
"name": "demo-crossing",
"shot": "Screenshots/demo/review-demo-crossing.png",
"size": [
1920,
1080
],
"references": [
"Art/reference/hollywood/walk-of-fame-terrazzo-ovation-day-cc0-2019-06-20.jpg"
],
"expect": "The designated mid-block crossing: continental zebra bars on the asphalt, the south frontage and hero colonnade behind, daylight"
},
{
"name": "demo-north-buildings",
"shot": "Screenshots/demo/review-demo-north-buildings.png",
"size": [
1920,
1080
],
"references": [
"Art/reference/north-chinese/01-grauman-s-chinese-theatre-by-carol-highsmith.jpg"
],
"expect": "The two north-side buildings (TCL Chinese Theatre, Madame Tussaud's) seen from across Hollywood Boulevard, full height in frame, daylight"
}
]
}