feat(qa): reference-vs-render iteration loop; screenshots are demo-only
Tools/visual_loop.py:
- --verify gates every capture against its expectation in Tools/visual_targets.json
(missing, blank, tone-less or wrong-size shots fail), so 'each screenshot shows
what was expected' is checked rather than assumed.
- --compare measures luminance, RMS contrast, edge density, palette distance and a
3x3 luma grid against the reference photograph, with a 0-1 similarity score.
- --iteration N writes Art/reference/iterations/N/{metrics.json,feedback.md}; the
feedback names the skill to consult (blender-modeler, archviz, materials, lookdev,
camera-cinematography), so the next modelling pass starts from a measurement.
- Iterations are gitignored until the owner confirms a shot; only confirmed shots
are promoted into Screenshots/.
Screenshots/ holds demo-map captures only now: render_screenshots.sh renders just
DemoMap, collects the shots and runs the verification gate (--verify); the review
staging map is no longer rendered for review and its PNGs are removed. The README
is rewritten around the four demo shots with per-shot expected content and a
verified column, and no longer claims the review staging set.
Tests: 67 pass (13 new) covering metrics on synthetic images, comparison scoring,
the feedback rules and the target spec (including that every target is a demo shot
and that iterations are gitignored).
This commit is contained in:
1 parent
ad5bb25009
commit
bd1f8a9c2e
25 files changed
+688
-121
No files matched your search
+3
-1
@@ -56,7 +56,9 @@ Geographic tools live in [`Geo/tools/`](../Geo/README.md) and the Blender-side b
|
||||
| Tool | Purpose |
|
||||
| --- | --- |
|
||||
| `render_screenshots.py` / `render_screenshots.sh` | Rebuilds the review map, runs the game, captures the shot list into `Screenshots/` |
|
||||
| `judge_screenshot.py` | Vision-model pre-check: judges a screenshot against an expected-scene description (`--image` + `--expect`, or `--shots` for every shot in the README). Reads `OPENROUTER_API_KEY` from `.env` (ignored). Optional helper — a human must still approve. |
|
||||
| `judge_screenshot.py` | Vision-model pre-check: judges a screenshot against an expected-scene description (`--image` + `--expect`, or `--shots` for every shot in the README). Reads `OPENROUTER_API_KEY` from `.env` (ignored). Optional, advisory only — the model is unreliable on this content and never replaces a human verdict. |
|
||||
| `visual_loop.py` | The reference-vs-render loop. `--verify` gates each capture against its expectation in `visual_targets.json` (blank/short/wrong-size/tone-less shots fail); `--compare` measures luminance, contrast, edge density, palette and 3×3 composition against the reference photograph and turns the deltas into modelling actions naming the skill to consult (`blender-modeler`, `archviz`, `materials`, `lookdev`, `camera-cinematography`); `--iteration N` writes the artefacts under `Art/reference/iterations/N/`, which is **gitignored until the owner confirms a shot**. |
|
||||
| `visual_targets.json` | Per-shot expectation and reference photograph for `visual_loop.py`. Every entry must be a demo-map shot. |
|
||||
|
||||
## Gameplay QA
|
||||
|
||||
|
||||
+32
-34
@@ -1,56 +1,54 @@
|
||||
#!/usr/bin/env bash
|
||||
# Render in-game screenshots for visual review into <repo>/Screenshots/ (gitignored).
|
||||
# Render the demo-map screenshots used for visual review into <repo>/Screenshots/demo/.
|
||||
#
|
||||
# Usage: Tools/render_screenshots.sh
|
||||
# Rebuilds /Game/Test/ReviewMap and /Game/Hollywood/DemoMap (shot lists), runs
|
||||
# the game on each map (the GigLineShotActor walks the cameras, captures, quits),
|
||||
# and copies the PNGs into Screenshots/. PNGs are never committed.
|
||||
# Usage: Tools/render_screenshots.sh [--verify]
|
||||
#
|
||||
# Only the **demo map** is captured: the review staging map exists to hold the
|
||||
# hero/camera list, not to produce review screenshots. Everything here is what a
|
||||
# player actually sees when the demo runs.
|
||||
#
|
||||
# --verify run Tools/visual_loop.py --verify afterwards and fail if a shot
|
||||
# does not show the content its expectation describes
|
||||
#
|
||||
# The PNGs are committed (see Screenshots/README.md): a reviewer must see exactly
|
||||
# what the build rendered.
|
||||
set -euo pipefail
|
||||
|
||||
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
ENGINE="${UE_ENGINE_DIR:-/run/media/system/DATA/Engines/UE_5.8.4}"
|
||||
PROJ="$REPO/Game/GigLine/GigLine.uproject"
|
||||
VERIFY="no"
|
||||
[ "${1:-}" = "--verify" ] && VERIFY="yes"
|
||||
|
||||
echo "== 1/4 build review + demo shot maps =="
|
||||
echo "== 1/3 stage the demo shot cameras =="
|
||||
cd "$ENGINE"
|
||||
./Engine/Binaries/Linux/UnrealEditor-Cmd "$PROJ" \
|
||||
-run=pythonscript -script="$REPO/Tools/ue_build_review_map.py" \
|
||||
-nullrhi -unattended -nopause -stdout
|
||||
cat /tmp/ue-review-map-result.txt | tail -3
|
||||
tail -3 /tmp/ue-review-map-result.txt 2>/dev/null || true
|
||||
|
||||
run_capture() {
|
||||
local map="$1" label="$2"
|
||||
echo "== capture: $label =="
|
||||
./Engine/Binaries/Linux/UnrealEditor-Cmd "$PROJ" "$map" \
|
||||
-game -windowed -ResX=1920 -ResY=1080 -nopause -nosplash -stdout
|
||||
}
|
||||
echo "== 2/3 capture the demo map =="
|
||||
./Engine/Binaries/Linux/UnrealEditor-Cmd "$PROJ" /Game/Hollywood/DemoMap \
|
||||
-game -windowed -ResX=1920 -ResY=1080 -nopause -nosplash -stdout
|
||||
|
||||
echo "== 2/4 run game on review map (hero/scene shots) =="
|
||||
run_capture "/Game/Test/ReviewMap" "review"
|
||||
|
||||
echo "== 3/4 run game on demo map (walkable-demo shots) =="
|
||||
run_capture "/Game/Hollywood/DemoMap" "demo"
|
||||
|
||||
echo "== 4/4 collect screenshots =="
|
||||
echo "== 3/3 collect the demo screenshots =="
|
||||
mkdir -p "$REPO/Screenshots/demo"
|
||||
rm -f "$REPO"/Screenshots/review-*.png "$REPO"/Screenshots/demo/review-*.png
|
||||
rm -f "$REPO"/Screenshots/demo/review-*.png
|
||||
found=0
|
||||
for src in \
|
||||
"$REPO/Game/GigLine/Saved/Screenshots"/*/review-*.png \
|
||||
"$REPO/Game/GigLine/Saved/Screenshots"/*/*/review-*.png \
|
||||
"$HOME/.config/Epic/GigLine/Saved/Screenshots"/*/review-*.png; do
|
||||
[ -e "$src" ] || continue
|
||||
name=$(basename "$src")
|
||||
if [[ "$name" == review-?("")-demo-* || "$name" == review-*-demo-* || "$name" == review-0?-demo-* ]] \
|
||||
|| [[ "$name" == *demo-* ]]; then
|
||||
cp -f "$src" "$REPO/Screenshots/demo/$name"
|
||||
else
|
||||
cp -f "$src" "$REPO/Screenshots/$name"
|
||||
fi
|
||||
found=1
|
||||
case "$(basename "$src")" in
|
||||
*demo-*) cp -f "$src" "$REPO/Screenshots/demo/"; found=1 ;;
|
||||
esac
|
||||
done
|
||||
[ "$found" = "1" ] || { echo "ERROR: no review-*.png found"; exit 1; }
|
||||
echo "--- Screenshots/ ---"
|
||||
ls -1 "$REPO/Screenshots"
|
||||
echo "--- Screenshots/demo/ ---"
|
||||
ls -1 "$REPO/Screenshots/demo"
|
||||
[ "$found" = "1" ] || { echo "ERROR: no demo screenshots found"; exit 1; }
|
||||
ls -1 "$REPO/Screenshots/demo"
|
||||
|
||||
if [ "$VERIFY" = "yes" ]; then
|
||||
echo "== verify: does each shot show what it should? =="
|
||||
cd "$REPO"
|
||||
python3 Tools/visual_loop.py --verify
|
||||
fi
|
||||
@@ -0,0 +1,139 @@
|
||||
"""Tests for Tools/visual_loop.py — metrics, comparison and feedback rules.
|
||||
|
||||
Synthetic 8-bit images are written to a temp directory and measured, so the
|
||||
numbers can be reasoned about directly (a mid-grey flat frame has near-zero edge
|
||||
density and contrast; a checkerboard has a lot of both).
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from PIL import Image
|
||||
|
||||
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
|
||||
sys.path.insert(0, os.path.join(ROOT, "Tools"))
|
||||
|
||||
import visual_loop as vl # noqa: E402
|
||||
|
||||
|
||||
def write_image(path, kind, size=(64, 64)):
|
||||
img = Image.new("RGB", size)
|
||||
if kind == "flat":
|
||||
img.paste((128, 128, 128), (0, 0, size[0], size[1]))
|
||||
elif kind == "dark":
|
||||
img.paste((10, 10, 10), (0, 0, size[0], size[1]))
|
||||
elif kind == "checker":
|
||||
px = img.load()
|
||||
for y in range(size[1]):
|
||||
for x in range(size[0]):
|
||||
px[x, y] = (240, 240, 240) if (x // 4 + y // 4) % 2 else (20, 20, 20)
|
||||
elif kind == "sky":
|
||||
# bright top third, mid stone bottom: composition like an elevation
|
||||
px = img.load()
|
||||
for y in range(size[1]):
|
||||
for x in range(size[0]):
|
||||
px[x, y] = (150, 190, 230) if y < size[1] // 3 else (170, 165, 155)
|
||||
img.save(path)
|
||||
return path
|
||||
|
||||
|
||||
class MetricsTest(unittest.TestCase):
|
||||
def test_flat_frame_has_no_edges_and_no_contrast(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
m = vl.metrics(write_image(os.path.join(tmp, "a.png"), "flat"))
|
||||
self.assertEqual(m["size"], [64, 64])
|
||||
self.assertAlmostEqual(m["mean_luma"], 128.0, delta=1.0)
|
||||
self.assertLess(m["rms_contrast"], 1.0)
|
||||
self.assertLess(m["edge_density"], vl.MIN_EDGE_DENSITY)
|
||||
|
||||
def test_checker_frame_has_edges_and_contrast(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
m = vl.metrics(write_image(os.path.join(tmp, "b.png"), "checker"))
|
||||
self.assertGreater(m["edge_density"], 0.01)
|
||||
self.assertGreater(m["rms_contrast"], 50.0)
|
||||
|
||||
def test_composition_grid_is_nine_cells(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
m = vl.metrics(write_image(os.path.join(tmp, "c.png"), "sky"))
|
||||
self.assertEqual(len(m["grid_luma"]), 9)
|
||||
# sky row should be brighter than the stone rows
|
||||
self.assertGreater(m["grid_luma"][0], m["grid_luma"][8])
|
||||
|
||||
def test_dominant_palette_is_reported(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
m = vl.metrics(write_image(os.path.join(tmp, "d.png"), "flat"))
|
||||
self.assertTrue(m["dominant"])
|
||||
self.assertAlmostEqual(m["dominant"][0]["share"], 1.0, delta=0.05)
|
||||
|
||||
|
||||
class CompareTest(unittest.TestCase):
|
||||
def _metrics(self, kind):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
return vl.metrics(write_image(os.path.join(tmp, "x.png"), kind))
|
||||
|
||||
def test_identical_images_score_high(self):
|
||||
m = self._metrics("checker")
|
||||
delta = vl.compare(m, m)
|
||||
self.assertGreater(delta["score"], 0.9)
|
||||
self.assertEqual(delta["delta_luma"], 0.0)
|
||||
|
||||
def test_flat_against_checker_scores_low_on_edges(self):
|
||||
flat, checker = self._metrics("flat"), self._metrics("checker")
|
||||
delta = vl.compare(flat, checker)
|
||||
self.assertLess(delta["edge_ratio"], 0.5)
|
||||
self.assertLess(delta["score"], 0.9)
|
||||
|
||||
def test_dark_against_bright_reports_a_luma_delta(self):
|
||||
dark, bright = self._metrics("dark"), self._metrics("checker")
|
||||
delta = vl.compare(dark, bright)
|
||||
self.assertLess(delta["delta_luma"], -50)
|
||||
|
||||
|
||||
class FeedbackTest(unittest.TestCase):
|
||||
def test_low_edge_ratio_names_the_modelling_skills(self):
|
||||
notes = vl.feedback("x", {}, {}, {
|
||||
"delta_luma": 0, "contrast_ratio": 1.0, "edge_ratio": 0.3,
|
||||
"palette_distance": 10, "composition_correlation": 0.8, "score": 0.5})
|
||||
joined = " ".join(notes)
|
||||
self.assertIn("Relief", joined)
|
||||
self.assertIn("blender-modeler", joined)
|
||||
|
||||
def test_bright_and_compressed_flags_lighting_and_lookdev(self):
|
||||
notes = vl.feedback("x", {}, {}, {
|
||||
"delta_luma": 40, "contrast_ratio": 0.5, "edge_ratio": 1.0,
|
||||
"palette_distance": 10, "composition_correlation": 0.8, "score": 0.5})
|
||||
joined = " ".join(notes)
|
||||
self.assertIn("Too bright", joined)
|
||||
self.assertIn("lookdev", joined)
|
||||
|
||||
def test_framing_mismatch_names_the_camera_skill(self):
|
||||
notes = vl.feedback("x", {}, {}, {
|
||||
"delta_luma": 0, "contrast_ratio": 1.0, "edge_ratio": 1.0,
|
||||
"palette_distance": 10, "composition_correlation": 0.1, "score": 0.5})
|
||||
self.assertIn("camera-cinematography", " ".join(notes))
|
||||
|
||||
def test_no_delta_reports_no_action(self):
|
||||
notes = vl.feedback("x", {}, {}, {
|
||||
"delta_luma": 0, "contrast_ratio": 1.0, "edge_ratio": 1.0,
|
||||
"palette_distance": 5, "composition_correlation": 0.9, "score": 0.95})
|
||||
self.assertIn("No metric is outside its tolerance", notes[0])
|
||||
|
||||
|
||||
class TargetSpecTest(unittest.TestCase):
|
||||
def test_every_target_has_expectation_and_reference(self):
|
||||
for target in vl.load_targets():
|
||||
self.assertTrue(target["expect"].strip(), target["name"])
|
||||
self.assertTrue(target["references"], target["name"])
|
||||
self.assertTrue(target["shot"].startswith("Screenshots/demo/"),
|
||||
f"{target['name']} must be a demo-map shot")
|
||||
|
||||
def test_iteration_output_is_gitignored(self):
|
||||
# Iterations are deliberately uncommitted until the owner confirms a shot.
|
||||
ignore = open(os.path.join(ROOT, ".gitignore"), encoding="utf-8").read()
|
||||
self.assertIn("/Art/reference/iterations/", ignore)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,389 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Iterate a building against the reference photography and the engine render.
|
||||
|
||||
The loop this tool implements:
|
||||
|
||||
```
|
||||
reference photograph ─┐
|
||||
├─ compare ─► metrics + feedback ─► blender-skill actions
|
||||
engine screenshot ─┘ │
|
||||
└─ next iteration (uncommitted)
|
||||
```
|
||||
|
||||
It answers three questions with numbers instead of opinions:
|
||||
|
||||
1. **Does the shot show what it was supposed to?** (`--verify`) — every capture
|
||||
has an expectation in `Tools/visual_targets.json`; a shot that is blank, the
|
||||
wrong size, or wildly unlike its expectation is reported as a failure.
|
||||
2. **How far is the render from the reference?** (`--compare`) — luminance,
|
||||
contrast, edge density, palette and 3×3 composition are measured on both and
|
||||
reported as deltas with a 0–1 similarity score.
|
||||
3. **What should be modelled next?** (`--feedback`) — the deltas are translated
|
||||
into concrete actions phrased for the Blender skill packs (silhouette, relief,
|
||||
value range, hue, framing).
|
||||
|
||||
Iterations are **not committed**: every artefact goes under
|
||||
`Art/reference/iterations/<iteration>/`, which is gitignored. Promote a shot into
|
||||
`Screenshots/` only after the owner confirms it.
|
||||
|
||||
python3 Tools/visual_loop.py --verify # gate the current set
|
||||
python3 Tools/visual_loop.py --compare --iteration 3 # measure vs references
|
||||
python3 Tools/visual_loop.py --compare --iteration 3 --judge # + vision advisory
|
||||
|
||||
Metrics are computed with Pillow (already in requirements-tools.txt); no numpy.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
from PIL import Image, ImageFilter, ImageStat
|
||||
|
||||
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
|
||||
TARGETS = os.path.join(ROOT, "Tools", "visual_targets.json")
|
||||
ITER_ROOT = os.path.join(ROOT, "Art", "reference", "iterations")
|
||||
|
||||
# A shot must clear these floors or it is treated as broken, not merely ugly.
|
||||
MIN_EDGE_DENSITY = 0.010
|
||||
MIN_CONTRAST = 12.0
|
||||
MIN_MEAN_LUMA = 8.0
|
||||
|
||||
|
||||
# --- metrics ----------------------------------------------------------------
|
||||
|
||||
|
||||
def _luma_gray(image):
|
||||
return image.convert("L")
|
||||
|
||||
|
||||
def metrics(image_path):
|
||||
"""Measure one image. Everything is on the 8-bit luma/RGB 0..255 scale."""
|
||||
with Image.open(image_path) as handle:
|
||||
image = handle.convert("RGB")
|
||||
width, height = image.size
|
||||
gray = _luma_gray(image)
|
||||
stat = ImageStat.Stat(gray)
|
||||
|
||||
# Edge density: fraction of pixels above a low threshold after FIND_EDGES.
|
||||
# Stands in for "how much relief and silhouette detail is in the frame".
|
||||
# Filter, then crop: FIND_EDGES samples outside the image as zero, so it leaves
|
||||
# its own border ring at a high value (236 of 3600 px on a 60x60 frame, 6.5%),
|
||||
# which would make a perfectly flat picture look detailed. Cropping *before*
|
||||
# filtering does not help — the new border gets the same treatment.
|
||||
filtered = gray.filter(ImageFilter.FIND_EDGES)
|
||||
inner = filtered.crop((2, 2, max(3, width - 2), max(3, height - 2)))
|
||||
hist = inner.histogram()
|
||||
area = float(inner.width * inner.height)
|
||||
edge_pixels = sum(hist[24:])
|
||||
edge_density = edge_pixels / area if area else 0.0
|
||||
|
||||
# Palette: quantise and keep the dominant colours with their share.
|
||||
total = float(width * height)
|
||||
quantised = image.quantize(colors=12, method=Image.Quantize.MEDIANCUT)
|
||||
palette = quantised.getpalette() or []
|
||||
counts = sorted(quantised.getcolors() or [], reverse=True)
|
||||
dominant = []
|
||||
for count, index in counts[:5]:
|
||||
base = index * 3
|
||||
if base + 2 < len(palette):
|
||||
dominant.append({
|
||||
"rgb": palette[base:base + 3],
|
||||
"share": round(count / total, 3),
|
||||
})
|
||||
|
||||
# 3x3 composition grid of mean luma, so "too much sky" is measurable.
|
||||
grid = []
|
||||
for row in range(3):
|
||||
for col in range(3):
|
||||
box = (int(col * width / 3), int(row * height / 3),
|
||||
int((col + 1) * width / 3), int((row + 1) * height / 3))
|
||||
grid.append(round(ImageStat.Stat(gray.crop(box)).mean[0], 1))
|
||||
|
||||
return {
|
||||
"path": os.path.relpath(image_path, ROOT),
|
||||
"size": [width, height],
|
||||
"mean_luma": round(stat.mean[0], 1),
|
||||
"rms_contrast": round(stat.stddev[0], 1),
|
||||
"edge_density": round(edge_density, 4),
|
||||
"dominant": dominant,
|
||||
"grid_luma": grid,
|
||||
}
|
||||
|
||||
|
||||
def _palette_distance(a, b):
|
||||
"""Mean nearest-colour distance between two dominant palettes, 0..441."""
|
||||
if not a or not b:
|
||||
return None
|
||||
total = 0.0
|
||||
for entry in a:
|
||||
pa = entry["rgb"]
|
||||
best = min(
|
||||
sum((pa[i] - pb["rgb"][i]) ** 2 for i in range(3)) ** 0.5 for pb in b)
|
||||
total += best
|
||||
return round(total / len(a), 1)
|
||||
|
||||
|
||||
def _correlation(values_a, values_b):
|
||||
n = len(values_a)
|
||||
mean_a = sum(values_a) / n
|
||||
mean_b = sum(values_b) / n
|
||||
var_a = sum((v - mean_a) ** 2 for v in values_a)
|
||||
var_b = sum((v - mean_b) ** 2 for v in values_b)
|
||||
if var_a <= 0.0 or var_b <= 0.0:
|
||||
return 0.0
|
||||
cov = sum((a - mean_a) * (b - mean_b) for a, b in zip(values_a, values_b))
|
||||
return round(cov / (var_a * var_b) ** 0.5, 3)
|
||||
|
||||
|
||||
def compare(shot_metrics, reference_metrics):
|
||||
"""Delta between a render and a reference, plus a 0..1 similarity score."""
|
||||
luma = shot_metrics["mean_luma"] - reference_metrics["mean_luma"]
|
||||
contrast_ratio = (shot_metrics["rms_contrast"] /
|
||||
reference_metrics["rms_contrast"]) if reference_metrics["rms_contrast"] else 1.0
|
||||
edge_ratio = (shot_metrics["edge_density"] /
|
||||
reference_metrics["edge_density"]) if reference_metrics["edge_density"] else 1.0
|
||||
palette = _palette_distance(shot_metrics["dominant"], reference_metrics["dominant"])
|
||||
composition = _correlation(shot_metrics["grid_luma"], reference_metrics["grid_luma"])
|
||||
|
||||
# Score: luminance within 25, contrast and edge ratios near 1, palette near 0,
|
||||
# composition correlated. Each term is clamped to 0..1 and equally weighted.
|
||||
def near(value, target, span):
|
||||
return max(0.0, 1.0 - abs(value - target) / span)
|
||||
|
||||
score = (
|
||||
near(luma, 0.0, 25.0)
|
||||
+ near(contrast_ratio, 1.0, 1.0)
|
||||
+ near(edge_ratio, 1.0, 1.5)
|
||||
+ (1.0 if palette is None else near(palette, 0.0, 120.0))
|
||||
+ max(0.0, composition)
|
||||
) / 5.0
|
||||
|
||||
return {
|
||||
"delta_luma": round(luma, 1),
|
||||
"contrast_ratio": round(contrast_ratio, 2),
|
||||
"edge_ratio": round(edge_ratio, 2),
|
||||
"palette_distance": palette,
|
||||
"composition_correlation": composition,
|
||||
"score": round(score, 3),
|
||||
}
|
||||
|
||||
|
||||
# --- feedback ---------------------------------------------------------------
|
||||
# Each rule maps a measured delta to an action phrased for the Blender skills.
|
||||
|
||||
|
||||
def feedback(name, shot, reference, delta):
|
||||
notes = []
|
||||
if delta["edge_ratio"] < 0.6:
|
||||
notes.append(
|
||||
f"**Relief and silhouette detail** are {1 - delta['edge_ratio']:.0%} below "
|
||||
f"the reference (`blender-modeler` / `archviz`): add window rhythm, "
|
||||
f"pilasters, cornices or quoins that cast a shadow line.")
|
||||
elif delta["edge_ratio"] > 1.8:
|
||||
notes.append(
|
||||
"**Too busy** against the reference: the render has far more edge detail. "
|
||||
"Check for stray kit pieces, z-fighting or duplicated geometry.")
|
||||
|
||||
if delta["delta_luma"] <= -15:
|
||||
notes.append(
|
||||
f"**Too dark** by {abs(delta['delta_luma']):.0f} luma (`materials` / "
|
||||
f"`lighting`): raise the material albedo or the exposure.")
|
||||
elif delta["delta_luma"] >= 15:
|
||||
notes.append(
|
||||
f"**Too bright** by {delta['delta_luma']:.0f} luma (`materials` / "
|
||||
f"`lighting`): lower the albedo or the exposure.")
|
||||
|
||||
if delta["contrast_ratio"] < 0.7:
|
||||
notes.append(
|
||||
"**Value range compressed** (`lookdev`): the reference has much stronger "
|
||||
"light/shadow separation. Deepen recesses or widen the material range.")
|
||||
elif delta["contrast_ratio"] > 1.5:
|
||||
notes.append(
|
||||
"**Value range too harsh** (`lookdev`): soften the key or the shadow "
|
||||
"contrast so the surface does not clip.")
|
||||
|
||||
if delta["palette_distance"] is not None and delta["palette_distance"] > 60:
|
||||
notes.append(
|
||||
f"**Hue drift** ({delta['palette_distance']:.0f} palette distance, "
|
||||
f"`materials`): compare the dominant colours in the iteration report "
|
||||
f"against the reference and correct the base colours.")
|
||||
|
||||
if delta["composition_correlation"] < 0.3:
|
||||
notes.append(
|
||||
"**Framing differs** (`camera-cinematography`): the 3x3 luma grid does "
|
||||
"not match the reference. Check the camera height, lens and how much "
|
||||
"sky/ground is in frame before changing the model.")
|
||||
|
||||
if not notes:
|
||||
notes.append(
|
||||
"No metric is outside its tolerance. Differences left are stylistic; "
|
||||
"confirm visually rather than by number.")
|
||||
return notes
|
||||
|
||||
|
||||
# --- verification -----------------------------------------------------------
|
||||
|
||||
|
||||
def verify(targets, judge=False):
|
||||
"""Check that each shot exists, is not broken, and looks like its expectation."""
|
||||
results = []
|
||||
for target in targets:
|
||||
shot = os.path.join(ROOT, target["shot"])
|
||||
entry = {"name": target["name"], "shot": target["shot"], "expect": target["expect"]}
|
||||
if not os.path.exists(shot):
|
||||
entry["status"] = "MISSING"
|
||||
results.append(entry)
|
||||
continue
|
||||
m = metrics(shot)
|
||||
problems = []
|
||||
if m["edge_density"] < MIN_EDGE_DENSITY:
|
||||
problems.append(f"almost flat (edge density {m['edge_density']})")
|
||||
if m["rms_contrast"] < MIN_CONTRAST:
|
||||
problems.append(f"no tonal range (contrast {m['rms_contrast']})")
|
||||
if m["mean_luma"] < MIN_MEAN_LUMA:
|
||||
problems.append(f"black frame (luma {m['mean_luma']})")
|
||||
expected_size = target.get("size")
|
||||
if expected_size and m["size"] != expected_size:
|
||||
problems.append(f"size {m['size']} != {expected_size}")
|
||||
entry["metrics"] = {k: m[k] for k in
|
||||
("size", "mean_luma", "rms_contrast", "edge_density")}
|
||||
entry["status"] = "FAIL" if problems else "OK"
|
||||
entry["problems"] = problems
|
||||
|
||||
if judge and os.environ.get("OPENROUTER_API_KEY") or (
|
||||
judge and os.path.exists(os.path.join(ROOT, ".env"))):
|
||||
entry["judge"] = run_judge(shot, target["expect"])
|
||||
results.append(entry)
|
||||
return results
|
||||
|
||||
|
||||
def run_judge(shot, expect):
|
||||
"""Advisory vision check. Never a gate: the model is unreliable on this content."""
|
||||
import subprocess
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[sys.executable, os.path.join(ROOT, "Tools", "judge_screenshot.py"),
|
||||
"--image", shot, "--expect", expect],
|
||||
capture_output=True, text=True, timeout=180, cwd=ROOT)
|
||||
out = (proc.stdout or "") + (proc.stderr or "")
|
||||
for line in out.splitlines():
|
||||
if line.startswith("verdict="):
|
||||
return line.strip()
|
||||
return "unavailable"
|
||||
except Exception as exc: # noqa: BLE001 - advisory only
|
||||
return f"error: {exc}"
|
||||
|
||||
|
||||
# --- reporting --------------------------------------------------------------
|
||||
|
||||
|
||||
def write_iteration(iteration, rows, args):
|
||||
"""Write the iteration artefacts (gitignored) and return the directory."""
|
||||
out_dir = os.path.join(ITER_ROOT, str(iteration))
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
for row in rows:
|
||||
shot = os.path.join(ROOT, row["shot"])
|
||||
if os.path.exists(shot):
|
||||
shutil.copy2(shot, os.path.join(out_dir, os.path.basename(shot)))
|
||||
with open(os.path.join(out_dir, "metrics.json"), "w", encoding="utf-8") as handle:
|
||||
json.dump({"iteration": iteration, "targets": rows}, handle, indent=2)
|
||||
|
||||
lines = [f"# Iteration {iteration} — reference vs render", ""]
|
||||
for row in rows:
|
||||
lines.append(f"## {row['name']}")
|
||||
lines.append("")
|
||||
lines.append(f"- shot: `{row['shot']}`")
|
||||
if row.get("reference"):
|
||||
lines.append(f"- reference: `{row['reference']}`")
|
||||
delta = row.get("delta")
|
||||
if delta:
|
||||
lines.append(
|
||||
f"- **score {delta['score']:.2f}** · luma {delta['delta_luma']:+.0f} · "
|
||||
f"contrast ×{delta['contrast_ratio']} · edges ×{delta['edge_ratio']} · "
|
||||
f"palette {delta['palette_distance']} · composition r={delta['composition_correlation']}")
|
||||
lines.append("")
|
||||
for note in row.get("feedback", []):
|
||||
lines.append(f"- {note}")
|
||||
lines.append("")
|
||||
path = os.path.join(out_dir, "feedback.md")
|
||||
with open(path, "w", encoding="utf-8") as handle:
|
||||
handle.write("\n".join(lines) + "\n")
|
||||
return out_dir, path
|
||||
|
||||
|
||||
def load_targets():
|
||||
with open(TARGETS, encoding="utf-8") as handle:
|
||||
return json.load(handle)["targets"]
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--verify", action="store_true",
|
||||
help="gate the current shots against their expectations")
|
||||
ap.add_argument("--compare", action="store_true",
|
||||
help="measure each shot against its reference(s)")
|
||||
ap.add_argument("--feedback", action="store_true", help="include action notes (default with --compare)")
|
||||
ap.add_argument("--judge", action="store_true", help="add the advisory vision verdict")
|
||||
ap.add_argument("--iteration", default=None, help="iteration label for the output directory")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
if not (args.verify or args.compare):
|
||||
args.verify = args.compare = True
|
||||
|
||||
targets = load_targets()
|
||||
|
||||
if args.verify:
|
||||
print("== verification ==")
|
||||
results = verify(targets, judge=args.judge)
|
||||
failed = [r for r in results if r["status"] != "OK"]
|
||||
for r in results:
|
||||
extra = f" ({'; '.join(r['problems'])})" if r.get("problems") else ""
|
||||
judge = f" [judge: {r['judge']}]" if r.get("judge") else ""
|
||||
print(f" {r['status']:7} {r['name']:32}{extra}{judge}")
|
||||
print(f" -> {len(results) - len(failed)}/{len(results)} shots OK")
|
||||
|
||||
if args.compare:
|
||||
print("\n== comparison ==")
|
||||
rows = []
|
||||
for target in targets:
|
||||
shot_path = os.path.join(ROOT, target["shot"])
|
||||
if not os.path.exists(shot_path):
|
||||
continue
|
||||
shot_m = metrics(shot_path)
|
||||
best = None
|
||||
for ref_rel in target.get("references", []):
|
||||
ref_path = os.path.join(ROOT, ref_rel)
|
||||
if not os.path.exists(ref_path):
|
||||
continue
|
||||
ref_m = metrics(ref_path)
|
||||
delta = compare(shot_m, ref_m)
|
||||
if best is None or delta["score"] > best[1]["score"]:
|
||||
best = (ref_rel, delta, ref_m)
|
||||
if best is None:
|
||||
continue
|
||||
ref_rel, delta, ref_m = best
|
||||
rows.append({
|
||||
"name": target["name"],
|
||||
"shot": target["shot"],
|
||||
"reference": ref_rel,
|
||||
"delta": delta,
|
||||
"feedback": feedback(target["name"], shot_m, ref_m, delta),
|
||||
"expect": target["expect"],
|
||||
})
|
||||
print(f" {target['name']:32} score {delta['score']:.2f} "
|
||||
f"luma {delta['delta_luma']:+.0f} edges ×{delta['edge_ratio']}")
|
||||
if args.iteration is not None:
|
||||
out_dir, feedback_path = write_iteration(args.iteration, rows, args)
|
||||
print(f"\n iteration written to {os.path.relpath(out_dir, ROOT)}")
|
||||
print(f" feedback: {os.path.relpath(feedback_path, ROOT)}")
|
||||
print(" (gitignored: promote a shot into Screenshots/ only after owner confirmation)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,53 @@
|
||||
{
|
||||
"comment": "What each demo-map capture must show, and which reference photograph it is measured against. Used by Tools/visual_loop.py --verify (does the shot show the expected content?) and --compare (how far is the render from the reference?).",
|
||||
"targets": [
|
||||
{
|
||||
"name": "demo-spawn-view",
|
||||
"shot": "Screenshots/demo/review-demo-spawn-view.png",
|
||||
"size": [
|
||||
1920,
|
||||
1080
|
||||
],
|
||||
"references": [
|
||||
"Art/reference/hollywood/el-capitan-facade-day-wikimedia-2009-10-27.jpg"
|
||||
],
|
||||
"expect": "The whole south block from the north sidewalk: the neoclassical hero facade between its neighbouring buildings, both sidewalks and the road in the foreground, daylight"
|
||||
},
|
||||
{
|
||||
"name": "demo-street-level",
|
||||
"shot": "Screenshots/demo/review-demo-street-level.png",
|
||||
"size": [
|
||||
1920,
|
||||
1080
|
||||
],
|
||||
"references": [
|
||||
"Art/reference/hollywood/el-capitan-theatre-blade-sign-day-wikimedia-2008-10.jpg"
|
||||
],
|
||||
"expect": "Street level from the north side: the hero colonnade with the adjacent frontage buildings either side, shopfronts at pavement level, daylight"
|
||||
},
|
||||
{
|
||||
"name": "demo-crossing",
|
||||
"shot": "Screenshots/demo/review-demo-crossing.png",
|
||||
"size": [
|
||||
1920,
|
||||
1080
|
||||
],
|
||||
"references": [
|
||||
"Art/reference/hollywood/walk-of-fame-terrazzo-ovation-day-cc0-2019-06-20.jpg"
|
||||
],
|
||||
"expect": "The designated mid-block crossing: continental zebra bars on the asphalt, the south frontage and hero colonnade behind, daylight"
|
||||
},
|
||||
{
|
||||
"name": "demo-north-buildings",
|
||||
"shot": "Screenshots/demo/review-demo-north-buildings.png",
|
||||
"size": [
|
||||
1920,
|
||||
1080
|
||||
],
|
||||
"references": [
|
||||
"Art/reference/north-chinese/01-grauman-s-chinese-theatre-by-carol-highsmith.jpg"
|
||||
],
|
||||
"expect": "The two north-side buildings (TCL Chinese Theatre, Madame Tussaud's) seen from across Hollywood Boulevard, full height in frame, daylight"
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in new issue
Block a user