3 Commits
Author SHA1 Message Date
zwitschi 81f7cd3991 feat(geo): cross-check block heights against Mapbox building data (#25)
Merged PR #25: Mapbox building-height cross-check

Co-authored-by: Zwitschi <zwitschi82@gmail.com>
2026-11-08 16:31:39 +01:00
zwitschi 68d8745534 docs(agents): tooling + architecture-upkeep rules, with tool tests (#24)
Merged PR #24: tooling + architecture-upkeep rules, tool tests

Co-authored-by: Zwitschi <zwitschi82@gmail.com>
2026-11-08 16:25:35 +01:00
zwitschi 420f863999 fix(geo): the three "guessed" block heights are a roof, a roof and a bus shelter (#23)
Merged PR #23: block height classification + hero proportions recorded

Co-authored-by: Zwitschi <zwitschi82@gmail.com>
2026-11-08 16:20:46 +01:00
16 changed files with 712 additions and 15 deletions

No files matched your search

+5
View File
@@ -42,6 +42,11 @@ Thumbs.db
.env.* .env.*
*.env *.env
# Mapbox-derived data — local only. Mapbox terms restrict persisting/redistributing
# their data, so the Tilequery cache and any generated report stay untracked.
/Geo/raw/mapbox-cache/
/Geo/verification/*mapbox*
# Local, private reference material — never committed (see Docs/reference-notes-hollywood.md) # Local, private reference material — never committed (see Docs/reference-notes-hollywood.md)
/Art/reference/ /Art/reference/
/Docs/data-inventory.md /Docs/data-inventory.md
+53
View File
@@ -110,6 +110,59 @@ Keep entries short and dated, with the evidence: the command, the log line, or t
Memory edits on an agent's branch are not shared live. Communicate important findings to the coordinator during work and reconcile them into the shared `main` copy when integrating. Memory edits on an agent's branch are not shared live. Communicate important findings to the coordinator during work and reconcile them into the shared `main` copy when integrating.
## Python and shell tooling
Every committed asset and every measured claim in this repository is produced by a script. Tools live in `Tools/` (Unreal, network, repository), `Geo/tools/` (projection, OSM/DEM parsing, metadata) and `Art/blender/*/` (Blender-side builders, next to their output).
**Fix the tool, not the artefact.** If an asset is corrected by hand, script the correction (or regenerate it) in the same change. A second manual fix of the same kind means the tool is missing a step.
**Improve over fork.** Extend the existing tool rather than adding a near-duplicate. When two or three scripts start sharing logic, factor the shared part into `Tools/_common.py`, `Geo/tools/osm_common.py` or `Art/blender/common.py`; keep the single metres→centimetres conversion where it already lives.
Rules that keep tools reusable and reviewable:
- **One script, one job.** Every input is an argument; no interactive prompts; deterministic output paths.
- **Fail loudly.** A tool that cannot do its job exits non-zero with a message. Never leave a partially written asset and a success code.
- **Result files for Unreal.** Unreal commandlets swallow `print()`; write the outcome to `/tmp/gigline/<tool>.txt` and read that instead of trusting the exit code alone.
- **Absolute paths into sandboxes.** Blender Flatpak (`LD_LIBRARY_PATH="$B/blender/lib"`) and Unreal do not share the repo's working directory; pass absolute paths.
- **Clean worktree for generators.** Tools that rewrite committed assets (site/street/hero builders, importers, map builders) run from a clean worktree; inspect the LFS diff before committing. Note that `.blend` and `.umap` check out read-only — `chmod u+w` first or the save silently fails.
- **Network and paid-API tools are human-triggered.** `Tools/commons_fetch.py`, `Tools/page_image_fetch.py`, `Tools/judge_screenshot.py` (OpenRouter) and anything similar never sit in a gate or CI. Handle HTTP 429 with back-off rather than retrying immediately.
- **Measure, don't assert.** Verification tools print the sampled numbers — counts, holes, Z ranges, worst step and its location, frame-time percentiles — so a claim in `TODO.md`, `CHANGELOG.md` or `Docs/` can be re-checked by re-running the command.
**Document every tool.** A new or changed tool gets a row in the inventory table of the README beside it: [`Tools/README.md`](Tools/README.md), [`Geo/README.md`](Geo/README.md), or [`Art/blender/README.md`](Art/blender/README.md). What it does, its arguments, and a typical command. A tool without a README row is unfinished.
**Test the pure parts.** `Tools/tests/` holds stdlib `unittest` tests (no Blender or Unreal process; they must not start a game or editor). Extract the testable core — argument parsing, geometry maths, licence gating, report formatting — into functions that a test can call directly.
```bash
python3 -m unittest discover -s Tools/tests -t Tools -p 'test_*.py'
```
Keep the tools standard-library-only where practical; if a third-party import is genuinely needed, add it to `requirements-tools.txt` with a version floor and mention it in the tool's README row.
## Architecture documentation
[`Docs/Architecture/`](Docs/Architecture/) is the arc42 documentation: twelve numbered section files, `about-arc42.md`, and the combined [`architecture.md`](Docs/Architecture/architecture.md). **Markdown is canonical** — the `.adoc` sources were removed on 2026-11-08.
Keep it current **in the same change as the implementation**, not in a later documentation pass. When work alters any of the following, edit the matching section:
| Change | Section |
| --- | --- |
| A new or restructured module, tool, data source or external interface | `05_building_block_view.md` |
| A runtime flow, tick/order dependency, or lifecycle (possession, map load, QA self-test) | `06_runtime_view.md` |
| Packaging, engine pin, checkout/build/launch procedure | `07_deployment_view.md` |
| A pattern, convention or cross-cutting rule (units, collision policy, provenance) | `08_concepts.md` |
| A decision that constrains future work | new ADR in `09_architecture_decisions.md` |
| A quality goal or a measured budget (frame time, traversal coverage) | `10_quality_requirements.md` |
| A known risk or a risk that retired | `11_technical_risks.md` |
| A new term, tool or abbreviation | `12_glossary.md` |
Rules:
- **Describe what exists.** No aspirational architecture: anything planned belongs in `ROADMAP.md` and `TODO.md`. Every structural claim names the file, script or flag that implements it.
- **ADRs are append-only.** Never rewrite an accepted decision; supersede it with a new ADR that says what it replaces and why.
- **Keep the combined document in sync.** `architecture.md` mirrors the section files; when a section changes, update the combined document too.
- **Images** live in `Docs/Architecture/images/` and are referenced, not duplicated.
- **Verify:** all Markdown must pass `prettier --check` (config in `.prettierrc`), and a command you quote must still run.
## Geographic & Art Pipeline ## Geographic & Art Pipeline
1. Save a versioned OSM extract; document date, bounds, and source. 1. Save a versioned OSM extract; document date, bounds, and source.
+4 -1
View File
@@ -60,8 +60,11 @@ Python scripts in `Tools/` drive the content pipeline:
| `ue_create_input_assets` | Enhanced Input actions + mapping context | | `ue_create_input_assets` | Enhanced Input actions + mapping context |
| `render_screenshots.sh` | Screenshot capture pipeline | | `render_screenshots.sh` | Screenshot capture pipeline |
| `judge_screenshot.py` | Vision-model pre-check (optional) | | `judge_screenshot.py` | Vision-model pre-check (optional) |
| `input_emulation.py`, `input_test.py` | OS-level virtual keyboard/mouse injection for input verification |
| `Tools/tests/` | Stdlib `unittest` tests for the pure parts of the tools (`requirements-tools.txt`) |
| `Geo/tools/mapbox_buildings.py` | Second height source: Mapbox `mapbox-streets-v8` building data vs our OSM heights |
Source data: `Art/blender/` (`.blend` files), `Geo/` (OSM extracts, DEM, verification docs). Source data: `Art/blender/` (`.blend` files), `Geo/` (OSM extracts, DEM, verification docs). Tool rules (one script one job, result files for commandlets, clean worktree for generators, documented inventory, tests for the pure parts) are in [`AGENTS.md`](../../AGENTS.md#python-and-shell-tooling).
### QA & Review ### QA & Review
+3
View File
@@ -199,6 +199,9 @@ Python scripts in `Tools/` drive the content pipeline:
| `ue_create_input_assets` | Enhanced Input actions + mapping context | | `ue_create_input_assets` | Enhanced Input actions + mapping context |
| `render_screenshots.sh` | Screenshot capture pipeline | | `render_screenshots.sh` | Screenshot capture pipeline |
| `judge_screenshot.py` | Vision-model pre-check (optional) | | `judge_screenshot.py` | Vision-model pre-check (optional) |
| `input_emulation.py`, `input_test.py` | OS-level virtual keyboard/mouse injection for input verification |
| `Tools/tests/` | Stdlib `unittest` tests for the pure parts of the tools (`requirements-tools.txt`) |
| `Geo/tools/mapbox_buildings.py` | Second height source: Mapbox `mapbox-streets-v8` building data vs our OSM heights |
Source data: `Art/blender/` (`.blend` files), `Geo/` (OSM extracts, DEM, verification docs). Source data: `Art/blender/` (`.blend` files), `Geo/` (OSM extracts, DEM, verification docs).
+1
View File
@@ -27,6 +27,7 @@ Tools that build 3D content from these sources live next to their output in `Art
| `elevation_probe.py` | Samples 3DEP for relief and records the baseline for the release checks | | `elevation_probe.py` | Samples 3DEP for relief and records the baseline for the release checks |
| `dem_stats.py` | Stdlib reader for the tiled GeoTIFF the 3DEP service returns; GDAL does not run on this host | | `dem_stats.py` | Stdlib reader for the tiled GeoTIFF the 3DEP service returns; GDAL does not run on this host |
| `asset_metadata.py` | Writes and audits `<asset>.geo.json` sidecars: position, provenance, conversion | | `asset_metadata.py` | Writes and audits `<asset>.geo.json` sidecars: position, provenance, conversion |
| `mapbox_buildings.py` | Cross-checks every building's height against Mapbox `mapbox-streets-v8` building data (Tilequery per centroid), merging our OSM height + source with theirs and flagging matches, close, disagreements, non-building volumes. Needs `MAPBOX_ACCESS_TOKEN` in `.env`; caches responses under `Geo/raw/mapbox-cache/`, and **both the cache and its report are gitignored** (Mapbox terms restrict redistribution) |
## Unit boundary ## Unit boundary
+26 -9
View File
@@ -22,6 +22,29 @@ import osm_common as oc
LEVEL_HEIGHT = 3.2 LEVEL_HEIGHT = 3.2
def classify_height(tags, level_height=LEVEL_HEIGHT):
"""Return (height_m, source) for one OSM way's tags.
Extracted from `main()` so the classification is testable: the roof/shelter
rule and the "height tag beats levels beats guess" order are the parts that
actually decide what the blockout should render.
"""
# Not building volumes: elevated roof/canopy structures and transit shelters.
# Found by cross-checking against Mapbox's building data
# (Geo/tools/mapbox_buildings.py), which returns roof/canopy/shelter for the
# same footprints instead of a storey count.
building = tags.get("building")
if building in ("roof", "canopy") or tags.get("amenity") == "shelter":
kind = building if building in ("roof", "canopy") else "shelter"
return 0.0, f"not a building volume ({kind})"
if tags.get("height"):
return float(tags["height"]), "height tag"
if tags.get("building:levels"):
return (float(tags["building:levels"]) * level_height,
f"levels x {level_height} m")
return 2 * level_height, f"GUESS: default 2 levels x {level_height} m"
def main(): def main():
ap = argparse.ArgumentParser() ap = argparse.ArgumentParser()
ap.add_argument("osm") ap.add_argument("osm")
@@ -43,14 +66,7 @@ def main():
pts = oc.way_points(nodes, refs) pts = oc.way_points(nodes, refs)
if not pts or not in_block(pts): if not pts or not in_block(pts):
continue continue
tag_height = tags.get("height") height, source = classify_height(tags)
levels = tags.get("building:levels")
if tag_height:
height, source = float(tag_height), "height tag"
elif levels:
height, source = float(levels) * LEVEL_HEIGHT, f"levels x {LEVEL_HEIGHT} m"
else:
height, source = 2 * LEVEL_HEIGHT, f"GUESS: default 2 levels x {LEVEL_HEIGHT} m"
nearest = min(oc.project(p, line, lat0, lon0, lengths)[0] for p in pts) nearest = min(oc.project(p, line, lat0, lon0, lengths)[0] for p in pts)
side = "S" if oc.project(pts[0], line, lat0, lon0, lengths)[2] < 0 else "N" side = "S" if oc.project(pts[0], line, lat0, lon0, lengths)[2] < 0 else "N"
rows.append((side, nearest, wid, tags.get("name", "") or tags.get("addr:housenumber", ""), rows.append((side, nearest, wid, tags.get("name", "") or tags.get("addr:housenumber", ""),
@@ -60,7 +76,8 @@ def main():
guessed = [r for r in rows if r[5].startswith("GUESS")] guessed = [r for r in rows if r[5].startswith("GUESS")]
print(f"{len(rows)} buildings in the block, {len(guessed)} with a guessed height\n") print(f"{len(rows)} buildings in the block, {len(guessed)} with a guessed height\n")
for side, setback, wid, name, height, source in rows: for side, setback, wid, name, height, source in rows:
flag = " <-- guessed" if source.startswith("GUESS") else "" flag = (" <-- guessed" if source.startswith("GUESS")
else (" <-- not a building volume" if source.startswith("not a building") else ""))
print(f"{side} way {wid:>10} setback {setback:5.1f} m height {height:5.1f} m " print(f"{side} way {wid:>10} setback {setback:5.1f} m height {height:5.1f} m "
f"({source:26}) {name[:34]}{flag}") f"({source:26}) {name[:34]}{flag}")
+317
View File
@@ -0,0 +1,317 @@
#!/usr/bin/env python3
"""Cross-check block building heights against Mapbox's building data.
Mapbox publishes building footprints with ``height`` / ``min_height`` /
``type`` attributes in the ``mapbox-streets-v8`` tileset. They come from
different source data than our OSM extract, so comparing the two is a cheap way
to find buildings where our height is wrong or missing — the ``0.1.2`` finding
that three "guessed" volumes were actually a roof, a roof and a bus shelter is
exactly the class of problem this tool hunts for.
Usage
-----
# token from the environment or the ignored .env file
python3 Geo/tools/mapbox_buildings.py \
--osm Geo/raw/osm-api-map-2026-11-06.osm \
--out Geo/verification/2026-11-09-mapbox-buildings.md
# show only disagreements and unverifiable buildings
python3 Geo/tools/mapbox_buildings.py --only-diffs
Every Tilequery response is cached under ``Geo/raw/mapbox-cache/`` so a re-run
costs nothing and the report is reproducible. The cache is gitignored along with
``Geo/raw/``.
Design notes
------------
* Standard library only (``urllib``), like the rest of ``Geo/tools``.
* One request per building, spaced by ``--delay`` (default 1.2 s) — Mapbox bills
per request and rate-limits bursts.
* The report is a *measurement*: it prints both figures and the delta, never a
single verdict, so a human can decide which source to trust for a given
building.
* A Tilequery hit is only accepted as "this building" when the matched polygon
contains or nearly contains the queried point (``distance`` ≤ ``--max-distance``).
Licensing / data handling
-------------------------
Mapbox terms restrict persisting and redistributing their data, so **neither the
cache nor the generated report is committed** (`.gitignore` covers
`Geo/raw/mapbox-cache/` and `Geo/verification/*mapbox*`). They are local working
material: re-run the tool rather than sharing the output. The token is read from
the environment or the ignored `.env` and is never written to the cache, the
report or the JSON.
"""
from __future__ import annotations
import argparse
import json
import math
import os
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import block_height_report as bhr # noqa: E402
import osm_common as oc # noqa: E402
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
CACHE_DIR = os.path.join(ROOT, "Geo", "raw", "mapbox-cache")
TILESET = "mapbox.mapbox-streets-v8"
LAYER = "building"
API = "https://api.mapbox.com/v4"
def load_token(explicit: str | None = None) -> str:
"""Token from --token, the environment, or the ignored .env file."""
if explicit:
return explicit.strip()
token = os.environ.get("MAPBOX_ACCESS_TOKEN")
if token:
return token.strip()
env_path = os.path.join(ROOT, ".env")
if os.path.exists(env_path):
with open(env_path, encoding="utf-8") as handle:
for line in handle:
line = line.strip()
if line.startswith("MAPBOX_ACCESS_TOKEN="):
return line.split("=", 1)[1].strip().strip('"').strip("'")
raise SystemExit(
"no Mapbox token: pass --token, set MAPBOX_ACCESS_TOKEN, or add it to .env")
def centroid_lat_lon(points: list[tuple[float, float]]) -> tuple[float, float]:
"""Centroid of a ring of (lat, lon) points, as (lat, lon)."""
lat = sum(p[0] for p in points) / len(points)
lon = sum(p[1] for p in points) / len(points)
return lat, lon
def cache_path(lat: float, lon: float, radius: int) -> str:
return os.path.join(CACHE_DIR, f"{lat:.6f}_{lon:.6f}_r{radius}.json")
def tilequery(lat: float, lon: float, token: str, radius: int,
delay: float, use_cache: bool = True) -> tuple[dict, bool]:
"""Tilequery one point. Returns (payload, from_cache)."""
path = cache_path(lat, lon, radius)
if use_cache and os.path.exists(path):
with open(path, encoding="utf-8") as handle:
return json.load(handle), True
query = urllib.parse.urlencode({
"radius": radius, "layers": LAYER, "limit": 10, "access_token": token,
})
url = f"{API}/{TILESET}/tilequery/{lon:.6f},{lat:.6f}.json?{query}"
request = urllib.request.Request(url, headers={"User-Agent": "gig-line-geo/1.0"})
payload = None
for attempt in range(4):
try:
with urllib.request.urlopen(request, timeout=30) as response:
payload = json.load(response)
break
except urllib.error.HTTPError as exc:
if exc.code == 429 or exc.code >= 500:
wait = delay * (2 ** attempt)
sys.stderr.write(f" HTTP {exc.code}; retrying in {wait:.1f}s\n")
time.sleep(wait)
continue
raise
except urllib.error.URLError as exc:
sys.stderr.write(f" network error ({exc.reason}); retrying\n")
time.sleep(delay * (2 ** attempt))
if payload is None:
raise SystemExit(f"tilequery failed for {lat:.6f},{lon:.6f}")
if token in json.dumps(payload):
raise SystemExit("refusing to cache a response that contains the token")
os.makedirs(CACHE_DIR, exist_ok=True)
with open(path, "w", encoding="utf-8") as handle:
json.dump(payload, handle)
time.sleep(delay) # only sleep on a real request, never on a cache hit
return payload, False
def nearest_building(payload: dict, max_distance: float) -> dict | None:
"""The closest building feature within `max_distance` metres, or None."""
best = None
for feature in payload.get("features", []):
props = feature.get("properties", {})
if props.get("tilequery", {}).get("layer") != LAYER:
continue
distance = props.get("tilequery", {}).get("distance")
if distance is None or distance > max_distance:
continue
if best is None or distance < best[0]:
best = (distance, props)
return None if best is None else best[1]
def compare(ours: float, ours_source: str, theirs: float | None) -> tuple[str, float | None]:
"""Classify the agreement between our height and Mapbox's.
Returns (verdict, delta_m). Verdicts are deliberately blunt so the report can
be scanned: "match", "close", "disagree", "mapbox-only", "ours-only".
"""
ours_is_volume = not ours_source.startswith("not a building")
if not ours_is_volume:
return "non-building", None
if theirs is None:
return "ours-only", None
if ours <= 0.0:
return "mapbox-only", theirs
delta = theirs - ours
if abs(delta) <= 0.5:
return "match", delta
if abs(delta) <= 2.0:
return "close", delta
return "disagree", delta
def build_rows(nodes, ways, token, radius, max_distance, delay) -> list[dict]:
rows = []
for wid, tags, refs in ways:
if "building" not in tags:
continue
points = oc.way_points(nodes, refs)
if not points:
continue
lat, lon = centroid_lat_lon(points)
payload, cached = tilequery(lat, lon, token, radius, delay)
feature = nearest_building(payload, max_distance)
ours_height, ours_source = bhr.classify_height(tags)
theirs = None if feature is None else feature.get("height")
verdict, delta = compare(ours_height, ours_source, theirs)
rows.append({
"way": wid,
"name": tags.get("name") or tags.get("addr:housenumber") or "",
"kind": tags.get("building"),
"osm_height_m": round(ours_height, 2) if ours_height else 0.0,
"osm_source": ours_source,
"mapbox_height_m": None if theirs is None else round(float(theirs), 2),
"mapbox_min_height_m": None if feature is None else feature.get("min_height"),
"mapbox_type": None if feature is None else feature.get("type"),
"mapbox_distance_m": None if feature is None
else round(feature.get("tilequery", {}).get("distance", 0.0), 1),
"verdict": verdict,
"delta_m": None if delta is None else round(delta, 2),
"cached": cached,
"lat": round(lat, 6),
"lon": round(lon, 6),
})
return rows
def render_markdown(rows: list[dict], args) -> str:
order = {"disagree": 0, "mapbox-only": 1, "ours-only": 2,
"non-building": 3, "close": 4, "match": 5}
rows = sorted(rows, key=lambda r: (order.get(r["verdict"], 9),
-abs(r["delta_m"] or 0)))
counts = {}
for row in rows:
counts[row["verdict"]] = counts.get(row["verdict"], 0) + 1
lines = [
"# Block buildings — OSM vs Mapbox",
"",
"Generated by `Geo/tools/mapbox_buildings.py`; re-run it to reproduce.",
"Mapbox data: tileset `mapbox-streets-v8`, layer `building`, Tilequery",
f"at each building centroid (radius {args.radius} m, accept ≤ "
f"{args.max_distance} m). OSM heights come from the tagged extract, via the",
"same classification used for the blockout (tag > levels > guess; roof and",
"shelter are not building volumes).",
"",
f"Verdicts: {', '.join(f'{k} {v}' for k, v in sorted(counts.items()))}.",
"",
"| Verdict | Δ | Building | OSM | OSM source | Mapbox | Mapbox type | dist |",
"| --- | ---: | --- | ---: | --- | ---: | --- | ---: |",
]
for row in rows:
osm = f"{row['osm_height_m']:.1f} m"
mb = "—" if row["mapbox_height_m"] is None else f"{row['mapbox_height_m']:.1f} m"
delta = "—" if row["delta_m"] is None else f"{row['delta_m']:+.1f} m"
dist = "—" if row["mapbox_distance_m"] is None else f"{row['mapbox_distance_m']:.0f} m"
name = row["name"] or f"way {row['way']}"
lines.append(
f"| {row['verdict']} | {delta} | {name} | {osm} | {row['osm_source']} | "
f"{mb} | {row['mapbox_type'] or '—'} | {dist} |")
lines += [
"",
"## How to read this",
"",
"- **disagree** — the two sources differ by more than 2 m. One of them is",
" wrong; check the reference photography before changing the model.",
"- **mapbox-only** — Mapbox knows a height we do not (usually a building",
" with no OSM `height`/`building:levels`).",
"- **ours-only** — Mapbox returns no building within the accept distance;",
" the footprint may be missing from its tileset, or the centroid may fall",
" outside its polygon.",
"- **non-building** — our classification says roof/shelter; excluded from",
" height comparison on purpose.",
"",
"Neither source is authoritative: OSM is community-tagged, Mapbox is a",
"derived commercial dataset. Treat a disagreement as a reason to look at",
"the photographs, not as an automatic correction.",
"",
]
if args.only_diffs:
lines.insert(0, "> Filtered: only disagreements and unverifiable buildings.\n")
return "\n".join(lines)
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--osm", default="Geo/raw/osm-api-map-2026-11-06.osm")
ap.add_argument("--out", help="Markdown report path; omit to print to stdout")
ap.add_argument("--json", help="Also write the merged rows as JSON here")
ap.add_argument("--token", help="Mapbox access token (default: env/.env)")
ap.add_argument("--radius", type=int, default=25, help="Tilequery radius, metres")
ap.add_argument("--max-distance", type=float, default=8.0,
help="accept a hit only within this distance of the centroid")
ap.add_argument("--delay", type=float, default=1.2, help="seconds between requests")
ap.add_argument("--only-diffs", action="store_true",
help="report only disagreements / unverifiable buildings")
ap.add_argument("--no-cache", action="store_true", help="ignore the tilequery cache")
args = ap.parse_args()
token = load_token(args.token)
nodes, ways = oc.load(args.osm)
rows = build_rows(nodes, ways, token, args.radius, args.max_distance, args.delay)
if args.no_cache:
for row in rows:
row["cached"] = False
if args.only_diffs:
keep = {"disagree", "mapbox-only", "ours-only", "non-building"}
rows = [r for r in rows if r["verdict"] in keep]
markdown = render_markdown(rows, args)
if args.out:
os.makedirs(os.path.dirname(os.path.abspath(args.out)), exist_ok=True)
with open(args.out, "w", encoding="utf-8") as handle:
handle.write(markdown + "\n")
print(f"wrote {args.out} ({len(rows)} buildings)")
else:
print(markdown)
if args.json:
payload = {
"generated_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"osm": args.osm,
"mapbox": {"tileset": TILESET, "layer": LAYER,
"radius_m": args.radius, "max_distance_m": args.max_distance},
"buildings": rows,
}
with open(args.json, "w", encoding="utf-8") as handle:
json.dump(payload, handle, indent=2)
print(f"wrote {args.json}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+20
View File
@@ -359,3 +359,23 @@ Scope: facts and traps that cost time to establish. Planned work lives in [TODO.
45 cm MaxStepHeight, so not blocking. 45 cm MaxStepHeight, so not blocking.
- Kerb height finalised at 0.15 m modelled / 0.13 m built (6 in LA standard; DEM - Kerb height finalised at 0.15 m modelled / 0.13 m built (6 in LA standard; DEM
and photos can't resolve finer). and photos can't resolve finer).
## 2026-11-09 — Mapbox building data as a second height source
- `Geo/tools/mapbox_buildings.py` (new, stdlib `urllib`) Tilequeries
`mapbox.mapbox-streets-v8` layer `building` at every building centroid in the
extract and merges the answer with our OSM height + source. Token comes from
`MAPBOX_ACCESS_TOKEN` in the ignored `.env`; responses cache to
`Geo/raw/mapbox-cache/`.
- **Mapbox terms restrict persisting/redistributing their data**, so `.gitignore`
now covers `/Geo/raw/mapbox-cache/` and `/Geo/verification/*mapbox*`. The tool
also refuses to cache a response that contains the token.
- **First run (72 buildings): 58 match, 5 close, 3 disagree, 6 non-building.**
- It independently classifies our roof/canopy/shelter volumes as `roof`/
`shelter` — corroborating the `0.1.2` finding that they are not buildings.
- Remaining disagreements are untyped `building=yes`/`apartments` rear
structures where our fallback guess (2 × 3.2 m = 6.4 m) meets Mapbox's 3.0 m.
- Every "close" case is a `building:levels × 3.2 m` building where Mapbox is
**0.6 m lower per 6 levels** → Mapbox implies **3.1 m per level**, not 3.2.
Worth revisiting `LEVEL_HEIGHT` if a measured storey height ever lands.
- Its query API surface: `height`, `min_height`, `type`, `extrude` per polygon.
+4 -4
View File
@@ -13,12 +13,12 @@ Actionable task list for this project. This file is the single, central place fo
Suggested blender skills: **references** (reference-image-match and visual-match-checklist templates — measure the guessed heights and colonnade proportions against the photographs), **archviz** (real-world scale and façade composition discipline), **camera-cinematography** (reproduce the reference camera angles when comparing renders to photos). Suggested blender skills: **references** (reference-image-match and visual-match-checklist templates — measure the guessed heights and colonnade proportions against the photographs), **archviz** (real-world scale and façade composition discipline), **camera-cinematography** (reproduce the reference camera angles when comparing renders to photos).
### Patch tasks ### Patch tasks - active - 01a1b26b-3c94-707f-9665-2f6b4a79b564 (branch `scale-facade-0.1.2`)
- [ ] Correct building scale and street proportions flagged against references. - [ ] Correct building scale and street proportions flagged against references. **Cross-check available:** `Geo/tools/mapbox_buildings.py` merges our OSM heights with Mapbox's building data for the same footprints (72 buildings: 58 match, 5 close, 3 disagree, 6 non-building). Three rear `building=yes`/`apartments` structures still render at our 2-level guess (6.4 m) against Mapbox's 3.0 m; and every `levels × 3.2 m` building is 0.6 m taller per 6 levels than Mapbox implies, i.e. **Mapbox suggests 3.1 m per level**. Neither is applied yet — decide from the reference photography.
- [ ] Refine hero façade details where reference photography shows discrepancies. - [ ] Refine hero façade details where reference photography shows discrepancies.
- [ ] **Replace the placeholder heights.** Three of the fifteen block buildings carry no OSM height and render at a guessed 6.4 m (`Geo/tools/block_height_report.py`). Measure them off the photographs and correct the blockout. - [x] **Replace the placeholder heights.** **Resolved 2026-11-08 — they were never unknown buildings.** The three "guessed" ways are not building volumes at all: `1360755125` and `1360755126` are `building=roof` at `layer=1` (elevated canopies) and `1417995292` is `amenity=shelter` / `shelter_type=public_transport` (a transit shelter). The report defaulted them to "2 levels × 3.2 m" and produced the bogus 6.4 m. `Geo/tools/block_height_report.py` now classifies roof/shelter explicitly and reports **0 guessed heights**: all 15 block volumes have a real `height` tag, a `building:levels` derivation, or are correctly excluded as non-building volumes. No photograph measurement was needed, and no frontage building was affected.
- [ ] Resolve the remaining unknown on the hero: the colonnade currently occupies about 45 % of the height against roughly 50–60 % in the photographs. Fix the vertical proportions. - [ ] Resolve the remaining unknown on the hero: the colonnade currently occupies about 45 % of the height against roughly 50–60 % in the photographs. Fix the vertical proportions. **Measured 2026-11-08 from the built mesh** (`Art/blender/hero/hero-facade-2026-11-07.blend`): envelope 0–14.50 m; stylobate 0–2.00; columns 2.00–7.95; capitals 7.95–8.90; entablature 8.90–12.30; parapet 12.30–14.50. So the colonnade (columns + capitals) is **6.90 m = 47.6 %** of the height — inside the range only at the low end. Bringing it to ~55 % needs ~+1.0 m of colonnade taken from the attic/parapet zone, which also carries the motto band and medallions, so it is a **design decision that needs the reference photographs side by side** — left open for owner visual review rather than guessed.
- [ ] Tag `v0.1.2` and write `CHANGELOG.md` entry; then remove this section. - [ ] Tag `v0.1.2` and write `CHANGELOG.md` entry; then remove this section.
## 0.1.3 — Packaging and setup fixes (patch) ## 0.1.3 — Packaging and setup fixes (patch)
+18 -1
View File
@@ -57,4 +57,21 @@ Geographic tools live in [`Geo/tools/`](../Geo/README.md) and the Blender-side b
| Command | Purpose | | Command | Purpose |
| --- | --- | | --- | --- |
| `UnrealEditor <uproject> /Game/Hollywood/DemoMap -game -qaselftest` | Headless behaviour check: drives forward input for 2 s, then resets and pauses. Logs `GigLine QA: ... PASS/FAIL` lines and a final `RESULT`. Runs only with `-qaselftest`, so normal play is unaffected. | | `UnrealEditor <uproject> /Game/Hollywood/DemoMap -game -qaselftest` | Headless behaviour check: drives forward input for 2 s, then resets and pauses. Logs `GigLine QA: ... PASS/FAIL` lines and a final `RESULT`. Runs only with `-qaselftest`, so normal play is unaffected. |
| `GigLine SIM` / `GigLine GROUND` log lines | Emitted every second by `AGigLineCharacter` (TEMP QA): position, fall state, movement mode, and what the ground probe hits underneath. Remove before release. | | `-qaselftest` phases | Walk, reset, pause/unpause, a 16-row seam transect (0 holes) and a ground-referenced wall probe (0/35), plus a frame-time capture writing `Saved/Profiling/perf-walk-frametimes.csv` and logging `GigLine PERF:`. |
| `gigline.SetupLogs` cvar | Spawn/input setup logging, **off by default** (`DefaultEngine.ini [ConsoleVariables]`); enable at runtime with `gigline.SetupLogs 1`. |
## Input injection
| Tool | Purpose |
| --- | --- |
| `input_emulation.py` | Pure-stdlib `ctypes`/`fcntl` driver that creates a virtual keyboard+mouse on `/dev/uinput`. The desktop treats it as real hardware, so it proves whether OS input actually reaches the game. Needs `/dev/uinput` writable (ACL on this host). |
| `input_test.py` | Injects a scripted sequence into the focused window: `--sequence move:2,look:3,jump:1,reset:1,pause:1,click`. |
## Tests
```bash
python3 -m pip install -r requirements-tools.txt
python3 -m unittest discover -s Tools/tests -t Tools -p 'test_*.py'
```
Stdlib `unittest`, no Blender/Unreal process. See [`tests/README.md`](tests/README.md): test the pure parts (encoding, classification, maths, parsing); if logic is buried in `main()`, extract it into a function so it can be tested.
+26
View File
@@ -0,0 +1,26 @@
# Tooling tests
Standard-library `unittest` tests for the host-side tools in `Tools/`. They never start Blender, Unreal, or the game; anything that needs those processes is verified by the tool itself and recorded in `Docs/release-checks.md` instead.
Run from the repository root:
```bash
python3 -m pip install -r requirements-tools.txt
python3 -m unittest discover -s Tools/tests -t Tools -p 'test_*.py'
```
The `-t Tools` top-level directory is required: the tests import the tool modules by their plain names (`import input_emulation`), which is why `Tools/tests/` carries an `__init__.py`.
Individual test:
```bash
python3 -m unittest Tools.tests.test_input_emulation -v
```
## What belongs here
Test the **pure** parts — argument parsing, key/coordinate encoding, geometry maths, licence gating, report formatting, decision classification. Network, Blender and Unreal calls stay out: a test that needs a paid API, a display, or an engine process is a manual check, not a unit test.
When a tool's logic is buried in `main()`, extract it into a function so it can be tested directly. That is a tooling improvement, not test scaffolding.
Geo (`Geo/tools/`) and Blender (`Art/blender/*/`) builders are tested the same way once their pure parts are extracted; note the module path in the test.
+1
View File
@@ -0,0 +1 @@
"""Package marker so `python3 -m unittest Tools.tests.<module>` resolves."""
+59
View File
@@ -0,0 +1,59 @@
"""Tests for Geo/tools/block_height_report.py::classify_height.
The Geo tools live outside `Tools/`, so the test adds `Geo/tools` to `sys.path`
before importing the module by name.
"""
import os
import sys
import unittest
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
sys.path.insert(0, os.path.join(ROOT, "Geo", "tools"))
import block_height_report as bhr # noqa: E402
class ClassifyHeightTest(unittest.TestCase):
def test_height_tag_wins(self):
height, source = bhr.classify_height({"building": "yes", "height": "18.1"})
self.assertEqual(height, 18.1)
self.assertEqual(source, "height tag")
def test_height_tag_beats_levels(self):
height, _ = bhr.classify_height(
{"building": "yes", "height": "12.5", "building:levels": "4"})
self.assertEqual(height, 12.5)
def test_levels_derive_height(self):
height, source = bhr.classify_height(
{"building": "yes", "building:levels": "4"}, level_height=3.2)
self.assertEqual(height, 12.8)
self.assertIn("levels x", source)
def test_roof_is_not_a_building_volume(self):
# building=roof at layer=1 is an elevated canopy, not a two-storey block.
height, source = bhr.classify_height({"building": "roof", "layer": "1"})
self.assertEqual(height, 0.0)
self.assertIn("not a building volume (roof)", source)
def test_canopy_is_not_a_building_volume(self):
# building=canopy at layer=1, found by the Mapbox cross-check.
height, source = bhr.classify_height({"building": "canopy", "layer": "1"})
self.assertEqual(height, 0.0)
self.assertIn("not a building volume (canopy)", source)
def test_transit_shelter_is_not_a_building_volume(self):
height, source = bhr.classify_height(
{"building": "yes", "amenity": "shelter", "shelter_type": "public_transport"})
self.assertEqual(height, 0.0)
self.assertIn("not a building volume (shelter)", source)
def test_missing_data_falls_back_to_a_flagged_guess(self):
height, source = bhr.classify_height({"building": "yes"})
self.assertEqual(height, 2 * bhr.LEVEL_HEIGHT)
self.assertTrue(source.startswith("GUESS"))
if __name__ == "__main__":
unittest.main()
+57
View File
@@ -0,0 +1,57 @@
"""Tests for Tools/input_emulation.py — the pure ioctl/key-code parts.
The device path itself needs /dev/uinput, so only the encoding helpers are
covered here; the end-to-end injection is verified by running the tool (see
Docs/release-checks.md and MEMORY.md).
"""
import unittest
import input_emulation as ie
class IocEncodingTest(unittest.TestCase):
"""_IOC must match the Linux ioctl bit layout."""
def test_layout_constants(self):
self.assertEqual(ie._IOC_NRSHIFT, 0)
self.assertEqual(ie._IOC_TYPESHIFT, 8)
self.assertEqual(ie._IOC_SIZESHIFT, 16)
self.assertEqual(ie._IOC_DIRSHIFT, 30)
def test_known_values(self):
# UI_SET_EVBIT = _IOW('U', 100, int) -> 0x40045564 on Linux.
self.assertEqual(ie.UI_SET_EVBIT, 0x40045564)
# UI_DEV_CREATE = _IO('U', 1) -> 0x5501.
self.assertEqual(ie.UI_DEV_CREATE, 0x5501)
def test_type_accepts_char_or_int(self):
self.assertEqual(ie._ioc(1, 4, "U", 100), ie._ioc(1, 4, ord("U"), 100))
class KeyCodeTest(unittest.TestCase):
"""KEYCODES must hold the Linux input-event codes the demo uses."""
EXPECTED = {
"w": 17, "s": 31, "a": 30, "d": 32,
"space": 57, "r": 19, "esc": 1,
}
def test_demo_controls(self):
for name, code in self.EXPECTED.items():
self.assertEqual(ie.KEYCODES[name], code, f"{name} code")
def test_lookup_is_case_insensitive(self):
dev = ie.InputDevice.__new__(ie.InputDevice) # no device opened
self.assertEqual(dev._key_code("W"), 17)
self.assertEqual(dev._key_code("SPACE"), 57)
self.assertEqual(dev._key_code(19), 19) # a raw code passes through
def test_unknown_key_raises(self):
dev = ie.InputDevice.__new__(ie.InputDevice)
with self.assertRaises(ie.UInputError):
dev._key_code("not-a-key")
if __name__ == "__main__":
unittest.main()
+117
View File
@@ -0,0 +1,117 @@
"""Tests for Geo/tools/mapbox_buildings.py — the pure merge logic.
No network calls here: Tilequery responses are represented as the small dicts
the API returns, and only the selection/comparison/formatting helpers are
exercised. The live call is verified by running the tool.
"""
import json
import os
import sys
import unittest
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
sys.path.insert(0, os.path.join(ROOT, "Geo", "tools"))
import mapbox_buildings as mb # noqa: E402
def feature(distance, height=None, min_height=0, type_="building", layer="building"):
props = {"tilequery": {"distance": distance, "layer": layer}}
if height is not None:
props["height"] = height
props["min_height"] = min_height
props["type"] = type_
return {"properties": props}
class NearestBuildingTest(unittest.TestCase):
def test_picks_the_closest_building(self):
payload = {"features": [feature(6.0, 10.0), feature(1.0, 20.0)]}
best = mb.nearest_building(payload, max_distance=8.0)
self.assertEqual(best["height"], 20.0)
def test_ignores_hits_beyond_max_distance(self):
payload = {"features": [feature(20.0, 10.0)]}
self.assertIsNone(mb.nearest_building(payload, max_distance=8.0))
def test_ignores_other_layers(self):
payload = {"features": [feature(0.0, 10.0, layer="road")]}
self.assertIsNone(mb.nearest_building(payload, max_distance=8.0))
def test_no_features(self):
self.assertIsNone(mb.nearest_building({"features": []}, max_distance=8.0))
class CompareTest(unittest.TestCase):
def test_match_within_half_a_metre(self):
self.assertEqual(mb.compare(14.5, "height tag", 14.2)[0], "match")
def test_close_within_two_metres(self):
verdict, delta = mb.compare(38.4, "levels x 3.2 m", 37.2)
self.assertEqual(verdict, "close")
self.assertAlmostEqual(delta, -1.2)
def test_disagree_beyond_two_metres(self):
verdict, delta = mb.compare(6.4, "GUESS: default 2 levels x 3.2 m", 3.0)
self.assertEqual(verdict, "disagree")
self.assertAlmostEqual(delta, -3.4)
def test_mapbox_only_when_we_have_nothing(self):
self.assertEqual(mb.compare(0.0, "height tag", 12.0)[0], "mapbox-only")
def test_ours_only_when_mapbox_returns_nothing(self):
self.assertEqual(mb.compare(12.0, "height tag", None)[0], "ours-only")
def test_non_building_is_excluded_from_comparison(self):
verdict, delta = mb.compare(0.0, "not a building volume (roof)", 3.0)
self.assertEqual(verdict, "non-building")
self.assertIsNone(delta)
def test_canopy_is_non_building(self):
verdict, _ = mb.compare(0.0, "not a building volume (canopy)", 3.0)
self.assertEqual(verdict, "non-building")
class CentroidTest(unittest.TestCase):
def test_centroid_of_a_square(self):
lat, lon = mb.centroid_lat_lon(
[(34.0, -118.0), (34.0, -117.0), (35.0, -117.0), (35.0, -118.0)])
self.assertAlmostEqual(lat, 34.5)
self.assertAlmostEqual(lon, -117.5)
class RenderTest(unittest.TestCase):
ARGS = type("A", (), {"radius": 25, "max_distance": 8.0, "only_diffs": False})()
def test_disagreements_sort_first(self):
rows = [
{"verdict": "match", "delta_m": 0.1, "way": 1, "name": "", "kind": "yes",
"osm_height_m": 10.0, "osm_source": "height tag", "mapbox_height_m": 10.1,
"mapbox_min_height_m": 0, "mapbox_type": "building",
"mapbox_distance_m": 0.0, "cached": True, "lat": 0, "lon": 0},
{"verdict": "disagree", "delta_m": -3.4, "way": 2, "name": "", "kind": "yes",
"osm_height_m": 6.4, "osm_source": "GUESS", "mapbox_height_m": 3.0,
"mapbox_min_height_m": 0, "mapbox_type": "building",
"mapbox_distance_m": 0.0, "cached": True, "lat": 0, "lon": 0},
]
text = mb.render_markdown(rows, self.ARGS)
self.assertLess(text.index("way 2"), text.index("way 1"))
def test_never_contains_a_token(self):
rows = []
text = mb.render_markdown(rows, self.ARGS)
self.assertNotIn("access_token", text)
self.assertNotIn("sk.", text)
class TokenTest(unittest.TestCase):
def test_explicit_token_wins(self):
self.assertEqual(mb.load_token("abc123"), "abc123")
def test_token_is_stripped(self):
self.assertEqual(mb.load_token(" abc123\n"), "abc123")
if __name__ == "__main__":
unittest.main()
+1
View File
@@ -0,0 +1 @@
requests>=2.28