zwitschi/gig-line
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(tools): group the sixty scripts by what they talk to

Browse Source 
One directory with sixty scripts stopped being navigable, and the grouping is also the sandbox boundary:

  Tools/unreal/    commandlets and the two map builders, with ue_common.py and ue_placement.py as the
                   shared imports (they need the engine)
  Tools/blender/   builders, exporters, the FBX preset, the MPFB character pipeline, blender_screenshots
                   (they need the Blender Flatpak)
  Tools/review/    the visual loop: render_screenshots.sh, judge_screenshot.py, hero_compare.py,
                   build_comparison_md.py, e2e_asset_check.py, hero_iterate.sh, visual_loop.py,
                   visual_targets.json
  Tools/gen/       image and texture generation: the ComfyUI driver, deterministic texture writers,
                   reference fetching
  Tools/input/     OS-level input injection
  root             _common.py, check-lfs.sh, addon_compat_probe.py, sibling_asset_audit.py, tests/

Every moved script now resolves the repository root as ../.. and puts the tools root plus its siblings on
sys.path, so 'import ue_common' still works from unreal/. The test suite bootstrap does the same for the five
folders. render_screenshots.sh takes the folder as a third argument to run_cmdlet, and the e2e driver,
hero_iterate.sh and the comparison generator were repointed.

51 README rows now link to their new location, the README gains a Layout section naming each folder's
sandbox, and the architecture notes and Blender README follow the moves.

Verified: 108 tests pass, prettier is clean, judge_screenshot.py and hero_compare.py run from their new
homes, and the e2e run was aborted mid-way (it had been rebuilding an empty DemoMap for 15 minutes) and is
being rerun against the new layout.
This commit is contained in:
zwitschi committed 2026-11-10 12:40:30 +01:00
1 parent f681d83736
commit 4d27109668
168 files changed
+799 -407
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Docs/Architecture/04_solution_strategy.md
+1 -1
View File
@@ -30,7 +30,7 @@ All modelling and geographic processing live outside the game project in `Art/`,
## Quality achievement
- **Realism first**: reference photography and measured data drive every façade decision; renders are compared against the photographs with `Tools/visual_loop.py`, and a façade is built from the shared 4.96 m module kit so the street wall reads consistently. Materials are authored in Unreal with world-aligned generated textures, and lighting is fixed daytime so a shot is comparable between runs.
- **Realism first**: reference photography and measured data drive every façade decision; renders are compared against the photographs with `Tools/review/visual_loop.py`, and a façade is built from the shared 4.96 m module kit so the street wall reads consistently. Materials are authored in Unreal with world-aligned generated textures, and lighting is fixed daytime so a shot is comparable between runs.
- **Performance as a budget, not a driver**: dynamic-only lighting (no build step, no static shadow maps), world-space-noise materials (cheap procedural variation), minimal overdraw for street-level geometry, and a 60 FPS at 1440p floor that every release must meet. It closes the quality list because it constrains how much realism fits, not what the block should look like.
- **Walkability**: complex-as-simple collision on terrain (C++ helper forces the flag on the asset), block-all collision on every site mesh, swept transect QA verifies zero raycast holes.
Docs/Architecture/05_building_block_view.md
+1 -1
View File
@@ -49,7 +49,7 @@ Static meshes and materials in `/Game/Hollywood/Site/`:
| `SM_HeroFacade` | El Capitan Entertainment Centre detailed façade |
| `SM_ElCapitanTheatre`, `SM_Budget_6822`, `SM_Hollywood_6806`, `SM_Hollywood_6902`, `SM_Hollywood_6922`, `SM_LeeDrug_6800` | South block frontage (6 buildings) |
| `SM_Madame_Tussauds`, `SM_TCL_Chinese_Theatre` | North-side buildings (within 50 m) |
| `SM_Retail_LeeDrug`, `SM_Retail_6806` | Retail trim over 6800 and 6806–6820, modelled from their own photograph (`Tools/retail_frontage_build.py`, staged as `RetailTrim_*`). `6822 Budget` stays a bare volume: no photograph exists. |
| `SM_Retail_LeeDrug`, `SM_Retail_6806` | Retail trim over 6800 and 6806–6820, modelled from their own photograph (`Tools/blender/retail_frontage_build.py`, staged as `RetailTrim_*`). `6822 Budget` stays a bare volume: no photograph exists. |
### Art Pipeline
Docs/Architecture/06_runtime_view.md
+3 -3
View File
@@ -18,7 +18,7 @@
## Screenshot capture
1. `Tools/render_screenshots.sh` compiles editor, builds `ReviewMap` from `Tools/ue_build_review_map.py`, launches game in `-game` mode.
1. `Tools/review/render_screenshots.sh` compiles editor, builds `ReviewMap` from `Tools/unreal/ue_build_review_map.py`, launches game in `-game` mode.
2. `AGigLineShotActor` iterates the camera list, sets view, calls `HighResShot 1280x720`, waits for capture.
@@ -48,6 +48,6 @@
1. Blender headless (`Tools/export_*.py`) reads `.blend` files, applies metre→centimetre conversion, exports FBX with `ue_metric_v1` preset.
2. Unreal headless (`Tools/ue_import_*.py`) imports FBX via Interchange, places meshes at geographic coordinates, sets collision profiles.
2. Unreal headless (`Tools/unreal/ue_import_*.py`) imports FBX via Interchange, places meshes at geographic coordinates, sets collision profiles.
3. `Tools/ue_build_demo_map.py` assembles `DemoMap`: creates/repositions actors, sets lighting, saves map and assets.
3. `Tools/unreal/ue_build_demo_map.py` assembles `DemoMap`: creates/repositions actors, sets lighting, saves map and assets.
Docs/Architecture/10_quality_requirements.md
+2 -2
View File
@@ -22,7 +22,7 @@ Ordered by priority: realism first, performance last. The order decides design t
| ID | Source/Stimulus | Response | Metric |
| --- | --- | --- | --- |
| QS-1 | Screenshot is compared against its reference photograph | The render reads as the same building | Similarity score and visual match confirmed by the owner; `Tools/visual_loop.py` reports the deltas |
| QS-1 | Screenshot is compared against its reference photograph | The render reads as the same building | Similarity score and visual match confirmed by the owner; `Tools/review/visual_loop.py` reports the deltas |
| QS-2 | A façade is placed from measured data | It sits on its verified footprint | World span probed against `Geo/verification/` within ~1 m |
| QS-3 | Player walks both sides of the demo | No fall-through, character stays at ground Z | 0 raycast holes in 84-sample transect |
| QS-4 | Player holds W for 2 s on flat road | Character moves forward | ≥ 200 cm displacement, Z constant within ±5 cm |
@@ -41,5 +41,5 @@ Ordered by priority: realism first, performance last. The order decides design t
| BR-1 | A building with a reference photograph has a review shot compared against it | Silhouette and proportion deltas recorded per building in `Screenshots/README.md`; owner approval per shot |
| BR-2 | Every building is either photo-matched or explicitly recorded as an unfounded blockout | List in `TODO.md`; no building in an in-between state |
| BR-3 | Every building has a collision proxy and at least one LOD | Count of buildings missing either = 0 |
| BR-4 | No surface carries a placeholder material | Material audit in `Tools/ue_audit_scene.py` reports 0 placeholder slots |
| BR-4 | No surface carries a placeholder material | Material audit in `Tools/unreal/ue_audit_scene.py` reports 0 placeholder slots |
| BR-5 | The realism pass does not break the floor | QS-9 still holds at 1440p, with the measured cost of the pass recorded |
Docs/build.md
+19 -19
View File
@@ -56,9 +56,9 @@ Both content-acceptance tests run headless and are the template for all asset im
```bash
# Static-mesh scale test (Blender export -> Unreal import -> measure)
# Blender: Tools/scale_test_export.py, Unreal: Tools/ue_scale_test_import.py
# Blender: Tools/blender/scale_test_export.py, Unreal: Tools/unreal/ue_scale_test_import.py
# MPFB character (generate -> rig -> bake -> export -> skeletal import)
# Blender: Tools/mpfb_character_export.py, Unreal: Tools/ue_character_import.py
# Blender: Tools/blender/mpfb_character_export.py, Unreal: Tools/unreal/ue_character_import.py
# Unreal side of any test:
./Engine/Binaries/Linux/UnrealEditor-Cmd \
@@ -78,22 +78,22 @@ The hero block and the demo level are built by a fixed sequence. Each step write
| Street | `Art/blender/street/street_mesh.py` | `Art/blender/street/street-*.blend` |
| Site assembly | `Art/blender/site/site_assembly.py` | `Art/blender/site/site-*.blend` |
| Site export | `Art/blender/site/export_site_fbx.py` | seven FBXs, one per mesh (terrain, road, both sidewalks, both kerbs, crossing) |
| Hero façade | `Tools/hero_facade_build.py` | `Art/blender/hero/hero-facade-*.blend` + JSON part report |
| Block frontage | `Tools/export_block_frontage.py` | six frontage FBXs from the blockout |
| North buildings | `Tools/export_north_buildings.py` | TCL and Tussaud's FBXs from the blockout |
| Site import | `Tools/ue_import_site.py` | `/Game/Hollywood/Site/*` |
| Frontage import | `Tools/ue_import_block_frontage.py` | `/Game/Hollywood/Frontage/*` |
| North import | `Tools/ue_import_north_buildings.py` | `/Game/Hollywood/North/*` |
| Hero import | `Tools/ue_hero_facade_import.py` | `/Game/Hollywood/ElCapitan/SM_HeroFacade` |
| Demo map | `Tools/ue_build_demo_map.py` | `/Game/Hollywood/DemoMap` |
| Façade kit assembly | `Tools/ue_apply_facade_kit.py` | kit actors in `DemoMap` |
| Theatre trim | `Tools/ue_stage_theatre_trim.py` | marquee + blade sign actors |
| Scene audit | `Tools/ue_audit_scene.py` | materials, placement spans, labels (gate) |
| Review shots | `Tools/ue_build_review_map.py` + `Tools/render_screenshots.sh` | `Screenshots/` |
| Hero façade | `Tools/blender/hero_facade_build.py` | `Art/blender/hero/hero-facade-*.blend` + JSON part report |
| Block frontage | `Tools/blender/export_block_frontage.py` | six frontage FBXs from the blockout |
| North buildings | `Tools/blender/export_north_buildings.py` | TCL and Tussaud's FBXs from the blockout |
| Site import | `Tools/unreal/ue_import_site.py` | `/Game/Hollywood/Site/*` |
| Frontage import | `Tools/unreal/ue_import_block_frontage.py` | `/Game/Hollywood/Frontage/*` |
| North import | `Tools/unreal/ue_import_north_buildings.py` | `/Game/Hollywood/North/*` |
| Hero import | `Tools/unreal/ue_hero_facade_import.py` | `/Game/Hollywood/ElCapitan/SM_HeroFacade` |
| Demo map | `Tools/unreal/ue_build_demo_map.py` | `/Game/Hollywood/DemoMap` |
| Façade kit assembly | `Tools/unreal/ue_apply_facade_kit.py` | kit actors in `DemoMap` |
| Theatre trim | `Tools/unreal/ue_stage_theatre_trim.py` | marquee + blade sign actors |
| Scene audit | `Tools/unreal/ue_audit_scene.py` | materials, placement spans, labels (gate) |
| Review shots | `Tools/unreal/ue_build_review_map.py` + `Tools/review/render_screenshots.sh` | `Screenshots/` |
**Order matters.** `ue_build_demo_map.py` rebuilds the site materials, so any actor staged by another tool must be (re)staged _after_ it, or it keeps a reference to a material that no longer exists — that is how 604 actors once ended up untethered from their materials while the map only looked slightly flat. `Tools/render_screenshots.sh` runs the full chain in the right order and gates on `ue_audit_scene.py`.
**Order matters.** `ue_build_demo_map.py` rebuilds the site materials, so any actor staged by another tool must be (re)staged _after_ it, or it keeps a reference to a material that no longer exists — that is how 604 actors once ended up untethered from their materials while the map only looked slightly flat. `Tools/review/render_screenshots.sh` runs the full chain in the right order and gates on `ue_audit_scene.py`.
Every Blender exporter calls [`Tools/fbx_export_preset.py`](../Tools/fbx_export_preset.py) (`ue_metric_v1` / `ue_character_v1`) instead of repeating the FBX argument block — one place for the scale conversion, the axis mapping and `mesh_smooth_type="FACE"` (without which every Unreal import warns about missing smoothing groups and round geometry arrives faceted).
Every Blender exporter calls [`Tools/blender/fbx_export_preset.py`](../Tools/blender/fbx_export_preset.py) (`ue_metric_v1` / `ue_character_v1`) instead of repeating the FBX argument block — one place for the scale conversion, the axis mapping and `mesh_smooth_type="FACE"` (without which every Unreal import warns about missing smoothing groups and round geometry arrives faceted).
Two rules that the pipeline enforces rather than documents:
@@ -103,7 +103,7 @@ Two rules that the pipeline enforces rather than documents:
## Known engine quirks (2026-11-07)
- One handled `PSOPrecache` ensure fires at packaged-build exit only; not blocking.
- Interchange 5.8 does not create material/texture assets for skeletal FBX imports — `Tools/ue_character_import.py` finishes that chain explicitly.
- Interchange 5.8 does not create material/texture assets for skeletal FBX imports — `Tools/unreal/ue_character_import.py` finishes that chain explicitly.
- Android Turnkey status errors and datarouter DNS warnings in logs are benign.
## Packaged Linux build
@@ -129,13 +129,13 @@ cd /run/media/system/DATA/Builds/GigLineQA/Linux
| `Add-on not loaded: "bl_ext.blender_org.mpfb"` | `mpfb_character_export.py` needs the MPFB add-on visible to Blender | export `BLENDER_USER_SCRIPTS` (see the invocation in [../Art/blender/README.md](../Art/blender/README.md)) |
| The packaged run exits after a couple of seconds during QA | `GigLineShotActor` runs its screenshot list and quits (`bQuitWhenDone`) | it skips its rounds when `-qaselftest` is on the command line — keep that guard |
| Headless `-nullrhi` traces report no hits for everything | the null RHI has no physics scene | verify traces in the **packaged** run (`-qaselftest`), never in a commandlet |
| Import logs "No smoothing group information" for every mesh | `mesh_smooth_type="OFF"` in the FBX export | use `Tools/fbx_export_preset.py` (`FACE`) |
| Import logs "No smoothing group information" for every mesh | `mesh_smooth_type="OFF"` in the FBX export | use `Tools/blender/fbx_export_preset.py` (`FACE`) |
## PIE input test (compare editor vs packaged input intake)
Use to isolate whether the "no controls respond" bug is the packaged standalone's input intake or the Enhanced Input wiring generally.
- `Tools/run_pie.sh` opens the editor on `/Game/Hollywood/DemoMap`. Press the green **Play** (PIE) button and walk / mouse-look.
- `Tools/input/run_pie.sh` opens the editor on `/Game/Hollywood/DemoMap`. Press the green **Play** (PIE) button and walk / mouse-look.
- Interpretation:
- PIE **works** -> the Enhanced Input wiring is fine and the standalone packaged build's SDL input intake is the environment-dependent bug.
- PIE **fails** too -> the Enhanced Input setup itself is broken; rebuild the input config / character from scratch.
Docs/controls.md
+3 -3
View File
@@ -19,7 +19,7 @@ Keyboard and mouse. The same list is shown in-game, bottom-right of the frame (`
- Movement is relative to where you are looking (controller yaw), so the capsule stays upright on the sloped street.
- The mouse is captured on launch, so looking works without clicking first. `Escape` toggles the capture as part of pause.
- Bindings are Enhanced Input actions in `/Game/Input` built by `Tools/ue_create_input_assets.py`; the mapping context is `IMC_Default`.
- Bindings are Enhanced Input actions in `/Game/Input` built by `Tools/unreal/ue_create_input_assets.py`; the mapping context is `IMC_Default`.
## Changes since `0.1.0`
@@ -48,10 +48,10 @@ The walking shot is driven per shot rather than by the switch: the shot actor en
## Testing input without a physical keyboard
`Tools/input_test.py` injects OS-level keyboard/mouse events through a virtual `/dev/uinput` device (see `Tools/input_emulation.py`), e.g.:
`Tools/input/input_test.py` injects OS-level keyboard/mouse events through a virtual `/dev/uinput` device (see `Tools/input/input_emulation.py`), e.g.:
```bash
python3 Tools/input_test.py --sequence move:2,look:3
python3 Tools/input/input_test.py --sequence move:2,look:3
```
It drives the focused game window exactly like real hardware, which is how the controls were regression-tested on the packaged build.
Docs/release-checks.md
+3 -3
View File
@@ -50,7 +50,7 @@ Measured at 1440p on the development workstation: Bazzite, Ryzen 7 7700X, 64 GB
## `0.1.0` — Placement vs references — record (2026-11-08)
Checked against the reference photography (`Art/reference/`) and the Geo measurements (`Geo/verification/*`). Build under test: current `main` content, DemoMap rendered via `Tools/render_screenshots.sh` (2026-11-08, Vulkan SM5, 1920×1080); engine positions probed from the DemoMap (commandlet, 2026-11-08).
Checked against the reference photography (`Art/reference/`) and the Geo measurements (`Geo/verification/*`). Build under test: current `main` content, DemoMap rendered via `Tools/review/render_screenshots.sh` (2026-11-08, Vulkan SM5, 1920×1080); engine positions probed from the DemoMap (commandlet, 2026-11-08).
### Placement and scale (in-engine ↔ reference)
@@ -82,7 +82,7 @@ Checked against the reference photography (`Art/reference/`) and the Geo measure
- **Crossing ramps are kerb flushes, not graded ramps** — the `0.1.1` Crossing ramp gradient item.
- Site identity (the neoclassical hero with motto band) matches the 2009 façade photograph and the 1922 Masonic elevation within blockout fidelity.
Pre-check method: `Tools/judge_screenshot.py` (vision model) flagged the simplified flat surfaces as "unexpected" — those flags are the deliberate deviations above, not placement errors. Final verdict is by human review of the shots in `Screenshots/demo/`.
Pre-check method: `Tools/review/judge_screenshot.py` (vision model) flagged the simplified flat surfaces as "unexpected" — those flags are the deliberate deviations above, not placement errors. Final verdict is by human review of the shots in `Screenshots/demo/`.
## `0.1.0` — Collision and packaging — audit note (2026-11-08)
@@ -90,7 +90,7 @@ Run on the packaged build (`GigLineQA`, Development, Vulkan SM5) with `-qaselfte
- **Transect (holes/gaps):** 1456 samples over 16 rows — lane centres **plus the kerb/road/sidewalk seam Ys** (−1382, −1053, +718, +1047 and neighbours) and X −20000..−2000. Result **0 holes**. (The old 84-sample check covered only 3 lane centres; the seam rows are what this check adds.)
- **Invisible-wall probe:** horizontal traces 30 m long, grounded at 160 cm above the local ground, along the road and both seam rows. Result **0/35 blocking hits inside 21 m** — no invisible walls or misplaced blockers in the street envelope.
- **Facade occlusion probe:** from the road centre, 40 m traces sideways at walking height toward each frontage, required only where the map stages a building (the spans are listed in `AGigLineCharacter` and mirror `Tools/ue_build_demo_map.py`). Result **0/9 directions without a blocking wall** — the player cannot walk through the hero or the frontage. This probe found the hero façade had **no simple collision** at all (complex-only, so traces and capsule sweeps passed through it); it is now `UseComplexAsSimple`. Auto-generated convex hulls were tried first and rejected: on a 95-part, 26 m façade they bulged into the street (the wall probe went 0/35 → 2/35). (First pass used a fixed Z 160 start; the street is at Z ~366, so the probe started inside the earth and reported every row as a dist-0 blocker. Grounding the probe fixed that measurement bug.)
- **Facade occlusion probe:** from the road centre, 40 m traces sideways at walking height toward each frontage, required only where the map stages a building (the spans are listed in `AGigLineCharacter` and mirror `Tools/unreal/ue_build_demo_map.py`). Result **0/9 directions without a blocking wall** — the player cannot walk through the hero or the frontage. This probe found the hero façade had **no simple collision** at all (complex-only, so traces and capsule sweeps passed through it); it is now `UseComplexAsSimple`. Auto-generated convex hulls were tried first and rejected: on a 95-part, 26 m façade they bulged into the street (the wall probe went 0/35 → 2/35). (First pass used a fixed Z 160 start; the street is at Z ~366, so the probe started inside the earth and reported every row as a dist-0 blocker. Grounding the probe fixed that measurement bug.)
- **Walk / reset / pause:** 761 cm in 2 s (Z steady 456), reset 0 cm, pause `IsGamePaused()=1`, unpause 0 — all PASS.
- Packaging: the packaged build launches, reaches gameplay, and the QA run quits cleanly. The `GigLineShotActor` screenshot rounds are suppressed under `-qaselftest` so they cannot quit the run before the audit phases.
- Caveat recorded: ground Z extremes (−12 m / 3000 cm) are trace-start/terrain-edge artifacts of the widened extent, not walkable holes — the walkable corridor is continuous (walk + transect both PASS).
Docs/setup.md
+1 -1
View File
@@ -91,7 +91,7 @@ Two tools talk to paid services and read their keys from the gitignored `.env`:
| Variable | Used by | Needed for |
| --- | --- | --- |
| `OPENROUTER_API_KEY` | `Tools/judge_screenshot.py`, `Tools/facade_detail_concepts.py` | vision pre-check and AI concept plates |
| `OPENROUTER_API_KEY` | `Tools/review/judge_screenshot.py`, `Tools/gen/facade_detail_concepts.py` | vision pre-check and AI concept plates |
| `MAPBOX_ACCESS_TOKEN` | `Geo/tools/mapbox_buildings.py` | the second building-height source |
Nothing in the build, the tests or the release path needs either key. Never commit `.env`; `git check-ignore .env` should succeed.