diff --git a/MEMORY.md b/MEMORY.md index 4395999..48d9a9e 100644 --- a/MEMORY.md +++ b/MEMORY.md @@ -4,6 +4,15 @@ Session memory for durable decisions, state, and gotchas. Keep concise, remove o ## Decisions +- **Gaussian splatting / photo-derived maps (unversioned local track):** expanded in + `TODO.md` "Gaussian splatting evaluation". Accepted photo channels are team captures, + permissioned user/volunteer photos and clearly licensed downloads; every set needs a + provenance/consent sidecar. Sparse web photos are reference/training supplements, not a splat + source. Five enhancement paths are ranked (reference/QA, texture bake, mesh proxy, distant + backdrop, single-facade user-photo supplement). Splats stay offline/development-only until the + ADR proves licence, privacy, Godot Compatibility/Web cost and mesh-based collision; no runtime + splat dependency in 0.10.0. + - **NPC model variety decision (2026-11-03, ADR-0015):** CC0 is acceptable for shipped assets; CC-BY only with attribution in `CREDITS.md`; no service assets with redistribution restrictions. The **procedural generator is the primary path**. Four crowd-variety specs were diff --git a/TODO.md b/TODO.md index be5f34e..e803d51 100644 --- a/TODO.md +++ b/TODO.md @@ -332,16 +332,92 @@ facades and AI-assisted composition. This is an unversioned development track, n - [ ] Integrate the result behind `WorldBuilder.VENUE_SPECS` with colliders, navigation obstacles, occlusion and the existing day/night materials. -### Gaussian splatting evaluation +### Gaussian splatting evaluation: photo intake and provenance -- [ ] Evaluate Gaussian splatting as a development tool: local capture, splat reconstruction, - Blender import and comparison against the concept/reference set. -- [ ] Test whether a splat can be used as a photoreal reference or backdrop, and measure file - size, render/import cost and Godot Compatibility support. -- [ ] Check capture permissions, privacy (people, plates, storefronts) and the licence of every - capture before using it in the repository. -- [ ] Record a go/no-go decision in an ADR. Keep splats development-only unless runtime - performance, collision/navigation and provenance are all proven. +Gaussian splatting is evaluated as a photo-driven pipeline: licensed or permissioned photos +become a reference, texture source, mesh proxy or distant backdrop. It is an unversioned local +track with no runtime dependency or release gate until the ADR closes it, and it extends the +reference-image licence and privacy gates above rather than replacing them. + +- [ ] Define the accepted photo channels: project/team captures, volunteer or user-supplied + photos with written permission, and downloaded images from clearly licensed archives + (Wikimedia Commons, Openverse, Mapillary and similar). Record which channel produced + each reconstruction. +- [ ] Write a one-page capture brief for user-supplied photos: overlapping arc or orbit around + the facade, 60–80% view overlap, 12 MP+ originals, fixed exposure/white balance, HDR and + filters off, no motion blur, no people or plates in frame where avoidable, overcast or + even light, and the facade height/doors/steps included in several frames. +- [ ] Build an intake checklist and provenance sidecar per photo set: contributor, consent or + property permission, licence and attribution text, source URL, capture date, camera/lens, + location, faces/plates/interiors present, redactions applied, file hashes and whether the + set is shipped or development-only. +- [ ] Set the licence gate: own photos need an explicit contributor licence; CC0/CC BY are + preferred; CC BY-SA triggers a share-alike review for derived textures and meshes; reject + Street View, social-media scrapes, unlicensed commercial imagery and AI-upscaled copies of + unknown originals. Attribution must be feasible in `CREDITS.md` before use. +- [ ] Set the privacy gate: blur or remove faces, minors, licence plates, screens, private + interiors and anything behind a closed door; obtain property permission where required; + keep a redacted copy as the reconstruction input and never commit the originals. +- [ ] Treat downloaded photo sets as sparse and of unknown intrinsics. Use them for reference, + palette and texture study, or as extra training views; do not expect a usable splat from + 10–30 casual web photos of a facade. +- [ ] Keep photos, masks, COLMAP databases, checkpoints and splats outside git; commit only + derived, licensed, provenance-stamped assets that pass the review gate. + +### Gaussian splatting: reconstruction and alignment + +- [ ] Pin a reproducible offline toolchain (for example COLMAP or GLOMAP for camera poses plus + gsplat, OpenSplat, Nerfstudio or Brush for training), with exact versions, GPU/VRAM/disk + needs and known failure modes recorded in the sidecar. +- [ ] Reconstruct two quality tiers: a dense 100–400-photo arc/orbit capture and a sparse + 10–30-photo downloaded set. Record camera-registration rate, training time, splat count, + file size and held-out PSNR/SSIM/LPIPS for both. +- [ ] Mask sky, people, vehicles and moving foliage before training; verify the masks do not + erase facade edges, signage or window detail, and keep the redacted input variant. +- [ ] Align the splat to game coordinates from the OSM footprint plus surveyed points (door + width, column spacing, step rise, storefront height) and record alignment residuals in a + Blender scene containing both the splat and the current venue geometry. +- [ ] Judge quality from held-out views and a render from the game’s day/night camera poses, + not from training views alone; reject splats that only look correct from the capture path. +- [ ] Maintain a coverage map per block: which facades, corners and rooflines have usable views, + and which areas must fall back to the facade kit or stylized geometry. + +### Gaussian splatting: map enhancement paths + +- [ ] **Path A — Reference and QA (lowest risk):** render the splat from game camera poses and + compare it with the realism captures; derive dimensions, signage placement, facade rhythm + and material colours. No splat data reaches the runtime. +- [ ] **Path B — Texture derivation:** project splat renders onto the facade kit to produce + albedo, normal, roughness and AO candidates; de-light and bake them, then verify UV scale, + tiling and readability under the Compatibility renderer. +- [ ] **Path C — Mesh proxy:** extract a mesh from the splat or dense reconstruction, retopo and + dress it in Blender, assign LODs, and keep it inside the venue triangle/material budgets + with mesh-based colliders and navigation. +- [ ] **Path D — Distant backdrop:** test a baked panorama, impostor or billboard generated from + the splat for skyline and opposite-street depth where the player cannot walk; measure file + size, loading and visible pop at the venue boundary. +- [ ] **Path E — User-photo supplement:** let a small supplied photo set improve one facade or + landmark (a signage patch, colour reference or hero detail) without rebuilding the venue; + define the minimum evidence and the same licence/privacy/review gate as a full capture. +- [ ] For every path record quality gain versus effort, licence compatibility of the derived + output, and whether the result may ship or stays development-only. Prefer the cheapest + path that closes the visible gap. + +### Gaussian splatting: runtime evaluation and decision + +- [ ] Probe Godot 4.7.2 Compatibility and Web support for splat rendering in a throwaway scene: + addon or importer availability, shader/sort cost, memory, load time and fallback behaviour. + Do not add a splat dependency to the release project during evaluation. +- [ ] Set hard limits before any runtime experiment: maximum per-splat file size and total + memory, allowed frame-time delta on the 24-NPC budget scene, acceptable Web load time, and + no compute-shader requirement for the Compatibility/Web path. +- [ ] Keep collision and navigation mesh-based even if a splat is displayed; a splat must never + become the playable floor or the access/occlusion authority. +- [ ] Write the ADR with a per-path go/no-go, the accepted licence/provenance rules, whether each + result ships or stays development-only, rejected alternatives and the evidence bundle. +- [ ] Exit: one licensed photo capture measurably improves a venue through a documented path and + passes visual, performance and provenance review; or the approach is rejected with + reproducible findings and removed from the release path. ### QA and exit @@ -372,6 +448,11 @@ facades and AI-assisted composition. This is an unversioned development track, n - Godot version changes can alter lighting/imports; do not combine an engine baseline move with a content/feature milestone. - Signed macOS builds and legal/AI provenance are release blockers, not polish. +- Photo-derived map assets add a second provenance risk: downloaded or user-supplied photos can + carry unclear licences, identifiable people, plates and property restrictions. No photo or + splat-derived asset ships without a completed sidecar, consent/licence review and redaction. +- Gaussian-splat training is GPU/disk-heavy and the toolchain churns quickly; keep it offline, + version-pinned and outside CI, and never make a runtime splat renderer a release dependency. - Keep the de-escalation-first rules and rare-emergency design unchanged while adding progression, difficulty and feedback.