chore: keep reference photography local and out of the repository
Reference images are development-only material: licensing is not a gate for local private use, so they are simply never committed. Art/reference/ is now gitignored; the licensed status of anything shipped is still recorded in THIRD_PARTY_NOTICES.md. Adds Tools/commons_fetch.py, which fetches Wikimedia Commons imagery by search or exact title and writes a provenance sidecar, plus Docs/reference-notes-hollywood.md recording what each local set is evidence for.
This commit is contained in:
zwitschi committed
1 parent
52b0eba910
commit
7a1eb0be66
7 files changed
+264
-27
No files matched your search
+6
-2
@@ -37,5 +37,9 @@ venv/
|
||||
Thumbs.db
|
||||
*~
|
||||
|
||||
# Environment variables
|
||||
.env
|
||||
# Credentials and environment — never committed
|
||||
.env
|
||||
*.env
|
||||
|
||||
# Local, private reference material — never committed (see Docs/reference-notes-hollywood.md)
|
||||
/Art/reference/
|
||||
@@ -59,7 +59,7 @@ Semantic Versioning: `MAJOR.MINOR.PATCH`.
|
||||
4. Import OSM blockouts through Blosm; clean and detail them in Blender.
|
||||
5. Preserve geographic coordinates in metadata; convert to local Unreal coordinates with explicit meter→centimeter conversion.
|
||||
6. Export modular buildings, sidewalks, roads, collision, and materials.
|
||||
7. Use licensed reference photography for manual façade work.
|
||||
7. Use reference photography for manual façade work, ignore licensing for internal development use and do not commit reference images to the repository.
|
||||
8. Treat OSM as **layout data only** — not authoritative façade, height, or pavement detail.
|
||||
|
||||
Always use a consistent scale test, named export presets, and repeatable import settings.
|
||||
|
||||
+2
-2
@@ -8,12 +8,12 @@ Editable sources and original textures. Exported, game-ready output goes to `Gam
|
||||
| --- | --- |
|
||||
| `Art/blender/` | `.blend` sources, one file per building or asset family |
|
||||
| `Art/textures/` | Original texture sources before baking or compression |
|
||||
| `Art/reference/` | Licensed reference photography, development only — see attribution rules |
|
||||
| `Art/reference/` | Reference photography, local only, **gitignored — never committed**. Fetch with `Tools/commons_fetch.py`; what it is evidence for goes in `Docs/reference-notes-hollywood.md` |
|
||||
|
||||
## Rules
|
||||
|
||||
- Every `.blend` saves with a real-world metric scene scale. No exceptions; scale drift is the main source of import defects.
|
||||
- Named export presets only. Record the preset name and its settings next to the asset that used it.
|
||||
- Textures enter as a documented source (author, licence, source URL) and are listed in [../THIRD_PARTY_NOTICES.md](../THIRD_PARTY_NOTICES.md).
|
||||
- Reference photography stays out of the shipped build. It is evidence, not content, and its licences are recorded before it is committed.
|
||||
- Reference photography is never committed and never shipped. It is evidence, not content. Licensing is not a gate for local private development, but anything that would derive from a reference in a shipped asset needs a licensing decision before release.
|
||||
- `.blend` files are LFS-tracked and lockable; pull before editing.
|
||||
+3
-3
@@ -12,7 +12,7 @@ Licensing is tracked in four separate layers. Do not merge them; each has differ
|
||||
## Rules
|
||||
|
||||
1. An asset whose licence cannot be verified does not get committed. Record it as unresolved in [THIRD_PARTY_NOTICES.md](../THIRD_PARTY_NOTICES.md) and replace it with a redistributable asset before the release it would ship in.
|
||||
2. Reference photography is development evidence, not shipped content. Keep it in the repository only when its licence permits redistribution, and state in the notices that it is not built into the game.
|
||||
2. Reference photography is dev-only material and is **never committed**. It lives in the gitignored `Art/reference/` tree, is used for internal, private development only, and is not shipped or baked into shipped textures. What the images are evidence for is recorded in [reference-notes-hollywood.md](reference-notes-hollywood.md); the images stay outside version control.
|
||||
3. OSM is layout data only. Do not treat it as authoritative for façades, heights, or pavement detail, and do not import it as a texture or panorama source.
|
||||
4. Never use extracted Google imagery, Street View, or geometry. No exception.
|
||||
5. Attribution: "© OpenStreetMap contributors" with a link to <https://www.openstreetmap.org/copyright> appears in the shipped build's UI (a `0.1.0` release check) and in the notices.
|
||||
@@ -23,8 +23,8 @@ Licensing is tracked in four separate layers. Do not merge them; each has differ
|
||||
- **Code licence.** MIT is proposed: short, compatible with the assets' separate terms, and no copyleft spill onto art. Decide before publishing the repository.
|
||||
- **Documentation licence.** Default to the code licence unless a reason appears to separate them.
|
||||
- **ODbL posture.** Decide whether distributed builds contain a derived geographic database (a modified OSM extract) or only rendered geometry. The answer changes the share-alike obligations for the shipped product. Record the decision here and in the release notes.
|
||||
- **Reference photography retention.** Decided 2026-11-06: the El Capitan reference set is committed under `Art/reference/hollywood/` with its `SIDECAR.md` provenance carried over verbatim, because every file is CC0, public domain, CC BY, or CC BY-SA and none is shipped. The CC BY-SA review note travels with the set. Re-check before any material is derived from a frame.
|
||||
- **Reference photography retention.** Decided 2026-11-06: reference images are **not** committed. Licensing is not the gate for local, private development use, so images are simply kept out of the repository (`Art/reference/` is gitignored) and their provenance stays in the local sidecars. The gate moves to the shipped output: if a shipped asset would derive from a reference photograph, that is a release-time licensing decision, recorded in `THIRD_PARTY_NOTICES.md` before the asset ships.
|
||||
|
||||
## Provenance sources
|
||||
|
||||
Primary record: [THIRD_PARTY_NOTICES.md](../THIRD_PARTY_NOTICES.md). The sibling project at `…/06-games/security-game/docs/screens/refs/hollywood/SIDECAR.md` holds the original per-file provenance for the El Capitan block photographs; a verbatim copy now lives at `Art/reference/hollywood/SIDECAR.md`.
|
||||
Primary record: [THIRD_PARTY_NOTICES.md](../THIRD_PARTY_NOTICES.md) for anything committed or shipped, and [reference-notes-hollywood.md](reference-notes-hollywood.md) for what the local-only reference images are evidence of. Per-file provenance for the local sets lives in `Art/reference/hollywood/SIDECAR.md` and in each folder's `sidecar.json` — local files, not committed.
|
||||
+2
-2
@@ -36,8 +36,8 @@ Status detail in [Docs/setup.md](Docs/setup.md); workstation audit in [Docs/envi
|
||||
### Asset production
|
||||
|
||||
- [ ] Establish a consistent scale test, named export presets, and repeatable import settings.
|
||||
- [-] Collect licensed reference photography for manual façade work.
|
||||
- [ ] Download licensed reference photography for the two north-side buildings inside the 50 m buffer — TCL Chinese Theatre (OSM way 424105908, 6925 Hollywood Blvd) and Madame Tussaud's (way 424105889, 6933 Hollywood Blvd). No photograph of either exists in the current set. Keep the licence/provenance sidecar pattern from `Art/reference/hollywood/`, and do not source them from an unlicensed image search. See `Docs/data-inventory.md` for the tooling available.
|
||||
- [-] Collect reference photography for manual façade work. Local only, never committed: `Art/reference/` is gitignored, and what each set establishes is written up in `Docs/reference-notes-hollywood.md`. Still missing: the east retail neighbours 6806–6822, a night frame of the hero façade, and a sidewalk-level view of the hero entry.
|
||||
- [x] Download reference photography for the two north-side buildings inside the 50 m buffer — TCL Chinese Theatre (OSM way 424105908, 6925 Hollywood Blvd) and Madame Tussaud's (way 424105889, 6933 Hollywood Blvd). Done 2026-11-06 with `Tools/commons_fetch.py` against the Wikimedia Commons API: eleven frames in the local `Art/reference/north-chinese/` and `Art/reference/north-tussauds/`, provenance in each `sidecar.json`, nothing committed.
|
||||
- [ ] Model hero façade for the El Capitan Entertainment Centre site.
|
||||
- [ ] Model surrounding block frontage.
|
||||
- [ ] Model simplified background buildings directly across Hollywood Boulevard.
|
||||
|
||||
+14
-17
@@ -25,26 +25,23 @@ Status values: **verified** (licence confirmed and permits our use), **pending**
|
||||
|
||||
Attribution string for any OSM-derived output: "© OpenStreetMap contributors", <https://www.openstreetmap.org/copyright>.
|
||||
|
||||
## Reference photography (development only, not shipped)
|
||||
## Reference photography (local only, not committed, not shipped)
|
||||
|
||||
Committed in `Art/reference/hollywood/` (Git LFS). Copied 2026-11-06 from `…/06-games/security-game/docs/screens/refs/hollywood/`, which holds the per-file provenance in `SIDECAR.md` — carried over verbatim with the images. These are development evidence and are not built into the game.
|
||||
Reference photographs are **not** part of this repository. They live in the gitignored `Art/reference/` tree on the development machine, are used only for internal, private development, and are never built into a build or shipped. `.gitignore` enforces this; nothing under `Art/reference/` may be added, staged, or committed.
|
||||
|
||||
| File | Subject | Author | Licence | Shipped |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `el-capitan-facade-day-wikimedia-2009-10-27.jpg` | El Capitan facade, day | woo | CC BY 2.0 | no |
|
||||
| `el-capitan-theatre-blade-sign-day-wikimedia-2008-10.jpg` | blade sign and temple frontage | Andreas Praefcke | CC BY 3.0 | no |
|
||||
| `el-capitan-theatre-marquee-neighbour-day-wikimedia-2009-07-11.jpg` | east neighbour marquee | Alan Light | CC BY 2.0 | no |
|
||||
| `premiere-staging-street-day-wikimedia-2006-02-12.jpg` | premiere staging | John Barreiros | CC BY 2.0 | no |
|
||||
| `masonic-temple-historic-pd-1922.jpg` | Masonic Temple, 1922 | Charles W. Beam | public domain | no |
|
||||
| `ovation-hollywood-across-street-pd-2006-12-28.jpg` | Ovation Hollywood, across street | Gary Minnaert | public domain | no |
|
||||
| `walk-of-fame-terrazzo-ovation-day-cc0-2019-06-20.jpg` | Walk of Fame terrazzo band | Luijtenphotos | CC0 | no |
|
||||
| `walk-of-fame-star-closeup-cc0-2022-07-08.jpg` | star close-up | Benoît Prieur | CC0 | no |
|
||||
| `hollywood-boulevard-night-context-ccbysa.jpg` | boulevard at night (near Vine) | Downtowngal | CC BY-SA 3.0 | no |
|
||||
| Local folder | Source | Provenance |
|
||||
| --- | --- | --- |
|
||||
| `Art/reference/hollywood/` | Copied 2026-11-06 from `…/06-games/security-game/docs/screens/refs/hollywood/` | That folder's `SIDECAR.md` (author, licence, URL, crop notes) |
|
||||
| `Art/reference/north-chinese/` | Fetched 2026-11-06 with `Tools/commons_fetch.py` from the Wikimedia Commons API | `sidecar.json` in the folder |
|
||||
| `Art/reference/north-tussauds/` | Same | `sidecar.json` in the folder |
|
||||
|
||||
Notes:
|
||||
- The CC BY-SA frame is used for colour and lighting study only. No texture, mesh, or material is derived from it. Any future derivative needs a fresh share-alike review before use.
|
||||
- Pre-existing crop and redaction notes live in the sidecar and apply to any reuse.
|
||||
- These photos are evidence for proportions, signage, and staging. They are licensed as photographs, not as building designs; do not treat them as permission to reproduce signage artwork at full fidelity. Venue lettering and trademarks are separate rights.
|
||||
What the images are used for, what they are evidence for, and the block's recorded measurements live in [Docs/reference-notes-hollywood.md](Docs/reference-notes-hollywood.md). That document is committed; the images are not.
|
||||
|
||||
Rules that still apply, because they are about shipped output rather than local working material:
|
||||
|
||||
- No reference photograph may be shipped, baked into a texture, or distributed. If a shipped asset would otherwise derive from one, that is a licensing decision to be made before the release, recorded here.
|
||||
- Venue lettering, signage artwork, and trademarks are separate rights from the photograph. Copying a façade's proportions is not the same as reproducing its signage.
|
||||
- A reference image whose provenance cannot be found locally must not be used at all.
|
||||
|
||||
## Assets
|
||||
|
||||
|
||||
Executable
+236
@@ -0,0 +1,236 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fetch reference photographs from Wikimedia Commons.
|
||||
|
||||
Local, private development reference only. Images fetched here are **not**
|
||||
committed: the output directory is gitignored. The tool records provenance
|
||||
(author, licence, source URL) in a JSON sidecar next to the images so a local
|
||||
working copy stays traceable.
|
||||
|
||||
# dry run: list what the search returns, download nothing
|
||||
python3 Tools/commons_fetch.py --query "Grauman's Chinese Theatre" --slug north-chinese --dry-run
|
||||
|
||||
# download up to 6 images at 1600 px wide
|
||||
python3 Tools/commons_fetch.py --query "Grauman's Chinese Theatre" --slug north-chinese --limit 6
|
||||
|
||||
# fetch specific files by title
|
||||
python3 Tools/commons_fetch.py --file "File:Grauman's Chinese Theatre, Hollywood.JPG" --slug north-chinese
|
||||
|
||||
Defaults: out dir `Art/reference/<slug>/`, 1600 px wide thumbnails, one request
|
||||
per second, JPEG/PNG only, at least 800 px wide on the original.
|
||||
|
||||
Requires only the standard library.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
API = "https://commons.wikimedia.org/w/api.php"
|
||||
UA = "gig-line-references/0.1 (local development reference fetch)"
|
||||
ALLOWED_MIME = {"image/jpeg", "image/png"}
|
||||
MIN_WIDTH = 800
|
||||
|
||||
|
||||
def api_get(params: dict, retries: int = 3) -> dict:
|
||||
query = urllib.parse.urlencode({**params, "format": "json", "formatversion": "2"})
|
||||
url = f"{API}?{query}"
|
||||
for attempt in range(retries):
|
||||
req = urllib.request.Request(url, headers={"User-Agent": UA})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=45) as resp:
|
||||
return json.loads(resp.read().decode("utf-8"))
|
||||
except Exception as exc: # noqa: BLE001 - report and retry, then fail
|
||||
if attempt == retries - 1:
|
||||
raise SystemExit(f"Commons API failed: {exc}")
|
||||
time.sleep(2 * (attempt + 1))
|
||||
return {}
|
||||
|
||||
|
||||
def strip_html(value: str) -> str:
|
||||
text = re.sub(r"<[^>]+>", " ", value or "")
|
||||
return re.sub(r"\s+", " ", text).strip()
|
||||
|
||||
|
||||
def search_titles(query: str, limit: int) -> list[str]:
|
||||
data = api_get(
|
||||
{
|
||||
"action": "query",
|
||||
"generator": "search",
|
||||
"gsrsearch": query,
|
||||
"gsrnamespace": "6",
|
||||
"gsrlimit": str(limit),
|
||||
"prop": "imageinfo",
|
||||
"iiprop": "url|size|mime|extmetadata",
|
||||
}
|
||||
)
|
||||
pages = data.get("query", {}).get("pages", [])
|
||||
# generator search returns unordered; keep relevance order from the index list
|
||||
order = [p.get("pageid") for p in pages]
|
||||
titles = []
|
||||
for page in pages:
|
||||
info = (page.get("imageinfo") or [{}])[0]
|
||||
if info.get("mime") not in ALLOWED_MIME:
|
||||
continue
|
||||
if info.get("width", 0) < MIN_WIDTH:
|
||||
continue
|
||||
titles.append(page["title"])
|
||||
return titles, order
|
||||
|
||||
|
||||
def image_info(titles: list[str], thumb_width: int) -> list[dict]:
|
||||
data = api_get(
|
||||
{
|
||||
"action": "query",
|
||||
"titles": "|".join(titles),
|
||||
"prop": "imageinfo",
|
||||
"iiprop": "url|size|mime|extmetadata|sha1|user",
|
||||
"iiurlwidth": str(thumb_width),
|
||||
}
|
||||
)
|
||||
out = []
|
||||
for page in data.get("query", {}).get("pages", []):
|
||||
info = (page.get("imageinfo") or [{}])[0]
|
||||
if not info or info.get("mime") not in ALLOWED_MIME:
|
||||
continue
|
||||
meta = info.get("extmetadata", {})
|
||||
out.append(
|
||||
{
|
||||
"title": page["title"],
|
||||
"page_url": info.get("descriptionurl"),
|
||||
"file_url": info.get("url"),
|
||||
"download_url": info.get("thumburl") or info.get("url"),
|
||||
"width": info.get("width"),
|
||||
"height": info.get("height"),
|
||||
"mime": info.get("mime"),
|
||||
"uploader": info.get("user"),
|
||||
"sha1": info.get("sha1"),
|
||||
"artist": strip_html(meta.get("Artist", {}).get("value", "")),
|
||||
"credit": strip_html(meta.get("Credit", {}).get("value", "")),
|
||||
"licence": strip_html(
|
||||
meta.get("LicenseShortName", {}).get("value", "")
|
||||
or meta.get("UsageTerms", {}).get("value", "")
|
||||
),
|
||||
"licence_url": strip_html(meta.get("LicenseUrl", {}).get("value", "")),
|
||||
"attribution_required": strip_html(
|
||||
meta.get("AttributionRequired", {}).get("value", "")
|
||||
),
|
||||
"date": strip_html(
|
||||
meta.get("DateTimeOriginal", {}).get("value", "")
|
||||
or meta.get("DateTime", {}).get("value", "")
|
||||
),
|
||||
"description": strip_html(meta.get("ImageDescription", {}).get("value", ""))[:400],
|
||||
}
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def safe_name(title: str, index: int) -> str:
|
||||
base = title.split(":", 1)[-1]
|
||||
base = re.sub(r"[^A-Za-z0-9._-]+", "-", base).strip("-").lower()
|
||||
ext = os.path.splitext(base)[1] or ".jpg"
|
||||
stem = os.path.splitext(base)[0][:60]
|
||||
return f"{index:02d}-{stem}{ext}"
|
||||
|
||||
|
||||
def download(url: str, path: str, retries: int = 3) -> str:
|
||||
for attempt in range(retries):
|
||||
req = urllib.request.Request(url, headers={"User-Agent": UA})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=90) as resp:
|
||||
payload = resp.read()
|
||||
with open(path, "wb") as handle:
|
||||
handle.write(payload)
|
||||
return hashlib.sha256(payload).hexdigest()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
if attempt == retries - 1:
|
||||
raise SystemExit(f"download failed for {url}: {exc}")
|
||||
time.sleep(2 * (attempt + 1))
|
||||
return ""
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--query", help="Commons search string")
|
||||
ap.add_argument("--file", action="append", default=[], help="exact File: title, repeatable")
|
||||
ap.add_argument("--slug", required=True, help="output folder name under the out dir")
|
||||
ap.add_argument("--out", default="Art/reference", help="base output directory")
|
||||
ap.add_argument("--limit", type=int, default=6)
|
||||
ap.add_argument("--width", type=int, default=1600, help="thumbnail width in px")
|
||||
ap.add_argument("--dry-run", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
if not args.query and not args.file:
|
||||
ap.error("need --query or --file")
|
||||
|
||||
titles = list(args.file)
|
||||
if args.query:
|
||||
found, _ = search_titles(args.query, args.limit * 3)
|
||||
titles += [t for t in found if t not in titles]
|
||||
|
||||
if not titles:
|
||||
print("no candidate files matched")
|
||||
return 1
|
||||
|
||||
print(f"{len(titles)} candidate file(s)")
|
||||
if args.dry_run:
|
||||
for title in titles[: args.limit]:
|
||||
print(f" {title}")
|
||||
return 0
|
||||
|
||||
out_dir = os.path.join(args.out, args.slug)
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
|
||||
selected = image_info(titles[: args.limit], args.width)
|
||||
if not selected:
|
||||
print("no downloadable candidates")
|
||||
return 1
|
||||
|
||||
records = []
|
||||
for index, item in enumerate(selected, start=1):
|
||||
name = safe_name(item["title"], index)
|
||||
path = os.path.join(out_dir, name)
|
||||
print(f" [{index}/{len(selected)}] {item['title']}")
|
||||
digest = download(item["download_url"], path)
|
||||
item.update(
|
||||
{
|
||||
"local_file": name,
|
||||
"sha256": digest,
|
||||
"bytes": os.path.getsize(path),
|
||||
}
|
||||
)
|
||||
records.append(item)
|
||||
time.sleep(1)
|
||||
|
||||
sidecar = {
|
||||
"fetched_utc": datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds"),
|
||||
"source": "Wikimedia Commons API",
|
||||
"api": API,
|
||||
"query": args.query,
|
||||
"requested_files": args.file,
|
||||
"slug": args.slug,
|
||||
"thumb_width": args.width,
|
||||
"tool": "Tools/commons_fetch.py",
|
||||
"note": "local development reference only; not committed to the repository",
|
||||
"images": records,
|
||||
}
|
||||
sidecar_path = os.path.join(out_dir, "sidecar.json")
|
||||
with open(sidecar_path, "w", encoding="utf-8") as handle:
|
||||
json.dump(sidecar, handle, indent=2, ensure_ascii=False)
|
||||
|
||||
print(f"\nwrote {len(records)} image(s) + sidecar.json to {out_dir}")
|
||||
for item in records:
|
||||
print(f" {item['local_file']} {item['licence'] or 'licence unknown'} {item['artist'][:40]}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user