feat(character): ComfyUI blueprints and a render-vs-blueprint evaluation
Two tools that turn character work into something measurable. - Tools/comfyui_character_blueprint.py generates a four-view sheet (front, side, back, head) plus a contact sheet from one description, with fixed seeds and a JSON record of prompts and prompt ids. First sheet: 'GigLine walker', lean pedestrian in a navy tee, jeans and sneakers. - Tools/character_match.py compares a rendered character against a sheet: figure aspect, fill and coverage deltas, mean CIE76 colour distance, dominant-palette distance, and a side-by-side montage. The deltas stay separate on purpose - a single similarity score between a photoreal sheet and a low-detail body would be a number with no meaning. - comfyui_generate.py grew wait_for_image() so both tools share the polling path instead of duplicating it. - 13 unit tests for the colour maths, masks and palette; 95 tool tests pass. First real result, recorded rather than smoothed over: aspect delta 1.24, fill delta 0.12, mean deltaE 31.9, palette deltaE 25.4. The montage shows the reason - the model is a nude body and the blueprint wears a tee and jeans, so clothing dominates every metric. Figure segmentation is explicit (--render-box): auto-detection picked backdrop blobs on every attempt, and a background plate is the proper fix.
This commit is contained in:
1 parent
1fc5835327
commit
85a8437cbc
56 files changed
+841
-170
No files matched your search
@@ -90,6 +90,10 @@ Geographic tools live in [`Geo/tools/`](../Geo/README.md) and the Blender-side b
|
||||
| --- | --- |
|
||||
| `input_emulation.py` | Pure-stdlib `ctypes`/`fcntl` driver that creates a virtual keyboard+mouse on `/dev/uinput`. The desktop treats it as real hardware, so it proves whether OS input actually reaches the game. Needs `/dev/uinput` writable (ACL on this host). |
|
||||
| `input_test.py` | Injects a scripted sequence into the focused window: `--sequence move:2,look:3,jump:1,reset:1,pause:1,click`. |
|
||||
| `comfyui_generate.py` | Local ComfyUI driver: list checkpoints, run txt2img through the HTTP API, and write a provenance sidecar (model, prompt, seed, full graph). Local, human-triggered. | `models`; `txt2img --prompt ... --out <png>` | `python3 Tools/comfyui_generate.py models` |
|
||||
| `comfyui_character_blueprint.py` | Generates a character blueprint sheet - front, side, back and head views plus a contact sheet - from one description, with fixed seeds and a JSON record. | `--desc`, `--out`, `--model`, `--size` | `python3 Tools/comfyui_character_blueprint.py --desc "lean pedestrian, navy tee, jeans" --out Art/characters/blueprints` |
|
||||
| `character_match.py` | Compares a rendered character against a blueprint: figure aspect, fill and coverage deltas, mean CIE76 colour distance, dominant-palette distance, and a side-by-side montage. | `--render`, `--blueprint`, `--render-box`, `--out`, `--montage` | `python3 Tools/character_match.py --render Screenshots/demo/review-demo-third-person-front.png --blueprint Art/characters/blueprints/front.png --render-box 0.42,0.46,0.56,0.97 --montage /tmp/gigline/match.png` |
|
||||
| `generate_character_textures.py` | Procedural skin-pore and hair-strand textures with sidecars; no model involved. | `--out` | `python3 Tools/generate_character_textures.py --out Art/textures/generated` |
|
||||
|
||||
## Tests
|
||||
|
||||
|
||||
@@ -0,0 +1,368 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Evaluate a rendered character against a generated blueprint.
|
||||
|
||||
The blueprint comes from `Tools/comfyui_character_blueprint.py`; the render comes from
|
||||
the review pipeline. This tool isolates the figure in both, then reports how far the
|
||||
model has drifted from the intended character:
|
||||
|
||||
python3 Tools/character_match.py \
|
||||
--render Screenshots/demo/review-demo-third-person-front.png \
|
||||
--blueprint Art/characters/blueprints/front.png \
|
||||
--out /tmp/gigline/character-match.json \
|
||||
--montage /tmp/gigline/character-match.png
|
||||
|
||||
Reported per pair:
|
||||
|
||||
aspect_delta |height/width difference| between the two figures, in ratio units
|
||||
fill_delta |figure area / bbox area| - bulk, catches a thin or bloated build
|
||||
coverage_delta |figure area / frame area| - how much of the frame the figure takes
|
||||
mean_delta_e mean CIE76 colour distance over the figure
|
||||
palette_delta_e mean distance between the five dominant colours of each figure
|
||||
|
||||
Why these and not a similarity score: an absolute "match percentage" between a
|
||||
photoreal generated sheet and a low-detail game body would be a number with no
|
||||
meaning. The deltas are separable, so a report says *what* drifted - build, framing
|
||||
or colour - instead of blending all three into one figure.
|
||||
|
||||
The figure mask is a heuristic and the tool says so in its report: for the blueprint
|
||||
it is "differs from the corner colour", for the render it is "more saturated or darker
|
||||
than the frame median", largest connected component. Pair the tool with a human look
|
||||
at the montage; it narrows where to look, it does not replace the review.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
|
||||
from PIL import Image
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- colour maths
|
||||
|
||||
def srgb_to_linear(value: float) -> float:
|
||||
value /= 255.0
|
||||
return value / 12.92 if value <= 0.04045 else ((value + 0.055) / 1.055) ** 2.4
|
||||
|
||||
|
||||
def rgb_to_lab(rgb) -> tuple[float, float, float]:
|
||||
"""sRGB (0-255) to CIELAB under D65."""
|
||||
r, g, b = (srgb_to_linear(v) for v in rgb[:3])
|
||||
x = r * 0.4124 + g * 0.3576 + b * 0.1805
|
||||
y = r * 0.2126 + g * 0.7152 + b * 0.0722
|
||||
z = r * 0.0193 + g * 0.1192 + b * 0.9505
|
||||
# D65 white point
|
||||
xn, yn, zn = 0.95047, 1.0, 1.08883
|
||||
|
||||
def f(t: float) -> float:
|
||||
return t ** (1.0 / 3.0) if t > 0.008856 else (7.787 * t) + (16.0 / 116.0)
|
||||
|
||||
fx, fy, fz = f(x / xn), f(y / yn), f(z / zn)
|
||||
return (116.0 * fy - 16.0, 500.0 * (fx - fy), 200.0 * (fy - fz))
|
||||
|
||||
|
||||
def delta_e76(lab_a, lab_b) -> float:
|
||||
return math.sqrt(sum((a - b) ** 2 for a, b in zip(lab_a, lab_b)))
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- masking
|
||||
|
||||
def luminance(rgb) -> float:
|
||||
return 0.2126 * rgb[0] + 0.7152 * rgb[1] + 0.0722 * rgb[2]
|
||||
|
||||
|
||||
def _subject_mask(image: Image.Image, saturation_threshold: int = 20,
|
||||
dark_ratio: float = 0.62) -> Image.Image:
|
||||
"""Mask the figure as "more saturated than the backdrop, or notably darker".
|
||||
|
||||
Both inputs have a desaturated backdrop - grey studio sweep in the blueprint,
|
||||
beige pavement and pale walls in the render - and a figure that carries colour
|
||||
(skin, denim, hair). Saturation separates them without needing a clean plate.
|
||||
The previous version masked the blueprint by "differs from the corner colour",
|
||||
which swallowed a gradient backdrop whole.
|
||||
"""
|
||||
rgb = image.convert("RGB")
|
||||
pixels = rgb.load()
|
||||
lums = [luminance(pixels[x, y])
|
||||
for y in range(0, rgb.height, 4) for x in range(0, rgb.width, 4)]
|
||||
lums.sort()
|
||||
median = lums[len(lums) // 2] if lums else 128.0
|
||||
|
||||
mask = Image.new("L", rgb.size, 0)
|
||||
out = mask.load()
|
||||
for y in range(rgb.height):
|
||||
for x in range(rgb.width):
|
||||
r, g, b = pixels[x, y]
|
||||
saturation = max(r, g, b) - min(r, g, b)
|
||||
if saturation > saturation_threshold or luminance((r, g, b)) < median * dark_ratio:
|
||||
out[x, y] = 255
|
||||
return mask
|
||||
|
||||
|
||||
def figure_mask_from_background(image: Image.Image, tolerance: int = 28):
|
||||
"""Figure mask for a blueprint sheet: a coloured subject on a grey sweep."""
|
||||
return _largest_component(_subject_mask(image, saturation_threshold=18,
|
||||
dark_ratio=0.55))
|
||||
|
||||
|
||||
def figure_mask_from_frame(image: Image.Image):
|
||||
"""Figure mask for a rendered frame: the body against pavement and stucco."""
|
||||
return _largest_component(_subject_mask(image, saturation_threshold=26,
|
||||
dark_ratio=0.72))
|
||||
|
||||
|
||||
def _largest_component(mask: Image.Image) -> Image.Image:
|
||||
"""Keep the largest plausible figure blob.
|
||||
|
||||
"Largest" alone is wrong twice over: in both inputs the backdrop is one huge
|
||||
connected region, so a plain largest-blob pick returned the background and the
|
||||
first report measured the whole frame. The filter is size-based instead - a figure
|
||||
occupies between 0.2 % and 55 % of the frame - which rejects the backdrop on one
|
||||
side and road markings on the other.
|
||||
"""
|
||||
width, height = mask.size
|
||||
frame_area = width * height
|
||||
source = mask.load()
|
||||
seen = bytearray(width * height)
|
||||
best = [] # largest blob fully inside the frame
|
||||
usable = [] # largest plausible blob, border or not
|
||||
fallback = [] # largest blob at all, used when nothing is plausible
|
||||
best_score = 0
|
||||
|
||||
for start_y in range(0, height, 3):
|
||||
for start_x in range(0, width, 3):
|
||||
index = start_y * width + start_x
|
||||
if seen[index] or source[start_x, start_y] == 0:
|
||||
continue
|
||||
stack = [(start_x, start_y)]
|
||||
blob = []
|
||||
seen[index] = 1
|
||||
while stack:
|
||||
x, y = stack.pop()
|
||||
blob.append((x, y))
|
||||
for nx, ny in ((x + 1, y), (x - 1, y), (x, y + 1), (x, y - 1)):
|
||||
if 0 <= nx < width and 0 <= ny < height:
|
||||
nindex = ny * width + nx
|
||||
if not seen[nindex] and source[nx, ny] != 0:
|
||||
seen[nindex] = 1
|
||||
stack.append((nx, ny))
|
||||
if len(blob) > len(fallback):
|
||||
fallback = blob
|
||||
share = len(blob) / frame_area
|
||||
if not (0.002 < share < 0.55):
|
||||
continue
|
||||
# Blobs glued to the frame edge are backdrop: the sky band in a render and
|
||||
# the studio sweep in a sheet both touch the border and both beat the
|
||||
# figure on raw area. Prefer fully enclosed blobs, fall back by area.
|
||||
xs = [p[0] for p in blob]
|
||||
ys = [p[1] for p in blob]
|
||||
enclosed = (min(xs) > 1 and min(ys) > 1
|
||||
and max(xs) < width - 2 and max(ys) < height - 2)
|
||||
if enclosed and len(blob) > best_score:
|
||||
best_score = len(blob)
|
||||
best = blob
|
||||
elif len(blob) > len(usable):
|
||||
usable = blob
|
||||
|
||||
keep = Image.new("L", mask.size, 0)
|
||||
out = keep.load()
|
||||
for x, y in (best or usable or fallback):
|
||||
out[x, y] = 255
|
||||
return keep
|
||||
|
||||
|
||||
def bbox(mask: Image.Image):
|
||||
return mask.getbbox()
|
||||
|
||||
|
||||
def mask_stats(mask: Image.Image) -> dict:
|
||||
box = mask.getbbox()
|
||||
if not box:
|
||||
return {"area": 0, "bbox": None, "aspect": None, "fill": None, "coverage": None}
|
||||
x0, y0, x1, y1 = box
|
||||
histogram = mask.histogram()
|
||||
area = sum(histogram[1:])
|
||||
box_area = max(1, (x1 - x0) * (y1 - y0))
|
||||
return {
|
||||
"area": area,
|
||||
"bbox": [x0, y0, x1, y1],
|
||||
"aspect": (y1 - y0) / max(1, (x1 - x0)),
|
||||
"fill": area / box_area,
|
||||
"coverage": area / (mask.width * mask.height),
|
||||
}
|
||||
|
||||
|
||||
def masked_colours(image: Image.Image, mask: Image.Image, limit=20000):
|
||||
rgb = image.convert("RGB")
|
||||
pixels = rgb.load()
|
||||
values = mask.load()
|
||||
collected = []
|
||||
step = max(1, (rgb.width * rgb.height) // limit)
|
||||
index = 0
|
||||
for y in range(rgb.height):
|
||||
for x in range(rgb.width):
|
||||
index += 1
|
||||
if index % step:
|
||||
continue
|
||||
if values[x, y]:
|
||||
collected.append(pixels[x, y])
|
||||
return collected
|
||||
|
||||
|
||||
def dominant(counts: dict, count: int = 5):
|
||||
"""Coarse palette: quantise to 32 levels per channel, then take the top bins."""
|
||||
bins: dict[tuple, int] = {}
|
||||
for colour in counts:
|
||||
key = (colour[0] // 32, colour[1] // 32, colour[2] // 32)
|
||||
bins[key] = bins.get(key, 0) + 1
|
||||
ranked = sorted(bins.items(), key=lambda item: -item[1])[:count]
|
||||
return [((k[0] * 32) + 16, (k[1] * 32) + 16, (k[2] * 32) + 16) for k, _ in ranked]
|
||||
|
||||
|
||||
def palette_delta_e(palette_a, palette_b) -> float:
|
||||
if not palette_a or not palette_b:
|
||||
return float("nan")
|
||||
labs_b = [rgb_to_lab(c) for c in palette_b]
|
||||
total = 0.0
|
||||
for colour in palette_a:
|
||||
lab = rgb_to_lab(colour)
|
||||
total += min(delta_e76(lab, other) for other in labs_b)
|
||||
return total / len(palette_a)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------- report
|
||||
|
||||
def evaluate(render_path: str, blueprint_path: str, montage_path: str | None = None,
|
||||
render_box: tuple[float, float, float, float] | None = None) -> dict:
|
||||
render_full = Image.open(render_path).convert("RGB")
|
||||
blueprint = Image.open(blueprint_path).convert("RGB")
|
||||
|
||||
# The figure box is explicit by default. Auto-segmentation on these shots failed
|
||||
# in a way worth recording rather than papering over: the backdrop (sky band, pale
|
||||
# wall, dark pavement) is more saturated and far larger than the subject, so every
|
||||
# size-based rule picked a backdrop blob. A box read off the shot is honest and
|
||||
# cheap; a background plate rendered without the character would be the proper fix.
|
||||
width, height = render_full.size
|
||||
box = render_box or (0.15, 0.02, 0.85, 0.99)
|
||||
region = (int(width * box[0]), int(height * box[1]),
|
||||
int(width * box[2]), int(height * box[3]))
|
||||
render = render_full.crop(region)
|
||||
|
||||
render_mask = figure_mask_from_frame(render)
|
||||
blueprint_mask = figure_mask_from_background(blueprint)
|
||||
|
||||
render_stats = mask_stats(render_mask)
|
||||
blueprint_stats = mask_stats(blueprint_mask)
|
||||
|
||||
render_colours = masked_colours(render, render_mask)
|
||||
blueprint_colours = masked_colours(blueprint, blueprint_mask)
|
||||
|
||||
def mean_lab(colours):
|
||||
labs = [rgb_to_lab(c) for c in colours]
|
||||
if not labs:
|
||||
return None
|
||||
return tuple(sum(l[i] for l in labs) / len(labs) for i in range(3))
|
||||
|
||||
lab_render = mean_lab(render_colours)
|
||||
lab_blueprint = mean_lab(blueprint_colours)
|
||||
mean_de = (delta_e76(lab_render, lab_blueprint)
|
||||
if lab_render and lab_blueprint else float("nan"))
|
||||
|
||||
palette_render = dominant(render_colours)
|
||||
palette_blueprint = dominant(blueprint_colours)
|
||||
|
||||
def delta(a, b):
|
||||
if a is None or b is None:
|
||||
return None
|
||||
return abs(a - b)
|
||||
|
||||
report = {
|
||||
"render": os.path.abspath(render_path),
|
||||
"blueprint": os.path.abspath(blueprint_path),
|
||||
"render_mask": "saturated-or-dark vs frame median, largest component",
|
||||
"blueprint_mask": "differs from corner colour",
|
||||
"render_region": list(region),
|
||||
"render_figure": render_stats,
|
||||
"blueprint_figure": blueprint_stats,
|
||||
"aspect_delta": delta(render_stats["aspect"], blueprint_stats["aspect"]),
|
||||
"fill_delta": delta(render_stats["fill"], blueprint_stats["fill"]),
|
||||
"coverage_delta": delta(render_stats["coverage"], blueprint_stats["coverage"]),
|
||||
"mean_lab_render": lab_render,
|
||||
"mean_lab_blueprint": lab_blueprint,
|
||||
"mean_delta_e": mean_de,
|
||||
"palette_render": palette_render,
|
||||
"palette_blueprint": palette_blueprint,
|
||||
"palette_delta_e": palette_delta_e(palette_render, palette_blueprint),
|
||||
}
|
||||
|
||||
if montage_path:
|
||||
report["montage"] = write_montage(render, render_mask, blueprint, blueprint_mask,
|
||||
montage_path)
|
||||
return report
|
||||
|
||||
|
||||
def write_montage(render, render_mask, blueprint, blueprint_mask, path, height=640):
|
||||
"""Render crop and blueprint crop side by side, same height, for a human look."""
|
||||
def crop(image, mask):
|
||||
box = mask.getbbox()
|
||||
return image.crop(box) if box else image
|
||||
|
||||
left = crop(render, render_mask)
|
||||
right = crop(blueprint, blueprint_mask)
|
||||
left = left.resize((max(1, int(left.width * height / max(1, left.height))), height))
|
||||
right = right.resize((max(1, int(right.width * height / max(1, right.height))), height))
|
||||
sheet = Image.new("RGB", (left.width + right.width + 12, height), (20, 20, 20))
|
||||
sheet.paste(left, (0, 0))
|
||||
sheet.paste(right, (left.width + 12, 0))
|
||||
os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
|
||||
sheet.save(path)
|
||||
return os.path.abspath(path)
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--render", required=True)
|
||||
ap.add_argument("--blueprint", required=True)
|
||||
ap.add_argument("--out", default="/tmp/gigline/character-match.json")
|
||||
ap.add_argument("--montage", default=None)
|
||||
ap.add_argument("--render-box", default=None,
|
||||
help="figure box as x0,y0,x1,y1 fractions of the frame (default: central 70 percent)")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
for path in (args.render, args.blueprint):
|
||||
if not os.path.exists(path):
|
||||
print(f"error: no such image: {path}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
box = None
|
||||
if args.render_box:
|
||||
box = tuple(float(v) for v in args.render_box.split(","))
|
||||
if len(box) != 4:
|
||||
print("error: --render-box needs four comma-separated fractions", file=sys.stderr)
|
||||
return 2
|
||||
report = evaluate(args.render, args.blueprint, args.montage, box)
|
||||
os.makedirs(os.path.dirname(os.path.abspath(args.out)) or ".", exist_ok=True)
|
||||
with open(args.out, "w") as handle:
|
||||
json.dump(report, handle, indent=2)
|
||||
|
||||
def show(value, digits=3):
|
||||
return "n/a" if value is None or (isinstance(value, float) and math.isnan(value)) \
|
||||
else f"{value:.{digits}f}"
|
||||
|
||||
print(f"render {report['render_figure']}")
|
||||
print(f"blueprint {report['blueprint_figure']}")
|
||||
print(f"aspect delta {show(report['aspect_delta'])}")
|
||||
print(f"fill delta {show(report['fill_delta'])}")
|
||||
print(f"coverage delta {show(report['coverage_delta'])}")
|
||||
print(f"mean deltaE {show(report['mean_delta_e'], 1)}")
|
||||
print(f"palette deltaE {show(report['palette_delta_e'], 1)}")
|
||||
print(f"report -> {args.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,118 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate character blueprints with the local ComfyUI instance.
|
||||
|
||||
A "blueprint" here is an art-direction sheet for one character: a front, side and back
|
||||
view, plus a head detail, generated from a fixed prompt and seed set so the sheet can
|
||||
be regenerated and compared against later. It is the reference the modelled character
|
||||
is evaluated against — see `Tools/character_match.py`.
|
||||
|
||||
python3 Tools/comfyui_character_blueprint.py \
|
||||
--character "GigLine walker" \
|
||||
--desc "lean adult pedestrian, short dark hair, olive skin, plain navy t-shirt, dark jeans, worn sneakers" \
|
||||
--out Art/characters/blueprints
|
||||
|
||||
Writes one PNG per view plus `contact-sheet.png` and `blueprint.json`, which records
|
||||
the prompt, model, seeds and the ComfyUI prompt ids. Everything is generated locally
|
||||
by `Tools/comfyui_generate.py`; no network service and no credential is involved.
|
||||
|
||||
The sheet is generated art, not photography, and it stays out of the reference photo
|
||||
folders on purpose: AGENTS.md forbids committing reference photography, and a
|
||||
generated concept should not blur that line.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import comfyui_generate as comfy # noqa: E402
|
||||
|
||||
from PIL import Image # noqa: E402
|
||||
|
||||
VIEWS = (
|
||||
# name, prompt suffix, seed
|
||||
("front", "full body front view, neutral standing pose, arms at sides, "
|
||||
"even studio light, plain light grey background, character sheet", 101),
|
||||
("side", "full body side profile view, neutral standing pose, arms at sides, "
|
||||
"even studio light, plain light grey background, character sheet", 202),
|
||||
("back", "full body back view, neutral standing pose, arms at sides, "
|
||||
"even studio light, plain light grey background, character sheet", 303),
|
||||
("head", "head and shoulders portrait, front view, neutral expression, "
|
||||
"even studio light, plain light grey background", 404),
|
||||
)
|
||||
NEGATIVE = ("multiple people, crowd, text, watermark, signature, blurry, low quality, "
|
||||
"cropped limbs, extra limbs, deformed hands, cartoon, anime, logo")
|
||||
|
||||
|
||||
def contact_sheet(paths, out_path, columns=4):
|
||||
images = [Image.open(p).convert("RGB") for p in paths if os.path.exists(p)]
|
||||
if not images:
|
||||
return None
|
||||
width = max(i.width for i in images)
|
||||
height = max(i.height for i in images)
|
||||
sheet = Image.new("RGB", (width * columns, height), (24, 24, 24))
|
||||
for index, image in enumerate(images[:columns]):
|
||||
sheet.paste(image.resize((width, height)), (index * width, 0))
|
||||
sheet.save(out_path)
|
||||
return out_path
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--character", default="GigLine walker")
|
||||
ap.add_argument("--desc", required=True,
|
||||
help="what the character wears and looks like; drives every view")
|
||||
ap.add_argument("--out", default="Art/characters/blueprints")
|
||||
ap.add_argument("--model", default=comfy.DEFAULT_MODEL)
|
||||
ap.add_argument("--size", default="832x1216")
|
||||
ap.add_argument("--steps", type=int, default=30)
|
||||
ap.add_argument("--cfg", type=float, default=5.0)
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
if not os.path.isdir(comfy.HOST) and "://" not in comfy.HOST:
|
||||
print(f"error: bad COMFYUI_HOST {comfy.HOST}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
os.makedirs(args.out, exist_ok=True)
|
||||
width, height = (int(v) for v in args.size.lower().split("x"))
|
||||
|
||||
record = {"character": args.character, "description": args.desc,
|
||||
"model": args.model, "views": {}}
|
||||
written = []
|
||||
for name, suffix, seed in VIEWS:
|
||||
prompt = f"{args.character}, {args.desc}, {suffix}"
|
||||
out = os.path.join(args.out, f"{name}.png")
|
||||
graph = comfy.workflow(prompt, NEGATIVE, args.model, width, height, seed,
|
||||
args.steps, args.cfg, "dpmpp_2m", "karras")
|
||||
queued = comfy.post("/prompt", {"prompt": graph})
|
||||
prompt_id = queued.get("prompt_id")
|
||||
if not prompt_id:
|
||||
print(f"error: ComfyUI rejected the {name} prompt: {queued}", file=sys.stderr)
|
||||
return 1
|
||||
print(f"{name}: queued {prompt_id} (seed {seed})")
|
||||
image = comfy.wait_for_image(prompt_id, timeout=600.0)
|
||||
if image is None:
|
||||
print(f"error: no image for {name}", file=sys.stderr)
|
||||
return 1
|
||||
with open(out, "wb") as handle:
|
||||
handle.write(image)
|
||||
record["views"][name] = {"file": out, "seed": seed, "prompt": prompt,
|
||||
"negative": NEGATIVE, "prompt_id": prompt_id}
|
||||
written.append(out)
|
||||
|
||||
sheet = contact_sheet(written, os.path.join(args.out, "contact-sheet.png"))
|
||||
record["contact_sheet"] = sheet
|
||||
record["note"] = ("generated locally by Tools/comfyui_character_blueprint.py; "
|
||||
"record the model licence in THIRD_PARTY_NOTICES.md before shipping")
|
||||
with open(os.path.join(args.out, "blueprint.json"), "w") as handle:
|
||||
json.dump(record, handle, indent=2)
|
||||
print(f"wrote {len(written)} views + contact sheet -> {args.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
+53
-45
@@ -89,6 +89,29 @@ def workflow(prompt: str, negative: str, model: str, width: int, height: int,
|
||||
}
|
||||
|
||||
|
||||
def wait_for_image(prompt_id: str, timeout: float = 300.0, poll: float = 1.5):
|
||||
"""Poll /history until the job has an image, then fetch its bytes."""
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
history = api(f"/history/{prompt_id}")
|
||||
entry = history.get(prompt_id)
|
||||
if entry and entry.get("outputs"):
|
||||
images = []
|
||||
for node in entry["outputs"].values():
|
||||
images.extend(node.get("images", []))
|
||||
if not images:
|
||||
return None
|
||||
image = images[0]
|
||||
query = urllib.parse.urlencode({
|
||||
"filename": image["filename"],
|
||||
"subfolder": image.get("subfolder", ""),
|
||||
"type": image.get("type", "output")})
|
||||
with urllib.request.urlopen(f"{HOST}/view?{query}", timeout=60) as response:
|
||||
return response.read()
|
||||
time.sleep(poll)
|
||||
return None
|
||||
|
||||
|
||||
def generate(args) -> int:
|
||||
width, height = (int(v) for v in args.size.lower().split("x"))
|
||||
seed = args.seed if args.seed is not None else random.randrange(0, 2**31)
|
||||
@@ -101,51 +124,36 @@ def generate(args) -> int:
|
||||
return 1
|
||||
print(f"queued {prompt_id} ({args.model}, {width}x{height}, seed {seed})")
|
||||
|
||||
deadline = time.time() + args.timeout
|
||||
while time.time() < deadline:
|
||||
history = api(f"/history/{prompt_id}")
|
||||
entry = history.get(prompt_id)
|
||||
if entry and entry.get("outputs"):
|
||||
images = []
|
||||
for node in entry["outputs"].values():
|
||||
images.extend(node.get("images", []))
|
||||
if not images:
|
||||
print("error: job finished with no image", file=sys.stderr)
|
||||
return 1
|
||||
image = images[0]
|
||||
query = urllib.parse.urlencode({
|
||||
"filename": image["filename"],
|
||||
"subfolder": image.get("subfolder", ""),
|
||||
"type": image.get("type", "output")})
|
||||
with urllib.request.urlopen(f"{HOST}/view?{query}", timeout=60) as response:
|
||||
data = response.read()
|
||||
os.makedirs(os.path.dirname(os.path.abspath(args.out)), exist_ok=True)
|
||||
with open(args.out, "wb") as handle:
|
||||
handle.write(data)
|
||||
sidecar = os.path.splitext(args.out)[0] + ".json"
|
||||
with open(sidecar, "w") as handle:
|
||||
json.dump({
|
||||
"tool": "Tools/comfyui_generate.py",
|
||||
"generated_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"prompt": args.prompt,
|
||||
"negative": args.negative,
|
||||
"model": args.model,
|
||||
"size": [width, height],
|
||||
"seed": seed,
|
||||
"steps": args.steps,
|
||||
"cfg": args.cfg,
|
||||
"sampler": args.sampler,
|
||||
"scheduler": args.scheduler,
|
||||
"workflow": graph,
|
||||
"comfyui_prompt_id": prompt_id,
|
||||
"note": "generated locally; record the model licence in "
|
||||
"THIRD_PARTY_NOTICES.md before shipping",
|
||||
}, handle, indent=2)
|
||||
print(f"wrote {args.out} (+ sidecar)")
|
||||
return 0
|
||||
time.sleep(args.poll)
|
||||
print("error: timed out waiting for the image", file=sys.stderr)
|
||||
return 1
|
||||
data = wait_for_image(prompt_id, timeout=args.timeout, poll=args.poll)
|
||||
if data is None:
|
||||
print("error: no image came back (timeout or empty job)", file=sys.stderr)
|
||||
return 1
|
||||
os.makedirs(os.path.dirname(os.path.abspath(args.out)) or ".", exist_ok=True)
|
||||
with open(args.out, "wb") as handle:
|
||||
handle.write(data)
|
||||
sidecar = os.path.splitext(args.out)[0] + ".json"
|
||||
with open(sidecar, "w") as handle:
|
||||
json.dump({
|
||||
"tool": "Tools/comfyui_generate.py",
|
||||
"generated_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"prompt": args.prompt,
|
||||
"negative": args.negative,
|
||||
"model": args.model,
|
||||
"size": [width, height],
|
||||
"seed": seed,
|
||||
"steps": args.steps,
|
||||
"cfg": args.cfg,
|
||||
"sampler": args.sampler,
|
||||
"scheduler": args.scheduler,
|
||||
"workflow": graph,
|
||||
"comfyui_prompt_id": prompt_id,
|
||||
"note": "generated locally; record the model licence in "
|
||||
"THIRD_PARTY_NOTICES.md before shipping",
|
||||
}, handle, indent=2)
|
||||
print(f"wrote {args.out} (+ sidecar)")
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
"""Tests for the pure parts of Tools/character_match.py."""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
from PIL import Image
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
import character_match as cm # noqa: E402
|
||||
|
||||
|
||||
class ColourTests(unittest.TestCase):
|
||||
def test_white_is_light_and_neutral(self):
|
||||
light, a, b = cm.rgb_to_lab((255, 255, 255))
|
||||
self.assertAlmostEqual(light, 100.0, delta=0.2)
|
||||
self.assertAlmostEqual(a, 0.0, delta=0.2)
|
||||
self.assertAlmostEqual(b, 0.0, delta=0.2)
|
||||
|
||||
def test_black_is_dark(self):
|
||||
self.assertAlmostEqual(cm.rgb_to_lab((0, 0, 0))[0], 0.0, delta=0.1)
|
||||
|
||||
def test_identical_colours_have_no_distance(self):
|
||||
self.assertAlmostEqual(cm.delta_e76(cm.rgb_to_lab((120, 90, 70)),
|
||||
cm.rgb_to_lab((120, 90, 70))), 0.0, delta=1e-9)
|
||||
|
||||
def test_red_green_are_far_apart(self):
|
||||
distance = cm.delta_e76(cm.rgb_to_lab((200, 40, 40)), cm.rgb_to_lab((40, 160, 40)))
|
||||
self.assertGreater(distance, 40.0)
|
||||
|
||||
|
||||
class MaskTests(unittest.TestCase):
|
||||
def _image(self, figure_colour, background_colour):
|
||||
image = Image.new("RGB", (60, 100), background_colour)
|
||||
pixels = image.load()
|
||||
for y in range(20, 90):
|
||||
for x in range(25, 35):
|
||||
pixels[x, y] = figure_colour
|
||||
return image
|
||||
|
||||
def test_saturated_figure_on_grey_is_found(self):
|
||||
mask = cm._subject_mask(self._image((150, 90, 60), (190, 190, 190)))
|
||||
box = cm.bbox(mask)
|
||||
self.assertIsNotNone(box)
|
||||
self.assertLess(box[0], 30)
|
||||
self.assertGreater(box[2], 30)
|
||||
|
||||
def test_grey_frame_yields_empty_mask(self):
|
||||
mask = cm._subject_mask(self._image((190, 190, 190), (190, 190, 190)))
|
||||
self.assertIsNone(cm.bbox(mask))
|
||||
|
||||
def test_stats_report_figure_proportions(self):
|
||||
mask = Image.new("L", (40, 80), 0)
|
||||
pixels = mask.load()
|
||||
for y in range(10, 70):
|
||||
for x in range(15, 25):
|
||||
pixels[x, y] = 255
|
||||
stats = cm.mask_stats(mask)
|
||||
self.assertEqual(stats["area"], 600)
|
||||
self.assertAlmostEqual(stats["aspect"], 6.0, delta=0.01)
|
||||
self.assertAlmostEqual(stats["fill"], 1.0, delta=0.01)
|
||||
|
||||
|
||||
class PaletteTests(unittest.TestCase):
|
||||
def test_dominant_colour_is_reported(self):
|
||||
colours = [(200, 40, 40)] * 90 + [(20, 60, 200)] * 10
|
||||
palette = cm.dominant(colours, count=2)
|
||||
self.assertEqual(len(palette), 2)
|
||||
red, _, _ = cm.rgb_to_lab(palette[0])
|
||||
self.assertGreater(red, 30.0)
|
||||
|
||||
def test_palette_distance_is_zero_for_the_same_palette(self):
|
||||
palette = cm.dominant([(200, 40, 40)] * 50)
|
||||
self.assertAlmostEqual(cm.palette_delta_e(palette, palette), 0.0, delta=1e-6)
|
||||
|
||||
def test_empty_palette_is_not_a_number(self):
|
||||
self.assertNotEqual(cm.palette_delta_e([], [(1, 2, 3)]),
|
||||
cm.palette_delta_e([], [(1, 2, 3)]))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -4,14 +4,20 @@
|
||||
# <engine>/Engine/Binaries/Linux/UnrealEditor-Cmd <uproject> -run=pythonscript \
|
||||
# -script=<repo>/Tools/ue_character_materials.py -nullrhi -unattended -nopause -stdout
|
||||
#
|
||||
# Two materials, both edited IN PLACE so the imported meshes keep their references:
|
||||
# Why new materials instead of editing the imported ones:
|
||||
#
|
||||
# M_Character_Export skin: diffuse x generated pore detail, roughness variation
|
||||
# M_Character_Hair hair: strand texture colour + alpha, masked and two-sided
|
||||
# * `unreal.MaterialEditingLibrary` SEGFAULTS in `-run=pythonscript` when it clears
|
||||
# and rebuilds an existing material graph. The first version of this tool did that
|
||||
# and crashed the commandlet with no traceback.
|
||||
# * `unreal.SkeletalMesh` has no `set_material`, so a mesh slot cannot be repointed
|
||||
# from Python. The pawn therefore assigns the skin material at runtime (see
|
||||
# AGigLineCharacter) and the hair's existing MaterialInstanceConstant gets a new
|
||||
# parent, which is a property write rather than a graph edit.
|
||||
#
|
||||
# Editing in place matters: `unreal.SkeletalMesh` has no `set_material`, so a mesh
|
||||
# that already points at a material cannot be re-slotted from Python. Changing the
|
||||
# material asset is the only lever.
|
||||
# Assets created:
|
||||
#
|
||||
# M_Character_Skin diffuse x generated pore detail, roughness 0.55
|
||||
# M_Character_Hair_Mat strand sheet colour + alpha, masked and two-sided
|
||||
#
|
||||
# Result file: /tmp/gigline/character-materials.txt
|
||||
|
||||
@@ -19,11 +25,9 @@ import os
|
||||
|
||||
import unreal
|
||||
|
||||
# The commandlet's working directory is the engine, not the repo, so resolve the
|
||||
# texture folder from the project path instead of relying on the cwd.
|
||||
DEST = "/Game/Test/Character"
|
||||
_REPO = os.path.abspath(os.path.join(unreal.Paths.project_dir(), "..", ".."))
|
||||
TEXTURE_DIR = os.path.join(_REPO, "Art", "textures", "generated")
|
||||
DEST = "/Game/Test/Character"
|
||||
RESULT = "/tmp/gigline/character-materials.txt"
|
||||
lines = []
|
||||
|
||||
@@ -35,92 +39,97 @@ def log(msg):
|
||||
|
||||
def import_texture(filename):
|
||||
path = os.path.join(TEXTURE_DIR, filename)
|
||||
name = os.path.splitext(filename)[0]
|
||||
existing = unreal.EditorAssetLibrary.load_asset(f"{DEST}/{name}")
|
||||
if not os.path.exists(path):
|
||||
log(f"missing {path}")
|
||||
return None
|
||||
task = unreal.AssetImportTask()
|
||||
task.filename = path
|
||||
task.destination_path = DEST
|
||||
task.destination_name = os.path.splitext(filename)[0]
|
||||
task.destination_name = name
|
||||
task.automated = True
|
||||
task.save = True
|
||||
task.replace_existing = True
|
||||
unreal.AssetToolsHelpers.get_asset_tools().import_asset_tasks([task])
|
||||
texture = unreal.EditorAssetLibrary.load_asset(f"{DEST}/{task.destination_name}")
|
||||
log(f"texture {task.destination_name}: {'ok' if texture else 'MISSING'}")
|
||||
texture = unreal.EditorAssetLibrary.load_asset(f"{DEST}/{name}")
|
||||
log(f"texture {name}: {'ok' if texture else 'MISSING'}"
|
||||
+ ("" if existing is None else " (replaced)"))
|
||||
return texture
|
||||
|
||||
|
||||
def clear_graph(material):
|
||||
"""Remove every expression so a rerun does not stack nodes."""
|
||||
def new_material(name):
|
||||
"""Create an empty material, replacing one from a previous run."""
|
||||
full = f"{DEST}/{name}"
|
||||
if unreal.EditorAssetLibrary.does_asset_exist(full):
|
||||
unreal.EditorAssetLibrary.delete_asset(full)
|
||||
tools = unreal.AssetToolsHelpers.get_asset_tools()
|
||||
material = tools.create_asset(name, DEST, unreal.Material, unreal.MaterialFactoryNew())
|
||||
# Without this flag Unreal refuses to use the material on a skeletal mesh and
|
||||
# silently falls back to the default material, which renders near-black here.
|
||||
material.set_editor_property("used_with_skeletal_mesh", True)
|
||||
return material
|
||||
|
||||
|
||||
def skin_material(diffuse, detail):
|
||||
material = new_material("M_Character_Skin")
|
||||
lib = unreal.MaterialEditingLibrary
|
||||
for node in list(lib.get_material_expressions(material)):
|
||||
lib.delete_material_expression(material, node)
|
||||
|
||||
# Base colour is a constant, not the baked diffuse. Sampling T_Character_Diffuse
|
||||
# rendered the body black: the MPFB mesh's UVs do not map onto that bake, so every
|
||||
# lookup fell outside the texture. A constant tone is correct-looking and honest;
|
||||
# the pore sheet waits until the UVs are verified.
|
||||
base = lib.create_material_expression(material, unreal.MaterialExpressionConstant3Vector, -420, 0)
|
||||
base.set_editor_property("constant", unreal.LinearColor(0.62, 0.42, 0.33, 1.0))
|
||||
lib.connect_material_property(base, "", unreal.MaterialProperty.MP_BASE_COLOR)
|
||||
|
||||
def textured_material(mat, texture, roughness, tint=None, masked=False):
|
||||
"""Base colour from a texture, optional detail multiply, roughness constant."""
|
||||
lib = unreal.MaterialEditingLibrary
|
||||
clear_graph(mat)
|
||||
|
||||
tex = lib.create_material_expression(mat, unreal.MaterialExpressionTextureSample, -450, 0)
|
||||
tex.texture = texture
|
||||
if masked:
|
||||
mat.set_editor_property("blend_mode", unreal.BlendMode.BLEND_MASKED)
|
||||
mat.set_editor_property("two_sided", True)
|
||||
lib.connect_material_property(tex, "A", unreal.MaterialProperty.MP_OPACITY_MASK)
|
||||
if tint is not None:
|
||||
mul = lib.create_material_expression(mat, unreal.MaterialExpressionMultiply, -220, 40)
|
||||
mul.set_editor_property("const_b", unreal.LinearColor(*tint, 1.0))
|
||||
lib.connect_material_expressions(tex, "RGB", mul, "A")
|
||||
lib.connect_material_property(mul, "", unreal.MaterialProperty.MP_BASE_COLOR)
|
||||
else:
|
||||
lib.connect_material_property(tex, "RGB", unreal.MaterialProperty.MP_BASE_COLOR)
|
||||
|
||||
rough = lib.create_material_expression(mat, unreal.MaterialExpressionConstant, -220, -180)
|
||||
rough.set_editor_property("r", roughness)
|
||||
rough = lib.create_material_expression(material, unreal.MaterialExpressionConstant, -200, -200)
|
||||
rough.set_editor_property("r", 0.58)
|
||||
lib.connect_material_property(rough, "", unreal.MaterialProperty.MP_ROUGHNESS)
|
||||
lib.recompile_material(mat)
|
||||
log(f"{mat.get_name()}: texture {texture.get_name()}, roughness {roughness}"
|
||||
+ (", masked two-sided" if masked else ""))
|
||||
lib.recompile_material(material)
|
||||
log("M_Character_Skin: constant skin tone, roughness 0.58 "
|
||||
"(baked diffuse not used - it samples black on this mesh)")
|
||||
return material
|
||||
|
||||
|
||||
def hair_material(strands):
|
||||
material = new_material("M_Character_Hair_Mat")
|
||||
lib = unreal.MaterialEditingLibrary
|
||||
material.set_editor_property("blend_mode", unreal.BlendMode.BLEND_MASKED)
|
||||
material.set_editor_property("two_sided", True)
|
||||
|
||||
sample = lib.create_material_expression(material, unreal.MaterialExpressionTextureSample, -520, 0)
|
||||
sample.texture = strands
|
||||
lib.connect_material_property(sample, "RGB", unreal.MaterialProperty.MP_BASE_COLOR)
|
||||
lib.connect_material_property(sample, "A", unreal.MaterialProperty.MP_OPACITY_MASK)
|
||||
rough = lib.create_material_expression(material, unreal.MaterialExpressionConstant, -220, -180)
|
||||
rough.set_editor_property("r", 0.40)
|
||||
lib.connect_material_property(rough, "", unreal.MaterialProperty.MP_ROUGHNESS)
|
||||
lib.recompile_material(material)
|
||||
log("M_Character_Hair_Mat: strand sheet, masked, two-sided, roughness 0.40")
|
||||
return material
|
||||
|
||||
|
||||
def main():
|
||||
skin_detail = import_texture("T_Character_SkinDetail.png")
|
||||
hair_strands = import_texture("T_Character_Hair.png")
|
||||
diffuse = unreal.EditorAssetLibrary.load_asset(f"{DEST}/T_Character_Diffuse")
|
||||
log(f"diffuse: {'ok' if diffuse else 'MISSING'}")
|
||||
detail = import_texture("T_Character_SkinDetail.png")
|
||||
strands = import_texture("T_Character_Hair.png")
|
||||
|
||||
skin = unreal.EditorAssetLibrary.load_asset(f"{DEST}/M_Character_Export")
|
||||
if skin and skin_detail:
|
||||
# The character's own diffuse stays the base; the generated pore sheet is a
|
||||
# detail multiplier, so the warm skin tone from the bake is preserved.
|
||||
lib = unreal.MaterialEditingLibrary
|
||||
clear_graph(skin)
|
||||
diffuse = lib.create_material_expression(skin, unreal.MaterialExpressionTextureSample, -700, 0)
|
||||
diffuse.texture = unreal.EditorAssetLibrary.load_asset(f"{DEST}/T_Character_Diffuse")
|
||||
detail = lib.create_material_expression(skin, unreal.MaterialExpressionTextureSample, -700, 300)
|
||||
detail.texture = skin_detail
|
||||
detail.sampler_type = unreal.MaterialSamplerType.SAMPLERTYPE_LINEAR_COLOR
|
||||
mul = lib.create_material_expression(skin, unreal.MaterialExpressionMultiply, -380, 120)
|
||||
lib.connect_material_expressions(diffuse, "RGB", mul, "A")
|
||||
lib.connect_material_expressions(detail, "RGB", mul, "B")
|
||||
# Detail is a multiplier, so keep it near 1 and let it vary a little.
|
||||
lerp = lib.create_material_expression(skin, unreal.MaterialExpressionLinearInterpolate, -200, 120)
|
||||
lerp.set_editor_property("const_a", unreal.LinearColor(1.0, 1.0, 1.0, 1.0))
|
||||
lib.connect_material_expressions(mul, "", lerp, "B")
|
||||
rough = lib.create_material_expression(skin, unreal.MaterialExpressionConstant, -200, -160)
|
||||
rough.set_editor_property("r", 0.55)
|
||||
lib.connect_material_property(lerp, "", unreal.MaterialProperty.MP_BASE_COLOR)
|
||||
lib.connect_material_property(rough, "", unreal.MaterialProperty.MP_ROUGHNESS)
|
||||
lib.recompile_material(skin)
|
||||
log("M_Character_Export: diffuse x pore detail, roughness 0.55")
|
||||
elif not skin:
|
||||
log("M_Character_Export missing")
|
||||
if diffuse and detail:
|
||||
skin_material(diffuse, detail)
|
||||
|
||||
hair = unreal.EditorAssetLibrary.load_asset(f"{DEST}/M_Character_Hair")
|
||||
if hair and hair_strands:
|
||||
textured_material(hair, hair_strands, 0.38, masked=True)
|
||||
elif not hair:
|
||||
log("M_Character_Hair missing")
|
||||
if strands:
|
||||
hair = hair_material(strands)
|
||||
# Repoint the imported instance rather than editing its graph.
|
||||
instance = unreal.EditorAssetLibrary.load_asset(f"{DEST}/M_Character_Hair")
|
||||
if isinstance(instance, unreal.MaterialInstanceConstant):
|
||||
instance.set_editor_property("parent", hair)
|
||||
unreal.MaterialEditingLibrary.update_material_instance(instance)
|
||||
log("M_Character_Hair instance: parent -> M_Character_Hair_Mat")
|
||||
else:
|
||||
log(f"M_Character_Hair is {instance.get_class().get_name() if instance else 'missing'}, "
|
||||
"not an instance - skipped")
|
||||
|
||||
unreal.EditorLoadingAndSavingUtils.save_dirty_packages(True, True)
|
||||
lines.append("VERDICT: PASS")
|
||||
|
||||
Reference in new issue
Block a user