Define release-candidate balance acceptance targets
Full build / Import + headless tests (push) Successful in 43s
Full build / Full desktop build + optional release (push) Skipped

This commit is contained in:
zwitschi committed 2026-11-03 07:40:45 +01:00
1 parent f48d87c84f
commit 46f1e41874
4 files changed
+132 -3

No files matched your search

+8
View File
@@ -4,6 +4,14 @@ Session memory for durable decisions, state, and gotchas. Keep concise, remove o
## Decisions
- **0.10.0 balance protocol:** `docs/balance-acceptance.md` defines targets and
denominator/cohort rules for the 36 venue/difficulty/phase cells, seeds 1–100,
access/emergency error rates and matched calm vs emergency-first comparisons.
These are acceptance criteria, not measured results; sweeps/playtests remain open.
Inspection found `_read_npc_truth()` is type-derived, not behavior-derived;
the pending emergency fairness review must cover this oracle mismatch rather
than treating internal truth as independent evidence.
- The visual-alignment path/loader/panel/HUD work and the realism lighting pass are complete and
recorded in `DONE.md`; remaining visual items are the realism follow-ups in `TODO.md`.
- **Character mesh cap (2026-11-02):** the eight library variants now use **8,000 triangles /
+8 -3
View File
@@ -49,7 +49,7 @@ release-candidate foundation, not yet a 0.10.0 release candidate.
refinement** and **0.9.6 integration** passes are complete (all moved to
`DONE.md`). The **0.10.0 release candidate** is next: content lock, external
playtest, balance, save migration and release/legal sign-off. The source version
is 0.9.6; the v0.9.6 release is published.
is 0.10.0-rc.1; the v0.9.6 release is published.
## Milestone 0.10.0 — Release candidate
@@ -79,12 +79,17 @@ verify every distribution path. This is the next minor release after the 0.9.x p
### Balance and fairness pass
- [ ] Define measurable targets for routine compliance, access-decision accuracy, peaceful
- [x] Define measurable targets for routine compliance, access-decision accuracy, peaceful
resolution, justified emergency calls and false-positive/false-negative rates.
(`docs/balance-acceptance.md` defines cohorts, denominators, 36 matrix cells,
seeds, thresholds and matched emergency-first comparisons. Documentation verified;
targets are not measured results.)
- [ ] Run scripted venue sweeps across easy/normal/hard and all four phases; compare outcomes and
tune event weights, reaction tables, compliance and report strictness where targets fail.
- [ ] Review emergency calls against actual danger: no routine behavior should make emergency the
dominant winning move, and genuine threats must remain answerable.
dominant winning move, and genuine threats must remain answerable. Include the
type-derived `_read_npc_truth()` mismatch with observed threats/forced entry;
use independently labelled behavior evidence, not the helper as its own oracle.
- [ ] Verify the reputation grades, debrief explanations and campaign progression still communicate
the de-escalation-first design after tuning.
- [ ] Record the balance evidence and any deliberate exceptions in `docs/development-status.md`
+1
View File
@@ -17,6 +17,7 @@ This folder holds the project documentation, visual references and generated QA
- [`accessibility.md`](accessibility.md) — accessibility statement and known gaps
- [`known-limitations.md`](known-limitations.md) — release-candidate blockers and accepted limits
- [`smoke-checklist.md`](smoke-checklist.md) — manual verification
- [`balance-acceptance.md`](balance-acceptance.md) — release-candidate balance targets and evidence protocol
- [`../CREDITS.md`](../CREDITS.md) — credits and asset provenance
- [`../PRIVACY.md`](../PRIVACY.md) — privacy note
- [`ui-copy.md`](ui-copy.md) — screen, copy and button inventory
+115
View File
@@ -0,0 +1,115 @@
# Release-candidate balance acceptance
These are the acceptance targets for the 0.10.0 balance pass, not measured results.
No reaction tables, event weights or scoring rules were changed to set them. The
[roadmap](../TODO.md) tracks the sweeps and external playtests still needed.
[Gameplay](gameplay.md) owns the mechanics; this page owns their balance-test protocol.
## Test matrix and evidence
Run Hollywood, Joke Shop and TV Studio on easy, normal and hard, in Early,
Midday, Evening and Late: **36 cells**. Use seeds 1–100 for each cell and retain
those seeds for before/after comparisons. Exercise each eligible authored event
at least once as a separate coverage pass; a random schedule can miss rare events.
Record scheduled event counts separately from forced coverage cases so the latter
do not distort event-frequency estimates.
For reaction measurements, use a fresh controller at the entrance for each sample.
Routine cases are tourists, confused visitors and delivery workers obstructing the
entrance without threatening behavior. Do not include authorized entry requests in
this redirect cohort. For peaceful-resolution measurements, send one polite
direction (L1), then one warning (L2) only if still unresolved. Do not reroll until
success. Observe return behavior for 60 simulated seconds before closing the case.
Do not issue emergency calls as part of this policy.
For access cases, follow the venue's required inspection and screening steps before
allow/deny. Include valid, expired, counterfeit, mismatched, missing and ambiguous
artifacts, known recognition, age boundaries, screening refusal, benign wand alerts
and prohibited items. Recognition must be independently justified by the fixture;
calling a stranger recognized is not evidence of correct admission. An unresolved
hold counts as unresolved, not as a correct refusal.
Evidence must identify the commit, engine, venue, difficulty, phase, seed, event ID,
cohort, visible evidence, policy action, reaction/state, return, access truth and
decision, emergency truth and call, report fields, reputation modifiers, final grade
and unresolved cases. Save raw rows plus numerator/denominator summaries for every
cell. Use at least **100 eligible cases per metric per cell**; if the schedule
provides fewer, add a separately labelled forced-case sample. Report zero eligible
cases as **N/A**, never as 100% success. Keep scheduled and forced results separate.
## Targets
Percentages below apply to each matrix cell, not only a pooled average. Scripted
access/report cases use complete evidence and are held to exact correctness;
first-time-player results measure whether that evidence is understandable.
| Metric | Numerator / denominator | Acceptance target |
| --- | --- | --- |
| First-direction routine compliance | First L1 reaction is `COMPLY` or `LEAVE` / eligible routine redirect cases | Easy ≥80%; normal ≥75%; hard ≥65% |
| Routine peaceful resolution | Routine cases leaving after at most L1 then L2, with no return during 60 s and no L3–L5 / eligible routine redirect cases | Easy ≥90%; normal ≥85%; hard ≥80% |
| Access accuracy | Correct final allow/deny decisions / all eligible access cases (including unresolved cases) | Scripted 100%; first-time players ≥90% pooled, with each error reviewed |
| Access false positive | Unauthorized people allowed / unauthorized access cases | Scripted 0%; first-time players ≤5% pooled |
| Access false negative | Authorized people denied / authorized access cases | Scripted 0%; first-time players ≤10% pooled |
| Emergency precision | Calls supported by immediate-danger evidence / all emergency calls | Scripted 100%; first-time players ≥95% pooled; zero calls is N/A |
| Emergency false positive | Routine/no-immediate-danger cases receiving an emergency call / all routine/no-immediate-danger cases | Scripted 0%; first-time players ≤5% pooled |
| Emergency false negative | Immediate-danger cases without an accepted emergency call within 15 simulated seconds of visible danger / all immediate-danger cases with radio initially available | Scripted 0%; first-time players ≤10% pooled |
| Report accuracy | Reports with correct weapon and danger fields / all submitted reports | Scripted 100%; first-time players ≥90% pooled |
For access, authorized means entitled under the venue policy **after required
screening**, without a prohibited item or an independent immediate-danger reason
to refuse. Compare to a hand-labelled fixture expectation, not only to
`AccessVerifier.verify()`: using the verifier as both decision and oracle would
hide policy bugs. Report unresolved access cases separately; they cannot improve
either error rate by being omitted from accuracy.
Immediate danger means an observed weapon, assault, active forced entry or a
credible immediate threat to occupants. Argument, filming, intoxication, a crisis,
a refused screen or a benign wand alert alone is not immediate danger. A mental
health label or clothing must never be used as the emergency oracle. The 15 s
window is a test-response target, not the responder-arrival time (currently 15 s
after an accepted emergency call). Separately test depleted battery and active
cooldown cases for safe containment/assistance; do not silently drop them from the
evidence or claim the initially-available-radio target covers them.
## Scoring and emergency dominance
Compare the calm policy with an emergency-first policy on the same routine events
and seeds. Each comparison starts with fresh radio, crowd and reputation state.
An accurate report about harmless behavior must not make its emergency call
justified. Require an unjustified-call investigation and a strictly lower rating
for emergency-first in every routine comparison. Inspect modifiers as well as the
final rating: the 0–100 clamp can hide rewards/penalties. Emergency-first must not
improve the campaign outcome or final grade. Grade boundaries remain A ≥90,
B ≥75, C ≥60, D ≥40, otherwise F; these are existing rules, not new tuning.
In genuine-danger cases, verify that the call is accepted, responders arrive,
the incident remains answerable and the debrief explains the outcome without an
unjustified-call investigation. Also compare assistance/containment routes where
appropriate; emergency need not be mandatory for every difficult interaction.
## Implementation cautions and sign-off
- `NPCController.roll_reaction()` samples per-level weights. Difficulty adds
+12/0/−8 to the `COMPLY` weight before normalization; `base_compliance` alone
is not the observed compliance percentage. Measure `COMPLY` plus `LEAVE`.
- `GameManager._read_npc_truth()` currently derives weapon/danger truth from the
NPC type (violent threat: weapon/3; everyone else: no weapon/1), rather than
observed behavior. This is a balance-review risk: test nonviolent profiles that
threaten or force entry, and violent profiles before visible danger. Do not
treat agreement with this helper as proof of fair emergency judgment.
- `RadioSystem.credibility_hits()` checks weapon and danger fields only;
location, description and behavior still need human review. Difficulty scales
report penalties, not the oracle.
- A terminal peaceful reaction is not sufficient evidence of an incident staying
resolved: returning NPCs and unresolved shift-review entries must be retained.
Do not tune to a pooled average that hides a failing venue, phase or difficulty.
Record each miss, its severity, the raw evidence and any approved exception before
changing weights. Re-run the affected cells and tutorial/campaign smokes after a
fix. Five first-time-player sessions must cover the tutorial and a complete venue
shift; report pooled counts and per-session errors, not statistical confidence
from five players. Scripted sweeps do not substitute for those sessions.
**Status:** targets defined; sweeps, emergency fairness review, grade/campaign
verification and external-player acceptance remain pending. No balance sign-off
or release approval is implied by this document.