diff --git a/MEMORY.md b/MEMORY.md index 6493603..0b2a1d4 100644 --- a/MEMORY.md +++ b/MEMORY.md @@ -4,6 +4,14 @@ Session memory for durable decisions, state, and gotchas. Keep concise, remove o ## Decisions +- **0.10.0 balance protocol:** `docs/balance-acceptance.md` defines targets and + denominator/cohort rules for the 36 venue/difficulty/phase cells, seeds 1–100, + access/emergency error rates and matched calm vs emergency-first comparisons. + These are acceptance criteria, not measured results; sweeps/playtests remain open. + Inspection found `_read_npc_truth()` is type-derived, not behavior-derived; + the pending emergency fairness review must cover this oracle mismatch rather + than treating internal truth as independent evidence. + - The visual-alignment path/loader/panel/HUD work and the realism lighting pass are complete and recorded in `DONE.md`; remaining visual items are the realism follow-ups in `TODO.md`. - **Character mesh cap (2026-11-02):** the eight library variants now use **8,000 triangles / diff --git a/TODO.md b/TODO.md index 5ccd605..e420908 100644 --- a/TODO.md +++ b/TODO.md @@ -49,7 +49,7 @@ release-candidate foundation, not yet a 0.10.0 release candidate. refinement** and **0.9.6 integration** passes are complete (all moved to `DONE.md`). The **0.10.0 release candidate** is next: content lock, external playtest, balance, save migration and release/legal sign-off. The source version -is 0.9.6; the v0.9.6 release is published. +is 0.10.0-rc.1; the v0.9.6 release is published. ## Milestone 0.10.0 — Release candidate @@ -79,12 +79,17 @@ verify every distribution path. This is the next minor release after the 0.9.x p ### Balance and fairness pass -- [ ] Define measurable targets for routine compliance, access-decision accuracy, peaceful +- [x] Define measurable targets for routine compliance, access-decision accuracy, peaceful resolution, justified emergency calls and false-positive/false-negative rates. + (`docs/balance-acceptance.md` defines cohorts, denominators, 36 matrix cells, + seeds, thresholds and matched emergency-first comparisons. Documentation verified; + targets are not measured results.) - [ ] Run scripted venue sweeps across easy/normal/hard and all four phases; compare outcomes and tune event weights, reaction tables, compliance and report strictness where targets fail. - [ ] Review emergency calls against actual danger: no routine behavior should make emergency the - dominant winning move, and genuine threats must remain answerable. + dominant winning move, and genuine threats must remain answerable. Include the + type-derived `_read_npc_truth()` mismatch with observed threats/forced entry; + use independently labelled behavior evidence, not the helper as its own oracle. - [ ] Verify the reputation grades, debrief explanations and campaign progression still communicate the de-escalation-first design after tuning. - [ ] Record the balance evidence and any deliberate exceptions in `docs/development-status.md` diff --git a/docs/README.md b/docs/README.md index 45e99aa..f2396f2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -17,6 +17,7 @@ This folder holds the project documentation, visual references and generated QA - [`accessibility.md`](accessibility.md) — accessibility statement and known gaps - [`known-limitations.md`](known-limitations.md) — release-candidate blockers and accepted limits - [`smoke-checklist.md`](smoke-checklist.md) — manual verification +- [`balance-acceptance.md`](balance-acceptance.md) — release-candidate balance targets and evidence protocol - [`../CREDITS.md`](../CREDITS.md) — credits and asset provenance - [`../PRIVACY.md`](../PRIVACY.md) — privacy note - [`ui-copy.md`](ui-copy.md) — screen, copy and button inventory diff --git a/docs/balance-acceptance.md b/docs/balance-acceptance.md new file mode 100644 index 0000000..09c79b2 --- /dev/null +++ b/docs/balance-acceptance.md @@ -0,0 +1,115 @@ +# Release-candidate balance acceptance + +These are the acceptance targets for the 0.10.0 balance pass, not measured results. +No reaction tables, event weights or scoring rules were changed to set them. The +[roadmap](../TODO.md) tracks the sweeps and external playtests still needed. +[Gameplay](gameplay.md) owns the mechanics; this page owns their balance-test protocol. + +## Test matrix and evidence + +Run Hollywood, Joke Shop and TV Studio on easy, normal and hard, in Early, +Midday, Evening and Late: **36 cells**. Use seeds 1–100 for each cell and retain +those seeds for before/after comparisons. Exercise each eligible authored event +at least once as a separate coverage pass; a random schedule can miss rare events. +Record scheduled event counts separately from forced coverage cases so the latter +do not distort event-frequency estimates. + +For reaction measurements, use a fresh controller at the entrance for each sample. +Routine cases are tourists, confused visitors and delivery workers obstructing the +entrance without threatening behavior. Do not include authorized entry requests in +this redirect cohort. For peaceful-resolution measurements, send one polite +direction (L1), then one warning (L2) only if still unresolved. Do not reroll until +success. Observe return behavior for 60 simulated seconds before closing the case. +Do not issue emergency calls as part of this policy. + +For access cases, follow the venue's required inspection and screening steps before +allow/deny. Include valid, expired, counterfeit, mismatched, missing and ambiguous +artifacts, known recognition, age boundaries, screening refusal, benign wand alerts +and prohibited items. Recognition must be independently justified by the fixture; +calling a stranger recognized is not evidence of correct admission. An unresolved +hold counts as unresolved, not as a correct refusal. + +Evidence must identify the commit, engine, venue, difficulty, phase, seed, event ID, +cohort, visible evidence, policy action, reaction/state, return, access truth and +decision, emergency truth and call, report fields, reputation modifiers, final grade +and unresolved cases. Save raw rows plus numerator/denominator summaries for every +cell. Use at least **100 eligible cases per metric per cell**; if the schedule +provides fewer, add a separately labelled forced-case sample. Report zero eligible +cases as **N/A**, never as 100% success. Keep scheduled and forced results separate. + +## Targets + +Percentages below apply to each matrix cell, not only a pooled average. Scripted +access/report cases use complete evidence and are held to exact correctness; +first-time-player results measure whether that evidence is understandable. + +| Metric | Numerator / denominator | Acceptance target | +| --- | --- | --- | +| First-direction routine compliance | First L1 reaction is `COMPLY` or `LEAVE` / eligible routine redirect cases | Easy ≥80%; normal ≥75%; hard ≥65% | +| Routine peaceful resolution | Routine cases leaving after at most L1 then L2, with no return during 60 s and no L3–L5 / eligible routine redirect cases | Easy ≥90%; normal ≥85%; hard ≥80% | +| Access accuracy | Correct final allow/deny decisions / all eligible access cases (including unresolved cases) | Scripted 100%; first-time players ≥90% pooled, with each error reviewed | +| Access false positive | Unauthorized people allowed / unauthorized access cases | Scripted 0%; first-time players ≤5% pooled | +| Access false negative | Authorized people denied / authorized access cases | Scripted 0%; first-time players ≤10% pooled | +| Emergency precision | Calls supported by immediate-danger evidence / all emergency calls | Scripted 100%; first-time players ≥95% pooled; zero calls is N/A | +| Emergency false positive | Routine/no-immediate-danger cases receiving an emergency call / all routine/no-immediate-danger cases | Scripted 0%; first-time players ≤5% pooled | +| Emergency false negative | Immediate-danger cases without an accepted emergency call within 15 simulated seconds of visible danger / all immediate-danger cases with radio initially available | Scripted 0%; first-time players ≤10% pooled | +| Report accuracy | Reports with correct weapon and danger fields / all submitted reports | Scripted 100%; first-time players ≥90% pooled | + +For access, authorized means entitled under the venue policy **after required +screening**, without a prohibited item or an independent immediate-danger reason +to refuse. Compare to a hand-labelled fixture expectation, not only to +`AccessVerifier.verify()`: using the verifier as both decision and oracle would +hide policy bugs. Report unresolved access cases separately; they cannot improve +either error rate by being omitted from accuracy. + +Immediate danger means an observed weapon, assault, active forced entry or a +credible immediate threat to occupants. Argument, filming, intoxication, a crisis, +a refused screen or a benign wand alert alone is not immediate danger. A mental +health label or clothing must never be used as the emergency oracle. The 15 s +window is a test-response target, not the responder-arrival time (currently 15 s +after an accepted emergency call). Separately test depleted battery and active +cooldown cases for safe containment/assistance; do not silently drop them from the +evidence or claim the initially-available-radio target covers them. + +## Scoring and emergency dominance + +Compare the calm policy with an emergency-first policy on the same routine events +and seeds. Each comparison starts with fresh radio, crowd and reputation state. +An accurate report about harmless behavior must not make its emergency call +justified. Require an unjustified-call investigation and a strictly lower rating +for emergency-first in every routine comparison. Inspect modifiers as well as the +final rating: the 0–100 clamp can hide rewards/penalties. Emergency-first must not +improve the campaign outcome or final grade. Grade boundaries remain A ≥90, +B ≥75, C ≥60, D ≥40, otherwise F; these are existing rules, not new tuning. + +In genuine-danger cases, verify that the call is accepted, responders arrive, +the incident remains answerable and the debrief explains the outcome without an +unjustified-call investigation. Also compare assistance/containment routes where +appropriate; emergency need not be mandatory for every difficult interaction. + +## Implementation cautions and sign-off + +- `NPCController.roll_reaction()` samples per-level weights. Difficulty adds + +12/0/−8 to the `COMPLY` weight before normalization; `base_compliance` alone + is not the observed compliance percentage. Measure `COMPLY` plus `LEAVE`. +- `GameManager._read_npc_truth()` currently derives weapon/danger truth from the + NPC type (violent threat: weapon/3; everyone else: no weapon/1), rather than + observed behavior. This is a balance-review risk: test nonviolent profiles that + threaten or force entry, and violent profiles before visible danger. Do not + treat agreement with this helper as proof of fair emergency judgment. +- `RadioSystem.credibility_hits()` checks weapon and danger fields only; + location, description and behavior still need human review. Difficulty scales + report penalties, not the oracle. +- A terminal peaceful reaction is not sufficient evidence of an incident staying + resolved: returning NPCs and unresolved shift-review entries must be retained. + +Do not tune to a pooled average that hides a failing venue, phase or difficulty. +Record each miss, its severity, the raw evidence and any approved exception before +changing weights. Re-run the affected cells and tutorial/campaign smokes after a +fix. Five first-time-player sessions must cover the tutorial and a complete venue +shift; report pooled counts and per-session errors, not statistical confidence +from five players. Scripted sweeps do not substitute for those sessions. + +**Status:** targets defined; sweeps, emergency fairness review, grade/campaign +verification and external-player acceptance remain pending. No balance sign-off +or release approval is implied by this document.