Define release-candidate balance acceptance targets
This commit is contained in:
1 parent
f48d87c84f
commit
46f1e41874
4 files changed
+132
-3
No files matched your search
@@ -4,6 +4,14 @@ Session memory for durable decisions, state, and gotchas. Keep concise, remove o
|
||||
|
||||
## Decisions
|
||||
|
||||
- **0.10.0 balance protocol:** `docs/balance-acceptance.md` defines targets and
|
||||
denominator/cohort rules for the 36 venue/difficulty/phase cells, seeds 1–100,
|
||||
access/emergency error rates and matched calm vs emergency-first comparisons.
|
||||
These are acceptance criteria, not measured results; sweeps/playtests remain open.
|
||||
Inspection found `_read_npc_truth()` is type-derived, not behavior-derived;
|
||||
the pending emergency fairness review must cover this oracle mismatch rather
|
||||
than treating internal truth as independent evidence.
|
||||
|
||||
- The visual-alignment path/loader/panel/HUD work and the realism lighting pass are complete and
|
||||
recorded in `DONE.md`; remaining visual items are the realism follow-ups in `TODO.md`.
|
||||
- **Character mesh cap (2026-11-02):** the eight library variants now use **8,000 triangles /
|
||||
|
||||
@@ -49,7 +49,7 @@ release-candidate foundation, not yet a 0.10.0 release candidate.
|
||||
refinement** and **0.9.6 integration** passes are complete (all moved to
|
||||
`DONE.md`). The **0.10.0 release candidate** is next: content lock, external
|
||||
playtest, balance, save migration and release/legal sign-off. The source version
|
||||
is 0.9.6; the v0.9.6 release is published.
|
||||
is 0.10.0-rc.1; the v0.9.6 release is published.
|
||||
|
||||
## Milestone 0.10.0 — Release candidate
|
||||
|
||||
@@ -79,12 +79,17 @@ verify every distribution path. This is the next minor release after the 0.9.x p
|
||||
|
||||
### Balance and fairness pass
|
||||
|
||||
- [ ] Define measurable targets for routine compliance, access-decision accuracy, peaceful
|
||||
- [x] Define measurable targets for routine compliance, access-decision accuracy, peaceful
|
||||
resolution, justified emergency calls and false-positive/false-negative rates.
|
||||
(`docs/balance-acceptance.md` defines cohorts, denominators, 36 matrix cells,
|
||||
seeds, thresholds and matched emergency-first comparisons. Documentation verified;
|
||||
targets are not measured results.)
|
||||
- [ ] Run scripted venue sweeps across easy/normal/hard and all four phases; compare outcomes and
|
||||
tune event weights, reaction tables, compliance and report strictness where targets fail.
|
||||
- [ ] Review emergency calls against actual danger: no routine behavior should make emergency the
|
||||
dominant winning move, and genuine threats must remain answerable.
|
||||
dominant winning move, and genuine threats must remain answerable. Include the
|
||||
type-derived `_read_npc_truth()` mismatch with observed threats/forced entry;
|
||||
use independently labelled behavior evidence, not the helper as its own oracle.
|
||||
- [ ] Verify the reputation grades, debrief explanations and campaign progression still communicate
|
||||
the de-escalation-first design after tuning.
|
||||
- [ ] Record the balance evidence and any deliberate exceptions in `docs/development-status.md`
|
||||
|
||||
@@ -17,6 +17,7 @@ This folder holds the project documentation, visual references and generated QA
|
||||
- [`accessibility.md`](accessibility.md) — accessibility statement and known gaps
|
||||
- [`known-limitations.md`](known-limitations.md) — release-candidate blockers and accepted limits
|
||||
- [`smoke-checklist.md`](smoke-checklist.md) — manual verification
|
||||
- [`balance-acceptance.md`](balance-acceptance.md) — release-candidate balance targets and evidence protocol
|
||||
- [`../CREDITS.md`](../CREDITS.md) — credits and asset provenance
|
||||
- [`../PRIVACY.md`](../PRIVACY.md) — privacy note
|
||||
- [`ui-copy.md`](ui-copy.md) — screen, copy and button inventory
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
# Release-candidate balance acceptance
|
||||
|
||||
These are the acceptance targets for the 0.10.0 balance pass, not measured results.
|
||||
No reaction tables, event weights or scoring rules were changed to set them. The
|
||||
[roadmap](../TODO.md) tracks the sweeps and external playtests still needed.
|
||||
[Gameplay](gameplay.md) owns the mechanics; this page owns their balance-test protocol.
|
||||
|
||||
## Test matrix and evidence
|
||||
|
||||
Run Hollywood, Joke Shop and TV Studio on easy, normal and hard, in Early,
|
||||
Midday, Evening and Late: **36 cells**. Use seeds 1–100 for each cell and retain
|
||||
those seeds for before/after comparisons. Exercise each eligible authored event
|
||||
at least once as a separate coverage pass; a random schedule can miss rare events.
|
||||
Record scheduled event counts separately from forced coverage cases so the latter
|
||||
do not distort event-frequency estimates.
|
||||
|
||||
For reaction measurements, use a fresh controller at the entrance for each sample.
|
||||
Routine cases are tourists, confused visitors and delivery workers obstructing the
|
||||
entrance without threatening behavior. Do not include authorized entry requests in
|
||||
this redirect cohort. For peaceful-resolution measurements, send one polite
|
||||
direction (L1), then one warning (L2) only if still unresolved. Do not reroll until
|
||||
success. Observe return behavior for 60 simulated seconds before closing the case.
|
||||
Do not issue emergency calls as part of this policy.
|
||||
|
||||
For access cases, follow the venue's required inspection and screening steps before
|
||||
allow/deny. Include valid, expired, counterfeit, mismatched, missing and ambiguous
|
||||
artifacts, known recognition, age boundaries, screening refusal, benign wand alerts
|
||||
and prohibited items. Recognition must be independently justified by the fixture;
|
||||
calling a stranger recognized is not evidence of correct admission. An unresolved
|
||||
hold counts as unresolved, not as a correct refusal.
|
||||
|
||||
Evidence must identify the commit, engine, venue, difficulty, phase, seed, event ID,
|
||||
cohort, visible evidence, policy action, reaction/state, return, access truth and
|
||||
decision, emergency truth and call, report fields, reputation modifiers, final grade
|
||||
and unresolved cases. Save raw rows plus numerator/denominator summaries for every
|
||||
cell. Use at least **100 eligible cases per metric per cell**; if the schedule
|
||||
provides fewer, add a separately labelled forced-case sample. Report zero eligible
|
||||
cases as **N/A**, never as 100% success. Keep scheduled and forced results separate.
|
||||
|
||||
## Targets
|
||||
|
||||
Percentages below apply to each matrix cell, not only a pooled average. Scripted
|
||||
access/report cases use complete evidence and are held to exact correctness;
|
||||
first-time-player results measure whether that evidence is understandable.
|
||||
|
||||
| Metric | Numerator / denominator | Acceptance target |
|
||||
| --- | --- | --- |
|
||||
| First-direction routine compliance | First L1 reaction is `COMPLY` or `LEAVE` / eligible routine redirect cases | Easy ≥80%; normal ≥75%; hard ≥65% |
|
||||
| Routine peaceful resolution | Routine cases leaving after at most L1 then L2, with no return during 60 s and no L3–L5 / eligible routine redirect cases | Easy ≥90%; normal ≥85%; hard ≥80% |
|
||||
| Access accuracy | Correct final allow/deny decisions / all eligible access cases (including unresolved cases) | Scripted 100%; first-time players ≥90% pooled, with each error reviewed |
|
||||
| Access false positive | Unauthorized people allowed / unauthorized access cases | Scripted 0%; first-time players ≤5% pooled |
|
||||
| Access false negative | Authorized people denied / authorized access cases | Scripted 0%; first-time players ≤10% pooled |
|
||||
| Emergency precision | Calls supported by immediate-danger evidence / all emergency calls | Scripted 100%; first-time players ≥95% pooled; zero calls is N/A |
|
||||
| Emergency false positive | Routine/no-immediate-danger cases receiving an emergency call / all routine/no-immediate-danger cases | Scripted 0%; first-time players ≤5% pooled |
|
||||
| Emergency false negative | Immediate-danger cases without an accepted emergency call within 15 simulated seconds of visible danger / all immediate-danger cases with radio initially available | Scripted 0%; first-time players ≤10% pooled |
|
||||
| Report accuracy | Reports with correct weapon and danger fields / all submitted reports | Scripted 100%; first-time players ≥90% pooled |
|
||||
|
||||
For access, authorized means entitled under the venue policy **after required
|
||||
screening**, without a prohibited item or an independent immediate-danger reason
|
||||
to refuse. Compare to a hand-labelled fixture expectation, not only to
|
||||
`AccessVerifier.verify()`: using the verifier as both decision and oracle would
|
||||
hide policy bugs. Report unresolved access cases separately; they cannot improve
|
||||
either error rate by being omitted from accuracy.
|
||||
|
||||
Immediate danger means an observed weapon, assault, active forced entry or a
|
||||
credible immediate threat to occupants. Argument, filming, intoxication, a crisis,
|
||||
a refused screen or a benign wand alert alone is not immediate danger. A mental
|
||||
health label or clothing must never be used as the emergency oracle. The 15 s
|
||||
window is a test-response target, not the responder-arrival time (currently 15 s
|
||||
after an accepted emergency call). Separately test depleted battery and active
|
||||
cooldown cases for safe containment/assistance; do not silently drop them from the
|
||||
evidence or claim the initially-available-radio target covers them.
|
||||
|
||||
## Scoring and emergency dominance
|
||||
|
||||
Compare the calm policy with an emergency-first policy on the same routine events
|
||||
and seeds. Each comparison starts with fresh radio, crowd and reputation state.
|
||||
An accurate report about harmless behavior must not make its emergency call
|
||||
justified. Require an unjustified-call investigation and a strictly lower rating
|
||||
for emergency-first in every routine comparison. Inspect modifiers as well as the
|
||||
final rating: the 0–100 clamp can hide rewards/penalties. Emergency-first must not
|
||||
improve the campaign outcome or final grade. Grade boundaries remain A ≥90,
|
||||
B ≥75, C ≥60, D ≥40, otherwise F; these are existing rules, not new tuning.
|
||||
|
||||
In genuine-danger cases, verify that the call is accepted, responders arrive,
|
||||
the incident remains answerable and the debrief explains the outcome without an
|
||||
unjustified-call investigation. Also compare assistance/containment routes where
|
||||
appropriate; emergency need not be mandatory for every difficult interaction.
|
||||
|
||||
## Implementation cautions and sign-off
|
||||
|
||||
- `NPCController.roll_reaction()` samples per-level weights. Difficulty adds
|
||||
+12/0/−8 to the `COMPLY` weight before normalization; `base_compliance` alone
|
||||
is not the observed compliance percentage. Measure `COMPLY` plus `LEAVE`.
|
||||
- `GameManager._read_npc_truth()` currently derives weapon/danger truth from the
|
||||
NPC type (violent threat: weapon/3; everyone else: no weapon/1), rather than
|
||||
observed behavior. This is a balance-review risk: test nonviolent profiles that
|
||||
threaten or force entry, and violent profiles before visible danger. Do not
|
||||
treat agreement with this helper as proof of fair emergency judgment.
|
||||
- `RadioSystem.credibility_hits()` checks weapon and danger fields only;
|
||||
location, description and behavior still need human review. Difficulty scales
|
||||
report penalties, not the oracle.
|
||||
- A terminal peaceful reaction is not sufficient evidence of an incident staying
|
||||
resolved: returning NPCs and unresolved shift-review entries must be retained.
|
||||
|
||||
Do not tune to a pooled average that hides a failing venue, phase or difficulty.
|
||||
Record each miss, its severity, the raw evidence and any approved exception before
|
||||
changing weights. Re-run the affected cells and tutorial/campaign smokes after a
|
||||
fix. Five first-time-player sessions must cover the tutorial and a complete venue
|
||||
shift; report pooled counts and per-session errors, not statistical confidence
|
||||
from five players. Scripted sweeps do not substitute for those sessions.
|
||||
|
||||
**Status:** targets defined; sweeps, emergency fairness review, grade/campaign
|
||||
verification and external-player acceptance remain pending. No balance sign-off
|
||||
or release approval is implied by this document.
|
||||
Reference in new issue
Block a user