[developer-ainative]

Story 12.1 — Nightly symmetry audit (8 pts, Must, Iteration 7) · closes #74

Feature file: backend/tests/features/symmetry_audit.feature

As a methodology owner, I want mirrored synthetic candidates and profiles scored every night, so that any partisan skew is caught.
Acceptance: Fails if mirrored scores differ by more than 0.5; results are written to audit_runs; a failure blocks the next release.

---

Why this is not already done

tests/test_alignment_formula_unit.py::test_the_score_is_antisymmetric_under_flipping_every_position (story 6.4) already proves the formula is antisymmetric: mirror every s_i and a score of x becomes exactly 100 − x. That test is right, and it is deliberately all it tests — four hand-written numbers per term, no database.

It is blind to almost everything a voter's number actually passes through:

Component	What it does	Visible to the formula test?
positions.compute_position	votes → signed s_value, via vote-type weights + omnibus discount	no
positions._stance_direction	which way a Yea points for this stance	no
service._terms_for	tier weights from the published methodology, intensity from the profile	no
scoring.badges	the confidence badge	no
scoring.dealbreakers	the warning shown above the score	no
scoring.reasons	the three explanations a voter reads	no

Six of those seven are newer than that test. "Nonpartisan by construction" is a claim about the whole pipeline, so this audit drives the whole pipeline: it writes a mirrored synthetic fixture into storage, recomputes positions through the real PositionComputer, scores both sides through the real AlignmentService, and compares the real results.

What "mirrored" means, exactly

Two independent sign flips exist in this pipeline, and the fixture exercises both — a bug in one would be invisible to a fixture that only flipped the other.

1. The candidate's record (mirrored_position): L votes Yea on every roll call R votes Nay on. Same roll calls, same vote_type, same is_bundled, same topics. _vote_direction is the only code reading position, so s_value negates.
2. The voter's stance (mirrored_direction_code): the mirrored voter picks the opposite stance card on the same topic (direction_code +1 vs −1), at the same tier and intensity. _stance_direction is the only code reading direction_code, so s_value negates again.

That gives two relationships a nonpartisan pipeline must satisfy, both asserted per pair:

• flip 1 alone → complementary: score(voter, L) + score(voter, R) == 100 — the end-to-end form of the formula's antisymmetry, now with real stored positions and real published tier weights in the path.
• flips 1+2 → identical: score(voter, L) == score(mirrored voter, R) — s_value double-negates. This is prd.md's own BDD scenario, word for word: "each voter's score for their aligned candidate is identical within 0.5 points."

Both are measured in points; the worse of the two is the pair's delta, so max_delta is one number against one threshold rather than two that could disagree.

It also fails on four things that are not scores

The acceptance criterion names the scores, and a literal reading would stop there. Each of these is a way to present one direction of the axis as better than the other while the numbers stay perfectly symmetric:

• the confidence badge — a mirrored record rests on exactly as much evidence, so a different badge tells a voter one side is better evidenced;
• the evidence count — the badge's input, checked separately so the two are distinguishable in triage;
• the reasons — the same interests must be named, and each must have moved the two candidates in opposite directions (a reason that "raised" both means one of the two explanations a voter reads is wrong);
• the dealbreaker warning — exactly one side of a mirrored dealbreaker pair can be in violation; both or neither means the check is not reading the sign.

This is not theoretical. I sabotaged four real pipeline components and confirmed each is caught — and two of the four leave max_delta at exactly 0.0, so a score-only audit would have passed them:

Sabotage (real component, not the formula)	Caught	max_delta
positions._vote_direction — Nay weighted 0.8 vs Yea's 1.0	yes	10.0
service._terms_for — tier weight doubled for positive positions	yes	17.3
reasons.top_reasons — direction always "raised"	yes	0.0
dealbreakers — the check disabled entirely	yes	0.0

The real live run

backend/scripts/verify_symmetry_audit_live.py, against the live informed project, real methodology 2026.3, real published tier weights {primary: 3.0, secondary: 1.5, nice_to_have: 0.5}:

symmetry_audit: PASS — 3 mirrored pair(s), max delta 0.0000 (tolerance 0.5), 0 asymmetry(ies), methodology 2026.3

pair 'unanimous-record'       L=100.0              R=0.0                L+R=100.0  mirrored voter=100.0              delta=0.0
pair 'weighted-and-mixed'     L=49.04632152588556  R=50.95367847411444  L+R=100.0  mirrored voter=49.04632152588556  delta=0.0
pair 'dealbreaker'            L=14.285714285714285 R=85.71428571428572  L+R=100.0  mirrored voter=14.285714285714285 delta=0.0

No asymmetry bug was found. Three things about that result are worth more than the headline:

• the delta is exactly 0.0, not merely under 0.5. L + R sums to 100.0 in IEEE754 for all three pairs, including the non-round 49.046…/50.954… pair, so the tolerance is headroom and not something the pipeline is currently consuming. The live verifier asserts == 0.0 for that reason: a nonzero delta that stayed inside 0.5 would mean something is diverging and happening to hide.
• the live numbers are bit-identical to the in-memory ones, so nothing drifts in the float round-trip through real ZeroDB — a real risk given story 12.2's finding that live tier weights can come back as strings.
• weighted-and-mixed scoring 49.046… rather than something round is the fixture doing its job: that pair carries all three tiers, mixed Yea/Nay, a procedural vote, a bundled vote and an abstention, so the vote-type weights, the omnibus discount and present contributing nothing are all in the measured path.

The run finished green on every check (all live checks passed). It also confirmed, against the real API:

• audit_runs accepts this row shape. Nothing had ever written to that table — its columns existed in TABLE_SCHEMAS and no code used one. Live row: {'id': 'aud:2026-10-20', 'passed': True, 'run_at': '2026-10-20T11:30:12Z', 'max_delta': 0.0, 'report_file_id': None, 'methodology_version': '2026.3'}
• max_delta round-trips as a real float and passed as a real bool — not strings. This is the round-trip that matters most: a string "false" is truthy, and the release gate reads that field.
• the row carries exactly datamodel.md's five documented fields plus id, and no stance_id, tier, intensity, interests, vp: or payload.
• the failing path works live. An 8-point skew injected into one side was caught against the real store, and wrote its own row with passed=False:

  weighted-and-mixed: the two candidates' scores are 57.04632152588556 and 50.95367847411444,
                      summing to 108.0 rather than 100 (delta 8.0000 > 0.5)
  weighted-and-mixed: the voter scored its aligned candidate 57.04632152588556 but the mirrored
                      voter scored its aligned candidate 49.04632152588556 (delta 8.0000 > 0.5)
  

• the release gate blocked and then allowed, reading real rows:

  symmetry_gate: BLOCK the next release, from the recorded audit_runs row; 1 violation(s)
  symmetry_gate: ALLOW the next release, from a fresh audit; 0 violation(s)
  symmetry_gate: ALLOW the next release, from the recorded audit_runs row; 0 violation(s)
  

  That is the whole acceptance criterion — "a failure blocks the next release" — demonstrated against live ZeroDB rather than asserted.
• cleanup is complete. Re-read id by id, then re-checked independently afterwards: 0 synthetic rows remain in any of the eleven touched tables, including the fabricated candidate_positions (the set that matters most), and no live row links a synthetic voter to a candidate. Exactly one audit_runs row is left, which is the deliberate keep.

The CI gate also ran for real on this branch and behaved as designed — self-check first, then the real check:

symmetry_audit: FAIL — 3 mirrored pair(s), max delta 8.0000 (tolerance 0.5), 4 asymmetry(ies)
self-check passed: a 8.0-point skew is correctly caught (max delta 8.0000 > 0.5)
symmetry_audit: PASS — 3 mirrored pair(s), max delta 0.0000 (tolerance 0.5), 0 asymmetry(ies)
PASS — every mirrored pair is symmetric within 0.5 points; the release is not blocked

Determinism was checked separately over 12 repeat runs: one distinct max_delta (0.0), bit-identical scores, identical reason orders. No flakiness risk from float ordering.

"A failure blocks the next release"

backend/scripts/check_symmetry_audit.py, following story 12.5's check_panel_gate.py shape rather than inventing a second release-blocking mechanism, and wired into ci.yml on every push and PR.

One genuine difference from the panel gate, and it is in this gate's favour: check_panel_gate --fake can only prove its own logic, because its real question is about deployed data. This gate's question is "is the scoring code symmetric?", and the code in CI is the code that ships — the fixture is synthetic by design, so there is no live data for it to be missing. The in-memory run therefore answers the real question completely.

It self-checks its failing path first: an 8-point skew is injected and MUST be caught, or the script exits 1 without reporting a result. A neutrality audit that rubber-stamps a skewed pipeline is worse than no audit, so a broken comparison fails loudly instead of going quietly green.

prd.md's pending Neutrality scenario is retired

tests/features/neutrality.feature's "Symmetry audit" scenario was xfailed against story 11.1, which was a mis-numbering twice over: that scenario is 12.1, and 11.1 is the verdict engine. Its steps now drive the real audit rather than common_steps.py's CandidateBuilder/VoterProfileBuilder sketches — a PRD acceptance scenario asserting on the builders would be asserting that the test fixture is symmetric, which was never prd.md's claim.

Files

File
app/neutrality/audit.py	the mirrored fixture, the auditor, the report, the gate (new package)
app/repositories/models.py	AuditRun — audit_runs had no model
app/scheduling/jobs.py	run_nightly_symmetry_audit, nightly, preconditioned on the profile root key
scripts/check_symmetry_audit.py	the release gate, self-checking
scripts/verify_symmetry_audit_live.py	the live run
.github/workflows/ci.yml	the gate, every push and PR
.github/workflows/scheduled-jobs.yml	ZERODB_PROFILE_ROOT_KEY for the new job
tests/features/symmetry_audit.feature + step defs	16 scenarios
tests/test_symmetry_audit_unit.py	44 unit tests

4026 tests pass; app.neutrality.audit coverage is 93%, above CLAUDE.md's 90% bar for scoring logic.

Judgment calls

• The audit writes synthetic data; nothing else in the job registry does. Auditing the real pipeline end to end cannot be done without putting something through it. Every row is labelled SYMMETRY-AUDIT in its own id and deleted in a finally (including on failure) — a fabricated candidate_positions row is a claim about how a named person voted, which under "evidence over opinion" is the worst thing this codebase could leave in its database. The audit_runs row is the one deliberate keep.
• Cleanup sweeps by marker, not just by the tracked list. Noticed while watching the live run: finally covers an exception but not a SIGKILL, and a multi-minute nightly job against a ~60 req/min ceiling is a realistic candidate for a CI timeout or an OOM. So cleanup does two passes — the tracked list, then a marker sweep over all eleven tables the fixture writes — and the next night's run removes an interrupted run's orphans. The audit self-heals rather than accumulating one interrupted run's worth of fabricated voting record at a time. _is_synthetic is scoped to the three id-bearing fields (id, person_id, entity_id), never the whole serialised row: a real stance card or roll-call question containing the words "symmetry audit" would match a whole-row check, and a sweep that can delete real data is far worse than one that occasionally misses a synthetic row — the next run catches a missed row, nothing brings back a deleted roll call.
• audit_runs is not append-only, unlike verdicts and candidate_positions. Those two must stay reproducible because someone can be shown the verdict that was in force when they read it; an audit run measures code, not a person, and datamodel.md's aud:{date} id has no room for a second run per day anyway. CLAUDE.md's append-only rule names three tables specifically and this is none of them.
• An empty run fails. An audit that scored nothing has proven nothing, and a pass there is how a neutrality check becomes decoration. Likewise an unscoreable pair reports inf, not 0.0 — two Nones comparing equal would report a perfect run over a fixture that produced nothing.
• direction_code 0 is refused a mirror. A stance that does not move along its axis is its own reflection and compute_position declines to score it, so a 0 pair could never produce two comparable scores.
• Both candidates get the same party string. A differing party is the one thing a nonpartisan scorer must not be able to see; giving the fixture one would make a passing audit less meaningful.
• The skew hook. Its only production value is 0.0 and no non-test caller passes anything else. It exists because otherwise the only way to see this check fail would be to introduce a real partisan bug — and it is applied after the real pipeline, so the sabotage can never be what the pipeline is tested through.
• One module, not two. audit.py is 1512 lines / 389 statements, which is heavily documented but in line with this codebase (positions.py is 975, jobs.py 880). The gate is the separable part, and panel_review does split gate.py from service.py — but that gate reads a ledger, where this one reads the report type defined beside it, so splitting would buy a file boundary at the cost of an import that goes straight back. Happy to split it if a reviewer prefers the symmetry with 12.5's layout.

Open questions for human review

1. It does not score real candidates yet. prd.md says "against all candidates" and that half is not built — not out of shyness: real candidate_positions do not exist in the live project at all, because roll_calls.topic_ids is [] until story 5.5 classifies roll calls. An audit over the real roster today would score every candidate None and report a vacuous pass, which is worse than not running. What this proves is that the pipeline is symmetric, which is what "no party-specific rules" names. Scoring the real roster is a follow-up gated on 5.5.
2. A symmetric skew is invisible to this audit, by design — a pipeline scoring everyone 10 points low passes, because both sides of the mirror move together. That is a calibration bug, not a partisan one, and conflating the two would make failures hard to read. If calibration needs a check, it is a different one.
3. report_file_id is always None. datamodel.md gives audit_runs that field and a stored report file would be the right home for per-pair detail, but writing one needs zerodb_upload_file and a retention decision for audit artefacts that no story has made. Per-pair detail is in the log line and the gate's violations today.
4. The nightly job needs ZERODB_PROFILE_ROOT_KEY in the prod Environment, and it must be the same key the deployed API uses (the keyring derives per-user data keys from it). Documented in scheduled-jobs.yml alongside the two existing missing-key gaps. Until it exists the job reports blocked, not failed — and note the release-blocking promise is kept today by the CI step, which needs no credentials at all.
5. BUNDLED_WEIGHT_FACTOR is still a module constant, not published methodology. Pre-existing debt flagged in story 6.2's own docstring; this audit now depends on it being symmetric (it is — it is a multiplier applied identically to both sides), which is one more reason to move it into methodology_versions.

🤖 Generated with Claude Code