Repository navigation
Conversation
A reusable workflow (axiom-parity.yml) checks the axiom: line on every PR that changes parameters, variables or reforms: queued issues must exist, carry pe-parity and be dispatch-ready; encoded PRs must exist and be open or merged; encoded-correct must name a module on rulespec main and a companion test that outputs it; n/a needs an allowed category, and metadata/docs-tests claims are checked against the diff. The finder maps changed files' reference URLs to corpus citations, looks up RuleSpec modules and open pe-parity issues, and drafts a dispatch-ready issue per source document when nothing covers a provision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
drain-plan groups dispatch-ready pe-parity issues by source document, extracts each issue's target citations and pasteable review_finding, and prices a weekly wave from measured encoder run costs. corpus-pin lists the citations with an operative body in the release a rulespec clone pins, read from axiom-corpus with git. docs/drain.md is the drain design. Calibration against every policy PR merged since 2026-09-26 fixed: fenced axiom lines, bullet continuations, n/a notes on queued lines, dict() references, year-0000 parameter dates, and moved or misspelled references inside dated values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 10, 2026
…t's open issue A citation with no module and no issue of its own, in a source document that already has an open pe-parity issue, now suggests extending that issue instead of drafting a duplicate (seen on policyengine-us#10040, where rulespec-us#1626 covers 20 CFR 416.1202 and the PR also cites 416.1160). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dling The PR and the tool are checked out into separate directories, so files in a PR can't stand in for the tool. In warn mode a tooling failure is a warning, not a failed job; in enforce mode it exits 2, distinct from a rule failure. The finder writes nothing for PRs that change no policy files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A differential property test found the drain could extract a YAML-form review_finding (review_finding: |-) that the lint missed. The lint now uses the drain's extractor as a fallback, so anything the drain would paste counts as present, and a batch is ready only if every issue has pasteable text. The extractor also reads a cue after a bold step label. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…int and drain Check - A formula local named label/reference is code, not metadata; only class attributes are descriptive. Inert bare dict() expressions and date-keyed bracket values are normalized. - Every changed policy file is checked for an n/a metadata claim (no cap). - encoded-correct needs a companion test for every module it names, and an unresolved relative module (/22) blocks. - n/a is a claim wherever it sits, so it can't share a line with others. - Modal and future status words (might be queued, to be queued) are prose. Lint and drain - An element with a TODO is missing, companion tests need expected values, and a prose back-reference isn't the element: finder drafts never pass. - One extractor decides the review_finding for both the lint and the drain. - Every citation a finding names is a target; a batch needs pasteable text. - corpus-pin skips repealed rows and reads rulespec-uk's real corpus pin. Finder - Issues are read with their comments (one GraphQL query per 50 issues). - Dated-value references are read; dashes are normalized; the comment is only updated when a bot wrote it. - URL mapping gains legislation.gov.uk /id/ and nested schedules, IRS rev procs, eCFR part URLs, govinfo and 14 state code sites, and is tested against 427 corpus-checked URL vectors. Workflow: pull_request only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A shell guard on the maintainers' machines blocks any command line that contains find run from a broad directory, so axiom-parity find could not be run by the agents who need it. The workflow and docs use suggest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…approvals drain-plan --blocked-modules takes modules that must not be a target or an import. Its first use is the 17 rulespec-us modules that carry the obsolete module.source_verification.values field and the modules that reference them: a target importing one fails to compile. The scarce resource is the production-signing approval, and one signed run can carry up to 16 citations (MAX_SOURCE_BUNDLE_CITATIONS), so the plan counts runs per source document and the doc budgets approvals per week. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Check and parser
- Any module reference that resolves to nothing fails an encoded-correct
claim, however it is written (corpus path, legal id, path or sibling).
- A companion named later on the same axiom line counts; a bare test file
name is searched beside every cited module.
- Negation stops at its clause ("is not encoded but is queued"); a status
used as an adjective ("the queued rule") is not a claim; n/a after
"and" or a full stop is still a claim; a reference-less status joins the
next only when the statuses agree.
Lint and drain
- Placeholders are matched in any case; a companion case needs a stated
outcome, not a digit; negative judgments count.
- A review_finding cue must introduce the finding in its own sentence; a
fence tagged review_finding is one. Outputs named in the finding count.
- Targets are the labelled citation plus everything the finding names.
Finder and behaviour
- Sibling issues are found from every open issue's own citations.
- Comments past 100 are read; the REST fallback reads comments too.
- dict(...) is inert only when the module leaves the builtin alone.
- Dashes are normalized in URLs, claims and cited test paths.
docs/drain.md states the taxonomy's populations exactly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The encoder owner asked to pause manual paths until axiom-encode#1821 merges, so drain-plan takes --blocked-citation-pattern (repeatable). docs/drain.md records that Max approved the standing approval (d1262). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
drain-plan --blocked-citations reads {blocked: [{citation, match, kind,
reason}]}, where match is exact or exact-or-descendant, and blocks an issue
whose target matches. It complements the module list and the pattern block.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wnership
Lint (issue_lint.py)
- Input facts are not expected outcomes: an `input:` block and an
"input x: 0" clause are dropped before outcomes are looked for.
- A required output named in a finding must be declared as an output and
tied to the identifier. "Add an input named a_b_c" declares an input.
- A finding cue must hand the finding over now. Deferred, unavailable and
empty cues hand over nothing, and an inline cue takes only the block
right below it.
- review_findings() returns every finding in a text once. Braced paths
(a/b/{1,2}.yaml) are expanded before modules and citations are read.
Axiom line (axiom_line.py, check.py)
- A negation left hanging before an aside governs the status after it:
"is not (yet) queued", "is not, in fact, queued", "(not yet) queued".
- Bold n/a after a claim, and "the already queued rule", parse correctly.
- A companion test named right after a negation is not evidence.
Drain (drain.py, corpus_pin.py, cli.py)
- A batch is one source document, so no run mixes jurisdictions (the
encoder refuses such a bundle). An issue spanning documents is in each.
- The blocked-module filter checks the declared module, modules a finding
uses, the target's path and the modules above it, and, with
--rulespec-dir, the modules on main that own the target citation.
- Findings in comments add to the body's. Modules a finding asks for work
on become targets; imports, deferrals and precedents do not.
- A pin file marks repealed rows; a repealed target is not in the pin.
Finder (finder/cli.py, finder/propose.py)
- The finder updates only a comment its own account wrote that opens with
the marker. Issue text is dash-normalized before citations are matched.
Behaviour (behaviour.py)
- A star import may rebind dict, except the PolicyEngine model_api preamble.
docs/drain.md
- Four citations from the measured runs have a manifest on rulespec-us
main, not two; one is a GPT-6 Sol encode. The "no fresh model encode has
landed" claim is withdrawn.
- With every hold applied, 16 documents (20 citations, 16 runs) are ready,
not 25.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 4 found new phrasings that the rules misread. Rather than add a rule per phrasing, the rules now guess towards the safe side where they have to guess, and the README says what they cannot tell. Drain (drain.py) - A module a finding names is requested work unless the words around it mark an import, a deferral, a precedent or a description. Each marker covers its own clause. A numbered list of modules under one "Encode these:" line is now read (rulespec-uk#429). - A line that only pairs a module with a citation declares both (rulespec-us#1618). - A requested module resolves to the citation the issue gives for it or for the provision above it, else to the one its path spells (federal US and UK). With neither, the issue is held and the plan names the module. - A precedent or a described module is not at stake for the block list. - Backticked sibling names, several brace groups and incomplete paths are handled. A pin row may hold spaces. Lint (issue_lint.py) - A list of input facts is not an outcome: only an announced result ends an input clause, and a bare "key: 0" or "x is 0" never counts. A result after a colon counts after a described case. - A YAML case counts whatever the order of its keys; a comment inside a code block is not a heading. - "Add an input rule a_b_c" declares no output. - Headings and bold labels obey the same negation and deferral rules as inline cues. Back-to-back finding labels keep both findings. Axiom line (axiom_line.py, check.py) - Emphasis is stripped before parsing. A hanging negation skips its aside, and its reach ends at a hard break, another status word, or a reference after the aside. - A test ruled out inside brackets is not evidence. docs - With every hold applied, 16 documents (17 citations, 16 runs) are ready, completing 11 issues, not 15. 13 lint-ready issues are held because a requested module has no citation. - README: a Limits section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The encoder owner prepared the 12 US documents that passed the plan for
dispatch and found 2 dispatchable that day, 2 runnable after an issue
update, 5 waiting on axiom-encode#1819, 1 needing a ruling and 2 whose
finding relies on a form outside the pin.
- --blocked-citations is repeatable and reads the owner's verdict file
({documents: [{citation, verdict, reason, until}]}); every row holds
its citation.
- docs/drain.md says plainly that passing the plan is necessary, not
sufficient, with the verdict table. With the verdicts applied the plan
lists 4 UK documents.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The encoder owner keeps a list of assembled pipelines: modules composed
from other modules, where the change an issue asks for is a reviewed hand
edit, not an encode (axiom-encode#1723 for rulespec-us; an ordinary PR in
rulespec-uk).
- --work-holds reads {rule, modules: [{module, evidence}]}. An issue is
held when a finding, or a line pairing a module with a citation, asks
for work on a listed module. Importing from one is not held.
- The hold names the module and the owner's evidence, and the plan prints
the owner's rule once. It replaces the missing-citation reason for those
modules: naming a citation would not make them dispatchable.
On the backlog of 2026-10-10 it holds 9 lint-ready issues.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 5 found phrasings one step from fixed ones. Each is fixed, and a
new test module builds phrasings from their parts so a rule is tried
against every combination of markers, wrappers and wordings near the ones
it was written for. Those tests found three more gaps before review did.
Axiom line
- Emphasis is removed completely, however nested: "is not ***queued***".
- A bullet that says anything about a status never inherits one.
- A negation can hang before an aside after an auxiliary ("cannot be
(yet) queued") and before a dash aside ("is not — for now — queued").
Check
- A negation rules out every companion test in its reach: its own bracket
group, nested groups included, up to a colon, semicolon, sentence end
or status word.
Lint
- A name made an input by the words after it is not an output: "Add a
rule a_b_c as a new input", "Add `x` to the inputs".
- A label's remainder is checked for deferral whatever its length, and a
finding that starts on the label's line does not take a quote below it.
- A result table needs a filled-in result cell.
Drain
- A request in a module's own clause makes it work whatever describes it:
"encode the deferred X", "add it to the guarantee that X composes",
"encode X as a module". Description verbs are listed, not guessed.
- "1. module — citation" pairs are read like bulleted ones.
- programs/ modules are recognised; work on one with no citation is held.
docs
- Missing or repealed corpus text explains 8 failures, not 10.
- The backlog table and held-issue counts follow the new rules and the
encoder owner's assembled-pipelines list.
- README Limits lists the evidence and finding-label rules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s no signed route The encoder owner prepared all 16 documents that pass the plan. Four were dispatched (Colorado and three New Mexico sections). Five wait on axiom-encode#1819, three rely on a source outside the pin, and the four UK documents cannot be encoded: the signed workflow reads .axiom/workflow-toolchain.toml, which rulespec-uk lacks, and rulespec-uk's layout test allows no encoding manifest. The document now carries that table and the UK finding, and the plan command holds every uk citation until the route is decided. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Six rounds found phrasings one step from fixed ones, most of them in how a
status word is read. The parser no longer guesses at negations.
Axiom line
- A status is claimed only when it follows its reference or legal
citation directly (separators, closing marks and is/are/now/already and
a few such words may stand between), or opens its clause, and nothing
before it in its piece denies or conditions it. "is not [yet] queued",
"isn’t queued" and "Not in Axiom yet, rulespec-us#1416 queued" claim
nothing, whatever their punctuation. Each status is judged in its own
piece, so a denied claim keeps its references to itself.
- A bullet list inherits "The rest is queued:" only when nothing in the
lead denies it.
- A status in prose with a reference after it is noted, not claimed.
- Curly apostrophes, Markdown links and comma citations ("43-1041(A),(H)")
are read before a status.
Check
- Test references are read whole, so "file.test.yaml::case" doesn't end a
negation's reach.
Lint
- An output named in a finding counts only in a closed set of declared
shapes; a name followed by any input designation ("as a scoped input")
is not one.
- A finding on its label's line is none when its first sentence defers
it, however long.
- Cells under an input column are inputs. "(1)" and "(a)" are list
markers.
Drain
- Root-first program paths, emphasis around paths, "repair the existing
X" and "add the import to X" read as work on X.
Tests: grids over the words between a reference and its status, denials
before it, input designations, deferred label lengths, input columns,
module layouts, wrappers and modifiers, list numbering, repeated
references and Markdown links. 268 tests.
Replay: UK 60 of 81, US 87 of 173 policy PRs pass (was 58 and 87).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
force-pushed
the
axiom-parity-check
branch
from
October 11, 2026 16:37
6113400 to
d09c652
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This makes the PolicyEngine→Axiom parity rule (CONTRIBUTING, "Mirror policy changes in Axiom") checked and assisted rather than manual, for every country model.
What's in it
1. A reusable PR check (
.github/workflows/axiom-parity.yml, packageaxiom-parity/). Country repos call it onpull_request.<package>/parameters/,variables/orreforms/. Markdown under those paths doesn't count.axiom:line is resolved against GitHub:queued: the issue exists in aTheAxiomFoundation/rulespec-*repo, carriespe-parity, is open (or closed as completed), and passes a dispatch-ready lint over its body and comments. The lint needs six things: module path, corpus citation (not a corpus release file), verbatim law, required outputs, a pasteablereview_finding, and companion tests.encoded: a rulespec PR that is open or merged. It warns when the PR carries no encoder manifest.encoded-correct: every module named exists on rulespecmain, and so does any#rulecited. For each module, a cited companion case or file exists and outputs a rule of that module.n/a: an allowed category (below), as the PR's only claim. Formetadataanddocs-tests, no changed policy file may change a parameter value or formula; every file is checked. Parameter YAML is compared with descriptive keys removed, and Python by AST with docstrings and class-level labels removed.needed: never passes.;,|, several lines, or a bullet list after a colon). Every claim must pass.mode: warnreports in the job summary and annotations without failing.mode: enforcefails the job.2. A parity finder (
axiom-parity suggest). It runs in the same workflow and keeps one comment on the PR while the check fails.referenceURLs on the changed parameters and variables and maps them to corpus citation paths. The mapping covers:pe-parityissues.encoded-correctcandidates together with the companion cases that output each module.3. A drain design (
axiom-parity/docs/drain.md), with two tools:corpus-pinlists the citations the encoder could read in the corpus release a rulespec clone pins: rows with an operative body. Rows marked repealed are listed apart, because the encoder refuses them. It readsaxiom-corpuswith git, using the encoder owner's procedure.drain-planextracts each issue's target citations andreview_finding(every finding block, body first, then comments), files each target under its source document, and prices a wave from measured run costs.main), held citations and its dispatch-preparation verdicts, source patterns, and assembled pipelines, where the change an issue asks for is a hand edit that a signed run does not make.CONTRIBUTING now lists the n/a categories and the issue headings, and names the check.
Allowed n/a categories (please check this list)
The rule names
infra,data,ui,microsim-onlyandemulator-mapping. Three more cover policy-path PRs that change no provision of law:docs-tests: documentation and tests;metadata: labels, references and units;not-law: contributed reforms, proposals, and announcements not yet legislated.The replay below shows these three accounted for most n/a PRs. A reason saying "Axiom doesn't encode this yet" is rejected unless an explicit category is given, because that is what
queuedis for. Three US PRs used exactly that reason: policyengine-us#9667, #9656 and #9588.Evidence
Replay over every PR merged since 2026-09-26 (
scripts/replay_history.py, diffing each merge commit locally):needed3, n/a mixed with other claims 2, n/a category 2queuedwith no issue 6, n/a category 5dict(title=, href=)references;0000-01-01parameter dates;metadata;maintoday, so a few old claims pass only because a module landed later.Finder on the three open UK PRs the earlier replay session checked by hand (#2240–#2242). It reproduces those findings exactly:
uk:statutes/ukpga/2004/12/190, with casesearnings_above_basic_amountandearnings_below_basic_amount.The check passes on #2240's merge ref: s.190 is verified against its module and both cases, and rulespec-uk#444 is dispatch-ready.
Backlog (
axiom-parity backlog):review_finding; 60 of them are among the 71 filed on 2026-10-09..axiom/workflow-toolchain.toml, which rulespec-uk lacks, and rulespec-uk's layout test allows no encoding manifest. The encoder owner has queued that route question for Max.drain-planreads those verdicts back as holds. With them and the UK hold applied, it lists no document that is ready and not already dispatched.Encoder costs (
axiom-parity/docs/drain.md), from the 453targeted-signed-reencoderuns between 2026-09-01 and 10-10:main. One is a GPT-6 Sol encode from 2026-10-01. An earlier version of the drain document said two citations and no fresh model encode; it had read only the top-level record of each manifest file.Invariants (property-tested with Hypothesis)
axiom:prefix.review_finding, the lint counts.tests/test_neighbourhoods.py): each free-text rule is tried over every combination of the markers, wrappers and wordings near the ones it was written for. Examples are 14 negations × 12 asides × 49 emphasis pairs for a denied status, and 8 list markers × 6 separators for a module-and-citation line.There are 268 tests in all, run by
.github/workflows/axiom-parity-tool.yml. actionlint is clean.Review
Independent GPT-6.1 Sol reviews, each with reproductions:
505822f) requested changes: 4 P1 and 9 P2 findings.b94b1bc) requested changes: 2 P1 and 10 P2 findings, plus 5 false positives in a 30-PR sample.9f04a68) requested changes: 6 P1 findings, 8 P2 and 1 P3. It reproduced the replay, lint, pin and cost figures, and found the five earlier false positives fixed.f858521) requested changes: 5 P1 findings and 6 P2. It reproduced every figure and found no misjudged PR in a fresh 20-PR sample.d787a40) requested changes: 8 P1 findings and 1 P2. It reproduced every figure and found no misjudged PR in a fresh 20-PR sample.6269fb0) requested changes: 8 P1 findings and 2 P2, most of them again in how status words were read.d09c652the parser stops guessing at negations. A status counts only in the canonical shape: right after its reference or citation (only separators, closing marks and a few words like "is" between), or opening its clause, with no denying word before it. A phrasing the grammar doesn't know is not a claim. A property test checks that for any words between a reference and a status, a claim is made exactly when every one is allowed.d09c652with a regression test.d09c652.Rollout
mode: warn: Run the Axiom parity check on pull requests (warn-only) policyengine-uk#2249, Run the Axiom parity check on pull requests (warn-only) policyengine-us#10048, Run the Axiom parity check on pull requests (warn-only) policyengine-canada#558, Run the Axiom parity check on pull requests (warn-only) policyengine-il#12 and Run the Axiom parity check on pull requests (warn-only) policyengine-ng#28. They also update or add the PR templates' axiom line. The uk and us callers point at this branch, so their runs exercised the workflow end to end; both ran green.mode: enforce.policyengine-nz isn't listed in the rule. Its hub can opt in with the same five-line caller.
axiom: n/a: infra: CI tooling for the parity rule; no country-model policy changes
🤖 Generated with Claude Code