Skip to content

perf(readers): read each module once per reader for diff --application's import-search checks (#1046) - #1137

Merged
pengfei-threemoonslab merged 10 commits into
mainfrom
claude/application-reader-speed
Nov 14, 2026
Merged

pengfei-threemoonslab merged 10 commits into
mainfrom
claude/application-reader-speed

Conversation

@pengfei-threemoonslab

@pengfei-threemoonslab pengfei-threemoonslab commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

diff --application became 7–13x slower with #964, and four development-corpus members exceed run.py's 900 s cap. Most of the time went to re-reading the same modules for every import-search question #964's constructor-ownership checks ask. This PR makes each reader read a module once for those questions, and gives identical answers (answer_id and every row) on all 20 members sampled, including the four that exceeded the cap and now finish in 435–768 s. The full development corpus confirms it: all 49 answers are identical to main's, and run.py exits 0 with every member under the cap. The slowest member takes 575 s, and member time falls from 7,982 s to 3,705 s (see Corpus comparison).

Refs #1046 (part of its run-scoped caching; not its 300-module budget or its BuilderCalls counter, so it stays open).

Symptom

Seconds per member from saved run.py runs (runs.json): 10-08 main before #964 (out-964-main) → #964's head (out-964-pr) → the 1.3.0 release build 04fc9ced (out-cut-04fc9ced, load ~10; its src/ is byte-identical to main). Summed over the 49 members: 1,374 s → 7,383 s → 7,982 s.

Member 10-08 main #964 head release
VidulaWickramasinghe/O.R.I.O.N#126 99 timeout timeout
alliance-genome/agr_ai_curation#842 / #858 / #860 157 / 177 / 208 timeout timeout
Lling0000/OpenCMO#41 56 670 757
liauto-siada/siada-cli#6 47 425 462
Tiendat2703/MIS_TALENT#7 23 133 168
SponsioLabs/Sponsio#173 17 115 141
ITMO-NSS-team/FEDOT.MAS#36 11 39 43

Profile and cause

cProfile of SponsioLabs/Sponsio#173 (diff --application, pinned base/head) on 10-08 main (c1c0691153, the parent of #964's merge), main (26bfa733b3) and this PR. The machine was shared, so the wall-clock columns are inflated by load (noted); the call counts are deterministic and are the comparison.

10-08 main main this PR
profiled seconds (load) 51 (37–51) 521 (50–97) 282 (41–69)
function calls 158 M 2,363 M 501 M
ast.walk calls 4.2 M 137.6 M 16.5 M
_import_search_context: calls, cumulative s — 318, 401 s (77%) 318, 58 s
_type_checking_only calls 18 28,840 1,576
_import_guarded calls 18 28,296 1,036
_imported_paths calls 131 217,193 11,207
_attribute_patches calls 96 1,054 385

Cause, all introduced by #964 (5a22373, "read returned tool factories with constructor ownership checks"; _import_search_context, _import_search_issue and IMPORT_SEARCH_PATCH first appear there):

  1. One closure per runner list, each re-reading every module. Every resolution the constructor-ownership proof checks calls _no_import_patch → _import_search_limit(runners). That answer is cached per runner list, but the lists differ from one resolution to the next (318 on Sponsio#173), and _import_search_context rebuilt each list's closure from scratch: for every module it reached it re-ran _type_checking_only and _import_guarded (two whole-tree walks plus provider checks), walked the tree again for imports, and re-resolved every import (_imported_paths, _enclosing_packages). The closures of one read overlap almost entirely: about 750 module reads were repeated 28,014 times. This is the superlinear term (closures × closure size). It dominates the slow members too: timers on Lling0000/OpenCMO#41 on main put 2,342 s of 3,224 s in _import_search_context and 3,122 s in _no_import_patch.
  2. One refusal reading per carrier set. _import_search_issue(module, carriers) was cached per (tree, carriers), and the carrier set differs between closures, so the same module was read again for each set: 4,833 readings for 1,613 module trees on Tiendat2703/MIS_TALENT#7, 42% of its CPU profile once (1) was fixed.
  3. One patch map per reader per file. Each source, side and reach check has its own ImportResolver, and each parses the closures it scans; _module derives _attribute_patches (which feat(readers): read returned tool factories with constructor ownership checks #964 extended with _import_search_effects) for every parse. Timers on agr_ai_curation#860 after (1): 19,270 module builds, _module 444 s of 908 s.

#1010 and #1016–#1019 add no comparable term in these profiles; the release build is 8% slower in total than #964's head, which these changes do not separately account for.

Fix

Three memos, each of a computation that does not depend on what varies between the calls that repeat it. Commits are separate for review.

  1. ImportResolver._import_search_edges — what one module adds to any closure (whether it imports sys/site, the caveat for each import no file provides, each module file an import runs) is read once per (path, parsed tree) per resolver and replayed in reading order by _import_search_context. The closure, the carrier set, the propagation and every verdict are still computed per invocation.
  2. ImportResolver._import_search_issue_uncomputed — the module's reading (_read_import_search_issue, the former body line for line) does not involve the carriers; they enter at two checks (a retained namespace, the files an unread import runs), which are now evaluated per call against a per-tree reading. Each return became a terminal event, each carrier check an event with its data and message.
  3. _module(..., parsed=True) — where the caller has just made tree with ast.parse(text) (ImportResolver.module, _patch_scan, _layout_module, and _RepositoryConstructorResolver.module, which wraps _layout_module's tree), the syntax-only _attribute_patches(tree) comes from a process-wide memo keyed by the text's SHA-256; every module gets its own copy. entry(), whose tree comes from the caller, is unchanged.

Why none of this can change an answer:

  • Same inputs. (1) uses only the resolver's structural lookups and the tree; none reads the runner list, the closure, _constructor_reference_mode or a read budget, and their own inputs (listings, the layout, provider proofs) were already cached per resolver (_runtime_attribute_patches already caches _type_checking_only per resolver and tree). (2) is the existing _import_search_issues contract with the carriers factored out: the reading depends on the tree and, through _constructor_reference, on resolution state that only changes by parsing; so it is used only while _import_search_budget_spent() is false, as the existing cache's store rule requires, and every call past a read bound reads afresh and keeps nothing. (3) is a pure function of the syntax tree, and ast.parse of the same text gives the same tree; the key is the exact text's digest.
  • Scope. (1) and (2) are dicts on each ImportResolver instance, keyed by the tree they hold (as _import_search_limits does). Base and head, each source, each _reach binding and each _CapturedImportSearchResolver / _RepositoryConstructorResolver have their own. (3) is content-addressed, so sharing it across readers, sides or runs cannot mix files; it starts over past 8,192 entries.
  • Same order, same limits, same side effects. Readings are lazy (_LazyReading): replaying the events read so far makes no lookup, and reading resumes only when a caller needs the next event. So a closure cut short by MAX_PATCH_SCAN_MODULES, or a carrier set refused at the first check, has made exactly the lookups the uncached code made, in the same order, and no lookup (a parse that spends MAX_MODULES, a directory listing an identity-bound snapshot records in scan/verify) happens earlier or more often than before. _import_search_context still calls _patch_scan for each module it dequeues and enqueue for each edge, so BFS order, unread order and both module bounds fire where they did. A reading that ends in a bound refusal (exceeds) is dropped, so a later call meets the bound again, as _import_search_issue requires.
  • Stops and errors. A _Stop raised while reading ends the reading and is re-raised at the same point for every caller that reaches it (stopped closures are still never stored). Any other exception is not kept: the record is marked broken and the next caller reads afresh. A reading is never re-entered: a nested call for a module whose reading is in progress reads afresh, uncached.

inputs/builder_calls.py is untouched. The openai_sdk_static and google_adk adapters (scan/verify) share python_imports; their tests and goldens pass unchanged.

Regression guard

Five deterministic tests in tests/test_import_search_ownership.py, no wall-clock budget. Each fails on main and on the commit before the change it guards (checked by running it against those sources).

  • test_each_module_is_read_once_for_every_closure_that_reaches_it — seven runner lists over one read (a shared package, an optional import, a typing-only import, an import above the scope, a search-path change): each list is still its own closure and gets a fresh resolver's answer, while each module's guards are walked once. Main: 31 walks for 7 modules.
  • test_a_stop_reading_a_module_is_raised_by_every_closure_that_reaches_it — an import of a symbolic link refuses every closure with a fresh resolver's reason and detail, each time; the module is read once.
  • test_a_closure_cut_short_reads_no_import_past_its_bound — with the module bound at 3, _imported_paths is asked about exactly the first three imports; with the bound restored, the next closure reads only the fourth.
  • test_a_module_is_read_once_for_every_carrier_set — five carrier sets over a module with both carrier checks give a fresh resolver's answer each (line 3 namespace, line 4 unread import, none, line 3, none), and each reference is followed once in all, as for one empty set. Before change 2: 18 follows for 6 references.
  • test_a_file_version_patch_map_is_computed_once_for_every_reader — four readers of one file compute its patch map once, each module gets an equal, equally ordered, separate copy, and an edited file is computed again.

Timings and identical answers

Every sampled member was run with benchmark/application-q2/run.py's exact per-member logic from detached pinned worktrees of 715f23e0 (this head's src/; the commits after it touch only CHANGELOG.md), and compared with --require-identical against the release build's answers (out-cut-04fc9ced) and, for the timeout members, main's uncapped answers (out-main-26bfa733-uncapped4). All 20 sampled answers are identical (answer_id and every row), including O.R.I.O.N#126 (453 rows), and all three agr_ai_curation members (#858 run directly, with function timers, rather than through run.py). The intermediate commits were checked too: change 1 alone (a3b3c74c) on all 18 members, changes 1–2 (1aeba50e) on 13; all identical.

Seconds (wall; run.py cap 900). "Release" is out-cut-04fc9ced at load ~10. This PR ran on a shared machine; its load is the 1-minute average at start, and CPU (user+sys) is given where load moved during the run.

Member 10-08 main release this PR this PR CPU s load at start
VidulaWickramasinghe/O.R.I.O.N#126 99 timeout 435 334 4
alliance-genome/agr_ai_curation#842 157 timeout 562 492 12
alliance-genome/agr_ai_curation#858 177 timeout 768¹ 563¹ 22–43
alliance-genome/agr_ai_curation#860 208 timeout 608 612 23
Lling0000/OpenCMO#41 56 757 372 284 20
liauto-siada/siada-cli#6 47 462 193 200 18
Tiendat2703/MIS_TALENT#7 23 168 64 60 11
SponsioLabs/Sponsio#173 17 141 39 40 12
mosnin/realestatecrm#612 20 129 119 79 10, rising
xju2/hepagent#74 18 195 230 96 57
ITMO-NSS-team/FEDOT.MAS#36 11 43 20 18 4
geroembser/livekit-agents#1 17 19 25 18 11
mezgoodle/Vesta#53 7 16 15 12 8
InseeFrLab/GRAAL#33 6 9 13 8 6
krevetka-is-afk/llm-agent-for-ai-studio#20 6 10 8 8 20
rustic-ai/rustic-ai#419 5 6 14 7 56
jpka/attest#3 4 6 7 5 20
hamzafarooq/multi-agent-course#64 3 4 6 4 21
cuppibla/adk_tutorial#9 3 4 4 3 12
JSebastianIEU/voice-freight-negotiator#10 2 3 5 3 19

¹ Run directly with function timers on, not through run.py.

Load rose to ~57 during realestatecrm#612, hepagent#74 and rustic-ai#419, so their wall times are not comparable; main on the same machine at load ~8 used 124, 187 and 6 CPU-seconds on them (126.6, 194.0 and 6.2 s wall), against this PR's 79, 96 and 7. The small members do not reach the import-search path and are within noise. Per change, at load ~8–14: change 1 alone took Sponsio#173 to 45 s, MIS_TALENT#7 to 92 s, realestatecrm#612 to 91 s, hepagent#74 to 97 s and agr_ai_curation#842 to 668 s; change 2 took MIS_TALENT#7 to 74 s; change 3 took agr_ai_curation#842 to 562 s.

What remains slow: alliance-genome/agr_ai_curation#858 and #860. #858 on this head took 768 s wall and 563 CPU-seconds at load 22–43 (with function timers on; answer identical to main's uncapped answer). Its time: _no_import_patch 414 s, of which _import_search_limit 266 s; _patch_scan 199 s over 214,298 calls and 18,461 module builds (each reader parsing the modules it scans; _module itself is down to 85 s from 444 s on #860 before change 3); _attribute_patches 152 s over 6,069 calls (memo misses, and calls with a reader's own exclusion set from _runtime_attribute_patches, which the per-text memo does not cover); _type_checking_only 50 s over 16,435 calls. #860 took ~910 s with change 1 alone (timers on, load ~8) and 608 s on this head (load 23 falling to 3). Both are under the cap, #858 with the least margin: it timed out at 900 s once while the machine's load was 56–85. More margin needs #1046's next step: one parse per file and side shared by every reader, rather than one per reader.

Corpus comparison (development, 49 PRs)

Run. benchmark/application-q2/run.py --jobs 4 from a detached worktree of 79b56830bf, over ~/.cache/shipgate-q2/repos with every pin present (no --fetch), 900 s cap. This head (6d728a9bfc) has the same src/; its last commit only moves the CHANGELOG entry. The run started at 08:08:23Z, at a one-minute load of 2.1, and finished at 08:24:21Z (16 minutes) at load 5.8. Output: ~/.cache/shipgate-q2/out-pr2-79b56830. run.py exited 0: all 49 members answered ok within the cap.

Current head (2026-11-10). The corpus was measured on 79b56830bf. The head is now 170995867f, which merges main's #1130 in; that merge's only conflict was CHANGELOG.md. #1130 changes core.trust_roots, core.agent_boundary and schemas.verification, which diff --application imports. So I re-ran 8 members on 170995867f: jpka/attest#3, surrealdb/agent-memory-openai-agents-sdk#1, cuppibla/adk_tutorial#9, ITMO-NSS-team/FEDOT.MAS#36, Tiendat2703/MIS_TALENT#7, kkmiecik-coder/CRM#5, cianfhoghlaim/tuatha#1 and mosnin/realestatecrm#612. All 8 answers are identical to the measured run (answer_id and every row; ~/.cache/shipgate-q2/out-pr2-17099586-sample).

Pre-fix answers (main):

Result: every answer is identical. answer_id is summarize.py's digest of an answer without the engine block and the summary reading. Rows are keyed by (scope, agent, agent source, tool).

Main (pre-fix) This head
Members answering within 900 s 45 (4 only uncapped) 49
Status 43 partial, 6 not_established the same, member by member
Rows 0 established, 868 not_established the same
Identical answer_id — 49/49
Rows whose change moved / rows on one side only / rows with other content changes — 0 / 0 / 0
Member seconds, summed 7,982 in the ledger run (a floor: four members stopped at 900) 3,705

Seconds for the 13 slowest members. The ledger run was at load 11→8; this run at load 2→6. Every other member takes at most 56 s.

Member 10-08 main (before #964) 1.3.0 ledger run This head
alliance-genome/agr_ai_curation#860 208 timeout 575
alliance-genome/agr_ai_curation#858 177 timeout 532
alliance-genome/agr_ai_curation#842 157 timeout 434
VidulaWickramasinghe/O.R.I.O.N#126 99 timeout 320
Lling0000/OpenCMO#41 56 757 260
carlospolop/chack-agent#25 84 550 219
liauto-siada/siada-cli#6 47 462 185
VRSEN/agency-swarm#804 77 457 184
VRSEN/agency-swarm#796 77 454 182
xju2/hepagent#74 18 194 79
mosnin/realestatecrm#612 20 128 72
Tiendat2703/MIS_TALENT#7 23 168 58
SponsioLabs/Sponsio#173 17 141 37

What this does not restore. The 13 members above still take two to five times as long as before #964 (Sponsio#173 2.2×, agr_ai_curation#860 2.8×, O.R.I.O.N#126 3.2×, OpenCMO#41 4.6×). The slowest, #860, finishes 325 s under the cap at this load. Under heavy load, #858 timed out once in the implementer's runs, at load 56–85. More margin needs #1046's next step, one parse per file and side shared by every reader, which this PR leaves open.

Reproduce:

python benchmark/application-q2/run.py --engine ./shipgate --corpus benchmark/application-q2/development.json \
  --clones ~/.cache/shipgate-q2/repos --out ~/.cache/shipgate-q2/out-pr2-79b56830 --jobs 4
python3 -I ~/.cache/shipgate-q2/scripts/compare_rows.py --corpus benchmark/application-q2/development.json \
  --summarize benchmark/application-q2/summarize.py \
  --base ~/.cache/shipgate-q2/out-cut-04fc9ced,~/.cache/shipgate-q2/out-main-26bfa733-uncapped4 \
  --cand ~/.cache/shipgate-q2/out-pr2-79b56830 --require-identical

The comparison exits 0. It reads runs.json and <slug>.json from each directory; a later directory in a comma list overlays an earlier one. The per-member table is saved as ~/.cache/shipgate-q2/pr2-vs-prefix.md.

Both PRs together. I merged #1135 and this PR onto main locally (63e7616938, not pushed, CHANGELOG folded into one group) and ran the full corpus at 08:28–08:50Z. run.py exits 0: all 49 members answer.

CHANGELOG (main-session review fix, 6d728a9bfc). The entry now sits under ### Application comparison ahead of ### Packaging, in 1.3.0's group order. #1135 adds the same heading at the same place. In the earlier position, merging both PRs would have left two ### Application comparison headings without a textual conflict. Whichever PR merges second now gets a CHANGELOG conflict, to be resolved as one group with two entries. The entry's timings now come from this corpus run.

Type

  • Check or risk-model change
  • Input adapter change
  • CLI or GitHub Action behavior
  • Report, schema, or SARIF output
  • Documentation only

Authorship

  • A coding agent wrote some or all of this change. Agent: Claude Code (Claude Opus 5.5)
  • A person read the whole diff before opening this pull request.

Verification

CI is authoritative for python -m ruff check ., python -m compileall -q src tests, and python -m pytest.

Additional local checks run:

  • ruff check . — passed.
  • 67 focused test files with -n 4 (test_builder_calls.py, test_application_diff*.py, test_constructor_*.py, test_*constructor*.py, test_openai*.py, test_google_adk*.py, test_import_search_ownership.py, test_python_import_resolution.py, test_imported_tool_*.py, test_adk_tool_factories.py, and every test file that names python_imports, ImportResolver, the SDK/ADK loaders or a golden) — exit 0 on this head; plus test_distribution_surface_parity.py, test_release_pipeline.py, test_release_advisory_pipeline.py and test_unread_changed_inputs.py for the CHANGELOG.
  • The identical-answer runs and timings above.

Release-readiness notes

  • No user-code import added to default scan paths
  • No network access added to default scan paths
  • New or changed check IDs are documented in docs/checks.md (none changed)
  • Report/schema changes are additive or documented in STABILITY.md (none)

🤖 Generated with Claude Code

…rison ahead of Packaging, with the full-corpus timings (#1046)

The 1.3.0 section lists Application comparison before Packaging, and #1135
adds the same heading at that point; placed after Packaging, merging both
would leave two Application comparison headings without a textual conflict.
The timings now come from one development-corpus run of this head (load 2-6,
every member under the cap, every answer identical to the pre-fix answers).
… into review-1137

# Conflicts:
#	CHANGELOG.md
#1135 and #1137 each opened an `### Application comparison` section under
Unreleased. Keep one heading, with #1135's entry first and #1137's after it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant