Skip to content

fix(census): a function's own library import is not an unresolved caller; name what a caveated reference lacks - #1135

Merged
pengfei-threemoonslab merged 4 commits into
mainfrom
claude/census-precision-local-imports
Nov 14, 2026
Merged

pengfei-threemoonslab merged 4 commits into
mainfrom
claude/census-precision-local-imports

Conversation

@pengfei-threemoonslab

@pengfei-threemoonslab pengfei-threemoonslab commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Two precision defects in the #963 caller census (BuilderCalls._census in src/agents_shipgate/inputs/builder_calls.py), seen in the v1.3.0 development run (04fc9ced, ~/.cache/shipgate-q2/out-cut-04fc9ced):

  1. A function's own import of another library counts as unresolved user code. Tiendat2703_MIS_TALENT-5 to -8 withhold rows on texts such as 'run' (run_risk_flow.py:35) has an unresolved reference at app/Agent/financeAgent.py:153: no file inside the read scope provides module 'agents' imported by app/Agent/financeAgent.py, and the same for uvicorn at app/ngrok.py:47. The census is looking for callers of the script entry run; it finds Runner.run(...) and uvicorn.run(...). At the head pin both imports are function-local (from agents import Runner inside a try: in run_finance_preflight_agent, import uvicorn in main(), read by the closure run_server), and the repository holds no agents or uvicorn module anywhere.
  2. Census texts end in : None. 1,376 occurrences of has an unresolved reference at …: None across 10 members of that run (359 in row uncertainty), mostly mosnin_realestatecrm-612 (1,051), then surrealdb_agent-memory-openai-agents-sdk-1, carlospolop_chack-agent-25, SponsioLabs_Sponsio-173, ITMO-NSS-team_FEDOT.MAS-30/36/38/42, liauto-siada_siada-cli-6, Lling0000_OpenCMO-41.

Cause

  1. _elsewhere already exempts a module_not_found reference whose name is bound only by another library's import (another_library → ImportResolver.repository_holds, which separates application code outside the read scope from a third-party package, Google ADK: trace bounded FunctionTool factories into agent tool bindings #865). But it looked the name up in module.bindings, the module scope. The resolver had followed the function's own binding (_resolve_reference → ScopeIndex.enclosing_bindings → resolve_local_import), so for a function-local import the lookup found nothing and the reference counted as unresolved. When a module-level library import had the same name, it was consulted in place of the local import the resolver actually followed — the opposite error (fail-open): from agents import Runner at module level plus from common.tools import Runner (application code outside the scope) in a function exempted Runner.run(...).
  2. Resolution.detail is None when a reference reaches a definition but caveats (code that runs first imports something unread that could reassign it) keep it from being established. The census formatted resolution.detail unconditionally.

Bug, not a deliberate rule

  • The LOCAL_BINDING branch directly above in _elsewhere already reads the scope-aware binding (self.scopes(module).enclosing_bindings(evaluation_site(...), head)); only the MODULE_NOT_FOUND branch read module scope.
  • The sibling checks apply another_library to function-local imports: _expanding_imports walks every import statement in the caller (ast.walk), including function-local ones, and list_expressions (borrower census, ~l.3930) resolves a non-top-level import with resolve_local_import and then applies another_library(statement, alias) to that statement.
  • docs/engineering/caller-supplied-agent-lists.md says the census "finds direct callers in the read scope, including local imports"; it states no rule that a local third-party import is unresolved.
  • The module-scope lookup is not even conservative: it fails open when a module-level library import shadows a function-local application import (test local_shadows_module_library).

Fix

  • _elsewhere: for module_not_found, read the binding the resolver followed. When the reference sits in a scope that binds its head, _local_library_import decides; otherwise the module-level rule is unchanged, byte for byte.
  • _local_library_import exempts only: exactly one binding of the name in the enclosing scope; that binding is an absolute import/from … import whose statement belongs to a function (def/async def), possibly under try/if; no nonlocal/global of the name anywhere inside that function; the repository does not hold the module (another_library, the existing Google ADK: trace bounded FunctionTool factories into agent tool bindings #865 notion, which covers a local agents/ package or uvicorn.py anywhere in the repository import roots); and no file in the read scope provides the import's top-level package (_absolute raises module_not_found), so the failure is the import itself and not a deeper hop through local code.
  • _unresolved_detail renders a None detail in words: '<reference>' reaches its definition at <ref>:<line>, but <caveats>; a defensive fallback names the resolver reason. Every non-None detail is returned unchanged.

What stays fail-closed, and why

Shape Why it stays a limit
The repository holds the module (uvicorn.py, agents/__init__.py, even outside the read scope) It is application code the census does not read; it could re-export the builder (repository_holds).
A second binding of the name in the function (Runner = …, a second import, del, a loop target …) Already local_binding before this change; _local_library_import also refuses len(local) != 1.
nonlocal/global of the name in any scope nested in the function A nested function can rebind the name without ScopeIndex listing it in the outer scope.
An import in a class body Class bodies use LOAD_NAME: before (or without) the class's own binding the name falls back to the module.
A relative import (from .vendor import x) Application code (another_library is false for level > 0).
A top-level package the read scope provides (from helpers import Runner where helpers.py imports uvicorn) The failure is deeper, inside local code that could re-export anything.
A function-local application import shadowing a module-level library import Now judged by the local import the resolver followed (previously fail-open).
Module-level bindings Unchanged rule.

A single function-local import under try/if is exempt on purpose: a name bound anywhere in a function is local to all of it, so if the import does not run the name is unbound (UnboundLocalError), never the module's or a builtin. That is why the conditional-binding rule the module path needs (CONDITIONAL_BINDING) does not apply here; a second, conditional rebinding still does.

Tests

  • tests/test_builder_calls.py
    • test_another_librarys_run_is_not_an_unresolved_caller (6): module-level SDK import (unchanged), function-local SDK import, under try in an async def, import uvicorn read by a closure, from uvicorn import run (same name as the builder), function-local google.adk.runners.
    • test_a_local_import_that_could_be_application_code_keeps_its_limit (9): repository module, repository package, local import shadowing a module-level library import, rebound, imported twice, nonlocal, class body, relative, scope module importing the library.
    • test_an_unresolved_reference_with_caveats_says_what_is_unknown: the exact new text, and no limit ends in None.
  • tests/test_builder_bindings.py::test_a_function_local_third_party_builder_is_a_different_callable (sdk/adk × rebound): the SDK and ADK readers keep a builder complete beside a function-local third-party build, and incomplete when the function rebinds it.

Against origin/main's builder_calls.py the new census tests fail as expected (the five local positives, the shadowing negative, and the text test), and the reader test's non-rebound cases fail. Removing each guard of _local_library_import in a scratch copy fails the matching negative (class body, nonlocal, another_library ×4, top-level-package); the len(local) != 1 guard is defensive (_resolve_reference already returns local_binding for several bindings).

Focused suites: tests/test_builder_calls.py, tests/test_builder_bindings.py, tests/test_application_diff*.py, tests/test_constructor_*.py and every other *constructor* suite, tests/test_openai_api.py, tests/test_google_adk.py, tests/test_adk_tool_factories.py, tests/test_agent_bindings.py, tests/test_imported_tool_bindings.py, tests/test_imported_tool_review.py, tests/test_list_expressions.py, tests/test_framework_matrix.py, tests/test_tool_reach.py and the other suites that name the census or the SDK reader (about 7,900 tests, -n 4): exit 0. ruff check .: clean.

Spot check (pinned copy of this head, run.py logic via the subset helper, 900 s cap)

Engine: a detached worktree of c6f19d67 (this PR's first commit; the second changes only CHANGELOG.md), against the 1.3.0 release answers in ~/.cache/shipgate-q2/out-cut-04fc9ced. Output: ~/.cache/shipgate-q2/out-pr1-c6f19d67.

Member Status Rows (established / not) Row status changes Content-only rows Census texts Seconds (1.3.0 → this head)
Tiendat2703_MIS_TALENT-5 partial → partial 0/14 → 0/14 0 1 6 agents/uvicorn limits gone, none new 143.7 → 407.2
Tiendat2703_MIS_TALENT-6 partial → partial 0/16 → 0/16 0 1 7 gone, none new 167.1 → 443.6
Tiendat2703_MIS_TALENT-7 partial → partial 0/17 → 0/17 0 1 8 gone, none new 168.0 → 433.6
Tiendat2703_MIS_TALENT-8 partial → partial 0/18 → 0/18 0 1 7 gone, none new 141.6 → 337.0
surrealdb_agent-memory-openai-agents-sdk-1 partial → partial 0/1 → 0/1 0 0 16 distinct : None texts reworded 4.1 → 8.8
ITMO-NSS-team_FEDOT.MAS-36 partial → partial 0/0 → 0/0 0 0 7 distinct : None texts reworded 42.8 → 142.6
mosnin_realestatecrm-612 partial → partial 0/43 → 0/43 0 43 4 distinct : None texts reworded (they appear in every row) 128.5 → 308.8
SponsioLabs_Sponsio-173 partial → partial 0/0 → 0/0 0 0 16 distinct : None texts reworded 141.1 → 414.9
  • No row changed status, no row appeared or disappeared, and no row became established, so there is no new established row to judge. No established row was lost.
  • MIS_TALENT-5 to -8: the one content-only row in each is build_risk_agent@app/Agent/risk_compliance_agent.py → process_risk_context. Every 'run' (run_risk_flow.py:35) has an unresolved reference at …: no file inside the read scope provides module 'agents'/'uvicorn' … limit is gone, and the run census is now clean. The constructor caller chain then stops one hop further, at a limit this PR does not touch: 'run_validated_pipeline' (app/Agent/validated_pipeline.py:208)'s retained namespace is used through computed access or as a value at app/api.py:25 (the feat(readers): follow caller-supplied agent lists #963 retained-namespace gate, part of the Application diff regression: CRM#5 and MIS_TALENT#7 lost established rows after 2026-10-05 (likely #963) #1001 policy question). So the rows stay not_established, for that reason only.
  • : None: 0 occurrences remain in the 8 answers (was 1,376 across 10 members in the full 1.3.0 run; 4 of those 10 are in this subset). Examples of the new wording:
    • 'main' (examples/01_quickstart.py:20) has an unresolved reference at examples/01_quickstart.py:41: 'main' reaches its definition at examples/01_quickstart.py:20, but examples/01_quickstart.py:17 imports 'agent_memory_openai_agents_sdk', which the repository holds outside the read scope, which is not read and could reassign it
    • 'make_chippi_agent' (chippi.py:295) has an unresolved reference at modal_app.py:813: 'make_chippi_agent' reaches its definition at chippi.py:295, but modal_app.py:153 changes or retains sys.path.insert; import search identity is unread
    • FEDOT.MAS-36: … reaches its definition at packages/fedotmas/src/fedotmas/mas/builder.py:41, but packages/fedotmas/src/fedotmas/common/llm.py:64 has unread import-search …
  • Apart from the removed MIS_TALENT limits, the only texts that changed in these answers are the : None ones (checked by normalising the detail and comparing every string; the summary causes[].pattern abstractions change with them).
  • The timings say nothing about speed. The machine was shared, with a load average of 190–290 during this run, and every member took 2–3.3x its release-run time, including surrealdb…-1 (4.1 → 8.8 s), whose answer changed only in text. Per exempted reference, the new check reads the top-level package's listings, which the resolver has already read, plus the repository import-root listings that repository_holds reads for the module-level rule, and it walks the function's AST once.

Corpus comparison (development, 49 PRs)

Run. benchmark/application-q2/run.py --jobs 4 from a detached worktree of this head (58992ff7f3), over ~/.cache/shipgate-q2/repos with every pin present (no --fetch), 900 s cap. It started at 06:32:56Z, once the one-minute load had fallen to 10.9 (after a 9-minute wait), and finished at 07:07:51Z at load 5.9. Output: ~/.cache/shipgate-q2/out-pr1-58992ff7. Baseline: the run behind the 1.3.0 ledger (04fc9ced, whose src/ is byte-identical to this PR's base), ~/.cache/shipgate-q2/out-cut-04fc9ced. The comparison keys rows by (scope, agent, agent source, tool) for each member.

Current head (2026-11-10). The corpus was measured on 58992ff7f3. The head is now 69b1dfeec7, which merges main's #1130 in; that merge's only conflict was CHANGELOG.md. #1130 changes core.trust_roots, core.agent_boundary and schemas.verification, which diff --application imports. So I re-ran 8 members on 69b1dfeec7: jpka/attest#3, surrealdb/agent-memory-openai-agents-sdk#1, cuppibla/adk_tutorial#9, ITMO-NSS-team/FEDOT.MAS#36, Tiendat2703/MIS_TALENT#7, kkmiecik-coder/CRM#5, cianfhoghlaim/tuatha#1 and mosnin/realestatecrm#612. All 8 answers are identical to the measured run (answer_id and every row; ~/.cache/shipgate-q2/out-pr1-69b1dfee-sample).

1.3.0 ledger run This head
Status 39 partial, 6 not_established, 4 timeout the same, member by member
Established rows (added/removed/changed) 0 0
not_established rows 415 415
Rows whose change moved — 0
Rows present on only one side — 0
Established rows lost — 0 (the ledger has none to lose)
Answers identical (answer_id) — 31; 4 unanswered on both sides; 14 differ in text only
: None in any answer file 1,388 raw occurrences (1,376 census texts plus the summary block's copies) 0
Timeouts O.R.I.O.N#126, agr_ai_curation #842/#858/#860 the same four; no new timeout
Member seconds (sum; max answered member) 7,982; 757 7,683; 680

run.py exits 1 on this head, as it does on main, only because those four members exceed 900 s. That slowdown dates from #964 and is a separate fix.

The 14 answers that differ, and how. I walked every string of each answer, leaving out the engine block and the summary reading, and compared base with head:

The four members this run could not answer. They time out on main, so their part of this change was measured with #1137, which changes no answer, merged on top. That was a local merge (63e7616938, not pushed), with a full corpus run at 08:28–08:50Z; run.py exits 0 and all 49 members answer.

  • O.R.I.O.N#126 (453 rows) and agr_ai_curation#858 and Route an enabled in-repository plugin's hook at a non-registry path to review (#809) #860 are identical to main's answers.
  • agr_ai_curation#842 changes in 2 strings, both former : None details.
  • Across all 49 members against main's answers, 645 former : None details and the 40 MIS_TALENT agents/uvicorn texts change. Nothing else changes, and no row changes status.
  • The 45 members answered above are identical in that run.

Q1/Q2. Both stay 0/49: no answer establishes a row on either side. This PR removes two wrong reasons. It does not change which gate withholds a row.

Reproduce:

python benchmark/application-q2/run.py --engine ./shipgate --corpus benchmark/application-q2/development.json \
  --clones ~/.cache/shipgate-q2/repos --out ~/.cache/shipgate-q2/out-pr1-58992ff7 --jobs 4

Then compare member by member against out-cut-04fc9ced:

python3 -I ~/.cache/shipgate-q2/scripts/compare_rows.py --corpus benchmark/application-q2/development.json \
  --summarize benchmark/application-q2/summarize.py \
  --base ~/.cache/shipgate-q2/out-cut-04fc9ced --cand ~/.cache/shipgate-q2/out-pr1-58992ff7

The script lives outside the repository. It reads runs.json and <slug>.json from both directories and uses summarize.py's answer_id. The per-member table is saved as ~/.cache/shipgate-q2/pr1-vs-release.md.

Closes #1134

🤖 Generated with Claude Code

The caller census exempted a module_not_found reference bound only by
another library's import, but looked the name up at module scope while
the resolver had followed the function's own binding. A function-local
`from agents import Runner` or `import uvicorn` therefore made
`Runner.run(...)` an unresolved reference to any builder named `run`
(development members MIS_TALENT-5 to -8), and a module-level library import of the same name
was consulted in place of a function-local application import (fail-open).

Read the binding the resolver followed. Exempt only one absolute import
inside a function of a module neither the read scope nor the repository
provides; second bindings, class bodies, nonlocal/global, relative
imports and modules the scope or repository holds stay limits. The
module-level rule is unchanged.

A reference that reaches a definition but is not established because of
caveats printed ": None"; name the definition and the caveats instead.

Closes #1134
@pengfei-threemoonslab pengfei-threemoonslab added bug Something isn't working area:adapter Framework input adapters and tool catalog construction workstream:application Manifest-free application-agent comparisons, binding coverage and Q1/Q2 proof. labels Nov 9, 2026
pengfei-threemoonslab added a commit that referenced this pull request Nov 9, 2026
…rison ahead of Packaging, with the full-corpus timings (#1046)

The 1.3.0 section lists Application comparison before Packaging, and #1135
adds the same heading at that point; placed after Packaging, merging both
would leave two Application comparison headings without a textual conflict.
The timings now come from one development-corpus run of this head (load 2-6,
every member under the cap, every answer identical to the pre-fix answers).
@pengfei-threemoonslab
pengfei-threemoonslab merged commit 18ebffa into main Nov 14, 2026
16 checks passed
pengfei-threemoonslab added a commit that referenced this pull request Nov 14, 2026
#1135 and #1137 each opened an `### Application comparison` section under
Unreleased. Keep one heading, with #1135's entry first and #1137's after it.
pengfei-threemoonslab added a commit that referenced this pull request Nov 14, 2026
…n's import-search checks (#1046) (#1137)

* perf(readers): read each module's import-search edges once per read (#1046)

* perf(readers): read each module's import-search refusal once per read, checked per carrier set (#1046)

* perf(readers): compute a parsed file version's patch map once per run (#1046)

* docs(changelog): diff --application import-search reads once per reader (#1046)

* docs(changelog): name what keeps agr_ai_curation#858 and #860 near the cap (#1046)

* docs(changelog): agr_ai_curation#860 measured on this change (#1046)

* docs(changelog): file the import-search entry under Application comparison ahead of Packaging, with the full-corpus timings (#1046)

The 1.3.0 section lists Application comparison before Packaging, and #1135
adds the same heading at that point; placed after Packaging, merging both
would leave two Application comparison headings without a textual conflict.
The timings now come from one development-corpus run of this head (load 2-6,
every member under the cap, every answer identical to the pre-fix answers).

* docs(changelog): fold the two Application comparison sections

#1135 and #1137 each opened an `### Application comparison` section under
Unreleased. Keep one heading, with #1135's entry first and #1137's after it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:adapter Framework input adapters and tool catalog construction bug Something isn't working workstream:application Manifest-free application-agent comparisons, binding coverage and Q1/Q2 proof.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Caller census: a function's own third-party import is an unresolved reference, and caveated references print ': None'

1 participant