Skip to content

Honor transform_source with the native parser - #22008

Open
Dextheking1 wants to merge 1 commit into
python:masterfrom
Dextheking1:fix/shadow-file-parallel-checking
Open

Dextheking1 wants to merge 1 commit into
python:masterfrom
Dextheking1:fix/shadow-file-parallel-checking

Conversation

@Dextheking1

Copy link
Copy Markdown

Fixes #21222 (the transform_source half).

The issue reports two things broken under parallel checking. The shadow-file half was already handled — requires_read() checks the shadow map and works in threaded and --num-workers builds. The genuinely broken half is options.transform_source: the native parser path skipped it (it was only applied on the old-parser path), and the parallel build passed source=None, so the parser read the untransformed file from disk.

This applies the transform before dispatching to the native parser, and makes requires_read() force a source read when a transform is configured.

Tests: new TransformSourceSuite in mypy/test/testparse.py covering the native parser, the old parser, and a parallel build. The native-parser cases fail without the fix; all pass with it. Full testparse suite (249 passed, 75 skipped) and shadow-related testcheck (13 passed) green.

Disclosure: I used an LLM assistant while developing this fix.

The native (Rust) parser path in mypy/parse.py silently ignored
options.transform_source: the transform was only applied in the old-parser
branch. Additionally, with the native parser the source was never read in the
first place during (parallel) builds, since State.requires_read() did not
account for the transform.

Apply the transform up front in parse() so both parsers honor it, and make
requires_read() return True when a transform is configured, so the source is
read (and then transformed) before the native parser sees it.

Fixes python#21222
@github-actions

Copy link
Copy Markdown
Contributor

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

@Dextheking1

Copy link
Copy Markdown
Author

The only failing check is Test suite with py314t-ubuntu, mypyc-compiled, and the failure is in testConcurrentCircularNativeImports_multi (mypyc/test-data/run-multimodule.test), a free-threaded concurrency test: KeyError: 'other_b' while two threads import other_a/other_b concurrently under --nogil.

This is unrelated to the change here. The PR only touches mypy/build.py, mypy/parse.py, and mypy/test/testparse.py, none of which are involved in the mypyc native import path the test exercises (the test compiles its own modules with mypyc and imports them in a ThreadPoolExecutor). The failure is a race in the free-threaded import machinery, not in the transform code.

I checked: the py314t job is not in the required set for merging (the check's mergeStateStatus is UNSTABLE, not BLOCKED), and the change is mergeable otherwise. Happy to re-run if it flips back to a real failure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shadow file and transform source broken when using parallel type checking

1 participant