Repository navigation
"Check if generated files are up to date" job misses regen-unicodedata and regen-limited-abi #126524
Description
Feature or enhancement
Currently our regen-all is missing several regen- targets:
@echo "Note: make regen-stdlib-module-names, make regen-limited-abi, "
@echo "make regen-configure, make regen-sbom, and make regen-unicodedata should be run manually"
While our CI job to check regenerated files does not check them:
cpython/.github/workflows/build.yml
Lines 120 to 124 in 2a6b6b3
This does not seem right, I think that we should add:
make regen-unicodedata(it requires internet access, but GitHub jobs always have access, see gh-96954: Don't run regen-unicodedata in regen-all #112120)make regen-limited-abi(needs to compile python to get the ABI)
How long does it work? Testing locally first:
make regen-unicodedata 9.44s user 0.47s system 27% cpu 35.719 total
make regen-limited-abi 2.35s user 0.76s system 77% cpu 4.019 total (with pre-existing cache)
I propose adding these two make targets to the test to be extra safe.
What do others think? :)
If agreed, I will send a PR.
Linked PRs
- gh-126524: Run
regen-unicodedataas a part of our CI #126682 - [3.13] gh-126524: Run
regen-unicodedataas a part of our CI (GH-126682) #127594 - [3.12] gh-126524: Run
regen-unicodedataas a part of our CI (GH-126682) #127595 - gh-126524: Revert "gh-126524: Run
regen-unicodedataas a part of our CI #126682" #138197 - [3.14] gh-126524: Revert "gh-126524: Run
regen-unicodedataas a part of our CI GH-126682" (GH-138197) #138218
Activity
sobolevn commentedon Nov 7, 2024
I don't touch
regen-configurehere, because it will require a different solution, will create a new issue about it later.encukou commentedon Nov 11, 2024
There's no need to add
regen-limited-abi, since the CI runsmake check-limited-abi.That checks that
regen-limited-abiwouldn't make changes, and tries to generate a nicer message than you'd get just by comparing the resulting diff.pythongh-126524: Run `regen-unicodedata` as a part of our CI
sobolevn commentedon Nov 11, 2024
When I add
make regen-unicodedatahere's how the CI speed changes:1m 59shttps://github.com/python/cpython/actions/runs/11717879421/job/32638269138?pr=1265262m 11shttps://github.com/python/cpython/actions/runs/11777998050/job/32803523310?pr=126682Does not seem like it is slow now.
gh-126524: Run `regen-unicodedata` as a part of our CI (#126682)
pythongh-126524: Run `regen-unicodedata` as a part of our CI (pythonG…
pythongh-126524: Run `regen-unicodedata` as a part of our CI (pythonG…
pythongh-126524: Run `regen-unicodedata` as a part of our CI (python#…
encukou commentedon Aug 27, 2025
Turns out
regen-unicodedatacan fail on a network issue: https://github.com/python/cpython/actions/runs/17265732493/job/48997259329?pr=138142And it's been doing so consistently. Perhaps Unicode has scraper protection now. Downloading the data on every CI run isn't a very nice thing for us to do, anyway.
sobolevn commentedon Aug 27, 2025
I will revert this change, sorry that it did not turn out as a net positive :(
pythongh-126524: Revert "pythongh-126524: Run `regen-unicodedata` as …
hugovk commentedon Aug 27, 2025
No objection to reverting, but of the recent runs, it only failed once due to a network issue. The others were genuine test failures.
gh-126524: Revert "gh-126524: Run `regen-unicodedata` as a part of ou…
pythongh-126524: Revert "pythongh-126524: Run `regen-unicodedata` as …
hugovk commentedon Aug 28, 2025
I think we're done here. Thanks for the attempt!
[3.14] gh-126524: Revert "gh-126524: Run `regen-unicodedata` as a par…
encukou commentedon Aug 28, 2025
Indeed, the failure was not consistent. Sorry for the misinformation.
Even so, we don't need to run this step every time. AFAIK, the only thing it checked was that the results of the unicodedata regen scripts are bit-by-bit reproducible.
[3.14] pythongh-126524: Revert "pythongh-126524: Run `regen-unicodeda…
pythongh-126524: Revert "pythongh-126524: Run `regen-unicodedata` as …