gh-151464: exclude '<>' token from tokenize output by skirpichev · Pull Request #154854 · python/cpython

skirpichev · 2026-07-29T06:08:03Z

Was:

$ echo '1 <> 2' | python -m tokenize
1,0-1,1:            NUMBER         '1'
1,2-1,4:            OP             '<>'
1,5-1,6:            NUMBER         '2'
1,6-1,7:            NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''

Now (regardless on __future__.barry_as_FLUFL import):

$ echo '1 <> 2' | ./python -m tokenize
1,0-1,1:            NUMBER         '1'
1,2-1,3:            OP             '<'
1,3-1,4:            OP             '>'
1,5-1,6:            NUMBER         '2'
1,6-1,7:            NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''

in accordance with the Grammar:
https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters

Also adds a custom error message for <> ("not equal" in Pascal and Python 2).

Issue: Tokenizer produces token '<>', missing in the language grammar (regardless on __future__.barry_as_FLUFL settings) #151464

Was: ``` $ echo '1 <> 2' | python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,4: OP '<>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` Now (regardless on ``__future__.barry_as_FLUFL`` import): ``` $ echo '1 <> 2' | ./python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,3: OP '<' 1,3-1,4: OP '>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` in accordance with the Grammar: https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters Also adds a custom error message for ``<>`` ("not equal" in Pascal and Python 2).

Aniketsy

Thanks for this work :)

I've just gone through these changes and changes looks good as per my understanding ofcourse i'm not expert 😀 , and it was really interesting to going through these changes, at first I found its bit tricky to understand some portion.

I'm excited to see the review process in this from experts and try to understand how it goes, also if you feel this comment as noise please feel free to mark as off-topic.

pablogsal

Left some comments

bedevere-app · 2026-08-10T11:35:36Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

skirpichev · 2026-08-11T01:22:13Z

Left some comments

I have made the requested changes; please review again

bedevere-app · 2026-08-11T01:22:18Z

Thanks for making the requested changes!

@pablogsal: please review the changes made to this pull request.

read-the-docs-community · 2026-08-11T01:24:10Z

Documentation build overview

📚 cpython-previews | 🛠️ Build #34646001 | 📁 Comparing e1fc4cc against main (219768f)

🔍 Preview build

398 files changed · + 9 added · ± 388 modified · - 1 deleted

+ Added

± Modified

- Deleted

library/dialog.html

skirpichev · 2026-09-08T06:33:32Z

@pablogsal, does it make sense for you at all? I doubt I can reduce patch further.

# Conflicts: # Grammar/python.gram # Parser/parser.c # Parser/pegen.c

miss-islington-app · 2026-09-25T18:52:04Z

Thanks @skirpichev for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14, 3.15.
🐍🍒⛏🤖

miss-islington-app · 2026-09-25T18:52:08Z

Sorry, @skirpichev and @pablogsal, I could not cleanly backport this to 3.15 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 198bc7683641f6b44d23fe5dc228f732a7cf3df8 3.15

miss-islington-app · 2026-09-25T18:52:13Z

Sorry, @skirpichev and @pablogsal, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 198bc7683641f6b44d23fe5dc228f732a7cf3df8 3.14

miss-islington-app · 2026-09-25T18:52:17Z

Sorry, @skirpichev and @pablogsal, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 198bc7683641f6b44d23fe5dc228f732a7cf3df8 3.13

bedevere-app · 2026-09-25T18:55:10Z

GH-158189 is a backport of this pull request to the 3.15 branch.

bedevere-app · 2026-09-25T18:56:55Z

GH-158190 is a backport of this pull request to the 3.14 branch.

bedevere-app · 2026-09-25T18:57:44Z

GH-158191 is a backport of this pull request to the 3.13 branch.

…onGH-154854) * pythongh-151464: exclude '<>' token from tokenize output Was: ``` $ echo '1 <> 2' | python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,4: OP '<>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` Now (regardless on ``__future__.barry_as_FLUFL`` import): ``` $ echo '1 <> 2' | ./python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,3: OP '<' 1,3-1,4: OP '>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` in accordance with the Grammar: https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters Also adds a custom error message for ``<>`` ("not equal" in Pascal and Python 2). * +1 * address review: lowercase and move invalid rule * address review: news * address review: move test_guido_as_bdfl_ineq_tokens() * address review: revert _PyTokenizer_From* changes * + revert unrelated change * address review: remove whatsnew entry (cherry picked from commit 198bc76)

…#158191) * gh-151464: exclude '<>' token from tokenize output Was: ``` $ echo '1 <> 2' | python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,4: OP '<>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` Now (regardless on ``__future__.barry_as_FLUFL`` import): ``` $ echo '1 <> 2' | ./python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,3: OP '<' 1,3-1,4: OP '>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` in accordance with the Grammar: https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters Also adds a custom error message for ``<>`` ("not equal" in Pascal and Python 2). * +1 * address review: lowercase and move invalid rule * address review: news * address review: move test_guido_as_bdfl_ineq_tokens() * address review: revert _PyTokenizer_From* changes * + revert unrelated change * address review: remove whatsnew entry (cherry picked from commit 198bc76) Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>

…#158189) * gh-151464: exclude '<>' token from tokenize output Was: ``` $ echo '1 <> 2' | python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,4: OP '<>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` Now (regardless on ``__future__.barry_as_FLUFL`` import): ``` $ echo '1 <> 2' | ./python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,3: OP '<' 1,3-1,4: OP '>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` in accordance with the Grammar: https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters Also adds a custom error message for ``<>`` ("not equal" in Pascal and Python 2). * +1 * address review: lowercase and move invalid rule * address review: news * address review: move test_guido_as_bdfl_ineq_tokens() * address review: revert _PyTokenizer_From* changes * + revert unrelated change * address review: remove whatsnew entry (cherry picked from commit 198bc76) Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>

…onGH-154854) * pythongh-151464: exclude '<>' token from tokenize output Was: ``` $ echo '1 <> 2' | python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,4: OP '<>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` Now (regardless on ``__future__.barry_as_FLUFL`` import): ``` $ echo '1 <> 2' | ./python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,3: OP '<' 1,3-1,4: OP '>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` in accordance with the Grammar: https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters Also adds a custom error message for ``<>`` ("not equal" in Pascal and Python 2). * +1 * address review: lowercase and move invalid rule * address review: news * address review: move test_guido_as_bdfl_ineq_tokens() * address review: revert _PyTokenizer_From* changes * + revert unrelated change * address review: remove whatsnew entry (cherry picked from commit 198bc76)

…#158190) * gh-151464: exclude '<>' token from tokenize output Was: ``` $ echo '1 <> 2' | python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,4: OP '<>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` Now (regardless on ``__future__.barry_as_FLUFL`` import): ``` $ echo '1 <> 2' | ./python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,3: OP '<' 1,3-1,4: OP '>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` in accordance with the Grammar: https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters Also adds a custom error message for ``<>`` ("not equal" in Pascal and Python 2). * +1 * address review: lowercase and move invalid rule * address review: news * address review: move test_guido_as_bdfl_ineq_tokens() * address review: revert _PyTokenizer_From* changes * + revert unrelated change * address review: remove whatsnew entry (cherry picked from commit 198bc76) Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>

pablogsal · 2026-09-25T20:39:07Z

Great work on the tokenizer fix, @skirpichev! Thanks a lot! ❤️

bedevere-app Bot mentioned this pull request Jul 29, 2026

Tokenizer produces token '<>', missing in the language grammar (regardless on __future__.barry_as_FLUFL settings) #151464

Closed

+1

c53a17f

skirpichev marked this pull request as ready for review July 29, 2026 08:49

skirpichev requested review from lysnikolaou and pablogsal as code owners July 29, 2026 08:49

bedevere-app Bot added the awaiting review label Jul 29, 2026

Aniketsy reviewed Jul 31, 2026

View reviewed changes

pablogsal requested changes Aug 10, 2026

View reviewed changes

Comment thread Misc/NEWS.d/next/Core_and_Builtins/2026-07-29-08-43-14.gh-issue-151464.o_mtmz.rst Outdated

Comment thread Lib/test/test_flufl.py Outdated

Comment thread Parser/lexer/state.h Outdated

Comment thread Parser/tokenizer/tokenizer.h Outdated

Comment thread Grammar/python.gram Outdated

bedevere-app Bot added awaiting changes and removed awaiting review labels Aug 10, 2026

skirpichev added 5 commits August 11, 2026 02:57

Merge branch 'master' into save-python/151464

e525b8f

address review: lowercase and move invalid rule

f4dd644

address review: news

ba5843d

address review: move test_guido_as_bdfl_ineq_tokens()

9ea138f

address review: revert _PyTokenizer_From* changes

3f9c57e

skirpichev requested a review from AA-Turner as a code owner August 11, 2026 01:20

bedevere-app Bot added awaiting change review and removed awaiting changes labels Aug 11, 2026

bedevere-app Bot requested a review from pablogsal August 11, 2026 01:22

skirpichev added 3 commits August 21, 2026 06:26

Merge branch 'main' into save-python/151464

a966092

Merge branch 'master' into save-python/151464

d81b19c

+ revert unrelated change

3ff8480

Merge branch 'master' into save-python/151464

d134772

# Conflicts: # Grammar/python.gram # Parser/parser.c # Parser/pegen.c

skirpichev closed this Sep 16, 2026

bedevere-app Bot added awaiting merge and removed awaiting change review labels Sep 25, 2026

pablogsal added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes needs backport to 3.15 pre-release feature fixes, bugs and security fixes labels Sep 25, 2026

pablogsal merged commit 198bc76 into python:main Sep 25, 2026
64 of 65 checks passed

bedevere-app Bot removed the awaiting merge label Sep 25, 2026

miss-islington-app Bot assigned pablogsal Sep 25, 2026

pablogsal mentioned this pull request Sep 25, 2026

[3.15] gh-151464: exclude '<>' token from tokenize output (GH-154854) #158189

Merged

bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Sep 25, 2026

pablogsal mentioned this pull request Sep 25, 2026

[3.14] gh-151464: exclude '<>' token from tokenize output (GH-154854) #158190

Merged

bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Sep 25, 2026

pablogsal mentioned this pull request Sep 25, 2026

[3.13] gh-151464: exclude '<>' token from tokenize output (GH-154854) #158191

Merged

bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label Sep 25, 2026

skirpichev deleted the save-python/151464 branch September 26, 2026 00:14

Uh oh!

Conversation

skirpichev commented Jul 29, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aniketsy left a comment

Choose a reason for hiding this comment

Uh oh!

pablogsal left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-app Bot commented Aug 10, 2026

Uh oh!

skirpichev commented Aug 11, 2026

Uh oh!

bedevere-app Bot commented Aug 11, 2026

Uh oh!

read-the-docs-community Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

skirpichev commented Sep 8, 2026

Uh oh!

Uh oh!

miss-islington-app Bot commented Sep 25, 2026

Uh oh!

miss-islington-app Bot commented Sep 25, 2026

Uh oh!

miss-islington-app Bot commented Sep 25, 2026

Uh oh!

miss-islington-app Bot commented Sep 25, 2026

Uh oh!

bedevere-app Bot commented Sep 25, 2026

Uh oh!

bedevere-app Bot commented Sep 25, 2026

Uh oh!

bedevere-app Bot commented Sep 25, 2026

Uh oh!

pablogsal commented Sep 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

skirpichev commented Jul 29, 2026 •

edited by bedevere-app Bot

Loading

pablogsal left a comment •

edited

Loading

read-the-docs-community Bot commented Aug 11, 2026 •

edited

Loading