gh-152907: Restore cooked output flags around the input hook in the new REPL by harjothkhara · Pull Request #153389 · python/cpython

harjothkhara · 2026-07-09T06:13:11Z

The problem: While the new REPL waits for input, it puts the terminal in raw mode, which turns off the usual \n → \r\n translation. It also runs input hooks during that wait — these are what GUI toolkits (Tkinter, Qt, …) use to keep their event loops alive at the prompt. So when a hook prints something, each line starts where the previous one ended instead of at the left margin, and the text "staircases" down the screen. The old readline REPL didn't do this, and PYTHON_BASIC_REPL=1 still works fine.

The fix: The REPL needs raw mode for drawing its own output, but the input hook runs at a moment when it isn't drawing. So this turns the newline translation back on just around the hook call, then returns to raw mode. Only the output flags are touched, so keystrokes are still not echoed at the prompt.

Checked with a pty that captures the exact bytes a hook writes:

before:  line1\nline2\nline3\n
after:   line1\r\nline2\r\nline3\r\n

There's a new test that fails without the fix, and ./python -m test test_pyrepl passes.

I used AI assistance for this and have reviewed the change.

Refs #152907

Issue: New REPL (pyrepl) calls PyOS_InputHook with the terminal in raw mode, so output from input-hook callbacks loses carriage returns #152907

… the new REPL pyrepl clears OPOST for its own cursor rendering but calls PyOS_InputHook from inside the raw-mode read loop, so output written by an input hook (GUI toolkit event loops, and any warning/traceback/print they emit) is emitted with bare '\n' and no '\r'. Restore the terminal's saved output flags around the hook call and re-enter raw mode afterwards; only oflag is toggled so ECHO/ICANON stay off at the prompt.

The Emscripten buildbot has the pty module but no pty devices, so pty.openpty() raises OSError("out of pty devices"). Guard the test class the same way Lib/test/test_pty.py does.

pablogsal · 2026-07-11T13:40:39Z

+        cooked.oflag = self.__svtermstate.oflag
+        self.__input_fd_set(cooked)
+        try:
+            posix._inputhook()


Can we return posix._inputhook() here? This keeps the Unix wrapper consistent with the console API and the Windows path.

pablogsal · 2026-07-11T13:40:39Z

+            self.assertFalse(_termios.tcgetattr(slave_fd)[1] & _termios.OPOST)
+        finally:
+            console.restore()
+            os.close(slave_fd)


Nit, feel free to ignore: relying on a background reader plus time.sleep(0.2) to observe the output tends to be a source of buildbot flakiness. Since the TCSADRAIN in __run_input_hook already guarantees the bytes reach the master before the hook returns, can we drain master_fd directly after hook() instead of sleeping?

Tried that first but it hangs on macOS — TCSADRAIN there doesn't return until the master side actually reads the output, so the mode switch blocks forever without a reader. Kept the reader thread but dropped the sleep: once the hook returns the output is already drained, so joining the reader is enough.

I removed the flaky part.
The hook now drains the PTY master itself, so the test can compare the complete output directly:

def fake_hook(): observed["oflag"] = _termios.tcgetattr(slave_fd)[1] os.write(slave_fd, b"line1\nline2\n") observed["output"] = drain() return 0

Draining after hook() returns is too late because __run_input_hook switches the terminal mode on the way in and on the way out, so it can block inside the call. The test drains after prepare() and again inside the hook.
The only remaining drainer thread is for teardown, where restore() can block before the caller gets a chance to drain.
On macOS 26.5, tcsetattr(TCSADRAIN) on a PTY slave waits until the master's unread data is drained.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

pablogsal · 2026-07-11T17:36:04Z

+            return self.__run_input_hook
+
+    def __run_input_hook(self):
+        # Input-hook callbacks (GUI toolkit event loops, and any warning,


This comment is too verbose, pls, simplify it a bit

pablogsal · 2026-07-11T17:36:21Z

+@unittest.skipIf(is_android or is_apple_mobile or is_wasm32,
+                 "pty is not available on this platform")
+class TestUnixConsoleInputHook(TestCase):
+    # gh-152907: pyrepl runs with OPOST disabled so it can drive the cursor


Thhese AI comments are too verbose pls keep only the relevant parts

Done, trimmed them down.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The reader thread no longer feeds any assertion: the hook drains the pty master itself, so the check is an exact comparison. A drainer is still needed for restore(), which writes before switching modes.

miss-islington-app · 2026-09-24T16:01:23Z

Thanks @harjothkhara for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14, 3.15.
🐍🍒⛏🤖

miss-islington-app · 2026-09-24T16:01:37Z

Sorry, @harjothkhara and @pablogsal, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 46ee3580c0d1c8b2d527d13b6956881b61b26d34 3.14

bedevere-app · 2026-09-24T16:01:39Z

GH-158112 is a backport of this pull request to the 3.15 branch.

miss-islington-app · 2026-09-24T16:01:43Z

Sorry, @harjothkhara and @pablogsal, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 46ee3580c0d1c8b2d527d13b6956881b61b26d34 3.13

bedevere-app · 2026-09-24T16:12:22Z

GH-158117 is a backport of this pull request to the 3.14 branch.

bedevere-app · 2026-09-24T16:12:27Z

GH-158119 is a backport of this pull request to the 3.13 branch.

…n the new REPL (GH-153389) (#158112) gh-152907: Restore cooked output flags around the input hook in the new REPL (GH-153389) * gh-152907: Restore cooked output flags around the input hook in the new REPL pyrepl clears OPOST for its own cursor rendering but calls PyOS_InputHook from inside the raw-mode read loop, so output written by an input hook (GUI toolkit event loops, and any warning/traceback/print they emit) is emitted with bare '\n' and no '\r'. Restore the terminal's saved output flags around the hook call and re-enter raw mode afterwards; only oflag is toggled so ECHO/ICANON stay off at the prompt. * Skip the input-hook test on platforms without pty devices The Emscripten buildbot has the pty module but no pty devices, so pty.openpty() raises OSError("out of pty devices"). Guard the test class the same way Lib/test/test_pty.py does. * Propagate the input hook's return value and drop the sleep from the test * Trim comments * Observe the hook's output synchronously instead of via the reader thread The reader thread no longer feeds any assertion: the hook drains the pty master itself, so the check is an exact comparison. A drainer is still needed for restore(), which writes before switching modes. --------- (cherry picked from commit 46ee358) Co-authored-by: Harjoth Khara <harjoth.khara@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…n the new REPL (GH-153389) (#158119) (cherry picked from commit 46ee358) Co-authored-by: Harjoth Khara <harjoth.khara@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

pablogsal · 2026-09-24T16:49:42Z

Great catch on the input hook, @harjothkhara. Thanks for the fix! 🤘

…n the new REPL (GH-153389) (#158117) (cherry picked from commit 46ee358) Co-authored-by: Harjoth Khara <harjoth.khara@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

harjothkhara requested review from ambv, lysnikolaou and pablogsal as code owners July 9, 2026 06:13

bedevere-app Bot added the awaiting review label Jul 9, 2026

bedevere-app Bot mentioned this pull request Jul 9, 2026

New REPL (pyrepl) calls PyOS_InputHook with the terminal in raw mode, so output from input-hook callbacks loses carriage returns #152907

Open

Skip the input-hook test on platforms without pty devices

1146b70

The Emscripten buildbot has the pty module but no pty devices, so pty.openpty() raises OSError("out of pty devices"). Guard the test class the same way Lib/test/test_pty.py does.

pablogsal reviewed Jul 11, 2026

View reviewed changes

Propagate the input hook's return value and drop the sleep from the test

a979156

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

pablogsal reviewed Jul 11, 2026

View reviewed changes

harjothkhara and others added 2 commits July 11, 2026 11:22

Trim comments

3436671

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Observe the hook's output synchronously instead of via the reader thread

bf6509e

The reader thread no longer feeds any assertion: the hook drains the pty master itself, so the check is an exact comparison. A drainer is still needed for restore(), which writes before switching modes.

harjothkhara requested a review from pablogsal September 2, 2026 21:09

pablogsal added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes needs backport to 3.15 pre-release feature fixes, bugs and security fixes labels Sep 24, 2026

pablogsal merged commit 46ee358 into python:main Sep 24, 2026
65 checks passed

bedevere-app Bot removed the awaiting review label Sep 24, 2026

miss-islington-app Bot assigned pablogsal Sep 24, 2026

bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Sep 24, 2026

This was referenced Sep 24, 2026

[3.14] gh-152907: Restore cooked output flags around the input hook in the new REPL (GH-153389) #158117

Merged

[3.13] gh-152907: Restore cooked output flags around the input hook in the new REPL (GH-153389) #158119

Merged

bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Sep 24, 2026

bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label Sep 24, 2026

Uh oh!

Conversation

harjothkhara commented Jul 9, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harjothkhara Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-app Bot commented Sep 24, 2026

Uh oh!

miss-islington-app Bot commented Sep 24, 2026

Uh oh!

bedevere-app Bot commented Sep 24, 2026

Uh oh!

miss-islington-app Bot commented Sep 24, 2026

Uh oh!

bedevere-app Bot commented Sep 24, 2026

Uh oh!

bedevere-app Bot commented Sep 24, 2026

Uh oh!

pablogsal commented Sep 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

harjothkhara commented Jul 9, 2026 •

edited by bedevere-app Bot

Loading

harjothkhara Aug 2, 2026 •

edited

Loading