feat(windows): UIA cursor context; pace SendInput keystrokes - #1137
Open
DepengWang wants to merge 6 commits into
Open
DepengWang wants to merge 6 commits into
DepengWang wants to merge 6 commits into
Conversation
Adds a Windows UI Automation implementation of read_around_cursor, reusing the existing HostContextAdapter -> cursor_context_input() pipeline instead of a parallel path. Prefers TextPattern2::GetCaretRange (active caret only), falls back to TextPattern::GetSelection for collapsed selections only, and never guesses a caret from ValuePattern. The text window is built by expanding a Range from the caret (never reading the whole document), then sliced to budget by Core's existing window_around_cursor/plan_window. Gated by the same password-field (CurrentIsPassword) and process-blocklist checks as the existing edit watcher, plus a focus-consistency recheck before returning a result so an Alt+Tab mid-read can't mix two apps' text. Runs on spawn_blocking with a 1s outer timeout; any failure degrades to no context, never affects dictation. Adds budget-split/redistribution/emoji-boundary tests to openless-core's window.rs (previously untested) and a pure is_blocked_process_name helper with tests in windows.rs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011h9QUkoFtCoDLsJjRDB6dX
Shows the Cursor context toggle on Windows as well as macOS (Linux still has no implementation, so it stays hidden there). Updates the cursorContextEnabled doc comment in types.ts and all 8 locale descriptions to drop the "macOS only" wording and mention password fields/terminals are excluded on both platforms. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011h9QUkoFtCoDLsJjRDB6dX
Updates the Cursor Context sections in README.md and README.zh.md to cover Windows: TextPattern2 caret-range priority, TextPattern selection fallback, and the "skip rather than guess" rule when the caret can't be reliably located. Password-field exclusion wording now also mentions the Windows UIA password-control flag alongside macOS Secure Input. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011h9QUkoFtCoDLsJjRDB6dX
TauriHostContextAdapter::capture previously logged nothing on success — only debug_read_cursor_context exercised the status/chars_before/ chars_after/elapsed_ms/app log line. Extends the same metadata-only log (no document body, per design doc §18/§19) to every real dictation that has cursor context enabled, so whether a capture actually happened is visible from the normal log file without a devtools round-trip. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011h9QUkoFtCoDLsJjRDB6dX
Windows SendInput text injection sent characters within a chunk back to back with zero delay, only pausing every 16 characters. Found while testing cursor context (windowsInsertionMode=sendInput): dictating mixed Chinese/English text dropped the leading character(s) of a Latin-script word right after the CJK-to-ASCII transition, e.g. "open" arrived as "pen" and "OpenLess" vanished entirely — well before the 16-char chunk boundary, so the existing chunk-level pause never applied. macOS already works around the same class of problem (its INTER_KEYSTROKE_DELAY comment: "Chromium / Electron / Tauri themselves drop characters when keyDown/keyUp have no delay"); Windows had no equivalent. Adds a 1ms gap after every character, not just at chunk boundaries — confirmed on hardware to stop the drops, with no perceptible typing latency. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011h9QUkoFtCoDLsJjRDB6dX
Confirmed on hardware: Chromium-based editors (VS Code) only keep their accessibility tree's caret position in sync once they detect an active accessibility client, so with default settings the position UI Automation reads back can be stale even though the read itself succeeds. Setting editor.accessibilitySupport to "on" (plus a window reload) fixed it. Documents this in both READMEs so users don't mistake it for a bug. Also adds a metadata-only diagnostic log (source pattern + requested vs. actual Move distances, no document content) to host_document/windows.rs, added while narrowing this down between VS Code/Chrome — kept because it is cheap, useful for future reports of "wrong context" on a given app, and fully compliant with the no-document-body logging rule. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011h9QUkoFtCoDLsJjRDB6dX
DepengWang
force-pushed
the
feature/windows-cursor-context
branch
from
October 2, 2026 14:40
5359d3f to
c0788fb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings Cursor Context to Windows. The feature already exists on macOS (via AX): it reads a small window of text around the host app's caret and passes it to LLM polish, which helps with homophones and domain terms. This PR adds the Windows half using UI Automation and reuses the existing
HostContextAdapter->cursor_context_input()pipeline, so there is no parallel path.It also includes one unrelated Windows fix found while testing (dropped characters in
SendInputinsertion mode). It is a separate commit and can be split out if preferred.Changes
Cursor context on Windows
host_document/windows.rs: UIA reader. PrefersTextPattern2::GetCaretRange, falls back toTextPattern::GetSelectionfor collapsed selections only, and never guesses a caret fromValuePattern. The text window is built by expanding a range from the caret, so the whole document is never read; Core's existingwindow_around_cursor/plan_windowthen slices it to budget.CurrentIsPassword) and process-blocklist checks as the existing edit watcher, plus a focus-consistency recheck before returning, so an Alt+Tab mid-read cannot mix text from two apps.spawn_blockingwith a 1s outer timeout. Any failure degrades to "no context" and dictation continues.types.tsdoc comment drop the "macOS only" wording.Unrelated fix:
SendInputdropped characters (unicode_keystroke.rs, commit50a02da2)windowsInsertionMode=sendInput, mixed Chinese/English dictation lost the leading characters of a Latin word right after a CJK-to-ASCII transition ("open" arrived as "pen"). Characters inside a 16-char chunk were sent with no delay.INTER_KEYSTROKE_DELAY.Testing
Verified on a real Windows 11 machine using Settings -> Experiments & Extensions -> cursor context probe:
<textarea>, VS Code: caret position correct, including with the caret in the middle of existing text.Blocked/secure_text_field.Blocked/blocked_app.SendInputfix: the dropped-character case no longer reproduces, with no noticeable typing delay.Not tested:
core_adapters.rs,host_document/mod.rsandopenless-core/src/host_document/window.rs.Known limitation
VS Code and other Chromium/Electron editors only keep the accessibility tree's caret in sync once they detect an accessibility client. With default settings the read succeeds but can return a stale position. Setting
editor.accessibilitySupporttoonand reloading the window fixes it. This is documented in both READMEs.Screenshot
Windows, Settings -> Permissions & data. The Cursor context (experimental) toggle is now available on Windows. Taken from a local build that also has #1133 merged, which is why the encrypted sync card below shows a self-hosted token sign-in.
🤖 Generated with Claude Code