Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
7886c9f
Rewrite inline ghost text rendering around the host's own font, basel…
Mason363 Sep 8, 2026
a0fe981
Keep suggestions alive through host-owned text and system alerts
Mason363 Sep 8, 2026
38b248f
Add debug-only forced suggestions, observer key tracing, and a focus-…
Mason363 Sep 8, 2026
691eab8
Identify a web host's typeface from its own pixels when it names none
Mason363 Sep 8, 2026
12c75c5
Document the ghost text pipeline and file the marked-text policy unde…
Mason363 Sep 8, 2026
00bc2bc
Make continuations land on the word the user is typing and stop showi…
Mason363 Sep 8, 2026
1ba3430
Place ghost text exactly in WebKit and narrow Electron fields
Mason363 Sep 9, 2026
d302334
Match code editors' faces and line spacing: SF Mono, Monaco container…
Mason363 Sep 9, 2026
23113f9
Wrap ghost rows over the host's text on opaque bands
Mason363 Sep 9, 2026
feae16c
Measure the line pitch from character bounds when line APIs give none
Mason363 Sep 9, 2026
dc47aa7
Finish the word the user is typing with a required-prefix constraint
Mason363 Sep 9, 2026
31b12f8
Place the caret inside wrapped paragraphs in CodeMirror editors
Mason363 Sep 9, 2026
ae2a676
Strip Chromium's inline address-bar completion instead of blocking
Mason363 Sep 9, 2026
2890752
Wrap on the caret box height, bind leading punctuation, widen the prefix
Mason363 Sep 9, 2026
f4d273f
Document wrapped-run carets, run pitch, and the address-bar completion
Mason363 Sep 9, 2026
1db554e
Keep the ghost alive when typing through a space in web editors
Mason363 Sep 9, 2026
3f733d4
Add a context-recall eval so prompt context can be measured, not assumed
Mason363 Sep 9, 2026
a413469
Double the document the model can see, fixing recall past the old window
Mason363 Sep 9, 2026
9f4e506
Document the recall eval, the widened context window, and the dev scheme
Mason363 Sep 9, 2026
23dab7a
Let the eval suites target a specific model for head-to-head comparison
Mason363 Sep 9, 2026
8e82670
Keep ghost text inline in Obsidian instead of falling back to the card
Mason363 Sep 9, 2026
cd50f2a
Stop ghost text sitting higher on some lines than others
Mason363 Sep 9, 2026
811f7ab
Hold the ghost typeface steady in hosts that name no font
Mason363 Sep 9, 2026
e76f842
Revert the three positioning changes; they made spacing worse in prac…
Mason363 Sep 9, 2026
470574c
Re-land only the Obsidian card fix, with a guard on the derived pitch
Mason363 Sep 9, 2026
aafe308
Revert the Obsidian inline anchor; it drew ghost text over the host's…
Mason363 Sep 9, 2026
b0578d7
Keep every line of a field on one baseline
Mason363 Sep 10, 2026
5abc5cb
Place the caret from the host's pixels in Obsidian, and hold the type…
Mason363 Sep 10, 2026
ba8e7d9
Document pixel-verified placement and the two presentation measurements
Mason363 Sep 10, 2026
27aa946
Revert the pixel caret, typeface evidence, and baseline consensus
Mason363 Sep 10, 2026
e26ae02
Decide the ghost's leading space against the live text, not the reque…
Mason363 Sep 10, 2026
e2de395
Prefetch the next suggestion into the cache before the current one ru…
Mason363 Sep 10, 2026
99fb0dd
Reject baseline readings taken from ink that is not a line of this font
Mason363 Sep 10, 2026
ff00ae6
Hold the ghost typeface steady in hosts that name no loadable face
Mason363 Sep 10, 2026
59a705c
Place the Obsidian caret from the host's pixels, and re-arm the prefe…
Mason363 Sep 10, 2026
f72d40c
Log the system face's own score in the pixel typeface match
Mason363 Sep 10, 2026
303a38d
Document the presentation-time measurements and the pixel verificatio…
Mason363 Sep 10, 2026
ab0eb78
Name the writer in the base prompt only at a sign-off
Mason363 Sep 10, 2026
2401b3d
Leave Mail's header rows to Mail and reach its compose body
Mason363 Sep 10, 2026
3190ebe
Match the host's face and size from pixel-aligned strips and read bas…
Mason363 Sep 10, 2026
928e431
Show a mid-line suggestion in the card and never paint a ghost over t…
Mason363 Sep 10, 2026
baf232e
Keep a typed-through ghost through the host's Accessibility lag and r…
Mason363 Sep 10, 2026
3f7614f
Size the ghost from the host's own caret advance when it answers no w…
Mason363 Sep 10, 2026
b37a616
Match a host's bundled face at its exact size and hold every pixel ma…
Mason363 Sep 10, 2026
43a124f
Read Chromium's non-breaking line-end space as a space in the caret a…
Mason363 Sep 10, 2026
a53e6fb
Move a refined size only for a real gain in the pixel typeface match
Mason363 Sep 10, 2026
d4c1db5
Keep a pixel-measured caret and size a pixel-matched face from the ho…
Mason363 Sep 10, 2026
1c33bdc
Read every CodeMirror caret from the host's pixels, at the last glyph…
Mason363 Sep 10, 2026
8aee50b
Never read a pixel caret under the visible ghost, and count a keystro…
Mason363 Sep 10, 2026
b11055a
Carry the pixel caret forward under the visible ghost, and read singl…
Mason363 Sep 10, 2026
f65dfae
Withdraw the engine abort once the prompt is decoded, so a cancel can…
Mason363 Sep 10, 2026
4200e8e
Read one-line CodeMirror paragraphs as one line, and scope a paragrap…
Mason363 Sep 10, 2026
117b340
Take the host's own caret bar as the caret in a pixel read
Mason363 Sep 10, 2026
07dd078
Fit a matched face's size to the host's glyph positions
Mason363 Sep 10, 2026
787cb89
Place the caret after a trailing space from the text's own ink
Mason363 Sep 10, 2026
3c44768
Trace the caret observations that width samples are made of
Mason363 Sep 10, 2026
1765add
Pair a caret that moves before its text with the text that follows
Mason363 Sep 10, 2026
b37fa23
Make width samples only from carets that measure glyph positions
Mason363 Sep 11, 2026
4ba0838
Take a base model's own word boundary when spacing the ghost
Mason363 Sep 11, 2026
f43622e
Keep the ghost's face and size steady across sampler resets and new f…
Mason363 Sep 11, 2026
b1f7725
Place wrapped ghost rows from the host's own line box in Chromium edi…
Mason363 Sep 11, 2026
fc08228
Read a Chromium field's paragraphs from its value, not its run-togeth…
Mason363 Sep 11, 2026
cd9c9f2
Keep the typed line breaks in the prompt, and give a new line's first…
Mason363 Sep 11, 2026
9dd55ee
Prefetch the next suggestion from the text the ghost will leave behind
Mason363 Sep 11, 2026
dfb1a49
End the ghost where the model opens a new turn on its own line
Mason363 Sep 11, 2026
78c8337
Keep the remembered host faces across launches
Mason363 Sep 11, 2026
f4b41cd
Snap a pixel-matched web size to the host's zoom step
Mason363 Sep 11, 2026
a298412
Keep a wrapped line's caret on its own line when the host's caret bar…
Mason363 Sep 11, 2026
246fa81
Size the ghost from the host's own caret advance where the host repor…
Mason363 Sep 11, 2026
c97d01c
Drop a browser page's style from the remembered faces when restoring …
Mason363 Sep 11, 2026
007922a
Keep the system face until a family fits a width sample clearly better
Mason363 Sep 11, 2026
65b5f5e
Measure Chromium marker lines from their glyphs, not their line boxes
Mason363 Sep 11, 2026
e3501d5
Snap the scaled system face to the host's zoom step when the host's f…
Mason363 Sep 11, 2026
b7e3996
Hold a ghost whose caret still sits behind the text the host already …
Mason363 Sep 11, 2026
bf481ed
Anchor the mid-line card under a caret that has no width
Mason363 Sep 11, 2026
501d81c
Refuse line pitches no text of the ghost's size can have
Mason363 Sep 11, 2026
cfd4bf9
Find a mid-line caret among a paragraph's lines from its pixels
Mason363 Sep 11, 2026
396e363
Keep a ghost to one row in a single-line field
Mason363 Sep 11, 2026
208639a
Keep a monospaced field's ghost in a monospaced face
Mason363 Sep 11, 2026
988f8db
Take a browser paragraph's one-line box for its pitch before its text…
Mason363 Sep 11, 2026
fd768a5
Resolve a calibrated ghost's face the way a present does
Mason363 Sep 11, 2026
57edd65
Size the address bar's ghost from its own caret's advance
Mason363 Sep 11, 2026
895b5ee
Start a Chromium caret line where its first glyph is, not where its r…
Mason363 Sep 11, 2026
76691f0
Hold the ghost while Gmail shows its Smart Compose suggestion
Mason363 Sep 11, 2026
37d53da
Read a line's baseline under its letters, not under a capital's top c…
Mason363 Sep 11, 2026
b9e11b0
Bring the branch under SwiftLint strict and drop local references
Mason363 Sep 11, 2026
fa296d3
Regenerate Cotabby.xcodeproj from project.yml
Mason363 Sep 11, 2026
ed674fa
Say that Screen Recording also matches ghost text to the host
Mason363 Sep 12, 2026
034ed40
Merge main into fix/ghost-text-engine, keeping main's engine path and…
FuJacob Sep 27, 2026
2364e8c
Address review: no default writer name, metadata-only key trace, stre…
FuJacob Sep 27, 2026
251ed84
Split the caret-advance trace literal so Xcode 26.3 can type-check it
FuJacob Sep 27, 2026
723b0e3
Split the remaining large log literals and ink comparisons for Xcode …
FuJacob Sep 27, 2026
f59864c
Use .magnitude for the ink contrast so Xcode 26.3 does not see an amb…
FuJacob Sep 28, 2026
e0fa993
Use .magnitude for the font weight check so Xcode 26.3 does not see a…
FuJacob Sep 28, 2026
dc33835
Give the new main-actor presentation classes a nonisolated deinit so …
FuJacob Sep 28, 2026
54da2c8
Merge remote-tracking branch 'origin/main' into integrate/ghost-text-…
FuJacob Sep 28, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 74 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,23 @@ Focus and geometry live in:

Accessibility data is eventually consistent and app-specific. Browser editors, Electron apps,
native AppKit fields, and secure fields expose different AX shapes. Preserve stale-result guards,
`focusChangeSequence`, and capability checks unless the change explicitly replaces them.
`focusChangeSequence`, and capability checks unless the change explicitly replaces them. Known
shapes worth keeping in mind: CodeMirror (Obsidian) exposes a wrapped paragraph as one static-text
run plus a single-space spacer run per line, and its value runs paragraphs together with no
separator, so a paragraph is found from where its run was anchored in the value, never from the
last line break (`WrappedRunAnchor` hands both that run and a single-line paragraph run to
`PixelCaretLocator`, which reads the caret from the run's pixels, a one-line run in its own line
box, and carries it forward by the typed advance while the ghost covers the run);
Chromium's address bar keeps its inline completion selected after the caret (stripped by the
resolver and held as the host's marked text, never treated as a user selection) and answers every
bounds query with a zero rect (the caret comes from its pixels, `PixelCaretLocator`); Mail's compose
header rows are `AXTextField`s with identifiers `Mail.toField`, `Mail.ccField`, `Mail.subjectField`
where Tab moves to the next field, so they are blocked (`MailHeaderFieldDetector`), and the body is
an `AXWebArea` described "message body". A Chromium or Electron host reports its font size in CSS
pixels and nothing of its zoom (the Claude desktop composer reports 14 and paints 15.4 at 110%),
and a contenteditable answers no width query: the ghost's size there follows the advance measured
from the caret's own movement (`CaretAdvanceSampler`), and the faces such a host bundles in its
own `Contents/Resources` are registered for the pixel match (`HostBundledFontRegistry`).

## Visual Context And OCR

Expand Down Expand Up @@ -174,7 +190,13 @@ The Swift generation loop owns the maximum output-token budget.

## UI And Overlays

- `OverlayController` owns the ghost-text panel lifecycle and positioning.
- `OverlayController` owns the ghost-text panel lifecycle and positioning. Ghost glyphs are laid
out by `GhostTextLayout` in the font `GhostFontResolver` resolves, on the baseline
`GhostBaselinePolicy` (or a `HostBaselineCalibrator` measurement) gives, wrapping onto the host's
next lines at the measured pitch. A caret with text after it on its line gets the card under the
caret instead (`CompletionRenderModePolicy`): a ghost painted over the host's own characters
reads as overwriting them. Change those pure helpers and their tests rather than nudging offsets
in the controller.
- `SuggestionOverlayPresenter` decides whether a suggestion should be shown or hidden.
- `ActivationIndicatorController` owns the optional caret/field-edge indicator.
- `FocusDebugOverlayController` is for developer visibility and should stay gated behind debug
Expand Down Expand Up @@ -237,6 +259,10 @@ Console.app stream.
- `~/Desktop/cotabby-ax-dump.txt` — most recent Chrome AX tree snapshot. Overwritten on each
Chrome focus change (debounced by focused-element identity).
- Rotated previous logs: `*.jsonl.1` (one-step rotation when a file exceeds 10 MB).
- `~/Library/Logs/<app>/strips/` — every calibration strip the overlay captured, as PNG plus a JSON
sidecar (caret column, size, line text), written only while
`defaults write <bundle> cotabbyDumpCalibrationStrips -bool YES` is set at launch. Replay one
through `TypefaceMatcher` offline to see why a face was or was not matched.

**Correlation IDs.** Every prediction gets a `request_id` like `req_a3f9k2lq`, stamped on every log
line touching that request (coordinator state transitions, router selection, engine generation, LLM
Expand Down Expand Up @@ -297,6 +323,52 @@ Keep DerivedData in `build/DerivedData`, avoid concurrent builds in the same che
it after validation. Verify cleanup before reporting completion.
Never move DerivedData to `~/Library/Developer/Xcode/DerivedData/Cotabby-*`.

```bash
xcodebuild -project Cotabby.xcodeproj -scheme Cotabby -destination 'platform=macOS' build \
-derivedDataPath build/DerivedData
xcodebuild -project Cotabby.xcodeproj -scheme Cotabby -destination 'platform=macOS' build-for-testing \
-derivedDataPath build/DerivedData
```

Always pass `-derivedDataPath build/DerivedData` so the output lands in the repo-scoped `build/`
directory (already gitignored) instead of accumulating under
`~/Library/Developer/Xcode/DerivedData/Cotabby-*`, where every build leaves a fresh multi-GB module
cache and SwiftPM checkout that nothing trims. When a task is done and the artifacts are no longer
needed, `rm -rf build/DerivedData` before reporting completion.

Two model-backed eval suites measure suggestion quality and are local-only (they need a downloaded
GGUF and are gated behind the `RUN_LLAMA_EVAL` compile flag). `test_reportEvalSuite` scores ordinary
continuations; `test_reportRecallSuite` scores whether context the user did not type (earlier field
text, screen OCR, clipboard) actually reaches the completion. Run either with:

```bash
xcodebuild test -project Cotabby.xcodeproj -scheme Cotabby -destination 'platform=macOS' \
-only-testing:CotabbyTests/LlamaSuggestionEvalTests/test_reportRecallSuite \
SWIFT_ACTIVE_COMPILATION_CONDITIONS='$(inherited) RUN_LLAMA_EVAL' \
CODE_SIGNING_ALLOWED=NO -configuration Release ENABLE_TESTABILITY=YES \
-derivedDataPath build/DerivedData
```

Quote latency only from a Release build; Debug inflates per-token Swift work by an order of magnitude.
Any change to prompt content, context budgets, sampling, or the model should be justified with a
before/after on both suites over identical cases.

The dev app is a SEPARATE target and scheme, `Cotabby Dev` (product `Cotabby Dev.app`). Building the
`Cotabby` scheme leaves the dev bundle stale, so build the dev app explicitly and confirm the binary
timestamp moved before testing against it:

```bash
xcodebuild -project Cotabby.xcodeproj -scheme "Cotabby Dev" -configuration Release \
-destination 'platform=macOS' build -derivedDataPath build/DerivedData
```

Ghost placement changes are verified against the pixels the user sees, not against log counts: type
a fragment at the end of a field, capture the ghost, accept it, capture the host's own rendering of the
same words, and compare the two, the ghost's dx/dy in points per line. A placement fix is done when
every host checked (TextEdit, a Chrome contenteditable, an Obsidian note with one wrapped paragraph and
with several) stays within about 0.2pt on every line; a change that helps one host and moves another
is a regression even when its own log counters improve.

Run targeted tests for changed pure logic when available. If validation fails, report the exact
failure and distinguish a successful build-for-testing step from actual test execution.

Expand Down
Loading