Skip to content

fix: keep CJK text intact in CLI previews and the ZCode SessionStart hook - #171

Merged
Grivn merged 3 commits into
mnemon-dev:masterfrom
shixiaogaoya:fix/utf8-previews-zcode-hook
Oct 26, 2026
Merged

Grivn merged 3 commits into
mnemon-dev:masterfrom
shixiaogaoya:fix/utf8-previews-zcode-hook

Conversation

@shixiaogaoya

Copy link
Copy Markdown
Contributor

What

  • Add truncateRunes and use it at the four CLI preview sites in log.go, viz.go, and remember.go.
  • Read the ZCode SessionStart guide with Get-Content -Encoding UTF8 and set [Console]::OutputEncoding = [Text.Encoding]::UTF8 before writing the JSON response.
  • Add regression tests and run the PowerShell hook tests in the Windows CI job.

CLI previews now truncate at rune boundaries while preserving the existing ellipsis placement. Log details are bounded to 60 runes including the ellipsis; visualization previews retain up to 60 content runes before the ellipsis and category prefix; tag/entity error previews retain up to 50 runes. Input validation limits are unchanged.

Why

Byte-based preview truncation can split a multi-byte UTF-8 character and produce invalid UTF-8. On the affected Windows setup, this caused entire mnemon log rows to appear garbled. The same slicing pattern also affected visualization labels and tag/entity error previews.

Windows PowerShell 5.1 reads BOM-less files using the system ANSI code page, while stdout uses the console output encoding. Either can corrupt the UTF-8 guide when it differs from UTF-8. Correctly decoding the guide alone is insufficient if the JSON response is then encoded using a legacy console code page.

Release-note impact: preserve non-ASCII text in CLI previews and the ZCode SessionStart context, including on Windows systems using legacy code pages.

Regression tests

  • Table-driven truncation tests cover ASCII, CJK, mixed input, empty strings, and zero/negative limits. The UTF-8 property test uses an input where the original byte-based slice is explicitly verified to split a multi-byte character.
  • A platform-independent static check requires the guide's Get-Content call to keep -Encoding UTF8.
  • The Windows behavioral test executes the embedded hook with a BOM-less UTF-8 CJK guide, starts console output at cp936, and requires the guide to appear intact in the injected context. It uses an isolated environment without requiring a real Mnemon installation. Both PowerShell hook tests are included in the Windows CI job.

Testing

Local validation completed before submission:

  • Windows 11, cp936, Windows PowerShell 5.1:
    go test ./cmd/memory/... ./internal/memory/setup/assets/ -count=1
  • Negative checks on that cp936 Windows setup: removing either UTF-8 setting makes the behavioral test fail.
  • Linux, Ubuntu:
    go vet ./...
    and go test . ./cmd/... ./internal/... -count=1

Checklist

  • Deterministic tests pass (make test; full upstream CI pending)
  • New/changed behavior is covered by tests
  • User-facing release-note impact is described in this PR

Integration/process/Docker testing and user documentation changes are not applicable to these encoding fixes.

`mnemon log` built its detail preview with `detail[:57]`, `mnemon viz`
built node labels with `content[:60]`, and `mnemon remember` echoed an
over-long tag or entity as `t[:50]` / `e[:50]`. Slicing a string at a byte
offset cuts a multi-byte UTF-8 sequence in half, so the preview ended as an
invalid byte stream; a Chinese-locale Windows console then fell back to GBK
for the whole line and printed mojibake.

Add truncateRunes, which truncates on rune boundaries, and use it at all
three sites. The log and viz length guards now use
utf8.RuneCountInString so the guard and the truncation count the same unit.

cmd/memory/truncate_test.go covers ASCII, CJK, and mixed input at, under,
and over the limit, plus zero, negative, and empty limits, and asserts that
a truncated CJK preview with its ellipsis is still valid UTF-8.
The Windows SessionStart hook template loaded guide.md with
`Get-Content -Raw` but no `-Encoding` argument. Setup writes guide.md as
UTF-8 without a BOM, and Windows PowerShell 5.1 falls back to the system
ANSI code page for BOM-less files, so every non-ASCII character in the
guide was decoded as the wrong code point. The damaged text was then
injected into the session as additionalContext, so Chinese guidance
arrived as mojibake on a zh-CN host.

Pass `-Encoding UTF8` explicitly and record the reason in a comment so the
argument is not dropped again.
Copilot AI lite review requested due to automatic review settings October 26, 2026 12:33

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The reviewed changes have regression coverage, no unresolved review issues, and unanimous approval readiness.

No open issues 🎉

What changed in this PR

This pull request prevents UTF-8 corruption in CLI previews and the Windows ZCode SessionStart hook by using rune-safe truncation and explicit UTF-8 handling.

Changes:

  • Added rune-safe truncation for log, visualization, tag, and entity previews.
  • Configured the PowerShell hook to read UTF-8 guides and emit UTF-8 JSON.
  • Added regression tests and Windows CI coverage for encoding behavior.
File Description
internal/​memory/​setup/​assets/​zcode/​prime.ps1 Reads the guide as UTF-8 and configures UTF-8 stdout.
internal/​memory/​setup/​assets/​zcode_hook_test.go Adds static and Windows behavioral encoding tests.
cmd/​memory/​viz.go Uses rune-safe truncation for visualization labels.
cmd/​memory/​truncate.go Defines the shared rune-safe truncation helper.
cmd/​memory/​truncate_test.go Tests truncation boundaries, CJK text, and UTF-8 validity.
cmd/​memory/​remember.go Uses rune-safe previews in tag and entity validation errors.
cmd/​memory/​log.go Uses rune-safe truncation for operation details.
.github/​workflows/​ci.yml Runs the PowerShell hook tests in the Windows CI job.

Review effort: Lite

Keep the Windows environment required for PowerShell module discovery while clearing inherited Mnemon configuration and isolating profile fallbacks and PATH. The previous minimal environment stalls before the hook reaches its CJK assertion on hosted Windows runners. Include process context and stdout in failure diagnostics.

Validation: make test and go build -o mnemon . passed locally. Native Windows probes reproduced the minimal-environment timeout and completed with inherited system variables.
@Grivn
Grivn merged commit b4ad125 into mnemon-dev:master Oct 26, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants