Repository navigation
fix: keep CJK text intact in CLI previews and the ZCode SessionStart hook - #171
Merged
Grivn merged 3 commits intoOct 26, 2026
Merged
Conversation
`mnemon log` built its detail preview with `detail[:57]`, `mnemon viz` built node labels with `content[:60]`, and `mnemon remember` echoed an over-long tag or entity as `t[:50]` / `e[:50]`. Slicing a string at a byte offset cuts a multi-byte UTF-8 sequence in half, so the preview ended as an invalid byte stream; a Chinese-locale Windows console then fell back to GBK for the whole line and printed mojibake. Add truncateRunes, which truncates on rune boundaries, and use it at all three sites. The log and viz length guards now use utf8.RuneCountInString so the guard and the truncation count the same unit. cmd/memory/truncate_test.go covers ASCII, CJK, and mixed input at, under, and over the limit, plus zero, negative, and empty limits, and asserts that a truncated CJK preview with its ellipsis is still valid UTF-8.
The Windows SessionStart hook template loaded guide.md with `Get-Content -Raw` but no `-Encoding` argument. Setup writes guide.md as UTF-8 without a BOM, and Windows PowerShell 5.1 falls back to the system ANSI code page for BOM-less files, so every non-ASCII character in the guide was decoded as the wrong code point. The damaged text was then injected into the session as additionalContext, so Chinese guidance arrived as mojibake on a zh-CN host. Pass `-Encoding UTF8` explicitly and record the reason in a comment so the argument is not dropped again.
There was a problem hiding this comment.
🟢 Approval recommended
The reviewed changes have regression coverage, no unresolved review issues, and unanimous approval readiness.
No open issues 🎉
What changed in this PR
This pull request prevents UTF-8 corruption in CLI previews and the Windows ZCode SessionStart hook by using rune-safe truncation and explicit UTF-8 handling.
Changes:
- Added rune-safe truncation for log, visualization, tag, and entity previews.
- Configured the PowerShell hook to read UTF-8 guides and emit UTF-8 JSON.
- Added regression tests and Windows CI coverage for encoding behavior.
| File | Description |
|---|---|
internal/memory/setup/assets/zcode/prime.ps1 |
Reads the guide as UTF-8 and configures UTF-8 stdout. |
internal/memory/setup/assets/zcode_hook_test.go |
Adds static and Windows behavioral encoding tests. |
cmd/memory/viz.go |
Uses rune-safe truncation for visualization labels. |
cmd/memory/truncate.go |
Defines the shared rune-safe truncation helper. |
cmd/memory/truncate_test.go |
Tests truncation boundaries, CJK text, and UTF-8 validity. |
cmd/memory/remember.go |
Uses rune-safe previews in tag and entity validation errors. |
cmd/memory/log.go |
Uses rune-safe truncation for operation details. |
.github/workflows/ci.yml |
Runs the PowerShell hook tests in the Windows CI job. |
Review effort: Lite
Keep the Windows environment required for PowerShell module discovery while clearing inherited Mnemon configuration and isolating profile fallbacks and PATH. The previous minimal environment stalls before the hook reaches its CJK assertion on hosted Windows runners. Include process context and stdout in failure diagnostics. Validation: make test and go build -o mnemon . passed locally. Native Windows probes reproduced the minimal-environment timeout and completed with inherited system variables.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
truncateRunesand use it at the four CLI preview sites inlog.go,viz.go, andremember.go.Get-Content -Encoding UTF8and set[Console]::OutputEncoding = [Text.Encoding]::UTF8before writing the JSON response.CLI previews now truncate at rune boundaries while preserving the existing ellipsis placement. Log details are bounded to 60 runes including the ellipsis; visualization previews retain up to 60 content runes before the ellipsis and category prefix; tag/entity error previews retain up to 50 runes. Input validation limits are unchanged.
Why
Byte-based preview truncation can split a multi-byte UTF-8 character and produce invalid UTF-8. On the affected Windows setup, this caused entire
mnemon logrows to appear garbled. The same slicing pattern also affected visualization labels and tag/entity error previews.Windows PowerShell 5.1 reads BOM-less files using the system ANSI code page, while stdout uses the console output encoding. Either can corrupt the UTF-8 guide when it differs from UTF-8. Correctly decoding the guide alone is insufficient if the JSON response is then encoded using a legacy console code page.
Release-note impact: preserve non-ASCII text in CLI previews and the ZCode SessionStart context, including on Windows systems using legacy code pages.
Regression tests
Get-Contentcall to keep-Encoding UTF8.Testing
Local validation completed before submission:
go test ./cmd/memory/... ./internal/memory/setup/assets/ -count=1go vet ./...and
go test . ./cmd/... ./internal/... -count=1Checklist
make test; full upstream CI pending)Integration/process/Docker testing and user documentation changes are not applicable to these encoding fixes.