Skip to content

fix: script output survives a console encoding such as cp1252 - #92

Merged
917Dhj merged 1 commit into
developfrom
fix/research-memory-ascii-output
Nov 7, 2026
Merged

917Dhj merged 1 commit into
developfrom
fix/research-memory-ascii-output

Conversation

@917Dhj

@917Dhj 917Dhj commented Nov 7, 2026

Copy link
Copy Markdown
Owner

Summary

Agents read script output through a pipe. On Windows without UTF-8 mode, that pipe uses the locale encoding, for example cp1252 or GBK. Printing non-ASCII JSON then raises UnicodeEncodeError, and the script fails. CI never saw this because the workflow sets PYTHONUTF8=1.

 stdout JSON
-  json.dumps(payload, ensure_ascii=False)     # 长视频理解 -> crash under cp1252
+  json.dumps(payload)                         # "长..." -> reads back as 长视频理解
 --output file
   json.dumps(payload, ensure_ascii=False) written as UTF-8   (unchanged)
 run_pipeline name=path lines
+  stdout reconfigured to UTF-8 (escaped paths would be unusable)

The fix covers these scripts:

  • deeppapernote: common.emit, run_state.py, user_configuration.py, run_pipeline.py
  • paper-glossary: glossary_common.emit, configure_glossary._emit
  • research-memory: aggregate_reading.py, write_research_memory.py

Evidence

The new tests run the scripts with PYTHONIOENCODING=cp1252 and PYTHONUTF8=0:

  • tests/test_console_encoding.py: 4 tests;
  • research-memory: 5 existing write and aggregate CLI tests now run under cp1252, plus 1 new test.
Before fix After fix
tests/test_console_encoding.py 4 failed (UnicodeEncodeError: 'charmap' codec can't encode…) 4 passed
research-memory tests 5 failed 38 passed
python3 -m pytest -q — 1751 passed

organize-wiki already prints ASCII JSON, fixed in #91.

Merge Danger

Door: two-way.

Blast Radius: stdout format.

  • JSON parsers see identical values.
  • Only a human reading raw stdout sees \uXXXX instead of Chinese.
  • Files written with --output are unchanged.

🤖 Generated with Claude Code

Scripts printed non-ASCII JSON to stdout, which raises UnicodeEncodeError
when an agent reads them through a pipe on Windows without UTF-8 mode.
JSON on stdout is now ASCII-escaped in deeppapernote, paper-glossary, and
research-memory; files written with --output stay UTF-8. run_pipeline's
name=path lines switch stdout to UTF-8 instead, since escaped paths would
not be usable. Tests run the scripts under PYTHONIOENCODING=cp1252.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@917Dhj
917Dhj merged commit b8e51e3 into develop Nov 7, 2026
3 checks passed
@917Dhj
917Dhj deleted the fix/research-memory-ascii-output branch November 9, 2026 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant