Skip to content

feat(ENG-4342): decode UTF-8 encoded values in every field, for every tier - #82

Merged
hiskudin merged 3 commits into
mainfrom
feat/eng-4342-decode-utf8
Nov 18, 2026
Merged

hiskudin merged 3 commits into
mainfrom
feat/eng-4342-decode-utf8

Conversation

@hiskudin

@hiskudin hiskudin commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

What

Port of StackOneHQ/defender#160. Encoded values are now decoded wherever they are, for every tier:

  • UTF-8 decodes: a base64 decode is kept when its bytes are valid UTF-8, not only when it is pure ASCII. Control characters other than tab, LF, VT, FF and CR become spaces. A leading BOM is kept, as in TS.
  • Tier 1 on every field: Tier 1 decodes and scans encoded values in every string field, not only risky-named ones. Plain text in other fields is still not scanned.
  • Tier 3 reads decoded text: encoded spans are decoded in the reviewer's input (not ROT13/ROT47), and coverage counts the decoded text.
  • UTF-16 heads: the 50,000-character analysis head (Tier 1 values and keys, Tier 2 and Tier 3 decoding) is now cut in UTF-16 units, as in TS. Counting code points reached encoded text past TS's head when astral characters came first.

Only the analysis copies change. Measurements are in the TS PR.

Tests

  • Conformance synced to the defender#160 branch: two tier1-gate cases and two tier3-serialize cases.
  • Decoder tests: UTF-8 decodes; binary stays encoded.
  • Tier 1 tests on a non-risky field.
  • A UTF-16 head test, and control bytes in a decode.
  • uv run pytest: 811 passed. uv run ruff check: clean.

🤖 Generated with Claude Code

… tier

Port of the TypeScript change:
- A base64 decode is kept when its bytes are valid UTF-8 text without
  control characters, not only when it is pure ASCII.
- Tier 1 decodes and scans encoded values in every string field.
- Tier 3 reads encoded strings decoded; coverage counts the decoded text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… heads

- Tier 3 doesn't ROT13- or ROT47-decode, and control bytes in a base64
  decode become spaces (ports of the TS fixes).
- The 50,000-character analysis head (Tier 1 values and keys, Tier 2
  and Tier 3 decoding) is cut in UTF-16 units, as in TS. Code points
  reached encoded text past TS's head when astral characters came first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t of cumulative risk

Ports of the TypeScript fixes: cascade's chunk review reads the top
chunk decoded, and a match found by decoding a non-risky field
escalates that field only, not the cumulative tracker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hiskudin
hiskudin merged commit a636865 into main Nov 18, 2026
15 checks passed
@hiskudin
hiskudin deleted the feat/eng-4342-decode-utf8 branch November 18, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant