Skip to content

feat(ENG-4342): decode UTF-8 encoded values in every field, for every tier - #160

Merged
hiskudin merged 4 commits into
mainfrom
feat/eng-4342-decode-utf8
Nov 18, 2026
Merged

hiskudin merged 4 commits into
mainfrom
feat/eng-4342-decode-utf8

Conversation

@hiskudin

@hiskudin hiskudin commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

What

Encoded values are now decoded wherever they are, for every tier. Example: a Gmail message whose body comes base64url-encoded in payload.body.data.

  • UTF-8 decodes. A base64 decode was kept only if it was pure ASCII, so an email with one accented letter, dash or curly quote wasn't decoded at all. Now a decode is kept when its bytes are valid UTF-8. Control characters other than tab, LF, VT, FF and CR become spaces, so a NUL appended to an encoded instruction no longer keeps it encoded. Tier 1 and Tier 2 share this decoder.
  • Tier 1 on every field. Tier 1 decoded only risky-named fields. It now decodes and scans encoded values in every string field. A decode counts only when it trips a real attack pattern, so a benign body that merely uses the word "ignore" isn't flagged. Plain text in non-risky fields is still not scanned.
  • Tier 3 reads decoded text. The whole-result serializer gives the reviewer encoded spans decoded, in place of the encoded value, and coverage counts the decoded text. ROT13 and ROT47 are left as they are for the reviewer: they rewrite the whole string, so plain text with one ROT13 word would otherwise reach it scrambled. Tier 1 and Tier 2 still decode them.

Only the analysis copies change. The returned result is never decoded.

Measured

On Modal, 0.9.6 main (fe80823) vs this branch, on the same eval sets as #155 plus new Gmail-shaped sets. Each body was base64url-encoded in payload.body.data, as ASCII, as UTF-8 with non-ASCII punctuation, and as a multipart text+HTML message.

Set Main This PR
Gmail attacks, UTF-8 (1,000) 3 412
Gmail attacks, multipart (1,000) 4 433
Gmail attacks, ASCII (1,000) 404 479
Encoded attacks: base64 / base64 in prose (3,229 each) 1,834 / 2,870 2,056 / 2,996
Benign blocks on existing sets (20,817) 1,490 1,490
Gmail benign, UTF-8 / multipart (1,162 each) 2 / 4 88 / 107
  • Gmail attack recall now matches the same attacks sent as plain text (about 44% for these email-style attacks).
  • The new Gmail benign blocks match the rate at which Tier 2 blocks the same texts sent plain (CUAD 4.6–5.2% encoded vs 7.1% plain; Toucan 6.4–6.6% vs 5.2%). Before this change Tier 2 never read these bodies.
  • No benign changes on the existing sets.
  • Latency is unchanged on existing sets. Calls with an encoded body pay for scoring the decoded text: UTF-8 Gmail p50 5.5 → 17.7 ms; multipart, two bodies, 10 → 41 ms.

Tests

  • Decoder: a UTF-8 base64url body decodes; binary data (invalid UTF-8, control bytes) stays encoded.
  • Tier 1, model-free: an encoded attack in payload.body.data is flagged; an encoded benign body and plain text in non-risky fields aren't.
  • Tier 3: the reviewer gets the decoded body line by line, coverage isn't degraded, plain text with a ROT13 word isn't scrambled, and the returned result is unchanged.
  • Decoder: an encoded instruction with control bytes appended still decodes.
  • Conformance:
    • tier1-gate: utf8-base64-in-any-field and utf8-base64-benign;
    • tier3-serialize: tier3-reads-encoded-strings-decoded and tier3-no-rot13-of-plain-text;
    • the decode rule in conformance/README.md.
  • The Python port follows in StackOneHQ/defender-py.

pnpm test, pnpm lint and pnpm test:typecheck pass.

🤖 Generated with Claude Code

hiskudin and others added 2 commits November 18, 2026 16:35
… tier

- A base64 decode is kept when its bytes are valid UTF-8 text without
  control characters, not only when it is pure ASCII. A Gmail body with
  an accented letter or a dash was never decoded.
- Tier 1 decodes and scans encoded values in every string field, not
  only risky-named ones. Plain text in other fields is still not scanned.
- Tier 3 reads encoded strings decoded. Coverage counts the decoded text.
  Only the reviewer's copy changes; the returned result doesn't.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread src/core/tool-result-sanitizer.ts Resolved
Comment thread src/core/prompt-defense.ts Resolved
hiskudin and others added 2 commits November 18, 2026 16:51
… spaces

- Tier 3 no longer ROT13- or ROT47-decodes: those rewrite the whole
  string, so plain text with one ROT13 word reached the reviewer
  scrambled. Tier 1 and Tier 2 still decode them.
- A base64 decode is kept when it's valid UTF-8; control characters
  become spaces. Before, appending a NUL to an encoded instruction kept
  it encoded for every tier.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t of cumulative risk

- In cascade chunk review, the top-scoring chunk can be an encoded copy
  Tier 2 also scored; the reviewer now reads it decoded.
- A match found by decoding a non-risky field escalates that field only.
  Those fields aren't counted as processed in the cumulative tracker, so
  feeding it their matches let a few of them escalate the whole result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread src/core/tool-result-sanitizer.ts Resolved
@hiskudin
hiskudin merged commit c9931e9 into main Nov 18, 2026
5 checks passed
@hiskudin
hiskudin deleted the feat/eng-4342-decode-utf8 branch November 18, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant