Repository navigation
feat(ENG-4342): decode UTF-8 encoded values in every field, for every tier - #160
Merged
Merged
Conversation
… tier - A base64 decode is kept when its bytes are valid UTF-8 text without control characters, not only when it is pure ASCII. A Gmail body with an accented letter or a dash was never decoded. - Tier 1 decodes and scans encoded values in every string field, not only risky-named ones. Plain text in other fields is still not scanned. - Tier 3 reads encoded strings decoded. Coverage counts the decoded text. Only the reviewer's copy changes; the returned result doesn't. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… spaces - Tier 3 no longer ROT13- or ROT47-decodes: those rewrite the whole string, so plain text with one ROT13 word reached the reviewer scrambled. Tier 1 and Tier 2 still decode them. - A base64 decode is kept when it's valid UTF-8; control characters become spaces. Before, appending a NUL to an encoded instruction kept it encoded for every tier. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t of cumulative risk - In cascade chunk review, the top-scoring chunk can be an encoded copy Tier 2 also scored; the reviewer now reads it decoded. - A match found by decoding a non-risky field escalates that field only. Those fields aren't counted as processed in the cumulative tracker, so feeding it their matches let a few of them escalate the whole result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Encoded values are now decoded wherever they are, for every tier. Example: a Gmail message whose body comes base64url-encoded in
payload.body.data.Only the analysis copies change. The returned result is never decoded.
Measured
On Modal, 0.9.6 main (fe80823) vs this branch, on the same eval sets as #155 plus new Gmail-shaped sets. Each body was base64url-encoded in
payload.body.data, as ASCII, as UTF-8 with non-ASCII punctuation, and as a multipart text+HTML message.Tests
payload.body.datais flagged; an encoded benign body and plain text in non-risky fields aren't.tier1-gate:utf8-base64-in-any-fieldandutf8-base64-benign;tier3-serialize:tier3-reads-encoded-strings-decodedandtier3-no-rot13-of-plain-text;conformance/README.md.pnpm test,pnpm lintandpnpm test:typecheckpass.🤖 Generated with Claude Code