[exKAZUu]

Why

The external scanner stopped every string and text block fragment after 256 characters (MAX_FRAGMENT_LENGTH), so class A { String s = "<700 × a>"; } yielded three string_fragment nodes, and a 440-character literal yielded [17–273] and [273–439]. Other WillBooster grammars (JavaScript, C#, Go) yield one fragment per run of text. The bound was added in #83 to stop error recovery from scanning the rest of a malformed record header as one long fragment; since #93, the scanner reads no string fragment during error recovery, so the bound only splits valid literals.

Requirements

• required: a string literal's contiguous plain text yields one string_fragment (one multiline_string_fragment in text blocks); escape sequences and other legitimate splits stay as they are
• required: check text blocks, character literals, and any other token the scanner caps the same way, and fix them all
• required: corpus tests with literals longer than the cap
• required: parse time stays linear for very long literals (e.g., 1 MB)
• required: valid input still parses correctly, and invalid input never crashes or hangs and parses in linear time
• required: a tree-shape change is a breaking fix!: commit so that it is released as a new major version
• chosen: remove the length bound instead of merging fragments elsewhere, because the bound now only splits valid literals (error recovery reads no string fragment since fix!: keep syntax errors local to the mistake #93)

Customer Summary

A long string literal or text block now has one fragment node per run of plain text instead of one every 256 characters. This changes the tree shape, so it needs a major release.

Technical Summary

scan_fragment_characters in src/scanner.c now reads up to the next quote, backslash, or (outside text blocks) line terminator without a length bound. It served both string literals and text blocks, including text block fragments that start with one or two quotes. No other token is capped: character literals are lexed by the generated lexer as one token, and the scanner's comment, whitespace, and identifier scans have no length bound (the identifier's fixed spelling buffer only disables the keyword comparison once it overflows). src/parser.c changed only because the generation profile includes the new corpus entry.

Testing

• bun run verify passed.
• bun wb test on the performance, corpus, string literal, text block escape, error location, real-world example, Unicode literal, and string template tests passed (113 tests), including the existing linear-time check for malformed record components that motivated the bound.
• Added a corpus entry long string literals (342-character runs in a string, around an escape, in a multi-line text block, and after a leading quote in a text block); it fails with the previous scanner.
• Added linear-time checks for 100 KB versus 1 MB string literals and text blocks, and for an unterminated 1 MB text block. script/tree-sitter parse --time on 0.18 MB, 1.8 MB, and 18 MB inputs took 1.7 ms, 12 ms, and 119 ms.