[exKAZUu]

Why

Since the scanner started to recognize Unicode escapes before tokenization, it reads every identifier, whitespace run, and comment through its decoding path, and the generated lexer only gets what the scanner rejects. On the same runtime, a sample of 200 Java files (UTF-16 input, min of 7 rounds × 5 repetitions) parses in 59.2 ms with v18.0.0 and 73.5 ms with v32.0.0; scan_token takes about 16% of the parse time in a profile. Most of what it does there is per-character overhead that the decoding does not need: an eof call before every character, a call of mark_end after every comment character, the decoding call for a plain ASCII letter, and, for operators and separators, reading one character only to reject it.

This change removes that overhead without changing which tokens the scanner produces. Letting the generated lexer read plain identifiers was rejected: the scanner can tell that a word has no escape only after reading the whole word, so it would read every identifier twice.

Requirements

• required: Investigate why parsing became slower and tune performance as much as possible (requester's instruction).
• required: Every tree, for valid and invalid input, stays identical.
• required: Parsing never crashes or hangs and takes time linear in the input size (Maintenance Scope).
• chosen: Change only src/scanner.c and keep its token decisions, because any grammar change would regenerate the parser and needs its own proof of identical trees.

Customer Summary

Java parsing is about 6% faster (73.5 → 69.2 ms on the 200-file sample; 2996 → 2822 ms on the whole examples/ corpus). Long comments parse 30–40% faster. Trees are unchanged.

Technical Summary

All changes are in src/scanner.c:

1. scan_token reads a plain ASCII identifier start without next_code_point, and returns false before advancing for any other plain ASCII character that starts no token it reads (anything but whitespace, /, \, and U+001A). A false return is undone by the runtime's lexer reset, so reading less before it changes no token.
2. eof is called only when the lookahead is 0, which the runtime sets at the end of the input. The checks that follow a test for a non-zero character (hex digits, raw whitespace) drop it.
3. A comment body reads plain characters in skip_comment_characters and marks its end only before a character that may end it (\, CR, LF, /, or 0). The last mark therefore lands where marking after every character put it.
4. next_identifier_part holds the per-character step of scan_identifier, with one call of identifier_part, so that the compiler inlines its ASCII path. spell returns early once the spelling overflows, and is_keyword compares first characters before calling strcmp.

What to check hardest: whether the early rejection in item 1 covers every character for which the following code could produce a token, and whether the lazy end marking in item 3 leaves the same token end for terminated, unterminated, and malformed-escape comments.

Testing

• Hashes of every node (kind, flags, field, byte range, points, and parse states), for UTF-8 input and for UTF-16 input, are identical to main on the whole examples/ corpus (7,940 files), on the inputs of every test/corpus case (166), and on 16,152 generated files. The generated files are mutated corpus inputs and example files with random Unicode escapes (\uuuu0041, escaped keywords, escaped quotes, \u000a and \u000d, escaped comment delimiters, odd and even backslash runs, lone and paired surrogates, malformed escapes, NUL, U+001A, ignorable characters, invalid UTF-8), plus random snippet mixes. About 95% of them have syntax errors.
• Inputs of 0.5 MB and 2 MB of long identifiers, escaped identifiers, \uuu… runs, long line, block, and unterminated comments, backslash runs, whitespace, encoded whitespace, strings, surrogates, and repeated statements take about 4× as long at 4× the size, both before and after this change, with identical trees.
• bun run verify and bun wb test (unit and e2e, with the wasm rebuilt from this scanner) pass.
• CI passes, including the Robustness workflow, which regenerates src/ and finds no drift on three platforms and runs the fuzzer, and the Rust tests.