combine utf-16 surrogate pairs when decoding strings

[iefa-m]

The decoder turns each \u escape into one code point, so an astral character written as a surrogate pair like "\ud83d\ude00" comes back as two lone surrogate code points rather than the character they stand for. That disagrees with the json module and with ECMAScript string semantics, it breaks round-tripping of dumps() output (which emits astral characters as surrogate pairs under ensure_ascii), and the resulting string raises UnicodeEncodeError the moment anything tries to encode it to utf-8.

Combine well-formed high/low surrogate pairs back into a single code point when walking decoded string values and object keys. Doing it in the ast walk keeps it at the semantic-conversion layer rather than in the generated parser, and it covers both values and keys in one place. Lone surrogates are deliberately left as they are, which is what the json module does too.

[dpranke]

Hi,

Thanks for the bug report, but I'm having difficulty reproducing what you claim. From what I can tell, it seems like the json module leaves the surrogates alone (using 3.14.2):

>>> import json
>>> import json5
>>> json.dumps('\ud83d\ude00')
'"\\ud83d\\ude00"'
>>> json5.dumps('\ud83d\ude00')
'"\\ud83d\\ude00"'
>>> json.dumps('\ud83d\ude00', ensure_ascii=False)
'"\ud83d\ude00"'
json5.dumps('\ud83d\ude00', ensure_ascii=False)
'"\ud83d\ude00"'
>>> json.dumps('\ud83d\ude00') == json5.dumps('\ud83d\ude00')
True
>>> json.dumps('\ud83d\ude00', ensure_ascii=False) == json5.dumps('\ud83d\ude00', ensu\
re_ascii=False)
>>> json5.loads(json5.dumps('\ud83d\ude00')) == '\ud83d\ude00'
True
>>> json5.loads(json5.dumps('\ud83d\ude00', ensure_ascii=False)) == '\ud83d\ude00'
True

It seems like Javascript does as well, although I'm not certain. Using the console in Chrome's devtools, JSON.parse('"\d83d\ude00"').length returns 2.

That said, I admit my understanding of how surrogates work in either language is weak, so I might easily be missing something? I do see text in the Ecmascript standard (https://tc39.es/ecma262/#surrogate-pair) that suggests that the surrogates are supposed to be combined, but it doesn't seem like Chrome/V8 at least is doing that? But maybe by using escape sequences in the literals instead of the actual code points I'm not doing it right?

[iefa-m]

The confusing part is that in Python source, '\ud83d\ude00' is already a string of two lone surrogate code points (its len() is 2), not U+1F600, so every dumps() call in your session starts from a string that already contains unpaired surrogates, and both encoders escape those identically. The divergence is on the loads side, when the escaped pair appears in the JSON text itself:

>>> import json, json5
>>> json.loads('"\\ud83d\\ude00"')
'\U0001f600'     # one code point, the stdlib combines the pair
>>> json5.loads('"\\ud83d\\ude00"')
'\ud83d\ude00'   # two lone surrogates

Your round-trip test actually shows the disagreement from the other side: json.loads(json.dumps('\ud83d\ude00')) == '\ud83d\ude00' is False with the stdlib (you get back the combined character), while json5 returns True.

The case that bites users starts from a real astral character:

>>> s = '\U0001f600'
>>> json.loads(json.dumps(s)) == s
True
>>> json5.loads(json5.dumps(s)) == s
False
>>> json5.loads(json5.dumps(s)).encode('utf-8')
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed

On the Chrome side, a length of 2 is expected and doesn't mean the pair was left uncombined: JS strings count UTF-16 code units, so '😀'.length is also 2. In JS the surrogate pair in a string is the astral character, there is no separate uncombined representation to distinguish. You can see it with JSON.parse('"\ud83d\ude00"').codePointAt(0).toString(16), which gives "1f600", or JSON.parse('"\ud83d\ude00"') === '😀', which is true. Python strings are sequences of code points rather than UTF-16 units, so the combining has to happen explicitly in the decoder, which is what the stdlib's scanstring does in Lib/json/decoder.py (the surrogate pair branch), and what this patch mirrors.

[dpranke]

Ah, right.

[dpranke]

Thanks for the fix!