C API: Detect invalid UCS4 characters in debug mode

[vstinner]

Currently, PyUnicode_FromKindAndData() and PyUnicodeWriter_WriteUCS4() create an invalid string if a character is not in the [U+0000; U+10ffff] range. Instead, I propose to raise an exception in this case.

Example creating an invalid string:

$ python3.15
>>> import _testcapi, sys
>>> invalid_ucs4 = (0x11_0000).to_bytes(4, byteorder=sys.byteorder)
>>> s = _testcapi.unicode_fromkindanddata(4, invalid_ucs4*5)

>>> list(s)
SystemError: invalid maximum character passed to PyUnicode_New

>>> s[0]
SystemError: invalid maximum character passed to PyUnicode_New

Linked PRs

• gh-158445: Reject invalid UCS4 in PyUnicode_FromKindAndData() #158447
• gh-158445: Detect invalid UCS4 characters in debug mode #158502

[vstinner]

With 32-bit wchar_t (ex: Unix), PyUnicode_FromWideChar() and PyUnicodeWriter_WriteWideChar() already raise a ValueError if a character is outside the [U+0000; U+10ffff] range.

[vstinner]

Other Python and C functions raising an exception if the code pointer is negative or is greater than 0x10ffff:

• Python chr() raises ValueError.
• PyUnicode_FromOrdinal(), PyUnicode_Fill() and PyUnicodeWriter_WriteChar() raise ValueError.
• PyUnicode_FromFormat("%c") raises OverflowError.
• PyUnicode_New() raises SystemError.

PyUnicode_WriteChar() raises ValueError if the character is greater than the string maximum character.

[vstinner]

Since checking for invalid characters at runtime was rejected, I repurpose the issue to "Detect invalid UCS4 characters in debug mode".