gh-159449 - Fix a free-threaded asyncio race that corrupts the task lists

[jblespiau]

See #159449

When a thread exits with pending asyncio Tasks, PyThreadState_Clear() moves them onto the shared interp->asyncio_tasks_head list, but each Task still records the _Py_ThreadId() of the thread that created it. unregister_task() skips the stop-the-world pause whenever that id matches the current thread. That also happens for a later thread that reuses the exited thread's pthread_t, and for destructors that run later in PyThreadState_Clear(). The resulting unlocked llist_remove() calls on the shared list race with each other and can leave an already-removed node (next == NULL) linked into it. asyncio.all_tasks(), which asyncio.run() calls on shutdown, then crashes with SIGSEGV in _asyncio_all_tasks(). Record the unique id (PyThreadState.id) of the creating thread state instead, and only take the lock-free path while that thread state is not being cleared. Also drop the assertion in add_tasks_llist() that every linked Task is pending. A Task's state is set before unregister_task() unlinks it, and that removal can be waiting for the stop-the-world pause during which the list is walked. add_one_task() already filters out Tasks that are done.

I don't know if the bug is also present in 3.15, it's worth checking. I am hoping that given we have a reprocuder in the bug, + a bug-fix, it should be doable for the maintainers to port it.

• Issue: AsyncIO thread-safety issue in Python 3.14's free-threading #159449

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[jblespiau]

I think I did it. I used blurb-it.

[jblespiau]

Note that it's not as far as I am aware a backport of a fix; I haven't looked at whether the bug is still here for newer versions. So I don't think it should be titled as "[3.14] ..."

[AnnaAr321]

This also reproduces in the Python 3.15 free-threaded build

[StanFromIreland]

Please open a PR against main.