Skip to content

Replay service cooperative cancellation at the canonical boundary - #90

Draft
rmcdaniel wants to merge 37 commits into
mainfrom
feat/cooperative-cancellation
Draft

rmcdaniel wants to merge 37 commits into
mainfrom
feat/cooperative-cancellation

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Cooperative cancellation source candidate

Implements shared cancellation issue 136
in the Python SDK. Default and published worker protocol remains 1.19.
Cooperation requires explicit source protocol 1.20, installed Server/Native
discovery and matching worker capabilities. This PR remains draft.

The candidate includes immutable cancellation context and lineage, deterministic
remaining-time helpers, operation policies for children and remote activities,
physical supervised callback stop, prepared sequential and atomic local
admission, stale-attempt fences and durable cleanup recovery after owner loss.

Current change

Current head d41e76deb76d3a04c25c5879984b85bb9c23cda3 retains the runtime
implementation from afa7cfaabd0d7fad8049f7b6e3b4c19ce15d0aa2 and corrects
only the connected assertion for valid historical policy omission. It adds explicit prepared local
Activity TryCancel and WaitCancellationCompleted. It preserves historical
omission, refuses local Abandon and unsupported/legacy policies before callbacks
or checkpoint submission, registers only discovered policy support, and checks
original canonical policy on cold replay. Queries, updates and validators use
the same negotiated replay consumer.

Local implementation qualification passes 2,040 non-integration cases, two existing skips,
zero failures/errors, plus Ruff and mypy. Dedicated policy coverage passes
38 cases with zero skips. Actual supervised callback tests cover omission
and both explicit policies with no application heartbeats.

Exact source qualification
passes all required jobs against Server dd8996fdd6d488b48d575b1b934821bd8bedc8ed and Native
ba1aa4770c4b94533548bd45bd3ba028229c54e8. Connected scenarios now cover
sequential and atomic cleanup SIGKILL recovery for omission, Try and Wait,
preserving the original request, delivery and 30-second budget. All six cases
pass. Connected integration finishes with 54 passed, one existing unavailable-CLI
skip and zero failures/errors. Ordinary exact-head CI also passes. Current
results and raw artifact 11260048832 are retained in the PR comments and Actions.

The preceding Source attempt finished with 52 passed, one existing unavailable
CLI skip and two historical-omission assertion failures. Both explicit policies
passed sequential and atomic SIGKILL recovery. The assertion mistakenly required
an explicit canonical field where omission means historical Try. The full suite
passes after this runner correction. Raw failed-run evidence
is retained through January 1, 2027. This is not claimed as a completed Source pass.

The preceding exact implementation head
0e1c427eae2f6120155a5cf3c9c1ae79ba7a5756
passed its source qualification.
Its evidence remains attached to its tested tuple and does not qualify new code.

Remaining gates

Complete the remaining model decisions and current mixed source qualification,
then qualify exact published Server/PHP/Python/Rust artifacts in the required
mixed-language cascade with one API/CLI/UI view and fair competitive evidence.
Shared #136 stays open. This candidate is not published or adopted by Cloud.

@rmcdaniel

Copy link
Copy Markdown
Member Author

All source checks have passed for 18ec381ea7c0d9378775df033893b9069bdb7802, including Python 3.10, 3.11 and 3.12, lint, regression corpus policy, built package, docs, public boundary and the existing Server integration suite. Target branch qualification completed successfully at 23:44 UTC September 30.

The connected cooperative worker/client path is still the next implementation step. Existing integration success verifies the preserved supported protocol 1.19 surface. It does not qualify the new delivery intent, heartbeat observation, acknowledgment recovery or active local activity fencing against Server PR291. This PR remains a draft and advertises no cooperative cancellation capability. The completed local test worktree and tool images are removed, with the active branch preserved.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Request and delivery client source

a3c909c403634061cea81ebeb200342f31f14d9e adds separate request and delivery carriers. Request uses control credentials, an optional selected-current-run route and Server's original request metadata. Missing/false capability discovery, malformed/older protocol, invalid timeout and mismatched acknowledgment fail before execution is treated as accepted. A handle retains its selected run. There is no fallback to immediate cancel.

Delivery requires explicit compatible protocol 1.20, worker credentials, the recorded lease owner/attempt and a canonical call/range. Success must acknowledge the exact task/request/boundary. Transport retry preserves that body. Lease and attempt refusals retain Server's machine reason and do not retry as another claim.

Local source check on Python 3.12.14:

pytest tests/test_cooperative_cancellation_client.py tests/test_client.py tests/test_cooperative_cancellation.py tests/test_replay_regression_corpus.py -q
ruff check src/ tests/
mypy src/durable_workflow/
python scripts/ci/validate-regression-corpus.py --base-ref f542486e125fdb2738d0d6185ff8fe748e89ddcd

All 323 tests pass, including 52 new carrier cases. Lint, strict typing and corpus policy pass. CI36793706333 is qualifying this head.

The next Worker change must consume claim and heartbeat observations, persist the replay delivery intent before throwing, then reload Server-issued canonical history with the same owner/attempt. Active local activity results must be fenced across delivery, death and reclaim. Default protocol/capability advertisement remains unchanged until that path, all three SDK implementations and the exact published tuple qualify.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Client source qualification complete

At a3c909c403634061cea81ebeb200342f31f14d9e, normal CI passed every gate, including all three Python versions, lint, corpus, package and existing Server integration. Docs and boundary checks passed too. Python 3.11 reports 1,577 passing tests, two skips and 31 integration cases deselected.

The new client carrier requires explicit compatible capability discovery for cooperative requests. A run-bound handle preserves its run fence. Worker delivery requires protocol 1.20, worker credentials, claim owner/attempt and the authored call range. Duplicate requests preserve Server's original identity and deadline. Lost acknowledgments retry the identical delivery, and malformed or mismatched acknowledgments are rejected. Earlier immediate cancellation and termination remain covered.

The completed source worktree, dependencies, caches, tool images and build context have been removed. The active draft branch remains. Next is Worker integration for claim/heartbeat observations, canonical history refresh, delivery persistence and active local-result fencing, then the accepted connected failure cases and equivalent PHP/Rust behavior. Default protocol and published capability claims remain unchanged.

@rmcdaniel

rmcdaniel commented Oct 1, 2026 •

Copy link
Copy Markdown
Member Author

Worker source gates completed

Exact head febb5906541f4df82cd3f4f3e2c11f41e20acff6 passed every normal CI gate in run 36797286205, including Python 3.10/3.11/3.12, lint, corpus, package, existing Server integration and target-branch qualification. The last gate completed on October 1 at 00:44:19 UTC. Docs run 36797286266 and public boundary checks passed as well.

The local suite passed 1,607 tests with two existing skips and 31 integration cases deselected. Thirty Worker cases exercise claim/heartbeat observation, canonical paging and delivery, earlier commands, acknowledgment loss, local work and lease fencing, shielded cleanup, cold replay and incapable registrations.

The draft remains protocol 1.19 by default. Connected candidate Server qualification, actual process death/reclaim, shutdown during shielded cleanup, synchronous and remote in-flight work, deadlines and termination remain required, along with PHP/Rust equivalents and coordinated publication. Existing Server integration covers the preserved current protocol surface.

Cleanup is complete: the clean source worktree, virtual environment, test logs, build context and both disposable tool images are removed. The remote branch and this evidence retain the implementation and reproducible commands.

@rmcdaniel

rmcdaniel commented Oct 1, 2026 •

Copy link
Copy Markdown
Member Author

Negotiated local cleanup shutdown and synchronous execution

Exact head 797e3935d7a738eb91fc98a396a772ced82bd31f drains negotiated local cleanup within the configured worker shutdown grace period. At the deadline, it fences local execution and abandons the workflow claim without producing an activity cancellation/failure or a completion. A replacement worker consumes the original committed marker, uses a new owner/attempt and resumes cleanup with the original request/deadline.

Synchronous local handlers execute off the event loop in a lazy pool bounded by max_concurrent_workflow_tasks. It is separate from replay threads, which wait for local results. Context variables preserve activity metadata through that boundary. Shutdown and request observation fence late heartbeats and unencodable results from both async cancellation-suppressing handlers and synchronous handlers. Python cannot forcibly stop a running thread, and external effects retain their at-least-once idempotency/reconciliation requirement.

Local evidence: 1,611 tests passed, two existing skips, 31 integration cases deselected, 26.77seconds. All 34 focused Worker cases pass. Ruff, strict mypy for all 26 source modules and the revision-aware corpus guard pass. The synchronous observation case deliberately limits replay to one thread to expose a shared-pool deadlock. Replacement source cases preserve canonical request/history and use a different lease owner and attempt5 without redelivery.

Execution uses Python3.12.14, UID/GID1000:1000, two CPUs, 1GiB memory with no additional swap allowance and pids256. Python base index is sha256:f77ac9e44ae96ef2c90b8053ea08c31f8be030f824196b0ae4db6d462c84e51f. Git-only tool image config is sha256:eb1288c6265cdae40a2ea36b8e07a627e99c7ae4bfacf71647d899a6a1357dfa. After installing the editable package with its dev dependencies, commands were:

ruff check src/ tests/
mypy src/durable_workflow/
pytest tests/ -m "not integration" -q
python scripts/ci/validate-regression-corpus.py --base-ref f542486e125fdb2738d0d6185ff8fe748e89ddcd

Normal CI for the exact head completed successfully at 01:25:59 UTC on October 1. All jobs pass, including Python 3.10/3.11/3.12, existing Server integration, lint, corpus, package and target-branch qualification. Docs and public boundary checks also pass at the same head.

The draft remains pending connected candidate Server qualification, actual native process replacement, native shutdown/in-flight work, deadline/termination and acknowledgment-loss cases, PHP/Rust equivalents and coordinated publication. Default protocol1.19 and current registration advertisement remain intact. Completed local worktree, dependencies, build context, diagnostics and task images have been removed. No published support claim is added by these source checks.

@rmcdaniel

rmcdaniel commented Oct 1, 2026 •

Copy link
Copy Markdown
Member Author

Connected Python/Server source qualification

At October 1, 2026, 02:25 UTC, Python 119747bb2fb838f6dff64c9938c780414a074b99 and Server 2e2d6b31df981e98c1a054a8953e9efe95d17e6e pass all 11 connected cases on SQLite in 20.70 seconds. This uses the actual authenticated API, real database-backed queue worker, actual claim/heartbeat/history endpoints and separate OS processes. Protocol 1.20 is configured explicitly. Published defaults remain 1.19.

Cases Verified outcome
Request before claim, ordinary response Duplicate requests keep the original ID and deadline. One canonical delivery precedes cleanup.
Request before claim, discarded successful delivery response Canonical history proves the committed delivery. No second delivery call or cleanup duplication.
Already waiting on a timer Delivery cancels the pending timer, then shielded cleanup completes.
Leased remote activity The heartbeat refuses continuation during cleanup. Late result and failure receive typed run_cancelled responses and add no history.
Active local work, async and synchronous Actual lease observation catches the request without a user heartbeat. The late unencodable result is discarded. Only cleanup records completion.
Shutdown within grace Shielded cleanup drains and the run closes as cancelled.
Shutdown after grace The old local claim is abandoned. Deregistration records the ordinary repair, with the original cancellation history unchanged. A different owner and later attempt resumes the original request/deadline.
SIGKILL and cold process replacement The first child is killed with exit -9 during cleanup. A new child claims a later attempt and consumes the existing marker. One cleanup result and one terminal cancellation remain.
Cleanup deadline Expiry closes the run and fences the still executing local result.
Termination during cleanup The run closes as terminated and the still executing local result is fenced.

The host used isolated containers at UID/GID 1000, 2 CPU and 1 GiB memory with no additional swap per tool/service. Runtime base is published Server 2.4.34 index sha256:1b660a7228be15a777b46dfb720ba85b083cf8ee70d6d1cd29853118bb1b2715, overlaid with the candidate Server checkout and its unchanged Composer lock. Workflow is published 2.3.0. PHP is 8.3.35 and Python is 3.13.5. The disposable Git-enabled tool config is sha256:8827adb0b23799616f93f587c6555debc559926fefcca61c4e871f20598ee32a. These are source results, not published capability qualification.

Local checks:

  • pytest tests/integration/test_cooperative_cancellation.py -v, with DURABLE_WORKFLOW_COOPERATIVE_QUALIFICATION=1 and worker protocol 1.20, passes all 11 cases at the exact heads above.
  • pytest tests -m 'not integration' -q passes 1,618 tests, two existing skips and 42 integration cases deselected, in 13.37 seconds. Ruff and strict mypy pass for 26 source modules.
  • Server's discovery/request/cluster/success filter passes 145 tests and 5,165 assertions, in 15.258 seconds. PHPUnit also reports 11 annotation metadata deprecations. Pint passes for the four changed PHP files.

Normal Python CI and the explicit Python/MySQL candidate run pass every job at the exact head. The candidate integration completed at 02:29:45 UTC, with 33 passed and one existing CLI skip in 80.19 seconds. All eleven cooperative cases are present and passed in the downloaded JUnit artifact, which expires October 8. The log verifies the exact Server SHA above, and stack teardown passed. Supported Python 3.10/3.11/3.12, lint, corpus, package, docs and public boundaries pass. Server source/corpus CI passes 2,279 tests and 47,800 assertions. MySQL replay/query topology passes. Polling bounded-growth smoke and its performance qualification also pass at this exact head. All normal Server gates are complete. PHP/Rust equivalents, coordinated specification/capability activation and exact published tuple conformance remain required.

Both PRs remain drafts. Local stack and disposable source/dependency/proof/tool resources are removed. Downloaded CI diagnostics and transport files are removed after updating this record.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Next source phase: supervise actual cooperative remote activity callbacks through the qualified Server readonly observation route (Server073a516bbd4063f57d4ad65ad732ec423131c8ff). Keep worker registration and only authored activity progress, observe canonical cancellation/attempt/session/deadline fences, abandon on failed observation or shutdown expiry, and reject late heartbeat/result/failure publication. Qualify blocked async callbacks and synchronous handlers while preserving event-loop responsiveness and the existing bounded thread-pool model. Python threads cannot be forcibly stopped, so prove their late publication is fenced and document the remaining external-effect/process-supervisor responsibility. Extend the connected actual-worker cases and retain exact gates before claiming parity. Ordinary protocol1.19 registration and published capabilities stay unchanged.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Remote worker qualification is running at Python 88f0e31bf1736271deaabcc67d74df9ce98df491 against Server 073a516bbd4063f57d4ad65ad732ec423131c8ff and the published Workflow package selected by that Server source.

The exact-head ordinary CI and connected integration pass. The opt-in candidate run 36825588696 passes 39 connected cases, fails the new killed-remote-owner recovery case, and retains the existing CLI skip. All six new async/sync blocked-callback and shutdown cases pass. The replacement process in the kill case times out before receiving its workflow claim. Its result is not a qualified recovery claim.

Next action: reproduce that case in an isolated local stack using the same Server commit and inspect task leases, registration state and durable history. Determine whether the failure is a fixture bound or a recovery defect before changing it, then rerun the exact-head candidate gate. The failed runner completed teardown successfully and retained JUnit in artifact 11145516155 until October 8. Protocol 1.20 remains explicitly opt-in and this PR remains draft.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The isolated reproduction identifies the failure as Workflow #599. A still-issued workflow poll claims the new task for the killed process. Native repair then hides that expired workflow lease behind the older activity lease and repeatedly returns repair_not_needed.

A native regression using the actual workflow and activity bridges fails on the published baseline, while its fresh-lease case passes. Prioritizing expired leases in the native run summary makes both cases pass, with 33 assertions checking that only the workflow claim changes and the activity task, execution, attempt and history stay unchanged.

The unchanged Python SIGKILL case passes in 18.28 seconds when the isolated Server loads that corrected source. Its 30-second claim bound is unchanged. The replacement workflow task has attempt 2 and repair count 1, commits canonical delivery/cleanup, and the dead activity owner's late completion and failure are rejected. This local source result is the counterfactual for the defect. The full cooperative source suite, Workflow quality cycle, patch publication and exact dependent candidate gate are still running or pending. Protocol 1.20 remains opt-in.

Next action: qualify and publish the Workflow repair patch, update the Server candidate to that exact package, then rerun Python's entire connected candidate suite and retire the local qualification resources.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The cooperative Server candidate now consumes published Workflow 2.3.1 at fb3f3e59a4342fdebf8ced6160798906c3ee4387. Exact Server head is c32434cc784a1bf870dcf19568bef944a320f6a7. It merges the reviewed Server #292 dependency/release-metadata change into the previous 073a516bbd4063f57d4ad65ad732ec423131c8ff candidate. The candidate's service implementation is unchanged and protocol defaults remain 1.19. Normal source checks have restarted for this exact tuple.

Workflow's complete source matrix and all 16 published Laravel/PHP upgrade combinations pass. Server #292 is separately qualifying stable image 2.4.35. A focused ordinary-protocol signal drill reproduces the expired-workflow/older-active-activity defect on published Server 2.4.34 and PHP SDK 2.1.6, so this repair also affects existing service callers.

The entire Python candidate CI is dispatched at unchanged Python 88f0e31bf1736271deaabcc67d74df9ce98df491, with server_commit=c32434cc784a1bf870dcf19568bef944a320f6a7 and cooperative_qualification=true. It must pass the actual remote-worker supervision and SIGKILL cases using the published native correction. This is source-candidate qualification, separate from Server #292's published-image PHP/Python/Rust follow-through and shared #136's activation/reclaim/Rust gates.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Exact native correction requalification complete

Python head 88f0e31bf1736271deaabcc67d74df9ce98df491 passes connected CI 36831983386 against Server c32434cc784a1bf870dcf19568bef944a320f6a7 and published Workflow 2.3.1 at fb3f3e59a4342fdebf8ced6160798906c3ee4387. Every job passes. Integration has 41 cases, 40 passing and one existing CLI absence skip, in 129.300 seconds. All 18 cooperative cases pass, including actual async/synchronous remote callbacks with and without user heartbeats, readonly observation without progress renewal, stale publication fencing, actual shutdown and real remote-owner SIGKILL followed by cold replacement. The killed remote-owner case passes in 11.603 seconds. JUnit is retained through October 8. Job cleanup succeeds.

Server's current exact head passes feature/corpus with 2,300 tests and 47,995 assertions, MySQL HTTP topology, MySQL/Postgres rolling, bounded polling, chart validation/install and public boundaries.

The correction is also published independently in stable Server 2.4.35 with its default protocol 1.19. Its focused published recovery test and all 12 published portable lifecycle cells pass. These candidate source results do not activate cooperative protocol 1.20. Next action is finish Rust's equivalent actual remote-worker path and active activity-attempt reclaim qualification, then coordinate publication and exact published tuple gates. PHP/Python/Server PRs remain drafts through that work.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Requalified against the current Server and published Native package

Python 88f0e31bf1736271deaabcc67d74df9ce98df491 passes the complete opt-in CI workflow against exact Server 555cf7ab4ed4e9a79b52475be76a6dfcf886dd70, which embeds published Workflow 2.3.1 at fb3f3e59a4342fdebf8ced6160798906c3ee4387.

The connected suite reports 40 passed, 1 skipped in 128.61 seconds, including all 18 cooperative cases. These include actual async/synchronous remote callbacks with and without authored heartbeats, accepted owner registration heartbeats, shutdown fencing, canonical waiting/delivery, lost replies, local callback result suppression, shielded cleanup replacement, real remote-owner SIGKILL, cleanup reclamation in a new process, and cleanup deadline/termination fencing. Ordinary smoke, payload, session and memo integration cases also pass. The supported Python matrix, package, corpus and Avro checks pass in the same run.

Raw connected JUnit is retained for seven days. CI removed its isolated stack, network, payload volume and task images. The Server prefix successor fix is included in this candidate.

Next: complete active remote-attempt replacement qualification and the remaining shared per-language scenarios, then qualify the exact published tuple. This is source qualification, and this PR remains a draft. Published protocol/capability defaults are unchanged.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Child waits release the Python claim

Source 2b28ccad73b7d2ad76d9e5a4dad44a0bd33f2769 now accepts only the explicit
cancellation_waiting_for_child response with claim_released: true, the
matching task, and no claimed delivery boundary. It rejects malformed release
flags, mismatched tasks, fabricated delivery fields and unrelated timer waits.

The worker returns to polling without publishing completion or failure and
without attempting history paging on the completed claim. Its source fixture
executes another workflow task, then replays parent cleanup on a new task while
preserving the original request and deadline. Successful delivery still needs
matching canonical history before authored cleanup.

Focused local source checks passed: 151 tests, Ruff, and mypy over all 26
source files. Exact-head CI is running:
https://github.com/durable-workflow/sdk-python/actions/runs/36930582866

Connected child-policy qualification remains required. This is a source draft,
not a published capability. Default protocol stays 1.19, cooperative 1.20 is an
explicit unfrozen opt-in, and shared cancellation 136 remains open for the
strong model and exact published 30-second mixed-language cascade.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Prepared sequential local consumer qualified against the exact Source tuple

Component Exact commit
Python SDK 5150044c3bec86b8d83e206f5468dbe8a0057d8e
Server c1bda7f81323e9939ef433c7ebca66f62365cdfd
Native readonly overlay 850a64050b57d47181abeae86d163962c9047317

Source CI 37001051027 passed every job. Connected integration passed 43 cases with one existing CLI-binary skip, zero failures or errors, in 480.41 seconds. Both new prepared-local cases passed with no skip.

Durable sequential execution

The 2.775-second case checkpoints memo metadata before admission, executes two distinct backend attempts in supervised processes, records exactly two real application heartbeats, completes both activities and the workflow, then cold-replays the results without callbacks. Canonical history contains one memo event, two Scheduled/Started/Completed activity sequences and one workflow completion.

Original 30-second cancellation and cleanup recovery

The case stops the original local callback without application heartbeats and verifies physical exit before cancellation acknowledgment. It delivers cancellation once, starts shielded cleanup, SIGKILLs the workflow owner, verifies the surviving supervisor stopped the cleanup callback, and runs a replacement owner. Native records unknown stop for the old cleanup attempt, releases the claim for a durable retry, and admits one new cleanup attempt. Both admissions retain the original root/local request IDs, delivery history ID and deadline. Duplicate cancellation preserves request identity, request time and deadline.

Observation UTC
Original request 2026-10-02T11:38:55.742903Z
Original deadline 2026-10-02T11:39:25.742903Z
Workflow Cancelled 2026-10-02T11:39:21.774404Z

Completion was 26.031501 seconds after the request, leaving 3.968499 seconds. The whole test took 28.242 seconds including setup. The qualification stack uses a 10-second workflow-task timeout and a 10-second repair cadence. Independent callback control polls at most once per second and bounds transport by five seconds and original authority. This is one measured Source scenario, with no claim about arbitrary workloads or a published capacity/SLA change.

The original and replacement cleanup admissions share delivery ID 01m3y6j21cy792qvarbzxqyjbp. History contains one request, one delivery, one original activity cancellation, one physical-stop acknowledgment, one unknown-stop retry, one successful cleanup completion and one WorkflowCancelled. It contains no application heartbeat. The two cleanup backend attempt IDs are distinct. Physical exit assertions passed for original work, killed cleanup and replacement cleanup callbacks.

Retained evidence and additional checks

  • Raw histories, receipts, JUnit, image authority and exact source provenance, retained through December 31, 2026. Artifact digest: 5741881d97c424a72d83b0cf87d8b3d21932b366bcd4707d271341a9b10254fe.
  • Connected JUnit SHA256: 6a8f11cfb56059b018c7626dfbca07b93545e80282dc6529b70a86051b364a00.
  • Exact-head full local suite: 1,904 passed, 2 existing skips, 52 connected cases deselected, 119.79 seconds. Local JUnit SHA256: 34aceee7476e3b72fefac609054beb0481ea3182057a96142e2d1bf26bc92c67.
  • Prepared-local checks: 52 passed, no skips, 12.40 seconds. Local JUnit SHA256: 06acfbf0cc564e811338bfb5ce1edc8bd5e559cd4d6af014f8964117172220be.
  • Hosted Python 3.10–3.12, Ruff, mypy, package, corpus and central action policy passed. All current ordinary PR checks also pass.
  • Immutable Source cold-results fixture SHA256: 465e6753bf6b6cfd25bdda2f3a4bc91aedfc3e2d37b64af53b00e3c89d578a1e.
  • Hosted stack/image/volume teardown succeeded.

Reproduce with gh workflow run ci.yml --repo durable-workflow/sdk-python --ref feat/cooperative-cancellation -f server_commit=c1bda7f81323e9939ef433c7ebca66f62365cdfd -f cooperative_qualification=true -f native_commit=850a64050b57d47181abeae86d163962c9047317, after verifying the ref still resolves to the exact Python commit above.

The image retains published Composer Native authority 2.3.3 / 70d4fe48efd7dd796c35c1078b3d5ac43f738f4f; the Source qualification deliberately replaces that package directory with the readonly exact Native overlay identified above. This qualifies the Source consumer, not a published tuple.

PR stays draft. Python atomic local group consumption, operation policies, scopes, the exact published PHP parent → Python child → Rust remote activity / PHP local activity cascade, one inspection view and fair competitive proof remain outstanding. Shared issue 136 stays open. Published artifacts, default protocol 1.19 and Cloud adoption are unchanged.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Atomic prepared local group consumer implemented, connected qualification running

Candidate 0a7c68b140d4b74b73120716b2dff97773ae58ae adds explicit prepared_local_activity_groups negotiation and complete ordinary parallel admission before any callback starts. Nested local/remote/child/timer membership is bounded to 100 leaves. Canonical opening history and execution IDs must agree with the retained checkpoint. All local attempts must validate before spawning. Completed siblings replay stored results. Unfinished Started siblings use Native recovery before new admission.

Group interruption joins all callbacks before workflow replay proceeds. Unknown stop retains capacity and worker registration. The tests exposed cancellation interrupting a supervisor join. Confirmed joins and prepared-finalization now finish before cancellation propagates, with a dedicated process regression test. Result receipts remain required, and Native owns retries and deadlines. Selection and turn-closing waits remain explicitly refused by this admission path.

Local Python 3.12.15 verification passed:

  • 112 focused checks, zero skips, including atomic refusal, partial history, immutable partial completion, original cleanup authority, concurrent callbacks and physical group stop without application heartbeats, 39.81 seconds. JUnit SHA256 43f2c9d96db52b605db55eca90b6fda4660ef39a99da7bbfc1750117ef6c8122.
  • 1,927 full-suite checks, two existing skips, 54 connected cases deselected, 131.45 seconds. JUnit SHA256 782114a1d874861948ee8e94c52854c734734e399e77e60ceb9829382451c83e.
  • Ruff, mypy and corpus policy passed. New immutable group fixture SHA256 8b74c615edc472d8483475efc6b5cbaae0dfb73f919afd9a2924a4295152b45f. Replay corpus grows from 35 to 36, codec corpus remains 41.

Exact Source qualification 37005040454 is running against Server c1bda7f81323e9939ef433c7ebca66f62365cdfd and Native readonly overlay 850a64050b57d47181abeae86d163962c9047317. It includes a nested mixed local/timer group and owner SIGKILL during two-member cleanup under the original 30-second deadline, with raw physical PID evidence. Results are pending.

PR stays draft and shared issue 136 stays open. The earlier sequential report qualifies its own earlier head. This candidate is not published. Operation policies, scopes, the exact published PHP → Python → Rust cascade, one inspection view and fair competitive proof remain required.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Source group qualification failed, nested admission diagnosis in progress

Run 37005040454 completed with 44 passing cases, one failure and one existing CLI-binary skip. Exact tuple:

  • Python 0a7c68b140d4b74b73120716b2dff97773ae58ae
  • Server c1bda7f81323e9939ef433c7ebca66f62365cdfd
  • Native readonly Source overlay 850a64050b57d47181abeae86d163962c9047317

The nested mixed local/timer case failed after a refused prepared operation and a local heartbeat deadline. The original warning did not retain the refusal reason. This run does not qualify atomic groups. The next candidate retains the exact refusal and final canonical history rather than weakening or skipping the scenario.

The two-member cancellation case passed on this tuple. Two active callback processes stopped without application heartbeats. Two cleanup processes exited with their SIGKILLed owner. Two replacement cleanup processes exited after completion. All six physical exit assertions passed. Both unknown-stop recoveries preserved the original root request, local request, delivery event and cleanup deadline. Duplicate requests returned the original identity and deadline. Canonical terminal time was 12:22:28.224484Z, before the original deadline 12:22:41.989114Z. The history contains one request, one delivery, two activity cancellations, two stop acknowledgements, two unknown-stop retries, two cleanup completions and one WorkflowCancelled.

Raw connected artifact, retained until December 31, 2026. Artifact SHA256 af6606ae9c176f1710521f5255ae8f402319abec540025a9752f572a5763f3c2. JUnit SHA256 956336fd0e54b22fff2706198d941fb2f0481655cb5082d565316c015b4f2a1f. Teardown succeeded.

Diagnostic head e256cbdb951cd9da48dde05ea272324740b74779 passes 21 group checks, Ruff and mypy locally. Exact diagnostic Source run 37007105955 is in progress against the same Server and Native commits. Next action is to identify and repair the refused admission, then qualify the corrected exact tuple. PR remains draft and shared issue 136 remains open. No release or Cloud adoption occurred.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Diagnostic result narrows the group failure to application heartbeat

Diagnostic Source run 37007105955 at Python e256cbdb951cd9da48dde05ea272324740b74779 recorded 44 passed, one failed, one existing CLI-binary skip, zero errors, 503.474 seconds. Both nested group local preparations succeeded. An application heartbeat then raised ServerError with no machine-readable reason. This corrects the earlier suspected admission failure. The exact response status/body was not retained in that diagnostic revision.

Raw canonical histories, receipts and source provenance, retained until December 31, 2026. Artifact SHA256 b70b357f4affe977a1d3481d138885cdc5dd3584f0b8199220588beb4e0afcd8. JUnit SHA256 6cead844b9044c761d15a72f0bcd25ff3397207cd2e80f2676bac6f63e4cafec. Teardown passed.

Inspection also found a separate concrete progress-loss defect. Python sent authored heartbeat data under details. The prepared Server API consumes progress, so the earlier sequential case checked heartbeat count while losing its values. Current head f927f698aa2af0c882c0a701975fb485199153a1 sends progress and requires both connected fixtures to recover their authored phase values from canonical heartbeat history. Failed fixture transport now retains HTTP status and response body.

Local Python 3.12.15 checks pass 104 affected cases, zero skips, 20.64 seconds, plus Ruff and mypy. JUnit SHA256 bb1674e0b7d2925fbe7c0b1ce2d75740c043364104db140e0550bd01fdbea90a. The supervised progress regression passes current source (one pass, 2.70 seconds) and fails with the preceding runner bound read-only (one failure, zero errors, 2.02 seconds). It exercises the real supervised callback's heartbeat and canonical progress. The changed connected fixtures add the Server boundary proof.

Exact Source run 37008870970 is running with this head, Server c1bda7f81323e9939ef433c7ebca66f62365cdfd and Native readonly overlay 850a64050b57d47181abeae86d163962c9047317. This field correction does not yet establish the cause or repair of the nested HTTP error. Next action is to inspect that response if it recurs and qualify the corrected exact tuple. PR stays draft and shared 136 stays open. No release or Cloud adoption occurred.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Repairing prepared group heartbeat persistence and SDK progress

Two concrete defects are corrected in the Source candidates:

  1. Native's prepared heartbeat producer includes the complete authored group path, but ActivityHeartbeatRecorded did not permit its six canonical membership fields. The write throws before the heartbeat can persist. Runtime repair 673343b3194b8bf29a10eca29c493932db028bc1 admits those existing fields, adds a real nested local/local/timer heartbeat case, and retains an immutable heartbeat/cancellation replay fixture. The preceding 850a64050b57d47181abeae86d163962c9047317 validator rejects all six fields. The corrected validator accepts the fixture and still rejects an unknown producer key. PHP syntax checks pass. Fixture SHA256 cd65bf379e38af3da018978ef7d3e5135977d283de52344f12b280a8a203b5d6.
  2. Python sent heartbeat values under the remote route's outer details field, while the prepared route expects progress.details. The earlier field-only correction exposed the missing inner envelope through an explicit invalid_local_activity_heartbeat refusal. Current Python a6b243e84532e56d04d799e7b1e2c387874c5e3e matches the established PHP/Server format. The connected sequential and nested cases now require authored phase values in canonical history. Failed fixture transport retains response status and body.

Intermediate Python run reported 43 passed, two failed, one existing CLI-binary skip, zero errors, 494.250 seconds. Raw results and provenance, retained until December 31, 2026. Artifact SHA256 cce4306a240659764a5e1ba6358f134eabf7d50cfbd41f5c277426caff15a9b1. JUnit SHA256 3d61735c3278fe2e8737d62f32268ea5493d7f60c3b80cd4c32b41fb80a40a88. Teardown passed.

Corrected Python passes 104 affected tests, zero skips, 20.64 seconds, Ruff and mypy locally. JUnit SHA256 d55d7322954c13ba6d416fe4496a3e878a08b17a4994ebc732fa2c7a18be94a0. Native's PR check passed MySQL smoke, unit contracts and executable corpus, with a formatting-only failure. Current Native 9c937051f61f9a99701d2d7d9f89da634a3bbf83 applies exactly the formatter's changes. Runtime and resource files are identical to 673343b3194b8bf29a10eca29c493932db028bc1.

Native full Source qualification and Python connected Source qualification are running. Python binds SDK a6b243e84532e56d04d799e7b1e2c387874c5e3e, Server c1bda7f81323e9939ef433c7ebca66f62365cdfd and Native 673343b3194b8bf29a10eca29c493932db028bc1. The superseded unformatted Native full run was cancelled. Next action is to finish and repair these exact qualifications, then retain the complete group/cancellation proof. Both PRs remain drafts and shared 136 stays open. No release, Cloud adoption or paid provider change occurred.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Prepared groups and heartbeat persistence pass connected Source qualification

Run 37010610371 passes all jobs at Python a6b243e84532e56d04d799e7b1e2c387874c5e3e, Server c1bda7f81323e9939ef433c7ebca66f62365cdfd and Native readonly overlay 673343b3194b8bf29a10eca29c493932db028bc1. Python 3.10–3.12, lint, package, corpus and branch gates pass. Connected results are 45 passed, zero errors or failures, one existing unavailable-CLI skip, 509.950 seconds. The unrelated CLI skip is named in the JUnit artifact.

Both sequential and nested mixed local/timer cases persist the authored first and second heartbeat values through progress.details. Nested heartbeat history retains the complete outer and inner authored group positions. Results preserve binary bytes and cold replay without invoking completed callbacks again. This verifies the Python envelope correction together with Native's admission of the six canonical heartbeat membership fields.

Cancellation cases independently verify physical process exit without application heartbeats, owner SIGKILL during shielded cleanup, replacement-worker recovery, one canonical delivery, immutable duplicate identity/time/deadline and timely Cancelled closure.

Case Original request Original deadline Cancelled Elapsed Physical callback exits
Sequential 2026-10-02T13:16:41.352767Z 2026-10-02T13:17:11.352767Z 2026-10-02T13:17:05.739330Z 24.386563 s 3
Atomic two-member group 2026-10-02T13:17:08.060390Z 2026-10-02T13:17:38.060390Z 2026-10-02T13:17:33.402430Z 25.342040 s 6

The group has one request, one delivery, two activity cancellations, two stop acknowledgments, two unknown-stop recoveries, two cleanup completions and one WorkflowCancelled event. Four cleanup admissions use distinct attempts but one unchanged request/root/delivery/deadline tuple. Unknown stop state remains explicit after owner loss. No application heartbeat or renewed cancellation budget appears in either cancellation history.

Raw histories, receipts, results, source provenance and image digests are retained until December 31, 2026. Artifact SHA256 7b7ae6fed325d2fabbd94d98593f85f877738529dc022784fd30d5c1fedd5f55. JUnit SHA256 f9886503a506ed41b92e56094bd80cf59e3c9c0c17c28486933d765571477562. Hosted test containers and network were removed successfully.

Current Native 16b722e74a10451217b1c16f25b5b1bcb1ebc45d differs from the qualified overlay only in test formatting. Runtime and resource files are byte-identical. Its PR quality, replay, unit and MySQL smoke gates pass. Local Python checks pass 104 affected cases with zero skips, Ruff and mypy. Earlier failed runs and counterexample evidence remain in this PR's comments.

Next action is portable activity cancellation policies and deterministic scopes, followed by the exact published PHP parent → Python child → Rust remote activity plus PHP local activity cascade and one coherent API/CLI/UI inspection view. This PR remains draft and shared issue 136 remains open. Published protocol 1.19 and packages are unchanged. Protocol 1.20 remains an explicit, unfrozen Source opt-in.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Pending Activity cancellation response qualified

Exact Python SDK source bf931959af5cc6615cb42ccd0ed279e7b3d33024, Server 2fc521b5f132c5a6d0909ec6239f2fbbc435bae6, Native readonly overlay bf047ce2e88e6c565eb6b80fae6559216524d41b.

The client validates pending Activity cancellation replies for Activity, LocalActivity, Parallel and SelectionHandle boundaries. It retains child pending validation and requires explicit claim release, matching task identity and null delivery fields. Malformed fields, unknown or non-string reasons, timers and unrelated child/activity kinds are refused before the worker defers delivery.

Exact Source CI passes every job, including Python 3.10–3.12, lint/type checks, package, Avro, corpus and target-branch qualification. The unit matrix covers seven valid reason/kind pairs, malformed fields for both reasons and unrelated-kind refusal. Ordinary exact-head CI also passes. The preceding runtime commit's Source run stopped at two Ruff line-length findings in tests, which this final head corrects. Its skipped integration job is not used as qualification evidence.

Connected qualification uses pytest tests/integration/ -v --capture=tee-sys --junitxml=integration-results.xml. It passes 45 cases, zero errors or failures, with one existing unavailable-CLI skip, in 490.243 seconds. It rechecks real callback stop without application heartbeats, local and nested group history, original cancellation identity/deadline, worker SIGKILL during shielded cleanup, fresh replacement and resumed cleanup on this tuple. Source qualification does not turn the unavailable CLI case into a pass.

Raw histories, process observations, JUnit, image identities and source provenance are retained until 2026-12-31. Artifact SHA256 24bbddc080f6c8ca74a19fb6638b0683de66aa34912cb8e6ea5571ec09a44be5. JUnit SHA256 dd5a52a3d4f1e2b3f0fb4e4bafde236721a58ae91b2f4da68bba89d544405923. Hosted task containers, images and network were removed successfully. No local Python runtime or package cache was created for these edits.

This qualifies the reply validator and rebinds existing connected component cases. Explicit portable Activity policy authoring and the required published mixed-language cascade remain separate gates. Protocol 1.20 remains an unfrozen Source opt-in, the published package is unchanged, this PR remains draft and shared issue 136 stays open. Next: author and admit explicit Activity policies before running their actual worker scenarios.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Remote Activity policies qualified in Python Source

Exact Python de0c476760524dcac3df35572bb7bd19d818a2d0, Server 8940fb8a866e24d7874947e850f7fc864fbdfd3c, Native readonly Source overlay 9f8cb61947997c1c60f294bdf559d02d8258cc18.

ScheduleActivity and WorkflowContext.schedule_activity() accept the typed CancellationPolicy enum and its portable strings. Both command encoders retain the explicit policy. Invalid policy and unbounded Abandon fail before suspension, and encoding revalidates mutable commands. Missing negotiated worker capability is diagnosed before submission. Omission preserves the historical Try default and wire shape. Local policy authoring remains unavailable.

The replayer previously accepted a change from recorded Wait to authored Try. A focused reproduction fails before repair and passes afterward. Replay now keeps the original canonical Activity policy through later events that omit it, checks ordinary/group/selection matching and cancellation delivery, and rejects malformed or conflicting history. The minimal activity-cancellation-policy-changed.json Source fixture executes through the official replayer. Frozen 1.19 fixtures are unchanged.

Complete exact Source CI passes every required job, including Python 3.10–3.12, full lint/type analysis, package and corpus. Connected qualification passes 50 cases, zero errors or failures, one existing unavailable-CLI skip, 562.214 seconds. JUnit contains 51 cases including that skip. Hosted stack teardown passes.

All five new connected policy cases pass without skips. Explicit Try and Wait physically stop both async and sync callbacks without application heartbeats, preserve original stop receipts and refuse stale completion/failure. Wait checks that the receipt precedes canonical workflow delivery. Bounded Abandon proves the callback process remains live after parent Cancelled closure, then commits independent completion under the original total deadline, retains parent cancellation and refuses a second outcome from the completed attempt. The four Try/Wait times are 11.225, 11.180, 11.656 and 11.652 seconds. Abandon takes 27.052 seconds. These are Source policy cases with distinct cleanup and total Activity budgets.

Local Python 3.12 focused replay, child-policy, cancellation and corpus qualification passes 181 cases, zero errors, failures or skips. Full Ruff and mypy pass. Commands: pytest tests/test_activity_cancellation_policies.py tests/test_child_workflow_policies.py tests/test_cooperative_cancellation.py tests/test_replay_regression_corpus.py -q --junitxml=..., ruff check src/ tests/, mypy src/durable_workflow/. The connected command is the existing complete integration suite using the exact server_commit and native_commit workflow inputs.

Raw scenarios, JUnit, source provenance and image authority are retained until 2026-12-31. Artifact SHA256 2b52d8eb89b69cc027136b61a9c174f99b90ec19f3b078157905949d86d20736. JUnit SHA256 ce705951fbc09a4b1cc6542c400fae14a215508bc6e926eb9ff60e4abfac0050.

PHP's matching Source policy cases are qualified. Rust is qualifying its matching consumer. Explicit local policies, independent local Abandon lifetime, scopes, coherent inspection and the published mixed-language cascade remain required. This PR remains draft, shared issue 136 stays open, protocol 1.20 remains explicit/unfrozen and published artifacts are unchanged.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Python deterministic remaining-time Source candidate

Head 8c1e937d2bab234a8a9cd1c1cea3aed7265aa9db adds CancellationContext.remaining() on the consumed replay clock while preserving the existing start-time WorkflowContext.now(). The deadline, root identity, reason, requester and lineage retain their portable metadata schema.

The helper initializes at committed delivery, advances through consumed blocking cleanup outcomes, and preserves synchronous side-effect and inline-local decisions between first execution and cold replay. Selection follows its committed winner, awaited handles use their own durable result or first cancellation receipt, and failed groups exclude later sibling outcomes. Recorded clock skew cannot increase the budget. Fractional seconds are retained and expiry clamps to zero. A detached snapshot, ended replay, or missing/invalid boundary timestamp fails without using host time.

Local affected checks passed 206 cases, including 12 helper cases and the two Git-backed public-boundary scanner cases, zero errors, failures or skips. Full Ruff and mypy across 29 source files passed. Affected JUnit SHA-256: 99536e899690bd8ad7d286aa38391caf3e4e3659f57464c4ade882667182d39b. The earlier broad non-integration run passed 1,993 cases with two existing skips, but its two scanner cases could not run in a container without Git. Both scanner cases passed in the final isolated Git-equipped container. A full current-suite pass is pending hosted CI.

The actual Python SIGKILL scenario now records the delivery budget in the original worker, requires an identical value and metadata in the replacement, and checks the post-cleanup value against committed delivery/completion timestamps and the original deadline. Source run 37065839981 is pending against Server 70807856b416b8792099108712cdd031e2c66c89 and Native 88ea6e46499332427d0f9a439768ae0bf86e8702. Ordinary current-head CI is pending too.

The qualified PHP helper and mixed cascade remains bound to its exact tuple, including preceding Python de0c476760524dcac3df35572bb7bd19d818a2d0. Next: inspect Python CI and actual remaining-time observations, then port Rust and audit embedded Native synchronous replay time. Explicit local policies, nested scopes, competitive qualification and the exact published mixed-language scenario remain required. All candidates remain drafts and shared #136 stays open. Protocol 1.20 remains unfrozen and published/default 1.19 is unchanged.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Python remaining-time qualification now covers both cleanup paths

Current head 0e1c427eae2f6120155a5cf3c9c1ae79ba7a5756 corrects the connected qualification without changing the helper implementation.

Preceding Source run 37065839981 passed all ordinary jobs, including Python 3.10–3.12, lint/types, package and regression corpus. Connected qualification had 49 passes / 1 failure / 1 existing unavailable-CLI skip. The failing assertion expected legacy inline cleanup to consume its later persistence timestamp. Its delivery and post-inline values were correctly equal at 119.462336 seconds, matching the documented first-execution/cold-replay contract. That assertion now requires equality and verifies the original delivery timestamp.

The existing sequential and atomic prepared-cleanup tests now capture the actual original and replacement worker's delivery values and metadata, require equality after SIGKILL, and verify the final value against committed cleanup completion and original deadline. Both already require stopped callbacks without application heartbeats, unknown-stop recovery, duplicate original identity/deadline, and Cancelled convergence before the original 30-second deadline. The new observations add no workflow commands.

Current local Ruff, mypy across 29 source files, all 12 helper cases, collection of the 28 affected connected cases and whitespace checks passed. Helper JUnit SHA-256: 315dd46d775a28ded6dd91f210e1e9cbef812b1932f155244f66d76ca26aa803. The unchanged core helper and broader replay selection passed 206 focused cases at the preceding head.

Current Source run 37067939883 is pending against Server 70807856b416b8792099108712cdd031e2c66c89 and Native ae1513bf78a175f4965e4fb8fc473fab60c35ec1. Current ordinary CI is pending. Preceding failure evidence remains retained until December 31, ZIP SHA-256 7a57250ed61bb39067fe8335a7f225ad702898cbd3085bdf9cf41f8977368775, JUnit 0038bcb79df81cb3a9c78e7739b3c753d2208aacb02e2a311e04bd96766de327.

Local venv, generated caches and the temporary Python image are removed after hosted handoff. Next: inspect actual current prepared/inline observations and Native qualification, then port Rust and finish explicit local policies/scopes/competitive and final published closure proof. All candidates remain drafts, shared #136 stays open, protocol 1.20 remains unfrozen and published/default 1.19 is unchanged.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Python consumed cleanup budget qualified

Exact Python 0e1c427eae2f6120155a5cf3c9c1ae79ba7a5756 passes Source 37067939883 against Server 70807856b416b8792099108712cdd031e2c66c89 and Native ae1513bf78a175f4965e4fb8fc473fab60c35ec1. Ordinary CI 37067887319 also passes. Python 3.10–3.12, lint/types, package, corpus and connected gates all pass.

Connected JUnit records 50 passes, zero errors/failures, one existing unavailable-CLI skip, 561.247 seconds. Both prepared-cleanup SIGKILL cases pass without skips. The actual original and replacement worker observations are byte-equivalent metadata and equal remaining time:

Cleanup Original / replacement remaining After committed cleanup Cancelled after original request
Sequential prepared 28.858494 s / 28.858494 s 4.153142 s 25.952260 s
Atomic prepared group 28.857760 s / 28.857760 s 13.957172 s 16.197347 s

Each uses the original 30-second deadline, physically stopped callbacks without application heartbeats, one canonical delivery, actual owner-process SIGKILL during cleanup, fresh replacement recovery, unknown stop state preserved, and duplicate requests retaining original identity/deadline. The final helper value matches original deadline minus the latest consumed committed cleanup completion within one microsecond. Sequential/group histories contain zero application heartbeat records and respectively one/two callback stop acknowledgments. Both contain one Cancelled terminal before the original deadline.

The legacy inline recovery case correctly retains 119.502138 seconds at original delivery, replacement delivery and post-inline callback. Inline persistence is synchronous, so it does not change first-execution/cold-replay cleanup decisions. The earlier incorrectly written assertion remains recorded in its preceding failed run. The current core helper is unchanged by the qualification correction.

Raw artifact 11253209659 retains JUnit, actual observations, histories, callback process IDs, recovery receipts, exact source provenance, images and package provenance until December 31. SHA-256:

artifact-ZIP 57b27d263e0134ca3ebfd36319c6f7821faabe42f23498d3267a4bb526a1f78a
JUnit 452bdb7a37e9402d4a3d0028c490640b141bd47a261025b4a6c6e6873746de67
scenarios 9ef92456fd645f0992ca13c160827eaca2d627cc6d6be8a1ee8e92ad9d42481e
source-provenance a94ea560d37494dd08fa96a984e8e97dbfc8c6932e18a45280061e6bc9b5b658
image-provenance 7130ad258d735c3670c825ab09cc2d2dc81c071608857746dd459ab726bfa08b
package-provenance f158d21b57295c937223ee49838088cb006cda3a1f1beaa58f3caeedef164260

Native's exact-head full qualification is retained separately. Rust's consumed-budget helper is now under local qualification. Explicit local policies, nested scopes, fair competitive qualification and the complete exact published mixed-language cascade remain required. All cancellation candidates remain drafts, shared #136 remains open, protocol 1.20 is unfrozen and published/default 1.19 is unchanged.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Next: explicit prepared local Activity cancellation policies in Python

PHP's current source qualification now exercises the Native and Server prepared
local Activity policy contract. Python must consume the same discovery and
preserve authored semantics on cold replay.

Add optional TryCancel and WaitCancellationCompleted policies to local Activity
authoring and prepared descriptors. Preserve historical omission as TryCancel.
Refuse local Abandon, legacy inline execution with an explicit policy, and
unsupported policies before callback execution or checkpoint/group mutation.
Register the policy consumer only when discovery advertises installed support.
Compare the recorded policy on replay and reject changed authoring.

Test fresh admission, completed and unresolved cold replay, capability refusal,
whole-group refusal before side effects, and real supervised callback stop
without application heartbeats. Extend existing connected cleanup/reclaim cases
to cover explicit policies and the original cancellation identity and deadline.
Run ordinary CI and source qualification against exact Native/Server commits.

This remains part of draft #90 and unfrozen protocol 1.20. Published/default
protocol 1.19 is unchanged. Shared #136 remains open until the published
mixed-language cascade, inspection and competitive acceptance criteria pass.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Python prepared local policies implemented, qualification running

Candidate afa7cfaabd0d7fad8049f7b6e3b4c19ce15d0aa2 adds explicit prepared
local Activity TryCancel and WaitCancellationCompleted authoring, descriptors,
installed-policy discovery and consumer registration. Historical omission
retains Try. Local Abandon, legacy inline explicit policies and unavailable
policies refuse before callback execution or checkpoint submission, including
nested groups. Replay checks the original policy before ordinary recovery,
completed results and cancellation delivery. Queries, updates and validators
carry the same negotiated replay consumer.

Local Python 3.12.15, Git 2.47.3, Ruff and mypy pass. The full non-integration
suite passes 2,040 cases, two existing skips, zero failures or errors,
130.69 seconds. Dedicated policy coverage passes 38 cases with zero skips,
including completed/unresolved replay, canonical nested snapshots, changed
policies, malformed authoring, real Worker refusal before a prefix checkpoint,
registration and query/update replay. Existing process tests now exercise both
explicit policies and omission for actual callback completion and joined stop
without application heartbeats.

Full JUnit SHA-256 fab67eed5448479f2d67933688972821ed4e4b09a4c48cb75f8696caba33281f.
Policy JUnit SHA-256 b088b3032b2195169a1eb1a4e604eee4825244132fb869090d25281fd81d7ea0.
The preceding broad attempt had two fixture setup errors because the slim
container lacked Git. The corrected complete run includes those checks.

Exact source qualification
is running against Server dd8996fdd6d488b48d575b1b934821bd8bedc8ed and Native
ba1aa4770c4b94533548bd45bd3ba028229c54e8. Its existing sequential and atomic
SIGKILL/recovery scenarios now cover omission, Try and Wait. They must preserve
the same request, delivery, root and original 30-second deadline, inspect
canonical policies, physically stop callbacks without application heartbeats
and check Wait acknowledgment ordering before delivery. No connected pass is
claimed before that run completes.

Protocol 1.20 remains opt-in and unfrozen, published/default 1.19 unchanged.
This PR stays draft and shared #136 open. The published mixed-language cascade,
full inspection and competitive/model gates still remain required.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Historical-omission assertion corrected, full Source retry running

Source run 37084482121
at afa7cfaabd0d7fad8049f7b6e3b4c19ce15d0aa2 finishes with 52 passed,
one existing unavailable-CLI skip and two failed assertions
, 644.00 seconds.
The whole run is not a pass.

Both historical-omission cleanup/recovery variants reach an assertion that
indexes activity.cancellation_policy. Omitted policy is a valid historical
Try default and is absent from those canonical snapshots. The new assertion
incorrectly required an explicit field. It raised KeyError. The four explicit
Try/Wait sequential/atomic variants pass, including actual callback stop without
application heartbeats, owner SIGKILL, canonical replacement delivery and the
original immutable 30-second budget. The failed historical variants are not
counted as qualified cases.

Current head d41e76deb76d3a04c25c5879984b85bb9c23cda3 changes only that
qualification assertion to read the valid historical Try default and formats
it within the existing style limit. Runtime and package files are unchanged
from afa7cfa. This is a runner assertion repair, not a product behavior fix.
No replay/codec defect or regression corpus entry is claimed.

Full exact-head Source retry 37085616916
is running against the same Server/Native tuple. Cancellation of the redundant
preceding-head retry is requested before it starts the connected stack. Shared #136 remains
open, the PR draft and protocol 1.20 unfrozen. The coherent current mixed tuple
will run after full Python qualification passes.

Failed-run artifact 11259778858
is retained through January 1, 2027, including JUnit, complete scenarios, exact
provenance and image authority. ZIP SHA-256
0d97120172ee79ff7a88506b47739c5588c88f51011e554cba42c4c83da85746.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Current Python Source tuple qualified

Source run 37085616916
passes every required job at d41e76deb76d3a04c25c5879984b85bb9c23cda3, against
Server dd8996fdd6d488b48d575b1b934821bd8bedc8ed and Native
ba1aa4770c4b94533548bd45bd3ba028229c54e8. Ordinary exact-head CI
37085610278,
docs and boundary checks also pass.

Connected integration finishes with 54 passed, zero errors/failures and one
existing unavailable-CLI skip
, 643.359 seconds. The missing CLI scenario is
not counted as qualified by this run. The current PHP mixed qualification
supplies the exact CLI consumer separately.

All six prepared-local cancellation cases pass: historical omission, explicit
TryCancel and explicit WaitCancellationCompleted, each with sequential and
atomic-group admission. Each case stops the actual callback without application
heartbeats, SIGKILLs its workflow owner during cleanup, starts a fresh replacement,
replays the original delivery, rejects stale publication, retains duplicate
identity/deadline and records WorkflowCancelled with completed cleanup before
the original 30-second deadline. Wait additionally checks the matching stop
receipt before delivery. These are real connected tests, not virtual-clock
or lease-row substitutions.

Policy and admission Original/replacement remaining seconds Remaining after cleanup result
Historical omission, sequential 28.881331 / 28.881331 4.615620
Historical omission, atomic group 28.831439 / 28.831439 13.207976
TryCancel, sequential 28.866122 / 28.866122 15.994677
TryCancel, atomic group 28.804223 / 28.804223 13.248940
WaitCancellationCompleted, sequential 28.859150 / 28.859150 14.958525
WaitCancellationCompleted, atomic group 28.803752 / 28.803752 14.403743

These values are the workflow's deterministic remaining-time observations.
Canonical terminal timestamps and cleanup outcomes are retained in the raw
histories. This recovery profile uses a supported 10-second workflow lease and
unthrottled 10-second repair. It does not qualify the default 60-second lease
for a 30-second SIGKILL recovery target. The separately passing remote-owner
default-lease recovery case takes 308.843 seconds and is not a 30-second claim.

Raw artifact 11260048832
is retained through January 1, 2027. It contains JUnit, complete scenarios and
histories, source/package provenance and image authority.

  • ZIP SHA-256: 9e23f5c9e9556bca6998f58e2ee72231e7677049b2320aa4301c0b11ddd82894
  • JUnit: 10ec41bd2f8b6a3431fe4ecac817cd59e82dd7c8adb9ff2325490938871944bb
  • Scenarios: 047c95b48fe7581ed7ca94b39c0697c2316918c8b8488ee553e18dffaaf9b7a4
  • Source provenance: 642e6eeddac62fb07a838f3444ecf840fa18cd36f76ccbfaa37fe416489dc752

The preceding failed Source run remains classified as two historical-omission
runner assertion failures. Its raw artifact is retained. The full corrected
Source pass above qualifies this current tuple. Redundant preceding-head run
37085585468 is confirmed completed/cancelled without creating an integration
stack.

Current PHP mixed Source run 37086658222
is running with PHP 488550a1bd5857a91693515a8fd604145347076a, this Python
head, qualified Rust 14a1e9b329dfa1358b98e2aa6411183dc37f26d7, the same
Server/Native and CLI 933c554f8bffa2bf7d188ced712b69b20844ac2d.

Shared #136 stays open, the PR stays draft and protocol 1.20 stays unfrozen.
These source passes do not replace the required published-artifact cascade,
remaining model decisions or fair competitive evidence.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants