feat: add cooperative cancellation requests and claim-bound delivery

[rmcdaniel]

Customer outcome

Add service workflow cooperative cancellation under shared issue 136, with Native PR 603 and the first-party SDK drafts. Existing terminal operations preserve their recorded behavior.

Implementation

• Serialize request admission and claiming on the run lock. Bind actual claims to immutable registration capability and protocol snapshots. Refuse incompatible active claims before accepting a request.
• Preserve original identity and deadline through duplicates, cached polls, claims and heartbeats. Deliver through the engine's owner/attempt-fenced authored-call marker.
• Issue a run-bound token for bounded canonical history refresh after uncertain acknowledgments.
• Observe actual remote ownership without renewing leases, registrations, sessions or application progress. Validate namespace, task, attempt, owner, execution deadlines and session lifetime.
• Return HTTP 200 for a parked child wait with delivered: false and claim_released: true. Canonical child terminal history wakes the successor task.
• Preserve and validate cancellation_policy through HTTP completion and Native normalization. Refuse cooperative child/parent-close policies when the server/request protocol, immutable claim capability or installed runtime cannot honor them. Recheck admission under the actual completion lock. Diagnostics name the unavailable component and worker.
• Bind activity claims to the original worker capability and request protocol atomically with leasing. A later registration or request cannot upgrade an old claim.
• Add an original-owner remote callback-stop acknowledgment endpoint. Check the namespace, task, attempt, immutable claim and canonical cancellation snapshot under the activity lock order. Duplicates return the original receipt. No lease, heartbeat, publication authority or deadline is renewed.
• Expose the canonical request/root identity, original deadline and stop receipt through read-only activity status. Fenced publication with no worker report remains callback_state: unknown. Late receipt remains late. Local callbacks require their separate workflow claim authority.
• Advertise acknowledgment support only when the candidate protocol and installed Native primitive are available. Permit receipts during storage draining and refuse writes when storage is fenced.
• Keep legacy terminal request_cancel on its older protocol path. Candidate worker OpenAPI is version 27, with protocol defaults unchanged.

Current verification

Current Server source: dae9bb04d1828b0d45fa8d65ebe0a94d2d555937, based on Server 2.4.38 / chart 0.1.134 and pinned published Native 2.3.3.

Local source binding against Native 94aabbc3b57fe331f1305ab4539b1818a81f3f2c
passes 134 tests / 1,670 assertions, no errors, failures or skips. It includes
receipt identity, duplicate and late behavior, read-only observation, stale result
refusal, immutable registration and protocol claims, namespace/owner/attempt/request
fences, storage draining/fencing, legacy protocol and operator-role refusal.
The receipt regression now checks run inspection and diagnostics remain HTTP
200, and stale completion/failure remain HTTP 409 without adding history.
Native corrects the exhaustive timeline summary that failed in the preceding
connected runs. Current feature CI,
replay/query HTTP
and growth/polling smoke
all passed. Corrected connected source qualification completed in
PHP and
Rust.
PHP passed 18 / 815 plus memo restart 1 / 67. Rust passed 13 of 14 cases.
Its remaining fixture now waits for the valid asynchronous late stop receipt
before comparing full history after stale publication. The
corrected Rust run
passes all 14 cases in 348.24 seconds. Both connected jobs removed their
test stacks successfully. The normal dependency is still published Native 2.3.3.

Preceding affected polling/fence/retrier tests pass 22 / 332. Pint checked all eight
affected PHP files and formatted five blank lines, followed by another passing
134 / 1,663 run. The normal dependency remains published Native 2.3.3 and returns
explicit installed-runtime refusal for the candidate receipt primitive.

Preceding exact-head 546b1ca456b7d31bee1e8cd4a563da61fc584ce8 feature and repository gates
pass 2,326 tests / 48,335 assertions, with six existing PHPUnit deprecations and
six receipt cases requiring the candidate Native backend. Those six cases all run
in the passing source binding above. OpenAPI evolution 23 → 27 passes.
Concurrent replay/query HTTP
also passes. Bounded growth and polling smoke
also passes, including both the polling smoke and bounded-growth contract.

Exact-head affected feature CI and concurrent replay/query HTTP checks passed. Local source-binding qualification against Native 2b308d33389fdb0cf9d0a9d868b90cd5c5d5f35d passed 45 tests / 868 assertions, including the actual HTTP policy-to-history path, immutable old claim refusal, protocol floor and legacy policy. Pint and OpenAPI evolution 23 → 26 passed. Native's subsequent a3147d13c2ca5df8485056da2e47806c61ca4f7a changes two test comparisons only. The published Native 2.3.3 dependency has no child-policy implementation, so this source intentionally refuses that feature until qualified backend adoption. Connected published-image qualification remains required.

Connected single-worker child waiting passes at PHP 509a9daf04aa7254179fa4041f78414f8b00a93a, Server eafa77721b6840234e0de11e18a4a2aa10deb299 and Native a3147d13c2ca5df8485056da2e47806c61ca4f7a: MySQL/Redis 18 cancellation cases / 700 assertions and memo restart 1 / 67. A live worker and an actual SIGKILL/fresh-registration replacement both complete child cleanup before parent delivery within the original 30 second budget. Raw source provenance, histories, HTTP delivery observations and JUnit are retained by that run. It predates the new callback-stop receipt.

Preceding connected gates against Server 7675ab976c7c24b7aadfc940fd8930ef2e907a7c / Native 2.3.3 passed PHP 16 cases / 552 assertions, Python 19 cancellation cases and Rust 14 actual-worker cases.

Required before publication

The exact-head bounded-growth and polling smoke also passed. The affected feature run completed 2,310 tests / 48,221 assertions with six existing PHPUnit deprecations and no failures or errors.

Emit the remote receipt from SDKs only after actual callback stop/join, qualify it through connected workers, add local receipts and activity operation policies, and build one cascade inspection view. Continue mixed-language policy/context qualification and the complete PHP parent / Python child / Rust remote activity / PHP local activity published scenario, including cleanup SIGKILL and recovery within the original 30-second budget. Freeze and publish the shared specification, then verify exact artifact digests.

This stays a draft. Default protocol is 1.19. 1.20 is an explicit, unfrozen source opt-in. No cooperative RC or Cloud adoption is claimed.

[rmcdaniel]

Connected Python/Server source qualification now passes eleven SQLite cases at Python 119747bb2fb838f6dff64c9938c780414a074b99 and Server 2e2d6b31df981e98c1a054a8953e9efe95d17e6e. Scenario evidence covers actual SIGKILL/cold process replacement, shutdown grace and expiry, original request/deadline and one canonical marker, waiting timer, local async/synchronous execution, remote heartbeat/result/failure fencing, deadline, termination and a discarded successful delivery acknowledgment.

The current default remains protocol 1.19. Explicit source candidate protocol 1.20 now discovers the capability through the same policy used for request admission. Python's normal suite passes 1,618 local tests, Ruff and strict mypy. Server's affected filter passes 145 tests and 5,165 assertions, with Pint passing.

Normal Python CI, Server CI and an explicit exact-pair MySQL qualification are running. The candidate run's three supported Python, package, corpus and lint jobs pass. It retains connected JUnit evidence and removes its stack.

Next: finish those source gates, then PHP/Rust equivalents, coordinated protocol/specification activation and the exact published service tuple. These source drafts do not authorize a published cooperative capability claim. Completed local task resources are being removed.

[rmcdaniel]

Connected Python qualification complete

At Python 119747bb2fb838f6dff64c9938c780414a074b99 and Server 2e2d6b31df981e98c1a054a8953e9efe95d17e6e, normal Python CI, docs and boundaries pass. The explicit candidate MySQL run also passes every job. Its integration finished at 02:29:45 UTC on October 1, with 33 passed and one existing CLI-binary skip in 80.19 seconds. All eleven cooperative cases are present and passed in the downloaded JUnit artifact, retained through October 8. The log verifies the exact Server SHA and teardown passed.

Full scenario evidence covers original identity/deadline/one marker, a discarded successful delivery response, real SIGKILL and cold process replacement, local async/synchronous and remote fencing, shutdown grace/expiry, waiting timer, deadline and termination. Local SQLite passes all eleven cases. All local task containers, worktrees, dependencies, proofs and images are removed. Downloaded CI reports and transport files are removed after this handoff.

Server's source/corpus CI passes 2,279 tests and 47,800 assertions. MySQL replay/query topology passes. Polling bounded-growth smoke remains running. Both PRs remain drafts and default protocol remains 1.19.

Next: PHP/Rust request, delivery, canonical replay and shielded cleanup parity, remaining backend concurrency, coordinated specification/capability activation and the exact published Server/SDK tuple. No published cooperative capability is authorized by these source-only results.

[rmcdaniel]

Remote activity observation qualified at 073a516bbd4063f57d4ad65ad732ec423131c8ff

The additive candidate1.20 POST /worker/activity-tasks/{taskId}/status is implemented and all normal Server gates pass at this exact head. Full feature/corpus and source qualification runs 2,300 tests / 47,999 assertions, with six existing PHPUnit deprecations. MySQL replay/query HTTP topology, polling bounded-growth smoke/performance qualification and both public boundaries pass. Local PHP8.4.26 passes 184 tests / 7,982 assertions, Pint on all four changed PHP files, worker OpenAPI evolution23→25 and git diff --check.

The21 new status cases cover exact namespace/task/attempt/owner fencing, stale attempts, canonical activity cancellation, terminal cancel/terminate, lease expiry and authored heartbeat/execution deadlines before timeout repair. They prove preservation of attempt, execution, task, worker registration, required session and history. Required sessions must remain active, owned and inside their lease/TTL. Missing/replaced/closed/expired sessions refuse continuation without repair. Backend pressure returns retryable503 with the attempted fence and no continuation grant. Observation remains available under draining/fenced storage admission.

This uses the existing published Workflow2.3.0 attempt observation primitive. It never renews an attempt lease, user heartbeat, registration or required session. The existing five-minute activity lease and authored user heartbeat renewal behavior remain. A pending cooperative request is not delivered cancellation. Publication must independently validate the attempt fence, and a prior observation is not a reservation.

The worker OpenAPI candidate advances to25. Default released protocol1.19 and terminal endpoint behavior remain unchanged. PHP PR91 is wiring the actual owning Worker to bounded observations and the existing process supervisor, with user heartbeats proxied separately. Actual blocked remote callbacks, shutdown, owner death and cold workflow replacement are the next connected qualification. This Server result does not by itself establish that SDK lifetime behavior. The draft remains gated by per-language parity and exact published-tuple conformance before capability activation or release.

[rmcdaniel]

The cooperative Server candidate now consumes published Workflow 2.3.1 at fb3f3e59a4342fdebf8ced6160798906c3ee4387. Exact Server head is c32434cc784a1bf870dcf19568bef944a320f6a7. It merges the reviewed Server #292 dependency/release-metadata change into the previous 073a516bbd4063f57d4ad65ad732ec423131c8ff candidate. The candidate's service implementation is unchanged and protocol defaults remain 1.19. Normal source checks have restarted for this exact tuple.

Workflow's complete source matrix and all 16 published Laravel/PHP upgrade combinations pass. Server #292 is separately qualifying stable image 2.4.35. A focused ordinary-protocol signal drill reproduces the expired-workflow/older-active-activity defect on published Server 2.4.34 and PHP SDK 2.1.6, so this repair also affects existing service callers.

The entire Python candidate CI is dispatched at unchanged Python 88f0e31bf1736271deaabcc67d74df9ce98df491, with server_commit=c32434cc784a1bf870dcf19568bef944a320f6a7 and cooperative_qualification=true. It must pass the actual remote-worker supervision and SIGKILL cases using the published native correction. This is source-candidate qualification, separate from Server #292's published-image PHP/Python/Rust follow-through and shared #136's activation/reclaim/Rust gates.

[rmcdaniel]

Heartbeat ownership interleaving reproduced and fixed at source

The ordinary protocol 1.19 regression fails against unchanged main d0692b171a0cabe6eefdc8bfc0f82d249e16128d with locked published Native Workflow 2.3.1. Its two HTTP kernels perform actual Native status, claim and renewal. The fixture only provides an IPC barrier and clocks crossing the one-second lease expiry. It does not edit lease rows or fabricate Native results.

Observed baseline:

{"heartbeat":{"workflow_task_attempt":1,"lease_owner":"original","renewed":true},"replacement_claim":{"workflow_task_attempt":2,"lease_owner":"replacement"}}

The same Native task ID was present in both responses. The positive heartbeat therefore identified a claim that no longer owned the task.

The fix in #294 head 226025ffe3e0c1af98a77c01932a50f41799d8d8 holds the namespace-scoped task lock across ownership validation and Native renewal. Local Pint and 79 focused protocol, ownership/error, success and poll-pressure cases pass with 4,286 assertions. The controlled race now retains the original durable owner/attempt and the replacement poll returns no task. Command: php vendor/bin/phpunit --filter 'WorkflowTaskHeartbeatRaceTest|WorkerProtocolOwnershipErrorContractTest|WorkerProtocolSuccessContractTest|SqliteWorkerPollLockPressureTest|WorkerPollBackpressureTest'.

Cooperative cancellation draft #291 carries the same locked renewal, plus the pending observation read inside that transaction, at 6be6cd39 (full exact revision available in its CI). Its 24 focused cancellation protocol and concurrency cases pass locally with 545 assertions, preserving the original request/deadline and one canonical request without inventing delivery.

Normal CI is running for both heads. This source result does not claim a published image or Cloud deployment. Next finish exact-head CI/review, publish Server 2.4.36/Helm 0.1.132 independently of cooperative cancellation and verify the affected published worker cells. Rust coordinator qualification then uses the requalified #291 head. Default protocol 1.19 and ordinary capabilities remain unchanged.

[rmcdaniel]

Published Server 2.4.35 reproduces the ownership defect

The same committed regression ran inside the unchanged published Server image durableworkflow/server@sha256:49560f7f861271931125348a9d01638cd722e13ee976b5b732ef122548f15dab with its own PHP 8.3.35, Laravel and Native runtime code. PHPUnit 11.5.55 and Mockery 1.6.12 were mounted as separate test tools, together with the repository's tests. No application or vendor file was replaced. Reflection receipts verify that the controller and Native bridge load from the artifact, and both artifact provenance records name Native 2.3.1 at fb3f3e59a4342fdebf8ced6160798906c3ee4387.

The two actual HTTP kernels reproduce:

{"heartbeat_status":200,"heartbeat":{"workflow_task_attempt":1,"lease_owner":"original","renewed":true,"reason":null},"replacement_claim":{"workflow_task_attempt":2,"lease_owner":"replacement"}}

Both responses identify the same actual Native task. The unchanged published image therefore fails the ownership assertion, with 16 assertions reached. SQLite clocks and an IPC barrier control the interleaving; the harness does not edit any lease, owner or attempt row.

The original full CI result exposed an overly strict test assertion, not a valid positive acknowledgment. PHP 8.3 uses deferred SQLite transactions even when transaction_mode is configured, allowing replacement to win. The fixed transaction retries and returns a correct owner-mismatch refusal. The regression now requires either a positive acknowledgment matching the durable original claim or that exact refusal together with proof of the replacement claim. The same change preserves the cancellation request and cleanup deadline in #291. This still fails the old published image.

Next complete revised-head CI and review, publish Server 2.4.36, repeat this exact-image regression and verify the published PHP/Python/Rust lifecycle tuple before closing #293.

[rmcdaniel]

The ordinary-protocol heartbeat ownership fix has shipped independently as Server 2.4.36, and #293 now has exact published before/after regression and published PHP/Python/Rust lifecycle evidence.

This draft incorporates merged main at 898c51375ddcb2c25588eb63e41fed26a3fd61b2. Current draft head is 24b54c9f8b3f84cad93ec58d5d7cd2f95160458d. Ownership validation, renewal and the candidate cancellation delivery read share the locked transaction. Default protocol 1.19 and the explicit candidate protocol 1.20 boundary remain unchanged.

Local formatting and the focused ownership/cancellation cases pass 25 cases and 565 assertions. Exact-head feature CI passes 2,302 cases and 48,041 assertions. Boundary and replay/query checks pass. Polling smoke is still running at this handoff, so no completed all-checks claim is made yet.

Next product action remains Rust #55's real claim/heartbeat cancellation carrier, canonical delivery refresh and replay coordinator, including callback ownership, replacement and physical lifetime checks. Candidate Server/Native and per-language connected gates must use the updated exact source tuple before the final published tuple is released. This stable heartbeat patch does not qualify or activate cooperative cancellation.

[rmcdaniel]

The updated Server cancellation draft head 24b54c9f8b3f84cad93ec58d5d7cd2f95160458d now passes every normal exact-head check. Feature CI passes 2,302 cases and 48,041 assertions, and the formerly pending polling smoke passes. This source includes the independently released Server 2.4.36 ownership fence.

Rust #55 has advanced to 936303ddcea57789daf76889f5944d243758d7ce, with actual immutable claim/observation capture and bounded fenced history loading. All normal Rust exact-head CI passes, with 303 library cases and 24 integration/consumer/corpus cases.

Next action remains connecting that carrier to the Rust delivery/canonical-refresh/replay coordinator, preserving earlier command prefixes and proving committed delivery before application cancellation. The connected language and physical callback lifetime/ownership/replacement gates remain required before cooperative release or activation. Published Native remains Workflow 2.3.1.

[rmcdaniel]

Pending cancellation remains runnable after a command prefix

The real Rust Worker qualification found a missing Server successor. At Server 24b54c9f8b3f84cad93ec58d5d7cd2f95160458d, a worker correctly committed a side effect before its cancellation boundary. The run then stayed waiting, with the original request undelivered and its only workflow task completed. No successor existed. The other four connected scenarios passed.

Fixed in 555cf7ab4ed4e9a79b52475be76a6dfcf886dd70. Successful nonterminal completion by a cancellation-capable actual claim now ensures one workflow task remains available for an undelivered request. The check and creation share the existing fenced completion transaction and run lock. Existing ready/leased tasks are reused, delivered requests do not create successors, and terminal runs or expired cleanup authority cannot resume. The original request and deadline remain unchanged. Workflow remains the published 2.3.1 package at fb3f3e59a4342fdebf8ced6160798906c3ee4387.

Two regression cases cover request before and after claim, two prefix side effects, repeated completion rejection, a different successor owner, delivery at sequence 3, and no leftover runnable task after terminal cancellation.

Raw counterfactual on the preceding Server head:

test_prefix_completion_keeps_pending_cancellation_deliverable [before claim]
Failed asserting that actual size 0 matches expected size 1.
test_prefix_completion_keeps_pending_cancellation_deliverable [after claim]
Failed asserting that actual size 0 matches expected size 1.
Tests: 2, Assertions: 16, Failures: 2.

Focused protocol suite after the fix:

PHP 8.3.35 / PHPUnit 11.5.55
OK (23 tests, 496 assertions)

Local feature/Nexus/corpus/OpenAPI run:

Time: 03:24.539, Memory: 56.00 MB
Tests: 2304, Assertions: 47873, PHPUnit Deprecations: 6, Skipped: 9.

The local run omits nine external-service cases. Full Server CI supplies those services and is running. Exact-candidate connected qualification is also running for Rust, PHP, and Python.

Next: inspect those results, address any remaining actual Worker/recovery failures, then qualify the complete published tuple before activating the capability. This PR remains a draft. Default Worker protocol remains 1.19.

[rmcdaniel]

Current source qualification

Server 3c15bfb0f98e038febb657412f1ca97bbdb23ad8 retains the qualified successor-task fix for a pending cancellation after a prefix completion. Ordinary claims without the cooperative capability skip the extra task refresh.

All current normal gates passed:

• Feature suite: 2,304 tests, 48,099 assertions, six deprecations.
• MySQL concurrent replay and query.
• Polling growth and performance.
• Public boundary.

Connected source checks use this exact Server commit and published Workflow 2.3.1:

• Rust: seven actual Server cases, including managed callbacks with and without authored heartbeats.
• Python: 40 passed, one skipped, including all 18 cooperative cases. Raw artifact.
• PHP: 13 cases and 447 assertions with zero warnings, including SIGKILL while an actual workflow claim remains leased, expiry-bound recovery and rejection of late completion. Both diagnostic reads use the selected run, and warnings now fail the connected gate.

Shared recovery and replay gates and complete published-tuple qualification remain open. This remains a draft. Published artifacts and ordinary Worker defaults remain unchanged.

[rmcdaniel]

Candidate 7675ab9 now pins the freshly published Workflow 2.3.3 at 70d4fe48efd7dd796c35c1078b3d5ac43f738f4f. Only that package changed in the lockfile. The Native repair fix passed its full merged-source matrix, published Laravel upgrade gates and a separate fresh Composer install (15 repair tests, 117 assertions). PHP/Python/Rust connected source qualification has been dispatched against this exact Server candidate. This checks the existing candidate behavior. The required published mixed-language 30 second cascade and stronger cancellation model remain open in shared issue 136 and Native PR 603. No cooperative protocol publication or Cloud deployment is claimed.

[rmcdaniel]

Exact candidate 7675ab9 with published Workflow 2.3.3: Rust connected source qualification passed all 14 cases (run 36920369742), and Python connected integration passed with its existing CLI-availability skip (run 36920366894). PHP run 36920364544 passed durable deadline closure but failed worker shutdown in one cleanup-expiry case (15/16, 555 assertions). The original callback and relay stopped and the workflow closed 0.852 seconds later. PHP then treated expected claim expiry as an execution failure. PHP PR 91 is correcting this authority-loss handling before qualification is repeated. No claim of a qualified published mixed-language cascade is made.

[rmcdaniel]

Parent claims park while child cleanup finishes

Holding a parent's workflow claim while awaiting child cancellation could occupy
the only PHP workflow worker and prevent it from running the child. The source
candidate now parks the parent in Server, completes its current task claim, and
returns claim_released: true. Canonical child terminal history wakes the parent
into a new task. That task replays the same authored boundary before cancellation
delivery and parent cleanup. The original root identity and deadline are retained.

Local source checks:

• Native authored-child and portable bridge: 45 tests / 391 assertions, no
  failures, errors or skips. Full PHPStan passed.
• PHP transport, worker and replay: 101 tests / 408 assertions, no failures,
  errors or skips. PHPStan passed. The transport fixture checks that one worker
  returns to polling, handles another workflow task, and replays the parent on
  a new claim. Connected single-worker qualification remains required.
• Server response/schema: 24 tests / 576 assertions, no failures or errors.
  Pint passed. Worker OpenAPI evolution
  passed from main's version 23 to candidate version 25.

PHP source is ac434c444a4330155a882c00eac0ff7bb7ed02e2. Server source is
4d9a4248d740579b99668ed97e94c893959fba0a. Both are pushed for exact-head CI.
Python and Rust still need this deferred-claim contract. Portable policy/context
exposure, genuine cooperative parent-close policy, activity policies, cascade
inspection and the complete mixed-language published 30-second SIGKILL scenario
remain required. Defaults stay protocol 1.19. Protocol 1.20 remains an unfrozen,
explicit source opt-in. No cooperative release or closure is claimed.