Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,42 @@
# Changelog

## 0.17.8 - 2026-10-12

- Back off repeated interrupted-message recovery using `retryDelayMilliseconds`.
The first claimed attempt recovers immediately; after attempts 2, 3, 4,
and onward, the delay callback receives 1, 2, 3, and onward (defaults:
1, 2, 4 seconds, capped at 60 seconds). Total persisted claim attempts,
including earlier handler failures, determine the delay. The ready timestamp
survives worker replacement and preserves mailbox order while other actors
can proceed. Lease loss alone still does not dead-letter a message; handler
failure attempt limits, setting names, defaults, and schema stay unchanged.
- Yield to timers and input between busy passes in actor, effect, reminder,
and broadcast workers and drain helpers, including repeated lease loss.
Timer-driven aborts can stop the runtime between attempts; an instrumentation
abort stops before another pass. Synchronous handlers remain non-preemptible.
- Report `activation.lost` when a handler failure or rejection cannot commit
because its lease expired. These attempts recover with the same fencing and
backoff as lost completions, and instrumentation can stop the worker.

- Give overlapping synchronous caller passes separate, reusable workers so
cleanup cannot discard another turn's staged effects or other intents.
Calls to different actors still execute concurrently; workers are reused
once their pass and activation cleanup finish.
A caller interrupted by test reset cannot return its stopped worker to
the pool and stall subsequent calls.
- Re-establish actor, effect, reminder, and broadcast process registrations
before each pass, including after stale-process cleanup and retention.
Explicitly stopped workers stay stopped. No schema or setting changes.
- Cover caller concurrency and process retention on SQLite, PostgreSQL, and
MySQL, with `maxAttempts: 1` and both default and single-message pass limits.
Keep adapter and worker-role selection explicit in the regression fixtures.
- Drain recovered turns in the failure-recovery demo before checking committed
state. Verify completion and allow additional fenced retries when a worker
loses its lease on a delayed runner.
- Advance the database test clock after acquiring commit locks in PostgreSQL
and MySQL fencing tests, so runner delays cannot expire the lease before
the commit barrier is reached.

## 0.17.7 - 2026-10-12

- Renew retained activations between short turns and across worker passes,
Expand Down
15 changes: 15 additions & 0 deletions compatibility/telemetry-events.json
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,21 @@
"sequence"
]
},
{
"name": "activation.lost",
"attributes": [
"actorId",
"actorType",
"attempt",
"deliveryMode",
"durationMilliseconds",
"instanceId",
"messageId",
"operation",
"requestId",
"sequence"
]
},
{
"name": "activation.started",
"attributes": ["actorId", "actorType", "generation", "instanceId", "ownerId"],
Expand Down
3 changes: 3 additions & 0 deletions docs/agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,9 @@ Obey these rules in actor code:
- Do not call an external API in an operation. Use `this.emit()` and an effect
handler.
- Write each operation so that it can run again. Delivery is at least once.
- Repeated lease losses back off but do not become dead letters. Observe
`activation.lost` and set an application stopping policy if a hard bound is
required; handler-failure attempt limits do not bound lease loss.
- A reminder changes state only when it runs, and it runs only while a process
calls `runtime.run(signal)`. Do not compute expiry from the clock in a query;
read the state that the reminder committed.
Expand Down
26 changes: 24 additions & 2 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,11 @@ through `runtime.ref(ActorClass, actorId)`. Both validate options immediately.
| `maxStateBytes` | `5_242_880` | Positive persisted actor-state byte limit. |
| `warnStateBytes` | `131_072` | Positive persisted actor-state byte threshold that reports one warning. |
| `maxResultBytes` | `1_048_576` | Positive operation-result byte limit. |
| `maxAttempts` | `5` | Positive maximum operation, effect, and broadcast attempts. |
| `maxAttempts` | `5` | Positive attempt limit evaluated on handler failure. |
| `maxMessagesPerActivationPass` | `50` | Positive integer turn budget before fairness yield. |
| `maxActivationDurationMilliseconds` | `5_000` | Positive elapsed-time budget before fairness yield. |
| `claimScanLimit` | `100` | Positive integer ready-actor candidates considered per global claim. |
| `retryDelayMilliseconds` | exponential 1-60 seconds | Receives the next attempt number and returns its delay. |
| `retryDelayMilliseconds` | exponential 1-60 seconds | Receives a retry index; returns milliseconds (see below). |

The limits measure normalized JSON encoded as UTF-8. `maxResultBytes` also
limits each computed snapshot getter and effect-handler result. Explicit
Expand All @@ -36,6 +36,22 @@ names enter the broadcast outbox. `retryDelayMilliseconds` should return a
non-negative finite number; an invalid application callback will fail the
affected failure path rather than schedule an invalid timestamp.

Interrupted actor messages use the existing `retryDelayMilliseconds` callback
on recovery. After the first claimed attempt they are immediately eligible;
after persisted attempts 2, 3, 4, ... the callback receives 1, 2, 3, ... . Its
default delays are 1, 2, 4 seconds up to 60 seconds. This is the total claim count,
including earlier handler failures, not a separate lease-loss counter. The
availability timestamp is stored transactionally in ready membership, survives
worker replacement and stale-process cleanup, and keeps the message ahead of
later messages for that actor. Other actors remain eligible. Custom delays,
including zero, remain authoritative.

Lease loss alone never creates a dead letter, including with `maxAttempts: 1`.
A handler failure still uses the full attempt count with `maxAttempts` and
passes that count to `retryDelayMilliseconds`, as before. Applications needing
a hard retry or elapsed-time bound must observe `activation.lost` and stop or
intervene. No setting or schema change is required.

`maxStateBytes` is a hard limit that fails the turn. `warnStateBytes` is a soft
threshold that keeps the turn. A commit above the threshold reports one
`solid_objects.state.large` instrumentation event with the actor type, the actor
Expand Down Expand Up @@ -76,6 +92,12 @@ and `false` for a timeout. An older adapter that returns `void` remains
compatible and keeps the fast cadence. Wake-ups reduce latency, while database
polling remains the correctness path.

Busy SQL workers and testing drains give the event loop a turn between passes.
In Node this lets timers and input run even with synchronous SQLite calls.
An instrumentation callback can abort `runtime.run()` before its next pass.
This does not interrupt a synchronous handler: responsiveness is bounded by
one pass, or by a single overlong handler that must return first.

## Retention and cleanup

| Option | Default | Contract |
Expand Down
31 changes: 16 additions & 15 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,21 +44,22 @@ result. When an exporter or observer raises, both runtimes log
Delivery is best effort and synchronous callbacks should be short;
JavaScript does not await exporters. Telemetry is not a durable audit trail.

| Event | Meaning |
| ------------------------------------------- | ---------------------------------------------------------------------------- |
| `activation.started/completed/failed` | Local actor activation hook lifecycle |
| `message.started/completed/failed/rejected` | One attempt's execution outcome |
| `message.retry` | Failed attempt durably queued for another attempt |
| `dead_letter.created` | Message exhausted retries or failed permanently |
| `commit_action.started/completed/failed` | One registered commit action inside the commit transaction |
| `mailbox.depth` | On-demand diagnostic sample; `depth` is null when the sample is truncated |
| `reminder.enqueued` | Due reminder dispatch; lateness is measured from due time |
| `outbox.age` | Delivery observation; age is time since the item's current availability time |
| `recovery.reclaimed` | A previously claimed, interrupted message begins another attempt |
| `recovery.completed/failed` | Durable effect recovery callback commits or enters the dead-letter queue |
| `snapshot.read` | Authorized snapshot constructed without exposing its contents |
| `realtime.connected/disconnected` | Actor subscription added or removed |
| `payload_broadcast.failed` | One personalized payload failed; `payload` names it |
| Event | Meaning |
| ------------------------------------------- | ------------------------------------------------------------------------------------- |
| `activation.started/completed/failed` | Local actor activation hook lifecycle |
| `message.started/completed/failed/rejected` | One attempt's execution outcome |
| `activation.lost` | A message turn lost its activation fence; includes message ID and total attempt count |
| `message.retry` | Failed attempt durably queued for another attempt |
| `dead_letter.created` | Message exhausted retries or failed permanently |
| `commit_action.started/completed/failed` | One registered commit action inside the commit transaction |
| `mailbox.depth` | On-demand diagnostic sample; `depth` is null when the sample is truncated |
| `reminder.enqueued` | Due reminder dispatch; lateness is measured from due time |
| `outbox.age` | Delivery observation; age is time since the item's current availability time |
| `recovery.reclaimed` | A previously claimed, interrupted message begins another attempt |
| `recovery.completed/failed` | Durable effect recovery callback commits or enters the dead-letter queue |
| `snapshot.read` | Authorized snapshot constructed without exposing its contents |
| `realtime.connected/disconnected` | Actor subscription added or removed |
| `payload_broadcast.failed` | One personalized payload failed; `payload` names it |

Events describe local observations. Concurrent deletion, crashes, and failed
exporters can omit events. Never infer exactly-once delivery from event counts.
Expand Down
3 changes: 2 additions & 1 deletion docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,8 @@ for the same time emits nothing.

## Dead letters, retry, and redrive

A message that exhausts its attempts becomes a dead letter. An effect or a
A handler failure that exhausts its attempts becomes a dead letter.
Lease loss alone remains pending with [recovery backoff](configuration.md). An effect or a
broadcast that exhausts its attempts stays in its own table with
`status = 'dead'`. All three are read and retried through one receiver, which
carries the kind:
Expand Down
Loading
Loading