A durable execution engine for AI agents, in TypeScript on Postgres.
keel runs background jobs and multi-step agent workflows so they survive crashes, retry intelligently, stay within budget, and hand off to a person when they should.
Most job engines treat an AI agent as just another long-running job: when a step fails, they retry it. Agents fail in ways that make blind retries wrong:
- A hallucinated tool call won't fix itself on retry. The model needs to be told what it got wrong.
- A refund over an approval limit shouldn't be retried at all. A person has to decide.
- A runaway agent can burn through an LLM budget faster than any rate limit notices.
keel works out why a step failed and acts on it. It treats tokens and dollars as a scheduling resource, and makes human approval a normal part of a workflow.
| Durable steps | ctx.step.run() stores each step's result. After a crash or retry, completed steps replay from storage instead of running again. |
| Failure classification | Every failure is classified as transient, bad_input, bad_output, needs_human, over_budget or fatal, and a policy picks the action: back off, retry with a corrective hint, fall back to a cheaper model, fail, or escalate. |
| Jev classifier | An optional classifier that reads error messages, the failing step and the rejected model output, not just status codes, using TypeSafe's Jev model, with a rule-based fallback. |
| Budgets | Per-run budgets stop a run before its next step, or before a step whose declared maximum cost could cross the budget, and can fall back to a cheaper model instead of stopping. Tenant and per-task daily budgets defer runs instead of failing them. |
| Human in the loop | Approvals inside workflows, and escalated failures that wait for a reviewer's decision. A run that failed wrongly can be retried by hand (engine.retryRun) and resumes after its completed steps. |
| Safe side effects | Each step gets a stable idempotency key, so a step re-run after a crash can't charge a card twice. |
| Waits and events | Durable sleeps and waits for external events that free the worker in the meantime. |
| Dashboard | pnpm dashboard: a run list, a logbook of each run (steps, costs, errors, decisions), Approve/Reject for pending approvals, and Retry for failed runs. No dependencies, no JavaScript. |
| Serverless mode | worker.runOnce({ deadlineMs }) works through due runs and returns, for cron jobs and serverless functions. A run cut off by the deadline resumes from its last stored step on the next call. |
| Crash safety | Postgres SKIP LOCKED claims, leases with heartbeats, fenced writes, and dead-lettering of runs that keep crashing their workers. |
Requires Node.js 24+, pnpm 12+ and Docker. With nvm, nvm use picks Node 24 from .nvmrc.
git clone git@github.com:yodablocks/keel.git && cd keel
pnpm install
cp .env.example .env # optionally add TYPESAFE_API_KEY for the Jev classifier
pnpm db:up && pnpm db:migrate # Postgres 17 on localhost:5433
pnpm demo --offline # the end-to-end agent demo, no API key neededimport { createEngine } from "keel"; // see "Use it in your project" below
const engine = createEngine({ connectionString: process.env.DATABASE_URL! });
const worker = engine.createWorker({
queue: "agents",
tasks: {
refund: async ({ orderId, amount }, ctx) => {
const order = await ctx.step.run("load-order", () => orders.get(orderId));
if (amount > 500) {
const decision = await ctx.approval.request("manager-ok", {
prompt: `Refund $${amount} for order ${orderId}?`,
timeoutMs: 24 * 3600_000,
});
if (decision.status !== "approved") return decision.status;
}
return ctx.step.run("refund", ({ idempotencyKey }) =>
payments.refund(order.chargeId, amount, { idempotencyKey }),
);
},
},
});
worker.start();
await engine.enqueue("refund", { orderId: 42, amount: 900 }, { queue: "agents", budget: { usd: 2 } });If the worker crashes after load-order, another worker resumes the run without loading the order again. While the run waits for approval it holds no worker at all.
No long-lived process? Call await worker.runOnce({ deadlineMs: 25_000 }) from a cron job or a serverless function instead of worker.start().
See the guide for every feature: steps, waits, budgets, the classifier, approvals, and the rules that keep replay correct.
keel is not published to npm. Build a tarball and add it to your project:
cd keel && pnpm pack # builds dist/ and writes keel-0.0.0.tgz
cd ../your-app && pnpm add ../keel/keel-0.0.0.tgz pgThe package ships compiled JavaScript with type declarations, the SQL migrations, and the TypeScript sources for source maps. Run migrate(connectionString) once at startup, or pnpm db:migrate from the keel checkout.
pnpm dashboard # http://127.0.0.1:4400Each run has a logbook: steps with their cost, failures with how they were classified and what the policy did, and waits and decisions, in the order they happened. Pending approvals and escalations can be decided right there.
The dashboard has no login. It listens on 127.0.0.1 by default; don't expose it to a network. Pages allow no scripts, forms carry a per-process token, and requests for other hostnames are refused.
pnpm bench on the same Apple M4 laptop (10 cores, 32 GB), Postgres 17 in Docker on the same machine. Each run executes one durable step; 2,000 runs per configuration; all workers in one Node process. Runs per second:
| Workers | Sep 2026: Darwin 25.6, Node 25 | Oct 2026: Darwin 27.0, Node 26 |
|---|---|---|
| 1 | 445 to 471 | 265 to 278 |
| 2 | 861 to 873 | 510 to 576 |
| 4 | 1,365 to 1,442 | 800 to 934 |
| 8 | 1,618 to 1,683 | 1,059 to 1,471 |
| 16 | 1,897 to 1,904 | 1,151 to 1,395 |
| 32 | 1,974 to 2,023 | 1,176 to 1,438 |
Enqueue to completion on idle workers: p50 23 to 27 ms, p99 51 to 60 ms in both. That is mostly the default 50 ms polling interval.
-
The October drop is the machine, not keel. keel's September code, benchmarked in alternation with the current code on the same day, scored the same (1,336 to 1,393 runs per second at 32 workers). Between the two dates macOS was upgraded (Darwin 25.6 to 27.0), which is the likely cause: Docker runs Postgres in a virtual machine, and one worker, which is bound by database round trips, lost the most (about 40%). A macOS security scanner (
syspolicyd) was also busy during the October runs. Battery power was ruled out. -
Throughput scales to 32 workers. An earlier version plateaued at 8 workers and dipped at 32. Profiling showed Node using under half of one core, and the cause was every step updating the same daily spend row per task. Spend is now sharded over 16 rows, which lifted 32 workers from about 1,600 to 2,000 runs per second (September), at the cost of about 12 to 15% at 4 to 8 workers.
-
These are single-machine numbers with the database next to the workers, from a small number of runs each. Network latency to a managed Postgres, more workers per process, or heavier steps change them. Rerun
pnpm benchon your own setup; raw results are inbench-results/.
Runs are rows in Postgres. Workers claim them with FOR UPDATE SKIP LOCKED and hold a lease that heartbeats extend. Completed steps, waits and usage are stored per run, so any worker can resume any run by replaying its handler.
stateDiagram-v2
[*] --> queued: enqueue
queued --> running: worker claims (lease)
running --> completed: handler returns
running --> queued: transient failure, backoff
running --> waiting: wait, approval, escalation, tenant or task over budget
waiting --> running: timer due, event, decision, budget allows
running --> running: lease expired, another worker resumes
running --> failed: policy stops (bad input, bug, rejected)
running --> dead: maxAttempts exhausted
completed --> [*]
failed --> [*]
dead --> [*]
When a handler throws, the error goes through a classifier (what kind of failure is this?) and then a policy (what should happen?):
| Failure | Kind | Default action |
|---|---|---|
| HTTP 408/429/5xx, timeouts, connection resets | transient |
retry with exponential backoff and full jitter |
| Model output that can't be used (invalid JSON, unknown tool) | bad_output |
retry at once, with the error passed to the handler as ctx.hint |
| Invalid input (HTTP 400/422, validation errors) | bad_input |
fail |
| Approval limits, refusals, missing permissions | needs_human |
escalate to a reviewer |
| Run over its budget | over_budget |
fall back to a cheaper model once, if configured; otherwise escalate to a reviewer |
| Anything else | fatal |
fail |
pnpm demo runs one agent (research, plan, draft, send, follow-up) across two worker processes, with a scripted fake model and tools that misbehave on cue. In a single run:
- Hallucinated tool. The model calls
serch_web. Jev classifies the plain error asbad_output(confidence 0.97), and the retry's hint fixes the plan. - Crash. Worker A is
SIGKILLed mid-draft. Worker B resumes, and the earlier steps are replayed from storage, not called again. - Rate limit. The email API returns 429. The run backs off and retries with the same idempotency key.
- Budget. With $0.13 of its $0.15 budget spent, the follow-up model call could cost up to $0.03 (its
maxUsd). It never starts, so the budget is not overshot, and the run escalates instead of failing. - Approval. A reviewer raises the budget and approves, and the run completes.
The full recording, made with real Jev (jev-latest, November 2026), is in docs/demo-transcript.txt. Earlier recordings, from before maxUsd, are kept: October (jev-1.13.0) and September. pnpm test runs the offline version and checks every moment.
keel-compare builds the demo agent above twice more, in LangGraph.js and in Temporal, against the same scripted model and tools, and records what each one handles for you and what has to be written by hand.
- Temporal has the best crash recovery: the server notices a dead worker by its missed heartbeat and hands the step to another one.
- LangGraph has the cleanest human-in-the-loop API:
interrupt()in any node, resume with any value. - Neither handles bad model output or budgets: both retry a hallucinated tool call with the same input unless you write the loop yourself.
I wrote keel and designed it for these five cases, so the comparison says where the other two do better, and ends with what I would change in keel.
pnpm openroles:sync runs a workload of mine through keel: it reads every job board that openroles, my own job
search tool, tracks, one run per board, against the live Ashby, Greenhouse and Lever APIs. First run, October 2026:
- 447 runs, 8,287 jobs read, 6 minutes at one request per second per host
- 446 completed, 1 failed: a Lever board that answered 404, which keel failed without retrying
- The first offline test of this workload found a bug: rules marked
fetchnetwork errors asfatal, becausefetchputs the code onerror.cause. Fixed before any eval was run
It is real work, but it is not an agent: there are no model steps, and nothing crashed. It shows the engine running something other than the demo, not how classification performs in production.
eval-sets/public-issues.json: error messages copied from GitHub issues of agent
and model SDKs (openai-node, anthropic-sdk-typescript, vercel/ai, langchainjs, openai-agents-js, the MCP SDK,
undici), each linked to its issue. I labelled them and committed the labels before any classifier ran.
pnpm eval:classifier --cases eval-sets/public-issues.json, jev-1.13.0, October 2026:
| Classifier | Right kind | Right action |
|---|---|---|
| Rules only | 28 / 75 (37%) | 38 / 75 (51%) |
| Jev alone | 50 to 51 / 75 (67 to 68%) | 63 to 64 / 75 (84 to 85%) |
keel cascade (JevClassifier: rules for explicit signals, Jev for the rest, rules below 0.5 confidence) |
46 to 47 / 75 (61 to 63%) | 59 to 60 / 75 (79 to 80%) |
- Ranges are two passes over identical input in one run, so a difference of one case is run-to-run noise.
- Right action counts a verdict as right when keel's default policy would do the same thing as for the label:
bad_inputandfatalboth fail. I chose this measure after seeing the results, so the kind column is the headline. 13 of Jev's 25 misses are betweenfatalandbad_input. - Rules are wrong in the costly direction: 36 of their 37 wrong actions fail a run that should have been retried (14) or retried with a correction (22).
- The cascade did worse than Jev alone. Below 0.5 confidence it uses the rule verdict, and rules answer
fatalwhen they see no explicit signal. That changed 7 verdicts here: 1 for the better, 4 for the worse, and 2 wrong either way. Changing the fallback on the strength of this set would have been tuning on the test set, so the change was checked on a fresh set instead (below), where it lost.
Caveats: the messages were written by other people, but the labels are one person's, and that person wrote
the classifier prompt. Most errors were posted while developing, not hit in production. The mix has 1
needs_human case, so the set says almost nothing about escalation, and there is no step context. Details, and
the two labelling notes, are in eval-sets/README.md.
The finding above suggested a change: when Jev is unsure and the rules found no signal, keep Jev's answer instead
of the rules' fatal default. To check it without tuning on the public set, pnpm eval:harvest sampled 60 new
error lines from issues in 13 other repositories (Temporal, Trigger.dev, Inngest, LangGraph.js, Mastra, E2B and
others), with fixed queries, exclusions and seed, so no case was picked by hand
(eval-sets/fresh-issues.json). I labelled them and committed the labels before
any classifier ran. The decision rule, written before the set existed: ship the change only if it scores at least
as well as the current cascade on the same Jev answers.
| Classifier | Right kind | Right action |
|---|---|---|
Always answer fatal (baseline) |
43 / 60 (72%) | 48 / 60 (80%) |
| Rules only | 43 / 60 (72%) | 48 / 60 (80%) |
| Jev alone | 43 to 44 / 60 (72 to 73%) | 52 to 53 / 60 (87 to 88%) |
| keel cascade, as shipped | 48 to 49 / 60 (80 to 82%) | 55 / 60 (92%) |
| keel cascade with the change | 43 to 44 / 60 (72 to 73%) | 52 to 53 / 60 (87 to 88%) |
- The change lost, so it was reverted. It changed 8 to 9 verdicts: 1 to 2 for the better, 6 to 7 for the
worse. The rules'
fatalfallback stays. - The result doesn't depend on my labels. Three models (Opus 5.5, Sonnet 5.5 and Haiku 4.5), working blind,
relabelled the 60 cases. The change loses under every one of their labellings, by kind and by action
(
eval-sets/model-labels/, protocol committed before labelling). A fourth requested model ran as Opus 5.5 again, so it doesn't count. - The two sets disagree about the fallback: it cost about 3 cases on the public set and gained about 5 here. Every
labeller marks 46 to 48 of these 60 as errors that should fail the run, mostly configuration and API misuse,
against 32 of 75 in the public set. A fallback to
fatalis right more often here. Neither set is large enough to settle it. - A classifier that always answers
fatalmatches rules and Jev on the kind here, so the kind score on this set says little on its own. The action column separates them better. - Same caveats as the public set: one labeller who wrote the classifier prompt, errors mostly from development,
no step context, and no
needs_humancases at all.
Accuracy counts every miss the same, but failing a run that would have succeeded loses it, while retrying one
that was going to fail wastes a couple of attempts. After two readers of the results made that point, I wrote a
cost table, in units of one wasted retry (eval-sets/COSTS.md), and committed it before
scoring anything with it. Costs:
- a lost run costs 5
- a needless escalation costs 2
- retrying something that needed a person costs 20
pnpm eval:cost <result file>, summed over both passes, lower is better:
| Public set (75) | Fresh set (60) | |
|---|---|---|
| Rules only | 370 | 120 |
| Jev alone | 59 | 39 |
| keel cascade, as shipped | 121 | 42 |
| keel cascade with the reverted change | not recorded | 39 |
- Exploratory, not a decision. Both sets were seen before the costs were written, and one reader had already shown the change wins on the fresh set once a lost run costs more than 3 to 4 retries. The table sets it at 5.
- By cost, the change wins on the fresh set (39 against 42), but only just. With a lost run at 2 instead of 5, the shipped cascade wins (18 against 21). So on this set the answer depends on the costs.
- Neither set tests the most expensive mistake. No case in either set that needed a person was retried, so the 20 never applies. The fresh set has no such cases at all.
- The next fresh set will be scored on this table, with the decision rule in
COSTS.md, and will include approval and permission errors so that the expensive cell can be measured.
pnpm eval:classifier on 30 hand-labelled agent failures written for keel:
| Classifier | M7 (Sep 2026) | M11 (Sep 2026) | Oct 2026 (jev-1.13.0) |
|---|---|---|---|
| Rules only (status codes, error codes, error classes) | 14 / 30 (47%) | 14 / 30 (47%) | 15 / 30 (50%) |
| Jev, error and task only | 29 / 30 (97%) | 30 / 30 (100%) | 30 / 30 (100%) |
| Jev with the failing step and model output | 30 / 30 (100%) | 30 / 30 (100%) | |
| keel cascade | 29 / 30 (97%) | 30 / 30 (100%) | 30 / 30 (100%) |
- Rules gained one case in October because they now read the network code on
error.cause(thefetch failedcase). - The M7 miss, an ambiguous tool-argument error, is classified correctly in the later runs even without step context, so the fix can't be credited to context alone. The M7 run did not record the concrete model version behind
jev-latest, so a model update can't be ruled out; runs now record it. - Step context raised confidence: on that ambiguous case from 0.60 to 0.75, and the mean across all 30 from 0.90 to 0.92 (up on 9 cases, down on 4). The public set above shows Jev varies by about one case in 75 between identical calls, so shifts this small may be noise.
This set was written and labelled by the classifier's author, and at 100% it is too easy to show anything more; the public set above is the harder test. To build a set from your own runs, pnpm eval:export --blind writes their failures to a file, pnpm eval:label labels it, and pnpm eval:classifier --cases <file> scores it. Full results are in eval-results/.
keel is an experimental project, and the name is not final. All planned milestones are implemented and tested (PLAN.md). Apart from one real workload of mine, it hasn't been used, and never in production. Known limitations include:
- Budgets can overshoot by one step, because a step's cost is known only after it runs, unless the step declares the most it can cost with
maxUsd. A step that fails counts only the cost it reported before failing, withcall.reportUsage. - Idempotency keys protect external calls only for services that accept them.
- Retention is opt-in: without
retentionon a worker or calls toengine.purge, finished runs are kept forever. - A single Postgres instance is the throughput ceiling: about 1,200 to 2,000 runs per second on one laptop in the benchmark above, lower since a macOS upgrade.
- The dashboard has no authentication, so it is for local or internal use only.
- On error messages from public issues, the shipped classifier picks the right kind about 6 times in 10, and the right action about 8 times in 10 (results).
The full list is under Known risks, each tagged with the milestone that addresses it.
Roadmap: phases 1 and 2 are complete; see PLAN.md for what was built, the measured results, and the deliberate non-goals.
pnpm test # node:test against the real Postgres from docker-compose (no database mocks)
pnpm typecheck # tsc --noEmit
pnpm db:migrate # applies db/migrations/*.sql in order
pnpm build # compiles src/ to dist/ (JavaScript and type declarations)
pnpm bench # throughput and latency, results in bench-results/
pnpm openroles:sync # the real workload (needs an openroles database)
pnpm eval:classifier [--cases file] # classifier eval; needs TYPESAFE_API_KEY
pnpm eval:export --blind / pnpm eval:label <file> # build and label an eval setCI (.github/workflows/ci.yml) runs the type check and the full test suite against Postgres 17, on Node 24 and Node 26, on every pull request and every push to main.
Node runs the TypeScript sources directly, so there is no build step. Tests include real crash recovery: worker processes are killed with SIGKILL mid-run.
src/ engine, failure classification, Jev classifier, migrations
db/ SQL migrations
test/ unit, integration and crash tests
scripts/ demo, classifier eval and labelling, real workload, migrate
eval-sets/ labelled eval sets beyond the synthetic one
docs/ guide and demo transcript
PLAN.md milestones, acceptance criteria, known risks

