Repository navigation
feat!: optimize ship from saved runs with overlays and a blind judge - #83
Merged
Vzlentin merged 5 commits intoNov 1, 2026
Merged
Conversation
…iles
A run could replace prompts of a child run, but not its agent
profiles or the files its sessions read. optimize needs both to try
a candidate without changing the disk.
- `run(name, { overlay })` takes an object of path to text or null,
checked with a TypeBox schema in `childInput`. A bad value fails
with `run <name>: overlay is not an object of path to text or
null`. `overlay` is a reserved arg and does not reach the script.
- A key `prompts/<name>.md` replaces or removes that template, and a
key `agents/<name>.md` that profile, through `profileOf`. The
overlay wins over `prompts` for the same template. A key for a
name that the bundle does not have fails the call.
- Every other key is a file overlay. The child's input saves it as
`overlay`, merged over the caller's, so child runs inherit it.
`sessionOf` gives it as `spec.overlay` of the children library.
A run with no file keys saves no `overlay` field.
- Take @vzlentin/harness ^0.6.2 for `Spec.overlay`, and bump
`TASK.version` to 18 for the new input field.
- docs/spec.md describes `overlay` in Child runs, and points to it
from Ship, RPI, Implement, Thermoloop and Judge.
- New tests cover the replaced template and profile, the session
overlay of a child and a grandchild, the unchanged disk file, a
run with no overlay, removed templates and bad values.
…undle
optimize needs the tasks, baselines and surfaces of earlier ship
runs. A script could not read the store, and a child run could only
use the caller's prompts and profiles.
- New script call `pastRuns(workflow, { runs, limit })` gives the
completed root runs of `workflow`, newest first: run ID, item,
repository, start, saved prompts, profile texts and result. An ID
of `runs` that is not a completed root run fails with
`no completed <workflow> run <id>`. The call is saved, so a resume
replays it.
- It scans the `workflows.run` tasks of all versions in a read-only
commit of the host task, so it takes no lock of its own. Child
runs and runs in progress are left out.
- `run(name, { from: <runId> })` gives the child the prompts and
profiles of that saved root run. Scripts and lib stay the
caller's, and `prompts` and `overlay` apply on top. An unknown ID
fails with `run <name>: no run <id>`. `from` is a reserved arg and
does not reach the child script.
- src/past.ts holds the scan helpers that src/store.ts also uses
now. The saved input shape does not change, so `TASK.version`
stays the same.
- docs/spec.md, workflows/script.d.ts and the tool guidelines in
pi/index.ts describe both.
- New tests cover the list, `limit`, `runs`, failed and unknown IDs,
the replay on resume, and a child run from a saved bundle under
an overlay.
…gents global optimize scored candidates with a reflect turn and score(). It could not start from past ship runs, edit agent profiles, or compare two diffs of one item side by side. - optimize.js takes its baselines from `pastRuns` or from an `--items` folder. It reruns each baseline once with `from`, asks the INV turn of prompts/inv.md for edits, and runs the rollouts with `from` and `overlay`. - A blind A/B selector on prompts/select.md picks the better diff of each item. A candidate is promoted at 0.75 wins with at least 4 items. The run writes `best/`, `baseline/` and `overlay.diff`. - prompts/reflect.md and `score()` are gone. - The new script global `agents` gives the profile texts of the bundle, from `profileTexts` in src/past.ts. An `--items` run puts `agents/<name>.md` into the baseline texts, so INV can edit a profile. - workflows/script.d.ts, pi/index.ts and docs/spec.md describe the new optimize and `agents`. - The optimize tests cover past runs, an items folder, the rerun, promotion and the A/B order. The items test reads `overlay.diff` and checks the diff of `agents/researcher.md` and its `edit` header line, so a profile edit that does not reach the diff fails.
optimize could only edit the ship prompts and profiles. Each INV edit went straight into the overlay, and nothing checked the candidate before its rollouts. - A DOM turn on prompts/dom.md proposes edits to skills (`.agents/skills` or `.pi/skills` under the worktree, or the user skill folders) and to `AGENTS.md`. A path that is not a skill or `AGENTS.md`, or that INV also edits, fails the run. - A Compose turn on prompts/compose.md merges the INV and DOM edits into one candidate overlay. It gives one `kept | revised | dropped -> <edit> -> <moment>` note per edit, in the report and in `<runDir>/notes.txt`. - The candidate is checked before any rollout. An empty overlay, a skill without `name` or `description`, a removed skill that `AGENTS.md` names, or an `AGENTS.md` over the cap makes the candidate invalid. The run then starts no rollout, reports each reason and exits 0. - The new `--always-on-max` sets the cap. By default it is the length of the `AGENTS.md` at the start. - docs/spec.md, README.md, pi/index.ts and src/entry.ts describe DOM, Compose, the notes and the validation. - New optimize tests cover the DOM and Compose turns, the notes and each invalid candidate.
optimize kept no run memory and gave no lessons. A candidate that changes more than one file cannot tell which change made a verdict, so its rows must not teach a lesson. - optimize.js opens a run memory with `memory()`. After the verdict of each item, it notes one row `item <subject>: files <paths>, verdict <win|tie|loss>, confounded <yes|no>`. The paths are the overlay keys, and `confounded` is `yes` when the overlay has more than one path. - The run returns through `notes.finish(meta.description, result)`, so the result has `lessons` and the report has the lesson lines. A confounded overlay gives `lessons: []` with no harvest turn. An invalid candidate notes no row, so it gives `lessons: []` with no turn. - The `harvest` prompt gives no lesson that rests only on `confounded yes` rows. - docs/spec.md describes the rows, `confounded`, the lessons of optimize and the `memory/` folder of the run. README.md has one sentence about them. - New optimize tests cover the confounded rows of a two-file overlay with no lesson, the harvested lessons of a one-file overlay, and a resume after Ctrl+C that gives the same result, A/B order and `best/` files with no extra child run. The invalid candidate tests expect no harvest turn.
Vzlentin
deleted the
feat/optimize-ship-from-saved-runs-with-overlays-and
branch
November 1, 2026 00:19
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
optimize shipnow improves the whole surface of a run (prompts, profiles, skills,AGENTS.md) from the ship runs that already happened, and keeps an edit only when a blind judge prefers what it produced.run(name, { overlay }):prompts/<n>.mdandagents/<n>.mdkeys replace or remove bundle entries. Every other key goes to the sessions asSpec.overlay(harness 0.6.2), relative to the worktree or absolute, and child runs inherit it. Nothing on disk changes.run(name, { from: <runId> })starts a child with the prompts and profiles of a saved root run, so the overlay is the only difference with the baseline.pastRuns(workflow, { runs?, limit? })is a saved script call over the completed root runs of the store. Theagentsglobal gives the profile texts of the run's bundle.AGENTS.mdnames,AGENTS.mdat most--always-on-max(default the baseline length), and a non-empty overlay. An invalid candidate ships nothing and reports why.--improve,--trials,prompts/reflect.mdand the rubric score are gone.--items <folder>still works, and--runsand--limit(default 8) pick saved runs.TASK.versiongoes to 18, so runs saved by an earlier version do not resume.