Skip to content

feat!: optimize ship from saved runs with overlays and a blind judge - #83

Merged
Vzlentin merged 5 commits into
mainfrom
feat/optimize-ship-from-saved-runs-with-overlays-and
Nov 1, 2026
Merged

Vzlentin merged 5 commits into
mainfrom
feat/optimize-ship-from-saved-runs-with-overlays-and

Conversation

@Vzlentin

@Vzlentin Vzlentin commented Nov 1, 2026

Copy link
Copy Markdown
Owner

optimize ship now improves the whole surface of a run (prompts, profiles, skills, AGENTS.md) from the ship runs that already happened, and keeps an edit only when a blind judge prefers what it produced.

pastRuns("ship")  ->  baseline (item, start, bundle, result)   no rerun unless the head is gone or it did not ship
INV  -> prompts/*.md, agents/*.md edits ┐
DOM  -> skills/*/SKILL.md, AGENTS.md    ├-> Compose -> one overlay + kept|revised|dropped notes -> validate
                                        ┘
run("ship", { from: <baseline run>, overlay })  per item
select(item, diff A, diff B)  blind, stable A/B order, tie = 0
promote when wins/items >= 0.75 and items >= 4  ->  best/, baseline/, overlay.diff, notes.txt
lessons through notes.finish, none when the overlay changes more than one file (confounded)
  • run(name, { overlay }): prompts/<n>.md and agents/<n>.md keys replace or remove bundle entries. Every other key goes to the sessions as Spec.overlay (harness 0.6.2), relative to the worktree or absolute, and child runs inherit it. Nothing on disk changes.
  • run(name, { from: <runId> }) starts a child with the prompts and profiles of a saved root run, so the overlay is the only difference with the baseline.
  • pastRuns(workflow, { runs?, limit? }) is a saved script call over the completed root runs of the store. The agents global gives the profile texts of the run's bundle.
  • Validation before any rollout: skill frontmatter, no removed skill that AGENTS.md names, AGENTS.md at most --always-on-max (default the baseline length), and a non-empty overlay. An invalid candidate ships nothing and reports why.
  • Breaking: --improve, --trials, prompts/reflect.md and the rubric score are gone. --items <folder> still works, and --runs and --limit (default 8) pick saved runs. TASK.version goes to 18, so runs saved by an earlier version do not resume.

…iles

A run could replace prompts of a child run, but not its agent
profiles or the files its sessions read. optimize needs both to try
a candidate without changing the disk.

- `run(name, { overlay })` takes an object of path to text or null,
  checked with a TypeBox schema in `childInput`. A bad value fails
  with `run <name>: overlay is not an object of path to text or
  null`. `overlay` is a reserved arg and does not reach the script.
- A key `prompts/<name>.md` replaces or removes that template, and a
  key `agents/<name>.md` that profile, through `profileOf`. The
  overlay wins over `prompts` for the same template. A key for a
  name that the bundle does not have fails the call.
- Every other key is a file overlay. The child's input saves it as
  `overlay`, merged over the caller's, so child runs inherit it.
  `sessionOf` gives it as `spec.overlay` of the children library.
  A run with no file keys saves no `overlay` field.
- Take @vzlentin/harness ^0.6.2 for `Spec.overlay`, and bump
  `TASK.version` to 18 for the new input field.
- docs/spec.md describes `overlay` in Child runs, and points to it
  from Ship, RPI, Implement, Thermoloop and Judge.
- New tests cover the replaced template and profile, the session
  overlay of a child and a grandchild, the unchanged disk file, a
  run with no overlay, removed templates and bad values.
…undle

optimize needs the tasks, baselines and surfaces of earlier ship
runs. A script could not read the store, and a child run could only
use the caller's prompts and profiles.

- New script call `pastRuns(workflow, { runs, limit })` gives the
  completed root runs of `workflow`, newest first: run ID, item,
  repository, start, saved prompts, profile texts and result. An ID
  of `runs` that is not a completed root run fails with
  `no completed <workflow> run <id>`. The call is saved, so a resume
  replays it.
- It scans the `workflows.run` tasks of all versions in a read-only
  commit of the host task, so it takes no lock of its own. Child
  runs and runs in progress are left out.
- `run(name, { from: <runId> })` gives the child the prompts and
  profiles of that saved root run. Scripts and lib stay the
  caller's, and `prompts` and `overlay` apply on top. An unknown ID
  fails with `run <name>: no run <id>`. `from` is a reserved arg and
  does not reach the child script.
- src/past.ts holds the scan helpers that src/store.ts also uses
  now. The saved input shape does not change, so `TASK.version`
  stays the same.
- docs/spec.md, workflows/script.d.ts and the tool guidelines in
  pi/index.ts describe both.
- New tests cover the list, `limit`, `runs`, failed and unknown IDs,
  the replay on resume, and a child run from a saved bundle under
  an overlay.
…gents global

optimize scored candidates with a reflect turn and score(). It
could not start from past ship runs, edit agent profiles, or
compare two diffs of one item side by side.

- optimize.js takes its baselines from `pastRuns` or from an
  `--items` folder. It reruns each baseline once with `from`, asks
  the INV turn of prompts/inv.md for edits, and runs the rollouts
  with `from` and `overlay`.
- A blind A/B selector on prompts/select.md picks the better diff
  of each item. A candidate is promoted at 0.75 wins with at least
  4 items. The run writes `best/`, `baseline/` and `overlay.diff`.
- prompts/reflect.md and `score()` are gone.
- The new script global `agents` gives the profile texts of the
  bundle, from `profileTexts` in src/past.ts. An `--items` run puts
  `agents/<name>.md` into the baseline texts, so INV can edit a
  profile.
- workflows/script.d.ts, pi/index.ts and docs/spec.md describe the
  new optimize and `agents`.
- The optimize tests cover past runs, an items folder, the rerun,
  promotion and the A/B order. The items test reads `overlay.diff`
  and checks the diff of `agents/researcher.md` and its `edit`
  header line, so a profile edit that does not reach the diff fails.
optimize could only edit the ship prompts and profiles. Each INV
edit went straight into the overlay, and nothing checked the
candidate before its rollouts.

- A DOM turn on prompts/dom.md proposes edits to skills
  (`.agents/skills` or `.pi/skills` under the worktree, or the user
  skill folders) and to `AGENTS.md`. A path that is not a skill or
  `AGENTS.md`, or that INV also edits, fails the run.
- A Compose turn on prompts/compose.md merges the INV and DOM edits
  into one candidate overlay. It gives one
  `kept | revised | dropped -> <edit> -> <moment>` note per edit, in
  the report and in `<runDir>/notes.txt`.
- The candidate is checked before any rollout. An empty overlay, a
  skill without `name` or `description`, a removed skill that
  `AGENTS.md` names, or an `AGENTS.md` over the cap makes the
  candidate invalid. The run then starts no rollout, reports each
  reason and exits 0.
- The new `--always-on-max` sets the cap. By default it is the
  length of the `AGENTS.md` at the start.
- docs/spec.md, README.md, pi/index.ts and src/entry.ts describe
  DOM, Compose, the notes and the validation.
- New optimize tests cover the DOM and Compose turns, the notes and
  each invalid candidate.
optimize kept no run memory and gave no lessons. A candidate that
changes more than one file cannot tell which change made a verdict,
so its rows must not teach a lesson.

- optimize.js opens a run memory with `memory()`. After the verdict
  of each item, it notes one row
  `item <subject>: files <paths>, verdict <win|tie|loss>,
  confounded <yes|no>`. The paths are the overlay keys, and
  `confounded` is `yes` when the overlay has more than one path.
- The run returns through `notes.finish(meta.description, result)`,
  so the result has `lessons` and the report has the lesson lines.
  A confounded overlay gives `lessons: []` with no harvest turn. An
  invalid candidate notes no row, so it gives `lessons: []` with no
  turn.
- The `harvest` prompt gives no lesson that rests only on
  `confounded yes` rows.
- docs/spec.md describes the rows, `confounded`, the lessons of
  optimize and the `memory/` folder of the run. README.md has one
  sentence about them.
- New optimize tests cover the confounded rows of a two-file
  overlay with no lesson, the harvested lessons of a one-file
  overlay, and a resume after Ctrl+C that gives the same result,
  A/B order and `best/` files with no extra child run. The invalid
  candidate tests expect no harvest turn.
@Vzlentin
Vzlentin merged commit 0a7b35b into main Nov 1, 2026
3 checks passed
@Vzlentin
Vzlentin deleted the feat/optimize-ship-from-saved-runs-with-overlays-and branch November 1, 2026 00:19
@vzlentin-release vzlentin-release Bot mentioned this pull request Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant