Skip to content

fix(agent-context): tune prompts and codegen skill triggers with eval set - #438

Merged
dmealing merged 4 commits into
mainfrom
fm/mo-skills-audit-trigger-evals
Oct 14, 2026
Merged

dmealing merged 4 commits into
mainfrom
fm/mo-skills-audit-trigger-evals

Conversation

@dmealing

@dmealing dmealing commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

What Changed

  • Narrows the metaobjects-prompts skill description so it loads only for LLM-text code in projects that declare template.prompt, not for every prompt edit. The same description is also shipped into adopter .claude/skills through the agent-context conformance fixtures, which are regenerated to match.
  • Extends the metaobjects-codegen description to name the concrete trigger (editing the gen config in metaobjects.config.ts to choose generators, targets or the SQL dialect). The existing "ANY output derived from the MetaObjects model" wording is kept: the eval showed no over-trigger from it (every negative case stayed silent in every pass).
  • Adds a trigger evaluation set under agent-context/evals/skill-triggers/ (25 should-trigger and should-not-trigger cases for the two skills) with agent-context/evals/run-skill-triggers.sh to run it manually against a real Claude login, plus a README describing how to run and re-tune it.

Eval results (per pass)

Pass rates (runs passed / runs; 3 runs per case; Claude login, single arm; the case count is per skill):

  • Pass 1, original descriptions, 12+12 cases: codegen 36/36, prompts 32/36 (case 05: 0/3, case 10: 2/3).
  • Pass 2, prompts narrowed to projects that declare template.prompt, case 05 reworded: codegen 36/36, prompts 36/36.
  • Pass 3 (diagnostic, case 05 only, ORIGINAL wording, tuned description): 1/3; the other two runs loaded metaobjects-verify.
  • Pass 4, after the review fix (declare/adopt exempt from the exclusion, +case 13) and the codegen tune: codegen 36/36 (case 06 3/3), prompts 39/39.
    Case 05 rewording: from "meta verify --templates reports prompt drift on the ReviewDraft prompt after I renamed a field. Explain how to fix it." to "I renamed a field on the payload projection of the ReviewDraft template.prompt. Update the prompt template and its payload so rendering stays in sync." Original wording named the verify command, which metaobjects-verify owns, so loading verify was a defensible answer; bad test, not a real miss.
    Case 06 (dialect config): passed 3/3 in passes 1 and 2 against the old codegen description, then the pipeline's own test run scored 0/3 twice. Case file unchanged. Cause not established: run variance or a difference in the pipeline's test environment (its agent runs through a different provider router); the codegen description was tuned anyway (names editing the gen config / dialect) and case 06 is 3/3 in pass 4.
    Eval passes: 4 on the Claude login (over the usual three because the codegen rerun was added); pass 4 ran all 25 cases because --case matches names, which repeat across the two skills.
    Validation runs: run 1 failed in the Test step (fix agent: all providers failed); this is run 2.

The codegen-case-06 line of the Testing section below (2 of 3) is the pipeline's own spot check; the full rerun in pass 4 above is the record. The parse-only CI test was dropped at the operator's decision, so the evaluation is manual-only (agent-context/evals/run-skill-triggers.sh).

Validation

Risk Assessment

✅ Low: The change adds manual-only eval cases and a runner script, and edits two skill description lines. I confirmed that all 10 fixture mirrors match the source descriptions, and that the 13 prompts cases and 12 codegen cases exist. The README counts, the removal of the source-grepping SDK test and the recorded user decisions are all honored.

Testing

Drove the two changed skill descriptions through real Claude sessions on the subscription login (about 13 agent sessions, a few minutes of quota, no metered spend). Codegen dialect-config case 06 fired the skill 2 of 3 runs, up from 0 of 6 before the tune; the operator should rerun codegen cases to confirm the rate. New adopt-first-prompt case 13 fired 3 of 3. Over-trigger negatives 07 (f-string, no MetaObjects) and 08 (LangChain system prompt) stayed silent 3 of 3 each. Description text matches across the source skill and all five conformance mirrors. The full server TypeScript suite ran with 0 failures, but that run is non-live and is recorded as untested, not as a live pass. Worktree clean; evidence under the evidence directory only.

  • Live validation: ✅ go - 4 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Dialect/generator config request (postgres to sqlite) loads metaobjects-codegen skill ✅ pass live triggers/result.json (case 06-pos-dialect-config, 2 of 3 runs fired Skill)
New adopter project with zero template.prompt nodes asks to move f-string LLM prompt onto MetaObjects prompts loads metaobjects-prompts skill ✅ pass live triggers-prompts/adopt13/result.json (case 13-pos-adopt-first-prompt, 3 of 3 fired)
Non-MetaObjects project editing f-string LLM prompt does not load metaobjects-prompts skill (over-trigger guard) ✅ pass live triggers-prompts/neg07/result.json (case 07-neg-fstring-no-metaobjects, 0 Skill calls in 3 of 3 runs)
LangChain system-prompt edit in non-MetaObjects project does not load metaobjects-prompts skill ✅ pass live triggers-prompts/result.json (case 08-neg-langchain-system-prompt, 3 of 3 passed as non-trigger)
Full repository regression suite (server TypeScript bun test) passes after the change ⏸️ untested no Non-live regression run only: the bun test log reports 11026 tests, 10976 pass, 50 skip, 0 fail. That is not a live product run, so it is not counted as a pass for this scenario.
Evidence: Codegen case 06 live trigger result (2/3 fired)
{
  "schemaVersion": 1,
  "claudeVersion": "2.1.299",
  "startedAt": "2026-10-14T16:59:04.682Z",
  "durationSeconds": 8,
  "costUsd": 0.010226090000000002,
  "partial": false,
  "suite": {
    "root": "~/.no-mistakes/evidence/01M4XMMB6AV00HWH6FEMDGTA0R/triggers/plugin",
    "ablation": "none",
    "caseFilter": "*06-pos-dialect-config*",
    "threshold": 0,
    "concurrency": 4,
    "plugins": [
      {
        "name": "mo-skills",
        "version": "0.0.0",
        "path": "~/.no-mistakes/evidence/01M4XMMB6AV00HWH6FEMDGTA0R/triggers/plugin"
      }
    ]
  },
  "cases": [
    {
      "name": "06-pos-dialect-config",
      "dir": "evals/metaobjects-codegen/06-pos-dialect-config",
      "source": "prose",
      "promptMarkdown": "This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). Change the dialect that code generation uses from postgres to sqlite in the gen configuration.",
      "runsPerCase": 3,
      "timeoutSeconds": 180,
      "maxTurns": 3,
      "graders": [
        {
          "name": "trigger",
          "type": "tool_used",
          "weight": 1,
          "config": {
            "tool": "Skill",
            "input_match": "metaobjects-codegen",
            "min": 1
          }
        }
      ],
      "arms": {
        "with": [
          {
            "score": 1,
            "passed": true,
            "turns": 4,
            "costUsd": 0.0040053400000000005,
            "judgeCostUsd": 0,
            "durationSeconds": 7,
            "startedAt": "2026-10-14T16:59:04.716Z",
            "error": "exit 1: Reached maximum number of turns (3)",
            "tracePath": "/tmp/claude-eval-Xid3Fp/out/trace.jsonl",
            "skippedPaidGraders": false,
            "graders": [
              {
                "name": "trigger",
                "passed": true,
                "weight": 1,
                "explanation": "Skill called 1x (expected 1..∞)",
                "withOnly": false,
                "scored": true
              }
            ]
          },
          {
            "score": 0,
            "passed": false,
            "turns": 4,
            "costUsd": 0.0022048200000000006,
            "judgeCostUsd": 0,
            "durationSeconds": 5,
            "startedAt": "2026-10-14T16:59:04.716Z",
            "error": "exit 1: Reached maximum number of turns (3)",
            "tracePath": "/tmp/claude-eval-1NtkIL/out/trace.jsonl",
            "skippedPaidGraders": false,
            "graders": [
              {
                "name": "trigger",
                "passed": false,
                "weight": 1,
                "explanation": "Skill called 0x (expected 1..∞)",
                "withOnly": false,
                "scored": true
              }
            ]
          },
          {
            "score": 1,
            "passed": true,
            "turns": 4,
            "costUsd": 0.004015930000000001,
            "judgeCostUsd": 0,
            "durationSeconds": 8,
            "startedAt": "2026-10-14T16:59:04.716Z",
            "error": "exit 1: Reached maximum number of turns (3)",
            "tracePath": "/tmp/claude-eval-4mudw1/out/trace.jsonl",
            "skippedPaidGraders": false,
            "graders": [
              {
                "name": "trigger",
                "passed": true,
                "weight": 1,
                "explanation": "Skill called 1x (expected 1..∞)",
                "withOnly": false,
                "scored": true
              }
            ]
          }
        ]
      },
      "aggregates": {
        "score": 0.6666666666666666,
        "passRate": 0.6666666666666666
      }
    }
  ],
  "aggregates": {
    "casesTotal": 1,
    "casesPassed": 1,
    "overallScore": 0.6666666666666666,
    "overallPassRate": 0.6666666666666666
  }
}
Evidence: Codegen case 06 run log
Wrote ~/.no-mistakes/evidence/01M4XMMB6AV00HWH6FEMDGTA0R/triggers/result.json
Report: ~/.no-mistakes/evidence/01M4XMMB6AV00HWH6FEMDGTA0R/triggers/runs/report.html
Evidence: Prompts case 13 adopt-first-prompt live result (3/3 fired)
{
  "schemaVersion": 1,
  "claudeVersion": "2.1.299",
  "startedAt": "2026-10-14T16:59:32.669Z",
  "durationSeconds": 7,
  "costUsd": 0.01084939,
  "partial": false,
  "suite": {
    "root": "~/.no-mistakes/evidence/01M4XMMB6AV00HWH6FEMDGTA0R/triggers-prompts/adopt13/plugin",
    "ablation": "none",
    "caseFilter": "*13-pos-adopt*",
    "threshold": 0,
    "concurrency": 4,
    "plugins": [
      {
        "name": "mo-skills",
        "version": "0.0.0",
        "path": "~/.no-mistakes/evidence/01M4XMMB6AV00HWH6FEMDGTA0R/triggers-prompts/adopt13/plugin"
      }
    ]
  },
  "cases": [
    {
      "name": "13-pos-adopt-first-prompt",
      "dir": "evals/metaobjects-prompts/13-pos-adopt-first-prompt",
      "source": "prose",
      "promptMarkdown": "This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files) but has not declared any template.prompt yet. Our ticket summarizer builds its LLM prompt with an f-string in summarize.py. I want to start declaring prompts the MetaObjects way, so move that prompt onto it.",
      "runsPerCase": 3,
      "timeoutSeconds": 180,
      "maxTurns": 3,
      "graders": [
        {
          "name": "trigger",
          "type": "tool_used",
          "weight": 1,
          "config": {
            "tool": "Skill",
            "input_match": "metaobjects-prompts",
            "min": 1
          }
        }
      ],
      "arms": {
        "with": [
          {
            "score": 1,
            "passed": true,
            "turns": 4,
            "costUsd": 0.00351283,
            "judgeCostUsd": 0,
            "durationSeconds": 6,
            "startedAt": "2026-10-14T16:59:32.691Z",
            "error": "exit 1: Reached maximum number of turns (3)",
            "tracePath": "/tmp/claude-eval-YRyy8o/out/trace.jsonl",
            "skippedPaidGraders": false,
            "graders": [
              {
                "name": "trigger",
                "passed": true,
                "weight": 1,
                "explanation": "Skill called 1x (expected 1..∞)",
                "withOnly": false,
                "scored": true
              }
            ]
          },
          {
            "score": 1,
            "passed": true,
            "turns": 4,
            "costUsd": 0.0037549000000000003,
            "judgeCostUsd": 0,
            "durationSeconds": 6,
            "startedAt": "2026-10-14T16:59:32.691Z",
            "error": "exit 1: Reached maximum number of turns (3)",
            "tracePath": "/tmp/claude-eval-PZJaRt/out/trace.jsonl",
            "skippedPaidGraders": false,
            "graders": [
              {
                "name": "trigger",
                "passed": true,
                "weight": 1,
                "explanation": "Skill called 1x (expected 1..∞)",
                "withOnly": false,
                "scored": true
              }
            ]
          },
          {
            "score": 1,
            "passed": true,
            "turns": 4,
            "costUsd": 0.0035816600000000004,
            "judgeCostUsd": 0,
            "durationSeconds": 6,
            "startedAt": "2026-10-14T16:59:32.691Z",
            "error": "exit 1: Reached maximum number of turns (3)",
            "tracePath": "/tmp/claude-eval-iHyStx/out/trace.jsonl",
            "skippedPaidGraders": false,
            "graders": [
              {
                "name": "trigger",
                "passed": true,
                "weight": 1,
                "explanation": "Skill called 1x (expected 1..∞)",
                "withOnly": false,
                "scored": true
              }
            ]
          }
        ]
      },
      "aggregates": {
        "score": 1,
        "passRate": 1
      }
    }
  ],
  "aggregates": {
    "casesTotal": 1,
    "casesPassed": 1,
    "overallScore": 1,
    "overallPassRate": 1
  }
}
Evidence: Prompts case 07 f-string negative live result (3/3 silent)
{
  "schemaVersion": 1,
  "claudeVersion": "2.1.299",
  "startedAt": "2026-10-14T16:59:39.791Z",
  "durationSeconds": 7,
  "costUsd": 0.006202840000000001,
  "partial": false,
  "suite": {
    "root": "~/.no-mistakes/evidence/01M4XMMB6AV00HWH6FEMDGTA0R/triggers-prompts/neg07/plugin",
    "ablation": "none",
    "caseFilter": "*07-neg-fstring*",
    "threshold": 0,
    "concurrency": 4,
    "plugins": [
      {
        "name": "mo-skills",
        "version": "0.0.0",
        "path": "~/.no-mistakes/evidence/01M4XMMB6AV00HWH6FEMDGTA0R/triggers-prompts/neg07/plugin"
      }
    ]
  },
  "cases": [
    {
      "name": "07-neg-fstring-no-metaobjects",
      "dir": "evals/metaobjects-prompts/07-neg-fstring-no-metaobjects",
      "source": "prose",
      "promptMarkdown": "This is a plain project; it does not use MetaObjects. In src/summarize.py I build an OpenAI request with an f-string: f\"Summarize this ticket: {ticket.title}\". Add the customer's plan tier to the text.",
      "runsPerCase": 3,
      "timeoutSeconds": 180,
      "maxTurns": 3,
      "graders": [
        {
          "name": "trigger",
          "type": "tool_used",
          "weight": 1,
          "config": {
            "tool": "Skill",
            "input_match": "metaobjects-prompts",
            "min": 0,
            "max": 0,
            "arm": "both"
          }
        }
      ],
      "arms": {
        "with": [
          {
            "score": 1,
            "passed": true,
            "turns": 4,
            "costUsd": 0.0020966,
            "judgeCostUsd": 0,
            "durationSeconds": 5,
            "startedAt": "2026-10-14T16:59:39.811Z",
            "error": "exit 1: Reached maximum number of turns (3)",
            "tracePath": "/tmp/claude-eval-imv0ed/out/trace.jsonl",
            "skippedPaidGraders": false,
            "graders": [
              {
                "name": "trigger",
                "passed": true,
                "weight": 1,
                "explanation": "Skill called 0x (expected 0..0)",
                "withOnly": false,
                "scored": true
              }
            ]
          },
          {
            "score": 1,
            "passed": true,
            "turns": 4,
            "costUsd": 0.00203886,
            "judgeCostUsd": 0,
            "durationSeconds": 7,
            "startedAt": "2026-10-14T16:59:39.811Z",
            "error": "exit 1: Reached maximum number of turns (3)",
            "tracePath": "/tmp/claude-eval-HgsmCJ/out/trace.jsonl",
            "skippedPaidGraders": false,
            "graders": [
              {
                "name": "trigger",
                "passed": true,
                "weight": 1,
                "explanation": "Skill called 0x (expected 0..0)",
                "withOnly": false,
                "scored": true
              }
            ]
          },
          {
            "score": 1,
            "passed": true,
            "turns": 4,
            "costUsd": 0.00206738,
            "judgeCostUsd": 0,
            "durationSeconds": 5,
            "startedAt": "2026-10-14T16:59:39.811Z",
            "error": "exit 1: Reached maximum number of turns (3)",
            "tracePath": "/tmp/claude-eval-DcoUFt/out/trace.jsonl",
            "skippedPaidGraders": false,
            "graders": [
              {
                "name": "trigger",
                "passed": true,
                "weight": 1,
                "explanation": "Skill called 0x (expected 0..0)",
                "withOnly": false,
                "scored": true
              }
            ]
          }
        ]
      },
      "aggregates": {
        "score": 1,
        "passRate": 1
      }
    }
  ],
  "aggregates": {
    "casesTotal": 1,
    "casesPassed": 1,
    "overallScore": 1,
    "overallPassRate": 1
  }
}
Evidence: Prompts case 08 LangChain negative live result (3/3 silent)
{
  "schemaVersion": 1,
  "claudeVersion": "2.1.299",
  "startedAt": "2026-10-14T16:59:19.991Z",
  "durationSeconds": 5,
  "costUsd": 0.00565908,
  "partial": false,
  "suite": {
    "root": "~/.no-mistakes/evidence/01M4XMMB6AV00HWH6FEMDGTA0R/triggers-prompts/plugin",
    "ablation": "none",
    "caseFilter": "*08-neg-langchain*",
    "threshold": 0,
    "concurrency": 4,
    "plugins": [
      {
        "name": "mo-skills",
        "version": "0.0.0",
        "path": "~/.no-mistakes/evidence/01M4XMMB6AV00HWH6FEMDGTA0R/triggers-prompts/plugin"
      }
    ]
  },
  "cases": [
    {
      "name": "08-neg-langchain-system-prompt",
      "dir": "evals/metaobjects-prompts/08-neg-langchain-system-prompt",
      "source": "prose",
      "promptMarkdown": "This is a plain project; it does not use MetaObjects. It is a LangChain app. Tighten the wording of the system prompt in agent/prompts.py so the bot answers more briefly.",
      "runsPerCase": 3,
      "timeoutSeconds": 180,
      "maxTurns": 3,
      "graders": [
        {
          "name": "trigger",
          "type": "tool_used",
          "weight": 1,
          "config": {
            "tool": "Skill",
            "input_match": "metaobjects-prompts",
            "min": 0,
            "max": 0,
            "arm": "both"
          }
        }
      ],
      "arms": {
        "with": [
          {
            "score": 1,
            "passed": true,
            "turns": 4,
            "costUsd": 0.00184464,
            "judgeCostUsd": 0,
            "durationSeconds": 4,
            "startedAt": "2026-10-14T16:59:20.057Z",
            "error": "exit 1: Reached maximum number of turns (3)",
            "tracePath": "/tmp/claude-eval-pStdml/out/trace.jsonl",
            "skippedPaidGraders": false,
            "graders": [
              {
                "name": "trigger",
                "passed": true,
                "weight": 1,
                "explanation": "Skill called 0x (expected 0..0)",
                "withOnly": false,
                "scored": true
              }
            ]
          },
          {
            "score": 1,
            "passed": true,
            "turns": 4,
            "costUsd": 0.00190422,
            "judgeCostUsd": 0,
            "durationSeconds": 5,
            "startedAt": "2026-10-14T16:59:20.057Z",
            "error": "exit 1: Reached maximum number of turns (3)",
            "tracePath": "/tmp/claude-eval-bsle8F/out/trace.jsonl",
            "skippedPaidGraders": false,
            "graders": [
              {
                "name": "trigger",
                "passed": true,
                "weight": 1,
                "explanation": "Skill called 0x (expected 0..0)",
                "withOnly": false,
                "scored": true
              }
            ]
          },
          {
            "score": 1,
            "passed": true,
            "turns": 4,
            "costUsd": 0.00191022,
            "judgeCostUsd": 0,
            "durationSeconds": 5,
            "startedAt": "2026-10-14T16:59:20.057Z",
            "error": "exit 1: Reached maximum number of turns (3)",
            "tracePath": "/tmp/claude-eval-13qzL3/out/trace.jsonl",
            "skippedPaidGraders": false,
            "graders": [
              {
                "name": "trigger",
                "passed": true,
                "weight": 1,
                "explanation": "Skill called 0x (expected 0..0)",
                "withOnly": false,
                "scored": true
              }
            ]
          }
        ]
      },
      "aggregates": {
        "score": 1,
        "passRate": 1
      }
    }
  ],
  "aggregates": {
    "casesTotal": 1,
    "casesPassed": 1,
    "overallScore": 1,
    "overallPassRate": 1
  }
}
Evidence: Full server TypeScript bun test log (0 fail, non-live)
bun test v1.3.14 (0d9b296a)

packages/metadata/test/registry-coverage.test.ts:

[registry-coverage] subtypes: 53/65 exercised (81.5%), 12 UNTESTED
[registry-coverage] untested subtypes:
  attr.boolean
  attr.class
  attr.double
  attr.expression
  attr.filter
  attr.int
  attr.intMap
  attr.long
  validator.atLeastOne
  validator.comparison
  validator.presentIff
  validator.requiredWhen
[registry-coverage] exercised subtypes with untested attrs: 28

packages/codegen-ts/test/format.test.ts:
[codegen-ts] Biome formatter: 2 diagnostics; first: expected `,` but instead found `'x'`

packages/cli/test/migrate-drop-unmanaged.test.ts:
meta: migrate: refusing to drop public.theirs — absent from the committed schema snapshot, so this toolchain never managed it and the migration could not replay against a database where it never existed. Re-run with '--allow drop-unmanaged' if the drop is intended.
meta migrate — sqlite, file:/tmp/drop-unmanaged-tWclZe/t.db

  Changes: 1 drop-table

  Written:
    /tmp/drop-unmanaged-tWclZe/.metaobjects/migrations/20261014171257-x/up.sql
    /tmp/drop-unmanaged-tWclZe/.metaobjects/migrations/20261014171257-x/down.sql

meta migrate — sqlite, file:/tmp/drop-unmanaged-wbcCK5/t.db

  Changes: 1 drop-table

  Written:
    /tmp/drop-unmanaged-wbcCK5/.metaobjects/migrations/20261014171257-x/up.sql
    /tmp/drop-unmanaged-wbcCK5/.metaobjects/migrations/20261014171257-x/down.sql

meta migrate — sqlite, file:/tmp/drop-unmanaged-yKQcqz/t.db

  Changes: 1 drop-table

  Written:
    /tmp/drop-unmanaged-yKQcqz/.metaobjects/migrations/20261014171257-x/up.sql
    /tmp/drop-unmanaged-yKQcqz/.metaobjects/migrations/20261014171257-x/down.sql

meta: migrate: refusing to drop public.orders.orders_job_id_fk — absent from the committed schema snapshot, so this toolchain never managed it and the migration could not replay against a database where it never existed. Re-run with '--allow drop-unmanaged' if the drop is intended.
meta migrate — sqlite, file:/tmp/drop-unmanaged-fk-7WhMNE/t.db

  Changes: 1 drop-fk

  Written:
    /tmp/drop-unmanaged-fk-7WhMNE/.metaobjects/migrations/20261014171257-x/up.sql
    /tmp/drop-unmanaged-fk-7WhMNE/.metaobjects/migrations/20261014171257-x/down.sql

meta migrate — sqlite, file:/tmp/drop-unmanaged-fk-445VMH/t.db

  Changes: 1 drop-fk

  Written:
    /tmp/drop-unmanaged-fk-445VMH/.metaobjects/migrations/20261014171257-x/up.sql
    /tmp/drop-unmanaged-fk-445VMH/.metaobjects/migrations/20261014171257-x/down.sql


packages/cli/test/help-and-exit.test.ts:
meta gen — codegen TS targets from your declared metadata

USAGE:
  meta gen [<entity>...] [flags]

FLAGS:
  --dry-run             Compute and print, don't write
  --list                Print the generator CATALOG (name, layer, tier, what it emits,
                        what to install) and exit. Codegen is opt-in: nothing runs until
                        you wire it, and this is where you choose. No config or metadata
                        required — add --format json for the same catalog, machine-readable.
  --probe               With --list: construct every catalog generator and dry-run it
                        against YOUR model, reporting how many files each would emit.
                        A count per generator beats any category label — and it cannot go
                        stale, because it runs the generators. Needs a project.
  --baseline <default|adopt|fresh>
                        First-time-on-existing-file behavior. Default: refuse a file that
                        cannot be proved to be generated output. "adopt" records the files
                        you have as the merge base and writes NOTHING — the first run a
                        project with no committed .hashes.json can perform. "fresh"
                        overwrites them with fresh output and re-baselines (discards edits).
  --no-antipatterns     Suppress the advisory "hand-rolled what MetaObjects can model" pass
  --limit <n|all>       Advisory lines TEXT output prints before truncating (default 20)
  --format <toon|json|text>   Output format (global flag; default toon off-TTY)
  <entity> [<entity>]   Positional filter on entity names
  --help, -h            Print this help

A real write run (not --dry-run) also runs an ADVISORY anti-pattern pass: it scans
your authored source for hand-rolled aggregates, money-as-float, and CHECK-IN enums
and points you at the construct that models them (origin.aggregate / field.currency /
field.enum). Warnings only — it never fails the build. Opt out with --no-antipatterns
or META_NO_ANTIPATTERNS=1.

Text output caps that list at --limit lines (default 20) and says how many it held
back. --format toon / --format json carry EVERY finding in the payload's
antiPatterns block, uncapped — --limit is a display cap for a terminal, never a
limit on the machine-readable result.

NOTE: outDir, dialect, dbImport, extStyle are read from metaobjects.config.ts

meta migrate — diff metadata vs live DB; emit migration SQL files

USAGE:
  meta migrate [baseline|apply-pending] [flags]

SUBCOMMANDS:
  baseline             Snapshot an EXISTING database's schema as the reference point
                       (use with --from-db). NOTE: for a brand-new/empty database use
                       the greenfield example below, NOT baseline — an offline baseline
                       records your metadata as already-applied and emits no CREATE TABLE.
  apply-pending        Replay committed migration files against --db (no diff).
                       Provisions a fresh/CI database when the chain BUILDS the
                       schema — 'meta verify --replay' proves that it does. A
                       database adopted via 'baseline --from-db' has no such
                       chain. postgres/sqlite only.

MIGRATE FLAGS:
  --db <url>           DB connection URL (required for live-introspect / --apply / --rollback)
                       Supports: file:, libsql:, postgres:, postgresql:
  --dialect sqlite|postgres|d1
                       Optional dialect override (auto-detected from URL scheme)
  --migration-format default|flyway
                       Migration file layout (default: default). 'flyway' emits
                       V<N>__<slug>.sql + U<N>__<slug>.sql for a Flyway runner;
                       --apply/apply-pending/--rollback are refused under it
                       because Flyway owns apply and flyway_schema_history.
                       Also settable once as migrate.format in
                       .metaobjects/config.json.
  --out-dir <path>     Migration directory (default: ./.metaobjects/migrations;
                       flyway: src/main/resources/db/migration)
  --slug <name>        Required when changes are present (e.g., --slug add-user-shipping)
  --allow <csv>        Comma-separated destructive-change permissions:
                       drop-column,drop-table,type-change,drop-index,drop-fk,
                       drop-check,drop-view,drop-view-cascade,
                       adopt-view,nullable-to-not-null,drop-identity-default,
                       drop-unmanaged
                       drop-column DELETES the column and every value in it. If
                       the column was RENAMED, use --rename-column instead: it
                       keeps the data.
                       drop-unmanaged permits dropping a table/view the committed
                       snapshot never contained — one this toolchain never managed.
                       Without it that drop is refused, because the migration it
                       writes cannot replay against a database where the object
                       never existed.
  --on-ambiguous abort|rename|drop-add
                       How to handle ambiguous renames (default: abort)
  --rename-table [schema.]old=new
                       Declare a table rename (repeatable). Resolves the drop+create
                       as ALTER TABLE ... RENAME TO whatever the rename heuristic
                       makes of the names; refused if no such pair is pending.
  --rename-column [schema.]table.old=new
                       Declare a column

... [267453 bytes truncated] ...

                  # neutral model + API docs (on by default)
  6. Create your tables: meta migrate --from-db --db file:dev.sqlite --dialect sqlite --slug init --apply
     A new database later (a fresh clone, CI, tests) replays the committed migrations:
       meta migrate apply-pending --db file:other.sqlite

meta: no package.json found — create one with `npm init -y`, then set `"type": "module"`: MetaObjects scaffolds and generates ESM, which will not compile in a CommonJS project.
Refreshed the MetaObjects agent context in place (35 files): .metaobjects/AGENTS.md + .metaobjects/CLAUDE.md + .claude/skills/metaobjects-*.
Your metadata, .metaobjects/config.json, codegen/generators/ and metaobjects.config.ts were left untouched.

packages/cli/test/unit/reporting-inert.test.ts:
meta docs --site — wrote 7 page(s) → /tmp/reporting-inert-docs-8KxLYF/shop/out--site/site
meta docs --site — wrote 11 page(s) → /tmp/reporting-inert-docs-MZE6MM/shop/out--site/site
meta docs --site — wrote 7 page(s) → /tmp/reporting-inert-docs-FNHBeG/shop/out--site/site
meta docs --site — wrote 11 page(s) → /tmp/reporting-inert-docs-dxF0ZV/shop/out--site/site
meta docs --site — wrote 7 page(s) → /tmp/reporting-inert-docs-Z29em2/shop/out--site/site
meta docs --site — wrote 11 page(s) → /tmp/reporting-inert-docs-xqUdHr/shop/out--site/site

packages/cli/test/unit/migrate-d1.test.ts:
migrate: wrote /tmp/migrate-d1-cmd-6gC5X3/migrations/0001_init.sql
migrate: wrote /tmp/migrate-d1-cmd-6gC5X3/migrations/.down/0001_init.sql
  create-table: 1
migrate: wrote /tmp/migrate-d1-cmd-sVMCsc/migrations/0001_init-programs.sql
migrate: wrote /tmp/migrate-d1-cmd-sVMCsc/migrations/.down/0001_init-programs.sql
  create-table: 2
  add-fk: 1
  create-view: 1
meta: migrate: no wrangler.toml found in /tmp/migrate-d1-cmd-dIqQHl or parents; pass --d1 <binding> to bypass
meta: migrate: --db / DATABASE_URL is not used for dialect 'd1' — wrangler.toml owns connection

packages/cli/test/unit/verify-strict.test.ts:
meta verify — running --templates (default). Explicit subverbs: --templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift), --docs (docs drift), --deps (dependency drift), --replay/--replay-snapshot (the committed migration chain replays from empty).
meta: failed to load metadata: ERR_UNKNOWN_ATTR: Unknown attribute '@madeUpAttr' on field.string 'email' — not declared by any registered provider for field.string
  in meta.users.json at $['metadata.root'].children[0]['object.entity'].children[1]['field.string']
meta: meta verify is strict (ADR-0023): every authored @attr must be declared. Fix: register the attr on a metadata provider, OR move arbitrary author-supplied properties into an `attr.properties` bag, OR re-run with `meta verify --lax` to keep the legacy open-attr load.
meta verify — running --templates (default). Explicit subverbs: --templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift), --docs (docs drift), --deps (dependency drift), --replay/--replay-snapshot (the committed migration chain replays from empty).
meta verify — no template.* nodes found; the template gate had nothing to check.
meta verify — not run: schema (--db <url>), codegen (--codegen), docs (--docs), deps (--deps), replay (--replay) — a bare 'meta verify' runs only the template gate (plus the requirement ledger); name each gate to run it
meta verify — running --templates (default). Explicit subverbs: --templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift), --docs (docs drift), --deps (dependency drift), --replay/--replay-snapshot (the committed migration chain replays from empty).
meta verify — running --templates (default). Explicit subverbs: --templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift), --docs (docs drift), --deps (dependency drift), --replay/--replay-snapshot (the committed migration chain replays from empty).
meta verify — running --templates (default). Explicit subverbs: --templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift), --docs (docs drift), --deps (dependency drift), --replay/--replay-snapshot (the committed migration chain replays from empty).
meta verify — no template.* nodes found; the template gate had nothing to check.
meta verify — not run: schema (--db <url>), codegen (--codegen), docs (--docs), deps (--deps), replay (--replay) — a bare 'meta verify' runs only the template gate (plus the requirement ledger); name each gate to run it

packages/cli/test/unit/verify-output.test.ts:
meta verify — running --templates (default). Explicit subverbs: --templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift), --docs (docs drift), --deps (dependency drift), --replay/--replay-snapshot (the committed migration chain replays from empty).
meta verify — 1 template(s) clean.
meta verify — not run: schema (--db <url>), codegen (--codegen), docs (--docs), deps (--deps), replay (--replay) — a bare 'meta verify' runs only the template gate (plus the requirement ledger); name each gate to run it
meta verify — running --templates (default). Explicit subverbs: --templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift), --docs (docs drift), --deps (dependency drift), --replay/--replay-snapshot (the committed migration chain replays from empty).
meta: failed to load metadata: ERR_INVALID_TEMPLATE: template "Broken" @payloadRef "DoesNotExist" does not resolve to an object.value or sourceless object.projection at root
  in meta.ai.json at $['metadata.root'].children[0]['template.output']
meta verify — running --templates (default). Explicit subverbs: --templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift), --docs (docs drift), --deps (dependency drift), --replay/--replay-snapshot (the committed migration chain replays from empty).
meta verify — 1 template(s) clean.
meta verify — not run: schema (--db <url>), codegen (--codegen), docs (--docs), deps (--deps), replay (--replay) — a bare 'meta verify' runs only the template gate (plus the requirement ledger); name each gate to run it
meta verify — running --templates (default). Explicit subverbs: --templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift), --docs (docs drift), --deps (dependency drift), --replay/--replay-snapshot (the committed migration chain replays from empty).
meta: [Welcome] (email/html) ERR_VAR_NOT_ON_PAYLOAD: nonExistentField (prompts/e/html:1)
meta: meta verify — 1 drift error(s) across 1 template(s).
meta verify — not run: schema (--db <url>), codegen (--codegen), docs (--docs), deps (--deps), replay (--replay) — a bare 'meta verify' runs only the template gate (plus the requirement ledger); name each gate to run it
meta verify — running --templates (default). Explicit subverbs: --templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift), --docs (docs drift), --deps (dependency drift), --replay/--replay-snapshot (the committed migration chain replays from empty).
meta verify — 1 template(s) clean.
meta verify — not run: schema (--db <url>), codegen (--codegen), docs (--docs), deps (--deps), replay (--replay) — a bare 'meta verify' runs only the template gate (plus the requirement ledger); name each gate to run it
meta verify — running --templates (default). Explicit subverbs: --templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift), --docs (docs drift), --deps (dependency drift), --replay/--replay-snapshot (the committed migration chain replays from empty).
meta: [Doc] (document) ERR_VAR_NOT_ON_PAYLOAD: missingField (prompts/o/x:1)
meta: meta verify — 1 drift error(s) across 1 template(s).
meta verify — not run: schema (--db <url>), codegen (--codegen), docs (--docs), deps (--deps), replay (--replay) — a bare 'meta verify' runs only the template gate (plus the requirement ledger); name each gate to run it

 10976 pass
 50 skip
 0 fail
 4 snapshots, 37459 expect() calls
Ran 11026 tests across 1046 files. [886.24s]
  • Outcome: ⚠️ 1 info across 1 run (33m24s)

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

⚠️ **Test** - 1 info
  • ℹ️ agent-context/skills/metaobjects-codegen/SKILL.md:3 - Case 06-pos-dialect-config fired metaobjects-codegen 2 of 3 live runs (0 of 6 before the description tune). Improved, not solid. Operator should rerun the codegen cases to confirm the rate.
  • Live validation: ✅ go - 4 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Dialect/generator config request (postgres to sqlite) loads metaobjects-codegen skill ✅ pass live triggers/result.json (case 06-pos-dialect-config, 2 of 3 runs fired Skill)
New adopter project with zero template.prompt nodes asks to move f-string LLM prompt onto MetaObjects prompts loads metaobjects-prompts skill ✅ pass live triggers-prompts/adopt13/result.json (case 13-pos-adopt-first-prompt, 3 of 3 fired)
Non-MetaObjects project editing f-string LLM prompt does not load metaobjects-prompts skill (over-trigger guard) ✅ pass live triggers-prompts/neg07/result.json (case 07-neg-fstring-no-metaobjects, 0 Skill calls in 3 of 3 runs)
LangChain system-prompt edit in non-MetaObjects project does not load metaobjects-prompts skill ✅ pass live triggers-prompts/result.json (case 08-neg-langchain-system-prompt, 3 of 3 passed as non-trigger)
Full repository regression suite (server TypeScript bun test) passes after the change ⏸️ untested no Non-live regression run only: the bun test log reports 11026 tests, 10976 pass, 50 skip, 0 fail. That is not a live product run, so it is not counted as a pass for this scenario.
  • scripts/ci-local.sh --only ts-fast --only ts-unit --strict-toolchains
  • git diff --stat 66056aef..57fa8b1b (scope check)
  • per-skill description md5 check: source SKILL.md vs all 5 fixtures/agent-context-conformance mirrors, codegen and prompts, identical
  • agent-context/evals/run-skill-triggers.sh <evidence>/triggers --case '*06-pos-dialect-config*' (3 live sessions)
  • agent-context/evals/run-skill-triggers.sh <evidence>/triggers-prompts/adopt13 --case '*13-pos-adopt*' (3 live sessions)
  • agent-context/evals/run-skill-triggers.sh <evidence>/triggers-prompts/neg07 --case '*07-neg-fstring*' (3 live sessions)
  • agent-context/evals/run-skill-triggers.sh <evidence>/triggers-prompts --case '*08-neg-langchain*' (3 live sessions)
  • cd server/typescript && bun test (non-live: 11026 tests, 10976 pass, 50 skip, 0 fail)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

…kill descriptions

Add should-trigger and should-not-trigger cases (12 each) for metaobjects-prompts and
metaobjects-codegen, a runner that wraps the skills in a throwaway plugin for
claude plugin eval, and a CI test that the cases parse and name existing skills.

First run: codegen 36/36, prompts 32/36 (one run of an entities-only near-miss
fired the skill). Narrow the prompts description to projects that declare
template.prompt; codegen needed no change. Rerun: 36/36 and 36/36.
…uation is manual-only

A test that only parses the case files proves nothing about trigger behaviour, so the cases
and the documented run command are the evidence. README no longer claims a CI check.
@dmealing
dmealing merged commit 5230ec3 into main Oct 14, 2026
1 check passed
@dmealing
dmealing deleted the fm/mo-skills-audit-trigger-evals branch October 14, 2026 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant