Repository navigation
fix(agent-context): tune prompts and codegen skill triggers with eval set - #438
Merged
Merged
Conversation
…kill descriptions Add should-trigger and should-not-trigger cases (12 each) for metaobjects-prompts and metaobjects-codegen, a runner that wraps the skills in a throwaway plugin for claude plugin eval, and a CI test that the cases parse and name existing skills. First run: codegen 36/36, prompts 32/36 (one run of an entities-only near-miss fired the skill). Narrow the prompts description to projects that declare template.prompt; codegen needed no change. Rerun: 36/36 and 36/36.
…uation is manual-only A test that only parses the case files proves nothing about trigger behaviour, so the cases and the documented run command are the evidence. README no longer claims a CI check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Changed
metaobjects-promptsskill description so it loads only for LLM-text code in projects that declaretemplate.prompt, not for every prompt edit. The same description is also shipped into adopter.claude/skillsthrough the agent-context conformance fixtures, which are regenerated to match.metaobjects-codegendescription to name the concrete trigger (editing the gen config inmetaobjects.config.tsto choose generators, targets or the SQL dialect). The existing "ANY output derived from the MetaObjects model" wording is kept: the eval showed no over-trigger from it (every negative case stayed silent in every pass).agent-context/evals/skill-triggers/(25 should-trigger and should-not-trigger cases for the two skills) withagent-context/evals/run-skill-triggers.shto run it manually against a real Claude login, plus a README describing how to run and re-tune it.Eval results (per pass)
Pass rates (runs passed / runs; 3 runs per case; Claude login, single arm; the case count is per skill):
Case 05 rewording: from "
meta verify --templatesreports prompt drift on the ReviewDraft prompt after I renamed a field. Explain how to fix it." to "I renamed a field on the payload projection of the ReviewDraft template.prompt. Update the prompt template and its payload so rendering stays in sync." Original wording named the verify command, which metaobjects-verify owns, so loading verify was a defensible answer; bad test, not a real miss.Case 06 (dialect config): passed 3/3 in passes 1 and 2 against the old codegen description, then the pipeline's own test run scored 0/3 twice. Case file unchanged. Cause not established: run variance or a difference in the pipeline's test environment (its agent runs through a different provider router); the codegen description was tuned anyway (names editing the gen config / dialect) and case 06 is 3/3 in pass 4.
Eval passes: 4 on the Claude login (over the usual three because the codegen rerun was added); pass 4 ran all 25 cases because --case matches names, which repeat across the two skills.
Validation runs: run 1 failed in the Test step (fix agent: all providers failed); this is run 2.
The codegen-case-06 line of the Testing section below (2 of 3) is the pipeline's own spot check; the full rerun in pass 4 above is the record. The parse-only CI test was dropped at the operator's decision, so the evaluation is manual-only (
agent-context/evals/run-skill-triggers.sh).Validation
Risk Assessment
✅ Low: The change adds manual-only eval cases and a runner script, and edits two skill description lines. I confirmed that all 10 fixture mirrors match the source descriptions, and that the 13 prompts cases and 12 codegen cases exist. The README counts, the removal of the source-grepping SDK test and the recorded user decisions are all honored.
Testing
Drove the two changed skill descriptions through real Claude sessions on the subscription login (about 13 agent sessions, a few minutes of quota, no metered spend). Codegen dialect-config case 06 fired the skill 2 of 3 runs, up from 0 of 6 before the tune; the operator should rerun codegen cases to confirm the rate. New adopt-first-prompt case 13 fired 3 of 3. Over-trigger negatives 07 (f-string, no MetaObjects) and 08 (LangChain system prompt) stayed silent 3 of 3 each. Description text matches across the source skill and all five conformance mirrors. The full server TypeScript suite ran with 0 failures, but that run is non-live and is recorded as untested, not as a live pass. Worktree clean; evidence under the evidence directory only.
Evidence: Codegen case 06 live trigger result (2/3 fired)
Evidence: Codegen case 06 run log
Evidence: Prompts case 13 adopt-first-prompt live result (3/3 fired)
Evidence: Prompts case 07 f-string negative live result (3/3 silent)
Evidence: Prompts case 08 LangChain negative live result (3/3 silent)
Evidence: Full server TypeScript bun test log (0 fail, non-live)
Pipeline
Updates from git push no-mistakes
✅ **Intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
agent-context/skills/metaobjects-codegen/SKILL.md:3- Case 06-pos-dialect-config fired metaobjects-codegen 2 of 3 live runs (0 of 6 before the description tune). Improved, not solid. Operator should rerun the codegen cases to confirm the rate.scripts/ci-local.sh --only ts-fast --only ts-unit --strict-toolchainsgit diff --stat 66056aef..57fa8b1b (scope check)per-skill description md5 check: source SKILL.md vs all 5 fixtures/agent-context-conformance mirrors, codegen and prompts, identicalagent-context/evals/run-skill-triggers.sh <evidence>/triggers --case '*06-pos-dialect-config*' (3 live sessions)agent-context/evals/run-skill-triggers.sh <evidence>/triggers-prompts/adopt13 --case '*13-pos-adopt*' (3 live sessions)agent-context/evals/run-skill-triggers.sh <evidence>/triggers-prompts/neg07 --case '*07-neg-fstring*' (3 live sessions)agent-context/evals/run-skill-triggers.sh <evidence>/triggers-prompts --case '*08-neg-langchain*' (3 live sessions)cd server/typescript && bun test (non-live: 11026 tests, 10976 pass, 50 skip, 0 fail)✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.