fix(agent-context): tune prompts and codegen skill triggers with eval set

agent-context/evals/README.md

@@ -0,0 +1,26 @@
+# Skill trigger evaluations
+
+Should-trigger and should-not-trigger cases for the `description` of two shipped skills,
+`metaobjects-prompts` and `metaobjects-codegen`. A description decides when a skill's whole
+body is loaded, so an over-eager one costs tokens on every unrelated edit. Each case is a
+one-message prompt plus a `tool_used: Skill` grader: positive cases require the skill to
+fire, negative cases (near-misses: LLM code in a project that does not use MetaObjects
+prompts, model edits that need no generated output) forbid it.
+
+- `skill-triggers/<skill>/<NN>-pos-*|neg-*/` — `prompt.md` and `graders/trigger.md`, in the
+  `claude plugin eval` case format.
+- `run-skill-triggers.sh` — wraps `agent-context/skills` in a throwaway plugin (all seven
+  skills, so routing competes realistically) and runs the cases, single-arm.
+
+## Running it (manual)
+
+```
+agent-context/evals/run-skill-triggers.sh <out-dir>
+```
+
+It starts real Claude sessions on your own login (25 cases x 3 runs = 75 sessions, a few
+minutes with `-j 4`) and writes `<out-dir>/result.json`. Use `--case '<glob>'` to run a
+subset. It is manual-only and not part of CI, because model behaviour is not deterministic.
+
+Re-run it after editing either description. A `Reached maximum number of turns` note on a
+case is expected: the runs are capped at 3 turns because only the first tool call matters.

agent-context/evals/run-skill-triggers.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+# Run the skill trigger evaluation (agent-context/evals/skill-triggers) with `claude plugin eval`.
+# Manual, not CI: it drives real model sessions on your own Claude login (subscription quota).
+#
+# usage: agent-context/evals/run-skill-triggers.sh [OUT_DIR] [extra `claude plugin eval` flags]
+#   OUT_DIR defaults to a fresh temp dir; the scored result is written to OUT_DIR/result.json.
+#   e.g. agent-context/evals/run-skill-triggers.sh /tmp/triggers --case 'metaobjects-codegen/*'
+set -euo pipefail
+here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+skills="$here/../skills"
+out="${1:-$(mktemp -d)}"; [ $# -gt 0 ] && shift
+mkdir -p "$out"
+plugin="$out/plugin"
+rm -rf "$plugin"
+mkdir -p "$plugin/.claude-plugin"
+# `claude plugin eval` refuses an evals/ link that points outside the plugin, so copy.
+echo '{"name":"mo-skills","version":"0.0.0","description":"MetaObjects skills, wrapped for trigger evaluation"}' >"$plugin/.claude-plugin/plugin.json"
+cp -R "$skills" "$plugin/skills"
+cp -R "$here/skill-triggers" "$plugin/evals"
+cd "$plugin"
+# --ablation none: one arm, so the `tool_used: Skill` graders count toward the score.
+exec claude plugin eval . --ablation none --no-publish --trust-plugin -j 4 --threshold 0 \
+  --output-dir "$out/runs" --json "$out/result.json" "$@"

agent-context/evals/skill-triggers/metaobjects-codegen/01-pos-write-openapi-generator/graders/trigger.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-codegen
+min: 1
+---

agent-context/evals/skill-triggers/metaobjects-codegen/01-pos-write-openapi-generator/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). I need an OpenAPI document generated from the model. Write a generator for it.

agent-context/evals/skill-triggers/metaobjects-codegen/02-pos-eject-reference-generator/graders/trigger.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-codegen
+min: 1
+---

agent-context/evals/skill-triggers/metaobjects-codegen/02-pos-eject-reference-generator/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). The reference Zod generator does not emit what I need. Eject it so I can own and change it.

agent-context/evals/skill-triggers/metaobjects-codegen/03-pos-configure-generators/graders/trigger.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-codegen
+min: 1
+---

agent-context/evals/skill-triggers/metaobjects-codegen/03-pos-configure-generators/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). Configure which generators run and where `meta gen` writes each target in metaobjects.config.ts.

agent-context/evals/skill-triggers/metaobjects-codegen/04-pos-regen-hand-edits/graders/trigger.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-codegen
+min: 1
+---

agent-context/evals/skill-triggers/metaobjects-codegen/04-pos-regen-hand-edits/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). I hand-edited a generated entity file and now `meta gen` reports a merge conflict. How do I keep my edit?

agent-context/evals/skill-triggers/metaobjects-codegen/05-pos-json-schema-output/graders/trigger.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-codegen
+min: 1
+---

agent-context/evals/skill-triggers/metaobjects-codegen/05-pos-json-schema-output/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). Our frontend team needs JSON Schema files for every entity, derived from the model and kept current.

agent-context/evals/skill-triggers/metaobjects-codegen/06-pos-dialect-config/graders/trigger.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-codegen
+min: 1
+---

agent-context/evals/skill-triggers/metaobjects-codegen/06-pos-dialect-config/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). Change the dialect that code generation uses from postgres to sqlite in the gen configuration.

agent-context/evals/skill-triggers/metaobjects-codegen/07-neg-add-entity-field/graders/trigger.md

@@ -0,0 +1,8 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-codegen
+min: 0
+max: 0
+arm: both
+---

agent-context/evals/skill-triggers/metaobjects-codegen/07-neg-add-entity-field/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). Add a nullable `nickname` string field to the User entity. Do not generate anything yet.

agent-context/evals/skill-triggers/metaobjects-codegen/08-neg-verify-failure/graders/trigger.md

@@ -0,0 +1,8 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-codegen
+min: 0
+max: 0
+arm: both
+---

agent-context/evals/skill-triggers/metaobjects-codegen/08-neg-verify-failure/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). `meta verify --db` reports the live schema drifted from the metadata. Explain what that means.

agent-context/evals/skill-triggers/metaobjects-codegen/09-neg-openapi-no-metaobjects/graders/trigger.md

@@ -0,0 +1,8 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-codegen
+min: 0
+max: 0
+arm: both
+---

agent-context/evals/skill-triggers/metaobjects-codegen/09-neg-openapi-no-metaobjects/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This is a plain project; it does not use MetaObjects. Generate an OpenAPI document from our FastAPI routes.

agent-context/evals/skill-triggers/metaobjects-codegen/10-neg-prisma-codegen/graders/trigger.md

@@ -0,0 +1,8 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-codegen
+min: 0
+max: 0
+arm: both
+---

agent-context/evals/skill-triggers/metaobjects-codegen/10-neg-prisma-codegen/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This is a plain project; it does not use MetaObjects. Our `prisma generate` step fails in CI. Fix the CI job.

agent-context/evals/skill-triggers/metaobjects-codegen/11-neg-prompt-template/graders/trigger.md

@@ -0,0 +1,8 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-codegen
+min: 0
+max: 0
+arm: both
+---

agent-context/evals/skill-triggers/metaobjects-codegen/11-neg-prompt-template/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). Declare a template.prompt for summarizing tickets with a typed payload.

agent-context/evals/skill-triggers/metaobjects-codegen/12-neg-runtime-query/graders/trigger.md

@@ -0,0 +1,8 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-codegen
+min: 0
+max: 0
+arm: both
+---

agent-context/evals/skill-triggers/metaobjects-codegen/12-neg-runtime-query/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). Use the ObjectManager at runtime to list all active users, sorted by name.

agent-context/evals/skill-triggers/metaobjects-prompts/01-pos-fstring-prompt/graders/trigger.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-prompts
+min: 1
+---

agent-context/evals/skill-triggers/metaobjects-prompts/01-pos-fstring-prompt/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). It declares a template.prompt named SummarizeTicket. In src/summarize.py I build the LLM request with an f-string: f"Summarize this ticket: {ticket.title}\n{ticket.body}". Add the customer's plan tier to what the model sees.

agent-context/evals/skill-triggers/metaobjects-prompts/02-pos-declare-template-prompt/graders/trigger.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-prompts
+min: 1
+---

agent-context/evals/skill-triggers/metaobjects-prompts/02-pos-declare-template-prompt/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). Declare a new template.prompt for classifying support tickets, with a typed payload projection and a response model.

agent-context/evals/skill-triggers/metaobjects-prompts/03-pos-regex-reply-parse/graders/trigger.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-prompts
+min: 1
+---

agent-context/evals/skill-triggers/metaobjects-prompts/03-pos-regex-reply-parse/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). It declares a template.prompt whose reply we read. src/triage.ts reads the model's answer with a regex like /Category: (\w+)/. The regex breaks when the model adds a preamble. Fix the parsing.

agent-context/evals/skill-triggers/metaobjects-prompts/04-pos-test-harness-prompt/graders/trigger.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-prompts
+min: 1
+---

agent-context/evals/skill-triggers/metaobjects-prompts/04-pos-test-harness-prompt/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). It declares template.prompt nodes. A test script in tests/eval_harness.py concatenates strings to build the text it sends to the LLM. Add one more instruction line to that text.

agent-context/evals/skill-triggers/metaobjects-prompts/05-pos-prompt-drift-verify/graders/trigger.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-prompts
+min: 1
+---

agent-context/evals/skill-triggers/metaobjects-prompts/05-pos-prompt-drift-verify/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). I renamed a field on the payload projection of the ReviewDraft template.prompt. Update the prompt template and its payload so rendering stays in sync.

agent-context/evals/skill-triggers/metaobjects-prompts/06-pos-review-json-parse/graders/trigger.md

@@ -0,0 +1,6 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-prompts
+min: 1
+---

agent-context/evals/skill-triggers/metaobjects-prompts/06-pos-review-json-parse/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This project uses MetaObjects (it has .metaobjects/config.json and a metaobjects/ directory of meta.*.json files). It declares template.prompt nodes. Review this diff: it adds JSON.parse(completion.text) in src/agent.ts to read an LLM reply.

agent-context/evals/skill-triggers/metaobjects-prompts/07-neg-fstring-no-metaobjects/graders/trigger.md

@@ -0,0 +1,8 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-prompts
+min: 0
+max: 0
+arm: both
+---

agent-context/evals/skill-triggers/metaobjects-prompts/07-neg-fstring-no-metaobjects/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This is a plain project; it does not use MetaObjects. In src/summarize.py I build an OpenAI request with an f-string: f"Summarize this ticket: {ticket.title}". Add the customer's plan tier to the text.

agent-context/evals/skill-triggers/metaobjects-prompts/08-neg-langchain-system-prompt/graders/trigger.md

@@ -0,0 +1,8 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-prompts
+min: 0
+max: 0
+arm: both
+---

agent-context/evals/skill-triggers/metaobjects-prompts/08-neg-langchain-system-prompt/prompt.md

@@ -0,0 +1,8 @@
+---
+max_turns: 3
+timeout_seconds: 180
+allowed_tools: [Skill]
+runs: 3
+---
+
+This is a plain project; it does not use MetaObjects. It is a LangChain app. Tighten the wording of the system prompt in agent/prompts.py so the bot answers more briefly.

agent-context/evals/skill-triggers/metaobjects-prompts/09-neg-json-reply-no-metaobjects/graders/trigger.md

@@ -0,0 +1,8 @@
+---
+type: tool_used
+tool: Skill
+input_match: metaobjects-prompts
+min: 0
+max: 0
+arm: both
+---