Skip to content

test(examples): run each example file against the mocks - #412

Open
glebedel wants to merge 4 commits into
mainfrom
s2d2/audit-execute-real-examples
Open

glebedel wants to merge 4 commits into
mainfrom
s2d2/audit-execute-real-examples

Conversation

@glebedel

@glebedel glebedel commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Supersedes #381 by @willleeney, rebased on current main and finished. Will's commit is kept as the first commit and credited as co-author on the rework.

Summary

  • Each example test now imports the real file from examples/ and runs it, instead of re-implementing what the example does. tests/examples/run-example.ts points the example at the mocks with STACKONE_BASE_URL, captures console.log and console.error, and turns process.exit into a thrown error so the vitest worker survives.
  • The OpenAI (Chat Completions and Responses) and Anthropic mocks run a real tool round: the model asks for workday_list_workers, the example executes it against the MCP mock, and the model answers once the tool result comes back.
  • vitest.setup.ts fails any request to a host other than 127.0.0.1 that no handler serves. Before this, MSW only warned, and running the Anthropic example from a test reached the real api.anthropic.com (it got a 401 back with the dummy key).
  • Merging main into test(examples): execute the real example files #381: the conflicts in mocks/handlers.ts and the claude-agent-sdk, openai and openai-responses tests resolve to main. The tests for the defender-config and search-tools examples are dropped because main deleted those examples. The claude-agent-sdk test stays as it is on main, because query() there spawns a subprocess.

Test plan

  • Each test asserts the example's exact output and its skip path (Skipping: X_API_KEY is not set, exit 0), and checks which tools reached the MCP mock with which arguments.
  • auth-management serves a discovered account that has workday_list_workers, so section 5 runs setAccountId / getAccountId instead of its fallback. Its full output is pinned with an inline snapshot.
  • Mutation check: with each of the openai, ai-sdk and anthropic examples given a filter that drops the tool, the matching test fails.
  • With the Anthropic handler removed, the test fails with MSW's unhandled-request error instead of reaching the network.
  • Locally on the head commit (Node 24.22.3, pnpm 11): pnpm lint is clean. pnpm test passes 50 files, 1332 tests (4 skipped as on main) with no type errors. The examples project passes 9 files and 18 tests. pnpm build and publint are OK.

Review gates

Gate Result
Forge (loop and slop meter) Comment lines 47 to 47, longest block 6, copied blocks 2 to 1 (the account-serving helper now lives in mocks/node.ts and the toolset tests use it). The one copy left is 3 lines of MCP mock wiring that also appear in src/toolsets.test.ts, outside this change.
Security review (Claude security reviewer, standing in for the Codex pass, which was unavailable) No findings. No real credentials (only test-key style placeholders). The dynamic import takes a hardcoded example name at every call site.
Code review (Claude test reviewer) Six findings. Fixed: the auth-management test only covered the fallback branch, the Anthropic mock ignored text blocks, two test titles claimed more than they asserted, and the stubExampleEnv doc was wrong. Left as they are: a second, event-based unhandled-request check (the setup guard already fails the test), and the single tool round in the mock (all the examples make one).
Aikido Runs on this PR.

Requested by guillaume@stackone.com in Slack (repo audit thread).

🤖 Generated with Claude Code


Summary by cubic

Example tests now run the real files from examples/ instead of re-implementing their logic, so a broken example fails CI instead of staying green.

  • tests/examples/run-example.ts imports each example, points it at the mocks, captures console output, and turns process.exit into an assertable exit code.
  • The OpenAI and Anthropic mocks run a real tool round: the model asks for workday_list_workers, the example executes it against the MCP mock, and the model answers once the tool result is back.
  • vitest.setup.ts fails any unmocked request to a remote host; previously the Anthropic test reached the real api.anthropic.com.
  • Added the workday_* tools to the MCP catalog, which the examples filter on but were missing, so the filters matched zero tools.
  • The claude-agent-sdk example is excluded because query() spawns a subprocess MSW cannot intercept.

Written for commit 0450545. Summary will update on new commits.

View guided diff Turn on auto-fix

willleeney and others added 4 commits September 14, 2026 08:38
`tests/examples/` reimplemented each example's logic against `../../src` and
never imported `examples/*.ts`. An example could rot or break outright and CI
stayed green — and four of the eight had no test at all.

Examples do their work at import time via top-level await, so importing one runs
it. `tests/examples/run-example.ts` makes that safe under vitest:

- `STACKONE_BASE_URL` points the SDK at the MSW mock (toolsets.ts:630 reads it),
  so an example constructing `new StackOneToolSet()` with no explicit baseUrl
  still hits the mock rather than production. No example needed changing.
- `process.exit` becomes a throw that `runExample` catches and reports as
  `exitCode`, so an example bailing on missing config is an assertable outcome
  instead of a torn-down vitest worker.

Seven of the eight examples now execute for real. The eighth,
claude-agent-sdk-integration, still cannot: `query()` spawns a claude-code
subprocess that MSW cannot intercept. Its existing setup-level test is kept and
the exclusion is now stated in the file rather than implied.

Mock gaps that blocked execution, all of which made the examples exercise
nothing:

- The three `workday_*` actions every example filters on were absent from the
  MCP catalog, so the filters matched zero tools.
- No Anthropic handler existed at all (mocks/handlers.anthropic.ts).
- The OpenAI handlers keyed off prompts none of the examples send; the examples
  all prompt "List the first 5 employees".
- Semantic search (`/actions/search`) was unmocked, so search-tools hit the
  real network. Added scoped to that test via `server.use` — a global handler
  pre-empts src/semantic-search.test.ts, which asserts against an unmocked
  endpoint.

Tool-count assertions are exact rather than `> 0`. A `> 0` check cannot fail:
`fetchTools` appends tool_feedback *after* filtering, so a filter matching
nothing still returns one tool. Verified by mutation — pointing an example at a
nonexistent action now fails with `expected 1 to be 4`, where before it passed.

defender-config keeps all seven original wire-payload tests, which assert the
exact RPC body per defender mode; the four execution tests are added alongside.

fetch-tools and tanstack-ai-integration describe examples that do not exist.
Relabelled in place rather than moved — the `examples` vitest project globs only
`tests/examples/**` and `root` globs only `src/`+`scripts/`, so relocating them
would have dropped them from both projects silently.

348 passing, up from 329. The one failure is pre-existing and environmental
(`spawn tsc ENOENT`); it fails identically on origin/main.
Conflicts resolved to main's side: mocks/handlers.ts and the
claude-agent-sdk, openai and openai-responses example tests. The
defender-config and search-tools tests go with the examples main removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Guillaume Lebedel <10488548+glebedel@users.noreply.github.com>
Builds on the original approach (run the real files, not a copy of their
logic) for the examples on main today:

- tests/examples/run-example.ts runs examples/<name>.ts, captures what it
  prints and turns process.exit into an exit code; recordToolCalls()
  serves the mock catalog and records each tools/call
- auth-management, openai, openai-responses, ai-sdk and anthropic each run
  for real, asserting their exact output and, where the example executes a
  tool, the tools/call that reached the mock; each skip path is covered
- the openai and openai-responses tests no longer reimplement the example
- search-and-execute uses the shared helpers
- mock models answer the prompt the examples send, return a tool call,
  then answer once the tool result is back; adds the Anthropic handler
- vitest.setup.ts fails any unmocked request to a remote host, so an
  example can no longer reach a real API from the test suite (it did:
  the Anthropic example called api.anthropic.com)

Co-Authored-By: Will Leeney <will.leeney@outlook.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Guillaume Lebedel <10488548+glebedel@users.noreply.github.com>
…del turn

- auth-management: serve a discovered account that has workday_list_workers, so
  section 5 runs setAccountId/getAccountId instead of its fallback, and pin the
  whole output with an inline snapshot
- Anthropic and OpenAI mocks share one nextTurn, and read text blocks as well as
  input_text parts
- serveAccounts moves to mocks/node so the example and toolset tests share it
- reword the stubExampleEnv doc: an empty value is set, and the examples treat it
  as unset

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Guillaume Lebedel <10488548+glebedel@users.noreply.github.com>
@glebedel
glebedel requested a review from a team as a code owner November 13, 2026 12:03
@glebedel glebedel self-assigned this Nov 13, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-13T12:04:47.752373Z 0450545 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@pkg-pr-new

pkg-pr-new Bot commented Nov 13, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/StackOneHQ/stackone-ai-node/@stackone/ai@412

commit: 0450545

Comment thread vitest.setup.ts Resolved
@glebedel

Copy link
Copy Markdown
Contributor Author

Forge meter: comment lines 47 added, longest block 6, type escapes 0, copies 1 (recordToolCalls in run-example.ts vs a test-local serveTool in src/toolsets.test.ts; they differ in account tool wiring and sit at two copies, so no extraction yet). No rewrite warranted on head 0450545. vitest: 50 files, 1327 passed, 5 skipped, no type errors.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants