Repository navigation
Conversation
`tests/examples/` reimplemented each example's logic against `../../src` and never imported `examples/*.ts`. An example could rot or break outright and CI stayed green — and four of the eight had no test at all. Examples do their work at import time via top-level await, so importing one runs it. `tests/examples/run-example.ts` makes that safe under vitest: - `STACKONE_BASE_URL` points the SDK at the MSW mock (toolsets.ts:630 reads it), so an example constructing `new StackOneToolSet()` with no explicit baseUrl still hits the mock rather than production. No example needed changing. - `process.exit` becomes a throw that `runExample` catches and reports as `exitCode`, so an example bailing on missing config is an assertable outcome instead of a torn-down vitest worker. Seven of the eight examples now execute for real. The eighth, claude-agent-sdk-integration, still cannot: `query()` spawns a claude-code subprocess that MSW cannot intercept. Its existing setup-level test is kept and the exclusion is now stated in the file rather than implied. Mock gaps that blocked execution, all of which made the examples exercise nothing: - The three `workday_*` actions every example filters on were absent from the MCP catalog, so the filters matched zero tools. - No Anthropic handler existed at all (mocks/handlers.anthropic.ts). - The OpenAI handlers keyed off prompts none of the examples send; the examples all prompt "List the first 5 employees". - Semantic search (`/actions/search`) was unmocked, so search-tools hit the real network. Added scoped to that test via `server.use` — a global handler pre-empts src/semantic-search.test.ts, which asserts against an unmocked endpoint. Tool-count assertions are exact rather than `> 0`. A `> 0` check cannot fail: `fetchTools` appends tool_feedback *after* filtering, so a filter matching nothing still returns one tool. Verified by mutation — pointing an example at a nonexistent action now fails with `expected 1 to be 4`, where before it passed. defender-config keeps all seven original wire-payload tests, which assert the exact RPC body per defender mode; the four execution tests are added alongside. fetch-tools and tanstack-ai-integration describe examples that do not exist. Relabelled in place rather than moved — the `examples` vitest project globs only `tests/examples/**` and `root` globs only `src/`+`scripts/`, so relocating them would have dropped them from both projects silently. 348 passing, up from 329. The one failure is pre-existing and environmental (`spawn tsc ENOENT`); it fails identically on origin/main.
Conflicts resolved to main's side: mocks/handlers.ts and the claude-agent-sdk, openai and openai-responses example tests. The defender-config and search-tools tests go with the examples main removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Guillaume Lebedel <10488548+glebedel@users.noreply.github.com>
Builds on the original approach (run the real files, not a copy of their logic) for the examples on main today: - tests/examples/run-example.ts runs examples/<name>.ts, captures what it prints and turns process.exit into an exit code; recordToolCalls() serves the mock catalog and records each tools/call - auth-management, openai, openai-responses, ai-sdk and anthropic each run for real, asserting their exact output and, where the example executes a tool, the tools/call that reached the mock; each skip path is covered - the openai and openai-responses tests no longer reimplement the example - search-and-execute uses the shared helpers - mock models answer the prompt the examples send, return a tool call, then answer once the tool result is back; adds the Anthropic handler - vitest.setup.ts fails any unmocked request to a remote host, so an example can no longer reach a real API from the test suite (it did: the Anthropic example called api.anthropic.com) Co-Authored-By: Will Leeney <will.leeney@outlook.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Guillaume Lebedel <10488548+glebedel@users.noreply.github.com>
…del turn - auth-management: serve a discovered account that has workday_list_workers, so section 5 runs setAccountId/getAccountId instead of its fallback, and pin the whole output with an inline snapshot - Anthropic and OpenAI mocks share one nextTurn, and read text blocks as well as input_text parts - serveAccounts moves to mocks/node so the example and toolset tests share it - reword the stubExampleEnv doc: an empty value is set, and the examples treat it as unset Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Guillaume Lebedel <10488548+glebedel@users.noreply.github.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
commit: |
Contributor
Author
|
Forge meter: comment lines 47 added, longest block 6, type escapes 0, copies 1 (recordToolCalls in run-example.ts vs a test-local serveTool in src/toolsets.test.ts; they differ in account tool wiring and sit at two copies, so no extraction yet). No rewrite warranted on head 0450545. vitest: 50 files, 1327 passed, 5 skipped, no type errors. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #381 by @willleeney, rebased on current
mainand finished. Will's commit is kept as the first commit and credited as co-author on the rework.Summary
examples/and runs it, instead of re-implementing what the example does.tests/examples/run-example.tspoints the example at the mocks withSTACKONE_BASE_URL, capturesconsole.logandconsole.error, and turnsprocess.exitinto a thrown error so the vitest worker survives.workday_list_workers, the example executes it against the MCP mock, and the model answers once the tool result comes back.vitest.setup.tsfails any request to a host other than127.0.0.1that no handler serves. Before this, MSW only warned, and running the Anthropic example from a test reached the realapi.anthropic.com(it got a 401 back with the dummy key).maininto test(examples): execute the real example files #381: the conflicts inmocks/handlers.tsand the claude-agent-sdk, openai and openai-responses tests resolve tomain. The tests for the defender-config and search-tools examples are dropped becausemaindeleted those examples. The claude-agent-sdk test stays as it is onmain, becausequery()there spawns a subprocess.Test plan
Skipping: X_API_KEY is not set, exit 0), and checks which tools reached the MCP mock with which arguments.auth-managementserves a discovered account that hasworkday_list_workers, so section 5 runssetAccountId/getAccountIdinstead of its fallback. Its full output is pinned with an inline snapshot.pnpm lintis clean.pnpm testpasses 50 files, 1332 tests (4 skipped as onmain) with no type errors. The examples project passes 9 files and 18 tests.pnpm buildand publint are OK.Review gates
mocks/node.tsand the toolset tests use it). The one copy left is 3 lines of MCP mock wiring that also appear insrc/toolsets.test.ts, outside this change.test-keystyle placeholders). The dynamic import takes a hardcoded example name at every call site.stubExampleEnvdoc was wrong. Left as they are: a second, event-based unhandled-request check (the setup guard already fails the test), and the single tool round in the mock (all the examples make one).Requested by guillaume@stackone.com in Slack (repo audit thread).
🤖 Generated with Claude Code
Summary by cubic
Example tests now run the real files from
examples/instead of re-implementing their logic, so a broken example fails CI instead of staying green.tests/examples/run-example.tsimports each example, points it at the mocks, captures console output, and turnsprocess.exitinto an assertable exit code.workday_list_workers, the example executes it against the MCP mock, and the model answers once the tool result is back.vitest.setup.tsfails any unmocked request to a remote host; previously the Anthropic test reached the realapi.anthropic.com.workday_*tools to the MCP catalog, which the examples filter on but were missing, so the filters matched zero tools.query()spawns a subprocess MSW cannot intercept.Written for commit 0450545. Summary will update on new commits.