Repository navigation
Uptake bc-eval 0.6.0: call the M365 LLM API instead of CAPI - #962
Merged
Merged
Conversation
bc-eval 0.6.0 removed CAPI. The bcal bridge now builds the LLM API client and passes the locally staged certificate through the new credential hook instead of monkey-patching bc-eval internals; bceval metrics calculate drops --use-capi and defaults to the LLM API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Code Coverage OverviewLanguages: Python Python / code-coverage/pytestThe overall line coverage in commit c1ec162 in the Show a line coverage summary of the most impacted files.
Updated |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The result-affecting provider migration needs the repository-required benchmark version bump and lockfile refresh.
1 open finding
What changed in this PR
Migrates BCal and result publishing from bc-eval’s CAPI integration to the M365 LLM API in bc-eval 0.6.0.
Changes:
- Replaces the CAPI bridge with an M365 LLM API bridge and certificate authentication.
- Updates workflows, configuration, documentation, and package pins.
- Replaces CAPI bridge tests with LLM API coverage.
| File | Description |
|---|---|
.env.sample |
Documents the new bridge and LLM API settings. |
.github/workflows/bcal-evaluation.yml |
Runs BCal through bc-eval 0.6.0 and M365 LLM API. |
.github/workflows/summarize-results.yml |
Updates result publishing to bc-eval 0.6.0. |
docs/code-review.md |
Updates the documented bc-eval version. |
pyproject.toml |
Updates unresolved-import context. |
src/bcbench/agent/bcal/bc_eval_capi_bridge.py |
Removes the CAPI bridge. |
src/bcbench/agent/bcal/bc_eval_llm_api_bridge.py |
Adds the M365 LLM API bridge. |
tests/test_bcal_capi_bridge.py |
Removes obsolete CAPI tests. |
tests/test_bcal_llm_api_bridge.py |
Tests requests, credentials, and client invocation. |
tests/test_result_writer.py |
Updates the bc-eval version reference. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The BCal agent and lm_checklist judge now call the M365 LLM API instead of CAPI, which may affect results, so this is a minor bump per the versioning policy. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b24c001b-216b-4506-950a-112b6f7a0302
bc-eval 0.6.0 defaults the LMChecklist judge to gpt-56-ceres and no longer routes claude-* models, so the LLM API cannot serve the Claude option. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b24c001b-216b-4506-950a-112b6f7a0302
bc-eval 0.6.1 only fixes release publishing. The LLM API rejects gpt-55-chat-2026-04-29 as an unknown model; gpt-55-chat is its name there. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b24c001b-216b-4506-950a-112b6f7a0302
martinsrui-msft
marked this pull request as ready for review
October 9, 2026 09:23
martinsrui-msft
approved these changes
Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.