[sergiopaniego]

Summary

Two fixes to validate_llm that came up running openenv harbor rollout against the HF Inference Providers router at scale. A probe error that names no parameter (a bare 400, a 5xx, a timeout) aborted the run instead of being retried or falling back to an eval rollout, and the CLI rejected provider-suffixed model ids such as Qwen/Qwen3.5-397B-A17B:novita.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation
• New environment
• Refactoring

What was wrong

Probe. harbor rollout sometimes aborted before running the task:

Error: inference endpoint is not usable:
  probe failed: HTTP 400: {"message":"invalid request error trace_id: ...","type":"invalid_request_error"}
...
Any other OpenAI-spec endpoint (OpenAI, Anthropic, HF Inference Providers) still works, as an EVAL backend: ...

Once transiently (1 task in ~150, the same endpoint served the others), and deterministically when the router's default provider for a model rejects return_token_ids with a 400 that names no parameter, although the message itself says such an endpoint works as an eval backend. The probe loop (src/openenv/core/harness/capture/validate_llm.py:633) only recovers from a 400 that compat.diagnose maps to a parameter. Anything else ends the probe on the first attempt with reachable=False, and prepare() raises.

Provider suffix. --llm-url https://router.huggingface.co/v1 --model Qwen/Qwen3.5-397B-A17B:novita failed with model ... is not served here. The router routes <id>:<provider> (and :fastest, :cheapest) but /v1/models lists bare ids. validate_llm.py:577 already strips the suffix, but only for provider == "hf", which the web UI passes (src/openenv/harbor/ui.py:599) and the CLI can't.

Fix

• 401, 403 (except the known logprobs 403, still diagnosed), 404 and connection errors fail fast, as before.
• An HTTP error that names nothing to fix, or a timeout, is retried with the same request up to PROBE_RETRIES = 2 times, 1 s apart, so a transient error on a vLLM endpoint still grades tokens.
• If it persists, the probe tries once without return_token_ids, logprobs and top_logprobs. If that answers, the level is graded from the reply (text), the run is an eval rollout, and the dropped params show in param_fixes, which harbor info and harbor rollout already print.
• The router is also recognised by its host, so <listed id>:<suffix> is accepted there when <listed id> is listed, and the suffixed id is what gets sent. Unlisted models are still reported, and other endpoints still need an exact match.

This follows RFC 012: with purpose="auto" (what harbor rollout uses) or "eval", an endpoint without token capture runs as eval. With purpose="train", rollout_type_for still rejects anything below tokens.

RFC Status

• Not required (bug fix, docs, minor refactoring)

Test Plan

tests/envs/test_harbor_capture_level.py, 12 new tests with a scripted fake _post and a faked /models list, no network:

• a transient 400 with the router's body is retried with an identical request and stays trainable, and a 503 and a timeout are retried
• a persistent bare 400 falls back to text/eval without the capture params, and a 400 that survives the fallback is still reported after PROBE_RETRIES + 2 calls
• 401, 403 and 404 fail after exactly one call, and purpose="train" still refuses text
• the router accepts Qwen/Qwen3.5-397B-A17B:novita and every probe body carries the suffixed id (failed before the fix), still rejects an unlisted suffixed model, and a non-router URL still rejects <listed id>:novita

pytest tests/envs/test_harbor_capture_level.py: 53 passed. Across tests/envs/test_harbor_*.py, the only failure (test_harbor_capture_server.py::test_start_refuses_a_port_held_by_a_non_capture_listener) also fails on main. ruff format, ruff check and usort are clean.

Not yet run end to end against the router with the fix: the errors above come from ~3,000 rollouts of Qwen3.5-397B-A17B with OpenCode in HF Sandboxes before it.

Claude Code Review

N/A

---

Note

Medium Risk
Touches core endpoint validation for Harbor rollouts (classification, retries, when capture params are dropped); wrong branching could mislabel backends, though behavior is narrow and heavily covered by new tests.

Overview
Harbor validate_llm: The completion probe now treats unnamed HTTP failures as transient first: it resends the same body up to two times with a short delay, then—only for 400/422 after retries exhaust—drops return_token_ids, logprobs (and thus top_logprobs) via ParamFix, records that in param_fixes, and grades the reply so hosted routers that reject capture params still land as eval instead of failing the run. 401/403/404 still stop immediately; 429/503 and similar never trigger that param strip, so busy backends are not downgraded to eval. Timeouts (including nested on HTTPError) get the same retry budget.

Model listing: When the base host is router.huggingface.co, /v1/models matching uses the id without the :provider suffix (same behavior as provider=="hf"), so CLI URLs like model/id:novita validate while the suffixed id is still sent in requests.

Tests: test_harbor_capture_level.py adds coverage for retries, eval fallback, no-fallback on overload codes, fast-fail auth, router suffix acceptance/rejection, and rollout_type_for train vs auto.

^{Reviewed by Cursor Bugbot for commit 8c77816. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit c90083f. Configure here.}