Add iq-routing submission - #209
George-Avila wants to merge 3 commits into
Conversation
Judge-based complexity routing over a fixed 5-model pool (gemini-3-flash-preview, gemini-3.8-flash, gpt-5-nano, gpt-5.6-luna, gpt-5.6-terra). 8400 rows + 420 routing-only robustness rows. Adds 3 required model rows to universal_model_names.py and model_cost.json at public list prices.
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Evaluation completed by RouterArena automated workflow |
Re-served the 15 rows the evaluation flagged as having no valid generation (4 HTTP 503, 1 read timeout, 9 cap-truncated generations, 1 output-blocked 400) through the same routing policy; 13 now carry valid generations (one via the gpt-5.6-luna fallback rung), 2 remain cap-truncated and stay failed. The two GeoBench image rows remain honest failed rows. All other rows byte-identical.
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Evaluation completed by RouterArena automated workflow |
…e v2 The routing rule's multiple-choice detector required the literal header "Options:" before reading the lettered option block; the robustness split renames that header on 253 of 420 prompts while keeping the options, which flipped 248 rows. Version 2 of the feature drops the header requirement. It selects exactly the same rows on all 8,400 original prompts (the 8,400-row prediction file is unchanged), and the 420-row routing file is recomputed from the recorded judge decisions with no new model calls.
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Evaluation completed by RouterArena automated workflow |
|
The current automated result excludes the routing-decision call to |
2 participants
Add IQ Routing submission
Router
IQ Routing (https://iq-routing.com) is a closed-source, general-purpose production
gateway. It is not a research prototype built for this benchmark — its main workloads are coding
turns and cache protection, not QA benchmarks. The answers in this submission were generated by
IQ's routing pipeline, executed in an offline experiment harness against IQ's own provider
accounts. The gateway build carrying this
configuration is being deployed separately.
For this benchmark, each prompt was answered by exactly one of the five pool models, chosen by
IQ's routing. One routing decision and one generation per query — no cascade, no answer-side
ensembling, and no re-routing on a wrong answer.
The one exception is a provider-side content block: on 4 of the 8,400 rows (four NarrativeQA
queries) the selected model returned HTTP 400 for prohibited content before generating anything, and
IQ's fallback rung (the same rung the production gateway uses) served
gpt-5.6-lunainstead. Theblocked attempt produced no tokens and no cost; those rows are billed only at the model that actually
answered, which every row reports in
model_used.After the first
/evaluaterun flagged 17 rows with no valid generation, the 15 that had failedtransiently were re-run once through the same routing policy and the prediction file was updated in place (commit
b9ce3f0); 13 nowcarry valid generations and 2 still have no valid generation. The two GeoBench image rows stay failed.
Robustness file recomputed (2026-09-22).
iq-routing-robustness.jsonwas regenerated after correcting adefect in how one of our routing features read reworded prompts. The 8,400 predictions in
iq-routing.jsonare unchanged: the corrected feature makes the same decisions on all 8,400 originalprompts, so the served model, accuracy and cost on every full-split row are untouched. Only the
robustness split moves, recomputed offline from the robustness pass's already-recorded routing decisions
with no new generations (commit
dab1fe6). The same feature version was deployed in the productiongateway on 2026-09-22.
All five pool models were served internally, against IQ's own provider accounts (BYOK), by that
same pipeline. Costs on this leaderboard are therefore computed by RouterArena from reported token
counts at public list prices, not from anything IQ was billed.
gemini-3-flash-previewgemini-3.8-flashgpt-5-nanogpt-5.6-lunagpt-5.6-terraFiles
router_inference/predictions/iq-routing.jsongenerated_resultrouter_inference/predictions/iq-routing-robustness.jsongenerated_resultrouter_inference/config/iq-routing.jsonuniversal_model_names.pygemini-3.8-flash,gpt-5.6-luna,gpt-5.6-terramodel_cost/model_cost.jsonThe two registry additions are required: without them the three models do not resolve in
ModelNameManager, and_lookup_cost_info's substring fallback would silently billgpt-5.6-lunaandgpt-5.6-terraat thegpt-5rate andgemini-3.8-flashat no rate at all.Prices used are the public list prices in force on the run date
(
gemini-3.8-flash$0.75/$3.75 per 1M is Google's promotional tier, published as in forcethrough 2026-12-31;
gpt-5.6-luna$0.20/$1.20;gpt-5.6-terra$2.00/$12.00).Two GeoBench queries (
GeoBench_841,GeoBench_1080) reference an image that the text-onlydataset does not carry. The production gateway answers those with a "no image attached" notice
rather than guessing. They are submitted honestly as failed rows:
success: false,empty
generated_answer,error: "no_image_attached", zero tokens. They score 0 and contributeno cost. We did not substitute a guessed answer.
Evaluation Integrity
learned router, no classifier fitted to RouterArena labels, and no weights of any kind derived
from this dataset. The label files were used for scoring only.
turns and cache protection, not QA benchmarks. The outputs submitted here come from a fixed
routing configuration of that gateway, served internally through IQ's own provider accounts.
recomputes cost from the reported token counts at public list prices.
Reproducibility
Every row carries the provider-reported
model_used,provider, and token counts from theoriginal generation.
token_usagereportsinput_tokens,output_tokens, andtotal_tokens;total == input + outputon every row, so RouterArena's reasoning-token residual is zero byconstruction.
Prompts are byte-identical to
dataset/router_data.json. The routing-only robustness file wasproduced by the same routing, over the 420 prompts of
dataset/router_robustness.json.