Repository navigation
eval: 拆出入口画像重复运行与配对比较工具 - #132
Merged
Merged
Conversation
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
从 #128 的
94a7778单独拆出memory_runner repeat / summarize / compare和工具测试,使重复评测与按案例配对比较可以独立使用。repeat每次使用独立数据库与目录;离线汇总保留全部运行,核对输入和材料指纹,输出逐次结果、均值、逐案例分布、覆盖差、违规/禁止合计、调用量与耗时。汇总和比较继续为描述统计,不作门槛通过声明。删除
--union-nomination,没有新增运行设置覆盖参数;新序列直接使用现有单次运行的默认设置,settings={}。历史 settings 仅作为离线元数据透传,仍可复现 EXP2。产品并集逻辑、设置与相关产品测试未移植,附EXP2-report.md历史汇总。产品代码、提示词、评分、计分合并与单次运行器均保持 main 行为;17 个既有非 CLI 函数 AST 完全一致。验证(基线
93d1be9,全部使用假模型或既有封存材料,本回合没有真实模型调用):学习对照使用仓库
compare_learning_requests.py的外部副本,仅把 v5 加入语料清单以覆盖全部 116 段,其余字节一致。召回只剔除返回中的 recall_id 与计时字段;请求正文不规整。原始材料及验证日志:/Users/cassia/Local/Code/iris-eval-artifacts/EF-29-eval-tooling-20261020。本 worktree 的.pytest-tmp已删除。命令用法(本回合未运行真实 repeat):
每次评分继续使用既有 score 命令;summarize 不允许省略无效或缺失运行,compare 要求两组语料、源码、评分、模型、重放模式及运行次数一致。