Skip to content

docs(evaluation):补充合成消融 v1 数据集规模、分布与查询倾向说明 - #133

Merged
VirusPC merged 5 commits into
mainfrom
docs/synthetic-dataset-shape
Nov 17, 2026
Merged

VirusPC merged 5 commits into
mainfrom
docs/synthetic-dataset-shape

Conversation

@VirusPC

@VirusPC VirusPC commented Nov 17, 2026

Copy link
Copy Markdown
Owner

做了什么

为合成记忆消融 v1 数据集(RESULTS.md 的分数来源)新增 synthetic/DATASET.md,并在 README.md、RESULTS.md 加入口链接。只改文档,未改数据、生成器或 DATA_REPORT.*(后者是生成器产物,hash 绑定在 BUILD_MANIFEST.json)。

新增内容

  1. 规模与分布:193 个 scope(根 1×10 条、业务域层 96×3 条、服务层 96×4 条)、682 条记忆(可见 586、未来 96)、96 个业务域构成;标题分布(留存期限 384 / 发布门槛 192 / 回退条件 96 / 归档 5 / 通知 5);四种版本时间窗口;一个真实业务域的目录树,以及 scoped / root 两组工作区的落盘方式。
  2. 查询任务:五类题 dev/test 数量、每题 gold 数(共 85)、起始 scope、gold 相对起点的位置、task_mode 分布。
  3. 倾向性与泄露:问题只写业务名和主题,不写路径或 ID(生成器有校验);Agent 会被明确告知当前 scope,属于部分指路;同名标题与业务名字面重叠情况;词法基线在 test 上逐类的 recall@5 和漏检明细。
  4. 时间过滤:规则继承自真实历史实验,runner 只写入 available_at < query_at 的记忆,未来记录 Agent 看不到;真正考验时间判断的是同时可见的过期版和当前版。
  5. 样例:每类一道 test 原题、gold 的 scope、标题和正文摘录,并说明为何属于该类。

所有数字都由脚本从 data-v1.json / DATA_REPORT.json 重算,与现有 README / DATA_REPORT 一致,未发现冲突。

验证

python3 -B -m unittest discover -s tests -p 'test_synthetic.py':32 个测试通过。

@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Central YAML; Path: .coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f4c8fe13-3735-4810-98a5-f7493ae10849
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@VirusPC
VirusPC merged commit 8b2a424 into main Nov 17, 2026
2 checks passed
VirusPC added a commit that referenced this pull request Nov 17, 2026
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants