Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,8 @@ details are in [benchmark comparison](docs/benchmark-comparison.md).
Ordinary project optimization stays in the current Agent conversation. Follow
the [shared optimization workflow](docs/optimization-workflow-unification.md):
prepare explicit workload, quality/measurement scripts, metric, gates, declared
project inputs and budget, then use `campaign start/knowledge/round/status/final`.
project inputs and budget, then use `campaign start/diagnose/knowledge/round/status/final`.
Record the measured bottleneck before optional knowledge retrieval or candidate search.
Harness freezes those supplied conditions and assesses the retained results;
it does not infer acceptance from the user's goal. Ordinary optimization does
not depend on Benchmark. Use `up_to` with a recorded stop reason or declare exact rounds.
Expand All @@ -129,7 +130,8 @@ campaigns generate dashboard views from that control record; do not use legacy

For a request to run an optimization task through RSI, follow the
[current-Agent benchmark workflow](docs/optimization-benchmarks.md).
Use `benchmarks session` to freeze the budget, submit each changed candidate,
Use `benchmarks session` to freeze the budget, record the measured bottleneck,
submit each changed candidate,
read its feedback, and independently retest the selected candidate with `final`.
Report the session summary after the declared rounds and final are complete.
`benchmarks run` is the single-candidate measurement operation inside that workflow.
Expand Down
10 changes: 10 additions & 0 deletions docs/benchmark-comparison.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,16 @@ The CLI neither starts a hidden Agent CLI nor invents a conversation identity.
6. Generate a progress or final comparison with `--action report`. Read
`--action status` to resume. Reporting never executes missing rounds/finals.

New comparisons freeze the same diagnosis requirement for both arms. Before
search, each session profiles the frozen baseline or uses measurements from the
same workload and environment. It records the measured bottleneck with
`benchmarks session diagnose --session <campaign> --bottleneck '<observed-bottleneck>'
--evidence <measurement-summary.txt>`. The UTF-8 input should contain the
measured values and scope. Knowledge retrieval is optional when enabled and
follows the diagnosis. The comparison JSON includes diagnosis IDs for each round.
Collection verifies the retained summaries and the round-to-diagnosis links
against the original round observations.

All non-prepare actions require `--output`. Dispatch/bind/record/collect also
require `--trial`. Jobs are created just before dispatch so waiting for earlier
trials does not consume their host deadlines. Native deadline cancellation
Expand Down
14 changes: 9 additions & 5 deletions docs/optimization-benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,17 @@ Final evaluation reserves one evaluation timeout from the total budget by
default; `--final-reserve-seconds` changes that explicit reservation. Use
`--knowledge off` for a condition that disables managed knowledge retrieval.

## Retrieve knowledge and propose a change
## Diagnose the bottleneck and propose a change

When the chosen experimental condition permits RSI knowledge, retrieve it through
the session. This calls the existing KnowledgeReader and retains the exact
delivered context with its hash.
Use profile data or existing measurements for the same workload and environment.
Record the bottleneck and a UTF-8 summary of its measured evidence before
the first candidate round. When the condition allows knowledge retrieval, use
it after diagnosis for a specific implementation question. The session retains
the exact delivered context with its hash.

```bash
inferrsi benchmarks session diagnose --session ./outputs/qwen-rsi \
--bottleneck '<measured-bottleneck>' --evidence ./measurement-summary.txt
inferrsi benchmarks session knowledge --session ./outputs/qwen-rsi \
--query 'Ascend denoising conditioning reuse normalization fusion' \
--context-bytes 16384
Expand Down Expand Up @@ -95,7 +99,7 @@ attempt prevents a second launch after an interrupted coordinator.
Repeat this operation until the declared round budget is consumed. Use
`session status --session ./outputs/qwen-rsi` to inspect progress. Each call
evaluates a newly retained candidate snapshot and returns independent feedback.
Profiling is available to the Agent as an optional diagnostic.
Refresh the diagnosis when new measurements identify a different bottleneck.

### A failed experiment can leave a usable best candidate

Expand Down
12 changes: 10 additions & 2 deletions docs/optimization-workflow-unification.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ sequenceDiagram
loop 每次命令独立进入相同的执行与判定链路
alt 继续搜索:准备一轮候选
A->>A: 分析反馈,按需 profiling,明确本轮问题
A->>C: campaign diagnose:记录瓶颈及其测量依据
opt 需要资料且知识开关允许
A->>C: campaign knowledge:提交查询、检索方式和上下文预算
C->>H: session_command:传入查询、KnowledgeReader 和记录回调
Expand Down Expand Up @@ -143,8 +144,8 @@ Harness 没有常驻 Agent 驱动。Agent 逐次调用这些操作,代码在

| 场景 | 入口 | 候选与停止规则 |
| --- | --- | --- |
| 普通项目优化 | `campaign start/knowledge/round/status/final/recover` | 声明多个文件或目录;默认 `up_to` 轮次上限,final 必须说明结束原因;也支持 `exact` |
| 题库优化 | `benchmarks session start/knowledge/round/status/final/recover` | 题包定义候选和评测;当前 Qwen 是单个 Python 文件、固定轮数 |
| 普通项目优化 | `campaign start/diagnose/knowledge/round/status/final/recover` | 声明多个文件或目录;默认 `up_to` 轮次上限,final 必须说明结束原因;也支持 `exact` |
| 题库优化 | `benchmarks session start/diagnose/knowledge/round/status/final/recover` | 题包定义候选和评测;当前 Qwen 是单个 Python 文件、固定轮数 |
| 题库默认新会话 | `agents prepare/dispatch/bind/record` + 宿主工具 | 先派发任务;新会话创建 campaign 并执行到 final,父会话回收结果 |
| 一次测量 | 原有 `run` 或 `benchmarks run` | 保留独立使用方式;不会自动生成多轮候选 |
| 旧式手动 campaign | 原有 `campaign init/record/decide` | 兼容已有记录;不允许写进新控制器管理的 campaign 形成第二套状态 |
Expand Down Expand Up @@ -196,6 +197,7 @@ inferrsi campaign start --directory <campaign-dir-outside-project> \
准备后,Agent 在当前会话循环:

```bash
inferrsi campaign diagnose --directory <campaign-dir> --bottleneck '<observed-bottleneck>' --evidence <measurement-summary.txt>
inferrsi campaign knowledge --directory <campaign-dir> --query '<current-question>' --context-bytes <budget>
inferrsi campaign round --directory <campaign-dir> --candidate <editable-project> --hypothesis '<what-and-why>'
inferrsi campaign status --directory <campaign-dir>
Expand All @@ -205,6 +207,12 @@ inferrsi dashboard --input <campaign-dir> --output <report.html>

两个受管 `knowledge` 入口都用 Find 为当前问题排序,再按预算装配选中的正文和需要的资料,留存引用、版本和省略原因。显式传入旧 `--relevance-method` 可以复用对应的排序。知识检索和 profiling 按当前问题需要进行。冻结条件为 `--knowledge off` 时,受管 `knowledge` 操作拒绝执行。

Harness 将证据优先的顺序默认写入新会话计划,适用于知识开启和关闭。Agent 先用同条件的
profile 或测量确定瓶颈,提交包含 workload 和观测值的 UTF-8 摘要,再按需检索和
提交候选。Harness 保留摘要,并把检索和候选轮次关联到诊断记录。已有记录可以
继续使用,瓶颈变化时重新运行 `diagnose`。service-reuse 的 reference epoch 改变后
需要记录新诊断。已有会话按其冻结计划继续。

## 实际保留什么、保护什么

`campaign.json` 关联计划,`plan.json` 固定协议、预算和候选格式,`state.json` 是唯一可写控制状态。`summary.json` 是可重建摘要,dashboard 从同一记录生成展示;不额外写一份 `rounds.jsonl`。原有知识交付和优化日志格式继续复用。
Expand Down
7 changes: 6 additions & 1 deletion docs/service-reuse-campaigns.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,7 @@ quality-command and sampling flags. Mixing both parameter sets is an error.
```text
inferrsi campaign start --evaluation-mode service_reuse --evaluation-config <protocol.json> ...
inferrsi campaign baseline --session <campaign>
inferrsi campaign diagnose --session <campaign> --bottleneck <description> --evidence <summary.txt>
inferrsi campaign knowledge --session <campaign> --query <cue> --context-bytes <bytes>
inferrsi campaign round --session <campaign> --candidate <committed-project> --hypothesis <text>
inferrsi campaign profile --session <campaign> --round <round-number>
Expand All @@ -127,7 +128,11 @@ inferrsi campaign final --session <campaign> --reason <stop-reason>
```

`start` prepares the campaign records. The baseline establishes reference
measurements. A candidate first passes functional checks and service readiness,
measurements. Record measured bottleneck evidence before knowledge retrieval
or candidate search, and again after a reference refresh. This summary can use
existing measurements. Managed profiling remains optional.

A candidate first passes functional checks and service readiness,
then completes its warmup. While that service remains usable and evaluation time
remains, the harness runs all formal scenarios and repetitions, including after
a screening rejection or screening command error. Only formal performance
Expand Down
6 changes: 5 additions & 1 deletion src/inferrsi/apps/cli/arguments.py
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ def _optimization_actions(stages, *, project=False):
start.add_argument("--knowledge", choices=("on", "off"), default="on")
start.add_argument("--evaluation-prefix", default="", help="Quoted allocator argv prefix")
start.add_argument("--python", default=sys.executable)
actions = ("status", "round", "final", "recover", "knowledge")
actions = ("status", "diagnose", "round", "final", "recover", "knowledge")
if project:
actions += ("baseline", "refresh-references", "resume", "profile")
for name in actions:
Expand All @@ -111,6 +111,10 @@ def _optimization_actions(stages, *, project=False):
if name == "profile":
stage.add_argument("--round", type=int, required=True,
help="Capture profiling for the last settled attempt of this round")
if name == "diagnose":
stage.add_argument("--bottleneck", required=True)
stage.add_argument("--evidence", type=Path, required=True,
help="UTF-8 profile or measurement summary from this workload")
if name == "round":
stage.add_argument("--candidate", type=Path, required=True)
stage.add_argument("--hypothesis", required=True)
Expand Down
20 changes: 17 additions & 3 deletions src/inferrsi/apps/workflows/benchmark_compare.py
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,7 @@ def _prepare_at(config, sources, runtime, task_settings, root):
"condition": condition,
"workspace": workspace.relative_to(root).as_posix()})
plan = {"schema_version": 1, "focus": "knowledge", "config": config,
"diagnosis_required": True,
"parameter_sources": sources, "task_settings": task_settings,
"runtime_files": {key: value for key, value in runtime_files.items()
if key not in knowledge_files},
Expand Down Expand Up @@ -280,6 +281,14 @@ def _prompt(root, plan, trial):
"--evaluation-prefix", plan["task_settings"][trial["task"]]["evaluation_prefix"],
"--evaluation-args", config["task_arguments"][trial["task"]],
"--python", config["python"]]
diagnosis = ""
if plan.get("diagnosis_required"):
diagnosis = (
"Before searching, profile the frozen baseline or inspect existing measurements "
"for the same workload and environment. Record the bottleneck, workload scope and "
"observed measurements with benchmarks session diagnose --session <campaign> "
"--bottleneck <description> --evidence <UTF-8-measurement-summary>. "
)
return (
f"Run complete RSI optimization: {trial['task']}, condition {trial['condition']}.\n"
f"Work only in {workspace}. Set PYTHONPATH to {runtime}; disable bytecode. "
Expand All @@ -302,10 +311,12 @@ def _prompt(root, plan, trial):
"For remote execution, copy this exact runtime to the prepared target. Use a private "
"campaign directory there; return its entire unchanged tree to workspace/campaign above. "
"Preserve bytes and relative paths. Exclude other trials and runtimes.\n"
"Read frozen TASK.md. Analyze feedback and choose a justified optimization each round. "
f"Read frozen TASK.md. {diagnosis}"
"Analyze feedback and choose a justified optimization each round. "
"Submit candidate, --strategy and --hypothesis; cite delivered --knowledge-ref paths when "
"used and retain failures. When enabled, retrieve/read relevant material with session "
"knowledge. Complete frozen rounds and independently retest the selected candidate "
"used and retain failures. When knowledge is enabled, use session knowledge for a "
"specific implementation question after diagnosing the bottleneck. Complete frozen "
"rounds and independently retest the selected candidate "
"using session final. Never replace final with a search score.\n"
"Return campaign, selected code, final delivery and environment observation. Report "
"blocked/failed execution honestly. The initiator collects host receipts and actual usage."
Expand Down Expand Up @@ -440,6 +451,8 @@ def _trial_result(root, plan, trial, host):
"frozen_files": settings["files"]}
if any(campaign_plan.get(key) != value for key, value in wanted.items()):
result["gaps"].append("Campaign task, condition, protocol or budget differs from plan")
if plan.get("diagnosis_required") and campaign_plan.get("diagnosis_required") is not True:
result["gaps"].append("Campaign did not require measured bottleneck evidence")
if campaign_plan.get("runtime_identity") != plan["session_runtime_identity"]:
result["gaps"].append("Campaign runtime identity differs from the frozen runtime")
result["knowledge_deliveries"] = control["contexts"]
Expand All @@ -465,6 +478,7 @@ def _trial_result(root, plan, trial, host):
assessment = record["assessment"]
result["trace"].append({
"round": index, "strategy": record.get("strategy") or record["hypothesis"],
"diagnosis_id": record.get("diagnosis_id"),
"hypothesis": record["hypothesis"],
"knowledge_refs": record.get("knowledge_refs"),
"available_contexts": record["contexts"],
Expand Down
10 changes: 7 additions & 3 deletions src/inferrsi/apps/workflows/campaign.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ def capture_context(root: Path, *, context_id: str, reader, query: str = "",
kind: str | None = None, repo: str | None = None, tier: str | None = None,
backend: str | None = None, scenario: str | None = None,
limit: int = 5, context_bytes: int = 65536,
sources: tuple[str, ...] = ("package",)) -> tuple[Path, str]:
sources: tuple[str, ...] = ("package",),
diagnosis_binding: dict | None = None) -> tuple[Path, str]:
"""Run ``collect`` with a context budget and keep its complete envelope.

The capture is produced by the same ``collect`` operation the agent uses,
Expand Down Expand Up @@ -87,9 +88,10 @@ def capture_context(root: Path, *, context_id: str, reader, query: str = "",
context_bytes_used=result.get("context_bytes"),
contexts=contexts,
assembled_text=assemble_context_text(contexts),
diagnosis_binding=diagnosis_binding,
)
return save_retrieval(root, context_id=context_id, result=result, request=request,
wrapper="campaign capture-context")
wrapper="campaign capture-context", diagnosis_binding=diagnosis_binding)


def _supplies_no_observation(result: dict) -> bool:
Expand All @@ -116,7 +118,8 @@ def _contexts(result: dict) -> list[dict]:


def save_retrieval(root: Path, *, context_id: str, result: dict,
request: dict | None = None, wrapper: str | None = None) -> tuple[Path, str]:
request: dict | None = None, wrapper: str | None = None,
diagnosis_binding: dict | None = None) -> tuple[Path, str]:
"""Explicitly retain a search/collect result without executing another search.

The caller owns this local capture. Review/redact the bundle before sharing;
Expand Down Expand Up @@ -150,6 +153,7 @@ def save_retrieval(root: Path, *, context_id: str, result: dict,
contexts=contexts,
assembled_text=assemble_context_text(contexts),
retrieval=retrieval,
diagnosis_binding=diagnosis_binding,
# A result without an observation has no recorded entry point. It is
# labelled unknown; a plausible current default is not evidence.
captured_by=f"inferrsi {(retrieval or {}).get('entrypoint', 'unknown')} {__version__}",
Expand Down
Loading
Loading