Skip to content

Make profiling optional in service-reuse campaigns - #521

Merged
jiangkuaixue123 merged 5 commits into
InferMatrixLab:mainfrom
lirx-pd:feat/service-reuse-optional-profiling
Nov 20, 2026
Merged

jiangkuaixue123 merged 5 commits into
InferMatrixLab:mainfrom
lirx-pd:feat/service-reuse-optional-profiling

Conversation

@lirx-pd

@lirx-pd lirx-pd commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Governing Issue and delivery slice

Refs #513

  • Accepted scope / delegation: Make profiling supplemental for ordinary managed campaigns using service_reuse. @Dong1017 accepted the scope and assigned delivery coordination to @lirx-pd. Issue decision
  • What this PR delivers: Formal performance and quality determine round acceptance. Each round reports profile availability, and operators can collect a profile separately after the round. Only substantiated profiling time is excluded from the evaluation budget.

Knowledge authorship declaration

{
  "schema_version": 1,
  "head_sha": "47239e4864b3a3694dd7320d50852922bdd75233",
  "human_only": false,
  "material_ai_authors": [
    {
      "tool": "Codex",
      "model_id": "gpt-6.2-astra",
      "effort": "xhigh",
      "role": "author"
    }
  ]
}

Changed artifacts and evidence

Requirement Actual change Check / evidence and result
Finish formal performance testing Screening is diagnostic. After functional checks, readiness and warmup, the campaign runs the configured formal measurements while the service is usable and budget remains. A formal performance rejection skips quality and settles as FAIL. test_project_service_reuse.py covers screening rejection, script error, missing formal evidence and valid gate failures.
Preserve the verdict across capture failure and recovery The executor saves formal evidence, the quality outcome and the assessment before inline capture. Required candidate and reference receipts determine settlement, selection and final admission. Campaign and reference evidence tests cover recovery without remeasurement, progression after profile loss and blocked decisions after required evidence loss.
Report availability and return errors to the agent CLI and dashboard summaries expose available and the campaign-relative receipt and artifact paths when usable. The agent receives the failed stage, a bounded excerpt and available details or log paths. Full collector output stays in its receipt. The 32 KiB limit applies to the encoded capture interval and artifact locations. test_service_profile_diagnostics.py covers large receipts and configurations, invalid captures and fallback to usable evidence.
Backfill a settled round inferrsi campaign profile --session <campaign> --round <number> starts a fresh service for the last settled PASS or FAIL attempt. Publication requires validated artifacts and verified cleanup. Round availability uses the latest valid capture. Selection records resolve the original inline profile by attempt ID, including when reading historical records. Campaign and backfill tests cover retries, lock contention, publication interruption and capture after the original deadline or finalization.
Protect the evaluation budget and final reserve Capture-ID accounting covers inline capture and standalone preparation, capture and publication. Recovery credits only documented work. A durable start alone earns zero credit, and the configured final reserve stays fixed. Controlled-clock tests cover the original deadline, delayed recovery and repeated recovery of the same capture.

Inline capture runs when configured after either a formal performance rejection or a combined performance and quality pass. A quality rejection settles as FAIL and skips capture. Missing required evaluation evidence yields INCONCLUSIVE. The CLI contract, exit codes and recovery examples are in docs/service-reuse-campaigns.md. The command is also listed in docs/operations.md.

Checks and dependencies

  • Checks actually run (result and tested scope): 109 tests and 43 subtests passed on WSL with Python 3.11 and a synthetic service. They cover profiling diagnostics, backfill after final, the project campaign lifecycle, Apps module migrations and import boundaries. Ruff and git diff --check passed.
  • Dependencies / shared interfaces / integration assumptions (if applicable): Uses the existing campaign lock, recorded revision and configuration, service ownership and cleanup receipts. The optional sampling.profile_timeout_seconds defaults to sampling.stage_timeout_seconds. A conclusive final after a condition change still requires a candidate accepted under the new conditions.
python -m pytest \
  tests/test_apps_module_migrations.py tests/test_target_module_import_gate.py \
  tests/test_service_fingerprint_isolation.py tests/test_service_profile_diagnostics.py \
  tests/test_project_optimization.py::test_project_lifecycle_never_imports_benchmark \
  tests/test_project_service_reuse.py::test_backfill_after_final_preserves_original_evidence_and_selection

Remaining work and Issue closure

@Dong1017 owns the repository acceptance decision for #513. Scope and ownership

Handoff (only when an actual handoff or decision changed)

@Dong1017 Dong1017 added area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice labels Nov 19, 2026

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bounded supplemental profile can still make the authoritative round unreadable.

Comment thread src/inferrsi/harness/project_service.py Resolved
zengchuang-hw
zengchuang-hw previously approved these changes Nov 19, 2026

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

首次独立检视,范围为 #513 已接受的设计。正式评估在 inline capture 前持久化,核实清理后从 checkpoint 恢复,不重跑测量;profiling 不参与 verdict/选优。预算按 capture ID 幂等结算,缺结束凭据仅返还已证明时长,单有起点返还零秒;调整后时间用于准入及执行,final reserve 不变。backfill 定位最后已结算 PASS/FAIL,独立路径/索引,失败不丢旧 profile 或改写旧 verdict。新增回归覆盖中断和重复恢复、锁/清理以及损坏证据等边界。未发现批准前必做项。

静态核对了 17 个文件和回归代码。作者的本地 CPU 合成服务结果不代表真设备或全量套件;我未运行代码、测试或查询 CI。本结论不改变 Draft,也不等于 #513 整体验收。

检视 head:586c56e4bb8676a84a569afd24262c862ec7aed6。

@lirx-pd
lirx-pd force-pushed the feat/service-reuse-optional-profiling branch from cb4c91c to 9fa23fb Compare November 19, 2026 10:48
@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by an agent on behalf of @Dong1017 (incremental review only; points already raised by others are not repeated).

DO NOT PASS — reviewed head 9fa23fbf95d2e8d13b5039fcf6c16593027a5ee9 (a rebase of cb4c91ce; project_service.py, service_backfill.py and service_evidence.py are identical between the two). One new blocking point, introduced by the receipt-size fix in cb4c91ce. @david6666666's duplicated-receipt concern looks addressed: the round record no longer embeds the collector receipt and reads it back from execution/profile/result.json.

Blocking

  1. The 32 KiB profile-metadata cap is mostly spent on campaign conditions, not on collector output, so a moderately large effective_config disables profiling for the whole campaign.
    validated_profile (src/inferrsi/harness/project_service.py, the PROFILE_METADATA_LIMIT check) counts the encoded profile dict, which carries conditions and binding. binding itself repeats conditions and effective_config, so effective_config is counted three times. With an effective_config of roughly 11 KB or more, every capture fails with profile: metadata exceeds 32768 bytes, however small the collector receipt and artifact list are. This affects inline capture, every campaign profile backfill, and read-time validation alike. The capture can never become available, and the error points the operator at the profiler rather than at the protocol's config size.
    Reproduced with the existing test_project_service_reuse.py fixture. I added 600 engine.option_NNNN keys to conditions.effective_config (21,619 bytes encoded) and left everything else unchanged. The round settled PASS with profiling: {"available": false} and profiling_error: "profile: metadata exceeds 32768 bytes". campaign profile --round 1 exited 2 with the same message. The unmodified fixture (19-byte config) gave available: true and a backfill exit of 0.
    Suggestion: apply the cap only to the parts the collector controls, i.e. the artifact locations and any receipt-derived fields, since conditions and binding are already bounded and validated elsewhere. Alternatively, store a reference to the attempt's existing request.json instead of re-embedding conditions and binding. Please add a regression test with a large effective_config and a tiny receipt, covering both inline capture and backfill.

Non-blocking

Scope checked: the full 9b81c9ba1..586c56e4 diff (17 files; project_service.py, optimization.py, service_backfill.py, service_evidence.py, service_observations.py, CLI/dashboard wiring, docs) and the 586c56e4..cb4c91ce delta (unchanged in the rebase to 9fa23fbf); #513 body and comments; existing PR reviews and inline threads. Ran the six targeted modules from the PR body locally at 586c56e4 (all passed), all six modules at cb4c91ce (all passed), and the reproduction above in an isolated worktree on Python 3.12 / Git 2.43. Not run: the full suite, real profilers or devices. PR-head CI full-check was still pending when this was written.

zengchuang-hw
zengchuang-hw previously approved these changes Nov 19, 2026

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

增量检视 586c56e4bb8676a84a569afd24262c862ec7aed6 → 2786151d681af2d1362233d1789bcf9495c2e01f。对照 上轮检视 和 #513。

新记录不再把大体积 profiler receipt 重复嵌入观察与引用,仍能按路径回读并核验原始 receipt、配置、服务绑定、完成状态及区间。32 KiB 上限只限制新增区间和 artifact 元数据。旧的大体积记录仍可通过逐字流式比对进入后续轮次和 final,不改写历史原件;补充 profile 仍不计入性能或质量裁决。回归用例覆盖大 receipt、大配置、超限元数据保留旧捕获、旧记录继续运行和篡改拒绝。

批准前必做:无。已静态核对变更与回归用例,未执行代码、测试或实机捕获。本检视不改变 Draft 状态,也不代表 #513 的完整验收。

@lirx-pd

lirx-pd commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

cc @jiangkuaixue123

@lirx-pd
lirx-pd marked this pull request as ready for review November 19, 2026 11:56
@jiangkuaixue123

Copy link
Copy Markdown
Collaborator

基于 2786151 的静态检视,有三条可维护性建议。2,814 行新增里,1,363 行是测试和 fixture,1,342 行是运行时代码,109 行是文档。#513 明确要求历史补采、按采集 ID 恢复预算,以及采集失败后仍能结算正式评测。建议保留这些行为,优先收紧下面前两处。

  1. 让在线采集和补采共用一个验证入口。 目前调用链是 optimization._service_profiling → service_backfill.latest → _validated → optimization._service_profile。可以把最后一层对 attempt、revision 和持久化 profile 的核验移到已有的 project_service.validated_profile 附近,让两条路径直接调用。补采模块继续负责验证原轮次、补采实例和索引。这样读“这个 profile 是否可用”就少绕一次主控制器。主控制器 · 补采

  2. 把“丢 profile 仍能继续”单独测一次。 当前针对四个必需阶段的参数化用例,每次都会先删 profile、确认能继续、再提交一轮。可以单独保留这条端到端路径,再把“丢正式凭据会阻止下一轮和 final”按现有的来源和阶段组合参数化,共用阻断断言。采集进程中断、索引发布前中断和发布后中断各有不同的恢复行为,建议分别保留。混合了两种行为的用例 · 已有的凭据缺失用例

  3. 后续可减少选优记录携带的 profile 数据。 choice() 把 profile 写进选择记录,references_reason() 核验时又跳过它,reference_snapshot() 还要处理历史内嵌回执。可以先核对 best.profile 的调用方,再考虑让新选择记录按 attempt 引用轮次,在输出时组装所需字段,并保留旧记录的读取能力。记录构造 · 引用核验

如果希望分批交付,可以先独立审查“正式结果先落盘、profile 失败后恢复原判、后续选优和 final 只依赖必需证据”,再分别审查 screening、预算恢复和历史补采,最后按 #513 的完整要求验收。本次反馈来自代码与测试源码阅读,未重跑测试。

Signed-off-by: lirx-pd <616517220@qq.com>
Agent-assisted-by: Codex / gpt-6.2-astra [effort=xhigh] (author)
Signed-off-by: lirx-pd <616517220@qq.com>
Agent-assisted-by: Codex / gpt-6.2-astra [effort=xhigh] (author)
Signed-off-by: lirx-pd <616517220@qq.com>
Agent-assisted-by: Codex / gpt-6.2-astra [effort=xhigh] (author)
Signed-off-by: lirx-pd <616517220@qq.com>
Agent-assisted-by: Codex / gpt-6.2-astra [effort=xhigh] (author)
Signed-off-by: lirx-pd <616517220@qq.com>
Agent-assisted-by: Codex / gpt-6.2-astra [effort=xhigh] (author)
@lirx-pd
lirx-pd force-pushed the feat/service-reuse-optional-profiling branch from 2786151 to 47239e4 Compare November 20, 2026 01:15
@lirx-pd

lirx-pd commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

@jiangkuaixue123 Thanks for the review. Inline and backfill profile reads now share the same validator. Backfill still verifies the target attempt, capture instance, and publication index. The tests now cover optional profile loss separately from required evidence loss, with distinct recovery cases for interruptions during capture, before index publication, and after publication. New selection records reference the original attempt by ID. CLI and dashboard output reconstruct the original inline profile, while round availability reflects the latest usable backfill. Existing campaign records remain readable. On WSL, 109 tests and 43 subtests passed, along with Ruff and git diff --check.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

增量 2786151d → 47239e48,上轮检视。选择快照不再重复嵌入可选 profile;读历史选择时只按原 attempt 恢复当时的 inline profile,当前轮可用 profiling 则可以独立选最新有效 backfill。共享验证仍检查原凭据、服务绑定、条件和 artifact;必需的性能/质量凭据缺失仍阻断后续评估。新增回归覆盖历史选择、条件刷新、停滞、旧大记录及必要凭据缺失。

批准前无必修项。静态核对源码及测试定义;未执行测试、设备采集或查询 CI。这不等于 #513 整体验收。

@jiangkuaixue123
jiangkuaixue123 self-requested a review November 20, 2026 02:02
@jiangkuaixue123
jiangkuaixue123 merged commit b5756cd into InferMatrixLab:main Nov 20, 2026
12 checks passed

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 493416520f537590eb8f2a4e74f276f82e488307 and head 47239e4864b3a3694dd7320d50852922bdd75233.

Independent third-party verify of tip (zengchuang-hw Approves on earlier heads are dismissed after tip moved). Scope: #513 — profiling optional / supplemental for service_reuse campaigns.

Verified:

  • Formal performance/quality stay authoritative; profile capture is optional, bounded (PROFILE_METADATA_LIMIT), and does not duplicate large receipts into campaign records (addresses prior inline concern).
  • Tip commits keep selection profiles resolved from original attempts (service_observations.selection_view / reference_view) instead of embedding mutable profile blobs in selection snapshots.
  • Local targeted: test_project_service_reuse + service backfill/capture/profile/reference suites → 128 passed.
  • Ready CI green on tip (full-check 3.10/3.11, declarations, installed-package, dashboard-browser, pr-references).

Eligibility: cursor[bot] collaborator permission is none. This Approve is technical/third-party verify only — not a write+ eligible Approver under docs/agent-collaboration.md, and not a claim that dual tip / merge-ready is satisfied. Merge still needs a write+ independent Approve and steward integration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants