Skip to content

task: Trace CANNBot knowledge through optimization rounds and the Dashboard #522

Open

Description

@jiangkuaixue123

Current state: Implementation is open for review in #523. Local CLI, worker and Dashboard checks passed with synthetic inputs.
Decision needed: Issue acceptance from the requester after review and integration.
Next action: Review #523 and follow the repository checks. Verify with an approved production CANNBot grant and Ascend workload when evaluating the integration on hardware.

Goal and acceptance

  • An explicitly configured Ascend optimization session can query the local corpus and the existing approved CANNBot channel in the same round. Sessions without that configuration retain their current behavior.
  • Dashboard identifies the round and query, distinguishes local and CANNBot results, and shows channel outcomes even for an empty or unavailable response. Source rankings remain separate.
  • The agent can cite the knowledge it used for a submitted candidate. Dashboard connects those explicit references to the candidate, assessment, and measured score. A result or read alone does not imply use.
  • Retained CANNBot card evidence is limited to source and identity metadata, including locator, revision, and digest. Dashboard and retained evidence exclude card content, local checkout paths, and worker diagnostics.
  • Automated checks exercise the managed path from query through candidate submission and Dashboard, including a local-only session and unavailable external results.

Scope and constraints

Ownership and authorization

  • Requester and goal-acceptance owner: @jiangkuaixue123.
  • Acting implementer: Codex, delegated by the requester in the implementation chat to open this Issue and deliver the scoped PR.
  • The requester retains completion and closure authority.

Activity

  1. added
    kind:taskBounded delivery task with a delegated slice
    area:harnessHarness module: profile, A/B loop, quality gates, evidence reports
    area:knowledgeKnowledge line: packs, skill library, knowledge admission
    area:usabilityUsability and collaboration: issue navigation, templates, readable reports
    on Nov 19, 2026
  2. Dong1017 commented on Nov 19, 2026

    @Dong1017
    Collaborator

    Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

    Assessment: the slice is well-scoped for this repository — concrete, checkable acceptance bullets, an explicit non-goals list, reuse of the #476/#484 read-only channel and grant instead of a new external dependency, and an explicit refusal to claim performance (Retrieval ranking changes, historical run backfill, and performance claims are outside the slice) which is consistent with our evidence rules. It is materially reviewable as written. What it is not yet is decidable: several acceptance bullets name outcomes without naming the record field, channel state or check that proves them, so "accepted" would currently have to be re-derived by the reviewer from #523.

    Gaps to close before acceptance can be recorded

    • Configuration surface is unnamed. "An explicitly configured Ascend optimization session" and "Sessions without that configuration retain their current behavior" need the actual key/flag, its default (off), and the named pre-existing behavior it must not disturb. As written the regression clause has no observable target.
    • "Cite" vs "read" has no schema. The bullet correctly says a result or read alone does not imply use, but that distinction must live in the managed session record — e.g. retrieved/returned source ids separate from agent-declared cited source ids — or the Dashboard cannot enforce it and will infer use from retrieval.
    • Channel outcome states are not enumerated. "shows channel outcomes even for an empty or unavailable response" should pin the terminal set (ok / empty / denied / timeout / not-configured) and require the unavailable state be distinguishable from empty. Otherwise "unavailable" is shown as zero results.
    • Retention allow-list needs to be a test, not a sentence. "Exclude card content, local checkout paths, and worker diagnostics" should be an explicit permitted-field set (locator, revision, digest, channel identity) with a negative assertion that the excluded classes are absent from both Dashboard payloads and retained evidence — including error strings, which is how checkout paths usually leak in.
    • "Source rankings remain separate" is ambiguous. Confirm it means existing rankings are untouched in this slice (consistent with ranking changes being out of scope), not that a new ranking view is introduced.

    Hardware verification and acceptance scope — please settle explicitly

    The body defers real verification to "an approved production CANNBot grant and Ascend workload," while local CLI/worker/Dashboard checks passed on synthetic inputs. If hardware verification is inside the goal-acceptance gate, this issue cannot close until that grant exists; if it is not, say so and state that acceptance is decidable from the managed-path checks alone. I'd recommend the latter, with hardware verification tracked as a separate follow-up against #476/#484 — but that call belongs to @jiangkuaixue123 as goal-acceptance owner.

    Process notes

    Suggested next step: in #523, add a short acceptance table mapping each of the five acceptance bullets to the concrete check, artifact path or test name that demonstrates it, plus the configuration key and the channel-outcome enum. With that, the acceptance decision on this issue becomes a read of the table rather than an independent audit, and the hardware question above resolves to one line.

    — Dong-bot (authorized delegate)

  3. zengchuang-hw commented on Nov 19, 2026

    @zengchuang-hw
    Contributor

    PASS

    这是对 #522 的范围和验收设计的判断,不是 #523 的代码结论。建议保留同一份 managed session 控制记录,Dashboard 只读展示;检索命中、主动引用、候选评估应分开,不能由分数或命中推断 CANNBot 收益。

    实现验收建议覆盖三点:① 无配置、非 Ascend、knowledge-off、grant 撤销或 checkout 不匹配均不得绕过准入;成功但无命中与未启用、超时、索引过期应分别呈现。② 引用只接受当前 session 留存的精确 capture/card 身份,保留 round 和 candidate 关联;缺失、跨 session、篡改和历史无记录不能补成“已使用”。③ 验证原始留存 JSON、离线 HTML、在线刷新及错误分支的内容最小化,不只看页面。正文、frontmatter、私有路径和 worker 诊断不应落盘到留存记录;外部排序分数不能替代 evaluator verdict。

    已对照主干 c573f40a 的控制器、外部渠道准入及 trace 契约。未运行 CLI、worker、Dashboard 或 Ascend 实验。

  4. david6666666 commented on Nov 19, 2026

    @david6666666
    Collaborator

    Could we add one degradation check to the acceptance table? Run the same query and limits with CANNBot off and with an approved worker that times out. The opted-in run should show timeout separately, preserve the local result IDs and order, and still settle a candidate citing only local evidence when time remains, without consuming the reserved final-evaluation time. This carries #476's degraded-mode guarantee through the managed path; rendering an unavailable channel alone doesn't verify it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area:harnessHarness module: profile, A/B loop, quality gates, evidence reportsarea:knowledgeKnowledge line: packs, skill library, knowledge admissionarea:usabilityUsability and collaboration: issue navigation, templates, readable reportskind:taskBounded delivery task with a delegated slice

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions