Skip to content

knowledge: add cold EPD GPU provisioning case - #506

Draft
Kelvin-Al wants to merge 2 commits into
InferMatrixLab:mainfrom
Kelvin-Al:knowledge/epd-cold-case
Draft

Kelvin-Al wants to merge 2 commits into
InferMatrixLab:mainfrom
Kelvin-Al:knowledge/epd-cold-case

Conversation

@Kelvin-Al

@Kelvin-Al Kelvin-Al commented Nov 18, 2026 •

Copy link
Copy Markdown

Refs #505

This adds an owner-local Knowledge case for choosing an integer E:PD GPU split from independently estimated stage capacities. It preserves the completed cold MMMU observations as external author-reported evidence with INCONCLUSIVE formal campaign acceptance.

The text distinguishes the integer maximizer of min(e * mu_E, (N - e) * mu_PD), conditional on constant scalable stage rates, from empirical applicability. In the cited four-H800 cold comparison, 2E:2PD ranked first in both reversed-order rounds. The case retains preprocessing, model/runtime identity gaps, fixed generation length, finite measurement window, queueing, first-attempt failure and corrected rerun boundaries.

Rebase onto the #516 layout

Rebased onto main at 1b826b5f9666a8b734000e1eb2990e582cdef876 after #516. The existing inference optimization casebook declares cases/epd-cold-provisioning.md in its SKILL.md and links it under recorded diagnostic options. Both case files retain their previous bytes and source pins. The compiled catalog inputs and coverage page were regenerated from source commit 55cb1c5a945bae7e9dbd211b9f6662af143472dd. Review request · Casebook registration

On head 3c0dc17adfad7e04819025372f4cee74c7e55ff9, inferrsi find "EPD cold MMMU GPU allocation stage capacity" --limit 5 --json returns case.epd-cold-provisioning first. Its read_args opens the case, including the position-balanced follow-up. The catalog and coverage freshness checks and targeted Ruff passed. Knowledge and CLI tests reported 203 passed, 2 skipped because the Windows host could not create symlinks. The wheel built from a clean detached checkout of this head was installed outside the source tree. Five resource hashes matched Git. Installed find opened the case and its source record, and the published wheel was downloaded and rehashed. JUnit, build report, script and wheel

Position-balanced cold MMMU follow-up

The original orders [1, 2, 3] and [3, 2, 1] kept 2E:2PD in second place in the schedule. The follow-up rotated encoder counts as [1, 2, 3], [2, 3, 1] and [3, 1, 2]. All nine waves ran in one four-H800 allocation on the original node, using the same cold 720-image MMMU cohort, seed, 64-token output and 34.27 requests/s target. Inline review · Frozen plan

2E:2PD ranked first in all three rounds, at 26.04, 26.44 and 26.67 PD completions/s from first, second and third position. Its lead over each round's runner-up was 10.7%–13.8%. All waves completed 720 requests with 720 visual forwards and zero EC or resident-cache hits. Each split-position cell has one observation, and the cyclic schedule has limited predecessor coverage. Formal campaign acceptance remains INCONCLUSIVE. Nine-wave results · Interrupted attempt

Earlier validation at 255887c

Head 255887cd3633a7e3eb59bd9bfca461641aa76f50 uses generated projection source 2dabbfe50ddddc872c030ba962b570fa2488e770. Author checks completed for 133 targeted Knowledge/index tests, generator freshness and targeted Ruff. The five follow-up source files were downloaded from their immutable public revision and rehashed. A clean detached checkout of this head built a wheel and installed it outside the checkout. Five installed resources matched the committed blobs, and the EPD discovery result opened the case. The published wheel was downloaded and rehashed. Build report, script, wheel and JUnit

Scope and delegation

User-authorized preparation of the Knowledge contribution. The diff covers the case and source record, its casebook entry, two navigation matrices, inventory/discovery tests, compiled catalog inputs and the generated coverage page. Runtime interfaces and benchmark execution are outside this slice.

Source snapshot epd-provisioning@3ed184e3.
External experiment issue,
frozen cold plan,
cold results.

The source record owns seven retrieved-file SHA-256 values. A case-specific test on the final head checks normalized source metadata, required-resource byte identity, missing-resource failure and invalidation of a stale content revision. This import is frozen. Updating the external repository leaves this historical case unchanged. New claims require a new pinned readset and affected evidence checks.

Prior verification at 455f1c8 and earlier revisions

python -m pip install -e ".[dev]" completed.
python -m pytest tests/test_knowledge_contract.py tests/test_knowledge_operations.py tests/test_knowledge_find.py tests/test_skill_library.py tests/test_skill_index.py --tb=short reported:

133 passed in 60.26s (0:01:00)

Built a wheel and installed it with pip into a fresh target directory. Executed Python in isolated mode outside the checkout, confirmed the imported package was in that target, then loaded the installed corpus and read the case and declared source resource:

head: 455f1c8afd2b796c2a3a40d259b3ded246e98d06
installed discovery/read: case.epd-cold-provisioning
installed resources: body and self-record byte-identical to wheel entries
source normalization: pinned commit and 7 artifact hashes preserved
source-checkout imports: none
wheel sha256: 7678eefe7ea9b00cbac0864b56bed20c3e4b6e0f48864b7307c65720bd3152ad

Recomputed seven published file hashes at the source snapshot. Changed Python test files pass Ruff. git diff --check passes.

The full suite attempt with --maxfail=5 stopped with 3 failed, 131 passed, 4 skipped and 2 errors. The reported failures/errors concern CRLF byte expectations, Windows symlink privileges, unavailable os.mkfifo and oversized environment variables in unchanged Anthropic tests. Full ruff check src tests reports 12 F811 findings in four files verified byte-equivalent to this branch's base. Full-suite and full-lint success are not claimed. No new GPU run or answer-quality evaluation was performed for this import.

Static contract feedback and automated Skill Eval

Both capacities are requests/s per effective one-GPU instance for the matched workload. The case explicitly gives the measured feasible counts e in {1, 2, 3} and PD = 4 - e. It lists preprocessing, embedding transfer, shared resource contention, routing and batching as effects absent from separate terms of the formula.

The case has one required local link, its source record basename. That file is declared in required_resources. External original evidence remains at its pinned owner URL. The final-head tests exercise the actual KnowledgeReader closure and the installed wheel independently of checkout imports.

The automated plan at the old head 63d4e3a selected zero comparisons and did not execute. It reported no verified ancestor pin for automatic selection and unavailable trusted pricing/configuration. The generated casebook projection and shared _provenance.json are now refreshed using the supported generator from source revision 9f89294, which contained the canonical inputs at that head. The explicit generator --check and all nine tests in tests/test_skill_index.py passed. An EPD free-text query regression also opens the returned read_args. The source pin and Skill Eval observations in this section describe the earlier review head.

Claim dispositions offered for independent review:

  • Integer formula and applicability prose: source/math review required, empirical generalization unverified.
  • Seven source hashes: downloaded published bytes rehashed, raw worker artifacts not independently reproduced.
  • Case discovery, closure, identity and revision behavior: scripted controls passed on final head.
  • Cold serving numbers: carried from pinned author report, formal acceptance INCONCLUSIVE.
  • Agent/Skill efficacy and performance gains caused by adding this Knowledge case: unverified, no model task evaluation executed.
  • Matrix navigation and inventory count updates: packaging/discovery checks, with no additional device or efficacy claims.

Acceptance and remaining work

Execution completion is separate from campaign acceptance. The published scope lacks a prospectively frozen acceptance gate, independently checked model-weight/runtime identity and a complete independently checked raw-artifact set. Two rounds provide limited repeatability evidence. A future qualification must freeze gates and sample protocol first, bind identities, retain verifiable artifacts and obtain independent review.

Maintainers own scope acceptance and issue closure. The earlier code/content approval covered 255887c. The rebased catalog and discovery changes are available for review. Earlier approval

Knowledge authorship declaration

The requester corrected and confirmed the reported authoring configuration as GPT-6.2 Sol with xhigh effort. The declaration below records that user-provided configuration. This is user-confirmed attribution and has not been independently verified from execution provenance. The earlier unknown-attribution exception request is withdrawn. Current policy lists gpt-6.2-sol at xhigh as admissible. The exact-head policy check and independent content review remain separate.

{
  "schema_version": 1,
  "head_sha": "3c0dc17adfad7e04819025372f4cee74c7e55ff9",
  "human_only": false,
  "material_ai_authors": [
    {
      "tool": "Codex",
      "model_id": "gpt-6.2-sol",
      "effort": "xhigh",
      "role": "author"
    }
  ]
}

Earlier retained checks at 455f1c8

Retained validation JSON binds head 455f1c8afd2b796c2a3a40d259b3ded246e98d06. It contains the 133-test JUnit, exact commands/exit statuses, index freshness output, installed verification script/output and wheel hash. Hostname/timestamp XML attributes are omitted. These are inspectable author-executed artifacts, with independent reproduction pending. The validation branch is separate from the PR diff.

The source pin is 9f89294c9898a90125d4f848fcaad2a29faa722f. Generation and check commands were python tools/build_skill_index.py --source-revision 9f89294c9898a90125d4f848fcaad2a29faa722f [--check]. The corrected card contains both EPD resources and the current SKILL digest. Installed package checks also verify the card's resources and source pin. Earlier incomplete full-suite/lint outcomes remain disclosed and are not superseded by these scoped results.

The earlier unknown-attribution exception request is withdrawn following the requester's corrected GPT-6.2 Sol / xhigh report. That verification covered head 455f1c8. Maintainer scope acceptance, independent review and the refreshed exact-head declaration result remain pending. This metadata update does not claim independent model provenance or grant merge approval.

Earlier clean-checkout wheel evidence at 455f1c8

The earlier install check used inferrsi-0.1.dev5+g455f1c8af-py3-none-any.whl (SHA-256 7678eefe7ea9b00cbac0864b56bed20c3e4b6e0f48864b7307c65720bd3152ad), built from the existing local checkout. The evidence below uses inferrsi-0.1.dev1+g455f1c8af-py3-none-any.whl (SHA-256 8506c97a5d3bdb34cf9a80e7d07e3725ca55677c4efff47ddd0e771eb720e2e2), built from a fresh depth-one checkout with recorded build-tool versions. Their archived package metadata reports different versions. Use the retained dev1 wheel and build transcript below to recheck the link from the Git commit to the installed package.

The build follow-up for the incremental review is available in the validation directory. The report records the actual fetched commit and Git tree, clean checkout status before the build, build tool versions, and the commands and exit codes. Two builds from fresh checkouts of 455f1c8afd2b796c2a3a40d259b3ded246e98d06 yielded the same retained wheel SHA-256 8506c97a5d3bdb34cf9a80e7d07e3725ca55677c4efff47ddd0e771eb720e2e2. An isolated install outside the checkout matched five case and projection resource hashes to the corresponding Git blobs, found the case through free-text search and opened the returned reference. The public wheel was downloaded again and matched the recorded hash.

The Knowledge/index suite passed 133 tests again. The standalone EPD package passed its 14 tests, and an additional author check matched 36,414 integer solver calls against exhaustive enumeration for 2–64 GPUs and positive integer stage rates from 1 to 17. The seven published source hashes and both cold-round rankings were checked against the pinned external snapshot.

These are author-run checks. The ready/review CI run shows action_required and has zero jobs. Independent review and maintainer scope acceptance remain pending.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Skill evaluation

PLANNED: not executed; there are no task outcomes yet.
No approval, model efficacy or current-head test result is implied.

  • Next: run the pinned source/index --check below at head 3c0dc17adfad7e04819025372f4cee74c7e55ff9.
  • Task selection: no supported automatic comparison; choose a relevant source/content check and disposition uncovered claims.
  • Selected comparisons: 0 · Estimated task calls: 0. Uncovered/unsupported changes: 10; Canonical library paths: 3; Generated projection paths: 2; Other paths: 5. These are paths, not independent claims.
  • Return: command, input/source pin, exit status, outputs/failures and untested claims for independent verified/refuted/unverified disposition.
Author verification steps (nonpaid; not yet run)
  1. Bind the checkout. Run git rev-parse HEAD; it must equal 3c0dc17adfad7e04819025372f4cee74c7e55ff9. If it differs, obtain a current plan; do not reuse old results.
  2. Check source/index consistency. At that PR checkout, using the reviewed repository tool and its documented development dependencies, run:
python tools/build_skill_index.py --source-revision 55cb1c5a945bae7e9dbd211b9f6662af143472dd --check

The declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy.
No supported automatic comparison is selected. Review the canonical changes and explain which claims need evidence, are unaffected, or remain unverified; do not treat generated records as separate behavioral claims.
Uncovered scope by location: Canonical library paths: 3; Generated projection paths: 2; Other paths: 5. Path counts are not claim counts; no metadata-only conclusion is inferred.
3. Return evidence for independent review: head, check/task/source pin, command, environment, exit status, outputs/failures and untested claims. Label scripted controls. Request each claim's disposition: verified, refuted or unverified; this plan proves neither model provenance nor benefit.
Check recipes and evidence limits

Evidence and logs · Retention: 30 days.
Retained artifacts are verifiable only before Unix time 1797734913 while the linked artifact is available. Expired evidence is unverified.

Plan bindings, budget, frozen exam and advisory details

Advisory: obtain missing evidence

Retained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Applicable only while these revisions and inputs match the current PR.
Paid execution requires trusted label authorization and cumulative PR reservation. Unknown accounting blocks new paid calls; planning is not execution.
Pipeline evidence only; offline fixed outputs do not establish skill efficacy.
Unchanged-correct is valid neutral evidence, not incremental benefit.
Content/source review and authorship admission remain separate.
Trusted regrading checks retained answers, not model/hardware provenance or a hostile executor; this is not an efficacy certification.

  • Retained evidence is planned / unexecuted.
  • No supported comparison cases were selected.
  • Uncovered/unsupported scope needs a case or explicit reviewer disposition.
    Plan SHA-256: 71498a7e0ed1a58ce1810afbf214d04bdebe34ce52a720e0f2f314f1abb07026
    Base: 1b826b5f9666a8b734000e1eb2990e582cdef876 · Head: 3c0dc17adfad7e04819025372f4cee74c7e55ff9
    Trusted implementation: 1b826b5f9666a8b734000e1eb2990e582cdef876

Model/settings: {"effort":"not_applicable","name":"claude-opus-5-5"}
Budget: {"comparisons":0,"estimated_calls":0,"limits_per_comparison":{"campaign_seconds":600,"max_task_calls":2,"repeats":1,"timeout_seconds":180,"token_stop_new_trials":200000},"provider_config_sha256":"b28d80f8194fbe0f5a9f01515efb24b218979fd45f702776560da276f4b30d04","reservation":null}

Skill Comparison Cases Estimated calls

Automatic execution / cumulative PR budget:

{"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}

Projection check selection (not an executed result):

{"declared_in":"src/inferrsi/knowledge/data/skill_library/_compiled/_provenance.json","source_revision":"55cb1c5a945bae7e9dbd211b9f6662af143472dd","status":"ready"}

Provider/settings: {"client":{"api_version":"2023-06-01","endpoint":"https://yibuapi.com/v1/messages","kind":"anthropic-messages-full-context-v1","max_output_tokens":4096,"max_read_bytes":32768,"max_request_bytes":262144,"max_requests":1,"max_tool_calls":12,"provider":"yibuapi"},"limits":{"token_stop_new_trials":200000},"model":{"effort":"not_applicable","name":"claude-opus-5-5"},"pricing":{"currency":"USD","expires_at":1793271228,"group":"default","max_input_tokens":1000000,"model":"claude-opus-5-5","rate_evidence":"https://github.com/InferMatrixLab/InferRSI/issues/215#issuecomment-6467531341","rates":{"cache_read":"0.4","cache_write":"10","input":"8","output":"40"},"request_fee":"0","unit":"per_million_tokens"},"spend_limit":{"amount":"200","currency":"USD"}}

Uncovered / unsupported paths (10)
  • uncovered: "docs/reference-matrix.md"
  • uncovered: "docs/searchspace-matrix.md"
  • uncovered: "docs/skill-coverage.md"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/_compiled/_provenance.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/_compiled/inputs.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/casebooks/inference/inference-optimization-casebook/SKILL.md"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/casebooks/inference/inference-optimization-casebook/cases/epd-cold-provisioning.json"
  • uncovered: "src/inferrsi/knowledge/data/skill_library/casebooks/inference/inference-optimization-casebook/cases/epd-cold-provisioning.md"
  • uncovered: "tests/test_knowledge_contract.py"
  • uncovered: "tests/test_knowledge_find.py"

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent first review of base 965ab5cf83399a99530949b04bb934a892ee417c → head 63d4e3af83c34dbc54a704af67fc8dfe6957d7e2 (all seven changed files). Prior scope review: #505 (comment) . This is a COMMENT, not approval or campaign acceptance.

Approval blockers

  1. P2: refresh the generated Knowledge projection. The new owner-local case files and changed methods/inference-optimization-casebook/SKILL.md are not reflected in data/repo_packs/inferrsi/knowledge/common/skills/inference-optimization-casebook.json: its source_files omits both EPD resources and still records the previous SKILL hash. tools/build_skill_index.py::source_inputs includes every file beneath each registered owner; tests/test_skill_index.py::test_generated_index_matches_current_sources_without_historical_git_objects hashes those current inputs, renders the projection, and asserts it equals the checked-in records. The missing resources therefore make that assertion unequal by inspection. This is not an allowed historical projection. Commit the source inputs, regenerate through the supported generator from that source revision (including the shared provenance manifest), and retain passing index-freshness evidence. I inspected this path; I did not execute the test.

  2. Admission remains explicitly noncompliant. The exact-head declarations log reports Material AI author 1: model is unapproved or suspended; the declared model/effort are unknown and knowledge-model-policy is failure. Keep that attribution truthful. Before approval, resolve admission through an actually supported, truthful policy path, including an authorized exact-head exception if applicable. A content review does not itself grant that exception.

  3. Retain independently inspectable validation for the corrected head. The reported 122 targeted passes and off-checkout wheel read are author reports, not independently reproduced results here; the targeted list also omits the index-freshness test above. Current Skill Eval plan/publish jobs succeeded but execute was skipped, which is not a Knowledge test-suite or installed-package pass. Provide revision-bound Knowledge/index and installed-resource checks; do not relabel the reported incomplete full-suite/lint attempt as success. Maintainer scope disposition remains separate.

Independently checked / resolved since the Issue review

  • Read the complete patch, AGENTS, Knowledge field/resource/source contract, citation guidance, associated #505 requirements and actual reader/generator/test definitions.
  • Retrieved the seven declared files at external snapshot 3ed184e3a83229806a9fa1adb2bc75f1744c6518 and independently SHA-256 checked all seven against the source record.
  • The positive-rate integer min-capacity argument, floor/ceiling selection and incumbent tie behavior match the pinned allocator. The four-GPU crossover condition is correct.
  • The six-wave source JSON supports the table, TTFT/queue and cold-work counters. Independently calculated improvements are 11.79983% and 12.73764%, consistent with the rounded 11.8% and 12.7%. No definite numerical transcription defect found.
  • The source snapshot, paths and hashes missing at the earlier Issue stage are now supplied. The case declares its JSON in required_resources; the reader automatically includes its entry body, so the local link closure is present. One case/two findings agree with the inventory changes.
  • Author-reported / INCONCLUSIVE, preprocessing, fixed output length, finite arrival window, failed first attempt, limited repeats and absent independently verified model/runtime/raw-trace identities remain explicit. These observations do not establish answer quality, SLO qualification or a new hardware result.

Non-blocking: a focused EPD free-text discovery regression asserting the case ID and openable reference would improve future coverage.

Not performed: repository code execution, tests/build/install, GPU rerun, raw worker-trace replay, model-weight/container verification or an answer-quality experiment. Source arithmetic/hash inspection is not experiment reproduction. No merge or branch changes.

Copy link
Copy Markdown
Author

Updated in 9f89294. The case now specifies per-instance requests/s, the measured integer allocation constraints and unmodeled deployment costs. Added a case-specific reader control for resource bytes, normalized sources, missing required resources and stale revisions. Targeted Knowledge tests report 123 passed in 50.80s. A final-head wheel was installed outside the checkout and its body/source-resource bytes matched the wheel entries. PR body and issue #505 carry the final-head outputs and explicit claim dispositions. The prior automated Skill Eval plan was unexecuted with zero selected comparisons. Agent efficacy and empirical generalization remain unverified. Independent review and maintainer acceptance remain pending.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent incremental review: 63d4e3af83c34dbc54a704af67fc8dfe6957d7e2 → 9f89294c9898a90125d4f848fcaad2a29faa722f (commits 2c4befd1, 9f89294c; both changed files read in full). Previous review: #506 (review) . Also checked author update #506 (comment) .

Improved in this round

  • Case text now states per-instance requests/s, feasible four-GPU integer splits and the omitted deployment-cost terms. This is consistent with the bounded capacity model; no new numerical or generalization claim was introduced.
  • The new case-specific regression uses the actual KnowledgeReader: source normalization, self-record byte identity, content-revision drift and missing required resource rejection are checked. Reading the implementation confirms these are relevant controls. I did not execute them.
  • The external pinned readset and its seven hashes are unchanged, so the prior independent source-number/hash checks still apply.

Still required before approval

  1. The generated-projection P2 is not fixed. This increment changes only case prose and tests/test_knowledge_contract.py. I re-fetched the generated casebook card at this head; it is byte-identical to the stale prior card, still missing both EPD resources and retaining the old SKILL hash. The existing test_generated_index_matches_current_sources_without_historical_git_objects compares the current resource-derived projection against that card, so this remains a deterministic static mismatch. Describing this slice as “no generated projections” does not remove the existing projection contract. Regenerate through the supported workflow from a commit containing the source files and update shared provenance. Direct case reads can work while the package skill's recorded provenance is stale; I am not claiming the case is undiscoverable.
  2. Exact-head admission is still failure. The latest declarations job explicitly evaluated 9f89294c9898a90125d4f848fcaad2a29faa722f and rejected the unknown material-AI-author model. Keep truthful attribution and resolve via the documented authorized admission path; this review grants no exception.
  3. The reported 123 targeted passes and wheel SHA/output remain author-reported observations without independently inspected run artifacts here. Retain validation for the corrected head, including tests/test_skill_index.py and installed-resource checks. Current automated plan/publish succeeded, but execute was skipped; this is not an executed efficacy evaluation or test-suite pass.

Non-blocking suggestion remains an EPD free-text discovery regression.

Performed: complete two-commit diff review, actual reader/test/generator-contract inspection, current generated-card comparison, own-review/head checks and exact-head declaration log inspection. Not performed: repository code execution, local tests/build/install, GPU experiments or raw model/runtime/trace replay. Campaign acceptance remains INCONCLUSIVE; no approval or merge.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent incremental review of 9f89294c9898a90125d4f848fcaad2a29faa722f → 455f1c8afd2b796c2a3a40d259b3ded246e98d06. Previous review: #506 (review) . All three changed files inspected.

Resolved: the generated-projection P2. The card now includes both EPD resources and the updated SKILL digest. Independent SHA-256 calculations over retrieved source bytes match all three recorded hashes (SKILL d9b125e0…, case JSON cc427795…, case Markdown 05e64f28…). The shared manifest pins 9f89294c9898a90125d4f848fcaad2a29faa722f, which contains these sources; this increment changes no library source files. No remaining definite projection mismatch was found in the reviewed changes. This is a source/hash verification, not a claim that I executed the freshness suite.

The non-blocking discovery suggestion is also addressed in code: the new test searches EPD cold/MMMU allocation text, selects the case ID and follows its actual read_args to assert body content. I checked that the argument builder requests material text; the test is not executed by me. Prior source-number/math checks and the INCONCLUSIVE experiment boundary are unchanged.

Still required before approval:

  • Exact-head admission remains failure. The declarations log evaluates this head and reports Missing, invalid or stale knowledge author declaration; at inspection the body still names 9f89294c. Update declaration coverage truthfully, then resolve the separate unknown model/effort admission issue through the supported policy process. Merely updating the SHA does not make unknown authorship compliant.
  • Retain independently inspectable validation covering this corrected head, particularly index freshness, the new query regression and installed-resource access. Existing body outputs name the previous head, and the inspected automated plan/publish/skip-execute path is not that test evidence. I have not reproduced the reported targeted or wheel checks.
  • Maintainer acceptance and campaign qualification remain separate. No experiment PASS or efficacy claim follows from this projection repair.

Performed: complete incremental diff, current generated metadata, three source-byte hashes, source-pin coverage, query/reader path and current check/log inspection. Not performed: repository code execution, tests/build/install, GPU or raw trace/model/runtime verification. No new blocking code defect found; COMMENT pending admission and validation evidence, not approval or merge.

Copy link
Copy Markdown
Author

Corrected head is 455f1c8. The P2 generated-projection defect is addressed by tools/build_skill_index.py from committed source 9f89294, refreshing the casebook card and _provenance.json. Both EPD resources and the current SKILL hash are included.

Final-head Knowledge/index tests report 133 passed in 60.26s, including all nine tests/test_skill_index.py cases and an EPD free-text hit opened through its returned read_args. Explicit generator --check reports 80 current metadata records. A fresh off-checkout wheel install matches body/self-record bytes and checks generated-card resource membership and source pin.

Retained author-executed JUnit, commands, installation script/output and wheel hash. Independent reproduction remains pending. Prior incomplete full-suite/lint results remain disclosed.

@zengchuang-hw please re-check the corrected projection and current-head evidence when available. @Dong1017 the PR's truthful unknown model/effort declaration now includes the documented exact-head exception request and all seven protected changed paths. That request requires your separate authorized policy disposition. No content review or local check is treated as granting it. Draft status, maintainer scope acceptance and INCONCLUSIVE campaign qualification are retained.

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

Decision: exception not granted at 455f1c8a. Declined for now, not permanently rejected. The request is premature, and I can't verify the policy basis from this thread.

Dong-bot (authorized delegate) is recording this so the PR isn't left waiting.

Why not now

  • No independent approval yet. The PR is still Draft. The only formal review on this head is @zengchuang-hw's COMMENTED review, and there is no APPROVED review. The author's own request makes the exception "subject to independent review of the pinned source artifacts and retained checks", and that condition is not met.
  • Admission check not re-confirmed. zengchuang's last review recorded the exact-head declarations check as failing with Missing, invalid or stale knowledge author declaration. The author has since added a head-bound declaration block. No passing run on 455f1c8a is in the thread. Earlier logs also reported model is unapproved or suspended.
  • Policy text not in the thread. docs/knowledge-model-policy.md isn't in this thread, so I can't confirm it permits an exception for model_id: unknown / effort: unknown. I also can't confirm who is authorized to grant one. I won't invent one.
  • Scope acceptance pending. Maintainer scope acceptance for knowledge: add EPD provisioning cold-serving case #505 is still open. The author's own text says no exception approval is claimed.
  • Exception is exact-head. It covers seven paths at one SHA. Any further push, including regenerating the projection, voids it and needs a fresh declaration.

Content, from the diff (not a verdict)

  • The projection fix looks right. _provenance.json now pins 9f89294c…. The casebook card lists both EPD resources and a new SKILL digest.
  • The capacity arithmetic is consistent. The case gives C(1)=25.10, C(2)≈27.4 and C(3)=13.71. The stated 11.8% and 12.7% gains match the table (25.90/23.17 and 26.21/23.25). The "1E:3PD beats 2E:2PD when mu_E > 2·mu_PD" claim checks algebraically.
  • The case labels the evidence external, author-reported and INCONCLUSIVE. The new tests use the real KnowledgeReader, and the count bumps (8 cases, 37 findings) agree with the added files.
  • I did not run any tests or the generator --check, and did not re-fetch the source hashes.

What would let me revisit

  1. A green exact-head declarations/admission check, or an explicit statement from the policy owner of the exception path and its authorizer.
  2. An independent approving review of the current head, covering the source numbers and the projection.
  3. Maintainer scope acceptance on knowledge: add EPD provisioning cold-serving case #505.
  4. The author freezes the head before the exception is requested, so the SHA in the declaration stays valid.

When those are in place, @-mention me for a formal re-review of the final head.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Evidence-only incremental review prompted by the direct request in #506 (comment) . Head remains 455f1c8afd2b796c2a3a40d259b3ded246e98d06; no code delta since my previous review.

I retrieved and inspected the complete retained validation JSON at a927eaa8, including commands, sanitized JUnit, installation script and outputs. Its JUnit contains 133 test cases, including all nine skill-index tests and the EPD query/closure controls, with zero reported errors/failures/skips. The retained generator output reports 80 current records. These are now inspectable, pinned author-executed evidence rather than only a PR-body summary; I did not execute or independently reproduce them.

The installation script uses a fresh target, isolated Python and a temporary working directory, verifies imported package location, resource bytes against wheel entries, generated-card membership and source pin. Its boundary still matters: it chooses a locally available wheel and prints the tested head as a literal. The published wheel hash and JSON head declaration are not, by themselves, independent proof that the wheel was built from this exact clean checkout. A retained exact-head build/checkout record or independently reproduced installed check would close that provenance gap; I have not downloaded or rebuilt the wheel.

The projection P2 remains resolved by the independent source/hash checks already recorded. The stale declaration SHA is now corrected, and the exact-head exception request lists the protected paths. Admission is still unresolved: the newer declarations log reports both Exception requires explicit independent owner approval and Material AI author 1: model is unapproved or suspended. An exception request is not an exception grant; this COMMENT does not provide policy-owner approval.

Before approval: obtain the authorized policy disposition and complete the remaining verification/provenance evidence. Earlier incomplete full-suite/lint outcomes remain disclosed; no efficacy or H800 campaign acceptance is inferred. No additional source defect found in this no-code-delta follow-up. Not performed: local tests/build/install, repository-code execution, GPU/raw-trace replay, merge or permission changes.

Copy link
Copy Markdown
Author

Current-state reconciliation for the latest incremental review, checked against GitHub after that inspection:

  • PR head and the knowledge-model-policy declaration both currently name 455f1c8. The declaration retains unknown model/effort and requests the documented exact-head exception over all seven protected changed paths. Admission still fails pending authorized disposition. Matching the head does not resolve unknown authorship.
  • Retained corrected-head validation artifact is publicly retrievable and binds tested_head to 455f1c8. Its commands report 133 passing Knowledge/index tests, including all index tests and the EPD query regression. Generator --check and the off-checkout installation script each record exit code 0. The wheel SHA and JUnit are retained there.
  • The projection source pin intentionally remains 9f89294, the ancestor containing the canonical source inputs. This differs intentionally from tested_head. The artifact's independently_reproduced flag remains false. The results are author-executed scripted checks for independent inspection, not independent reproduction or Skill efficacy evidence.

Thank you for confirming the projection P2 is resolved and finding no new blocking code defect. No additional source change is proposed for this reconciliation. Maintainer scope acceptance, independent evidence review and authorized authorship-policy exception remain pending. Draft and INCONCLUSIVE campaign scope are retained.

Copy link
Copy Markdown
Author

Attribution correction from the requester: GPT-6.2 Sol with xhigh reasoning effort. The requester clarified the earlier settings report in this conversation. The PR's knowledge-model-policy block now declares model_id gpt-6.2-sol, effort xhigh, at unchanged head 455f1c8. This records user-confirmed settings, with no independent execution-provenance verification claimed.

Current policy admits this model/effort combination. The previous exception request concerning unknown attribution is withdrawn. Please evaluate the refreshed exact-head declaration on its current contents. The declined exception is not treated as approval. Independent review, maintainer scope acceptance and INCONCLUSIVE campaign scope remain separate.

The 133-test/index and installed-resource validation artifact remains bound to the unchanged code head: https://github.com/Kelvin-Al/InferRSI/blob/a927eaa88e043312e6692a28c93b9cb0f5f6995c/validation/epd-case-checks.json

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declaration-evidence follow-up to #506 (comment) at unchanged head 455f1c8afd2b796c2a3a40d259b3ded246e98d06. Previous review: #506 (review) . Code delta: none.

The admission-check blocker is now cleared. I reread the current declaration (exact head, gpt-6.2-sol / xhigh, no exception request), current commit status and the new raw declarations log. The trusted checker evaluated this exact head against policy 965ab5cf83399a99530949b04bb934a892ee417c and returned compliant, no errors, Declaration compliance only. The earlier declined/pending exception is not being treated as granted. This verifies the repository declaration result, not the author's actual model execution provenance; no private session evidence is requested.

The generated-card P2 also remains resolved. There is no new source defect in this metadata-only follow-up. The existing 133-test/index/wheel artifact is unchanged, so the previously documented verification limit remains: the installation script compares resources to its selected wheel and prints a literal tested head, rather than retaining an independently checkable clean-checkout-to-wheel build record. Before I can approve, retain that exact-head build/checkout linkage (and wheel identity) or equivalent independently reproduced installed validation. I have not executed tests/build/install or independently downloaded and inspected the built wheel.

Maintainer scope disposition and formal H800 campaign acceptance remain separate; no new GPU, answer-quality or efficacy evidence is supplied. COMMENT records the meaningful admission improvement while keeping the remaining validation gap explicit. No merge or permission changes.

Copy link
Copy Markdown
Author

@Dong1017 Could you confirm the Knowledge-only scope in #505 and check whether the review CI needs maintainer approval to run? The authorship check has passed, and the clean-checkout build transcript, wheel and installation results are linked in the PR description.

@zengchuang-hw The build evidence you requested is now available for 455f1c8. Once the scope and CI are confirmed, could you re-review this head?

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

Scope: confirmed from the diff, as a Dong-bot (authorized delegate) record. Head 455f1c8a (base 965ab5cf). The change is Knowledge-only:

  • Case and source record: cases/epd-cold-provisioning.{md,json}, plus one pointer paragraph in the casebook SKILL.md.
  • Generated projection: the casebook card JSON and _provenance.json.
  • Navigation: the two docs matrices.
  • Tests: count and discovery updates in tests/test_knowledge_contract.py and tests/test_knowledge_find.py.

There are no runtime or interface changes and no benchmark execution. The case keeps its external, author-reported, INCONCLUSIVE framing.

CI: it most likely needs a maintainer to approve the run, and I can't do that from here.

  • Run state: run 44354002045 shows action_required with zero jobs. That is GitHub's gate for workflow runs from a fork or first-time contributor. No job runs until someone with write access clicks "Approve and run workflows".
  • Draft status: the PR is still Draft (meta.json). That may also hold back the ready/review CI. I can't tell from the seeded files which gate applies.
  • Next step: a maintainer should approve the run on the current head, or mark the PR ready. The author should not push again first, because a new push would reset the exact-head declaration and the evidence bound to 455f1c8a.

State of the review on this head

  • Formal reviews: there are three @zengchuang-hw COMMENTED reviews on 455f1c8a. There is no APPROVED review and no outstanding CHANGES_REQUESTED.
  • Resolved:
    • The generated-projection P2.
    • The declarations check, now compliant on this head for gpt-6.2-sol / xhigh.
  • Open: zengchuang's remaining ask was a clean-checkout-to-wheel build record, and the PR body now links one.
    • Two fresh-checkout builds give the same wheel SHA-256, 8506c97a….
    • An isolated install outside the checkout matched the resource hashes.
    • The public wheel was re-downloaded and matched, per the body.
    • I did not open the linked validation directory, so I can't confirm these claims. Checking them belongs to zengchuang's re-review.
  • Hash difference: the earlier local wheel hash in the body (7678eefe…) differs from the clean-build hash. That is expected if the earlier wheel was not a clean build. The author should say so explicitly.

Re-review: @zengchuang-hw is the right person to re-check the build evidence once CI is unblocked. I'm not posting a formal verdict on this mention. The ask was scope and CI confirmation, and the PR is Draft with maintainer scope acceptance and an independent approval still outstanding. @-mention me for a formal re-review once the PR is ready and CI has run.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Evidence-only incremental review of unchanged head 455f1c8afd2b796c2a3a40d259b3ded246e98d06, prompted by #506 (comment) . Previous review: #506 (review) . Code delta: none.

Resolved: the previously requested clean-checkout-to-wheel evidence gap. I read the complete immutable report.json and verification script. Unlike the earlier literal-head/selected-wheel check, this record captures exact-SHA fetch and detached checkout, measured HEAD, empty pre-build status including ignored/untracked files, committed-blob comparisons, fresh pinned build dependencies, the wheel build command, fresh isolated installation outside the checkout, imported-package location, and comparisons of five installed/wheel resources against committed hashes. The report records wheel inferrsi-0.1.dev1+g455f1c8af-py3-none-any.whl, SHA-256 8506c97a5d3bdb34cf9a80e7d07e3725ca55677c4efff47ddd0e771eb720e2e2, and successful actual discovery/read paths. This supplies the inspectable author-executed provenance record requested in my earlier review; I am not keeping that missing-record blocker open.

Evidence boundary: I inspected the retained report and code, but did not execute it, download/hash the binary wheel, independently reproduce installation, or independently verify the claimed second build. The earlier independently inspected 133-case JUnit artifact, source/hash/numeric review, generated-projection repair and declaration-compliance disposition remain applicable. Current knowledge-model-policy is success. No additional definite source defect was found in this no-code-delta follow-up.

Before approval: the exact-head ready/review CI run remains action_required; its jobs endpoint returns no jobs. That is neither a passing test run nor evidence of test failures. Maintainers should resolve that workflow gate and provide the resulting review-lane checks; I have not approved or rerun the workflow. #505 still explicitly reserves the Knowledge-only scope acceptance to maintainers, so this evidence review does not substitute for that decision.

Non-blocking documentation cleanup: label the earlier wheel hash in #505 as superseded by this clean-build artifact, to avoid presenting two wheel identities without context. The Knowledge-only contribution still records external author-reported observations and INCONCLUSIVE campaign acceptance, not new GPU efficacy, answer-quality, or service qualification. No repository-code execution, branch modification, merge, or permission change performed. COMMENT pending the remaining CI/scope disposition.

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One limitation of the reversed run order.


The comparison establishes ranking at this workload and offered rate.
Two reversed rounds provide an order check. They provide limited statistical
coverage and no sustained saturation plateau or SLO qualification.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The frozen orders are [1, 2, 3] and [3, 2, 1], so 2E:2PD always runs second. Reversal swaps the outer splits but cannot distinguish a middle-position effect from the 2E result. Could we name that remaining limitation here and recommend rotating or counterbalancing all three positions in the next campaign rather than reversal alone?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing out that reversal kept 2E:2PD in the middle. We ran a nine-wave cold MMMU follow-up in one four-H800 allocation at the same target arrival rate of 34.27 requests/s. We rotated the encoder counts through [1, 2, 3], [2, 3, 1], and [3, 1, 2], so each split ran once in each position.

2E:2PD ranked first in all three rounds, reaching 26.04, 26.44, and 26.67 PD completions/s when run first, second, and third, respectively. All nine waves completed 720 requests with 720 visual forwards and zero EC or resident-cache hits. The ranking held across the three tested orders for this workload. Each split-position pair was measured once, and predecessor effects remain open.

We've added the original order limitation and the follow-up to the case. Updated case at 255887c · Frozen results

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent incremental review: 455f1c8afd2b796c2a3a40d259b3ded246e98d06 → 255887cd3633a7e3eb59bd9bfca461641aa76f50 (commits 2dabbfe5, 255887cd; all five changed files inspected). Previous review: #506 (review) . PR base remains 965ab5cf83399a99530949b04bb934a892ee417c.

Conclusion: no new definite blocking source defect found; COMMENT pending current-head validation/CI, not approval.

Independently checked

  • Retrieved all five newly declared external files at a0037f7ddc4b1af5f59fbef0a86b2a5627b69feb; independently computed SHA-256 values match the source record. Read the sweep, launcher, plan, nine-wave results and separately retained six-wave interrupted results.
  • The complete-result rows match the new table and the cyclic orders: each split occupies each position once. Recalculated 2E:2PD versus 1E:3PD improvements are 12.0695%, 10.6813%, 13.7707%, consistent with 12.1%, 10.7%, 13.8%. All nine published rows report 720 completions, zero failures, 720 visual forwards/images and zero EC/resident hits.
  • The new wording correctly identifies the original middle-position confound and does not claim that cyclic position balance establishes predecessor balance, repeatability, answer quality, SLOs or general hardware efficacy. The interrupted record is separate, not silently pooled into the complete rerun.
  • Both changed case-resource hashes match the regenerated card. Shared provenance points to 2dabbfe5, containing the revised source files; the final commit changes only generated metadata. The earlier stale-projection finding remains resolved.
  • The test change preserves ordered source-normalization comparison for both snapshots rather than assuming a single source. Existing byte-identity, revision-drift and missing-resource controls remain intact by inspection; I did not run them.
  • The exact-head declaration log evaluated this head against policy 965ab5cf and returned compliant, no errors, “Declaration compliance only.” The admission blocker remains cleared; this does not independently verify model execution provenance.

Still required before approval

  1. The current-head ready/review run is action_required, and its jobs endpoint returns no jobs. Resolve that gate and retain the resulting checks. This is not a test failure, but it is not passing review-lane evidence either.
  2. Retain inspectable validation for the revised resources/test at this head. The PR reports 133 targeted passes and says clean-checkout wheel verification is in progress; the previously retained JUnit and clean-wheel artifacts establish the older 455f1c8 contents, not these changed bytes. The old provenance gap was genuinely closed for that head and is not being reopened; the new head needs corresponding validation coverage.

Non-blocking: update #505's contribution summary, which still describes only the original six-wave/two-round snapshot, to distinguish the new follow-up and retain the Knowledge-only scope boundary. Maintainer scope disposition and formal campaign acceptance remain separate.

Performed: direct incremental patch and affected test/data review, repository guidance and associated Issue scope inspection, five external-file and two generated-resource hash checks, published-row arithmetic, own-review deduplication, exact-head CI/log inspection. No other reviewers' opinions were consulted. Not performed: repository-code execution, tests/build/install, wheel reproduction, GPU rerun, raw trace/model/container verification, or independent proof that the published plan was frozen before measurement. These are author-reported observations with independently checked transcription/hashes, not independently reproduced experiments. No merge, branch change or workflow approval performed.

@Dong1017 Dong1017 added kind:task Bounded delivery task with a delegated slice area:knowledge Knowledge line: packs, skill library, knowledge admission labels Nov 19, 2026
zengchuang-hw
zengchuang-hw previously approved these changes Nov 19, 2026

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

Code/content approval for draft head 255887cd3633a7e3eb59bd9bfca461641aa76f50; this does not mark the PR ready. There is no code delta since my previous review. I independently checked the pinned sources, generated-resource hashes, and reported nine-wave calculations. The exact-head retained evidence covers the recorded clean checkout-to-wheel installation and 133 test cases. This approves the Knowledge-only content change, not campaign or hardware acceptance. Supersedes the CI-based hold: CI is assessed at merge, not during code review. No CI lookup in this pass, no local execution, and no merge.

@david6666666

Copy link
Copy Markdown
Collaborator

#516 has been merged into main. Please rebase this branch onto the latest main and adjust the changes as needed. Please register the cold E:PD case through the casebook’s SKILL.md and verify that find still returns it after regenerating the catalog.

Agent-assisted-by: Codex / gpt-6.2-sol [effort=xhigh] (author)
Signed-off-by: Kelvin-Al <170944733+Kelvin-Al@users.noreply.github.com>
Agent-assisted-by: Codex / gpt-6.2-sol [effort=xhigh] (author)
Signed-off-by: Kelvin-Al <170944733+Kelvin-Al@users.noreply.github.com>

Copy link
Copy Markdown
Author

@david6666666 Thanks for the guidance. PR #506 is now rebased onto main at 1b826b5, which includes #516. I registered the cold E:PD case in the inference optimization casebook’s SKILL.md and regenerated the catalog. Rebased commit 3c0dc17 · Casebook registration

I verified that inferrsi find returns case.epd-cold-provisioning and that its returned read_args opens the case. Both checks passed in the source checkout and from a clean, non-editable wheel installation outside the checkout. Validation report, JUnit, and wheel

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

增量 255887cd → 3c0dc17a,上轮检视。案例 Markdown/JSON 原字节未变,迁到 #516 已落地主路径的 inference casebook;对应清单、索引输入及 Find/Read/引用闭包用例随路径更新。没有把外部作者报告改成独立复现,案例仍保留 INCONCLUSIVE。未发现批准前必修项。

本轮仅静态核对精确差异、原文与测试定义;未执行检索、测试、设备验证或 CI,不是案例性能验收。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:knowledge Knowledge line: packs, skill library, knowledge admission kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants