Skip to content

feat(use): collect scored paths and probes (WIP) - #92

Draft
sysuTobo wants to merge 4 commits into
FeathBow:mainfrom
sysuTobo:feat/use-run-collection
Draft

sysuTobo wants to merge 4 commits into
FeathBow:mainfrom
sysuTobo:feat/use-run-collection

Conversation

@sysuTobo

@sysuTobo sysuTobo commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Status

Draft — not ready for review. This is the first phase of #80 (use run collection). We do not have an MLX host, and no real vLLM/MLX use run followed by use admit has collected a score and a probe yet, so the WIP marker and draft state remain until that end-to-end run is available.

Why

invar use run previously refused numerical requirements that need scored paths or full-vocabulary probes. Those records had to be collected manually with invar compare numerical / invar score, so a contract run and a training history could not reach the same admission path.

Refs #80.

What

  • Add a structured execution-plan collection: scored path sides, probe paths with frozen steps, and scorer endpoints.
  • Validate collection against the contract before launching anything: required paths, exact probe steps, scorer adapters, and HF probe refusal.
  • Collect each score/probe after the paired inference using Score.prepare / Score.bind / Score.Worker.run, retaining its own transcript.
  • Emit score/probe flags in runs.json; support multiple probe paths with --probe-set.
  • Retain source log digest, target policy, score-log digest, elapsed time and the recorded probe memory metric/scope.
  • Predict score/probe premises in use prepare.
  • Add collection validation and premise tests.

Memory reporting

cost.json reports probe memory readings with their call, path, target, metric, recorded scope, stage, host, pid, device and bytes. peak_probe_device_allocator_bytes is the largest reading over probe scoring intervals, not the peak of the whole run or process RSS. A plain score produces no such measurement; when no probe measurement exists the memory fields are null and memory_status says it was unmeasured.

Validation

  • cabal test: PASS.
  • Full HF worker suite: 225 tests OK (2 skipped).
  • fourmolu, hlint, test/types.sh ghc, cabal check: clean.

Not ready / remaining

The use execution plan now carries a structured collection: which side's path
is scored, which path and frozen steps are probed, and the scorer endpoint for
each implementation. The decoder accepts the old empty collection and refuses
non-empty legacy fields, whose values carried no scorer.

Collection is validated against the contract before any process starts:
requirements must have their score or probe entry, step lists must match the
contract, scorer adapters must match the declared policies, and HF probe plans
are refused with the named no-probe limitation. Score and probe calls reuse
Score.prepare/bind/admit and Worker.run, retain their own transcripts, and are
bound to the admitted inference they score. use run emits score/probe flags in
runs.json, retains partial records on failure, and reports call counts, wall
time and record/log sizes in cost.json.

Premise prediction now includes score and probe obligations, negative scored
and KL budgets are rejected, and the walkthrough documents the new collection
shape. Tests cover collection gaps, conflicting probe steps, HF refusal,
score/probe premises and negative budgets.

Signed-off-by: bohaotu <bohaotu@pony.ai>
Collection records now retain the source log digest, target policy, score log
digest and elapsed time, and plan.json records every allocated score/probe call
with its scorer endpoint. cost.json reports per-call seconds and the peak memory
reported by score/probe resource records, marking memory unmeasured when no
worker measurement exists.

A probe-set now rejects the old single-probe path and steps options instead of
silently ignoring them. Collection tests cover missing scorers, extra score
paths and duplicate probe paths.

Signed-off-by: bohaotu <bohaotu@pony.ai>
@sysuTobo
sysuTobo marked this pull request as ready for review November 4, 2026 13:27
@FeathBow

FeathBow commented Nov 6, 2026

Copy link
Copy Markdown
Owner

I found five issues on the current branch:

  • Transcript.file adds a newline to incomplete output, so the saved transcript no longer preserves the original bytes.
  • Collecting probes for both paths produces null endpoints in runs.json. Endpoint lookup needs to match both the source path and target.
  • Two KL requirements on the same path are rejected as conflicting even when their probe steps are identical.
  • KL comparison with --probe-set cannot run: the relation requires --probe-path, but the probe parser rejects that combination.
  • peak_rss_bytes contains device allocator measurements, not process RSS. Please retain the actual metric and its scope.

I reproduced the first four and traced the fifth to the worker measurements. The current tests miss these cases.

Transcript.file now writes complete records with their newline but writes the
incomplete final fragment without adding one, preserving the worker's bytes.

Probe-set endpoint lookup is scoped to the source path as well as the target,
so collecting both probe paths no longer produces null endpoints. Identical
probe steps on repeated KL requirements are no longer treated as conflicting,
and probe-path may accompany probe-set so KL relations can select their source
path.

Cost reporting now names the actual device allocator metric and scope instead
of calling it process RSS. Tests cover the transcript fragment and identical
KL steps.

Signed-off-by: bohaotu <bohaotu@pony.ai>

@FeathBow FeathBow left a comment •

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. I checked the new commit against the five points from my last comment.

Fixed:

  • The file transcript now keeps the trailing fragment byte for byte, and the new test covers it.
  • Probe endpoints are now looked up by both source path and target, so collecting both probe paths no longer produces null endpoints.
  • Repeated KL requirements with identical steps are accepted, and the new test covers it.
  • --probe-path can now accompany --probe-set, so a KL comparison over a probe set can run.

Still open:

  1. The help text now says "KL requires --probe-path or --probe-set". selectedRelation still requires --probe-path for both KL relations, so --probe-set on its own is refused. The --probe-path description also says it selects the source path for the probes, but with --probe-set it only selects the path of the KL claim. Something like "KL requires --probe-path; the probes come either from reference-probe-/candidate-probe- at --probe-steps or from --probe-set" would match the code.
  2. memory_scope is a fixed label ("mlx device allocator", "torch.cuda worker device allocator"), not the scope the measurement was recorded under. The records already carry it. Each vLLM worker entry has its own scope (permit to completion, with loading and serialization excluded) along with host, pid and device. The MLX reading covers only the cross_score stage. So peak_device_allocator_bytes is the largest reading taken over one probe's scoring interval, with loading excluded; it is not the peak of the run, which is how the name reads. Please carry the recorded scope or stage, and the call and device of each reading, into memory_measurements, and either describe the top-level figure that way or drop it. Also, only full-vocabulary probes carry measurements; a plain score's output admits none. So "A score or probe worker that reports a resource measurement" in the walkthrough, and score-probe-device-allocator, should both say probe.
  3. The PR description still says cost.json reports peak memory. Please update it to describe what is now reported.

The title and description still mark this as WIP, and no real use run followed by use admit has collected a score and a probe. Before the WIP marker comes off, I'd like to see one such run on MLX or vLLM, to show that the collected records are admitted end to end. I'll take another look after that.

KL now derives its source path from a probe set with exactly one path, and the
help text describes when --probe-path is required. Probe memory readings retain
their recorded stage, scope, call, path, target, host, pid and device; the
top-level figure is named as a max over probe intervals rather than run RSS,
and plain scores no longer claim to contribute memory measurements.

The walkthrough now documents the probe-only scope and the named device
allocator metric.

Signed-off-by: bohaotu <bohaotu@pony.ai>
@sysuTobo
sysuTobo marked this pull request as draft November 13, 2026 08:12
@FeathBow

Copy link
Copy Markdown
Owner

Thanks. I checked the new commit against the three open points.

The KL path selection is resolved. A probe set with one path now selects the KL path, a set with two paths asks for --probe-path, and the usage text says so. I ran both cases against the command. The PR description now describes the memory fields as well.

The memory readings still differ from the records in a few places, and the first one comes from a mistake in my last comment:

  1. I wrote that the MLX reading covers only the cross_score stage. That was wrong. Four records in an MLX probe's measurements carry peak_active: load, verify_before, cross_score and verify_after. memoryReadings turns all four into readings, so on MLX peak_probe_device_allocator_bytes can be the load reading, while the description and walkthrough call it the largest reading over probe scoring intervals. Keeping only the cross_score record would match the description. That is also the one record per probe that the resource parser checks.
  2. For vLLM, stage is read from each worker entry, but a worker entry has no stage field: the resource parser accepts exactly host, pid, device, seconds, peak_allocated, peak_reserved and scope. The stage is on the enclosing record, so every vLLM reading has "stage": null.
  3. MLX records carry no scope, host, pid or device. The reading fills in "mlx device allocator" as its scope and "mlx" as its device, where "mlx" is the record's allocator value. The walkthrough and description say each reading carries its recorded scope and device. I'd leave those fields null for MLX and report allocator from the record. On vLLM the parser requires the exact worker scope, so the "torch.cuda worker device allocator" fallback is never used and can go.
  4. Each side's scorer can be MLX or vLLM independently. With one of each, the top-level maximum compares peak_active with peak_reserved. memory_measurements already lists every reading with its call, path and target, so I'd drop the top-level figure and the fields that describe it. If you'd rather keep it, please restrict it to readings of one metric and describe it that way.

The build and tests pass on my side, and CI is green.

The WIP status and the request from my last comment stand: one real use run followed by use admit on MLX or vLLM that collects a score and a probe and has them admitted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants