Repository navigation
Evaluate every Atomic Skill with the v1 protocol - #775
Yuan-Xinyi merged 2 commits into
Conversation
Yuan-Xinyi
added
enhancement
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| "push_contact", | ||
| contact.start, | ||
| contact.stop, | ||
| frozenset({"object_pose"}), |
There was a problem hiding this comment.
The shipped push cases deliberately leave 2–3 mm of clearance at the end of contact. The object starts moving during push, but push_contact samples only contact. This reports contact_established=False even when the following push reaches its goal. Extend the measured-motion proxy into the part of push where contact actually happens.
Prompt To Fix With AI
This is a comment left during a code review.
Path: scripts/benchmark/motion_generation/scenarios/atomic_skill_providers.py
Line: 1860-1863
Comment:
**Successful pushes fail contact**
The shipped push cases deliberately leave 2–3 mm of clearance at the end of `contact`. The object starts moving during `push`, but `push_contact` samples only `contact`. This reports `contact_established=False` even when the following push reaches its goal. Extend the measured-motion proxy into the part of `push` where contact actually happens.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
| measurements={"grasp_established": moved > 1.0e-5}, | ||
| method="object_axis_motion_proxy", | ||
| source="measured_proxy", |
There was a problem hiding this comment.
grasp_established requires the object's axis to rotate during manipulate. A valid AxisAlign case can start with its axis already pointing at target_axis. The robot still grasps and lifts it, but no rotation is needed. That successful case fails grasped because moved is zero. Collect grasp evidence from the gripper rather than requiring an axis change.
Prompt To Fix With AI
This is a comment left during a code review.
Path: scripts/benchmark/motion_generation/scenarios/atomic_skill_providers.py
Line: 251-253
Comment:
**Aligned objects fail grasp**
`grasp_established` requires the object's axis to rotate during `manipulate`. A valid `AxisAlign` case can start with its axis already pointing at `target_axis`. The robot still grasps and lifts it, but no rotation is needed. That successful case fails `grasped` because `moved` is zero. Collect grasp evidence from the gripper rather than requiring an axis change.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| direction = torch.sign(grasp_qpos - open_qpos) | ||
| closed = (final - open_qpos) * direction | ||
| remaining = (grasp_qpos - final) * direction | ||
| return ((closed >= gap) & (remaining >= gap)).all(dim=-1) |
There was a problem hiding this comment.
_gripper_holding requires every joint to move between the open and grasp commands. For a valid multi-joint hand with one unchanged joint, direction is zero for that joint. Both distances are then zero, so .all() always returns false. The newly migrated single-arm grasp and retention checks reject successful grasps on these hands. Ignore unchanged joints when checking closure, while requiring evidence from the joints that do close.
Prompt To Fix With AI
This is a comment left during a code review.
Path: scripts/benchmark/motion_generation/scenarios/atomic_task.py
Line: 924-927
Comment:
**Unchanged joints reject grasps**
`_gripper_holding` requires every joint to move between the open and grasp commands. For a valid multi-joint hand with one unchanged joint, `direction` is zero for that joint. Both distances are then zero, so `.all()` always returns false. The newly migrated single-arm grasp and retention checks reject successful grasps on these hands. Ignore unchanged joints when checking closure, while requiring evidence from the joints that do close.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
49646b8
into
codex/spatial-generalization-tests
2 participants
Stack
codex/spatial-generalization-testsDescription
This PR migrates the atomic-task benchmark framework from #642 onto the evaluator protocol introduced by #712. It intentionally migrates framework only: no historical benchmark results, reports, JSONL artifacts, videos, or measured success rates are included.
Common infrastructure added for every skill:
StageEvidencegenerated only from replay measurements; missing or non-finite evidence fails only its environment row instead of aborting the batch.atomic-skill-v1stage criteria, standard failure codes, and focused positive, negative, boundary, terminal-settle, and partial-batch regression coverage.source="measured_proxy"; raw pose and joint measurements remainsimulation_state.Coverage by Atomic Skill
atomic-skill-v1stages and pass conditionsMoveEndEffectorMoveJointsPickUplifted: object lifted ≥ 4 cm and stayed above the no-drop floor.
held: object remained retained after settling.
MoveHeldObjectsettled: terminal object error remains ≤ 1 cm / 5° with no drop.
held: grasp remains retained after settling.
Placeplaced: full 3D position error ≤ 1 cm; tilt error ≤ 5° by default, or full rotation error ≤ 5° when target yaw is enabled; no drop.
stable: settle drift ≤ 1 cm / 5° with no drop.
Presspressed: peak signed stroke reaches ≥ 80% of the available stroke.
Slideeffect_joint_signdirection.slid: signed requested displacement error ≤ 1 cm.
released: gripper release proxy succeeds. Opposite-direction travel cannot pass.
Twisttwisted: signed executed TCP rotation about the requested axis has ≤ 5° error.
released: gripper release proxy succeeds. Opposite-direction rotation cannot pass.
OpenDooropened: final hinge-angle error ≤ 5°.
released: gripper release proxy succeeds.
Poursettled: terminal lift remains ≥ 4 cm with no drop.
held: grasp remains retained after settling.
AxisAlignaligned: actuation-end axis error ≤ 5°, lift ≥ 4 cm, and no drop.
settled: terminal axis error remains ≤ 5°, lift remains ≥ 4 cm, and no drop.
held: grasp remains retained.
PushObjectpushed: positive displacement toward the goal and actuation-end in-plane error ≤ 1 cm.
on_support_plane: settled in-plane error remains ≤ 1 cm, support-plane error ≤ 1 cm, and tilt ≤ 5°.
HandOvertransferred: both hands hold it with no drop.
handed_over: giver releases while receiver retains it.
receiver_released: receiver releases at the terminal target.
placed: final XY error ≤ 3 cm, signed Z error in [-10 cm, +3 cm], and no drop.
CoordinatedPickmentlifted: object lifted ≥ 4 cm with no drop.
moved: actuation-end object error ≤ 1 cm / 5° with no drop.
settled: terminal object error remains ≤ 1 cm / 5° with no drop.
held: both hands retain the object after settling.
CoordinatedPlacementreleased: placing hand releases while the support hand remains holding.
stable: settled relative pose still matches the target within 1 cm / 5°, settle drift stays within 1 cm / 5°, and no drop occurs.
Refs #642.
Type of change
Screenshots
Not applicable; this is benchmark framework and evaluator logic.
Validation
black --check --diff --color ./— 9007 files unchanged withblack==26.3.1tests/benchmark/motion_generation— 293 passed, 1 skippedpython docs/scripts/check_api_docs.py— 2487/2487 exports documentedpython .agents/skills/project-dev-context/scripts/context.py affected --base origin/codex/spatial-generalization-tests --explain— no affected topics in this PR layergit diff --checkThe focused tests use an in-process compatibility shim because the local DexSim 0.5.0 environment does not expose
dexsim.types.RTRenderMode; the shim is not part of this change. No live simulator/GPU benchmark run or result generation was performed.Checklist
black .command to format the code base.python docs/scripts/check_api_docs.py); no public API docs changed.