Skip to content

feat(cameras): ghost overlay of a saved reference frame to realign the rig - #139

Merged
nicolas-rabault merged 1 commit into
huggingface:mainfrom
ravediamond:camera-reference-overlay
Oct 29, 2026
Merged

nicolas-rabault merged 1 commit into
huggingface:mainfrom
ravediamond:camera-reference-overlay

Conversation

@ravediamond

@ravediamond ravediamond commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Closes #138

If you stash the arm and cameras between sessions, putting the framing back by eye is unreliable, and framing that drifts between episodes hurts policy training. The arm is in view, so one reference frame per camera captures both camera framing and arm pose.

In the recording dialog, each camera now has a Set reference button. It saves the frame currently shown in the preview. After that the reference is drawn as a translucent ghost over the live preview, with a slider for its opacity, so you can nudge the rig until the two line up. Clear removes it.

How it works:

  • Frontend only for the camera side. The frame is grabbed from the existing browser preview, so the backend never opens a camera and can't fight the browser or the recorder for the device.
  • The backend just stores the PNG: GET/PUT/DELETE /robots/{name}/camera-reference/{camera}, saved under ~/.cache/huggingface/lerobot/camera_references/<robot>/<camera>.png. Names go through the same check as robot names, and the body must be a PNG under 10 MB.
  • The ghost reuses the preview's rotation geometry, so it lines up with rotated cameras too.

Left out on purpose to keep this small: an alignment score and lighting match. They're in the standalone tool this came from (https://github.com/ravediamond/teleop-setup-align) and could follow if this is useful.

@nicolas-rabault
nicolas-rabault force-pushed the camera-reference-overlay branch from 8bd482f to 12231e4 Compare October 29, 2026 13:39

@nicolas-rabault nicolas-rabault left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this! The alignment score and lighting match you mentioned from teleop-setup-align would be very welcome as a follow-up PR.

@nicolas-rabault
nicolas-rabault merged commit 12d89c6 into huggingface:main Oct 29, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Camera reference overlay to realign the rig between sessions

2 participants