Skip to content

feat(cameras): live alignment score and lighting match against the reference frame - #151

Open
ravediamond wants to merge 1 commit into
huggingface:mainfrom
ravediamond:feat/camera-reference-scores
Open

ravediamond wants to merge 1 commit into
huggingface:mainfrom
ravediamond:feat/camera-reference-scores

Conversation

@ravediamond

Copy link
Copy Markdown
Contributor

Follow-up to #139, as suggested there: the alignment score and lighting match from teleop-setup-align.

Once a camera has a reference frame, the recording dialog now shows under its controls:

  • Alignment (0–100): how well the live framing and arm pose line up with the reference. The preview gets a green border at 90 or above and an orange one below, so you can watch it turn green while you nudge the camera or arm.
  • Lighting (0–100), with the brightness and color diffs it comes from as text, e.g. Lighting 72% (brightness 7.0%, color 0.3%). Green at 85 or above, amber from 60, red below.

How it works:

  • Still frontend only. Twice a second the scores are computed from the preview's own <video> pixels against the saved PNG, so the backend still never opens a camera. Both frames are raw stream frames, so rotation doesn't affect the comparison. No backend changes.
  • Alignment is normalized cross-correlation on 9×9-blurred grayscale. It subtracts each image's mean, so a brightness change barely moves it; only a change in framing does. The blur turns NCC's cliff at a 1–2 px shift into a gradual curve (on 640×480: ~93 at 10 px, ~73 at 40 px, ~46 at 80 px).
  • Lighting takes the larger of the brightness change and the color-balance shift, times 4. Auto-exposure hides most brightness changes, but not color temperature, and which signal moves depends on the camera. The gain and thresholds come from curtains-open vs curtains-closed tests on my two-camera SO-101 rig.
  • When the live frame and the reference differ in size, alignment shows – instead of a misleading number.

Tests: cameraReferenceScore.test.ts checks the port against numbers from the original OpenCV implementation on the same synthetic scenes (shift, brightness, color). CameraConfiguration.test.tsx covers the score line and the border. The full frontend suite passes (59 tests), and eslint is clean on the touched files.

Not yet checked: the browser version on the physical rig. The math matches the OpenCV tool, but I haven't run it on live cameras in LeLab.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant