Skip to content

Add drive and controller health diagnostics - #39

Open
Hamnah228 wants to merge 14 commits into
mainfrom
feature/drive-health-diagnostics-hamnah
Open

Hamnah228 wants to merge 14 commits into
mainfrom
feature/drive-health-diagnostics-hamnah

Conversation

@Hamnah228

@Hamnah228 Hamnah228 commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

What changed

  • Integrated DriveHealthMonitor with the active SimArmDrives path in waybionic_teleop, used by the arm in simulation and with the real MKS CAN drives.
  • Evaluate each configured MKS drive using:
    • commanded target
    • encoder feedback
    • age of the latest valid reply
    • commanded-motion state
  • Added health information to each drive.<name> diagnostics row:
    • health state
    • target-versus-encoder error
    • reply age
  • Added support for:
    • OK
    • FOLLOWING_ERROR
    • STALLED
    • NOT_RESPONDING
    • DISABLED
  • Added provisional drive-health thresholds to arm_drives.yaml:
    • 5° following-error threshold
    • <1° movement over 2 s for stall detection
    • 0.5 s reply timeout
  • Added simulation-only fault injection to SimulatedServo for:
    • stalled
    • not responding
    • disabled
  • Preserved the existing controller.link diagnostics and INPUT_STALE state in joy_udp_receiver.
  • Updated the diagnostics recording workflow to use ground_station.launch.py teleop:=true.

Why

The previous implementation evaluated health through the older waybionic_control host/mock-drive path, which the arm no longer uses.

This update moves the health monitoring onto the current arm-drive path so drive faults are detected from the same commanded targets, encoder feedback, and reply timing used by the simulated and real MKS drives.

This makes stalled, unresponsive, disabled, and off-target drives visible directly through the existing ROS diagnostics panel.

Diagnostics behavior

Each evaluated drive.<name> row reports:

  • state
  • target-versus-encoder error
  • reply age

Diagnostic levels follow the project diagnostics contract:

  • OK → OK
  • NOT_RESPONDING → STALE
  • FOLLOWING_ERROR → ERROR
  • STALLED → ERROR
  • DISABLED → ERROR

Existing setup/protocol fault precedence is preserved.

Validation

  • Full workspace build passed: 8 packages built
  • Full workspace tests: 1,077 passed, 0 failed, 0 errors, 0 skipped
  • Focused sim_arm_drives_node tests: 52 passed
  • Drive-health logic tested both:
    • without ROS
    • through the active drive node
  • Verified every drive row reports OK during normal simulated operation.
  • Verified injected faults appear within the required report period.
  • Verified each simulated fault returns to OK after clearing.
  • Flake8 passed.
  • git diff --check passed.
  • controller.link behavior remains covered by its existing tests.

Recorded sessions

New mock/simulated diagnostic sessions were recorded through the corrected SimArmDrives / waybionic_teleop path for:

  • healthy baseline
  • FOLLOWING_ERROR
  • STALLED
  • NOT_RESPONDING
  • DISABLED

Recording artifact:
task7-arm-drive-health-recordings-2026-10-17 (2).zip

Each recording was verified with ros2 bag info.

Limitations / notes

  • Fault injection is simulation-only and does not modify the real CAN path.
  • The real-drive path does not currently expose a reliable per-drive enabled/disabled signal, so DISABLED is validated using SimulatedServo.
  • Drive-health thresholds are provisional and configurable through arm_drives.yaml.
  • controller.link remains in joy_udp_receiver and was not changed as part of the drive-health rework.

yassinsolim and others added 9 commits September 26, 2026 13:06
The arm is a base yaw, three parallel pitch joints and a roll through the
tool tip, so kinematics.py solves any tip position and tool tilt in closed
form from the URDF. The new Cartesian group (Y) moves the tip along straight
lines in base_link with the tilt fixed; LB keeps only the strongest axis.
At a joint speed limit or the workspace edge the whole step shrinks, so the
tip stops on the line, and the roll turns against the yaw to keep the tool's
heading.

Each drive now gets the next setpoint one period ahead and the speed that
reaches it in exactly one period, measured from its encoder, so all drives
arrive together. acc is 255 because a gentle drive ramp adds the same rpm/s
to every drive and the one with the largest change falls behind. In
simulation a sideways cut at 30:1 gearing stays within 7-17 um of the line,
compared with up to 186 um with the old 1.5x speed rule. At the 1:1
placeholder gearing it stays within about 0.13 mm, one encoder count at the
tip.
D-pad left/right changes the tool pitch while the tip stays in place, so
the shoulder, elbow and wrist move around a fixed point. The tilt uses the
same closed-form step as straight moves, so a joint limit stops it with the
tip still in place. It runs at up to 30 deg/s, scaled by the speed level.
- Start is refused while a tilt button or A is held, like the sticks.
- Homing clears the Cartesian rates, so releasing A can't resume the last move.
- Cartesian roll ramps with the acceleration limit and scales with the accepted step.
- The drives' setpoint one period ahead stays within the URDF joint limits, which the
  drive node now reads from robot_description.
- If one drive would pass max_rpm, every drive slows by the same factor, so they still
  arrive together.
- The roll compensation is documented as stopping the tool spinning about its own axis;
  that keeps a blade's heading only when the tool points straight down.
- The Cartesian launch test is registered where it won't conflict with #22.
The tests could press Start before the teleop node had loaded the robot description or received controller input, so the press was lost on a busy machine. They now send idle packets until teleop reports its group and fresh input. Same fix as on #28.
A second robot_description replaced an enabled teleop with a disabled one without sending a hold, so the drives kept their last target.
@Hamnah228
Hamnah228 requested a review from yassinsolim as a code owner October 4, 2026 01:38
@coderabbitai

coderabbitai Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c0f8e264-3425-4a4f-a406-55c59385a4f7
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Hamnah228 Hamnah228 changed the title Add drive and controller health diagnostics / hamnah Add drive and controller health diagnostics Oct 4, 2026

@yassinsolim yassinsolim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Hamnah. drive_health.py and its boundary tests are solid, and all 785 tests pass. The problem is where it's wired: the monitor runs on the older waybionic_control host and its mock drives, which the arm doesn't use. The arm's drives run through sim_arm_drives_node in waybionic_teleop, both in simulation and on the real CAN bus.

  1. Evaluate each MKS drive in sim_arm_drives_node with DriveHealthMonitor, from the commanded target, the encoder replies and the reply age. Move the monitor into waybionic_teleop or import it.
  2. Add the state, the target-versus-encoder error and the reply age to each drive.<name> row, following the diagnostics contract: STALE when a drive isn't responding, ERROR for the other faults.
  3. Keep your thresholds as parameters in arm_drives.yaml, marked provisional: 5° following error, under 1° of movement in 2 s for a stall, and a 0.5 s reply timeout.
  4. Add test-only faults to SimulatedServo (stalled, not responding and disabled), and test each state without ROS, then with the drive node.
  5. Update the README steps to use ground_station.launch.py teleop:=true.

Keep the controller.link states in joy_udp_receiver; those are good. Once #27 is in main, rebase onto main and retarget this PR.

@yassinsolim
yassinsolim changed the base branch from feature/cartesian-teleop to main October 11, 2026 09:11
@yassinsolim

Copy link
Copy Markdown
Member

Hamnah, I merged main into this branch and retargeted the PR to main, so it now shows only your 11 files. Nothing was rewritten, so pull before you push again.

  • Controller link row: main's joy_udp_receiver.py already had a controller.link row, so there's still one row. It keeps main's fields and adds your state field and INPUT_STALE state.
  • Tests: your receiver tests moved unchanged to test_controller_link.py, because both test files defined a receiver fixture with different setups.
  • Health monitor: still wired to waybionic_control, as before. Moving it onto sim_arm_drives_node is still your task from the review.

The full suite passes: 1055 tests.

@Hamnah228
Hamnah228 requested a review from yassinsolim October 17, 2026 17:11

@yassinsolim yassinsolim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Hamnah. This is what the task asked for. The monitor runs in sim_arm_drives_node, and each drive.<name> row now shows the state, the error and the reply age. The thresholds are in arm_drives.yaml, and the simulated faults are tested both with and without ROS. CI passes. Three things before it merges:

  1. The error is measured at the motor, not the joint. to_counts() multiplies by each drive's gear_ratio, so (target - count) * 360 / COUNTS_PER_REV is the motor shaft's error. Every ratio is 1.0 today, so the numbers look right. With a 50:1 gearbox, though, a 0.1° lag at the joint is 5° at the motor, and every moving drive would report FOLLOWING_ERROR. Divide by drive.gear_ratio both for the positions you pass to the monitor and for target_encoder_error_deg. Add a test with a ratio other than 1.
  2. DISABLED only works in simulation. report() reads enabled from the simulated servo, which the host can't see on the real bus, so on the real arm it's always True. Use what the host knows instead: a drive is enabled once it has confirmed its ENABLE setup frame (mks_can.ENABLE not in self.unconfirmed[index]). A real drive that ignores the enable will then show as STALLED or fail its move, and the node already stops on a failed move. Keep the disabled fault so the tests cover that path.
  3. Merge main once #45 is in. It's merging today, and both PRs change sim_arm_drives_node.py. Ask me if the merge conflicts.

@yassinsolim

Copy link
Copy Markdown
Member

Hamnah, #45 is merged now, so arm_drives.yaml on main has the real gear ratios: 26:1 for the base, shoulder and elbow, and 3:1 for the wrist motors. Point 1 from my review will show up as soon as you merge main: every moving drive will report FOLLOWING_ERROR until the error is divided by gear_ratio.

@Hamnah228
Hamnah228 requested a review from Gkong5020 as a code owner October 17, 2026 19:48
@yassinsolim

Copy link
Copy Markdown
Member

@Hamnah228 this PR now has merge conflicts with main, so it can't be merged or re-reviewed until they're resolved. Since the branch last took in main (at #34), main has merged #18, #22, #26, #40, #45 and #46.

Merging origin/main into the branch reports 15 conflicted files, but 10 of them are files this PR doesn't change. They conflict only because the branch still carries the old feature/xbox-teleop commits, so it has two merge bases with main. Take main's version of those:

git fetch origin
git checkout feature/drive-health-diagnostics-hamnah
git merge origin/main
files="BuildInstructions.md waybionic_bringup/CMakeLists.txt
  waybionic_bringup/launch/ground_station.launch.py waybionic_bringup/scripts/camera_follower.py
  waybionic_teleop/setup.py waybionic_teleop/test/test_gamepad.py waybionic_teleop/test/test_mks_can.py
  waybionic_teleop/waybionic_teleop/gamepad.py waybionic_teleop/waybionic_teleop/mks_can.py
  waybionic_teleop/waybionic_teleop/xinput_bridge.py"
git checkout --theirs -- $files
git add -- $files

The other five need both sides kept:

  • waybionic_teleop/config/arm_drives.yaml: take main's side of both hunks (the new header comment and the carrier's serial speed). Your health_* thresholds merge in on their own.
  • sim_arm_drives_node.py: Drive the real arm through the carrier board #45 added the carrier board in the same four places as your drive-health code: the carrier import, CARRIER_TIMEOUT_S, the carrier state in __init__, and carrier_status(). Keep both sides of each.
  • sim_drives.py: Add UNO R4 CAN simulation and MKS drive bring-up tooling #40 and Drive the real arm through the carrier board #45 added bitrate to __init__ and split receive() into receive(data, address=None) and _handle(). Keep TEST_FAULTS and set_test_fault() next to them; your _responding() checks then end up in _handle().
  • joy_udp_receiver.py: Drive the real arm through the carrier board #45 added link quality (ping round trip, loss, jitter, longest gap, and a warning when the link is unstable) to the same controller.link row as your link states. Combine them into one row.
  • test_sim_drives.py: both sides added tests at the end of the file; keep both.

Then run the full suite (docker compose --progress plain build test), commit the merge and push. There's no need to rebase or force-push. Once it's green, re-request review.

@Hamnah228
Hamnah228 requested a review from yassinsolim October 17, 2026 23:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants