Robot-assisted spine surgery has been clinically validated worldwide, demonstrating enhanced performance, efficiency, and safety. However, current spine robotic systems rely on fiducial markers, requiring additional incisions that increase the risk of infection. Additionally, optical tracking systems are susceptible to line-of-sight obstructions. Previous research has demonstrated limitations in simplifying obstacles as single points, making complex obstacle environments unmanageable.
In this thesis, a multi-robot cooperative platform for surgery was developed, featuring a plug-in to integrate a markerless depth image segmentation network for globally optimal collision-free object tracking. The system enables a tracking robot arm to automatically adjust its pose to maintain visual contact with a moving target, while the second surgical robot arm acts as an obstacle. To achieve this, precise hand-eye calibration algorithms, including camera calibration and ArUco marker detection, were developed with ROS2. Its accuracy was assessed in a simulation environment before being tested on the real robot. The state-of-the-art cuRobo, a GPU-accelerated motion planning library, was translated by developing ROS2 packages for highly parallel inverse kinematics and trajectory optimization. Leveraging parallel computing enables realizing a global optimal solution for obstacle avoidance. The motion-tracking performance of the real robot on the generated trajectory was analyzed. Separate experiments and solve time analyses were conducted for both simple and complex obstacle avoidance scenarios. The pipeline was further developed into a continuous, efficient, collision-free object-tracking system. A tracking performance analysis showed that the system generated near-optimal, smooth, and collision-free paths. Finally, the algorithms were applied in a multi-robot marker tracking experiment, with a demo showcasing markerless knee tracking through collision-free paths.
The algorithm was tested in simulation, where the ground truth was known. We compared the calculated results with ground truth to ensure after nine distinctive samples, the error converged to zero.
Then similar robot and marker poses were repeated in real experiments to ensure accuracy.
We generated random positions around the obstacle to show that the robot was able to move freely in the environment without collision. With RTX 4060 GPU, in the simple cuboid environment, the average solve time was within 50ms, while in a more complex environment, it took around 500 ms. RTX 4090 could improve reduce solve time by 30%.
Here we show the cuRobo planner performance in object tracking compared with RRTConnect.
LBR iiwa 14 held a marker with pre-defined motion (straight line and Imperial 'I'), while iiwa 7 tracked the motion. Result demonstrated the robot's ability to follow both simple and complex patterns, with short and smooth paths.
More accurate end-effector traces were visualized. The arrow represents the orientation of the waypoint.
Camera intrinsics can be obtained either from internal ROS2 topics or via our camera calibration node. This package also contains ROS2 nodes for aruco marker pose estimations, robot transformation listener, and OpenCV hand-eye calibration estimation. We presented a demo in simulation to explore the accuracy of the algorithms.
Translating cuRobo using its python bindings and RTX 4060 GPU, we can plan obstacle -free, short and smooth path within 50 ms (cuboid) or 500 ms (cuboid + mesh). With RTX 4090, we can reduce solve time to 30 ms and 300 ms. Representing the obstacle robot as spheres further reduces solve time (under development).
We demonstrated the robot’s ability to accurately follow both simple and complex predefined motions, with near-optimal, smooth paths.
The algorithms were applied in a multi-robot marker tracking experiment, with a demo showcasing markerless knee tracking through collision-free paths.
KUKA LBR iiwa 7 ROS2 driver: lbr_fri_ros2_stack
KUKA LBR iiwa14 ROS1 driver: iiwa_stack
GPU-accelerated motion planning: cuRobo
@mastersthesis{zhuang2024multirobot,
author = {Zhuang, Shengyang},
title = {Multi-Robot System Prototyping for Cooperative Control in Robot-Assisted Spine Surgery},
school = {Imperial College London},
year = {2024},
}