Source-linked AI summary
A Holistic Approach to Reactive Mobile Manipulation
Jesse Haviland, Niko Sünderhauf, Peter Corke
TL;DR
Reactive systems are responsive to environmental change but typically struggle with complex constraints and useful real-world tasks, while planning can take significant time and run open-loop. A quadratic-program controller coordinates mobile-base and arm degrees of freedom holistically, while behaviour trees orchestrate sensor-based behaviours into tasks. Mean grasp time was 3 times faster than the motion planning approach and, while average pick and place time was 2 times faster than the motion planning approach.
Problem
Reactive systems are responsive to environmental change but typically struggle with complex constraints and useful real-world tasks, while planning can take significant time and run open-loop.
Source: §I · p. 1Method
A quadratic-program controller coordinates mobile-base and arm degrees of freedom holistically, while behaviour trees orchestrate sensor-based behaviours into tasks.
Source: §I · p. 1, §IV · p. 3Results
Mean grasp time was 3 times faster than the motion planning approach and, while average pick and place time was 2 times faster than the motion planning approach.
Source: C. Experiment 3 Results · p. 8Takeaways & Limitations
Combined with a behaviour tree, the reactive controller supports task execution and error recovery in a mobile pick-and-place task.
Source: §I · p. 1, §VII · p. 8
Abstract
from arXiv · showhide
We present the design and implementation of a taskable reactive mobile manipulation system. In contrary to related work, we treat the arm and base degrees of freedom as a holistic structure which greatly improves the speed and fluidity of the resulting motion. At the core of this approach is a robust and reactive motion controller which can achieve a desired end-effector pose, while avoiding joint position and velocity limits, and ensuring the mobile manipulator is manoeuvrable throughout the trajectory. This can support sensor-based behaviours such as closed-loop visual grasping. As no planning is involved in our approach, the robot is never stationary thinking about what to do next. We show the versatility of our holistic motion controller by implementing a pick and place system using behaviour trees and demonstrate this task on a 9-degree-of-freedom mobile manipulator. Additionally, we provide an open-source implementation of our motion controller for both non-holonomic and omnidirectional mobile manipulators available at jhavl.github.io/holistic.
I. INTRODUCTION
The paper addresses the gap between reactive behaviours and useful, complex tasks, while planning can be slow and open-loop. It combines holistic reactive control with behaviour-tree task orchestration to achieve fluid motion without planning pauses.
I. INTRODUCTION
- Planning can take a significant fraction of a second or more and executes open-loop, while reactive systems typically struggle with complex constraints and real-world tasks.Planning depends on the initial world state and accurate execution; reactive behaviours are responsive to environmental change but often simplistic.Source: §I · p. 1
- The controller extends reactive arm control to a non-holonomic mobile manipulator, coordinating all degrees of freedom in a quadratic program to command end-effector velocity.Motion is partitioned between arm and base to maximise arm dexterity and allow fluid whole-robot motion.Source: §I · p. 1
- Useful tasks require behaviour completion detection, switching based on world state and goals, and error recovery—capabilities beyond a single reactive behaviour.Source: §I · p. 1
- Behaviour trees orchestrate sensor-based behaviours into tasks, supporting complex tasks with continuous motion and no pauses for planning.The paper describes behaviour trees as easier to express than state machines and as having inherent error recovery.Source: §I · p. 1
- The contributions include a closed-loop mobile grasper using behaviour trees and validation on a 9-degree-of-freedom non-holonomic platform, with example code available online.Source: §I · p. 2
II. RELATED WORK
The related work contrasts planning-based mobile manipulation with reactive control and task coordination. The paper models mobile manipulators by representing base motion as virtual joints and using a holistic Jacobian and velocity controller for the differential-drive case.
II. RELATED WORK
- Prior systems often position the base before arm grasping; planning can be slow and open-loop, while reactive control and task coordination pose integration challenges.NEO-style reactive control is described as low-latency, while behaviour trees offer readable, scalable task coordination and error recovery.Source: §II · p. 2
III. MODELLING A MOBILE MANIPULATOR
- A mobile manipulator comprises a mobile base, either omnidirectional or non-holonomic, with one or more manipulators.Source: §III · p. 2
A. Two-Wheel Differential Drive Mobile Manipulators
- The desired end-effector spatial velocity feeds a QP that outputs joint velocities, splitting them between arm joints and base virtual joints transformed into wheel velocities.Arm velocities apply directly to the arm; the base rates are transformed using the unicycle kinematic model.Source: A. Two-Wheel Differential Drive Mobile… · p. 3
- For a differential-drive base, infinitesimal rotation and forward translation become virtual joints, yielding a holistic Jacobian that maps all axes to end-effector spatial velocity.Source: A. Two-Wheel Differential Drive Mobile… · p. 3
B. Omnidirectional Mobile Manipulators
Omnidirectional robots can translate and rotate instantaneously in the plane, represented by three virtual base-joint coordinates.
B. Omnidirectional Mobile Manipulators
- Omnidirectional robots can instantaneously translate and rotate in the plane, represented by virtual coordinates qb = (δx, δy, δθ).Source: B. Omnidirectional Mobile Manipulators · p. 3
IV. MOBILE MANIPULATOR MOTION CONTROLLER
The controller extends NEO to holistic mobile-manipulator control, producing joint velocities for a desired end-effector velocity while accounting for slack and optimization costs and constraints.
IV. MOBILE MANIPULATOR MOTION CONTROLLER
- The extended NEO controller outputs joint velocities for a mobile manipulator to achieve a desired end-effector velocity.It treats the robot as a holistic system.Source: §IV · p. 3
- A slack vector represents intentional error from the desired end-effector velocity, allowing the controller to meet constraints and minimize costs.Source: §IV · p. 3
- The quadratic program optimizes joint velocities and slack while incorporating manipulability, auxiliary tasks, and joint-position limits.Its decision variable is x = (q, δ)⊤.Source: §IV · p. 3
A. Velocity Control
Velocity control uses a desired end-effector velocity with slack to accommodate inequality constraints. The objective balances motion costs with manipulability and adjusts how the base and arm contribute as the end-effector approaches its goal.
A. Velocity Control
- Velocity control relates the desired end-effector spatial velocity to the actual velocity using a slack term that permits deviation to satisfy inequality constraints.Source: A. Velocity Control · p. 4
B. The Objective Function
- The objective minimizes joint-velocity and slack costs while maximizing robot manipulability.It also includes a cost for base-to-end-effector orientation.Source: B. The Objective Function · p. 4
- The manipulability Jacobian is calculated using arm joints so the arm can exploit its extra degrees of freedom for arbitrary end-effector velocity.The base contributes 2 degrees of freedom, or 3 for a holonomic platform, and has an infinite workspace; the arm typically has more degrees of freedom but a finite workspace.Source: B. The Objective Function · p. 4
- At θǫ ≃ 0°, the platform has infinite forward reach; at θǫ ≃ ±90°, its reach is limited to the arm.The controller orients the base toward the end-effector to maximize reach.Source: B. The Objective Function · p. 4, Figure 3 · p. 4
- A large slack cost limits slack, while a small cost risks slack canceling the desired velocity and leaving steady-state error.The chosen slack cost is λδ = 1.Source: B. The Objective Function · p. 4
- As end-effector distance error decreases, the optimizer increasingly restricts motion to ensure continued approach to the goal.The optimizer can avoid operational limits and maximize manipulability earlier in the trajectory.Source: B. The Objective Function · p. 4
- When end-effector error is large, the variable gain favors base motion; as error becomes small, it favors arm motion.Source: B. The Objective Function · p. 4
C. Joint Position Limit Avoidance
Joint position-limit avoidance uses velocity dampers to restrict motion as joints approach their limits, but does not apply this avoidance to the virtual base joints.
C. Joint Position Limit Avoidance
- Velocity dampers restrict joint velocities to reduce or nullify the rate of approach to a joint limit.Source: C. Joint Position Limit Avoidance · p. 4
- Position-limit avoidance is not performed on the virtual base joints.Source: C. Joint Position Limit Avoidance · p. 4
- The damper is parameterized by distance to the nearest joint limit, its influence distance, and a minimum permitted distance.Source: C. Joint Position Limit Avoidance · p. 5
D. Position-Based Servoing
Position-based servoing drives the end-effector from its current pose toward a desired pose. The experiments first test motion between poses, then demonstrate a mobile pick-and-place behavior task.
D. Position-Based Servoing
- Position-based servoing drives the end-effector from its current pose to a desired pose.Source: D. Position-Based Servoing · p. 5
- The servoing formulation uses a gain and a function that converts a homogeneous transformation matrix into a spatial displacement.Source: D. Position-Based Servoing · p. 5
V. EXPERIMENTS
- The experiments validate motion between poses before demonstrating the architecture in a mobile pick-and-place behavior task.Source: §V · p. 5
A. Platform Description
The study uses Frankie and simulated Omni-Frankie to validate holistic motion and random reaching across varied goals and controller settings.
A. Platform Description
- Frankie combines a differential-drive Omron LD-60 base with a 7-DoF Panda arm; an eye-in-hand RGB-D camera supports real-world visual feedback.Omni-Frankie, a holonomic-base model, is used in simulation.Source: A. Platform Description · p. 5
B. Experiment 1: Holistic Motion Controller Validation
- Simulation tests four goals from a shared starting pose, including front, right, behind, and a moving target; the 4 m distance requires base and arm motion.The moving target travels forward-left for 6 seconds, then right for 6 seconds before stopping.Source: B. Experiment 1: Holistic Motion… · p. 5
C. Experiment 2: Random Reaching Task
- A 1,000-pose random reaching task within 4 m compares holistic control with separate RRTConnect base-and-arm planning and varies kǫ and Ĵm.The comparison includes Frankie and Omni-Frankie; RRTConnect uses OMPL.Source: C. Experiment 2: Random Reaching Task · p. 5
D. Experiment 3: Repetitive Pick and Place Task
A behaviour tree orchestrates a visual, reactive pick-and-place task, with recovery and repeated attempts built into the sequence.
Task orchestration
- The behaviour tree uses tick-driven behaviours that return running, success, or failure, enabling reactive control flow through the task.The tree assumes known container and table locations in a two-dimensional map; GG-CNN proposes grasp poses from depth images.Source: D. Experiment 3: Repetitive Pick and… · p. 5
Recovery and repetition
- The tree first recovers from arm errors and restores ready and initial poses; unhandled failures return the robot to this initialization branch.After a successful cycle, the second branch repeats the pick-and-place task.Source: D. Experiment 3: Repetitive Pick and… · pp. 5–6
Visual grasping
- During grasping, GG-CNN goals update continuously until the target is 0.25 m away, when the grasp pose is locked for the attempt.Failed grasps are detected by a fully closed, empty gripper or an arm error, then retried.Source: D. Experiment 3: Repetitive Pick and… · p. 6
Planner comparison
- The comparison replaces holistic control with the Omron base planner and OMPL RRTConnect arm planner while keeping the behaviour tree and vision stack unchanged.Only the first GG-CNN output is used for the open-loop planner comparison.Source: D. Experiment 3: Repetitive Pick and… · p. 6
VI. RESULTS
Results compare holistic control with planning-based approaches in reaching and motion quality, and examine how controller settings affect failures and arm manipulability.
A. Experiment 1 Results
- 5.42, 6.17, 6.17, and 13.12 seconds were the holistic approach’s completion times for the four poses; planning took 14.32, 16.17, and 16.45 seconds for the comparable motions.The planning-based approach has no counterpart for the moving-goal experiment because planners are open-loop.Source: A. Experiment 1 Results · p. 7
- The base and arm cooperate to keep the arm well-conditioned and away from joint limits, including during smooth responses to a changing goal without a computed trajectory.The arm remains well-conditioned across all four motions.Source: A. Experiment 1 Results · p. 7
- Cumulative end-effector jerk is lower with holistic control than with planning-based control, indicating quantitatively smoother motion.The planning comparison ignores jerk from the initial robot motion.Source: A. Experiment 1 Results · p. 7
B. Experiment 2 Results
- Across holonomic and non-holonomic platforms, the holistic controller outperforms non-holistic planners in the 1,000 random reaching tasks.Holistic-control failures arise from local minima; planning failures occur when no solution is found within 5 seconds.Source: B. Experiment 2 Results · p. 7, Table I · p. 7
- kǫ values from 0.1 to 0.5 provide suitable results; larger values reduce the base-to-end-effector angle but increase failures from ill-conditioned arm configurations and local minima.Minimizing that angle supports follow-on tasks such as grasping.Source: B. Experiment 2 Results · p. 7, Table II · p. 7
- Low arm manipulability leads to operational failures; optimizing whole-platform manipulability or not optimizing manipulability leaves average final arm manipulability low and increases failures.The authors relate this finding to and emphasize exploiting the arm’s limited workspace.Source: B. Experiment 2 Results · p. 7, Table III · p. 7
C. Experiment 3 Results
In Experiment 3, the holistic controller completed pick-and-place trials with timing advantages over planning approaches and related work. The conclusion pairs reactive holistic control with behavior-tree tasking and identifies perception extensions for broader tasks.
C. Experiment 3 Results
- Across ten runs, the robot placed all 100 objects with 113 grasp attempts and matched motion-planning grasp performance; vision errors triggered behavior-tree retries.No failures were caused by motion control or robot actuation; incorrect grasp points from the GG-CNN network caused minor failures.Source: C. Experiment 3 Results · p. 7
- Because comparable published timing results were unavailable, timings for related approaches were reverse-engineered from supplementary videos and summarized in Table IV.Source: C. Experiment 3 Results · p. 7, Table IV · p. 8
- Mean grasp time was 3 times faster than motion planning and, and 6 times faster than; average pick-and-place time was 2 times faster than motion planning and 3 times faster than.Source: Table IV · p. 8, C. Experiment 3 Results · p. 8
VII. CONCLUSIONS
- The authors conclude that holistic reactive control with a behavior tree makes the system taskable and capable of error recovery, with smooth, intuitive, fast motion.Source: §VII · p. 8
- The authors identify object recognition and collision detection as future perception behaviors for widening the solvable task range, with more perception also supporting reactive collision avoidance.Source: §VII · p. 8