Source-linked AI summary
Real-time Perception meets Reactive Motion Generation
Daniel Kappler, Franziska Meier, Jan Issac, Jim Mainprice, Cristina Garcia Cifuentes, Manuel Wüthrich, Vincent Berenz, Stefan Schaal, Nathan Ratliff, Jeannette Bohg
TL;DR
High-DoF manipulation under uncertainty raises the question of how to integrate perception and motion generation. The paper compares three system architectures using shared components, finding that local feedback suits simpler tasks while reactive planning performs better in more complex environments.
Problem
For high-DoF manipulation with physical interaction, it remains open how perception and motion generation should be integrated under uncertainty.
Source: §I · p. 1, B. Visual Robot Tracking · p. 3Method
The paper empirically compares Sense-Plan-Act, Locally Reactive Control, and Reactive Planning, using the same components with different feedback rates or motion-planning horizons.
Source: §I · p. 1, §II · p. 1Results
Locally reactive control can be efficient and safe in simple tasks, while Reactive Planning performs better in more complex environments; Sense-Plan-Act performs well in static scenarios but has advantages over it.
Source: §VII · p. 7Takeaways & Limitations
The comparison concludes that integrating real-time feedback at different time scales is crucial for safe, successful execution in uncertain, dynamically changing environments.
Source: §I · p. 1
Abstract
from arXiv · showhide
We address the challenging problem of robotic grasping and manipulation in the presence of uncertainty. This uncertainty is due to noisy sensing, inaccurate models and hard-to-predict environment dynamics. We quantify the importance of continuous, real-time perception and its tight integration with reactive motion generation methods in dynamic manipulation scenarios. We compare three different systems that are instantiations of the most common architectures in the field: (i) a traditional sense-plan-act approach that is still widely used, (ii) a myopic controller that only reacts to local environment dynamics and (iii) a reactive planner that integrates feedback control and motion optimization. All architectures rely on the same components for real-time perception and reactive motion generation to allow a quantitative evaluation. We extensively evaluate the systems on a real robotic platform in four scenarios that exhibit either a challenging workspace geometry or a dynamic environment. In 333 experiments, we quantify the robustness and accuracy that is due to integrating real-time feedback at different time scales in a reactive motion generation system. We also report on the lessons learned for system building.
I. INTRODUCTION
Integrating perception with motion generation for high-DoF manipulation under uncertainty remains an open systems question. The paper evaluates three architectures and describes how their perception and motion-generation components interact.
I. INTRODUCTION
- The open question is how to integrate perception and motion generation for high-DoF physical interaction under uncertainty, despite advances in visual tracking and reactive planning.The paper notes that teams in the DARPA Robotics Challenge, and Amazon Picking Challenge, testified to this challenge.Source: §I · p. 1
- Visual perception tracks the target object and robot arm and represents workspace obstacles; the resulting state feeds a 1 KHz local controller and a 5–10 Hz motion optimizer.The paper evaluates adaptivity, accuracy, and robustness in four scenarios varying in geometric complexity and dynamicity.Source: §I · p. 1
- The paper evaluates system integration empirically, comparing systems that use the same components but differ in sensory update rates and motion-planning horizons.It proposes requirements for components rather than prescribing particular perception, planning, or control modules.Source: §I · p. 1
II. SYSTEM ARCHITECTURES
- Three compared architectures are Sense-Plan-Act, Locally Reactive Control, and Reactive Planning, with information flow between perception and motion generation shown across time steps.Reactive Planning combines local and global policies, enabling immediate reactions to local changes and looking ahead to larger changes.Source: §II · p. 1, Figure 1 · p. 2
A. Sense-Plan-Act
Sense-Plan-Act, local feedback control, and reactive planning trade off structured planning, immediate adaptation, and look-ahead. The paper compares these architectures as systems for manipulation under changing conditions.
A. Sense-Plan-Act
- Sense-Plan-Act subdivides manipulation into perception, motion planning, and control, suiting well-defined environments but coping less well with uncertainty and change.Sequential Sense-Plan-Act requests feedback at selected moments and is more robust to sensing and actuation uncertainty, but not fast environment or target dynamics.Source: A. Sense-Plan-Act · p. 2
B. Locally Reactive Control
- Locally reactive control uses local policies to compute the next command from nearby geometry, reacting immediately but risking local minima in complex workspaces.Visual servoing is a class of locally reactive control that closes the loop around visual data; this approach may be limited for tasks requiring planning.Source: B. Locally Reactive Control · p. 2
C. Reactive Planning
- Local control can react to new information but is susceptible to local minima, while motion planning can solve complex situations at computational cost that may break real-time requirements.The proposed hybrid addresses both capabilities by combining local policies with global optimization.Source: C. Reactive Planning · p. 2
- Reactive planning combines local control and global motion optimization, enabling local adaptation alongside replanning for larger changes.The two motion representations must be fused into one policy; fewer hybrid approaches have scaled to many-DoF robots manipulating the world.Source: C. Reactive Planning · pp. 2–3
- Prior hybrid examples include elastic-strip planning and controller funneling, which requires a-priori state-space partitions and switching conditions.The paper compares instantiations of the three architectures using the same components.Source: C. Reactive Planning · p. 3
III. REAL-TIME FEEDBACK MODULES
For dynamic-object manipulation in changing environments, feedback must be timely and compact enough to support continuous use. The system uses interchangeable components that meet these requirements.
III. REAL-TIME FEEDBACK MODULES
- Dynamic objects and environments make continuous feedback about target locations and workspace structure important, so feedback components must deliver information as fast as possible.A low-dimensional representation helps keep bandwidth requirements low.Source: §III · p. 3
- The feedback modules provide information about dynamic targets and workspace structure for use in manipulation scenarios.The passage identifies those information needs as target locations and workspace structure.Source: §III · p. 3
- The described methods are examples of integrated system components; other methods that fulfill the same requirements could also be used.The requirements are timely feedback and a low-dimensional representation to limit bandwidth.Source: §III · p. 3
A. Visual Tracking of Target Objects
The feedback modules estimate target and robot state and model obstacle geometry from depth images. Their outputs support manipulation despite occlusion, model error, and workspace obstacles.
A. Visual Tracking of Target Objects
- Object tracking uses depth images and known triangle meshes to estimate 6DoF poses at the camera frame rate, with a formulation robust to occluded object parts.The method is a probabilistic approach from prior work.Source: A. Visual Tracking of Target Objects · p. 3
B. Visual Robot Tracking
- Visual robot tracking fuses depth images and joint angles to estimate robot configuration relative to the camera at 1 kHz, correcting sensor-bias and camera-kinematics errors even under heavy occlusion and fast motion.The method also models delay between joint-sensor and camera measurements.Source: B. Visual Robot Tracking · p. 3
C. Modeling Unstructured Workspace Obstacles
- Workspace obstacles are reconstructed from depth images as a discrete occupancy grid, generated frame-by-frame at approximately 15 Hz.The grid is single-scale, noise-filtered, cropped to reachable space, and uses ray casting to mark occluded regions occupied.Source: C. Modeling Unstructured Workspace… · p. 3
- The occupancy grid represents obstacle regions, while its corresponding signed distance field encodes distance to the closest occupied cell.The figure depicts grid-cube colors as height and distance-field sphere colors as distance.Source: C. Modeling Unstructured Workspace… · p. 3, Figure 3 · p. 4
IV. FROM VISUAL FEEDBACK TO CONTINUOUS SIGNED DISTANCE FIELDS
Visual feedback is converted into distance-field geometry for collision reasoning, while motion generation combines local controllers across task spaces. The system also links these local controllers to longer-horizon motion optimization.
IV. FROM VISUAL FEEDBACK TO CONTINUOUS SIGNED DISTANCE FIELDS
- SDFs represent the target, table, unstructured workspace obstacles, and robot; local closest-point subsets let reactive control use workspace geometry without full-field communication.The full SDFs would require high bandwidth and cannot be leveraged in the 1 KHz control loop.Source: §IV · p. 4
- Target, table, and robot geometry is approximated with simple shapes; the target pose is visually estimated, while the table pose is assumed known and static.The robot configuration is updated from estimated joint angles.Source: §IV · p. 4
V. MOTION GENERATION
- Local controllers defined in task spaces are evaluated at joint angles and combined through a common Gauss-Newton-like approximation.The approximation removes the second-order term in the acceleration mapping.Source: §V · p. 4
- Local collision-avoidance controllers can be derived from motion policies optimized over longer horizons, allowing experiments across local control and higher-level optimization.The paper uses signed-distance gradients and Riemannian metrics to support obstacle-aware motion reasoning.Source: §V · p. 4, §IV · p. 4
A. Locally Reactive Control
Locally reactive control combines target convergence and obstacle avoidance, while continuous motion optimization supplies longer-horizon policies that are integrated with local control.
A. Locally Reactive Control
- Obstacle controllers generate workspace accelerations away from nearby obstacles with priority increasing as proximity increases, while a target controller pulls toward position and orientation goals.The target controller can operate without higher-level planning.Source: A. Locally Reactive Control · p. 4
- Default-posture and damping controllers help resolve arm redundancy and regulate system velocity.Damping is applied in configuration space and at the end effector.Source: A. Locally Reactive Control · p. 4
- Local control runs at 1 kHz, sending desired joint states to low-level execution at 100 Hz for interpolation and inverse-dynamics torque tracking.The low-level controller tracks the desired joint states.Source: A. Locally Reactive Control · p. 4
B. Reactive Planning and Continuous Motion Optimization
- Reactive planning continuously optimizes over a three-second horizon, updating at 5–10 Hz and sending LQR policies for integration with faster local controllers.The policies encode anticipatory actions around the local optimum, supporting coordination between planning updates.Source: B. Reactive Planning and Continuous… · p. 5
C. Grasping
The grasp task is divided into sequential states, and three system architectures are compared on a fixed-base, two-arm platform. Their perception and world-model inputs are varied in update and planning horizon.
C. Grasping
- Grasping is divided into approach, establish grasp, move object, release, and retract states, governed by local control, continuous optimization, or both.Grasping uses independent controllers, and grasp poses are manually defined for each object.Source: C. Grasping · p. 5
VI. EXPERIMENTS AND DEMONSTRATIONS
- Experiments use a fixed-base platform with two 7-DoF arms, Barrett Hands, an RGB-D camera, and torque-controlled components.The platform runs real-time inverse-dynamics control with 1 ms worst-case execution times.Source: §VI · p. 5
A. System Architecture Realizations
- The architectures share perception algorithms and vary information update frequency and planning horizon: one-shot sense-plan-act, continuous local control, or continuous replanning plus local control.The world model treats the table pose as known and other environment geometry as unstructured obstacles.Source: A. System Architecture Realizations · p. 5
B. Scenarios
Four scenarios evaluate manipulation under challenging geometry or changing environments, with each experiment instance run at least three times. In static pick-and-place, reactive planning finds feasible paths more often in the hardest setting, while execution times rise with difficulty.
B. Scenarios
- The paper evaluates four scenarios spanning workspace geometry and dynamic conditions, with each experiment instance repeated at least three times.The static pick-and-place scenario is one of the four scenarios.Source: B. Scenarios · p. 5, 1) Pick and Place in Static… · p. 5
1) Pick and Place in Static Environments of Increasing Difficulty:
- The static pick-and-place task moves a Pringles box around a brown-box obstacle, with difficulty increased by moving the obstacle closer to the robot base.Fig. 5 identifies the experimental scenarios, and Table I reports success rates for the pick-and-place task.Source: 1) Pick and Place in Static… · p. 5, Figure 5 · p. 6, Table I · p. 6
- Reactive planning finds a feasible path more often in the hardest static setting because it can replan during execution; sense-plan-act fails more often there.Table I reports success rates, and position 15 is where every architecture failed at least once across three trials per complexity level.Source: 1) Pick and Place in Static… · pp. 5–6, Table I · p. 6
- Execution time increases with difficulty as confined workspaces slow convergence, especially for locally reactive control; sense-plan-act is significantly slower overall.Locally reactive control and reactive planning are on par in simple settings; Table II reports successful-trial execution times.Source: 1) Pick and Place in Static… · p. 6, Table II · p. 6
2) Pick and Place in Dynamic Environments:
In dynamic pick-and-place, locally reactive control and reactive planning respond to obstacle changes, but only reactive planning completes the placement. For moving targets, both reactive systems adapt across the tested position and orientation changes, while sense-plan-act succeeds only in narrower circumstances.
2) Pick and Place in Dynamic Environments:
- The dynamic pick-and-place test moves an initially unmodeled obstacle between pickup and goal during execution.The obstacle is captured by the octomap and considered by locally reactive control and reactive planning.Source: 2) Pick and Place in Dynamic… · p. 6
- Reactive planning avoids the moved obstacle and completes placement; locally reactive control gets stuck after grasping, while sense-plan-act collides when the obstacle moves.Locally reactive control has lower clearance, whereas sense-plan-act begins with higher clearance but does not respond to environmental change during motion.Source: 2) Pick and Place in Dynamic… · p. 6
3) Grasping with Dynamic Targets:
- Both reactive systems grasp successfully across all tested target-position and target-orientation changes, while sense-plan-act succeeds only when the new target position is near its original path.For the repositioned-and-flipped target, sense-plan-act's successful grasp does not reflect adaptation to orientation; the target's symmetry makes the grasp work.Source: 3) Grasping with Dynamic Targets · p. 7
- For the repositioned target, locally reactive control is slightly faster; for the flipped target, reactive planning is faster because it adapts its approach continuously.Locally reactive control trades off position and orientation convergence, while reactive planning uses a planning horizon to adapt to the rotated approach.Source: 3) Grasping with Dynamic Targets · p. 7
- The experiments emphasize continuous target-pose feedback for adapting grasps and show a benefit from a longer planning horizon.Source: 3) Grasping with Dynamic Targets · p. 7
4) Pointing in Dynamic Environments with Dynamic Target:
The pointing experiment tests simultaneous changes to obstacles and target pose. Both reactive architectures avoid collisions, while sense-plan-act collides in two obstacle levels and does not adapt to the target rotation.
4) Pointing in Dynamic Environments with Dynamic Target:
- The task aligns the robot's fingertip with the drill tip across four levels, from a static baseline to changing obstacles and target orientation.Source: 4) Pointing in Dynamic Environments… · p. 7
- Neither reactive architecture collides; sense-plan-act collides with the blocking obstacle in Levels 2 and 3 and reaches only the drill tip's initial position in Level 4.Success is defined as reaching the drill tip within 3 cm without collision with an environmental obstacle.Source: 4) Pointing in Dynamic Environments… · p. 7
- Success meant reaching within 3 cm of the drill tip without collision; in Level 4, sense-plan-act lacked perceptual feedback, while reactive methods shortened paths using the removed obstacle; optimal performance requires target tracking and obstacle updates.Source: 4) Pointing in Dynamic Environments… · p. 7
VII. CONCLUSION, LESSONS LEARNED, FUTURE WORK
The paper compares three architectures to assess how perception and motion generation integration affects high-DoF manipulation. It concludes that local feedback suits simpler tasks, while reactive planning performs better in complex environments; the authors also identify sensing and modeling directions for future work.
VII. CONCLUSION, LESSONS LEARNED, FUTURE WORK
- Locally reactive control can be efficient and safe in simple tasks, while reactive planning performs better in complex environments by looking ahead.Sense-plan-act performs well in static scenarios, but the reactive systems can start moving earlier while continuing to consume feedback.Source: §VII · p. 7
- Fast feedback is more beneficial than accurate world representations on the observed accuracy–speed trade-off, especially in dynamic, uncertain manipulation.The authors describe model-based visual tracking, querying signed distance fields at a subset of robot points, and synchronizing sensory and information streams across three computers.Source: §VII · pp. 7–8
- In the dynamic pick-and-place test, reactive planning avoids the obstacle and completes placement, whereas locally reactive control gets stuck and sense-plan-act collides after the obstacle moves.Source: Figure 7 · p. 8
- The system mostly uses visual and joint-encoder feedback; the authors identify tactile or force/torque sensing, contact constraints, learning, and online inverse-dynamics learning as possible extensions.Source: §VII · p. 8