Source-linked AI summary
Let's Push Things Forward: A Survey on Robot Pushing
Jochen Stüber, Claudio Zito, Rustam Stolkin
TL;DR
Robotic manipulation in unstructured environments must contend with incomplete state descriptions and uncertain physical effects. This paper reviews forward models for pushing, classifying the literature across analytical, hybrid, dynamic, physics-engine, data-driven, and deep-learning approaches. The surveyed physics-engine studies report no clear overall winner, and evaluated contact models offer useful but not highly accurate predictions.
Problem
In unstructured environments, noisy sensing, calibration difficulty, and unmodeled variables make the state and outcomes of manipulation uncertain.
Source: §2 · p. 3Method
The paper reviews forward-model research and organizes it into six approach classes, from analytical methods to deep learning.
Source: §2 · p. 3Results
Physics-engine comparisons found no clear overall winner, while evaluated contact models made useful but not highly accurate predictions.
Source: §3.2.2 · pp. 9–10Takeaways & Limitations
Deep-learning approaches are described as promising but typically require massive amounts of data, often synthetic data from physics engines.
Source: §5 · p. 18Although motion-prediction theory is established and has been applied in simple structured scenarios, it has not yet been combined with industrial applications.
Source: §5.5 · p. 19
Abstract
from arXiv · showhide
As robot make their way out of factories into human environments, outer space, and beyond, they require the skill to manipulate their environment in multifarious, unforeseeable circumstances. With this regard, pushing is an essential motion primitive that dramatically extends a robot's manipulation repertoire. In this work, we review the robotic pushing literature. While focusing on work concerned with predicting the motion of pushed objects, we also cover relevant applications of pushing for planning and control. Beginning with analytical approaches, under which we also subsume physics engines, we then proceed to discuss work on learning models from data. In doing so, we dedicate a separate section to deep learning approaches which have seen a recent upsurge in the literature. Concluding remarks and further research perspectives are given at the end of the paper.
1 INTRODUCTION
Pushing extends robots’ manipulation repertoire, especially when objects cannot or should not be grasped. The paper reviews models that predict pushed-object motion and related uses in planning and control.
1 INTRODUCTION
- Pushing can move obstacles to clear a path when grasping is unavailable or impractical, including for robots without grippers or with objects too large or heavy to grasp.The introduction illustrates this with a household robot clearing a route to a bottle in a fridge and robots operating in mines, on the moon, or in rescue missions.Source: §1 · p. 1
- Pushing is also used to manipulate objects under uncertainty, bring them into graspable configurations, and perform dexterous tasks such as robot soccer.Source: §1 · p. 1
- An inverse model selects an action to reach a desired next state, whereas a forward model predicts the state after an action.Source: Figure 1 · p. 2, Figure 2 · p. 2
- Pushing motion is difficult to predict because friction is often unknown, dynamics are nonlinear and sensitive to initial conditions, and models that generalize to novel objects are scarce.Source: §1 · p. 2
- The review focuses on predicting pushed-object motion, also covering planning and control, and surveys analytical approaches, data-driven models, and deep learning.Source: §1 · p. 2
2 PROBLEM STATEMENT
The problem statement frames pushing as prediction and planning under idealized models as well as incomplete or uncertain states. It organizes forward-model research by approach and by the assumptions, motion, aims, and models considered.
2 PROBLEM STATEMENT
- A forward model can estimate a system’s next state from its current state and executable actions, but exploring all action-state combinations for planning is computationally intractable in continuous spaces.Source: §2 · pp. 2–3
- In unstructured environments, noisy sensors, calibration difficulty, and unmodeled variables create uncertainty in both the current state and action outcomes.The text distinguishes uncertainty in physical effects from uncertainty in sensory information.Source: §2 · p. 3
- The review classifies forward-model work into six classes: purely analytical, hybrid, dynamic analysis, physics engines, data-driven, and deep learning.Source: §2 · p. 3
- For each work, the review tracks assumptions, motion type, aim, and model type, including quasi-static assumptions, dimensionality, prediction or planning goals, and analytical, data-based, or simulator models.Source: §2 · pp. 3–4
3 ANALYTICAL APPROACHES
Analytical approaches model planar pushing through mechanics under simplifying assumptions, while data-driven methods estimate uncertain quantities from observations. Dynamic analysis and physics engines extend the discussion beyond quasi-static models, with engine evaluations reporting task-dependent performance and limits on prediction accuracy.
3.1 Quasi-Static Planar Pushing
- Quasi-static planar pushing reduces motion to the horizontal plane and neglects inertia, but the unknown support-pressure distribution leaves friction forces indeterminate.The assumption makes models more tractable; instantaneous motion follows the balance of contact forces, friction, and gravity.Source: §3.1.1 · p. 4, §3.1 · p. 4
3.1.1 Purely Analytical Approaches
- Analytical work ranges from Mason’s voting theorem and limit-surface models to ellipsoidal approximations and closed-form kinematics for single-point pushing.The voting theorem predicts rotation sense without pressure-distribution knowledge; limit surfaces map applied generalized force to instantaneous velocity when support distribution is specified.Source: §3.1.1 · p. 4
3.1.2 Complementing Analytical Approaches with Data-Driven Methods
- Data-driven approaches estimate friction and other quantities from observations, addressing uncertain parameters and assumptions in analytical models.Methods include friction estimation, convex-polynomial force-motion models, and learned predictions of likely push outcomes and their variability; some learned models remain specific to an object and material.Source: §3.1.2 · pp. 7–9
3.2 Physics Engines and Dynamic Analysis
- Dynamic pushing models are relevant when the quasi-static assumption is unsuitable, and studies examine pushing motions that produce equilibrium or manipulate objects under applied forces.Examples include catching an object by pushing it and estimating object motion and minimum pushing force for in-hand manipulation with frictional patch contacts.Source: §3.2 · p. 9, §3.2.1 · p. 9
3.2.1 Dynamic Analysis
- Dynamic-analysis studies address catching, frictional interactions, and quasi-dynamic in-hand manipulation, with assumptions varying across settings.For patch-contact in-hand manipulation, complex contact geometries are represented as rigid networks of point contacts.Source: §3.2.1 · p. 9
3.2.2 Physics Engines
- Physics engines support dynamic and 3D simulation, but comparisons find task-dependent performance, sensitivity to initial conditions, and prediction-accuracy limits.Engine comparisons report no clear overall winner; MuJoCo’s soft constraints improve efficiency but limit accuracy, especially for rigid contacts and sticking-slipping transitions. Applications include forward-simulating pushes in planners and learning physical parameters from observations; engines require explicit object models and extensive tuning.Source: §3.2.2 · pp. 9–10
4 LEARNING TO PREDICT FROM EXAMPLES
The review moves from qualitative learned effects to metrically precise forward models, then treats deep learning separately. Across these approaches, it covers prediction and related planning or control uses.
4.1 Qualitative Models
- Affordance models map object features to possible actions for planning, but tend not to generalise well to novel objects and actions.Other qualitative approaches include unsupervised learning from random pushes and regression from shape features to push-stability or rotate-push scores.Source: §4.1 · pp. 11–12
4 LEARNING TO PREDICT FROM EXAMPLES
- The review separates qualitative affordance models from metrically precise models, which predict effects such as six-DOF rigid-body motion.Qualitative models can support planning, while some applications require more precise predictions.Source: §4 · p. 10, §4.1 · p. 12
4.2 Metrically Precise Models
- Data-driven metrically precise models learn mappings from pushes to motion, sometimes encoding contact constraints with probabilistic product-of-experts models.Kopicki et al. (2017) report outperforming PhysX tuned on the same data, while requiring object point clouds; contact-specific transfer and case-based planning are also reviewed.Source: §4.2 · pp. 13–14
4.3 Deep Learning Approaches
- Deep-learning studies use neural networks to estimate physical properties, learn dynamics, and support pushing prediction, planning, or control.The review describes object-centric models, including the Visual Interaction Network, and approaches that predict distributions over outcomes.Source: §4.3 · pp. 14–15
- SE3-NETS predict 3D rigid-body motion from depth images and actions, outperform flow-based networks on simulated tabletop data, and also perform well on real Baxter depth images.The method requires point associations between depth images; Agrawal et al. (2016) instead learn forward and inverse models for poking, with training from over 100,000 actions across more than 400 hours.Source: §4.3 · pp. 15–16
- Finn et al. (2016) predict pixel motion conditioned on actions without object-centric representations, using previous frames to support generalisation to unseen objects.Their dataset contains 59,000 robot-pushing motions; the review notes that these models require large amounts of data for real-world performance.Source: §4.3 · pp. 16–17
- Deep reinforcement-learning control approaches often use dynamics models and are treated as complementary to the reviewed prediction work.The review points readers to separate overviews rather than providing a detailed review of this active field.Source: §4.3 · p. 17
5 FINAL REMARKS
The concluding remarks organise pushing research by approach and identify assumptions and open challenges. They highlight gaps in perception, sensing, uncertainty handling, cooperative pushing, and practical industrial use.
5 FINAL REMARKS
- The review classifies approaches as analytical, hybrid, dynamic-analysis, physics-engine, data-driven, and deep-learning, and notes that data-driven and deep-learning methods differ in their modelling focus.The review describes progress toward generalising prediction to unseen objects, alongside data demands for deep learning and commonly assumed knowledge of object properties.Source: §5 · pp. 17–18
5.1 Understanding and Semantic Representation
- Robots need to distinguish pushable from static objects, but human labelling is expensive; converting images into geometric and semantic representations is identified as beneficial.The proposed direction would let robots prioritise probable dynamic objects.Source: §5.1 · p. 18
5.2 Sensory Fusion and Feedback
- Vision alone may be insufficient for complex manipulation; the review calls for fusing tactile, proprioceptive, and visual feedback to support failure recovery.Source: §5.2 · p. 18
5.3 Explicitly Modelling Uncertainty in the Model
- The review identifies selecting actions that account for uncertainty in the object's initial pose as a direction that may benefit pushing and push-and-grasp.The cited grasping work empirically suggests that reasoning about uncertainty can yield more robust reach-to-grasp trajectories under pose uncertainty.Source: §5.3 · p. 18
5.4 Cooperative Robots and Multiple Contacts Pushing
- Multiple-contact pushing is hard to predict when several agents act, making control and decision-making key challenges for cooperative robots.The discussion frames this in settings such as warehouses moving large-scale objects.Source: §5.4 · p. 19
5.5 Real-world Applications
- Existing motion-prediction methods have not yet been combined with industrial applications, despite work in simple, structured scenarios.The review notes that theoretical solutions are rarely reliable in practical engineering applications and calls for sophisticated practical approaches.Source: §5.5 · p. 19
FUNDING
The work was supported by EPSRC for the National Centre for Nuclear Robotics.
FUNDING
- EPSRC support for the National Centre for Nuclear Robotics funded this work.The stated grant number is EP/R02572X/1.Source: FUNDING · p. 19