PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms

Oliver Kroemer, Scott Niekum, George Konidaris

arXiv:1907.03146v3cs.RO

TL;DR

Robots need to learn manipulation because real environments vary too much to model their objects, dynamics, and required skills in advance. The paper formalizes manipulation learning and reviews representative machine-learning research, concluding that reusable, robust skills remain unresolved despite progress across learning methods.

  • Problem

    Real-world variation prevents robots from assuming accurate models of environments, objects, or manipulation skills in advance.

  • Method

    The paper synthesizes manipulation-learning research into a formal framework and surveys representative approaches spanning representations, perception, models, policies, skills, and abstractions.

  • Results

    The review shows that skill policies can be learned quickly from demonstrations, while mastery often requires additional model-based or model-free reinforcement-learning experience.

  • Takeaways & Limitations

    Reusable skills become more robust when robots learn their preconditions and effects and monitor errors and goals in unstructured environments.

  • Takeaways & Limitations

    Robust and versatile manipulation skills remain far from resolved, and safe learning is constrained by high data demands and accurate-model requirements.

Abstract

from arXiv · show

A key challenge in intelligent robotics is creating robots that are capable of directly interacting with the world around them to achieve their goals. The last decade has seen substantial growth in research on the problem of robot manipulation, which aims to exploit the increasing availability of affordable robot arms and grippers to create robots capable of directly interacting with the world to achieve their goals. Learning will be central to such autonomous systems, as the real world contains too much variation for a robot to expect to have an accurate model of its environment, the objects in it, or the skills required to manipulate them, in advance. We aim to survey a representative subset of that research which uses machine learning for manipulation. We describe a formalization of the robot manipulation learning problem that synthesizes existing research into a single coherent framework and highlight the many remaining research opportunities and challenges.

1. Introduction

Robot manipulation is central to robotics because robots must effect change in the world, but useful autonomy requires handling unforeseen environmental variation. This review formalizes manipulation learning, surveys representative research, and identifies remaining challenges.

  • Motivation: Manipulation enables robots to effect change across applications including healthcare, factories, space, restaurants, services, and homes.These settings contain pervasive and unsystematic environmental variation, even in specialized tasks.
  • Research scope: Manipulation learning research spans demonstrations, abstract task descriptions for planning, and discovering object functionality through interaction.The review positions these as part of a broader range of learning objectives.
  • Review goals: The review formalizes robot manipulation learning into a coherent framework and surveys representative research across diverse manipulation problems.It also highlights research opportunities and challenges that remain.
  • Review structure: The review organizes technical challenges around learning state spaces, transition models, motor-control policies, skill conditions and effects, and higher-level abstractions.These topics cover perception, dynamics, control, and planning or transfer.

2. Common Concepts in Learning for Manipulation

Manipulation learning is shaped by physical, hierarchical, object-centered, and interactive structure. Exploiting these properties can support reusable skills, task-family generalization, and learning about novel concepts, while creating additional modeling challenges.

  • Physical structure: Manipulation is physical, so contact, gravity, object mass, irreversible processes, and other physical constraints provide prior structure for learning.Distinct objects cannot occupy the same space, and gravity applies mass-dependent forces.
  • Physical structure: Manipulation systems are often underactuated because environmental objects add independent state variables without increasing the robot’s action space.The robot must reach states from which it can alter object states.
  • Physical structure: Manipulation can be modeled as a hybrid system with continuous dynamics within discrete modes, often switched by making or breaking contacts.Mode-transition conditions may correspond to subgoals or obstacles.
  • Hierarchical task decompositions: Hierarchical decomposition divides tasks into smaller subtasks and skills, reducing learning complexity and enabling incremental policy construction.Lower-level skill policies can serve as action bases for higher-level tasks.
  • Skill reusability: Modularity and task families allow skills to be interchanged and generalized across similar tasks, but require suitable task decomposition.Discovering this structure autonomously remains a major manipulation research topic.
  • Object-centric generalization: Object-centric representations can generalize across task instances, while cross-object generalization requires adaptation to variations in shape, properties, and appearance.Abstract representations can treat substantially varying objects as equivalent or identical.

3. Formalizing Manipulation Learning Tasks

The review models manipulation learning as a task family: a distribution of MDPs whose shared and task-specific structure supports policies that generalize across tasks. This framework incorporates object-factorized states, structured transitions, hierarchical skills, and context information, while noting that most problems are more accurately POMDPs.

  • Task-family formulation: Manipulation learning targets a policy that generalizes across an entire task family rather than solving only one MDP.A task family is a distribution P(M) over manipulation MDPs, each representing a task.
  • Scope and assumptions: Most robot learning problems are more accurately characterized as POMDPs, but POMDPs remain uncommon because they are difficult to solve, especially while learning.The review therefore commonly uses MDP formulations as a practical simplification.
  • Structured dynamics: Manipulation transitions can be modular and hybrid, with piecewise-continuous dynamics changing across discrete contact-related modes and guard regions.Only subsets of objects and state variables may be affected from a given state.
  • Hierarchical actions: Higher-level skills can be represented as options with initiation sets, termination conditions, and low-level policies.Skills may be provided in advance or discovered during learning.
  • Generalization: A central challenge is transferring information across tasks whose state spaces may differ, using context variables for task-specific information that remains fixed during execution.Context can be monolithic or factored into object contexts.
  • Task-family formulation: Each task MDP includes state, action, reward, transition, discount, and task-specific context components.The action space is shared across tasks, while state, transition, and reward functions may vary by task.
  • Structured representations: Environment states are commonly factorized into relevant object states, enabling object-centric models and policies to transfer across environments.Factorization supports reuse when new environments contain similar objects.

4. Learning Object and Environment Representations

The review treats environment representation as a learned, object-based foundation for manipulation. Robots discover objects and properties through passive and interactive perception, then select features relevant to particular task components.

  • Section overview: Object-based representations provide the basis for learning transition models, skill policies, and skill pre- and postconditions.The representation must support both modeling tasks and generalizing manipulation skills.
  • Section overview: Robots can define and learn object-centered state and context spaces while discovering objects and estimating their properties.Perception may be passive or interactive.
  • Section overview: Feature selection is necessary because extracted object properties may be irrelevant to particular manipulation-learning components.The section discusses selecting and learning relevant features.

4.1 Object Representations

Object representations organize task variation hierarchically and distinguish manipulable within-task state from fixed across-task context. Point-, part-, object-, and group-level representations provide different abstraction and generalization trade-offs.

  • State and context: Within-task variation belongs in the state space, while fixed attributes that vary across tasks belong in the context space.For stacking blocks, shapes and sizes can be fixed within a task yet support generalization across tasks.
  • Object variation: Object variation includes pose, shape, size, material, mass, and robot-object or object-object interactions, with placement depending on whether it changes during a task.Interactions can impose constraints and cause mode switches that add or remove them.
  • Representation hierarchy: Hierarchical object models use point-, part-, and object-level layers, decreasing detail while increasing abstraction and enabling different forms of generalization.Geometric features describe where entities are; non-geometric features describe what they are.
  • Point-level representations: Point-level representations capture detailed object geometry using point clouds, pixels, or voxels and can associate color, material, segmentation, and contact features.Point correspondences support generalization across task instances through key points, registration, or geometry warping.
  • Part-level representations: Part-level representations group contiguous points into manipulation-relevant components, allowing correspondences across different object types.A mug’s handle, opening, bowl, and bottom illustrate parts associated with distinct interactions.
  • Object-level representations: Object-level representations group properties such as pose, mass, shape, and material because robots generally select objects rather than individual features.Groups of objects, such as clutter, may also be represented as a single entity when individual modeling is unnecessary.

4.2 Passive and Interactive Perception

Robot perception may be passive, relying on observation without physical interaction, or interactive, using actions to reveal otherwise latent environmental properties. Interactive perception costs more time and energy but can reduce uncertainty and provide supervision for passive estimators.

  • Passive perception: Passive perception observes the environment without exploiting physical interaction, such as recognizing and localizing objects from camera images.Changing the camera viewpoint remains passive when the robot does not apply force or alter the environment.
  • Interactive perception: Interactive perception physically interacts with the environment to improve estimates, using effects observed through haptic, tactile, visual, or audio sensing.Pushing can reveal constraints, while lifting can estimate weight.
  • Interactive perception: Interactive perception requires more time and energy than passive perception but can disambiguate scenarios and reveal latent properties, thereby reducing uncertainty.It can test whether objects are rigidly connected or merely in contact.
  • Informative actions: Selecting informative actions can accelerate environmental learning because different actions produce different information about an object or its contents.Shaking a container usually reveals more about its contents than pushing.
  • Interactive learning: Interactive perception can supervise passive property estimation, such as learning to predict object mass from appearance after measuring it through interaction.This self-supervised process allows robots to gather information autonomously in unfamiliar environments.

4.3 Learning About Objects and Their Properties

Learning about objects combines identifying object boundaries, discovering articulated degrees of freedom, recognizing interaction-relevant classes, and estimating properties through passive or interactive perception.

  • Segmentation distinguishes individual objects, while probabilistic beliefs and viewpoint selection can resolve ambiguous object identities.
  • Kinematic degrees of freedom and articulated connections establish object state spaces and support robust pose tracking.
  • Manipulation-oriented object classes are derived from affordances such as being containers, graspable, or stackable.
  • Passive and interactive perception estimate properties including position, shape, and material characteristics after object recognition.

4.4 Feature Learning and Selection

Feature learning selects or constructs task-relevant representations from rich sensory and object information, using unsupervised dimensionality reduction or supervised models such as neural networks.

  • Only a few object properties or sensory signals are usually relevant to a given manipulation task, and selecting them can improve robustness and generalization.
  • Unsupervised dimensionality reduction captures data correlations, discards noisy signal components, and produces compact representations of objects and their variations.
  • Supervised methods learn features within model or skill learning, commonly using decision trees and neural networks.
  • Architectural priors such as auto-encoders, spatial softmax layers, convolutions, and segmentation masks help neural models learn useful manipulation features.

5. Learning Transition Models

Transition-model learning represents how actions change task states across continuous, discrete, hybrid, and partially observable settings, balancing data efficiency, expressiveness, generalization, and transfer.

  • A transition model maps current states and actions to deterministic or stochastic next-state predictions, optionally conditioned on context for cross-context generalization.
  • Continuous Models: Continuous models predict state changes using regression, with local linear and nonparametric models trading broad generalization for data efficiency and flexibility.
  • Continuous Models: Hierarchical models such as neural networks and decision trees learn task-specific features that can improve generalization.
  • Continuous Models: Predictive state representations model future observations from observation history, capturing latent information without explicitly inferring latent state.
  • Continuous Models: Physics engines provide prior structure that can improve generalization and data efficiency, but mapping real scenarios into simulation remains difficult.
  • Discrete Models: Discrete models represent high-level task transitions and probabilistic outcomes, while tabular representations hinder generalization and transfer.
  • Discrete Models: Proposition-based operators compactly represent states through Boolean propositions and generalize actions across unmentioned proposition values.
  • Discrete Models: Relational and first-order models extend compact action representations across tasks with different numbers of objects and states.

6. Learning Skill Policies

Skill-policy learning chooses representations and control interfaces for behaviors that achieve manipulation goals, while reinforcement learning and transfer methods address data, generalization, and safety challenges.

  • Skill controllers are policies that map states to actions or action probabilities for accomplishing desired manipulation tasks.
  • Policies may output actuator commands directly or specify desired values for an intermediate controller operating at a higher frequency.
  • Cartesian and object-relative task-frame actions can improve workspace generalization by separating task execution from location.
  • Policies can adapt controller gains to vary robot compliance or stiffness during task execution.
  • Restricting policy representations according to task structure can improve generalization and data efficiency without significantly affecting asymptotic performance.
  • Nonparametric policies offer flexible, data-driven representations but typically require large datasets for high-quality performance.
  • Actor-only policy search avoids value-function learning but trades REINFORCE's high variance or gradient-free methods' sample efficiency for policy flexibility.
  • Actor-critic methods combine actors with critics to reduce gradient-estimate variance and have become state of the art in many manipulation tasks.

7. Characterizing Skills by Preconditions and Effects

Manipulation skills are characterized by preconditions that govern when they can execute and postconditions that describe their outcomes. These conditions can be represented, learned, grounded in sensory data, and monitored to support sequencing, transfer, and robustness.

  • Preconditions specify when a skill can execute, while postconditions describe the state-space regions reached after execution.
  • Predicates represent preconditions and postconditions as Boolean properties of objects, object relations, or manipulation modes.Examples include AtTableB, Full(mug, water), Grasped(mug, hand), and On(obj1,obj2).
  • Grounding maps continuous low-level state and context to a predicate’s truth value, commonly through a binary classifier.Probabilistic representations can model distributions over states and contexts where a predicate is true.
  • Modular predicates support knowledge sharing across skills and tasks, although task-specific discrepancies may require additional learning.A general grasp predicate may need to capture distinctions such as the type of grasp required for a task.
  • Precondition labels can be learned from successful and unsuccessful executions, while distinct postconditions can be discovered by clustering skill outcomes.Probabilistic classifiers capture stochastic transitions, and clustering can distinguish effects such as a box remaining upright or toppling.
  • Goal and error predicates enable classification-based monitoring, with action and sensor information from an execution helping detect success or failure.Interactive perception can verify predicate changes, while high-level arguments adapt skill execution to task needs.

8. Learning Compositional and Hierarchical Task Structures

Compositional and hierarchical task structures decompose manipulation into reusable skills, reducing the complexity of learning and planning while supporting transfer across settings. A central challenge is discovering skill libraries and their boundaries, especially during task learning rather than after complete trajectories are available.

  • 8. Learning Compositional and Hierarchical Task Structures: Task decomposition produces sequences of lower-dimensional subtasks whose recurring skills provide more relevant data and opportunities for generalization.Modularity also permits predefined and learned components to coexist within a hierarchy.
  • 8. Learning Compositional and Hierarchical Task Structures: Hierarchical structure centers on identifying component skills from which complete manipulation solutions can be assembled.
  • 8. Learning Compositional and Hierarchical Task Structures: An option consists of an initiation set, termination condition, and policy, linking skill execution to preconditions, goals, and control.The termination condition specifies when execution may cease, whereas the effect describes where execution actually terminates.
  • 8. Learning Compositional and Hierarchical Task Structures: Skill libraries collect reusable skills extracted from demonstrations or autonomous behavior, but identifying their constituent skills remains difficult and under-specified.The literature primarily segments trajectories or incorporates skill specification directly into the learning problem.
  • 8. Learning Compositional and Hierarchical Task Structures: Trajectory segmentation can use repeated-task similarity, policy or value-function similarity, latent reward similarity, salient sensory events, or transitions between interaction modes.Repeated subtasks can reduce library size and increase data available for learning each skill.
  • 8. Learning Compositional and Hierarchical Task Structures: Discovering skills while solving tasks can make otherwise infeasible decompositions possible, but fewer successful approaches exist because the problem is harder than retrospective segmentation.
  • 8. Learning Compositional and Hierarchical Task Structures: Hierarchical and compositional approaches remain promising for reducing sample complexity and improving generality, with major research breakthroughs still needed.

9. Conclusion

The review organizes robot manipulation learning around representations, perception, transition models, policies, and hierarchical skills. It concludes that robust and versatile manipulation remains unresolved, with open challenges spanning integration, safety, sensing, exploration, transfer, and sample efficiency.

  • The review synthesizes methods for representing objects and features, estimating properties through passive and interactive perception, and learning continuous, discrete, and hybrid transition models.Data-collection strategies and model choices affect learning speed and generalization to new scenarios.
  • Skill policies can often be learned quickly from demonstrations, while mastery commonly requires additional model-based or model-free reinforcement-learning experience.
  • Learning preconditions and effects helps capture skill utility across scenarios, while detecting goals and errors adds robustness in unstructured environments.
  • Modular skills can be incorporated into larger hierarchies, promoting reusability and robustness across tasks.
  • Robust and versatile manipulation skills remain far from resolved despite access to broad machine-learning methods.
  • Open challenges include integrating learning into complete control systems, in situ learning, safe learning with guarantees, multimodal sensing, and improved exploration.
  • Further priorities include exploiting common-sense physical knowledge, transferring across substantially different task families, and improving policy-learning sample complexity without empirical hyperparameter tuning.
  • Modularity, hierarchy, object generalization, and autonomous discovery are expected to remain central themes in effective manipulation-learning solutions.
Loading 1907.03146v3…