PaperlayerView exact revision on arXiv

Source-linked AI summary

Interactive Perception: Leveraging Action in Perception and Perception in Action

Jeannette Bohg, Karol Hausman, Bharath Sankaran, Oliver Brock, Danica Kragic, Stefan Schaal, Gaurav Sukhatme

arXiv:1604.03670v3cs.RO

TL;DR

Static-image perception is often under-constrained, motivating robotic approaches that use environmental interaction to obtain richer sensory signals and action–response regularities. This survey defines Interactive Perception, synthesizes and categorizes existing approaches and applications, and discusses open questions. It concludes that IP has two central aspects—novel signals and regularity in S × A × t—but lacks a single framework covering all relevant challenges.

  • Problem

    Static-image perception is often under-constrained, whereas robots can physically interact with environments to obtain additional constraints and informative signals.

  • Method

    The survey defines Interactive Perception through novel signals and action–perception regularity, then analyzes, categorizes, and reviews existing approaches and applications.

  • Results

    Interactive Perception is characterized by forceful interactions that create novel sensory signals and regularities in S × A × t supporting signal interpretation.

  • Takeaways & Limitations

    The survey presents Interactive Perception as a defined field spanning applications such as object segmentation, manipulation skills, and object-dynamics learning.

  • Takeaways & Limitations

    Existing Interactive Perception approaches do not yet provide one framework addressing all relevant challenges.

Abstract

from arXiv · show

Recent approaches in robotics follow the insight that perception is facilitated by interaction with the environment. These approaches are subsumed under the term of Interactive Perception (IP). It provides the following benefits: (i) interaction with the environment creates a rich sensory signal that would otherwise not be present and (ii) knowledge of the regularity in the combined space of sensory data and action parameters facilitate the prediction and interpretation of the signal. In this survey we postulate this as a principle and collect evidence in support by analyzing and categorizing existing work in this area. We also provide an overview of the most important applications of Interactive Perception. We close this survey by discussing remaining open questions. Thereby, we hope to define a field and inspire future work.

I. INTRODUCTION

Perception is an active, exploratory process, and interaction supplies sensory information and action–response regularities that static-image approaches lack. Interactive Perception therefore frames robotic perception as an integrated perception–action problem.

  • Active biological perception: Biological perception depends on active exploration and learning the relationship between self-produced movement and sensory feedback.Active kittens developed visually guided behavior, whereas passive kittens receiving the same visual stimuli did not; rotating and touching objects also improved recognition.
  • Limits of static vision: Static-image perception is often under-constrained because it seeks semantic annotation with minimal assumptions or prior knowledge.Large annotated datasets provide constraints that help address this difficulty, but the approach remains data-intensive.
  • Interactive Perception: Robots can exploit physical interaction to create informative sensory signals concurrent with action and use action–sensory regularities to simplify interpretation.This integrated approach may reduce the need for very large training datasets.
  • Interactive Perception: Interactive Perception is presented as a robotics approach that uses forceful interaction to enhance perception and support robust perceptually guided manipulation.Its two stated benefits are novel sensory signals and simpler, more robust prediction and interpretation in the combined action–sensory space.

A. Forceful Interactions

Interactive Perception focuses on forceful interactions that change or probe the environment, rather than locomotion or mapping. These interactions create informative signals and regularities linking sensory data, actions, and time.

  • A. Forceful Interactions: Forceful interaction is any action that exerts a potentially time-varying force on the environment.It may involve contact, gravitational or magnetic forces, and local or global effects.
  • A. Forceful Interactions: The survey considers physical interactions that change the environment or explore its properties while excluding locomotion and environment mapping.Perceptive Manipulation is treated as equivalent to Interactive Perception, emphasizing their blurred boundary.
  • B. Benefits of Interactive Perception: Forceful interactions create novel sensory signals useful for estimating manipulation-relevant quantities such as weight, material, and rigidity.These signals can include haptic, acoustic, and visual data correlated over time.
  • B. Benefits of Interactive Perception: Action–Perception Regularity describes structure in the combined space S × A × t of sensor information, action parameters, and time.Using this regularity supports prediction, latent-property updating, and inference of the action that produced an observed signal.
  • III. HISTORICAL PERSPECTIVE: Interactive Perception relates to Active Perception by incorporating physical effects on the environment in addition to sensory-apparatus actions.The survey positions IP within broader perception approaches that differ in how they use sensory and action spaces.

A. Sensorless Manipulation

The survey distinguishes sensorless manipulation and related active-perception approaches from Interactive Perception by emphasizing sensory feedback and physical interaction. It also identifies limits in current models of environment-changing actions.

  • A. Sensorless Manipulation: Sensorless manipulation uses action sequences to reduce state uncertainty without receiving sensory feedback.Its goal is to move a system from an unknown state into a defined state, as in tray tilting for part orientation.
  • A. Sensorless Manipulation: Interactive Perception instead represents signals jointly over sensory information, action parameters, and time.For complex dynamical systems, sensorless approaches may lack sufficiently expressive forward models or fail to reduce uncertainty enough.
  • B. Perception of Visual Data: Computer Vision commonly interprets static images or video, although modeling temporal and action-related regularities can simplify recognition and restoration.Examples include activity recognition, recognizing objects during use, manipulation-action recognition, and separating foreground obstructions from backgrounds.
  • C. Active Perception: Active observers can make some vision problems uniquely solvable and linear when camera motion and associated images are known.Passive observation may instead require additional assumptions, regularization, or nonlinear optimization.
  • C. Active Perception: Models for predicting physical interactions that change the environment remain less developed than models for viewpoint changes.The survey specifically notes a need for richer, expressive, and tractable models.
  • C. Active Perception: Active Perception mainly explores the environment, whereas Interactive Perception also uses perception to monitor goal-directed motor actions.The distinction concerns how visual information supports exploration versus manipulation execution.

D. Active Haptic Perception

Active haptic perception interprets sequences of contact-based observations to estimate object or environmental properties. Most approaches explore without changing the environment, typically assuming rigid, static objects during contact.

  • Scope: Interactive Perception includes interpreting temporally ordered haptic observations produced through forceful, time-varying contact.An isolated haptic frame resembles static-image scene understanding rather than interactive perception.
  • Applications: Early active haptic methods reconstructed shape, recognized objects by tracing surfaces, and explored texture and material properties.Vision and touch were also combined for 3D object-shape reconstruction.
  • Recent methods: More recent approaches use machine learning to learn exploration strategies and feature representations for haptic perception tasks.Applications include haptic representations, object detection, pose estimation, shape reconstruction, and texture classification.
  • Scope: Most active haptic perception deliberately contacts the environment without aiming to change it, often assuming objects or surroundings remain rigid and static.This assumption simplifies interpretation during contact.
  • Relation to IP: Interactive Perception methods pursue estimation or manipulation goals, including forceful interaction for quantity estimation and actions that bring the environment to a desired state.Object segmentation illustrates how observed or self-generated interaction can provide motion cues and support action selection.

B. Articulation Model Estimation

Interactive Perception simplifies articulation-model estimation by replacing ambiguous static observations with sensory evidence from observed or self-generated interaction. Autonomous interaction additionally links actions to visual and haptic responses, enabling the correct model hypothesis.

  • Problem: Static images make it difficult to determine whether objects are constrained and whether their articulation is prismatic or revolute, even across multiple viewpoints.The robot must also estimate the joint-axis pose.
  • Survey context: Articulation-model estimation is one application area among the broader Interactive Perception literature categorized by application domains.The survey’s categorization places multi-application papers on boundaries between domains.
  • Observed interaction: Observing another agent lift a Lego block creates motion evidence favoring a prismatic joint, although that hypothesis can still be incorrect.The example illustrates that externally generated interaction adds information without revealing the robot’s applied force.
  • Autonomous interaction: Autonomous interaction produces correlated visual and haptic data that support articulation-model inference through regularities in S × A × t.Applying sufficient vertical force changes the evidence from rigid attachment toward a free-body articulation model.
  • Existing approaches: Interactive Perception approaches include offline marker-based or markerless estimation and online estimation during motion.Recent methods also reason about actions to actively reduce articulation-model uncertainty.

C. Object Dynamics Learning and Haptic Property Estimation

Interactive Perception supports estimating object dynamics and haptic properties by coupling controlled actions with resulting sensory responses. It also lets robots collect visual and tactile data for object recognition and multimodal model construction.

  • Dynamics estimation: Static observations and viewpoint changes do not provide enough information to estimate a sphere’s weight from its appearance alone.The example assumes the robot knows the relationship among push force, travel distance, and sphere weight.
  • Dynamics estimation: Observing a person push a sphere aids segmentation but leaves weight estimation uncertain because the applied force is unknown.The robot must marginalize over possible push forces.
  • Dynamics estimation: Controlling the push force and observing where the sphere stops enables estimation of its inertial properties through known action–sensory associations.The controlled interaction supplies the missing force information.
  • Haptic properties: Robots can estimate surface and material properties more accurately by moving haptic sensors along object surfaces.Other approaches move objects to estimate inertial properties or otherwise unobservable dynamics parameters.
  • Recognition: Interactive Perception can also reveal hidden object features by moving objects, potentially reducing reliance on very large training datasets for recognition and categorization.The survey identifies object recognition and categorization as additional applications.
  • Object models: Interactive Perception lets robots generate object-model data by holding, pushing, grasping, lifting, shaking, and combining visual with tactile observations.These actions support object-model completion and multimodal representation learning.

F. Object Pose Estimation

Interactive Perception addresses object pose estimation and grasp planning by using touch, movement, and exploratory actions to reduce uncertainty and handle clutter or partial information. Related manipulation-skill methods learn action regularities from demonstrations.

  • Pose estimation: Interactive Perception reduces object-pose uncertainty by touching or moving the object.Information gain can guide actions, with some policies providing optimality guarantees.
  • Grasp planning: Interactive Perception treats clutter and premature interactions as opportunities to move objects or explore them for more successful grasp planning.This contrasts with treating clutter and interaction as obstacles to avoid.
  • Grasp planning: Tactile exploration can estimate unknown object shape and support grasp control by combining exploration with exploitation actions.A Gaussian Process implicit surface represents shape, while tactile sensing performs exploration.
  • Manipulation skills: Some Interactive Perception methods pursue manipulation skills that combine multiple predefined goals.These skills are treated as a separate manipulation-oriented application.
  • Manipulation skills: Demonstration-based methods encode manipulation behavior as action sequences, impedance strategies, or learned task models.A manipulation graph, kinesthetic demonstrations, and a Hidden Markov Model provide examples of these representations.
  • Representations: Some approaches learn mappings from raw high-dimensional sensory input to lower-dimensional state representations instead of fully pre-specifying them.Most Interactive Perception approaches still pre-specify sensory and latent-variable representations using prior system and task knowledge.

V. TAXONOMY OF INTERACTIVE PERCEPTION

The taxonomy distinguishes Interactive Perception approaches by whether they create novel sensory signals or exploit regularities linking sensory data, actions, and time. Existing methods span weakly constrained general approaches to methods relying on substantial interaction-specific prior knowledge.

  • Taxonomy structure: The taxonomy organizes paper sets by application and separates approaches according to whether they exploit CNS or APR.It also lists approaches separately when they pursue multiple goals, such as object segmentation and recognition.
  • How is the signal in S × A × t leveraged?: Interactive Perception approaches are grouped by their use of Create Novel Signals (CNS) and Action Perception Regularity (APR).CNS exploits sensory signals produced by forceful interaction, while APR additionally uses regularities in S × A × t to predict or interpret those signals.
  • CNS–APR spectrum: Visual tracking and optical flow use weak priors, whereas other approaches exploit regularities from arbitrary interactions or specific informative actions.Examples include rigid-body or planar-motion assumptions in the middle of the spectrum and action selection for informative object-pose observations at the APR end.
  • Create Novel Signals: CNS approaches use forceful interaction to produce informative sensory signals for tasks such as segmentation and object-model learning.Examples use general environmental assumptions, while other work obtains novel haptic signals through planar interactions or fixed action primitives.
  • Action Perception Regularity: APR approaches use structured interaction models and multimodal observations to predict dynamics, estimate object properties, and track pose during manipulation.Examples include robotic-arm dynamics estimation, attribute-based object similarity learning, and visual-tactile pose tracking during pushing.

B. What priors are employed?

Interactive Perception encodes action–sensory regularities through engineered or learned priors on dynamics and observations. These priors simplify prediction and interpretation but can constrain representations and object or motion assumptions.

  • Prior representation: The regularity in S × A × t may be programmed as a task prior, learned from interaction, or represented through a mixture of priors and learning.How this regularity is encoded and exploited is a central component of an IP system.
  • Types of priors: Dynamics priors model how actions change environmental state, while observation priors model how state produces raw sensory signals.Dynamics prediction can be costly, and observation models enable information extraction from sensor readings.
  • Engineered dynamics priors: Common engineered dynamics priors assume rigid, articulated, or deformable objects, or availability of action primitives such as pushing, pulling, and grasping.These assumptions simplify the dynamics model and reduce the cost of predicting action effects.
  • Motion assumptions: Planar-motion and quasi-static assumptions simplify scene segmentation and forward prediction of object motion during multi-step planning.Many object-segmentation approaches assume objects lie on or move within a table plane.
  • Learned dynamics priors: Some approaches learn dynamics models mapping states and actions to likely outcomes through robot interaction and trial and error.Examples include tray tilting and object pushing behaviors.
  • Observation priors: Observation models may be hand-designed from geometry or object databases, or learned as task-specific lower-dimensional representations from raw observations.Hand-designed states usually correspond to physical quantities, whereas learned representations are less interpretable.

C. Does the approach perform action selection?

Action selection in Interactive Perception chooses actions or policies using state estimates, dynamics, observations, and task rewards. Methods range from myopic decisions to multi-step or globally optimized policies, with computational tractability depending on uncertainty modeling.

  • Problem formulation: Optimal action selection requires a policy mapping the current state estimate to an action or action sequence that maximizes expected task-dependent reward.Perceptual rewards often reflect reduced uncertainty about the estimated quantity.
  • Problem formulation: Policy formalization depends on whether state is directly observable and whether system dynamics are deterministic or stochastic.These choices determine whether MDP, POMDP, or related formalisms are appropriate.
  • Dynamics models: Dynamics models support action selection by predicting action effects and expected rewards; deterministic settings can use forward or backward value iteration.Stochastic settings require expectations over possible future outcomes and can be modeled as MDPs.
  • Uncertainty and observability: POMDP-based action selection maintains beliefs over states from noisy observations, but finding an optimal policy is intractable for most real-world problems.Approximate methods are therefore used for many realistic settings.
  • Planning horizon: Action-selection methods differ by planning horizon, including myopic one-step decisions, fixed or variable multi-step look-ahead, and global policies.Planning horizons are often constrained by computation, uncertainty, or system costs.
  • Action granularity: Some approaches compute low-level controls online through global control laws, while others select high-level action primitives such as pushing or grasping.The action granularity determines whether reasoning concerns fine motor commands or outcomes of motion primitives.

D. What is the objective: Perception, Manipulation or Both?

Interactive Perception methods pursue perception goals, manipulation goals, or both. Their sensing and uncertainty choices shape how they estimate quantities, select actions, and trade robustness against computational cost.

  • Perception: Perception-oriented applications include object segmentation, recognition, pose estimation, multimodal object-model learning, and articulation-model estimation.These tasks use interaction to service perceptual objectives.
  • Manipulation: Manipulation-oriented methods exploit S × A × t regularities to select controls that move the system toward expected perceptual signals and manipulation goals.Examples include grasping and manipulation-skill learning.
  • Perception and manipulation: Some approaches jointly improve perception and manipulation by estimating object models or poses while selecting actions under uncertainty.This combines perceptual estimation with goals such as efficient grasping.
  • Sensing modalities: Approaches may use contact sensing, non-contact sensing, or multiple modalities spanning vision, proximity, sonar, tactile, and force-torque signals.Multimodal sensing is used to represent the interaction space more richly.
  • Uncertainty: Interactive Perception faces uncertainty from noisy sensors and variable environmental responses to interaction.Methods may model uncertainty in observations, dynamics, or both.
  • Uncertainty: Uncertainty modeling can improve robustness to noisy observations or dynamics but increases computation as the solution space grows.The survey labels deterministic and stochastic dynamics or observations, along with explicit uncertainty estimation, using DDM, SDM, DOM, SOM, and EU.

VI. DISCUSSION AND OPEN QUESTIONS

Interactive Perception still lacks a unified framework that coordinates information-seeking and task-directed actions across modalities, representations, and decision models. The survey identifies multimodal sensing, dynamic-scene representations, and action-selection assumptions as open challenges.

  • Open questions: Interactive Perception must balance information-gathering actions against manipulation goals while optimizing time, effort, and risk.This resembles the exploration–exploitation trade-off in reinforcement learning.
  • Open questions: Current IP research is mostly visual, but future systems must select and balance passive, active, and interactive sensing across modalities.Each sensing mode has different costs and expected information gains.
  • Open questions: Interactive Perception may require new sensory representations and tracking methods for dynamic scenes with occlusions, lighting changes, and appearing or disappearing objects.The survey questions whether existing visual features are suited to these conditions.
  • Frameworks: Interactive Perception departs from stand-alone perception because manipulation is integral and objectives can change across tasks over a robot’s lifetime.The process continuously trades off multiple sensing modalities and interaction types.
  • Frameworks: Existing formalisms address particular IP subproblems, but none currently covers all relevant challenges.Candidate frameworks include OACs for symbolic sensory-motor experience and MDPs, POMDPs, PSRs, and bandits for action selection.

C. New Application Areas

The survey extends Interactive Perception beyond object manipulation to broader robotic applications and defines criteria for identifying IP approaches. It organizes the field around interaction-generated signals and action–sensory regularities while proposing taxonomy-based benchmarks and open-problem identification.

  • New Application Areas: Interactive Perception can support whole-body, multi-contact exploration and manipulation in unstructured environments such as disaster sites.Physical probing with hands or legs can extract information, while current systems extensively rely on teleoperation and carefully designed interfaces.
  • New Application Areas: The survey reviews applications including object segmentation, manipulation-skill learning, and object-dynamics learning.These problems are described as commonly eased by Interactive Perception concepts.
  • Core criteria: IP is defined by forceful interaction that creates otherwise unavailable sensory signals and by prior knowledge linking actions with sensory responses in S × A × t.These two aspects determine whether work is included as related Interactive Perception research.
  • Core criteria: The proposed taxonomy is intended to help establish benchmarks for comparing approaches and identify open problems.The survey also compares IP with existing perception approaches and discusses formalisms for representing IP problems.
Loading 1604.03670v3…