Source-linked AI summary
Learning to infer and manipulate through distributed whole-arm interaction in a soft robot
Chuhan Zhang, Ebrahim Shahabi, Kseniia Khomenko, Wei Pan, Cosimo Della Santina
TL;DR
Soft-robotic control often treats interaction as a disturbance, although some manipulation tasks require task-relevant information to be acquired through contact. The paper introduces interaction inferential learning, a memory-based control framework in which interaction reveals information and organizes behavior, and demonstrates successful blind whole-arm grasping through interaction alone. Its applicability depends on the task and embodiment, particularly whether relevant information is already available before execution.
Problem
Existing control methods generally assume task-relevant information is available before interaction, leaving underexplored tasks where action-relevant information must be acquired physically.
Method
The paper introduces interaction inferential learning, using a single recurrent interaction inferential policy to infer task information and organize manipulation from proprioceptive interaction.
Results
The system successfully localizes and grasps diverse objects through interaction alone while autonomously organizing exploration, encounter, and whole-arm grasping.
Takeaways & Limitations
Physical interaction can serve as the process through which task-relevant information is generated and intelligent manipulation behavior is organized.
Takeaways & Limitations
The framework’s applicability depends on the task and embodiment, and is constrained when task-relevant information is already available before execution.
Abstract
from arXiv · showhide
In animals such as elephants and octopuses, acquiring non-visual information about an object and physically engaging with it are inseparable processes mediated by rich, large-area interactions between compliant appendages and the environment. Soft robots provide a natural platform for translating this principle into engineered systems. Yet current robotic intelligence makes limited use of physical interaction, treating it primarily as a disturbance to be rejected or, at best, as a means of compensating for object misalignment. Here, we introduce a physical intelligence framework in which distributed compliant interactions jointly reveal task-relevant information and organize manipulation behavior. This results in an intrinsically partially observable problem: key task-relevant information is never measured directly, but must instead be inferred from the history of physical interactions. We propose a reinforcement-learning architecture that addresses this challenge by learning a memory-based control policy end-to-end. The key innovations making this possible are (i) a pretrained exploration policy that provides a reference for broad workspace exploration, (ii) joint optimization that integrates exploration and grasping objectives within a single recurrent policy, and (iii) a two-stage sim-to-real adaptation including observation mapping and policy fine-tuning. We demonstrate this principle through blind whole-arm grasping with a hybrid rigid-soft robotic arm that we equip with IMUs embedded directly within its compliant structure, providing its only source of proprioceptive sensing. The learned policy successfully identifies and grasps various objects by autonomously coordinating workspace exploration, object encounter and localization, inference of grasp-relevant properties, and stable whole-arm wrapping.
INTRODUCTION
The paper frames physical interaction as both a source of task-relevant information and a means of organizing manipulation, addressing soft-robotic tasks where information must be acquired through contact. It introduces IIL and demonstrates blind whole-arm grasping driven by proprioceptive interaction alone.
- Motivation: Soft robots offer rich physical interaction through compliant bodies, but existing control often treats interaction as a disturbance or compensates for it.The paper argues that exploiting compliance requires control strategies that use embodied intelligence during interaction.
- Limitations of prior control: Existing interaction-based methods generally assume task-relevant information is available before execution through object states, progress variables, or human commands.Under this formulation, interaction helps execute a task but does not generate the information needed for later decisions.
- Interaction inferential learning: Interaction inferential learning jointly uses physical interaction to reveal task-relevant information and organize manipulation behavior without external task variables.The framework continuously infers what to perceive, how the task should proceed, and when grasp acquisition has been achieved.
- Experimental model: Blind grasping isolates this informational role because the controller must localize and grasp objects using physical interaction rather than externally available object states.Object poses, task-progress variables, and human-generated commands are withheld from the controller.
- System and demonstration: A proprioceptive soft robotic platform with distributed embedded sensing and one interaction inferential policy autonomously organizes exploration, object encounter, and stable whole-arm grasping.The system operates on proprioceptive observations without manually designed behavioral stages or external task inputs.
- Results: Across diverse object configurations, the system successfully localizes and grasps objects through interaction alone, turning interaction into the process that generates task-relevant information.This reframes interaction as organizing intelligent behavior rather than merely perturbing a fixed control trajectory.
RESULTS
The system achieved blind whole-arm grasping using proprioceptive sensing alone, while physical interactions reorganized behavior into stable, recurring wrapping strategies. Performance was strongest at initial encounter and less reliable during subsequent wrapping, with recurrent memory and IIL supporting broad exploration and interaction-driven learning.
- Blind grasping performance: 91.4% in simulation and 71.1% on the real physical platform were achieved across tested workspace regions.Successful trials were broadly distributed across object locations.
- Blind grasping performance: P(encounter) = 0.9914, while P(wrap | encounter) = 0.9366 and varied more across workspace locations.Failures therefore arose primarily during stable whole-arm wrapping after contact rather than initial object encounter.
- Interaction-dependent behavior: Different encounter locations and local contact geometries generated distinct proprioceptive responses and reorganized subsequent wrapping under the same policy.Sensor differences emerged immediately after encounter, and encounters occurred at varying times and arm locations.
- Learning interaction-driven behavior: The recurrent IIP maintained broad workspace exploration and higher training rewards, whereas MLP and recurrent ablations remained unstable or confined to limited regions.These comparisons indicate that recurrent memory alone was insufficient; IIL progressively organized interaction representations into stable grasping behavior.
- Learning interaction-driven behavior: 0.897 classification accuracy and AUC = 0.927 showed that physical interaction induced a linearly decodable transition in the recurrent policy’s latent representation.The latent transition persisted despite variation in encounter timing and spatial location.
DISCUSSION
The discussion reframes physical interaction as an active source of task-relevant information that jointly shapes control and manipulation. The framework is most applicable when task information is unavailable beforehand and embodiment produces informative sensory responses.
- Blind grasping requires the controller to acquire task-relevant information directly from ongoing contact because external object states and task-progress variables are unavailable.
- Physical interaction becomes the process through which information for subsequent decisions is generated, rather than merely part of manipulation dynamics.
- Under IIL, control commands both change the body-environment relationship and generate information that organizes future behavior.
- Physical regulation and information generation are inseparable aspects of the same closed-loop control process.
- IIL is advantageous when interaction provides information unavailable before execution; conventional formulations remain appropriate when task-relevant information is already available.
- Soft robotic compliance provides an informational substrate because distributed deformation produces proprioceptive responses encoding body configuration and environmental constraints.
MATERIALS AND METHODS
The paper trains a single recurrent interaction inferential policy to perform blind whole-arm grasping from proprioceptive interaction histories. Interaction inferential learning combines reference-guided exploration and contact-dependent grasp optimization, then adapts the policy to hardware through observation mapping and fine-tuning.
- The recurrent IIP integrates successive proprioceptive observations and maps its internal representation to tendon actuation and planar base motion.
- IIL jointly trains the same recurrent policy with reference-guided exploration before encounter and reinforcement learning for contact-dependent grasping.
- Because object position, contact state, and interaction phase are unobserved, task-relevant manipulation information must be inferred from temporal proprioceptive evolution.
- The policy updates a recurrent hidden state from each observation and samples actions conditioned on that state, thereby using observation history rather than instantaneous measurements.
- Sim-to-real transfer first trains a recurrent observation mapper and then fine-tunes the pretrained IIP using mapped observations before hardware deployment.
- During training, transition assignments shift pre-encounter emphasis from exploration toward grasp optimization, while physical interaction phase remains unavailable to the deployed policy.
- A pretrained SAC exploration policy promotes broad, stable workspace coverage by rewarding visits to unexplored grid cells and penalizing excessive tip oscillation.