PaperlayerView exact revision on arXiv

Source-linked AI summary

Contact-Guided Exploration for Non-Prehensile Locomanipulation with Multi-Critic RL

Simone Tolomei, Mayank Mittal, Franco Angelini, Manolo Garabini, Paolo Salaris, Marco Hutter

arXiv:2608.28140v1cs.RO

TL;DR

Non-prehensile locomanipulation is difficult because hybrid contact dynamics and sparse rewards impede effective exploration. The paper addresses this with a Multi-Critic strategy that uses dense contact guidance early and decays it toward task optimization, achieving strong simulated results and real-world chair-transportation validation.

  • Problem

    Non-prehensile locomanipulation remains challenging because hybrid contact dynamics, unilateral constraints, and sparse positive feedback complicate learning.

  • Method

    The method samples candidate interaction points with a grasping algorithm, applies a dense contact-seeking exploration reward, and decays its dedicated critic’s weight during training.

  • Results

    94.1% success rate, 4.4% tipover rate, and 9.2 s completion time are reported for the proposed method in chair transportation.

  • Takeaways & Limitations

    The approach supports contact-rich manipulation across simulated tasks and hardware trials, including generalization to novel objects and payloads.

Abstract

from arXiv · show

Non-prehensile manipulation offers versatile skills for moving and rearranging heavy or bulky objects, particularly when combined with a mobile manipulation platform. However, both model-based and model-free approaches struggle with the complex hybrid dynamics and the sparsity of the contact in these tasks. To address these challenges, we propose a contact-guided exploration strategy implemented within a Multi-Critic Reinforcement Learning (RL) framework. A dedicated exploration critic is trained with a dense contact-seeking reward that guides the end-effector toward meaningful contact points; its influence is progressively decayed to recover a task-optimal policy. We obtain candidate interaction points from a general-purpose grasping algorithm, enabling the exploration mechanism to generalise across various object geometries. We evaluate the approach on multiple tasks, including box pushing, chair transportation, and a dishwasher opening task. Finally, we validate the chair transportation policy through extensive experiments on a quadrupedal mobile manipulator, demonstrating deployable non-prehensile manipulation in the real world.

I. INTRODUCTION

Non-prehensile locomanipulation is difficult because hybrid contact dynamics and sparse feedback complicate exploration, especially under whole-body constraints. The paper proposes contact-guided exploration with a decaying exploration critic and evaluates it in simulation and hardware.

  • Non-prehensile locomanipulation must coordinate unilateral contact, friction, base stability, workspace limits, and object dynamics.
  • Sparse rewards can drive reinforcement-learning agents to avoid contact and settle in local minima dominated by regularization penalties.
  • The proposed strategy uses candidate interaction regions and a dense contact-seeking reward to guide exploration toward meaningful contacts.
  • A dedicated exploration critic is progressively decayed so learning transitions from guided contact synthesis to task-optimal execution respecting physical constraints.
  • The method is evaluated on box pushing and chair transportation, with an additional qualitative dishwasher-opening evaluation.
  • Hardware experiments on chair transportation demonstrate deployable behavior and zero-shot generalization across real-world object geometries and masses.

II. RELATED WORKS

Prior work uses model-based control, reinforcement learning, demonstrations, teleoperation, and visual pre-training for contact-rich locomanipulation. This paper instead biases learning from scratch with object-centric contact priors and decays that bias through a Multi-Critic schedule.

  • Model-based methods can struggle with uncertain friction, complex geometries, discontinuous contacts, model mismatch, and real-time replanning.
  • Reinforcement learning has addressed contact-rich tasks, including articulated-object opening and pushing large obstacles.
  • Sparse rewards make random exploration ineffective for complex manipulation, motivating demonstrations, teleoperation data, or visual pre-training in prior approaches.
  • The proposed approach learns from scratch using object-centric contact priors, then gradually decays their influence without assuming a specific morphology.
  • C. Multi-Critic and Curriculum Learning: Multi-Critic architectures separate value functions for reward components, enabling dynamic objective weighting and more stable gradient estimation.

III. METHODOLOGY

The methodology combines hierarchical whole-body control with contact-guided exploration for quadrupedal mobile manipulation. A high-level policy coordinates arm and base references across chair, box, and dishwasher tasks.

  • The tasks require moving chairs and boxes while preventing tipping and respecting the robot’s kinematic and dynamic limits.
  • A hierarchical controller uses a high-level policy to produce target arm joint positions, base SE(2) velocity, and base height.
  • Modulating base height helps keep the extended arm within a feasible configuration near ground-level objects and avoids shoulder-joint-limit failsafe activation.
  • Contact candidates are generated from object meshes through surface sampling or a grasping algorithm to guide exploration.
  • A frozen pre-trained locomotion policy converts base commands into low-level leg joint targets.
  • The architecture supports chair transportation, box pushing, and dishwasher opening, with randomized arm motions spanning the arm workspace during simulation training.

A. Exploration and Contact Sampling

The method uses grasp-derived contact candidates to provide dense exploration targets, then integrates exploration, task, and regularization objectives through multi-critic advantage weighting.

  • Contact-guided exploration: At each reset, a 3D candidate contact point is sampled and the policy is rewarded for reducing end-effector distance to it.The exploration reward is assigned to a dedicated critic whose weight is progressively decayed during training.
  • Contact sampling: Grasping-based mesh processing generates interaction candidates, while box-pushing tasks use uniformly sampled points on visible surfaces.An intermediate candidate count balances interaction diversity against sample efficiency and avoids excessive mesh-like sampling.
  • Multi-critic formulation: The return is divided into task performance, exploration incentives, and action regularization reward groups.This decomposition supports separate value estimates for competing objectives.
  • Multi-critic formulation: Separate value heads estimate each reward stream, whose advantages are combined with weights controlling their relative influence on policy updates.The weighted aggregation provides explicit control over exploration and regularization in the policy gradient.
  • Optimization: The policy is optimized with PPO using the composite advantage, while the collective value loss sums weighted mean-squared errors across critic heads.This extends PPO to multiple value-function approximations for distinct reward groups.

C. Training Details

Training uses large-scale randomized simulation and diverse chair assets to improve robustness across object properties and geometries.

  • Simulation setup: Training runs in Isaac Lab across 4096 parallel environments with randomized robot states, object properties, and goal positions at each reset.The simulation timestep is 0.005 s and the control timestep is 0.02 s.
  • Chair assets: Chair training uses 15 IKEA chairs combined with 100 procedurally randomized chairs spanning varied geometric dimensions.Randomized parameters include chair height, seat dimensions, thicknesses, and leg cross-section.

1) Asset Generation:

The system combines base and arm targets with actor observations of robot and object state, while critics additionally receive privileged dynamics and contact information.

  • Action space: The policy action space contains a 4D base command and 6D target joint angles for the arm.The base command includes motion and height components, while the arm output specifies target joint configurations.
  • Observation space: The actor observes proprioception and object state expressed in the robot’s base frame.Inputs include joint states, projected gravity, base velocity, and object pose and goal error.
  • Observation space: The critic receives privileged object velocities and the specific target contact point in addition to its other information.This creates an observation difference between actor and critic networks.
  • Goal command: Goal commands are 2D target positions relative to the object’s initial position and are sampled uniformly within a disk of radius 2 m.The command is resampled at each reset.
  • Network architecture: The actor uses an LSTM followed by an MLP, while the multi-critic network shares early layers and branches into three output heads.The actor LSTM has hidden dimension 256, and its MLP hidden layers are [256, 128, 64].

4) Network Architecture:

The evaluation uses simulation baselines, explicit failure-mode metrics, and a scheduled reward configuration to assess the proposed architecture. Results are aggregated across five random seeds and show improved success with Multi-Critic PPO plus weight scheduling.

  • Network Architecture: The critic weights use a linear schedule, with exploration annealed from 0.1 to 0.01 between 5k and 10k steps.The task weight remains fixed at wtask = 0.75, while the regularization weight increases from 0.15 to 0.24.
  • Quantitative Analysis: The simulation evaluation compares Multi-Critic PPO + WS against PPO, PPO + WS, and fixed-weight Multi-Critic PPO baselines.The fixed-weight Multi-Critic baseline uses wtask = 0.75, wexp = 0.1, and wreg = 0.15 throughout training.
  • Quantitative Analysis: Success Rate measures whether the object reaches the goal within 0.2m, while failure modes include Missed Contact, Tipover, and Timeout.Tipover is defined as tilting beyond 35 degrees, and Timeout as failing to reach the goal within the time limit.
  • Quantitative Analysis: The results aggregate outcomes from 5 random seeds, with the proposed methods showing lower success-rate variability than PPO.The reported standard deviations are 0.98% for the authors’ methods, 4.2% for Multi-Critic PPO, 1.2% for PPO + WS, and 9.7% for PPO.

1) The Exploration Bottleneck:

Removing the exploration reward prevents meaningful contact discovery, while fixed-weight Multi-Critic learning solves contact but over-prioritizes contact actions during transport. This produces unstable behavior and elevated Tipover risk.

  • The Exploration Bottleneck: 0% Success Rate is obtained by the variant without an exploration reward because it fails to discover meaningful contact with the object.
  • The Exploration Bottleneck: Fixed-weight Multi-Critic PPO solves contact discovery effectively but over-prioritizes contact actions at the expense of transport stability.Its exploration reward remains active throughout the episode, preventing the policy from learning smooth behavior and increasing Tipover.

2) Effects of Fixed-Weight Multi-Critic:

The scheduled method combines decoupled value estimation with weight scheduling and learns orientation-aware chair transportation behaviors. Hardware experiments test transfer, disturbances, and unseen object geometries, but performance drops because abrupt yaw commands degrade state estimation.

  • Effects of Fixed-Weight Multi-Critic: The proposed method achieves 94.1% success, 4.4% tipover, and 9.2 s completion time in simulation.Its completion time is comparable to PPO + WS at 9.1 s.
  • Effects of Fixed-Weight Multi-Critic: The policy selects contact points aligned with the motion direction and positions the mobile base to keep the object within the manipulator’s optimal workspace.This behavior is associated with maximizing control authority while avoiding singularities.
  • Object Generalization: 69.0% aggregate success is achieved on 40/58 trials across four unseen IKEA objects.The policy also manipulates an asymmetrical three-legged table by adapting to hook its legs and increases trajectory amplitude after missed or slipped hooks.
  • Object Generalization: Hardware success drops to 72.9% on ADDE and 57.1% on SANDSBERG despite 94.1% simulation success.The primary failure mode involves lateral approaches that produce abrupt yaw commands, degrade state estimation, and induce odometry errors that can tip objects over.
  • Object Generalization: The policy generalizes to a manually updated moving target by adjusting base velocity and contact forces along the changing trajectory.The authors identify this behavior as supporting use as a local controller for high-level path planners.

2) Dynamic Goal Tracking:

The policy handles changing contacts, disturbances, payloads, and object geometries during non-prehensile manipulation. It also transfers contact-guided behavior from simulation to physical objects without additional fine-tuning.

  • Dynamic Goal Tracking:: 6.5 kg chair transport exceeded the robot arm’s rated static payload when the mobile base and ground support were leveraged.The chair’s total mass was increased to 6.5 kg, beyond the arm’s rated static payload when fully extended.
  • Dynamic Goal Tracking:: After forced disengagement, the robot autonomously re-planned its approach, re-hooked the chair, and completed transport.This closed-loop recovery was observed when an operator physically pushed the chair during transport.
  • Dynamic Goal Tracking:: The policy switched from the dishwasher handle to the door panel as the door orientation changed, using distinct contacts across task stages.The handle initiated opening, while panel contact pushed the door to a fully open horizontal position.
  • Dynamic Goal Tracking:: The real-world policy transported multiple chair models and a backrest-free three-legged table without additional fine-tuning.For the table, the base reoriented to maintain an optimal contact normal, demonstrating transfer from simulation to physical geometries.
  • Dynamic Goal Tracking:: With an additional 5 kg payload or abrupt human disturbance, the system maintained transport robustness and immediately re-planned contact trajectories.The perturbation evaluation covered unseen changes in inertia and friction as well as real-time object shifts.
  • Dynamic Goal Tracking:: The method reduced by 59% the fraction of timesteps with an arm joint within 10% of its position limit.This reduction contrasts with a quasi-static handle-pulling strategy that often operated near joint limits.
  • Dynamic Goal Tracking:: After an initial slip or failed contact, the agent repositioned the base, increased arm swing amplitude, and re-established stable contact.The recovery sequence prevented entry into an unstable state before transportation resumed.

V. CONCLUSION

The conclusion presents Multi-Critic RL as a way to separate exploration from task performance, encouraging early contact-rich behavior before shifting toward task objectives. Experiments indicate robustness to perturbations, generalization to novel objects and payloads, and manipulation beyond the arm’s rated static payload, while broader domain testing and reduced reliance on motion capture remain open needs.

  • V. CONCLUSION: Multi-Critic RL decouples exploration and task value functions, encouraging contact-rich interactions early and gradually emphasizing task objectives.The architecture is designed to address sub-optimal local minima in complex manipulation tasks.
  • V. CONCLUSION: The learned policies were robust to external perturbations and generalized to novel objects and payloads across simulated environments and real-world hardware trials.The conclusion reports this pattern across the paper’s evaluations without reducing it to a single task or metric.
  • V. CONCLUSION: Objects exceeding the robotic arm’s rated static payload were successfully manipulated by leveraging contact dynamics and whole-body control for heavy-duty transport.This conclusion identifies the mobile system’s contact dynamics and whole-body behavior as central to the demonstrated heavy-load capability.
  • V. CONCLUSION: Broader domain-shift testing is still required, and reliance on external motion capture limits field deployment.The paper also identifies adaptive performance-based critic schedules as a possible way to reduce hyperparameter sensitivity and improve training autonomy.
Loading 2608.28140v1…