PaperlayerView exact revision on arXiv

Source-linked AI summary

MS-MEM: Multi-Skill Manipulation-Enhanced Mapping via Uncertainty- and Disturbance-Aware Action Selection

Yitian Shi, Jesper Mücke, Nils Dengler, Sicong Pan, Rania Rayyes, Maren Bennewitz

arXiv:2609.02493v1cs.RO

TL;DR

Confined, cluttered scenes challenge robots because occlusion and restricted access limit reliable scene understanding while manipulation can disturb mapped regions. MS-MEM integrates active viewing, pushing, and grasping with evidential uncertainty representations and disturbance-aware action selection. The reported results show improved mapping from combined skills and substantially less unnecessary scene change when collateral disturbance is penalized.

Abstract

from arXiv · show

Accurate scene understanding in confined, cluttered spaces such as shelves is essential for service robots, as many everyday tasks require them to locate and retrieve objects reliably. Yet, it remains challenging due to severe occlusions, restricted accessibility, and the need to avoid excessive scene changes. In this paper, we propose Multi-Skill Manipulation-Enhanced Mapping (MS-MEM), an evidential framework for uncertainty-aware mapping that integrates active viewpoint selection, object pushing, and grasping. MS-MEM combines scene-level metric-semantic evidential belief estimators with an uncertainty-aware grasp representation. This representation is learned using a novel full-evidential grasp estimator that models both grasp affordance and orientation uncertainty. In our framework, candidate perception and manipulation actions are evaluated within a unified action selection pipeline using a common information gain criterion. For manipulation actions, we further introduce a collateral disturbance constraint (CDC) that discourages excessive changes to confident regions of the scene belief. This enables MS-MEM to select actions that effectively reduce map uncertainty while limiting collateral scene changes. Experimental results show that, compared with single-skill and unconstrained baselines that ignore scene disturbance, MS-MEM achieves higher mapping accuracy while substantially reducing scene disturbance, highlighting the synergistic effects of active viewpoint selection, push, and grasp actions.

I. INTRODUCTION

MS-MEM addresses unreliable scene interpretation in cluttered, confined environments by coupling perception with manipulation. It unifies active viewpoint selection, pushing, and grasping to reduce uncertainty while preserving scene evidence.

  • Severe occlusion and restricted access make passive or active sensing alone insufficient for reliable scene interpretation.Source: §I · p. 1
  • Prior MEM formulated uncertain mapping with active views and pushing, but remained limited to pushing as its manipulation primitive.Source: §I · p. 1, A. Manipulation for Mechanical Search… · p. 2
  • MS-MEM selects active views, pushes, and grasps according to expected uncertainty reduction and scene preservation.Source: §I · pp. 1–2
  • Combining pushing and grasping improves geometric completeness and semantic accuracy over single-skill alternatives.Source: §I · p. 2, B. Push-Grasp Synergy · p. 2
  • Its DOIG objective compares heterogeneous actions under a common uncertainty-aware criterion while penalizing unnecessary scene changes.Source: §I · p. 2

C. Uncertainty-Aware Grasp Learning

The framework extends evidential grasp learning and scene mapping to represent uncertainty in both grasp predictions and metric-semantic beliefs. Learned belief updates support efficient prediction of post-action states for information-driven action evaluation.

B. Hierarchical Bayesian Grasp Representation

The hierarchical grasp representation decomposes each contact-point grasp into affordance and two orthogonal orientation components. A vMF distribution models baseline directional uncertainty for uncertainty-aware prediction and multi-view aggregation.

IV. METHODOLOGY

MS-MEM jointly evaluates viewpoint, push, and grasp actions over a shared evidential scene belief. Its DOIG objective balances uncertainty reduction against collateral disturbance, while manipulation actions are paired with subsequent viewpoint updates.

  • MS-MEM integrates grasping, pushing, and active view selection into a closed-loop decision process over a shared evidential scene belief.Source: §IV · p. 3
  • Candidate actions from all three modalities are generated and evaluated jointly under the common DOIG objective.This enables direct comparison between sensing and manipulation actions.Source: A. Overview · p. 3, B. Disturbance- and Occlusion-Aware… · p. 3
  • DOIG extends occlusion-aware information gain by accounting for uncertainty and predicted post-manipulation scene changes.For manipulation, the system predicts the resulting belief before evaluating information gain.Source: B. Disturbance- and Occlusion-Aware… · pp. 3–4
  • After selecting a manipulation action, MS-MEM executes its associated next-best viewpoint and integrates the resulting observation into the belief state.Source: B. Disturbance- and Occlusion-Aware… · p. 3, Figure 2 · p. 4
  • The collateral disturbance constraint penalizes semantic changes in confident regions to preserve voxel-level scene structure.The constraint targets disturbed confident voxels and is incorporated through a Lagrangian relaxation.Source: B. Disturbance- and Occlusion-Aware… · p. 4
  • For viewpoint actions, DOIG reduces to OIG, whereas manipulation is selected only when expected occlusion reduction outweighs predicted collateral disturbance.The grasped object’s intentional removal is excluded from the collateral-disturbance calculation.Source: B. Disturbance- and Occlusion-Aware… · p. 4

C. Uncertainty-Informed Grasp Selection

The uncertainty-informed grasp pipeline predicts evidential grasp hypotheses, refines them across observations, and jointly selects a grasp with a subsequent viewpoint using DOIG.

1) Full-Evidential Grasp Learning:

Full-Evidential Grasp Learning represents both grasp affordance and orientation uncertainty. FE-vMF uses evidential contact predictions and models the two orientation components independently before enforcing their orthogonality.

2) Full-Evidential Uncertainty-guided Multi-view Grasp Fusion:

FE-UMGF fuses evidential grasp predictions across time as the map belief evolves. It updates affordance and orientation evidence, then evaluates fused grasps and subsequent viewpoints under DOIG.

D. Multi-Skill Action Selection

MSAS compares candidate grasping, pushing, and viewpoint actions through predicted post-action beliefs and a shared DOIG objective, then executes the selected action and updates the evidential map.

V. EXPERIMENTS

The experiments assess multi-skill mapping and uncertainty-aware grasping in simulated confined shelf environments using generated training data and controlled evaluation scenes.

  • Experiments assess mapping performance, multi-skill action selection, scene preservation, and uncertainty-aware grasp reasoning.Source: §V · p. 6
  • Simulation uses PyBullet with a UR5 manipulator, Robotiq 2F-85 gripper, projection-based cameras, and 300 fixed viewpoint candidates.Source: 1) Data Generation · p. 6
  • FE-vMF is trained on 4 × 10^3 simulated shelf scenes with randomized object arrangements and 30%–45% occupancy.Source: 1) Data Generation · p. 6
  • The grasp CNABU training dataset contains 7000 simulated grasp instances with maps before and after execution and volume-based action representations.Source: 1) Data Generation · p. 6

B. Simulation Experiments

Simulation compares full MS-MEM with single-skill and unconstrained baselines, showing complementary push–grasp benefits and improved disturbance–accuracy trade-offs with CDC.

Simulation Experiments

  • Evaluation uses 25 hand-crafted challenging shelf scenes to analyze mapping performance and action selection under controlled clutter.Source: B. Simulation Experiments · p. 6
  • The baselines isolate multi-skill selection and CDC effects: Grasp Only, Push Only, w/o CDC, and full MS-MEM.Source: 1) Baselines and Metrics · p. 6
  • Mapping is measured with occupancy and semantic IoU, while scene disturbance is measured by cumulative unintended object displacement.Source: 1) Baselines and Metrics · p. 6

Mapping Performance

  • Combining pushing and grasping achieves the highest occupancy IoU and clearly improves semantic IoU over single-skill baselines.Source: 2) Mapping Performance · p. 7
  • Pushing can create grasp opportunities by separating clutter, revealing object boundaries, and increasing free space.Source: 2) Mapping Performance · p. 7
  • CDC substantially reduces object displacement while maintaining strong mapping performance, with a stronger reduction in pushes than grasps.Source: 2) Mapping Performance · p. 7

3) Evidential Grasp Learning Performance:

FE-vMF represents grasp affordance and orientation uncertainty, while qualitative and real-world results examine uncertainty behavior, grasp effects, and zero-shot transfer.

Evidential Grasp Learning Performance

Real-World Performance

  • The real-world evaluation transfers the action-selection pipeline zero-shot to a UR5 shelf setup with a Robotiq gripper, RealSense camera, and Vicon pose tracking.Source: C. Real-World Performance · p. 7
  • Compared with Push Only, scene position change decreases 28.0%, from 0.567 m to 0.408 m, while grasp success improves 30% over Grasp Only.Source: C. Real-World Performance · p. 8
  • MS-MEM jointly leverages pushing, grasping, and active view selection within an evidential mapping framework to balance occlusion reduction and collateral disturbance.Source: §VI · p. 8

Related papers

Loading 2609.02493v1…