Source-linked AI summary
MS-MEM: Multi-Skill Manipulation-Enhanced Mapping via Uncertainty- and Disturbance-Aware Action Selection
Yitian Shi, Jesper Mücke, Nils Dengler, Sicong Pan, Rania Rayyes, Maren Bennewitz
TL;DR
Confined, cluttered scenes challenge robots because occlusion and restricted access limit reliable scene understanding while manipulation can disturb mapped regions. MS-MEM integrates active viewing, pushing, and grasping with evidential uncertainty representations and disturbance-aware action selection. The reported results show improved mapping from combined skills and substantially less unnecessary scene change when collateral disturbance is penalized.
Problem
Severe occlusions, restricted accessibility, and potential scene changes make reliable mapping and object retrieval difficult in confined cluttered environments.
Method
MS-MEM combines evidential metric-semantic mapping, full-evidential grasp learning, unified information-gain action selection, and a collateral disturbance constraint.
Results
Combining pushing and grasping achieves higher mapping accuracy than single-skill alternatives, while CDC substantially reduces unnecessary object-position changes.
Takeaways & Limitations
Push-grasp synergy supports more geometrically complete and semantically accurate evidential maps while balancing occlusion reduction against collateral scene disturbance.
Abstract
from arXiv · showhide
Accurate scene understanding in confined, cluttered spaces such as shelves is essential for service robots, as many everyday tasks require them to locate and retrieve objects reliably. Yet, it remains challenging due to severe occlusions, restricted accessibility, and the need to avoid excessive scene changes. In this paper, we propose Multi-Skill Manipulation-Enhanced Mapping (MS-MEM), an evidential framework for uncertainty-aware mapping that integrates active viewpoint selection, object pushing, and grasping. MS-MEM combines scene-level metric-semantic evidential belief estimators with an uncertainty-aware grasp representation. This representation is learned using a novel full-evidential grasp estimator that models both grasp affordance and orientation uncertainty. In our framework, candidate perception and manipulation actions are evaluated within a unified action selection pipeline using a common information gain criterion. For manipulation actions, we further introduce a collateral disturbance constraint (CDC) that discourages excessive changes to confident regions of the scene belief. This enables MS-MEM to select actions that effectively reduce map uncertainty while limiting collateral scene changes. Experimental results show that, compared with single-skill and unconstrained baselines that ignore scene disturbance, MS-MEM achieves higher mapping accuracy while substantially reducing scene disturbance, highlighting the synergistic effects of active viewpoint selection, push, and grasp actions.
I. INTRODUCTION
MS-MEM addresses unreliable scene interpretation in cluttered, confined environments by coupling perception with manipulation. It unifies active viewpoint selection, pushing, and grasping to reduce uncertainty while preserving scene evidence.
- Severe occlusion and restricted access make passive or active sensing alone insufficient for reliable scene interpretation.
- Prior MEM formulated uncertain mapping with active views and pushing, but remained limited to pushing as its manipulation primitive.
- MS-MEM selects active views, pushes, and grasps according to expected uncertainty reduction and scene preservation.
- Combining pushing and grasping improves geometric completeness and semantic accuracy over single-skill alternatives.
- Its DOIG objective compares heterogeneous actions under a common uncertainty-aware criterion while penalizing unnecessary scene changes.
C. Uncertainty-Aware Grasp Learning
The framework extends evidential grasp learning and scene mapping to represent uncertainty in both grasp predictions and metric-semantic beliefs. Learned belief updates support efficient prediction of post-action states for information-driven action evaluation.
- Full-evidential grasp representations model uncertainty in grasp affordance and orientation rather than directional uncertainty alone.
- The scene uses metric-semantic evidential beliefs whose expected values provide occupancy and semantic maps.
- Occupancy is modeled with Beta distributions, while semantic probabilities use Dirichlet distributions over known classes.
- CNABU networks update evidential beliefs from actions and observations, enabling efficient prediction of post-action beliefs.
B. Hierarchical Bayesian Grasp Representation
The hierarchical grasp representation decomposes each contact-point grasp into affordance and two orthogonal orientation components. A vMF distribution models baseline directional uncertainty for uncertainty-aware prediction and multi-view aggregation.
- A grasp at contact point c is represented hierarchically through affordance, baseline, and approach components.
- The affordance score q indicates contact-point graspability.
- The approach direction a specifies gripper approach, while baseline direction b represents the closing direction between fingers.
- Baseline uncertainty is modeled with a von Mises-Fisher distribution whose mean direction and concentration parameter characterize directional predictions.
- This representation supports uncertainty-aware grasp prediction and Bayesian aggregation across multiple views.
IV. METHODOLOGY
MS-MEM jointly evaluates viewpoint, push, and grasp actions over a shared evidential scene belief. Its DOIG objective balances uncertainty reduction against collateral disturbance, while manipulation actions are paired with subsequent viewpoint updates.
- MS-MEM integrates grasping, pushing, and active view selection into a closed-loop decision process over a shared evidential scene belief.
- Candidate actions from all three modalities are generated and evaluated jointly under the common DOIG objective.This enables direct comparison between sensing and manipulation actions.
- DOIG extends occlusion-aware information gain by accounting for uncertainty and predicted post-manipulation scene changes.For manipulation, the system predicts the resulting belief before evaluating information gain.
- After selecting a manipulation action, MS-MEM executes its associated next-best viewpoint and integrates the resulting observation into the belief state.
- The collateral disturbance constraint penalizes semantic changes in confident regions to preserve voxel-level scene structure.The constraint targets disturbed confident voxels and is incorporated through a Lagrangian relaxation.
- For viewpoint actions, DOIG reduces to OIG, whereas manipulation is selected only when expected occlusion reduction outweighs predicted collateral disturbance.The grasped object’s intentional removal is excluded from the collateral-disturbance calculation.
C. Uncertainty-Informed Grasp Selection
The uncertainty-informed grasp pipeline predicts evidential grasp hypotheses, refines them across observations, and jointly selects a grasp with a subsequent viewpoint using DOIG.
- The uncertainty-informed grasp pipeline extends MEM with selective occluder removal through grasp actions.
- FE-vMF predicts contact-level grasp hypotheses with evidential uncertainty from the current occupancy belief.
- Uncertainty-guided multi-view grasp fusion accumulates and refines hypotheses over time.
- A grasp CNABU predicts the post-grasp belief, enabling joint selection of the grasp and subsequent viewpoint under DOIG.
1) Full-Evidential Grasp Learning:
Full-Evidential Grasp Learning represents both grasp affordance and orientation uncertainty. FE-vMF uses evidential contact predictions and models the two orientation components independently before enforcing their orthogonality.
- FE-vMF extends vMF-Contact to a full-evidential representation of the SE(3) grasp configuration.
- The UGS pipeline accumulates grasp hypotheses in a global buffer before DOIG selects the grasp and subsequent viewpoint.
- FE-vMF models baseline and intermediate approach directions as independent von Mises–Fisher distributions.This dual representation captures uncertainty in both orientation components.
- The final approach distribution is obtained by projecting the intermediate approach estimate onto the subspace orthogonal to the baseline.This enforces the grasp constraint a ⊥b.
- Grasp affordance at each contact is represented as a Beta distribution whose total evidence determines predictive affordance and epistemic uncertainty.
- The model predicts evidential grasp parameters for visible contact points from a frontier surface point cloud.The point cloud is processed with a PTv3 backbone followed by an MLP.
- The composite evidential loss supervises orientation distributions for positive contacts and affordance evidence for all contacts.
2) Full-Evidential Uncertainty-guided Multi-view Grasp Fusion:
FE-UMGF fuses evidential grasp predictions across time as the map belief evolves. It updates affordance and orientation evidence, then evaluates fused grasps and subsequent viewpoints under DOIG.
- FE-UMGF integrates evidential grasp predictions across time while maintaining consistency with the evolving map belief.
- A global grasp buffer is updated by matching current predictions with previously accumulated hypotheses.
- Fused affordance evidence aggregates newly assigned neighboring candidates while decaying historical evidence by factor γ.The decay factor gives greater relative weight to the current belief while retaining earlier evidence.
- Baseline and approach orientations are fused independently within each cluster, after which the approach direction is recovered by orthogonal projection.
- The predicted post-grasp belief is used to evaluate fused grasps and their best subsequent viewpoints under DOIG.
- Push candidates are likewise evaluated by predicting their post-push beliefs for the shared action-selection process.
D. Multi-Skill Action Selection
MSAS compares candidate grasping, pushing, and viewpoint actions through predicted post-action beliefs and a shared DOIG objective, then executes the selected action and updates the evidential map.
- MSAS selects among optimal grasping, pushing, and viewpoint candidates using predicted post-action beliefs and the shared DOIG objective.
- After manipulation selection, the system executes the associated next-best viewpoint and updates the global evidential belief with observation CNABU.
- Manipulation actions are disabled once the fraction of semantic map cells exceeding confidence threshold τ_conf is sufficiently high.
- The action-selection process terminates when the step budget T_max is exhausted.
V. EXPERIMENTS
The experiments assess multi-skill mapping and uncertainty-aware grasping in simulated confined shelf environments using generated training data and controlled evaluation scenes.
- Experiments assess mapping performance, multi-skill action selection, scene preservation, and uncertainty-aware grasp reasoning.
- Simulation uses PyBullet with a UR5 manipulator, Robotiq 2F-85 gripper, projection-based cameras, and 300 fixed viewpoint candidates.
- FE-vMF is trained on 4 × 10^3 simulated shelf scenes with randomized object arrangements and 30%–45% occupancy.
- The grasp CNABU training dataset contains 7000 simulated grasp instances with maps before and after execution and volume-based action representations.
B. Simulation Experiments
Simulation compares full MS-MEM with single-skill and unconstrained baselines, showing complementary push–grasp benefits and improved disturbance–accuracy trade-offs with CDC.
- Simulation Experiments: Evaluation uses 25 hand-crafted challenging shelf scenes to analyze mapping performance and action selection under controlled clutter.
- Simulation Experiments: The baselines isolate multi-skill selection and CDC effects: Grasp Only, Push Only, w/o CDC, and full MS-MEM.
- Simulation Experiments: Mapping is measured with occupancy and semantic IoU, while scene disturbance is measured by cumulative unintended object displacement.
- Mapping Performance: Combining pushing and grasping achieves the highest occupancy IoU and clearly improves semantic IoU over single-skill baselines.
- Mapping Performance: Pushing can create grasp opportunities by separating clutter, revealing object boundaries, and increasing free space.
- Mapping Performance: CDC substantially reduces object displacement while maintaining strong mapping performance, with a stronger reduction in pushes than grasps.
3) Evidential Grasp Learning Performance:
FE-vMF represents grasp affordance and orientation uncertainty, while qualitative and real-world results examine uncertainty behavior, grasp effects, and zero-shot transfer.
- Evidential Grasp Learning Performance: FE-vMF affordance uncertainty is high at partially observed contacts, collision-sensitive regions, and intrinsically ambiguous affordances near q_c ≈ 0.5.
- Evidential Grasp Learning Performance: Orientation uncertainty increases when multiple collision-free approach directions are plausible and decreases when obstacles constrain feasible orientations.
- Evidential Grasp Learning Performance: Quantitative calibration shows lower predicted directional precision for larger angular errors, indicating captured orientation uncertainty.
- Real-World Performance: The real-world evaluation transfers the action-selection pipeline zero-shot to a UR5 shelf setup with a Robotiq gripper, RealSense camera, and Vicon pose tracking.
- Real-World Performance: Compared with Push Only, scene position change decreases 28.0%, from 0.567 m to 0.408 m, while grasp success improves 30% over Grasp Only.
- Real-World Performance: MS-MEM jointly leverages pushing, grasping, and active view selection within an evidential mapping framework to balance occlusion reduction and collateral disturbance.