Source-linked AI summary
MMGNet: Multi-Modal 6-DoF Grasp Synthesis for Cluttered Scenes
Rui Cao, Jia-Hui Pan, Yichuan Li, Yunfan Yu, Chi-Wing Fu, Jiangliu Wang, Yun-Hui Liu
TL;DR
Scene-level 6-DoF grasp synthesis depends on point clouds for geometry, though real-world observations can be noisy or incomplete, and image cues are not fully used in scene representation and local reasoning. MMGNet injects multi-scale image features throughout 3D representation formation and uses rotation-aware local grouping. It reports 60.34 mean AP on RealSense scenes and an 8.46-percentage-point real-robot success-rate improvement over AnyGrasp on adversarial objects.
Problem
Scene-level 6-DoF grasp synthesis needs metric geometry, but point observations can be noisy, incomplete, and over-smoothed, while image cues face bottlenecks in scene representation and local grasp reasoning.
Source: §1 · p. 2Method
MMGNet injects aligned multi-scale image features throughout a sparse 3D backbone, then uses rotational graspness to select approach direction and in-plane rotation before gripper-frame local grouping.
Source: §1 · pp. 2–3Results
MMGNet achieves 60.34 mean AP on GraspNet-1Billion RealSense scenes and improves real-robot grasp success over AnyGrasp by 8.46 percentage points on adversarial objects.
Source: §1 · p. 3Takeaways & Limitations
Ablations show complementary gains from hierarchical image-to-3D feature injection and rotational graspness modeling.
Source: §1 · p. 3The model relies on accurate image–point correspondence and lacks an explicit fallback when RGB information becomes unreliable.
Source: §5 · p. 25
Abstract
from arXiv · showhide
Grasping is a foundational capability in robotic manipulation. Scene-level 6-DoF grasp synthesis relies on point clouds for metric geometry, yet real-world point observations are often noisy, incomplete, and over-smoothed. RGB images provide object-region, boundary, and appearance cues, yet two bottlenecks limit their effective use in scene representation and local grasp reasoning. We present MMGNet, a Multi-Modal Grasp Synthesis Network that addresses these bottlenecks through hierarchical image-to-3D feature injection and rotational graspness modeling. Hierarchical injection integrates multi-scale image features throughout 3D representation formation, while the rotational graspness jointly predicts the grasp approach direction and in-plane rotation before organizing local image-informed features through grasp-shaded grouping. Ablation studies show that these two modules provide complementary gains. MMGNet achieves a mean AP of 60.34 on GraspNet-1Billion with RealSense, surpassing all compared methods. In real-world experiments with adversarial grasp objects, MMGNet improves the grasp success rate over AnyGrasp by 8.46 percentage points. \emph{Code, models, and data will be released upon publication.}
1. Introduction
MMGNet addresses limits in image use for scene-level 6-DoF grasp synthesis with hierarchical image-to-3D feature injection and rotation-aware local reasoning. It reports strong benchmark and real-robot results, with complementary gains from its two modules.
1. Introduction
- Point-cloud methods provide metric geometry, but real-world observations can be noisy, incomplete, or over-smoothed, while RGB offers complementary region, boundary, and appearance cues.The introduction identifies limited use of image context during 3D representation formation and rotation-insensitive local representations as two bottlenecks.Source: §1 · p. 2
- Hierarchical image-to-3D feature injection aligns multi-scale image features with scene points and injects them at successive sparse-backbone stages.This lets image context shape the 3D representation throughout backbone formation.Source: §1 · pp. 2–4
- Rotational graspness jointly selects a grasp approach direction and in-plane rotation before grasp-shaded grouping organizes local image-informed features in the selected gripper frame.The operation head then predicts approach depth and gripper width.Source: §1 · pp. 3–4
- MMGNet reaches 60.34 mean AP on GraspNet-1Billion RealSense scenes and improves real-robot grasp success over AnyGrasp by 8.46 percentage points on adversarial objects.It also achieves 50.72 mean AP on Kinect scenes.Source: §1 · p. 3
- Ablations report gains of 5.39 AP points for hierarchical injection and 8.39 points for rotational graspness, plus a 2.19-point positive interaction.These results support the modules’ complementarity.Source: §1 · pp. 3–4
2. Related Work
Prior scene-level 6-DoF grasp methods use point clouds or richer visual information through varied fusion and reconstruction strategies. MMGNet combines multi-scale image-to-3D injection with rotation-aware local grasp prediction.
2.1. 6-DoF Grasp Synthesis for Cluttered Scenes
- Scene-level 6-DoF grasp work spans point-cloud candidate selection and methods using image cues, shape reconstruction, local modeling, or cross-view aggregation.Point-cloud methods remain dependent on partial and potentially noisy observations; MMGNet retains the scene-level point-cloud paradigm while adding multi-scale image information and rotation-aware local prediction.Source: §2.1 · pp. 4–5
2.2. Image-to-3D Fusion for 3D Perception
- Image-to-3D fusion has been studied at input, intermediate, and output levels, and MMGNet injects aligned multi-scale image features into successive sparse-backbone stages.Its selected grasp rotations then guide grouping of local image-informed features in corresponding gripper frames.Source: §2.2 · pp. 5–6
3. Method
MMGNet builds an image-informed 3D representation through hierarchical feature injection, then selects joint grasp rotations to guide gripper-aligned local aggregation.
3.1. Problem Formulation and Pipeline Overview
- Given a single-view RGB-D observation, MMGNet forms an image-informed 3D scene representation, identifies foreground points, and uses FPS to select spatially distributed grasp seeds.Source: §3.1 · p. 6, §3.2 · p. 9
3.2. Hierarchical Image-to-3D Feature Injection
- Multi-scale image features are aligned with scene points and injected at successive sparse-backbone stages, enriching 3D features without changing point coordinates.Source: §3.2 · p. 8
3.3. Rotational Graspness Model
- Rotational graspness assigns separate targets to joint view–in-plane rotations, selects the highest-scoring rotation, and guides local aggregation in its gripper frame.Source: §3.3 · p. 10
3.4. Optimization
- Training combines point-wise objectness, rotational-graspness, grasp-score, and width losses, applying width loss only to positive-score grasp poses.Source: §3.4 · p. 11
4. Experiments
MMGNet is evaluated on GraspNet-1Billion, through ablations and robustness tests, and in real-robot trials. Results compare its grasp performance across camera settings and show how image injection and rotational grasp reasoning contribute.
4.1. Implementation Details
- The benchmark uses 100 training and 90 testing scenes captured from 256 views with RealSense and Kinect cameras; AP follows the GraspNet-1Billion protocol.Testing scenes are split into seen, similar, and novel categories; evaluation applies NMS before retrieving the top 50 grasps.Source: §4.1 · pp. 12–13
4.2. Comparison with State-of-the-Art Methods
- 60.34 mean AP: MMGNet achieves the highest mean AP on RealSense, surpassing FlexLoG by 4.32; on Kinect it reaches 50.72 versus the prior 50.67 and has the highest Novel-split AP, 25.49.The comparison follows the GraspNet-1Billion evaluation protocol.Source: §4.2 · p. 13, Table 1 · p. 14
4.3. Ablation Studies and Representation Analysis
- Under hierarchical injection, rotational graspness raises mean AP from 51.95 to 56.87 with cylinder grouping, and grasp-shaded grouping adds 3.47 points to reach 60.34.The rotational model also improves over the View model without image features, from 48.75 to 54.95; reducing grouped samples from 32 to 8 costs it 1.08 points, versus 4.74 for cylinder grouping. Hierarchical injection also improves GSNet and Ma et al. by 3.20 and 3.33 AP points, respectively.Source: §4.3 · pp. 15, 16, 19, 20
- Hierarchical injection improves mean AP by 5.39 points and produces stronger cross-modal correspondence across stages and scales than early or late fusion.At the shallow s2 stage, CKA reaches 0.82/0.84 with p1/p2, compared with 0.58/0.52 for early fusion and approximately 0.10 for late-fusion variants.Source: §4.3 · pp. 18–19, Figure 3 · p. 19
4.4. Robustness under Point and RGB Corruptions
- 23.75 AP points: under strongest Gaussian noise, hierarchical injection raises AP from 15.53 to 39.28; MMGNet leads absolute AP under all point corruptions and remains best under RGB corruption.Late (Concat) and Late (Gate) fall below Point-only after RGB corruption; cutout causes the largest degradation among the reported RGB corruptions.Source: §4.4 · pp. 21–22
- 71.74 ms: MMGNet inference is slower than GSNet and AnyGrasp, at 33.82 and 44.55 ms, respectively, but faster than Ma et al., at 784.05 ms.MMGNet has 44.14M parameters and requires 304.47 profiled GFLOPs per sample.Source: §4.4 · p. 22
4.5. Real-world Evaluation
- MMGNet reaches 84.06% success on Household objects and 81.43% on Adversarial objects, exceeding AnyGrasp by 1.35 and 8.46 percentage points, respectively.Each object group is evaluated over 20 trials in cluttered scenes; the setup uses a Franka Panda arm and a gripper-mounted RealSense D415 camera.Source: §4.5 · pp. 22–23
4.6. Failure Mode Analysis
- Three representative real-world failures are execution-angle filtering that leaves a less reliable candidate, object slippage, and inaccurate grasp depth that can prevent force closure.The paper notes slippage on low-friction objects such as soap.Source: §4.6 · p. 24
5. Conclusion
MMGNet combines hierarchical image-to-3D feature injection with rotational graspness modeling for scene-level 6-DoF grasp synthesis. It reports gains on GraspNet-1Billion and in real-robot grasp success, while relying on accurate image–point correspondence and adding image-backbone computation.
5. Conclusion
- 60.34 mean AP on the RealSense setting of GraspNet-1Billion; MMGNet improves real-robot grasp success over AnyGrasp by 8.46 percentage points on challenging objects.The authors also report that hierarchical injection gains on other grasp models suggest broader applicability of the image-to-3D interface.Source: §5 · p. 25
- Hierarchical image-to-3D injection and rotational graspness modeling let multi-scale image context shape 3D representation and guide local feature aggregation by selected grasp rotation.Ablation and representation analyses show improved image–3D correspondence and retention of grasp-quality information among candidate rotations.Source: §5 · p. 25
- The model relies on accurate image–point correspondence, has no explicit fallback for unreliable RGB information, and incurs extra computation from its image backbone.Future work will explore reliability-aware and more efficient image-to-3D fusion, execution-aware grasp scoring, and online feedback for avoiding repeated failures.Source: §5 · pp. 25–26