Source-linked AI summary
MLCVNet: Multi-Level Context VoteNet for 3D Object Detection
Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang, Yiming Zhang, Kai Xu, Jun Wang
TL;DR
3D point-cloud object detection must handle noisy, missing, and occluded data, while existing approaches often recognize objects without modeling their relationships. MLCVNet extends VoteNet with patch-, object-, and scene-level context modules, achieving improved and state-of-the-art detection performance on SUN RGB-D and ScanNetV2. The paper concludes that multi-level contextual modeling improves 3D object detection, while richer scene-context analysis remains open for future work.
Problem
Existing 3D detection methods recognize objects individually and lack relationships between point patches, objects, and their surrounding scenes.
Method
MLCVNet extends VoteNet with PPC and OOC self-attention modules plus a GSC multi-scale feature-fusion branch for patch-, object-, and scene-level context.
Results
MLCVNet improves detection accuracy and outperforms state-of-the-art methods on both SUN RGB-D and ScanNetV2.
Takeaways & Limitations
Multi-level contextual information can improve 3D object detection in noisy, cluttered, and occluded indoor scenes.
Takeaways & Limitations
The paper identifies further room for improving global scene-context constraints and the mechanisms used to encode contextual information.
Abstract
from arXiv · showhide
In this paper, we address the 3D object detection task by capturing multi-level contextual information with the self-attention mechanism and multi-scale feature fusion. Most existing 3D object detection methods recognize objects individually, without giving any consideration on contextual information between these objects. Comparatively, we propose Multi-Level Context VoteNet (MLCVNet) to recognize 3D objects correlatively, building on the state-of-the-art VoteNet. We introduce three context modules into the voting and classifying stages of VoteNet to encode contextual information at different levels. Specifically, a Patch-to-Patch Context (PPC) module is employed to capture contextual information between the point patches, before voting for their corresponding object centroid points. Subsequently, an Object-to-Object Context (OOC) module is incorporated before the proposal and classification stage, to capture the contextual information between object candidates. Finally, a Global Scene Context (GSC) module is designed to learn the global scene context. We demonstrate these by capturing contextual information at patch, object and scene levels. Our method is an effective way to promote detection accuracy, achieving new state-of-the-art detection performance on challenging 3D object detection datasets, i.e., SUN RGBD and ScanNet. We also release our code at https://github.com/NUAAXQ/MLCVNet.
1. Introduction
MLCVNet extends VoteNet by modeling contextual relationships across point patches, object candidates, and whole scenes. The approach targets ambiguity and reduced accuracy caused by recognizing indoor objects independently.
- 3D point-cloud detection is difficult because unstructured data and missing or noisy observations complicate direct recognition.
- VoteNet uses Hough-voting-inspired regression to generate object centers and bounding boxes, but processes patches and objects independently.This omission leaves relationships between objects and their surrounding scenes unmodeled.
- Indoor contextual information can disambiguate incomplete point sets, such as identifying a chair from surrounding chairs and a dining-room table.
- MLCVNet models context at patch, object, and global-scene levels using PPC, OOC, and GSC modules.PPC and OOC use self-attention, while GSC uses multi-scale feature fusion.
- Experiments report improved detection accuracy and state-of-the-art performance on both SUN RGB-D and ScanNetV2.The contribution passage also reports that ablations demonstrate the effectiveness of the modules.
2. Related Work
Related work spans deep learning for 2D and 3D detection, contextual modeling, and point-cloud scene understanding. Prior 3D methods often rely on intermediate 2D detectors, while contextual approaches analyze relationships across visual or point-cloud elements.
- Deep convolutional networks substantially improved accuracy and efficiency in 2D object detection, while deep point-cloud learning later enabled 3D detection architectures.
- Many 3D detection methods use 2D detectors as an intermediate step, restricting generalization where those detectors perform poorly.
- Prior 3D work explored point-patch context, hierarchical 3D layout priors, and self-attention-based contextual modeling.
3. Approach
MLCVNet extends VoteNet with three complementary context modules that model relationships among point patches, object candidates, and the global scene. These modules use self-attention and multi-scale feature fusion to improve voting, proposal, classification, and final 3D detection predictions.
- 3. Approach: MLCVNet builds on VoteNet and adds PPC, OOC, and GSC modules for multi-level contextual modeling.The modules target patch, object, and scene contexts within the VoteNet framework.
- 3. Approach: VoteNet processes point patches and object candidates independently, whereas MLCVNet explicitly encodes their relationships.This contextual modeling is introduced to address VoteNet’s omission of surrounding patches, objects, and scene information.
- 3.2. PPC Module: PPC applies CGNL self-attention to point-patch features before voting, aggregating correlations to produce more accurate object-center votes.The module operates on a feature map A ∈ R^1024×D and models correlations between point patches and channels.
- 3.2. PPC Module: With PPC, voted centers are more concentrated on objects and more closely clustered than without the module.The comparison uses green votes with PPC, red votes without PPC, and pink annotated ground-truth centers.
- 3.3. OOC Module: OOC applies CGNL attention to pooled vote-cluster vectors, encoding affinities among object candidates before proposal and classification.The resulting contextual features support 3D bounding-box regression and classification.
- 3.3. OOC Module: With OOC, detected objects overlap less and their predicted positions become more accurate.The qualitative comparison is made between results with and without OOC.
- 3.4. GSC Module: GSC fuses pooled patch- and object-level features into a global scene representation that is combined with OOC features for final prediction.This multi-scale fusion makes bounding-box and class inference consider scene compatibility and reduces false detections.
4. Results and Discussions
MLCVNet is evaluated on SUN RGB-D and ScanNetV2, where it consistently improves detection over VoteNet and other competing methods. Ablations and qualitative comparisons indicate that its contextual modules improve center voting, box precision, and object compatibility.
- Efficiency: MLCVNet increases model size to 13.9MB from VoteNet’s 11.2MB and inference time to 0.14s from 0.13s.Training time also increases from around 40s to 42s per epoch on ScanNet with batch size 8.
- State-of-the-art comparisons: 59.8% overall mAP on SUN RGB-D validation, 2.1% higher than VoteNet.The comparison includes ten common object categories and attributes the improvement to contextual information helping detect occluded objects with missing parts.
- State-of-the-art comparisons: 64.5% mAP@0.25 on ScanNet validation, improving 5.9 absolute points over VoteNet; mAP@0.5 improves by 7.9 points.Improvements exceed 8 points for some categories, especially plane-like objects such as doors, windows, pictures, and shower curtains.
- Ablation study: Adding PPC and OOC raises the evaluation scores to 59.1 and 63.4, while all three contextual modules produce the highest mAP@0.25 scores on both datasets.The ablation study uses VoteNet as the baseline and adds the proposed modules incrementally.
- Qualitative results: Qualitative comparisons show more reasonable objects and more precise boxes on ScanNetV2 and SUN RGB-D, including improved detection in heavily occluded scenes.MLCVNet also avoids some VoteNet overlaps and misclassifications, although overlapped windows remain a failure case.
5. Conclusions
MLCVNet integrates multi-level contextual information into 3D object detection through self-attention, multi-scale feature fusion, and three contextual sub-modules. Experiments show improved detection performance, while future work remains in strengthening global scene context and contextual encoding.
- MLCVNet integrates patch-, object-, and scene-level context into 3D object detection using self-attention and multi-scale feature fusion.PPC models point-patch relationships, OOC captures object-candidate context, and GSC aggregates global scene context.
- Ablation studies demonstrate that the contextual sub-modules improve detection accuracy.
- Quantitative and qualitative experiments demonstrate improved 3D object detection performance.
- Future work includes strengthening global scene context through auxiliary scene-type prediction and developing more effective contextual encoding mechanisms.