Source-linked AI summary
MVImgNet: A Large-scale Dataset of Multi-view Images
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, Guanying Chen, Shuguang Cui, Xiaoguang Han
TL;DR
Real-world 3D vision lacks a generic, ImageNet-scale dataset because collecting and labeling 3D data is difficult. The paper introduces MVImgNet, a large multi-view image dataset collected from object videos, and derives MVPNet through dense reconstruction; pilot studies report promising effectiveness across 3D and 2D tasks.
Problem
Real-world 3D vision lacks a generic dataset comparable in scale to ImageNet because scanning and labeling 3D data are laborious.
Method
MVImgNet collects multi-view videos of real-world objects with masks, camera parameters, and point-cloud annotations, then derives MVPNet through dense reconstruction.
Results
Pilot studies report promising effectiveness for radiance field reconstruction, multi-view stereo, and view-consistent image understanding, while MVPNet benefits real-world 3D object classification.
Takeaways & Limitations
MVImgNet serves as a soft bridge between 2D and 3D vision, and MVPNet provides a labeled real-world 3D object point-cloud dataset.
Takeaways & Limitations
MVImgNet does not consider very complex backgrounds and cannot be straightforwardly adopted for complicated scene-level understanding.
Abstract
from arXiv · showhide
Being data-driven is one of the most iconic properties of deep learning algorithms. The birth of ImageNet drives a remarkable trend of "learning from large-scale data" in computer vision. Pretraining on ImageNet to obtain rich universal representations has been manifested to benefit various 2D visual tasks, and becomes a standard in 2D vision. However, due to the laborious collection of real-world 3D data, there is yet no generic dataset serving as a counterpart of ImageNet in 3D vision, thus how such a dataset can impact the 3D community is unraveled. To remedy this defect, we introduce MVImgNet, a large-scale dataset of multi-view images, which is highly convenient to gain by shooting videos of real-world objects in human daily life. It contains 6.5 million frames from 219,188 videos crossing objects from 238 classes, with rich annotations of object masks, camera parameters, and point clouds. The multi-view attribute endows our dataset with 3D-aware signals, making it a soft bridge between 2D and 3D vision. We conduct pilot studies for probing the potential of MVImgNet on a variety of 3D and 2D visual tasks, including radiance field reconstruction, multi-view stereo, and view-consistent image understanding, where MVImgNet demonstrates promising performance, remaining lots of possibilities for future explorations. Besides, via dense reconstruction on MVImgNet, a 3D object point cloud dataset is derived, called MVPNet, covering 87,200 samples from 150 categories, with the class label on each point cloud. Experiments show that MVPNet can benefit the real-world 3D object classification while posing new challenges to point cloud understanding. MVImgNet and MVPNet will be publicly available, hoping to inspire the broader vision community.
SSE, CUHKSZ FNii, CUHKSZ
MVImgNet is a large-scale multi-view image dataset collected from real-world object videos, while MVPNet is a derived real-world 3D object point-cloud dataset.
- MVImgNet provides 3D-aware signals through multi-view consistency, bridging 2D and 3D vision.
- Pilot studies evaluate MVImgNet on radiance field reconstruction, multi-view stereo, and view-consistent image understanding.
- MVPNet contains 87,200 labeled point clouds from 150 categories and benefits real-world 3D object classification.
- MVImgNet and MVPNet are intended for public release and broader community use.
1. Introduction
The introduction identifies a shortage of large-scale, generic real-world 3D datasets and proposes multi-view images as a practical bridge between 2D and 3D vision.
- ImageNet established large-scale pretraining as a standard for 2D visual representation learning.
- Existing 3D datasets are often synthetic or smaller than ImageNet because real-world 3D data are difficult to scan and label.
- MVImgNet uses crowd-sourced object videos to collect 6.5 million frames from 219,188 videos across 238 classes.
- Pilot experiments examine MVImgNet for radiance fields, multi-view stereo, and view-consistent image understanding.
- MVPNet is derived from MVImgNet with 87,200 point clouds spanning 150 categories and labeled by class.
2. Related Work
Prior datasets primarily target 2D understanding, specialized 3D tasks, or synthetic and limited-scale multi-view settings; MVImgNet is positioned as a broader real-world alternative.
- Single-view image datasets mainly support 2D understanding, limiting their direct application to 3D vision.
- Existing synthetic and specialized multi-view datasets have limited scale and category range for generic 3D learning.
- MVImgNet contains 219,000 videos of real-world objects spanning 238 categories.
- CO3D covers 19,000 videos and 50 categories, whereas MVImgNet is described as substantially larger.
- MVImgNet differs from CO3D by seeking a 3D counterpart to ImageNet and using multi-view images as a bridge between 2D and 3D.
- The later experiments report greater power for MVImgNet than CO3D across different visual challenges.
3. The Basis – MVImgNet Dataset
MVImgNet is built through efficient real-world video capture and automated reconstruction, producing large-scale multi-view data with comprehensive annotations.
- Raw Data Preparation: The acquisition pipeline is designed to collect multi-view object data efficiently from everyday videos.
- Raw Data Preparation: Captured videos are guided by requirements of approximately 10 seconds and non-blurred frames.
- Data Processing: Automatic processing obtains object masks, camera intrinsics and extrinsics, depth maps, and point clouds.
- Dataset Summary: MVImgNet contains 238 object classes, 6.5 million frames, and 219,188 videos with masks, camera parameters, and point-cloud annotations.
- Dataset Summary: The dataset uses WordNet taxonomy and focuses on objects found or used in daily life.
4. 3D Reconstruction
MVImgNet pretraining improves generalizable NeRF reconstruction and data-efficient multi-view stereo, with performance increasing as more MVImgNet data is used.
- Radiance Field Reconstruction: MVImgNet targets generalizable NeRFs by supplying large-scale real-world multi-view data for learning 3D priors.The evaluation measures generalization to unseen objects from few-shot source views.
- Radiance Field Reconstruction: MVImgNet-pretrained IBRNet improves generalization by a large margin over training from scratch across three test-set levels.The test set contains 100 objects from synthetic and real-world object-centric datasets.
- Radiance Field Reconstruction: MVImgNet pretraining performs better than CO3D pretraining, while increasing MVImgNet training data produces an apparent rise in generalization metrics.MVImgNet-small matches CO3D in scale, enabling the comparison.
- Multi-view Stereo: MVImgNet-pretrained MVS improves a model trained from scratch by a large margin when fine-tuned with limited DTU training data.Evaluation uses 5%, 15%, and 25% of DTU training samples and reports depth-map accuracy at different error thresholds.
- Multi-view Stereo: MVImgNet is stronger than CO3D for data-efficient MVS, while MVImgNet-small shows comparable power to CO3D.The comparison fine-tunes models using 5% of DTU training samples.
- Multi-view Stereo: The paper advocates benchmarking NeRF and MVS methods with pretraining on MVImgNet.
5. View-consistent Image Understanding
MVImgNet improves view consistency across supervised classification, contrastive learning, and salient object detection, including gains over CO3D-based training in image recognition.
- View-consistent Image Classification: Mixing MVImgNet views with ImageNet data improves view consistency and accuracy for image classification.For ResNet-50, accuracy rises from 48.76% to 73.88% while confidence variance falls from 0.225 to 0.122.
- View-consistent Image Classification: MVImgNet-based training provides more benefit than CO3D-based training for view-consistent image recognition.
- View-consistent Contrastive Learning: Using two frames from the same video as positive pairs enables MVImgNet fine-tuning of MoCo-v2 for view-consistent contrastive learning.Frames from other videos serve as negatives.
- View-consistent Contrastive Learning: MVImgNet fine-tuning with contrastive learning improves both view consistency and accuracy on the MVImgNet test set.The evaluation compares softmax-confidence variance and mean together with accuracy.
- View-consistent Image Classification: The paper recommends view consistency as a criterion for image recognition evaluation and MVImgNet for model pretraining.
- View-consistent SOD: 4.1% IoU improvement is obtained by fine-tuning U2Net on MVImgNet for hard-view salient object detection.The result is reported for test examples with IoU ≤0.7.
6. MVPNet for 3D Understanding
MVPNet derives a large real-world point-cloud dataset from MVImgNet, benefits point-cloud classification through pretraining, and introduces a challenging benchmark for 3D understanding.
- Dataset: MVPNet contains 87,200 real-world object point clouds spanning 150 categories.It is derived from dense reconstruction on MVImgNet and provides abundant common-object categories.
- Point Cloud Classification: MVPNet pretraining increases ScanObjectNN classification accuracy under most evaluation circumstances.Evaluation includes the PB T50 RS setting and a more challenging version with heavy rotation.
- Self-supervised Point Cloud Pretraining: PointMAE pretrained on MVPNet outperforms state-of-the-art methods after fine-tuning on ScanObjectNN.MVPNet pretraining is also reported as more powerful than CO3D for real-world point-cloud classification.
- MVPNet Benchmark: The MVPNet benchmark provides 64,000 training and 16,000 testing samples for real-world point-cloud classification.
- MVPNet Benchmark: MVPNet is reported as more challenging than ScanObjectNN and is suggested for future point-cloud-understanding research.
7. Conclusion
MVImgNet links 2D and 3D vision through large-scale multi-view data, while MVPNet extends it into labeled real-world point clouds. The datasets show promise but remain limited in category richness and complex-background coverage.
- MVImgNet is a large-scale multi-view image dataset collected by shooting videos of real-world objects.
- Its multi-view nature provides 3D-aware visual signals and connects 2D and 3D vision.
- Pilot studies report promising effectiveness for radiance field reconstruction, multi-view stereo, and view-consistent image understanding.
- MVPNet is derived from MVImgNet as a point cloud dataset that benefits real-world 3D object classification.
- MVImgNet focuses mainly on human-centric classes, limiting category richness relative to ImageNet and potentially weakening recognition of nature-centric classes.
- Complex backgrounds are not considered, so the data cannot be directly applied to complicated scene-level understanding.
Contributions
The listed contributions cover dataset construction, data processing, reconstruction experiments, view-consistent salient object detection, and broader 3D and image-understanding exploration.
- Xianggang Yu contributed to the dataset-building pipeline, including data acquisition and processing.
- Xianggang Yu conducted radiance field reconstruction experiments and designed and advised experiments on view-consistent SOD.
- Mutian Xu advised view-consistent image understanding and 3D understanding, organized and wrote the paper, and led most of the research.
- Yidan Zhang worked on data processing and implemented experiments on view-consistent image understanding.
Supplement for MVImgNet
Supplementary material presents dataset distributions and qualitative examples for MVImgNet and MVPNet, alongside visualizations of radiance-field reconstruction and view-consistent SOD.
- The supplementary contents include sections on 3D reconstruction, view-consistent image understanding, and 3D understanding.
- MVImgNet and MVPNet category taxonomies and per-category distributions are shown in the supplementary figures.The average size is 921 per class for MVImgNet and 581 per class for MVPNet.
- MVImgNet examples show multi-view images with corresponding class labels and differences between viewpoints.
- MVPNet examples show point clouds with distinct texture, noise, and pose, indicating real-world signals.
- Qualitative radiance-field results show the MVImgNet-pretrained model consistently outperforming the train-from-scratch model.
- Fine-tuning U2Net on MVImgNet produces better view-consistent SOD results than the original U2Net.
D. More Experiments of Data Scalability
The scalability experiments examine how increasing MVImgNet data affects multi-view stereo and view-consistent image classification, while supplementary material provides dataset-distribution visualizations and point-cloud transfer results.
- More Experiments of Data Scalability: The experiments add more data from the datasets to evaluate data scalability.
- Dataset distributions: MVImgNet category proportions are visualized through a taxonomy whose class angles represent actual data proportions.
- Dataset distributions: MVPNet category distribution is presented in a supplementary figure.
- Data scalability results: Using larger amounts of MVImgNet videos for pretraining yields higher multi-view stereo depth-map accuracy on the DTU evaluation set.
- Data scalability results: View-consistent image classification progressively adds MVImgNet videos to mixed ImageNet-MVImgNet training for ResNet-50 evaluation.Smaller Confidence Var and higher Accuracy indicate better view consistency.
- Point-cloud transfer: Fine-tuning on ScanObjectNN after MVPNet pretraining improves real-world point-cloud classification for supervised and self-supervised models.
E. More Discussions about Our Datasets
The datasets preserve real-world capture conditions while supporting analyses across 3D reconstruction, point-cloud classification, and view-consistent image understanding. MVImgNet data is also used to improve view consistency in image models through multi-view training and optical-flow-based SOD.
- Dataset construction: ∼219k videos are retained after filtering ∼260k raw videos for bad camera estimations, while MVPNet keeps 87k cleaned point clouds from 90k videos across 150 categories.The 150 most common categories are selected when building MVPNet.
- Dataset construction: Real-world captures retain objects in their original environments, producing data that is easier to collect and point clouds that commonly contain partial views.Examples include objects such as sofas remaining against walls rather than being staged for ideal 360° captures.
- 3D tasks: MVPNet is evaluated for real-world point-cloud classification with 25%, 50%, and 100% pretraining-data ratios under two ScanObjectNN settings.The reported metric is overall / average accuracy, but the supplied table captions do not include the numerical results.
- 3D tasks: MVImgNet is used to pretrain IBRNet before finetuning on the original IBRNet training datasets for radiance-field reconstruction.The protocol uses 8∼12 training views and 10 inference views per object.
- View-consistent image understanding: For view-consistent image understanding, MVImgNet frames are mixed with ImageNet data, sampled as same-video positive pairs, and used with optical-flow consistency for salient object detection.The hybrid datasets use 100k MVImgNet videos and 200k ImageNet images; the optical-flow term is added to the original SOD loss with τ=0.15.