MVImgNet: A Large-scale Dataset of Multi-view Images

SSE, CUHKSZ FNii, CUHKSZ

• MVImgNet collects multi-view images by filming real-world objects and uses multi-view consistency to bridge 2D and 3D vision.The figure caption describes MVImgNet as a large-scale dataset with 3D-aware signals.Source: Figure 1 · p. 1
• Dense reconstruction on MVImgNet also yields MVPNet, a large-scale real-world 3D object point-cloud dataset.Source: Figure 1 · p. 1

• ImageNet pretraining is standard in 2D, but existing 3D datasets are synthetic, limited in scale, or domain-specific, leaving no generic 3D counterpart and unclear benefits.ImageNet itself lacks 3D-aware signals.Source: §1 · p. 2
• MVImgNet contains 6.5 million frames from 219,188 videos of real-world objects across 238 classes, with object masks, camera parameters, and point clouds.The multi-view images can be collected by filming objects from different viewpoints with common mobile devices.Source: §1 · p. 2
• Pilot experiments report that MVImgNet pretraining benefits NeRF generalization and data-efficient multi-view stereo, and improves view consistency in classification and salient object detection.The view-consistency experiments include supervised and self-supervised contrastive classification.Source: §1 · p. 2
• Dense reconstruction on MVImgNet yields MVPNet, a dataset of 87,200 labeled point clouds spanning 150 categories that benefits real-world 3D classification and poses new challenges for point-cloud understanding.Source: §1 · p. 2
• The authors state that MVImgNet and MVPNet will be publicly available to support the broader vision community.Source: §1 · p. 2

Related Work

• Single-view image datasets mainly support 2D understanding, limiting their 3D-vision applications; video datasets primarily target video understanding rather than MVImgNet’s objective.The surveyed resources include image datasets such as ImageNet and MSCOCO, and video datasets such as Kinetics and HowTo100M.Source: §2 · pp. 2–3
• Earlier real-object multi-view datasets offered approximate models or few instances; Objectron lacked camera poses or point clouds for all its roughly 15k videos, and GSO had limited samples.Synthetic multi-view datasets also exist, while prior collections were described as small in scale and category range.Source: §2 · p. 3
• Existing 3D datasets include synthetic CAD models with low quality, no textures, and a domain gap from real objects, while real-object datasets remain limited in scale.The paper contrasts these resources with collecting multi-view images of real-world objects for scalable acquisition and 3D awareness.Source: §2 · p. 3
• 219k videos across 238 categories give MVImgNet conspicuously greater scale than CO3D’s 19k videos across 50 categories.The paper also reports that MVImgNet performs better than CO3D on different visual challenges in later experiments.Source: §2 · p. 3
• MVImgNet aims to be an ImageNet-like 3D counterpart, using multi-view images as a soft bridge between 2D and 3D rather than targeting only reconstruction transfer.The authors conduct pilot studies across varied visual tasks to inspire broader exploration.Source: §2 · p. 3

3.1. Raw Data Preparation

• MVImgNet uses around 1,000 collectors across professions and ages 20–50 to film in daily life; about 200 trained expert cleaners review each submission against capture requirements, supporting diversity and quality of raw videos.Guidelines call for videos about 10 seconds long, unblurred, capturing as much of a 180° or 360° view as possible, showing the object above 15% of the frame, containing one class of principal object, and capturing a stereoscopic object.Source: §3.1 · p. 4

3.2. Data Processing

• Automated processing produces object masks, camera intrinsics and extrinsics, depth maps, and point clouds using COLMAP SfM [74], CarveKit, and multi-view stereo.The point clouds are fused from masked object depths and manually cleaned to remove noisy reconstructions and backgrounds.Source: §3.2 · p. 4

3.3. Dataset Summary

• 6.5 million frames across 219,188 videos cover 238 human-centric everyday-object classes, with object masks, camera parameters, and point clouds; 65 classes overlap with ImageNet.The classes use the WordNet taxonomy and focus on objects found or used in daily life, unlike ImageNet’s nature-centric emphasis.Source: §3.3 · p. 4

4.1. Radiance Field Reconstruction

• MVImgNet pretraining improves IBRNet’s generalization to unseen scenes by a large margin over training from scratch.IBRNet was pretrained on the full MVImgNet dataset, then finetuned for a few iterations on its original training datasets; all methods used the same evaluation protocol.Source: Table 3 · p. 5, §4.1 · p. 5
• The NeRF generalization test set spans 100 objects from 56 categories, including synthetic and real-world 360° object datasets.It combines 4 diffuse synthetic objects, 8 realistic synthetic objects, and 88 real-world scenes under different lighting conditions.Source: §4.1 · p. 5
• MVImgNet shows greater power than CO3D for NeRF pretraining, and generalization metrics rise as the amount of MVImgNet training data increases.MVImgNet-small was a random subset matched to CO3D’s scale.Source: Table 4 · p. 5, Figure 5 · p. 5, §4.1 · p. 5

4.2. Multi-view Stereo

• MVImgNet-pretrained MVSNet improves depth-map accuracy over training from scratch when finetuned with limited DTU training samples.The data-efficient evaluation used 5%, 15%, and 25% of DTU’s 79 training samples and evaluated on 22 test samples.Source: §4.2 · pp. 5–6, Table 5 · p. 6
• With 5% of DTU training samples, MVImgNet pretraining is stronger than CO3D pretraining, while MVImgNet-small performs comparably to CO3D.The comparison evaluates MVS depth-map accuracy on the DTU evaluation set.Source: Table 6 · p. 6, §4.2 · p. 6
• The authors advocate benchmarking NeRF and MVS methods using MVImgNet pretraining.They motivate this as a way to study radiance-field reconstruction and data-efficient MVS, where collecting RGB-D training data is burdensome.Source: §4.2 · pp. 5–6

5.1. View-consistent Image Classification

• MVI-Mix combines ImageNet with five consecutive MVImgNet frames per video; MVI-Gap uses five non-consecutive frames, while MVI-Aug uses one frame with four augmentations.These training sets vary the sampled views or augmentation while mixing with ImageNet.Source: §5.1 · p. 6
• 48.76% accuracy and 0.225 Var on ImageNet-only versus 73.88% accuracy (25.12↑) and 0.122 Var (0.103↓) on MVI-Mix for DeiT-Tiny.The classification evaluation uses the MVImgNet test set; smaller Var and higher accuracy indicate better view consistency.Source: Table 7 · p. 6, §5.1 · p. 6
• At matched ImageNet mixing ratios, MVI-Mix brings more benefit than CO3DI-Mix for view-consistent recognition; CO3DI-Mix-to-MVImgNet performance drops exceed the reverse direction.CO3DI-Mix combines CO3D [73] with ImageNet.Source: Table 8 · p. 6, §5.1 · p. 6

5.2. View-consistent Contrastive Learning

• MoCo-v2 [17] finetuning uses two same-video frames as positives and other-video frames as negatives, improving view consistency and accuracy on the MVImgNet test set.The evaluation compares variance and mean of softmax confidence and accuracy.Source: §5.2 · pp. 6–7, Table 9 · p. 7
• The authors recommend using view consistency as an image-recognition evaluation criterion and using MVImgNet to pretrain models.Source: §5.2 · p. 7

5.3. View-consistent SOD

• On hard views (IoU ≤0.7), MVImgNet-finetuned U2Net improves IoU by 4.1%, using optical-flow-warped masks for cross-frame consistency while retaining DUTS-TR training data.The consistency loss compares a warped mask from one of two consecutive frames with the other frame’s mask.Source: §5.3 · p. 7

6.1. MVPNet Dataset

• MVPNet contains 87,200 real-world point clouds across 150 categories, with more real-world point clouds and broader everyday-object coverage than existing 3D object datasets.The dataset is derived from dense reconstruction on MVImgNet.Source: §6.1 · p. 7

6.2. 3D Point Cloud Classification

• Pretraining several models on MVPNet improves ScanObjectNN classification accuracy under most circumstances, including its standard PB T50 RS setting and a heavy-rotation variant.The study focuses on point cloud classification and argues for evaluating in real-world settings.Source: §6.2 · p. 7, Table 10 · p. 8

6.3. Self-supervised Point Cloud Pretraining

• PointMAE pretrained on MVPNet outperforms state-of-the-art methods on ScanObjectNN and surpasses CO3D pretraining for real-world point cloud classification.These comparisons support MVPNet’s benefit in the self-supervised learning regime.Source: §6.3 · p. 8, Table 10 · p. 8, Table 11 · p. 8

6.4. MVPNet Benchmark Challenge

• MVPNet’s benchmark challenge has 64,000 training and 16,000 testing samples, and cross-dataset tests show larger accuracy drops when training on ScanObjectNN and testing on MVPNet than in the reverse direction.The authors conclude MVPNet is more challenging than ScanObjectNN and propose it for future 3D point cloud understanding research.Source: §6.4 · p. 8, Table 13 · p. 8, Table 14 · p. 8

Conclusion

• MVImgNet’s multi-view signals bridge 2D and 3D vision, with promising results in radiance field reconstruction, multi-view stereo, and view-consistent image understanding.The authors describe these as pilot experiments and anticipate further exploration.Source: §7 · p. 8
• MVPNet, derived from MVImgNet, benefits real-world 3D object classification.Source: §7 · p. 8
• MVImgNet focuses on human-centric classes, has less category richness than ImageNet, and may perform worse on nature-centric recognition such as turtles and bears.Source: §7 · p. 8
• MVImgNet does not account for very complex backgrounds, limiting straightforward use in scene-level understanding; future domain adaptation or knowledge distillation may address this.The passage also points to commonsense knowledge and universal representations gained from the data as support for such methods.Source: §7 · p. 8

Research leadership

• Xiaoguang Han proposed and led the research; Mutian Xu led most of it, organized and wrote the paper, and advised exploration of view-consistent image and 3D understanding.Guanying Chen and Shuguang Cui advised the project.Source: Contributions · p. 9

Dataset construction

• Xianggang Yu contributed to dataset acquisition and processing; Yidan Zhang worked on data processing, while Zhangyang Xiong handled data collection and processing.Yushuang Wu worked on data collection and annotation; Tianyou Liang worked on data cleaning.Source: Contributions · p. 9
• Haolin Liu handled point-cloud labeling and cleaning, Chongjie Ye conducted data labeling, and Yushuang Wu contributed to data collection and annotation.Yidan Zhang collected and provided illustrations of dataset statistics and details; Yushuang Wu collected and wrote related work.Source: Contributions · p. 9

Experiments

• Experiments covered radiance-field reconstruction, multi-view stereo, 3D point-cloud classification, and view-consistent image classification, saliency-object detection, and contrastive learning.Xianggang Yu conducted radiance-field reconstruction; Chongjie Ye implemented multi-view stereo and 3D point-cloud classification experiments; Haolin Liu implemented point-cloud classification. Yidan Zhang implemented view-consistent image classification and saliency-object detection; Chenming Zhu conducted view-consistent contrastive learning.Source: Contributions · p. 9
• Xianggang Yu designed and advised the view-consistent saliency-object detection experiment, while Zizheng Yan suggested experiments on view-consistent image understanding.Tianyou Liang assisted experiments on view-consistent image understanding.Source: Contributions · p. 9

F. Implementation Details

• The supplement organizes implementation details into 3D reconstruction, view-consistent image understanding, and 3D understanding.These topics appear as subsections F.1–F.3.Source: F. Implementation Details · p. 14

A. Per-category Data Distribution

• Average category counts are 921 samples per class for MVImgNet and 581 per class for MVPNet.The supplement also presents category taxonomies and per-category distributions for both datasets.Source: A. Per-category Data Distribution · p. 14

B. More Visualizations of Data Samples

• MVImgNet examples show multiple views and broad category coverage, while MVPNet point clouds exhibit varied texture, noise, and pose.The supplement illustrates these examples in Figs. V and VI, respectively.Source: B. More Visualizations of Data Samples · p. 14

C. More Visualizations of Qualitative Results

• MVImgNet pretraining consistently outperforms training from scratch in the additional generalizable NeRF reconstruction visualizations.The supplement presents these comparisons in Fig. VII.Source: C. More Visualizations of Qualitative… · p. 14
• Finetuning U2Net on the MVImgNet test set improves view-consistent salient object detection over the original U2Net.Additional qualitative results are shown in Fig. VIII.Source: C. More Visualizations of Qualitative… · p. 14

Multi-view stereo

• Using more MVImgNet videos for pretraining yields higher MVS depth-map accuracy on the DTU evaluation set.Table I compares pretraining with 10k, 50k, and 100k videos.Source: D. More Experiments of Data Scalability · p. 14, Table I · p. 15

View-consistent image classification

• Adding more MVImgNet training data to MVI-Mix improves view consistency for ResNet-50 [43] image classification on the MVImgNet test set.MVI-Mix combines ImageNet data with MVImgNet data; Table II evaluates training sets using 20k, 40k, 80k, 100k, and 120k videos.Source: D. More Experiments of Data Scalability · p. 14, Table II · p. 15

Real-world point cloud classification

• Larger MVPNet pretraining data ratios improve ScanObjectNN [87] classification after fine-tuning for both supervised PointNet++ [71] and CurveNet [96] and self-supervised PointMAE [66].The task is real-world point-cloud classification.Source: D. More Experiments of Data Scalability · pp. 14–15

E. More Discussions about Our Datasets

• ∼219k videos were retained from ∼260k raw videos after filtering bad camera estimations; MVPNet uses 90k videos from 150 common categories, yielding 87k cleaned point clouds.Source: E. More Discussions about Our Datasets · p. 15
• Objects retain their real-world arrangements rather than being staged for ideal 360° captures, supporting easier collection, partial-view point clouds, and diverse scene backgrounds.The authors say this better matches real-world applications and offers potential for in-the-wild scene-level visual tasks.Source: E. More Discussions about Our Datasets · p. 15
• ScanObjectNN experiments compare MVPNet pretraining with 25%, 50%, and 100% of its data under Add Random Rotation and PB T50 RS, reporting overall and average accuracy.Source: Table III · p. 15, Table IV · p. 15

F.1. 3D Reconstruction

• IBRNet was pretrained on full MVImgNet then finetuned for 10k iterations on its original training datasets, with an otherwise identical author-released scratch model as comparator.Training uses 8–12 views per object and 10 inference views; pretraining took about 3 days on 8 RTX3090 GPUs.Source: §F.1 · p. 15
• For MVS, 200K frames from 100K MVImgNet videos were resized to 640 × 360 or 360 × 640, and JDACS used MVSNet pseudo-depth with multi-view consistency for self-supervision.Source: §F.1 · p. 15

View-consistent image classification.

• The hybrid image-classification set combines 500k MVImgNet frames from 100k videos with 200k ImageNet images across 1,100 non-overlapping categories; MoCo v2 uses same-video views as positive pairs.Five frames are randomly sampled per MVImgNet video for finetuning.Source: View-consistent image classification · pp. 15–16
• SOD training adds an optical-flow consistency loss so adjacent frames’ masks align after warping; τ is set to 0.15.Source: View-consistent image classification · p. 16

F.3. 3D Understanding

• 3D-understanding experiments follow the selected backbones’ original settings; supplementary figures show an IBRNet comparison and U2Net salient-object results, whose caption reports improvement after MVImgNet finetuning.Source: §F.3 · p. 16, Figure VII · p. 21, Figure VIII · p. 22