PaperlayerView exact revision on arXiv

Source-linked AI summary

DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision

Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, Xuanmao Li, Xingpeng Sun, Rohan Ashok, Aniruddha Mukherjee, Hao Kang, Xiangrui Kong, Gang Hua, Tianyi Zhang, Bedrich Benes, Aniket Bera

arXiv:2312.16256v2cs.CVcs.AI

TL;DR

Existing scene-level datasets are too synthetic or limited in real-world coverage for comprehensive NVS evaluation and learning universal 3D scene priors. The paper introduces DL3DV-10K, a large-scale annotated real-world scene dataset, and evaluates it through DL3DV-140 and generalizable NeRF experiments. The benchmark and pilot study report broad NVS analysis and enhanced NeRF generalizability after pretraining on DL3DV-10K.

  • Problem

    Existing scene-level datasets are restricted to synthetic environments or narrow real-world selections, limiting comprehensive NVS evaluation and universal-prior learning.

  • Method

    The paper constructs DL3DV-10K from high-resolution real-world videos, annotates scene complexity, samples DL3DV-140 for NVS benchmarking, and pretrains generalizable NeRF models.

  • Results

    Pretraining on DL3DV-10K enhances the generalizability of NeRF, while DL3DV-140 enables statistical evaluation of state-of-the-art NVS methods.

  • Takeaways & Limitations

    DL3DV-10K’s scale and diversity support learning a universal scene prior and provide a path toward foundational 3D representation models.

  • Takeaways & Limitations

    Some mobile-phone videos contain moving objects, introducing additional challenges for static view synthesis and motivating future dynamic NVS models.

Abstract

from arXiv · show

We have witnessed significant progress in deep learning-based 3D vision, ranging from neural radiance field (NeRF) based 3D representation learning to applications in novel view synthesis (NVS). However, existing scene-level datasets for deep learning-based 3D vision, limited to either synthetic environments or a narrow selection of real-world scenes, are quite insufficient. This insufficiency not only hinders a comprehensive benchmark of existing methods but also caps what could be explored in deep learning-based 3D analysis. To address this critical gap, we present DL3DV-10K, a large-scale scene dataset, featuring 51.2 million frames from 10,510 videos captured from 65 types of point-of-interest (POI) locations, covering both bounded and unbounded scenes, with different levels of reflection, transparency, and lighting. We conducted a comprehensive benchmark of recent NVS methods on DL3DV-10K, which revealed valuable insights for future research in NVS. In addition, we have obtained encouraging results in a pilot study to learn generalizable NeRF from DL3DV-10K, which manifests the necessity of a large-scale scene-level dataset to forge a path toward a foundation model for learning 3D representation. Our DL3DV-10K dataset, benchmark results, and models will be publicly accessible at https://dl3dv-10k.github.io/DL3DV-10K/.

1. Introduction

Existing scene-level datasets are too synthetic or narrow to comprehensively evaluate NVS in complex real-world settings or support universal 3D scene priors. DL3DV-10K addresses this gap with large-scale, diverse real-world scenes, a 140-scene NVS benchmark, and a pilot study of generalizable NeRF.

  • Motivation: Existing NVS datasets are restricted to synthetic environments or a narrow selection of real-world scenes, limiting comprehensive evaluation in complex scenarios.The cited gap includes non-Lambertian surfaces and other real-world conditions.
  • Motivation: Scene-level data scarcity also limits learning-based 3D models’ ability to learn universal priors from substantial real scenes.Per-scene NeRF optimization requires numerous calibrated views and substantial computation, while cross-scene models depend on broad scene coverage.
  • Dataset: DL3DV-10K contains 51.3 million 4K frames from 10,510 videos spanning 65 POI locations and annotated complexity indices.Annotations include environment, reflection, transparency, lighting, and texture frequency.
  • Benchmark: DL3DV-140 samples 140 scenes from DL3DV-10K to provide a diverse benchmark for evaluating state-of-the-art NVS methods.The evaluated methods include Nerfacto, Instant-NGP, Mip-NeRF 360, Zip-NeRF, and 3DGS.
  • Generalizable NeRF: Pretraining on DL3DV-10K enhances generalizable NeRF performance, supporting the importance of diversity and scale for learning a universal scene prior.The paper presents this as a demonstration of the dataset’s potential for deep 3D representation learning.

2. Related Work

Prior NVS benchmarks span synthetic object and scene datasets and real-world collections, but commonly lack either real-world effects, scene variety, scale, or consistent coverage. DL3DV-140 is positioned as a diverse real-world benchmark for evaluating challenging NVS conditions and comparing methods.

  • Novel View Synthesis: NeRF and 3DGS methods produce scene-specific radiance fields but require dense views and extensive computation, motivating learning-based universal-prior approaches.Learning-based methods train across numerous scenes to support universal priors and sparse-view synthesis.
  • NVS Benchmarks: Synthetic NVS benchmarks provide varied CAD geometry and textures but lack real-world noise and non-Lambertian effects.Examples include NeRF-synthetic, ShapeNet, and Objaverse.
  • NVS Benchmarks: Real-world benchmarks offer limited variety or collection constraints, including narrow fields of view, motion blur, inconsistent standards, and insufficient challenging lighting and material effects.LLFF contains 24 forward-facing cellphone-captured scenes, while other real-world datasets also have restricted coverage.
  • Multi-view Scene Datasets: DL3DV-140 covers view-dependent effects, reflective and transparent materials, outdoor unbounded environments, and high-frequency textures for comprehensive NVS assessment.The benchmark also provides extensive comparative analyses of NVS techniques.
  • Multi-view Scene Datasets: Existing scene-level datasets remain limited in scale or diversity across indoor and outdoor environments, constraining robust and universal 3D model training.RealEstate10k offers comparable scale but low resolution and limited diversity, while several alternatives focus mainly on indoor scenes.

3. Data Acquisition and Processing

DL3DV-10K combines a guided mobile-and-drone capture pipeline with broad POI diversity, quality controls, complexity annotations, and frequency estimation. DL3DV-140 samples static scenes and simplifies reflection and transparency classes for benchmark interpretation.

  • Data Acquisition: The acquisition pipeline integrates video capture, preprocessing, and analysis using consumer mobiles and drones to collect accessible real-world scenes efficiently.Collector guidelines target reduced motion blur, excluded exposure lights, and fewer moving objects.
  • Diversity: DL3DV-10K spans 16 primary and 65 secondary POI categories across indoor and outdoor settings and different cities.Categories include educational institutions, tourist attractions, restaurants, medical facilities, and transportation hubs.
  • Quality: The recording guidelines require broad scene coverage, 4K resolution, 30 or 60 fps, minimum video durations, limited moving-object duration, and avoidance of blur or overexposure.Videos should include horizontal views of at least 180° or 360° and objects should be stereoscopic.
  • Data Processing: Texture frequency is estimated by averaging normalized high-frequency wavelet energy across 100 sampled grayscale frames per video.The method applies a two-dimensional bi-orthogonal wavelet transform and normalizes the Frobenius norm by pixel count.
  • Data Processing: Scene annotations encode POI, device, lighting, indoor or outdoor setting, reflectivity, transparency, and high-frequency textures.Reflectivity and transparency are estimated from relevant pixel ratios and their duration in the video.
  • Benchmark Construction: DL3DV-140 samples 140 static scenes and merges reflection and transparency levels into more-versus-less categories for easier interpretation.The same simplification is applied to both properties.

4. Experiment

The experiments evaluate state-of-the-art NVS methods on DL3DV-140 and study DL3DV-10K for generalizable NeRF. Results show method-dependent strengths across metrics, scene complexities, visual artifacts, and pretraining settings.

  • NVS benchmark: Zip-NeRF, Mip-NeRF 360, and 3DGS consistently outperform Instant-NGP and Nerfacto across all evaluation metrics.
  • NVS benchmark: Zip-NeRF achieves the highest average PSNR and SSIM, but uses more GPU memory with its default batch size.Mip-NeRF 360 reaches PSNR 30.98 and SSIM 0.91, while 3DGS reaches SSIM 0.92.
  • Scene complexity: Outdoor unbounded scenes are most challenging, low-frequency scenes easiest, and more-transparent scenes harder than less-transparent scenes.Zip-NeRF leads in most settings, whereas Mip-NeRF 360 performs best and most robustly in low-frequency scenes.
  • Visual results: NeRF variants show grainy microstructure, whereas 3DGS produces elongated or splotchy Gaussian artifacts.NeRF variants also exhibit blurry distant backgrounds and aliasing on high-frequency objects; 3DGS handles aliasing better but artifacts remain in far-distance backgrounds.
  • View-dependent effects: 3DGS renders sharp reflections and transparent-object edges effectively but oversimplifies softer reflective effects.Zip-NeRF and Mip-NeRF 360 capture reflections more generally, while Nerfacto and Instant-NGP often produce floating artifacts under complex lighting.
  • Generalizable NeRF: Pretraining IBRNet on DL3DV-10K improves performance across all evaluation benchmarks, with larger DL3DV-10K inputs yielding consistently better results.ScanNet++ pretraining does not improve IBRNet on the existing benchmarks.

5. Conclusion

The conclusion presents DL3DV-10K as a diverse real-world multi-view dataset and DL3DV-140 as a challenging NVS benchmark. It also reports potential for improving generalizable NeRF and notes moving objects as a limitation of mobile-phone video collection.

  • Dataset and benchmark: DL3DV-10K contains high-resolution real-world multi-view videos, while DL3DV-140 samples 140 scenes for challenging NVS evaluation.
  • Dataset and benchmark: DL3DV-140 supports statistical analysis of NVS strengths and weaknesses across diverse scene complexities.The benchmark includes challenging view-dependent effects, reflective and transparent materials, outdoor environments, and high-frequency textures.
  • Generalizable NeRF: DL3DV-10K pretraining enhances NeRF generalizability and indicates potential for a foundational model for learning 3D representations.
  • Limitations: Some scenes contain moving objects because videos were collected with mobile phones, introducing additional challenges for static view synthesis.The authors note that these challenges may inform robustness studies and future dynamic NVS models.

6. Overview

The supplementary materials include a PDF, video demo, and HTML page with additional qualitative results. The paper also directs readers to sections covering data acquisition, distribution, benchmark details, and generalizable NeRF experiments.

  • Supplementary materials include a PDF, video demo, and HTML page with more qualitative results.
  • Section 7 covers data acquisition standards and DL3DV-10K distribution.
  • Section 8 provides benchmark experiment details, training details, additional qualitative results, and generalizable NeRF experiment details.

7. Data

DL3DV-10K captures real-world scenes through varied camera trajectories and annotates scene complexity, including reflection, transparency, environmental setting, lighting, and frequency. Its statistics characterize scene distributions across POI categories and complexity attributes.

  • Scene Capture: Scenes are captured through horizontal 180°–360° views from different heights, using panoramic or semicircular trajectories according to scene accessibility and occlusion.Panoramic coverage typically spans areas reachable on foot within 30 to 45 seconds; obstructed rear views motivate semicircular capture exceeding 180°.
  • Complexity Annotation: Reflection and transparency indices are manually assigned from the proportions of reflective or transparent pixels and their observed duration in videos.Transparency uses the same labeling rule as reflection.
  • Scene Statistics: Scene statistics are organized by secondary POI categories and complexity indices covering environmental setting, light condition, reflection, and transparency.The dataset links detailed secondary classes to their corresponding primary POI categories.
  • Scene Statistics: Reflective scenes are more common in some locations, while hiking trails are predominantly nonreflective and most scenes are non-transparent.The observed distributions are reported as consistent with common expectations for real-world scenarios.
  • Frequency and Duration: Frequency classes are delineated using the median value of the frequency metric, alongside kernel-density estimates of frequency and video duration.The frequency and duration distributions are presented in Figure 11.

8. Experiment

Experiments evaluate state-of-the-art NVS methods across POI categories and scene-complexity dimensions, with additional generalizable-NeRF experiments using different pretraining datasets. The setup includes specified implementations, precision choices, ray-batch settings, and substantial GPU usage.

  • Implementation: The benchmark uses implementations from nerfstudio, method authors, and a public Zip-NeRF implementation matching the reported paper performance.Nerfacto and Instant-NGP use nerfstudio; Mip-NeRF360 and 3DGS use author code.
  • Experimental Setup: 13,230 GPU hrs were required for the NVS experiments, with a default ray batch of 4096 and method-specific precision settings.Zip-NeRF was also evaluated with a ray batch of 65536 because it is sensitive to this parameter; near and far bounds were shared across NeRF methods.
  • POI Analysis: DL3DV-140 reports better NVS performance for Education institutions, Restaurants and Cafes, and Shopping Centers than for Tourist Attractions and Transportation Hubs.The comparison is summarized as average performance across six primary POI categories.
  • Environmental Setting: Indoor scenes are less challenging than outdoor scenes, with outdoor performance varying across 3DGS, Nerfacto, Instant-NGP, Zip-NeRF, and Mip-NeRF 360.The figure specifically identifies greater outdoor difficulty for 3DGS, Nerfacto, and Instant-NGP than for Zip-NeRF and Mip-NeRF 360.
  • Frequency: High-frequency scenes are more challenging than low-frequency scenes, while 3DGS consistently captures high-frequency details and shape edges.Among NeRF variants, Nerfacto and Instant-NGP are reported as more challenged by high-frequency details than Zip-NeRF and Mip-NeRF 360.
  • Transparency: Scenes with more transparent materials are more challenging, while 3DGS, Zip-NeRF, and Mip-NeRF 360 capture subtle transparent-object edges effectively.Nerfacto and Instant-NGP are reported to perform less consistently on these scenes.
  • Reflection: More reflective scenes are more challenging, with Zip-NeRF and Mip-NeRF 360 adept at subtle reflections and highlights.3DGS tends to smooth less intense reflections, while Nerfacto and Instant-NGP struggle with reflective scenes.
Loading 2312.16256v2…