PaperlayerView exact revision on arXiv

Source-linked AI summary

A Novel Recurrent Encoder-Decoder Structure for Large-Scale Multi-view Stereo Reconstruction from An Open Aerial Dataset

Jin Liu, Shunping Ji

arXiv:2003.00637v3cs.CV

TL;DR

Large-scale aerial MVS lacked suitable benchmarks, while existing learned approaches were aimed mainly at close-range scenes. The paper introduces the synthetic WHU dataset and RED-Net, a recurrent encoder-decoder network for depth inference. RED-Net outperformed comparable WHU methods and transferred to a different aerial dataset, with reported efficiency gains over commercial software.

  • Problem

    Deep-learning MVS was largely developed for close-range objects, and the lack of aerial benchmarks limited method development and comparison for large-scale Earth-surface reconstruction.

    Source: §1 · p. 1

  • Method

    The paper creates the synthetic WHU aerial dataset and introduces RED-Net, which sequentially regularizes multi-view cost maps with a recurrent encoder-decoder.

    Source: §3 · p. 3, §4 · p. 4

  • Results

    RED-Net outperformed all other methods on the WHU indicators, improving MAE by at least 50% over second-best R-MVSNet, and transferred directly to another aerial dataset with slightly better accuracy and 16-times improved efficiency over commercial software.

    Source: §1 · p. 2, §5.1 · p. 6

  • Takeaways & Limitations

    The authors conclude that the WHU dataset can supplement close-range datasets and that RED-Net supports efficient, full-resolution large-scale reconstruction and direct application to a different aerial dataset.

    Source: §7 · p. 8

Abstract

from arXiv · show

A great deal of research has demonstrated recently that multi-view stereo (MVS) matching can be solved with deep learning methods. However, these efforts were focused on close-range objects and only a very few of the deep learning-based methods were specifically designed for large-scale 3D urban reconstruction due to the lack of multi-view aerial image benchmarks. In this paper, we present a synthetic aerial dataset, called the WHU dataset, we created for MVS tasks, which, to our knowledge, is the first large-scale multi-view aerial dataset. It was generated from a highly accurate 3D digital surface model produced from thousands of real aerial images with precise camera parameters. We also introduce in this paper a novel network, called RED-Net, for wide-range depth inference, which we developed from a recurrent encoder-decoder structure to regularize cost maps across depths and a 2D fully convolutional network as framework. RED-Net's low memory requirements and high performance make it suitable for large-scale and highly accurate 3D Earth surface reconstruction. Our experiments confirmed that not only did our method exceed the current state-of-the-art MVS methods by more than 50% mean absolute error (MAE) with less memory and computational cost, but its efficiency as well. It outperformed one of the best commercial software programs based on conventional methods, improving their efficiency 16 times over. Moreover, we proved that our RED-Net model pre-trained on the synthetic WHU dataset can be efficiently transferred to very different multi-view aerial image datasets without any fine-tuning. Dataset are available at http://gpcv.whu.edu.cn/data.

1. Introduction

Deep-learning MVS had not been well suited to large-scale aerial reconstruction, in part because aerial benchmarks were lacking and existing networks required substantial GPU memory. The paper presents the WHU aerial dataset and RED-Net for large-scale MVS, reporting stronger accuracy and efficiency and direct transfer to a different aerial dataset.

1. Introduction

  • The lack of aerial benchmarks and the domain difference from laboratory imagery limited training and comparison of deep-learning MVS methods for Earth-surface reconstruction.High GPU-memory demand and fourfold downsampling in R-MVSNet also constrained large-scale, high-resolution reconstruction.Source: §1 · p. 1
  • The WHU dataset provides thousands of simulated aerial images across 6.7 × 2.2 km2, with depth ground truth and camera parameters, plus cropped blocks for single-GPU training.Its camera parameters are described as unbiased and its ground truths as complete, including occluded regions.Source: §1 · p. 1
  • RED-Net outperformed comparable methods on the WHU aerial dataset and transferred directly to a different aerial dataset, with slightly better accuracy and 16-times improved efficiency over commercial software.The introduction also reports higher accuracy and efficiency than R-MVSNet while using less GPU memory and maintaining unlimited depth resolution.Source: §1 · p. 2

2. Related Work

Prior MVS datasets largely covered close-range, laboratory, or other scenes, while accessible aerial datasets were few and small. Existing learned MVS methods traded memory, resolution, and contextual detail; RED-Net follows sequential depth processing with a recurrent encoder-decoder.

2.1. Datasets

  • Earlier stereo datasets were often too small for deep learning, while a large synthetic stereo dataset was difficult to transfer to real scenes because of heterogeneous data sources.The multi-view survey distinguishes laboratory, close-range, and complex outdoor benchmarks from aerial reconstruction needs.Source: §2.1 · p. 2
  • The WHU work adds a large synthetic aerial dataset with accurate camera parameters and complete ground truths, whereas the cited M¨unchen and Vaihingen aerial datasets contain only dozens of images and are not publicly accessible.The related-work survey also describes close-range and outdoor benchmarks, but not an accessible large-scale aerial benchmark.Source: §2.1 · p. 2

2.2. Networks

  • RED-Net replaces R-MVSNet’s stacked GRUs with a recurrent encoder-decoder to regularize 2D cost maps sequentially, targeting finer context at full resolution with lower memory demand.R-MVSNet sequentially regularizes 2D cost maps but uses small 3 × 3 GRU receptive fields and outputs depth maps down-sampled four times; earlier volumetric methods also had high memory costs.Source: §2.2 · pp. 2–3

3. WHU Dataset

The WHU dataset supplies synthetic aerial imagery and cropped subsets for high-resolution MVS training. It is built from a manually refined surface model and includes complete depth data, camera parameters, and multi-view and stereo crops.

3. WHU Dataset

  • The WHU dataset was created for large-scale, high-resolution Earth-surface reconstruction and includes complete aerial imagery alongside cropped subsets for deep-learning use.The imagery was simulated from a 3D surface model produced by software and refined manually.Source: §3 · p. 3

3.1. Data Source

  • The source DSM was reconstructed from real multi-view aerial images captured by an oblique five-camera UAV rig and manually edited to improve resemblance to the scene.It covers about 6.7 × 2.2 km2 over Meitan County at about 0.1 m ground resolution.Source: §3.1 · p. 3

3.2. Synthetic Aerial Dataset

  • The synthetic images were projected from a gridded 3D point set, with depth maps retrieved from the model and complete on building facades.The collection contains 1,776 images and depth maps at 5376 × 5376 pixels, plus 1,760 disparity maps; adjacent images have zero rotation and can serve as epipolar pairs.Source: §3.2 · p. 3

3.3. Sub-Dataset for Deep Learning

  • Training uses 261 images from Areas 1/4/5/6, while testing uses 93 images from Areas 2/3; five-view units use image 1 as reference and four surrounding images as searches.The multi-view and stereo subsets are cropped to 768 × 384 pixels for single-GPU use; stereo pairs are adjacent epipolar images, and a three-view set uses images 0, 1, and 2.Source: §3.3 · pp. 3–4, Figure 2 · p. 4, Figure 3 · p. 4

4. RED-Net

RED-Net combines shared multi-view feature extraction with recurrent encoder-decoder regularization of cost maps across spatial and depth dimensions. Its multi-scale design produces full-resolution depth maps while reducing memory use relative to a stack of GRUs.

4. RED-Net

  • Compared with the stack of GRUs, RED-Net's 2D recurrent encoder-decoder reduces memory use, improves computational efficiency, captures finer structures, and outputs input-resolution depth maps.Figure 4 depicts the network structure; W, H, and D denote image width, height, and depth sample number.Source: §4 · p. 4, Figure 4 · p. 5
  • Shared-weight 2D convolution branches extract features from each view, which plane sweeping warps into D cost maps for recurrent regularization.Each branch produces 16-channel feature maps at half the input width and height.Source: §4 · pp. 4–5
  • RED-Net sequentially regularizes cost maps with a multi-scale recurrent encoder-decoder across spatial and depth dimensions, aggregating contextual information for depth inference.At each spatial scale, convolutional GRUs regularize features; recurrent states carry context between successive depth maps.Source: §4 · p. 5

5. Experiments

On the WHU test set, RED-Net outperformed the compared methods across all reported indicators, while using less memory for full-resolution output. Tests on the München aerial set also reported strong performance for RED-Net pretrained on WHU data.

5.1. Experimental Settings and Results

  • CNN methods were trained on 3,600 WHU sub-units and evaluated on 1,360; WHU-3 and WHU-5 used three and five views, respectively, with D=200.The WHU test set contained 93 images; evaluations used depth maps without post-processing.Source: §5.1 · p. 6
  • At least 50% MAE improvement over second-best R-MVSNet; RED-Net also outperformed all methods on every indicator and exceeded them by at least 2% on 3-interval-error and 0.6 m threshold indicators.The WHU quantitative results are reported in Table 1.Source: Table 1 · p. 6, §5.1 · p. 6

5.2. GPU Memory and Runtime

  • RED-Net used nearly half R-MVSNet's memory while producing full-resolution depth maps 16 times larger; (R-)MVSNet ran slightly faster but output maps were downsampled by four.Table 2 compares memory requirement and runtime for the methods.Source: §5.2 · p. 6, Table 2 · p. 7

5.3. Generalization

  • On München, WHU-3-trained RED-Net led every indicator, exceeding other methods by at least 6% in 3-interval-error; WHU-5 performed almost as well, while DTU-trained CNNs lagged conventional methods.The evaluation used CNN models pretrained on DTU or WHU without fine-tuning.Source: Table 3 · p. 7, §5.3 · p. 7

6. Discussion

RED-Net retains advantages when output resolution is matched, exceeds R-MVSNet on the close-range DTU dataset, and supports fast inference over large areas.

6.1. Advantage of the Recurrent Encoder-Decoder

  • At matched cost-map and output sizes, RED-Net led on all measures across three aerial datasets, supporting gains from its encoder-decoder beyond output resolution.The authors attribute this to better learned spatial and contextual representations than stacked GRUs.Source: §6.1 · p. 7

6.2. Evaluation on DTU

  • 18%: RED-Net's overall score exceeded R-MVSNet's on DTU with identical photometric and geometric filtering.It also outperformed the results in using all four post-processing methods.Source: §6.2 · p. 8

6.3. Large-scale Reconstruction

  • 9.3 minutes: RED-Net inferred a 1.8 × 0.85 km2 scene, versus 150 minutes for SURE and 608 minutes for COLMAP.It also processed 7040 × 7040 three-view inputs on a 24GB GPU in 58 seconds with 128 depth samples.Source: §6.3 · p. 8, Figure 7 · p. 8

7. Conclusion

The conclusion presents the WHU synthetic aerial dataset for large-scale MVS and RED-Net for efficient, full-resolution reconstruction, with reported performance exceeding deep-learning methods and commercial software.

7. Conclusion

  • The WHU dataset was created for large-scale, high-resolution MVS reconstruction and, to the authors' knowledge, was the largest and only available multiview aerial dataset.The authors describe it as a supplement to close-range datasets for studying Earth-surface and city reconstruction.Source: §7 · p. 8
  • RED-Net achieved efficient, full-resolution reconstruction with relatively low memory requirements and exceeded both deep-learning methods and commercial software.The conclusion also reports direct applicability to a somewhat different aerial dataset after pretraining on WHU.Source: §7 · p. 8

Related papers

Loading 2003.00637v3…