PaperlayerView exact revision on arXiv

Source-linked AI summary

IM2HEIGHT: Height Estimation from Single Monocular Imagery via Fully Residual Convolutional-Deconvolutional Network

Lichao Mou, Xiao Xiang Zhu

arXiv:1802.10249v1cs.CV

TL;DR

Monocular remote-sensing height estimation is ambiguous, yet such imagery is common and the task has received little attention in the field. The paper proposes an end-to-end residual convolutional-deconvolutional network with a skip connection, reports improved height-map quality, and demonstrates building instance segmentation using predicted heights.

  • Problem

    A single remote-sensing image can correspond to infinitely many real-world scenes, making height estimation inherently ambiguous.

    Source: §I · p. 1

  • Method

    An end-to-end residual convolutional-deconvolutional network maps monocular remote-sensing images to height maps, with a skip connection carrying low-level visual information across the network.

    Source: B. Contributions · p. 3, C. Res. Conv-Deconv Net with Skip · p. 5

  • Results

    The skip-connected model improves object-edge and boundary quality over Eigen-Net and network variants; the paper reports accuracy increases of 0.0023 of MSE, 0.01 of MAE, and 0.1306 of SSIM.

    Source: C. Evaluation on Height Maps · p. 8

  • Takeaways & Limitations

    The paper demonstrates building instance segmentation using height maps predicted from a single monocular image.

    Source: §IV · p. 10

    Improving the accuracy of high-rise buildings remains a focus for future work.

    Source: §IV · p. 10

Abstract

from arXiv · show

In this paper we tackle a very novel problem, namely height estimation from a single monocular remote sensing image, which is inherently ambiguous, and a technically ill-posed problem, with a large source of uncertainty coming from the overall scale. We propose a fully convolutional-deconvolutional network architecture being trained end-to-end, encompassing residual learning, to model the ambiguous mapping between monocular remote sensing images and height maps. Specifically, it is composed of two parts, i.e., convolutional sub-network and deconvolutional sub-network. The former corresponds to feature extractor that transforms the input remote sensing image to high-level multidimensional feature representation, whereas the latter plays the role of a height generator that produces height map from the feature extracted from the convolutional sub-network. Moreover, to preserve fine edge details of estimated height maps, we introduce a skip connection to the network, which is able to shuttle low-level visual information, e.g., object boundaries and edges, directly across the network. To demonstrate the usefulness of single-view height prediction, we show a practical example of instance segmentation of buildings using estimated height map. This paper, for the first time in the remote sensing community, attempts to estimate height from monocular vision. The proposed network is validated using a large-scale high resolution aerial image data set covered an area of Berlin. Both visual and quantitative analysis of the experimental results demonstrate the effectiveness of our approach.

I. INTRODUCTION

Height helps describe scene geometry and supports remote-sensing tasks, but monocular height estimation remains challenging and relatively little addressed. Remote-sensing imagery also differs from conventional depth-estimation imagery in ways that constrain available cues and complicate prediction.

I. INTRODUCTION

  • Height information helps describe scene geometry and benefits remote-sensing tasks such as semantic labeling and change detection.Source: §I · p. 1
  • Although DSM acquisition has become affordable, height data may be unavailable for large collections of historical imagery, while monocular observations are common in practice.Source: §I · p. 1
  • A single remote-sensing image can fit infinitely many real-world scenarios, making its mapping from color or intensity to height inherently ambiguous.The paper defines a correct height map as one that is physically plausible in the real world.Source: §I · p. 1

A. Related Work

  • Computer-vision depth work includes CNNs for dense per-pixel prediction and image-transfer methods that require runtime access to datasets and expensive alignment.The transfer-method limitation is described in the context of monocular depth estimation.Source: A. Related Work · pp. 1–2
  • Remote-sensing images are often orthographic, limiting context, and their coverage, resolution, and tiny objects pose further height-prediction challenges.Source: A. Related Work · p. 2
  • Single-image height prediction has rarely been studied in remote sensing; an earlier CNN approach required pixel-wise segmentation masks, which are costly to obtain at scale.DSM data is described as widely accessible at reasonable cost.Source: A. Related Work · p. 2

B. Contributions

The paper targets monocular remote-sensing height estimation and trains an end-to-end network using DSM ground truth alone. Its residual convolutional-deconvolutional design includes a skip connection, and predicted height maps are also applied to building instance segmentation.

B. Contributions

  • The paper addresses single-image height estimation using DSM ground truth without additional pixel-wise segmentation masks.Source: B. Contributions · p. 2
  • An end-to-end residual network combines convolutional feature extraction, deconvolutional height generation, and a skip connection.The convolutional part forms abstract feature representations; the deconvolutional part generates the height map.Source: B. Contributions · p. 3, Figure 2 · p. 3
  • The paper demonstrates building instance segmentation using predicted height maps as an application of single-view height prediction.Source: B. Contributions · p. 3

II. METHODOLOGY

The methodology frames height prediction as learning a point-estimate mapping from remote-sensing images to DSMs. Under the stated loss, the minimizer is the conditional expected height map, and training fits the network weights to sampled loss.

II. METHODOLOGY

  • The method learns a point-estimate mapping from remote-sensing images to DSMs by minimizing an objective over the mapping.The paper frames the goal as estimating p(y|x) and uses a mapping f from image x to height y.Source: §II · p. 3
  • The loss minimizer is the conditional expectation, described as the expected height map.The passages identify the conditional expectation as the minimizer and equate it with the expected height map.Source: §II · p. 3
  • Training learns the weights Θ from paired remote-sensing images and DSMs by minimizing a Monte-Carlo estimate of the loss.The described end-to-end training approximates DSMs from their corresponding images.Source: §II · p. 3

A. Architecture: Fully Convolutional-Deconvolutional Net

Dense height maps require refining coarse feature representations produced by conventional CNN pooling. The proposed architecture uses a deconvolutional sub-network to upsample the whole coarse feature maps, transferring more high-level information to fine predictions than approaches that process only coarse predictions.

A. Architecture: Fully Convolutional-Deconvolutional Net

  • Conventional CNN pooling shrinks feature maps, so dense per-pixel height prediction requires refining the resulting coarse representations.The cited FCN work uses upsampling to refine coarse maps; Eigen et al.'s predicted depth maps are one-quarter input resolution and lose border areas, then receive an additional refinement network.Source: A. Architecture: Fully… · pp. 3–4
  • The figure illustrates plain convolutional and residual blocks as distinct block designs.The section's architecture description identifies the proposed network as convolutional-deconvolutional with a skip connection.Source: Figure 3 · p. 4, A. Architecture: Fully… · p. 4
  • The deconvolutional sub-network upsamples whole coarse feature maps rather than only coarse predictions, allowing more high-level information into fine predictions.The architecture pairs a convolutional feature extractor with a deconvolutional height generator.Source: A. Architecture: Fully… · p. 4

B. Plain Convolutional Net vs. Residual Net

The section develops the convolutional-deconvolutional model from VGG-style plain blocks to residual blocks, addressing optimization and height-map plausibility problems. Its decoder uses unpooling to restore spatial resolution.

B. Plain Convolutional Net vs. Residual Net

  • The convolutional sub-network follows VGG-style blocks, stacking 3×3 convolutions and applying 2×2 max-pooling with stride 2.The convolutional layers preserve feature-map resolution using stride 1 and padding 1.Source: B. Plain Convolutional Net vs.… · p. 4
  • The mirrored decoder uses unpooling followed by convolution, restoring spatial span with the encoder's max-pooling indices.The paper selects index-based unpooling, which it says theoretically enables more accurate location representation and boundary delineation.Source: B. Plain Convolutional Net vs.… · p. 4, Figure 4 · p. 4
  • Fig. 5 juxtaposes the ground truth with height maps from plain, residual, and skip-connected conv-deconv variants.The panels also include the remote sensing input image.Source: Figure 5 · p. 5
  • The plain conv-deconv net quickly converges to relatively high error and fails to produce physically plausible height maps.Its training and validation errors initially decrease before converging to the high error value.Source: B. Plain Convolutional Net vs.… · pp. 4–5
  • Residual blocks learn a residual function added to the input, and the authors report that this makes the network easier to optimize than plain blocks.The block output is described as F(φl) + φl rather than a direct mapping.Source: B. Plain Convolutional Net vs.… · p. 5, Figure 6 · p. 5

C. Res. Conv-Deconv Net with Skip

The skip-connected residual conv-deconv network is designed to carry low-level image details past the bottleneck. This addresses blurred object boundaries in the network without the skip connection.

C. Res. Conv-Deconv Net with Skip

  • Because the bottleneck forces information through all layers, object boundaries tend to blur in the residual conv-deconv network without a skip connection.The input image and height map are described as having roughly aligned structure despite differing surface appearance.Source: C. Res. Conv-Deconv Net with Skip · p. 5
  • Earlier approaches combined coarse or blurred segmentation maps with input-image superpixels to restore edges, but that strategy cannot be integrated into end-to-end training.The skip connection is proposed as a way to route low-level visual information around the bottleneck.Source: C. Res. Conv-Deconv Net with Skip · p. 6
  • A skip connection from the first block to the next-to-last block concatenates feature maps, directly carrying low-level boundaries and edges across the network.The final architecture uses residual blocks, and the caption identifies the skip connection's endpoints.Source: C. Res. Conv-Deconv Net with Skip · p. 6, Figure 7 · p. 6

III. EXPERIMENTS

The experiments use a Berlin aerial image and evaluate height estimation with scale-invariant error measures and structural similarity.

A. Data and Error Metrics

  • The Berlin aerial image and corresponding DSM were reconstructed using semi-global matching, resampled to about 70 cm ground spacing, and split two-thirds for training and one-third for testing.The test area is indicated in Fig. 9(a).Source: A. Data and Error Metrics · p. 6
  • Evaluation uses a scale-invariant error metric alongside MSE, MAE, and SSIM for estimated height-map quality.SSIM compares local pixel-intensity patterns normalized for luminance and contrast.Source: A. Data and Error Metrics · p. 6

B. Training Details

Training uses augmented RGB–DSM pairs, MAE loss, and early stopping to fit the network. The authors report the optimizer settings, data split, and training resources.

B. Training Details

  • MAE is the network’s loss because it yielded slightly better results than ℓ2 loss in the authors’ experiments.Training uses Nesterov Adam with a learning rate of 0.00002.Source: B. Training Details · pp. 6–7
  • Augmentation expands 840 training images to 8,568 samples through 90-degree rotations and horizontal and vertical flips.The training passage identifies 840 images; the augmentation passage specifies the transformations and resulting count.Source: B. Training Details · pp. 7–8
  • A 7,711/857 training-validation split, batch size 1, and early stopping are used for a network with 3,976,907 trainable parameters.Training took about two hours on one NVIDIA GeForce GTX TITAN with 12 GB of GPU memory.Source: B. Training Details · p. 8

C. Evaluation on Height Maps

On the Berlin test scene, the residual conv-deconv network with skip connection improves the reported MSE, MAE, and SSIM values. Visual comparisons show sharper boundaries than the alternatives, while the plain conv-deconv variant fails to learn realistic maps.

C. Evaluation on Height Maps

  • Across the illustrated Berlin land-use scenes, the proposed model improves object edges, boundaries, and structure definition over Eigen-Net and the residual conv-deconv net.The comparison includes dense residential, vegetative cover, park, commercial zone, river, railway station, and playground scenes.Source: C. Evaluation on Height Maps · p. 8
  • The plain conv-deconv net with skip connection collapses to nearly identical outputs, while Eigen-Net produces blurry maps; the proposed predictions need no post-processing.The authors report that dropout and ℓ2 weight regularization did not make the plain network learn a valid model.Source: C. Evaluation on Height Maps · p. 8
  • 0.0023 of MSE, 0.01 of MAE, and 0.1306 of SSIM are the reported accuracy increases for the residual conv-deconv net with skip connection.The statistics are for the Berlin test scene.Source: C. Evaluation on Height Maps · p. 8
  • Because the network has no fully connected layers, it accepts remote sensing images of arbitrary size; on the large-scale Berlin test zone, its skip connection yields height maps with object-boundary detail rather than blur.Source: C. Evaluation on Height Maps · p. 8

D. Application to Instance Segmentation of Buildings

The Berlin example uses predicted heights to segment building instances without pixel-wise segmentation masks. The pipeline thresholds elevated structures, filters vegetation, and then cleans the result.

D. Application to Instance Segmentation of Buildings

  • The method thresholds predicted heights to extract elevated structures, filters trees using vegetation index, then removes small areas and fills holes.The example frames height-based segmentation as an alternative to collecting expensive pixel-level segmentation masks.Source: D. Application to Instance… · p. 8
  • The Berlin test-scene segmentation is described as satisfactory despite using no pixel-wise labeled masks or supervised training for segmentation.The result is shown in Fig. 13, with two zoomed examples in Fig. 14.Source: D. Application to Instance… · p. 9
  • The authors identify relatively high-rise buildings as a common failure case and a main challenge for monocular height prediction.The cited examples compare input images, ground truth, and predicted height maps.Source: Figure 11 · p. 10

IV. CONCLUSION

The conclusion presents a residual convolutional-deconvolutional network with a skip connection for single-view height prediction. It also demonstrates building instance segmentation from predicted heights and identifies high-rise accuracy as future work.

IV. CONCLUSION

  • The convolutional sub-network extracts abstract features and the deconvolutional sub-network generates height maps, with a skip connection carrying low-level boundary information across the network.The skip connection links the first residual block and the next-to-last block to address blurred object boundaries.Source: §IV · p. 10
  • The paper demonstrates building instance segmentation using height maps predicted from a single monocular image.The conclusion characterizes this application as evidence of the predicted height map’s usefulness.Source: §IV · p. 10, Figure 13 · p. 10
  • Future work will focus on improving the accuracy of high-rise buildings.Source: §IV · p. 10

Related papers

Loading 1802.10249v1…