Source-linked AI summary
Predicting Ground-Level Scene Layout from Aerial Imagery
Menghua Zhai, Zachary Bessinger, Scott Workman, Nathan Jacobs
TL;DR
Aerial-image segmentation is hindered by expensive manual labels and limited transfer across imagery sources. The paper learns aerial features by predicting noisy semantic layouts from co-located ground imagery through an adaptive cross-view transformation. The resulting model supports rough aerial labeling without additional training, improves after finetuning, and extends to ground-image geolocation, orientation estimation, and panorama synthesis.
Problem
Manual aerial-image labeling is expensive, and labels often transfer poorly between aerial-image datasets and modalities.
Method
The method uses automatically segmented co-located ground images as weak supervision and jointly learns aerial semantic features with an adaptive transformation into the ground-level viewpoint.
Results
The learned model provides rough aerial semantic labeling without additional training, improves segmentation after finetuning over two initialization strategies, and supports localization, orientation estimation, and ground-panorama synthesis.
Takeaways & Limitations
Automatically labeled ground imagery can provide semantically meaningful aerial features for pixel-level labeling and multiple cross-view tasks.
Takeaways & Limitations
Ground-segmentation noise, domain mismatch, capture-time differences, and low synthesized-panorama resolution limit output quality.
Abstract
from arXiv · showhide
We introduce a novel strategy for learning to extract semantically meaningful features from aerial imagery. Instead of manually labeling the aerial imagery, we propose to predict (noisy) semantic features automatically extracted from co-located ground imagery. Our network architecture takes an aerial image as input, extracts features using a convolutional neural network, and then applies an adaptive transformation to map these features into the ground-level perspective. We use an end-to-end learning approach to minimize the difference between the semantic segmentation extracted directly from the ground image and the semantic segmentation predicted solely based on the aerial image. We show that a model learned using this strategy, with no additional training, is already capable of rough semantic labeling of aerial imagery. Furthermore, we demonstrate that by finetuning this model we can achieve more accurate semantic segmentation than two baseline initialization strategies. We use our network to address the task of estimating the geolocation and geoorientation of a ground image. Finally, we show how features extracted from an aerial image can be used to hallucinate a plausible ground-level panorama.
1. Introduction
The paper addresses costly, poorly transferable manual labeling for aerial imagery by using co-located ground-image semantics as weak supervision. It introduces a cross-view prediction strategy and positions dense ground-layout prediction as a new direction for aerial understanding.
- Motivation: Manual aerial-image annotations are expensive and often transfer poorly across datasets, while existing exceptions use coarse classes and limited coverage.These constraints have hampered deep-learning progress for aerial imagery.
- Approach: The proposed strategy predicts semantic labels from a centered aerial image using automatically segmented, geo-tagged ground images as weak supervision.The method uses existing ground-image segmentation methods rather than manually labeling aerial imagery.
- Novelty: The work is presented as the first attempt to predict dense pixel-level ground-image segmentation from an aerial image.This extends prior cross-view work beyond feature matching or other less dense representations.
- Contributions: The contributions include a CNN relating aerial appearance to same-location ground layout, aerial pre-training, localization, orientation estimation, synthesis, and evaluation on large real-world datasets.Together, these contributions extend deep-learning techniques toward aerial-image understanding.
2. Related Work
Prior work connects aerial and ground viewpoints through transformations, domain adaptation, and cross-view matching, but this paper focuses on predicting dense semantic ground layout. Its architecture learns an input-dependent spatial transformation jointly with aerial semantic features.
- Learning Viewpoint Transformations: Viewpoint-transformation methods range from space-variant linear filters to end-to-end learnable neural modules.These approaches motivate learning flexible mappings between image viewpoints.
- Relating Aerial and Ground-Level Viewpoints: Cross-view research has used aerial imagery for ground-image recognition, road segmentation, tree mapping, change detection, and geolocalization.Prior methods establish that co-located aerial and ground imagery can support several visual reasoning tasks.
- Relating Aerial and Ground-Level Viewpoints: The paper distinguishes its contribution by predicting the semantic layout of a ground image from an aerial image at dense pixel level.The authors identify this as the first work to explore that task.
- Network Architecture: The architecture uses VGG16 and PixelNet hypercolumns, then networks A, S, and F to produce semantic features, transformation controls, and the viewpoint mapping.Applying the learned transformation to aerial semantic features yields ground-level semantic labeling.
- Visual Domain Adaptation: Compared with related domain-adaptation work, the transformation operates across feature and spatial dimensions, preserves semantic meaning, depends on the input, and is learned jointly with source features.These design choices support adaptive cross-view mapping rather than a fixed transformation.
3. Cross-view Supervised Training
The method learns aerial-image features by predicting semantic layouts extracted from aligned ground panoramas, using an image-conditioned transformation between viewpoints. Training combines CNN feature extraction, adaptive cross-view mapping, and end-to-end segmentation supervision.
- Training data and objective: Ground panoramas are semantically labeled automatically and paired with georegistered aerial images, avoiding manual aerial-image annotations during training.The panorama is oriented to the aerial image before extracting four-class semantic layouts with an off-the-shelf segmentation method.
- Training data and objective: The aerial CNN predicts a semantic layout that is transformed into the ground viewpoint and compared with the ground layout during end-to-end training.The training procedure minimizes cross entropy between the extracted ground labels and the transformed aerial prediction.
- Adaptive transformation: The cross-view transformation applies a matrix channel-wise by reshaping the aerial label map, multiplying by M, and reshaping the result into the ground-label dimensions.This maps an aerial label tensor of size ha × wa × 4 to a ground label tensor of size hg × wg × 4.
- Adaptive transformation: Each transformation-matrix element is conditioned on aerial-image features and input/output pixel locations, allowing the mapping to adapt to image content and viewpoint coordinates.The transformation network computes matrix elements from aerial features together with normalized aerial and ground locations, followed by softmax normalization.
- Adaptive transformation: The adaptive representation reduces parameters when the transformation matrix is large and supports different aerial-image layouts without interpolation when feature-map resolutions change.These are stated advantages over treating the full transformation matrix as unconstrained learnable variables.
- Dataset and implementation: The dataset uses CVUSA ground panoramas and Bing Maps aerial images, with panorama warping based on camera extrinsic information to align the views.CVUSA contains approximately 1.5 million geotagged ground–aerial pairs before filtering for available aerial imagery.
4. Evaluation and Applications
The evaluation spans weakly supervised aerial labeling, supervised finetuning, geocalibration, and cross-view synthesis. The learned aerial features support semantic prediction and downstream localization, orientation estimation, and panorama hallucination, while synthesis remains limited by resolution and variable appearance.
- Weakly Supervised Learning: The network learned semantic features from aerial images without manually annotated aerial imagery.It was trained to predict ground-level semantic labeling and produced pixel-level aerial outputs.
- Weakly Supervised Learning: The training data uses noisy ground-image segmentation, visualized as road, vegetation, and man-made classes in aerial-image outputs.Figure 4 shows inferred pixel-level labels for test aerial images.
- Cross-view for Pre-training: Finetuning the proposed initialization outperformed random and ImageNet-pretrained VGG16 initialization on ISPRS aerial segmentation.The advantage was especially pronounced for Building, Low Vegetation, and Tree classes.
- Cross-view for Geocalibration: The learned ground-level feature maps were used for ground-image orientation and location estimation.Orientation estimation compares inferred ground labels with aerially predicted labels across candidate orientations, while geocalibration evaluates distributions over orientations at nearby locations.
- Cross-view for Pre-training: ISPRS evaluation compares initialization methods across training-set sizes using average precision for labeled pixels.The dataset contains true orthophotos from Vaihingen, Germany, with six pixel categories and evaluation at multiple image counts.
- Synthesizing Ground Images from Aerial Images: The network hallucinated plausible ground-level scene layouts, capturing roads, their orientations, trees, and grass but struggling with buildings and sky.The authors attribute these difficulties likely to highly variable appearance factors.
- Synthesizing Ground Images from Aerial Images: Synthesized ground-level panoramas had much lower resolution than the original panoramas.The authors identify higher-resolution adversarial generation and improved ground-image segmentation as routes toward more photo-realistic predictions.
5. Conclusion
The paper uses automatically labeled ground images as weak supervision to learn aerial-image representations and applies them to labeling, geocalibration, and ground-view synthesis.
- Automatically labeled ground images provide weak supervision for learning to understand aerial images.
- The method extracts semantically meaningful aerial features and refines them for more accurate pixel-level aerial labeling.
- The learned representations support estimating a ground image’s location and orientation.
- Aerial-image features can be used to synthesize novel ground-level views.
- The technique is also applicable to NIR, multispectral, and hyperspectral imagery, and future work will explore richer ground-image annotations.
Appendix
The appendix provides visualizations, qualitative examples, geocalibration results, and network specifications for the paper’s weakly supervised labeling, geocalibration, and ground-image synthesis applications.
- Figure 11 visualizes the learned transformation matrix and an alternative cell-based heat-map representation.The alternative visualization contains h_g × w_g cells.
- Figure 12 shows aerial images with pixel-level labels alongside ground-image labels inferred from aerial-image labels for road, vegetation, and man-made classes.
- Figure 13 presents fine-grained geocalibration visualizations, including feature maps, an orientation flow map, and predicted versus ground-truth frustums.Arrow direction indicates the optimal orientation and arrow length indicates its magnitude.
- Tables 2 and 3 specify the deep generator and deep energy network architectures used for ground-image synthesis.The generator uses extracted cross-view features and Gaussian noise, while the energy network receives the generated image and associated parameters.