Source-linked AI summary
Semantic Stereo for Incidental Satellite Images
Marc Bosch, Kevin Foster, Gordon Christie, Sean Wang, Gregory D Hager, Myron Brown
TL;DR
Incidental multi-date satellite imagery offers scalable 3D mapping but creates difficult stereo correspondence problems under seasonal appearance differences and scene change. The paper introduces US3D, a public multi-view, multi-band dataset with geometric and semantic labels, and evaluates lightweight and heavyweight baselines. Initial experiments show that combining semantic cues with stereo correspondence improves performance for challenging image pairs.
Problem
Incidental satellite image pairs lack a public labeled dataset and evaluation methodology for jointly exploiting semantic and stereo cues in remote sensing.
Method
The paper introduces US3D and evaluates heavyweight and lightweight CNN baselines for semantic segmentation, stereo correspondence, and their combination.
Results
Initial lightweight baseline experiments demonstrate the utility of combining semantic segmentation and stereo correspondence for image pairs with significant seasonal appearance change.
Takeaways & Limitations
US3D provides public data and baselines for advancing pairwise semantic stereo and multi-view semantic 3D reconstruction with incidental satellite imagery.
Takeaways & Limitations
Additional work remains to refine semantic labels and address remaining data issues before public release.
Abstract
from arXiv · showhide
The increasingly common use of incidental satellite images for stereo reconstruction versus rigidly tasked binocular or trinocular coincident collection is helping to enable timely global-scale 3D mapping; however, reliable stereo correspondence from multi-date image pairs remains very challenging due to seasonal appearance differences and scene change. Promising recent work suggests that semantic scene segmentation can provide a robust regularizing prior for resolving ambiguities in stereo correspondence and reconstruction problems. To enable research for pairwise semantic stereo and multi-view semantic 3D reconstruction with incidental satellite images, we have established a large-scale public dataset including multi-view, multi-band satellite images and ground truth geometric and semantic labels for two large cities. To demonstrate the complementary nature of the stereo and segmentation tasks, we present lightweight public baselines adapted from recent state of the art convolutional neural network models and assess their performance.
1. Introduction
Incidental satellite imagery is plentiful for global-scale 3D mapping but makes stereo correspondence difficult when image pairs differ in appearance. US3D addresses the lack of public data combining semantic and stereo cues for remote sensing.
- Motivation: Incidental satellite archives are more plentiful and less costly to collect than controlled coincident image sets.Well-selected multi-date pairs can approach coincident-stereo accuracy, but appearance differences make correspondence challenging.
- Motivation: Seasonal appearance differences make stereo correspondence difficult for incidental image pairs.When available image pairs are limited, more robust matching methods are required.
- Research Gap: CNN stereo matching and semantic labels are two promising directions for handling challenging satellite-image correspondence.Semantic priors appear suited to incidental pairs with substantial appearance differences.
- Research Gap: Before US3D, no public labeled dataset or evaluation methodology was available for jointly exploiting semantic and stereo cues in remote sensing.Existing challenge datasets separately advanced semantic labeling and multiview reconstruction.
- Contribution: US3D provides multi-view, multi-band imagery and geometric and semantic labels for pairwise semantic stereo and multiview semantic 3D reconstruction.The authors also provide lightweight and heavyweight baselines and an evaluation methodology.
2. Related Work
Prior benchmarks have advanced stereo, semantic segmentation, and semantic stereo in street-level or remote-sensing settings. US3D extends these efforts to semantic stereo for remote sensing.
- Stereo Benchmarks: Middlebury, KITTI, ETH3D, and MVS3DM established public benchmarks for stereo evaluation, including satellite-image 3D mapping.Data-driven methods increasingly dominate stereo benchmark leaderboards.
- Semantic Segmentation Benchmarks: VOC, COCO, Cityscapes, Mapillary Vistas, SpaceNet, Urban 3D Challenge, and DeepGlobe advanced semantic segmentation benchmarking.Remote-sensing competitions particularly supported building classification in satellite imagery.
- Semantic Stereo Benchmarks: KITTI, Cityscapes, SYNTHIA, and 3DRMS provide semantic stereo data mainly for autonomous driving or natural-environment navigation.These datasets supplied lessons for applying semantic stereo to remote sensing.
3. The US3D Dataset
US3D is an approximately 100-square-kilometer, largely automated dataset spanning Jacksonville and Omaha with incidental WorldView-3 imagery, lidar-derived labels, and stereo-ready products.
- Coverage and Curation: US3D covers approximately 100 square kilometers across Jacksonville, Florida, and Omaha, Nebraska.Its curation is automated except for manual editing of small subsets, supporting expansion to other cities.
- Incidental Satellite Images: The dataset contains 26 WorldView-3 images over Jacksonville and 43 over Omaha, collected across multiple years and months.Panchromatic imagery has approximately 30cm GSD and VNIR imagery approximately 1.3m GSD.
- Ground Truth Geometry: Airborne lidar supplies geometry for training and validation, while denser site-specific lidar and oblique imagery support fine-tuning and sequestered testing.The site-specific lidar has approximately 25cm point spacing and the airborne images approximately 5cm GSD.
- Semantic Labels: Semantic labels are derived from HSIP lidar products including DSM, DTM, tree points, and building polygons, with manual edits for reasonable accuracy.These products support semantic labels and height-related data.
- Image Products: Tiled images include orthorectified labels, XYZ coordinates, normalized DSM heights, aligned lidar intensity, pan-sharpened RGB and VNIR, and retained sensor metadata.Solar-angle metadata is used to cast shadows in lidar intensity images to improve matching reliability.
- Stereo Products: Epipolar rectified image pairs and ground-truth disparity and semantic labels are generated by projecting world coordinates through adjusted RPC models.This construction avoids degenerate correspondences in low-relief terrain.
4. Benchmark Tasks and Metrics
US3D benchmarks pairwise semantic stereo and multiview semantic 3D reconstruction, while also supporting semantic lidar classification and single-image height estimation. Metrics evaluate geometry, semantics, and their combination.
- Pairwise Semantic Stereo: Pairwise semantic stereo predicts semantic labels and disparity from epipolar rectified image pairs.The subtasks can be separated, sequenced and fused, or solved jointly, with semantics and stereo serving as priors for one another.
- Pairwise Semantic Stereo: Pairwise stereo uses average endpoint error (EPE) and fraction of erroneous pixels (D1), while segmentation uses mean intersection over union (mIoU).D1 counts disparity errors greater than three pixels as erroneous.
- Multi-View Semantic 3D Reconstruction: Multiview reconstruction combines multiple epipolar-rectified pairwise solutions and also supports unrectified RPC-based solutions.The unrectified products include semantic labels, XYZ coordinates, and normalized height coordinates.
- Multi-View Semantic 3D Reconstruction: Multiview reconstruction is assessed with stereo Z accuracy and completeness plus a combined mIoU requiring correct semantics and Z error below one meter.End-to-end NITF pipelines first require a 3D translation to account for bundle-adjustment inaccuracies.
- Additional Tasks: The dataset also supports single-view height estimation and semantic lidar point-cloud classification.Provided heights include DSM, DTM, and normalized DSM; point-cloud performance uses mIoU and overall accuracy.
5. Baseline Methods
The baselines adapt heavyweight and lightweight CNN architectures for stereo correspondence and semantic segmentation, including a semantic-label prior for stereo. They also provide a median-filter multi-view semantic stereo pipeline.
- The study adapts heavyweight state-of-the-art and lightweight real-time CNN architectures to assess stereo and segmentation baselines.The implementations are intended to validate the dataset and demonstrate the complementary nature of the two tasks.
- Stereo correspondence: iResNet performs cost calculation, aggregation, disparity estimation, and refinement end to end using multi-scale features, an encoder-decoder, and a refinement subnetwork.
- Stereo correspondence: DenseMapNet jointly learns correspondence and disparity regression in a compact DenseNet-inspired model with 290k parameters.
- Stereo correspondence: The adapted DenseMapNet removes disparity normalization, ignores unlabeled disparity values, and uses depth-wise separable convolutions to reduce the model to 90k parameters.
- Semantic segmentation: ICNet fuses supervised multi-resolution branches, while the adaptation uses Jaccard loss, ignores unlabeled pixels, supports arbitrary band counts, and applies depth-wise separable convolutions.
- Semantic stereo: DenseMapNet receives a fourth semantic-classification channel, enabling sequential execution and bootstrapping between semantic segmentation and stereo correspondence.
- Multi-view semantic stereo: The multi-view semantic stereo baseline triangulates pairwise correspondences, then median-filters reconstructed heights and associated semantic labels.
6. Experimental Evaluation
Experiments use sampled US3D image pairs to evaluate segmentation and stereo baselines, with model resources reported separately. CNN stereo models outperform SGM, while semantic priors reduce outliers after noisy pairs are filtered.
- The initial study samples 1,000 image pairs, using 80% for training and 20% for validation, while excluding sequestered test tiles.
- DeepLab and iResNet run on a Quadro M6000, whereas ICNet and DenseMapNet run on a GTX 1070; model sizes and runtimes appear in Table 1.
- Semantic segmentation: Segmentation baselines achieve reasonable mIoU scores but leave substantial room for improvement because labels are general, appearance varies, and training data are limited.
- Pairwise stereo: iResNet and DenseMapNet significantly outperform SGM, and DenseMapNet nearly matches iResNet with two orders of magnitude fewer parameters.
- Pairwise stereo: The incidental-image stereo task remains highly challenging, with a very high D1 error fraction and seasonal or man-made appearance changes.
- Semantic stereo: Filtering image pairs at a five-pixel SGM endpoint-error threshold reduces label noise, and adding semantic priors produces a notable reduction in outliers.
7. Conclusion
The paper presents US3D as a public dataset and automated labeled-data methodology for semantic stereo with incidental satellite imagery. Initial lightweight baselines support combining semantic segmentation and stereo, while geometric change motivates reliable image-pair selection.
- US3D provides multi-view, multi-band incidental satellite images with geometric and semantic labels, plus an automated methodology for extending labeled challenge data.
- Lightweight baseline experiments demonstrate the utility of combining semantic segmentation and stereo correspondence for image pairs with significant seasonal appearance change.
- Geometric change across many incidental image pairs emphasizes reliable image-pair selection for multi-view 3D reconstruction.
- Semantic labels and remaining data issues were still being refined before public release, while additional baselines were under development.