Source-linked AI summary
SatUnreal: A High-Precision Synthetic Dataset for Satellite Stereo Matching via Unreal Engine
Han-Gyeol Kim, JaeWan Park, Junmin Park, Darongsae Kwon
TL;DR
Satellite stereo datasets face spatio-temporal mismatch and challenges in generating data that preserves satellite orbital geometry. SatUnreal uses Unreal Engine to create a controlled synthetic dataset, and models trained on it achieve competitive zero-shot transfer to real-world benchmarks.
Problem
Satellite stereo datasets face spatio-temporal mismatch, while large-scale synthetic data generation must preserve the distinctive orbital geometry of Earth-observation satellites.
Source: §2.2 · p. 2, §2.3 · p. 3Method
SatUnreal uses Unreal Engine to simulate satellite stereo geometry and a controlled environment that holds scene conditions fixed across views.
Source: §1 · p. 2, §3.3 · p. 5Results
SatUnreal-trained models achieve competitive zero-shot generalization across US3D and WHU-Stereo, with comparable sub-pixel accuracy.
Source: §4.2 · p. 7, §5 · p. 8Takeaways & Limitations
Physically precise synthetic data effectively mitigates real-world supervisory noise in satellite stereo matching.
Source: §5 · p. 8Evaluation focuses on zero-shot transfer, leaving real-world fine-tuning for future work.
Source: §4.4 · p. 7
Abstract
from arXiv · showhide
3D reconstruction from satellite imagery is essential for large-scale topographic analysis, yet the lack of high-fidelity training datasets with accurate occlusion labels remains a primary bottleneck. Existing benchmarks, such as US3D and WHU-Stereo, face inherent challenges in spatio-temporal mismatch -- environmental changes and shadow displacements between multi-view acquisitions -- and provide ambiguous ground truth in occluded regions due to LiDAR sparsity. In this paper, we propose SatUnreal, a high-precision synthetic dataset designed to fundamentally overcome these limitations through an Unreal Engine-based simulation pipeline. SatUnreal provides 10,000 stereo pairs with high resolution (0.3m GSD) and is characterized by: (1) Physical Geometry Simulation, replicating realistic satellite orbits by systematically varying baselines and azimuths; (2) Spatio-temporal Consistency, eliminating temporal noise through fixed virtual environments; (3) Topographic Diversity, spanning dense urban canyons to low-texture natural terrains; and (4) Mathematical Label Integrity, utilizing a novel two-step linetrace algorithm to generate flawless occlusion masks. Experimental results using SOTA iterative models demonstrate that models trained exclusively on SatUnreal achieve superior zero-shot transfer performance on real-world benchmarks (US3D, WHU-Stereo) compared to those trained on real datasets. Our findings prove that physically accurate synthetic data provides a more effective supervisory signal for learning geometric features than complex real-world observations, establishing a new paradigm for Sim-to-Real transfer in Earth Observation. Code and dataset are available at https://github.com/jmp-Telepix/SatUnreal_A_High-Precision_Synthetic_Dataset_for_Satellite_Stereo_Matching_via_UnrealEngine
1. Introduction
SatUnreal addresses satellite stereo matching’s shortage of reliable training data with an Unreal Engine dataset designed to reproduce satellite geometry and provide accurate labels, diverse terrain, and temporally consistent views.
Introduction
- Satellite stereo matching remains challenging despite deep-learning advances on terrestrial datasets, limiting disparity estimation for satellite-based 3D reconstruction.Disparity maps are described as the fundamental step in the 3D reconstruction pipeline.Source: §1 · p. 1
- Real-world benchmarks are limited by multi-date shadow and terrain changes, while asynchronous terrain data creates ground-truth errors at object boundaries and in occlusions.Large-scale single-pass along-track acquisitions are prohibitively expensive.Source: §1 · p. 1
- SatUnreal uses a scaled aerial proxy and baseline rotation to reproduce satellite orbital geometry and simulate varied acquisition azimuths, covering urban structures occluded in single-heading datasets.The dataset is generated using Unreal Engine.Source: §1 · p. 2
- A two-step linetrace algorithm uses the engine’s geometry buffer to extract mathematically flawless occlusion masks, targeting a major source of real-world reconstruction errors.Source: §1 · p. 2
- SatUnreal spans dense urban areas and low-texture deserts and plains, while simultaneous multi-view image and label generation in a static frame eliminates temporal mismatch and supports superior zero-shot generalization on real imagery.Source: §1 · p. 2
2. Related Work
Related work traces stereo matching’s shift from hand-engineered optimization to learning, while identifying unreliable real-world supervision and satellite-specific geometry as barriers to effective training. Prior synthetic and deep-learning advances motivate SatUnreal’s focus on physically accurate satellite imagery and precise labels.
2.1. Evolution from Traditional to Learning-based Stereo Matching
- Satellite stereo matching moved from geometry-based optimization such as SGM toward deep learning, as traditional methods struggled with low texture and radiometric differences.The passage identifies high-quality training data as a primary bottleneck for continued progress [13].Source: §2.1 · p. 2
2.2. Satellite Stereo Datasets and Their Limitations
- Real satellite datasets such as US3D [3] and WHU-Stereo [15] are challenged by acquisition-time gaps that create shadow shifts and outdated LiDAR ground truth.Environmental changes can make LiDAR labels misrepresent the satellite scene, potentially teaching incorrect correspondences and weakening object-boundary preservation.Source: §2.2 · p. 2
2.3. Advancement of Synthetic Data in Remote Sensing
- Existing synthetic datasets span general stereo and remote sensing, but adapting their pipelines to satellite photogrammetry requires large-scale generation that preserves near-orthographic views and multi-azimuth orbital geometry.Prior work reports synthetic-data generalization to real domains [27] and its utility for topographic scarcity and augmentation.Source: §2.3 · p. 3
2.4. Deep Learning for Stereo Matching
- Iterative stereo models such as RAFT-Stereo [17] and IGEV [28], alongside satellite attention models Selective-IGEV [26] and IAASNet, depend on accurate fine-structure supervision.The paper argues that SatUnreal’s high-resolution ground truth and mathematically flawless occlusion labels support these architectures; foundation models also struggle with near-orthographic satellite geometry.Source: §2.4 · p. 3
3. SatUnreal Dataset Generation
SatUnreal’s generation pipeline combines diverse virtual environments, controlled stereo geometry, frozen scene conditions, and engine-derived disparity and occlusion labels.
3.1. Virtual Environment and Assets Overview
- Urban models from California, Toronto, and Venice, alongside custom natural landscapes, stress-test stereo matching under low texture, occlusions, and sharp depth changes.The natural environments include canyons, deserts, forests, fields, and coasts.Source: §3.1 · p. 3
3.2. Geometric Baseline and Scaled Simulation
- A 1km simulation altitude preserves the B/H ratio and enables 0.3m GSD; baselines of 50m–200m yield convergence angles of approximately 2.5°–12.0° with varied azimuths.Compressing atmospheric effects from a 500km column into 1km leaves a slight radiometric domain gap versus true orbital imagery.Source: §3.2 · pp. 3–5
3.3. Clean Spatio-temporal Supervision
- Freezing illumination, atmospheric effects, and dynamic objects makes camera viewing angle the only significant stereo variation, eliminating temporal noise from supervision.The paper says this clean signal helps models focus on geometric matching, improving convergence speed and reconstruction precision.Source: §3.3 · p. 5
3.4. High-Precision Occlusion and Disparity
- SatUnreal uses rendering-engine View-Projection matrices to generate mathematically precise occlusion labels, replacing estimation used by existing datasets [3].Source: §3.4 · p. 5
3.4.1. Matrix-based Disparity via Reprojection
- Reprojection back-projects left-image pixels into world space and projects them into the right view; rectification reduces disparity to d = uL − uR.The labels are derived from engine camera parameters rather than estimated, maintaining geometric consistency across baseline configurations.Source: §3.4.1 · p. 5
3.4.2. Two-Step LineTrace for Occlusion Masking
- The two-step LineTrace checks left-camera visibility, then ray-casts from each surface point toward the right camera to mark blocked correspondences invalid.Continuous world-space bidirectional ray-casting avoids Z-buffer thresholding failures in multilayered architecture and produces binary occlusion masks.Source: §3.4.2 · p. 5
3.4.3. Dataset Statistics and Composition
- The 10,000-pair dataset combines 20 baseline steps with 20 azimuth angles per pivot point, allocating 8,000 pairs to urban scenes and 2,000 to natural landscapes.Full-frame imagery retains non-overlapping regions as nulls, supporting uncropped training or cropping to overlapping regions for compatibility with pre-cropped benchmarks.Source: §3.4.3 · pp. 5–6
4. Experiments
The experiments assess whether training solely on SatUnreal’s synthetic data supports zero-shot transfer to real-world satellite imagery, compared with training on real-world datasets.
4.1. Experimental Setup
- Models were fine-tuned for 50k iterations with validity masking, and zero-shot evaluations cropped padding and normalized disparities for fair comparison.The study evaluates Selective-IGEV, RAFT-Stereo, and DLNR against US3D, WHU-Stereo, and SatUnreal splits of comparable training scale; metrics are EPE and 1px/3px error rates.Source: §4.1 · pp. 6–7
4.2. Quantitative Results
- SatUnreal-trained models achieve comparable sub-pixel zero-shot accuracy on US3D and WHU-Stereo, with stable transfer, though Selective-IGEV’s EPE can inflate from spatial misalignments despite sharp structural predictions.The authors attribute this sensitivity to mismatches between model boundaries and LiDAR-derived ground truth.Source: §4.2 · p. 7
4.3. Qualitative Analysis
- SatUnreal-trained models preserve crisp disparity boundaries in urban canyons and produce visually comparable reconstructions across domains despite using only synthetic training data.The authors attribute sharp geometric transitions to the two-step linetrace supervision.Source: §4.3 · p. 7
4.4. Discussion and Limitations
- The study isolates geometric supervision through zero-shot evaluation; real-world fine-tuning remains future work, and cross-domain training sometimes outperforms in-domain training.The discussion attributes this to noisy US3D data hindering generalized priors, while curated WHU or SatUnreal data provide stable geometry; real textures can also regularize high-capacity models against synthetic overfitting.Source: §4.4 · p. 7
5. Conclusion and Future Work
SatUnreal uses an Unreal Engine simulation pipeline to provide geometrically precise, multi-view-consistent satellite stereo data with accurate occlusion masks. The conclusion reports competitive zero-shot transfer to real-world imagery and outlines extensions to reduce the Sim-to-Real gap.
Conclusion and Future Work
- Qualitative evaluations show SatUnreal-trained models preserve architectural boundaries and handle complex occlusions across SatUnreal, US3D, and WHU-Stereo.The comparison also notes that DLNR’s global smoothing favors smooth predictions over high-frequency variations in US3D ground truth.Source: Figure 8 · p. 8
- The simulation achieves high practical fidelity through scaled geometric similitude, multi-view consistency, and mathematically precise occlusion masks from a two-step linetrace algorithm [1].It treats nonlinear epipolar curves as negligible for localized patches under orbital stability, validating piecewise-linear approximations.Source: §5 · p. 8
- SatUnreal-trained models achieve competitive zero-shot transfer on real-world imagery, supporting physically precise synthetic data as a way to mitigate real-world supervisory noise.The conclusion does not provide quantitative comparisons here.Source: §5 · p. 8
- Future work will add dynamic illumination, diverse biomes, atmospheric scattering, rigorous push-broom simulations, and support for arbitrary elevation datums (d < 0).The dataset, generation pipeline, and pre-trained models are slated for public release.Source: §5 · p. 8