GroundControl: Satellite-Conditioned Multi-View Diffusion for Reconstruction-Free Novel View Synthesis

1. Introduction

• Current novel-view diffusion models can hallucinate buildings and roads outside sparse input views, producing layouts that may not exist.Satellite views offer a spatial-layout prior, though limited information about ground-level appearance.Source: §1 · p. 1
• Figure 1 compares GroundControl with SEVA [3] and describes GroundControl as more faithful to the scene’s spatial layout.Source: Figure 1 · p. 1
• GroundControl combines satellite layout guidance with street-level appearance in a generative model for ground-level multi-view novel-view synthesis.The street and satellite images are complementary: satellite views provide spatial layout but may miss facades and small objects.Source: §1 · pp. 1–2
• Prior work generates ground-level images from satellite imagery alone, but doing so without ground-level information is highly challenging.Source: §1 · p. 1
• Combining satellite layout guidance with sparse street-level views addresses the challenge of generating street views with accurate textures and appearance.Source: §1 · pp. 1–2

Street view generation from oblique satellite and aerial

• Satellite-to-street methods use intermediate forms including textured meshes, 3D Gaussians, voxels, point clouds, triplanes, and depth estimates.The passage notes that these representations may lack large-scale supervision and introduce downstream artifacts that bottleneck quality.Source: Street view generation from oblique… · p. 2
• Prior work using orthorectified satellite views includes both intermediate-representation methods and direct satellite-to-ground image mappings.Source: Street view generation from oblique… · p. 2
• GroundControl directly conditions a multi-view generation model on satellite images without intermediate representations or supervision.The related-work comparison describes its support for street-level conditioning, orthorectified satellite views, pose control, and multiview-consistent outputs.Source: Street view generation from oblique… · p. 2, Table 1 · p. 2

2.2. Novel-view Synthesis and Multi-View Diffusion

• Classical NVS methods warp input pixels using geometry proxies, while deep-learning approaches also render explicit 3D scene representations.Source: §2.2 · p. 3
• Multi-view diffusion methods use ray-based positional encodings and sequence or joint-view training, but may still hallucinate layouts and semantic artifacts.GroundControl builds on the SEVA backbone to inject satellite-view priors.Source: §2.2 · p. 3

3. Method

• GroundControl adds ray-guided cross-attention to a pretrained multi-view diffusion model, combining satellite and street-view conditioning while retaining its generative prior.The satellite image is encoded with DINOv3, while the street-level images and noisy targets are encoded by the diffusion model.Source: §3 · p. 3

3.1. Multi-view Diffusion Models

• Camera-controlled multi-view diffusion generates target views from input images and their camera parameters, while diffusion training predicts noise in progressively noised samples.Latent diffusion models encode target images as lower-resolution latents and condition on input and target camera parameters.Source: §3.1 · p. 3

3.2. GroundControl: Satellite-conditioned Multiview Diffusion

• GroundControl conditions target-view generation on street images, their camera parameters, target cameras, and a satellite image, using specialized satellite parameters to preserve the pretrained multi-view prior.It assumes known metric relative poses and approximate latitude, longitude, and planar orientation to localize cameras in the satellite image.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Ray-guided Cross Attention

• Ray-guided cross-attention restricts satellite attention to tokens along each street-view token’s 3D ray, improving conditioning over attention to the full satellite image.Queries combine intermediate UNet features with Plucker-ray and timestep embeddings; the final MLP layer is zero-initialized to retain the pretrained latent prior.Source: §3.3 · p. 4, Figure 3 · p. 5

3.4. Training

• Training uses target-view L2 loss on predicted noise, geotagged multi-view datasets, and frozen SEVA weights while training only the satellite-conditioning blocks.Training samples use 21 views with 1–8 inputs; satellite crops cover 128m×128m at 4 pixels per meter, and inference uses 50 DDIM steps.Source: §3.4 · p. 5

3.5. Inference

• At inference, separate classifier-free guidance controls conditioning strength from the input views and satellite image.The model combines predictions conditioned on both inputs, only input images, and neither condition.Source: §3.5 · p. 5

4. Experiments

• Evaluation uses low-overlap target views on in-distribution Metropolis and nuScenes, while DL3DV tests out-of-distribution generalization with higher input-target overlap.Metropolis uses forward-facing input and left/right targets; nuScenes uses sequential front-camera inputs and targets from the other five cameras.Source: §4 · p. 5, Table 3 · p. 8
• GroundControl improves image quality and layout adherence on both in-distribution datasets; gains on nuScenes center on PSNR, SSIM, and semantic IoU, while Metropolis improves across all metrics.On Metropolis, PSNR rises from 10.88 to 12.81 and building IoU from 0.291 to 0.372.Source: Table 2 · p. 6, §4 · pp. 7–8
• On out-of-distribution DL3DV, GroundControl improves PSNR, SSIM, and multi-view consistency while remaining comparable to SEVA in LPIPS, SEG, and DINO.DL3DV has higher input-target overlap than Metropolis or nuScenes, leaving less room for satellite conditioning to help.Source: Table 3 · p. 8, §4 · p. 8

4.1. Ablations

• Ray-guided cross-attention outperforms street-token attention to all satellite tokens; encoder placement beats decoder placement, and raymap plus timestep query embeddings matter.Without those query embeddings, queries are noisy, especially early in diffusion.Source: Table 4 · p. 8, §4.1 · p. 8
• Treating the satellite image as another multi-view input does not condition generation as expected, while separate classifier-free guidance strengths can slightly improve PSNR.The proposed explanation is that satellite images differ substantially from ground views and lack specialized learned processing modules in that alternative design.Source: Table 4 · p. 8, §4.1 · p. 8

4.2. Qualitative Results

• Qualitative driving-scene comparisons show GroundControl better captures unseen scene structure, including placements of trees, buildings, and street furniture, than the baselines.Figure 5 compares GroundControl with SEVA and Sat2Density [50] at the same target viewpoints.Source: Figure 5 · p. 7, §4.2 · p. 8

5. Conclusion

• GroundControl enables effective satellite conditioning of multi-view diffusion models, and the evaluations support its utility for free-viewpoint satellite-conditioned novel-view synthesis.The conclusion reports a new state-of-the-art for this task.Source: §5 · p. 8

A.1. Performance across different input-target view overlap levels

• GroundControl improves PSNR across all tested visual-overlap levels, with noticeably consistent improvement below 200 SIFT correspondences.The analysis samples 20 Metropolis scenes and 20 random input-target configurations per scene.Source: §A.1 · p. 12

A.3. Additional baselines

• Sat2Density uses street-level inputs to derive a global appearance embedding, while SEVA is the primary multi-view diffusion baseline.GroundControl is built on SEVA and is comparable in parameter count and pretraining data.Source: §A.3 · p. 12

A.2. Robustness to satellite pose misalignment

• With up to 10 m translation and 10° rotation offsets, GroundControl suffers little degradation and remains competitive with SEVA, improving PSNR and SSIM.The paper notes that OrienterNet falls within this offset range, reporting 99.5% recall under 5 m and 5°.Source: Table I · p. 12, §A.2 · p. 12

A.3.1. Video generation models

• Vanilla image-to-video models cannot target specific viewpoints, while camera-conditioned versions require continuous target trajectories and typically use only one or two input views.Source: §A.3.1 · p. 12
• GroundControl and SEVA significantly outperform ViewCrafter on DL3DV, despite a comparison setup that favors ViewCrafter.ViewCrafter receives an additional target view and generates frames in batches of 25.Source: §A.3.1 · p. 12
• Across all Metropolis image-overlap regimes, GroundControl achieves better PSNR performance than SEVA.Source: Figure I · p. 13
• GroundControl is slightly better than RealCamI2V on continuous trajectories, and comparable to Veo on turns with better semantic-feature metrics.On straight trajectories, Veo performs better than GroundControl.Source: §A.3.1 · p. 13, Table II · p. 13

A.3.2. Feedforward Regression Models

• Feedforward reconstruction models such as LongLRM are unsuitable baselines because they do not generate content absent from the input views.LongLRM leaves unseen target regions blank, and visible regions also look poor in this extrapolative setting.Source: §A.3.2 · p. 13

• Satellite conditioning improves generations in most cases on both nuScenes and Metropolis, with positive PSNR gains in every Metropolis scene.The table reports the per-scene distribution of PSNRours−PSNRSEVA.Source: §A.4 · p. 13
• Only 3.7% of nuScenes scenes have GroundControl more than 0.5 dB worse than SEVA; common cases involve occlusion or stationary cameras that leave little ground-level context.Source: §A.4 · p. 13
• Flipped or mismatched satellite images degrade generation quality, but the degradation is not too significant.The authors suggest similar driving-scene layouts and the street-image condition’s strong prior may explain the limited degradation.Source: §A.4 · pp. 13–14, Table IV · p. 14

B. Additional Qualitative Results

• GroundControl’s qualitative results highlight multi-view consistency and stronger satellite adherence than SEVA, including challenging scenes with weather-reduced visibility.Source: B. Additional Qualitative Results · p. 14

B.1. Multiple Noise Samples

• Satellite conditioning reduces layout variance relative to SEVA while preserving diversity in building appearance and height.Those appearance and height properties are not observable from satellite imagery.Source: §B.1 · p. 14

B.2. Effect of Dual CFG

• Increasing satellite CFG improves agreement with mapped scene features, but very high values reduce image quality.Dual-CFG separately controls input-view and satellite conditioning strengths.Source: §B.2 · p. 14

B.3. Ablations

• The full model better matches overall scene structure than variants without satellite imagery or without ray-guided cross-attention.Source: §B.3 · p. 14

B.4. Visualizations of Attention Maps for Satellite Rays

• Attention-map visualizations show satellite-image attention for one map-conditioning block.Source: §B.4 · p. 14

C.1. Evaluation datasets

• Metropolis and nuScenes provide little same-timestep overlap between input and target views; overlap mainly comes from camera motion across timesteps.Metropolis uses 7 input and 14 target views per scene; nuScenes uses 7 input and 35 target views.Source: §C.1 · p. 14

C.2. Training datasets

• Satellite images are queried using dataset GPS locations; VIGOR images are recentered, Metropolis crops use the camera-view centroid, and Argoverse 2 uses Google Maps for Pittsburgh.The other stated Argoverse 2 and nuScenes queries use Esri World Imagery; the satellite queries use 4 pixels per meter over 128m × 128m, with parameters provided for reproducibility.Source: §C.2 · pp. 14–15
• Training on small datasets alone can overfit to their camera trajectories, so DL3DV scenes are added to retain performance beyond those datasets.The DL3DV benchmark scenes are removed from training, and satellite encoder features are not used for these scenes.Source: §C.2 · p. 15

D.1. Ray-guided Cross Attention

• Ray-guided satellite cross-attention and an MLP are inserted after SEVA’s intra-view and inter-view self-attention blocks, with residual application to diffusion embeddings.The resulting features pass to the next diffusion block.Source: §D.1 · pp. 17, 19
• Each block uses diffusion features, ray embeddings, a timestep embedding, and 20 satellite-feature samples per street-image ray, evenly spaced to a maximum depth of 62m.Ray and timestep embeddings are added to intermediate features to form queries; ablations find these embeddings important because early diffusion features are noisy.Source: §D.1 · pp. 17, 19
• With different noise samples, GroundControl shows less variation in generated layout than SEVA in regions outside input views, while still producing diverse samples.The qualitative comparison attributes the reduced layout variance to satellite-image conditioning.Source: Figure V · p. 18
• Increasing satellite CFG strengthens adherence to the satellite image but trades off against image quality; the ablation also reports better ground-truth layout fidelity for the full model.The ablation compares no satellite input, satellite input without ray-based cross-attention, and the full model.Source: Figure VI · p. 18, Figure VII · p. 19
• Each image token attends selectively to satellite tokens along its 3D ray, rather than to the full satellite image, enabling conditioning with added layers trained on relatively small paired datasets.For approximately gravity-aligned images, columns of street-image patches attend to the same satellite ray.Source: §D.1 · p. 19

E. Limitations

• The approach is limited to GPS-localized outdoor locations, and conditioning may affect pose consistency or image appearance, especially with out-of-distribution camera trajectories.The authors note that SEVA entangles camera pose and appearance representations; they balance datasets to minimize these effects.Source: E. Limitations · pp. 19–20
• Satellite conditioning can mislead generation through building shadows, while covered areas and large changes since the satellite image was captured may degrade performance.Examples of covered areas include under bridges, tree canopies, and parking lots.Source: E. Limitations · p. 20

F. Future work

• Future work could revisit the method with DiT-based diffusion, use satellite viewpoint metadata to enable joint satellite and oblique-aerial conditioning, or add vectorized semantic maps.The current model does not use satellite viewpoint metadata.Source: F. Future work · p. 20
• In the qualitative comparison, Veo produces highly photorealistic videos and handles dynamic objects, but may miss target camera poses and hallucinate scene content in incorrect locations.GroundControl generates target views in a single pass, whereas two separate Veo videos are generated to use the three conditioning frames.Source: Figure XI · p. 22