PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Photogrammetric Digital Surface Models with Pretrained Diffusion Models and Multimodal Conditioning

Antoine Lorentz, Stéphane May, Valentine Bellet, Dawa Derksen, Bastien Nespoulous

arXiv:2609.31199v1eess.IVcs.CV

TL;DR

Photogrammetric DSMs are cost-effective to produce but can contain noise, voids, and outliers, while aerial LiDAR is accurate but costly and limited in availability. The paper adapts Stable Diffusion 3 to refine DSMs using both DSM and Pléiades imagery, and reports lower Dense Urban RMSE in French test cities and held-out Bordeaux.

  • Problem

    Photogrammetric DSMs can contain noise, occlusion-related gaps, and matching errors, while high-accuracy aerial LiDAR is substantially more costly and less available for large-scale mapping.

    Source: §1 · p. 1

  • Method

    The approach adapts Stable Diffusion 3 with patch-wise normalization and separate ControlNets to condition DSM refinement on stereo DSMs and optical imagery.

    Source: §1 · pp. 2–3, §2.2 · p. 4

  • Results

    Dense Urban RMSE falls from 6.00 to 3.45 m in-context and from 4.16 to 2.77 m in Bordeaux with DSM + RGB conditioning.

    Source: §5 · p. 20

  • Takeaways & Limitations

    Optical imagery can complement stereo geometry with image boundaries and surface cues, with lower errors also reported in the held-out Bordeaux city.

    Source: §4 · p. 19

    The method assumes vertical co-registration and surface compatibility with LiDAR, and does not estimate an unknown vertical datum.

    Source: Failure Modes and Limitations · p. 19

Abstract

from arXiv · show

Large-scale Digital Surface Models (DSMs) can be produced cost-effectively from satellite images via stereo-photogrammetry. However, the resulting 3D maps are often contaminated by noise, outliers, and voids. On the other hand, aerial LiDAR provides high-accuracy elevation measurements at a substantially higher cost. In this work, we study diffusion models conditioned both on photogrammetric DSMs and Pl\'eiades imagery to refine vertically co-registered DSMs. We introduce a modified Stable Diffusion 3 architecture with a pruned text stream and a patch-wise normalization strategy, enabling stable training on LiDAR data and transfer from natural images to elevation maps. Experiments in French cities demonstrate that multimodal conditioning improves elevation accuracy, reducing Dense Urban RMSE from 6.00 to 3.45 m in the in-context cities and from 4.16 to 2.77 m in the held-out city of Bordeaux.

1. Introduction

Stereo-photogrammetry provides cost-effective large-scale DSMs, but noise, occlusions, and matching ambiguities limit their quality, while LiDAR is accurate but costly and less available. The study adapts Stable Diffusion 3 for multimodal DSM refinement, with the task scoped to vertically co-registered inputs representing the same surface.

1. Introduction

  • The motivation is to refine noisy, incomplete photogrammetric DSMs toward LiDAR-quality elevation despite LiDAR’s higher cost and limited availability.Photogrammetric errors include occlusions, low-texture regions, and matching ambiguities.Source: §1 · p. 1
  • Prior diffusion work focused on void filling or text-driven terrain generation, rather than comprehensive refinement of noisy photogrammetric data.Other compared approaches target missing regions, resolution changes, sparse depth, or LoD2-like elevations, so the calibrated stereo DSM is used as the task baseline.Source: §1 · p. 2
  • The task assumes vertical co-registration and that DSM and LiDAR represent the same physical surface; its estimand is local-error correction, not reconstruction from arbitrary inputs.Surface compatibility is not directly verified, and close acquisition dates do not guarantee it.Source: §1 · p. 2
  • The study adapts Stable Diffusion 3 with patch-wise normalization, a pruned text stream, and separate ControlNets for DSM and optical-image conditioning.The text-stream pruning reduces the transformer from approximately 2B to 1B parameters while retaining pretrained visual representations.Source: §1 · pp. 2–3
  • Experiments combine LiDAR-HD from 20 French cities with Pléiades-HR imagery and CARS DSMs for nine cities, holding out Bordeaux geographically.The study evaluates adaptation in this French urban setting; transfer to other countries, sensors, processing chains, and terrain types remains for future evaluation.Source: §1 · p. 3

2. Materials and Methods

This section describes the adaptation of pretrained SD3 for elevation refinement, the aligned data and city-based training and evaluation partitions, and the training configurations and computational costs.

2.2.1. Deriving the Loss Function

  • Rectified flow interpolates noise and normalized elevation along a straight path, trains a normalized velocity predictor, and generates elevations by ODE integration followed by denormalization.The training objective is divided by the estimated global scalar E[s^2] to keep its numerical scale near natural-image SD3 training without changing its minimizer.Source: §2.1 · p. 3, §2.2.1 · pp. 4–5, §2.2.2 · p. 6

2.2. Generating Elevation Maps

  • Patch-wise noise scaling addresses unstable training by shifting and scaling source noise to match local elevation-patch characteristics.For LiDAR-only adaptation, the offset is the target mean and the scale is its standard deviation divided by 0.25; for DSM-conditioned modeling, both statistics come from the conditioning DSM. No epsilon, clipping, or zero-variance fallback is used, though none of the reported runs showed scaling-attributable instability.Source: §2.2 · p. 4

2.3. Model Architecture

  • 2B parameters became 1B through text-stream pruning, retaining pretrained image weights and delivering approximately 1.4× higher inference throughput at batch size 32.The model replaces joint text-image attention with image self-attention and removes text-processing components. Single-channel elevation patches are duplicated across three channels for the VAE, whose decoded channels are averaged at inference; scale and offset embeddings are added to the timestep embedding.Source: §2.3 · p. 6, §2.3.1 · p. 7

2.3.2. Conditioning via MultiControlNet

  • Two ControlNets condition the pruned SD3 backbone on DSM and Pléiades RGB, with modality-specific residuals summed element-wise at each transformer block.Other fusion mechanisms were not ablated and are left for future work.Source: §2.3.2 · p. 7

2.4. Data

  • The dataset combines LiDAR-HD, CARS stereo DSMs, and Pléiades-HR across 9 paired cities, while LiDAR coverage spans 20 French cities; products use 512 × 512 patches at 0.5 m resolution.LiDAR point clouds were rasterized at 0.5 m by median aggregation; small connected NoData regions were iteratively filled, and target crops with remaining NaNs were discarded.Source: §2.4 · pp. 7–8, §2.4.1 · p. 8

2.4.2. Spatial Alignment and Partitioning

  • Geographic patches were extracted before splitting, with all fully paired Bordeaux patches reserved for held-out testing and no patch crossing train, validation, or test partitions.Rasters were reprojected to Lambert93 at 0.5 m; DSM acquisitions were vertically calibrated against LiDAR, and the conditional task assumes DSM and LiDAR represent compatible surfaces. Fully paired non-held-out cities used 70%/10%/20% train/validation/test splits.Source: §2.4.1 · p. 8, §2.4.2 · p. 9

2.4.3. Land Cover Stratification

  • Evaluation errors are stratified per 0.5 m pixel using nearest-neighbor labels from the 10 m Theia Land Cover 2021 map, grouped into five broader categories.The labels provide coarse rather than pixel-pure boundaries, and the same finite LiDAR/DSM support is used for every method's reported MAE and RMSE.Source: §2.4.3 · p. 11

2.5. Training

  • Training proceeds through LiDAR backbone adaptation, single-modality ControlNet training, and sequential multimodal ControlNet training; training order and alternative schedules were not controlled-ablation-tested.For multimodal conditioning, the Pléiades ControlNet is trained first and frozen before the DSM ControlNet is added; both stages use 20k steps at batch size 120 followed by 30k at batch size 360.Source: §2.5 · p. 12
  • Table 4 summarizes training configurations and computational cost, and defines CN as ControlNet.Source: Table 4 · p. 12

3. Results

Multimodal conditioning improves Dense Urban refinement, while results vary by land-cover group and test setting. The section also reports stronger generation quality from pretrained initialization and quantifies distortion from the pretrained VAE round trip.

3.1. Backbone

  • 692.8 versus 169.4 FDDINOv2: the pretrained configuration outperforms the best scratch configuration in the tested sweep.The comparison is reported without a training-seed study.Source: §3.1 · pp. 12–13, Table 5 · p. 13

3.2. Pretrained-VAE Round-Trip Diagnostic

  • 0.573 m RMSE in the VAE round trip: this diagnostic measures representation distortion, not error from the learned flow model.The VAE was not fine-tuned.Source: §3.2 · p. 13

3.3. Impact of Multimodal Conditioning Across Land Covers

  • 6.00 to 3.45 m Dense Urban RMSE: DSM + RGB improves on the calibrated input in the in-context cities, and Bordeaux results also show gains for Dense Urban and Roads.In-context Dense Urban RMSE falls further to 3.45 m with DSM + RGB from 3.85 m with DSM-only; Bordeaux conditioning can worsen some other groups.Source: §3.3 · p. 14, Table 6 · p. 14
  • In Bordeaux, DSM-only conditioning worsens Sparse Built-Up and Vegetation errors against input; RGB recovers Sparse Built-Up, but Vegetation MAE remains higher and RMSE is nearly unchanged.Vegetation MAE is 1.75 versus 1.69 m, and RMSE is 3.10 versus 3.12 m.Source: §3.3 · p. 14
  • 6.64 to 0.98 m Croplands RMSE with DSM + RGB in-context, versus 1.92 to 1.61 m in Bordeaux; the authors note possible explanations but did not measure them separately.RGB adds little to Croplands MAE in either test.Source: §3.3 · p. 14
  • Paired RMSE intervals show the clearest Bordeaux gains over calibrated input for Dense Urban and Roads, while Vegetation includes zero; the two tests use different bootstrap units.The in-context intervals resample cities, whereas Bordeaux intervals resample spatial blocks within one city.Source: §3.3 · p. 15, Figure 6 · p. 15
  • For original DSM voids, DSM + RGB has lower mean void RMSE than DSM-only in all five groups in both tests; these results are separate from the headline support.The calibrated DSM input is undefined in voids; on P90excluded patches, RGB raises Croplands RMSE relative to DSM-only, 2.240 versus 2.064 m.Source: §3.3 · p. 15
  • Figures 7–9 compare calibrated DSMs, predictions, LiDAR, and errors, documenting noise reduction, surface boundaries, structure placement, and remaining errors.Appendix examples include refinement and Bordeaux negative-transfer cases.Source: §3.3 · p. 15, Figure 7 · p. 16, Figure 8 · p. 17

4. Discussion

Multimodal DSM refinement improves Dense Urban accuracy and also supports void recovery, but the value of RGB varies by surface and input conditions. The discussion bounds these findings by the study’s assumptions, evaluation setting, and untested comparisons.

4. Discussion

  • With DSM + RGB, Dense Urban RMSE decreases by 43% in-context and 33% in Bordeaux relative to calibrated input; the benefit extends beyond training-city locations, though Bordeaux is one held-out city in the same acquisition and processing setting.Source: §4 · p. 19
  • RGB’s benefit varies: it changes Croplands MAE little on retained tests, raises RMSE on excluded high-error patches, and slightly worsens Bordeaux Vegetation MAE.The large in-context Croplands gain has a wide city-bootstrap interval; Bordeaux Vegetation RMSE is close to zero difference from the input.Source: §4 · p. 19
  • DSM + RGB lowers mean void RMSE relative to DSM-only in every evaluated group, making void recovery a secondary benefit.Void pixels have no observed input elevation against which to measure improvement.Source: §4 · p. 19
  • Paired-bootstrap intervals reveal between-city variation for in-context Croplands, while Bordeaux intervals describe within-city block variation, not cross-city transfer.Pretrained initialization yields a closer LiDAR feature distribution than the best tested scratch run, and the VAE round trip has 0.573 m RMSE distortion; neither isolates each component’s contribution to final accuracy.Source: §4 · p. 19

Failure Modes and Limitations

  • Interpretation assumes vertical co-registration and surface compatibility with LiDAR; RMSE filtering is only an indirect heuristic and changes evaluated-data composition.Dense Urban pixels are 1.81× more common among exclusions, and the effect of interpolating small LiDAR voids around roofs and canopies remains unquantified.Source: Failure Modes and Limitations · p. 19
  • Acquisition differences and residual misalignment complicate DSM–LiDAR comparisons, and performance on high-rise buildings, bridges, and steep terrain remains unevaluated separately.Bordeaux examples also document local degradation in vegetation and built areas despite aggregate gains.Source: Failure Modes and Limitations · p. 19
  • Bordeaux tests one held-out French city within the same Pléiades-HR/CARS setting; adjacent training and test patches can touch despite no pixel overlap.City-level bootstrap intervals do not remove split-level spatial dependence, and 10 m Theia labels provide coarse strata on the 0.5 m evaluation grid.Source: Failure Modes and Limitations · p. 20
  • The comparisons establish gains over calibrated DSM and DSM-only, not superiority over other enhancement methods; task-matched baselines and repeated training runs remain needed.Tile-boundary continuity also remains to be assessed for wider deployment.Source: Failure Modes and Limitations · p. 20

5. Conclusions

The study adapts a pretrained diffusion model to refine CARS DSMs under vertical co-registration and surface-compatibility assumptions. In the evaluated French cities, DSM + RGB improves Dense Urban RMSE, while broader geographic and sensor settings remain for future work.

5. Conclusions

  • The refined CARS DSMs are evaluated under vertical co-registration and surface compatibility with the LiDAR reference.These are stated input assumptions, not claims that the method estimates an unknown vertical datum.Source: §5 · p. 20
  • Dense Urban RMSE with DSM + RGB falls from 6.00 to 3.45 m in-context and from 4.16 to 2.77 m in Bordeaux.Dense urban areas and roads show clear gains, while Bordeaux Vegetation can be neutral or slightly harmful.Source: §5 · p. 20
  • The supported conclusion is multimodal DSM refinement within the tested data conditions; larger geographic scales, other cities and sensors, and unfiltered independently calibrated inputs remain future evaluations.Timestep distillation is proposed as a way to reduce sampling steps and make large-area processing more practical.Source: §5 · p. 20
  • Disaster-change mapping is a prospective application that would require checking whether refinement preserves genuine elevation changes.The adapted backbone could also be investigated as a pretrained representation for geospatial segmentation and object detection.Source: §5 · p. 20
  • LiDAR-HD, Pléiades-HR, and derived stereo-DSM imagery cannot be redistributed because of licensing restrictions.Source TIFFs, raw patches, model weights, and implementation are not redistributed; non-restricted summaries are available from the corresponding author upon reasonable request and subject to licenses.Source: §5 · p. 20

Abbreviations

The manuscript expands abbreviations used for its models, data, and evaluation metrics.

Abbreviations

  • DSM means Digital Surface Model, DEM means Digital Elevation Model, and LiDAR means Light Detection and Ranging.These terms name elevation products and the measurement technology referenced in the manuscript.Source: Abbreviations · p. 21
  • SD3 means Stable Diffusion 3, MM-DiT means Multimodal Diffusion Transformer, and VAE means Variational Autoencoder.The listed abbreviations cover the model and its architecture terminology.Source: Abbreviations · p. 21
  • MAE means Mean Absolute Error, RMSE means Root Mean Square Error, and RGB means Red-Green-Blue.The manuscript also expands FFN as Feed-Forward Network, MLP as Multilayer Perceptron, CN as ControlNet, and ODE as Ordinary Differential Equation.Source: Abbreviations · p. 21

Appendix A. Training and Inference Details

The appendix points to shared training hyperparameters and a separate procedure for comparing pretrained and scratch backbones.

Appendix A. Training and Inference Details

  • The appendix identifies Tables A1 and A2 as the sources for training settings and the backbone evaluation procedure.Table A1 is common across reported experiments; Table A2 concerns pretrained and scratch backbone comparison.Source: Appendix A · p. 21, Table A1 · p. 21, Table A2 · p. 21
  • Table A1 summarizes common training hyperparameters across the reported experiments.The listed settings include BF16 precision, 50 sampling steps, disabled classifier-free guidance, and a pretrained learning rate of 1 × 10−5.Source: Table A1 · p. 21, Appendix A · p. 21
  • Table A2 summarizes settings for pretrained-versus-scratch backbone comparisons using FDDINOv2.The appendix lists scratch learning-rate sweeps of {1, 2, 5} × 10−5 and {1, 2, 4} × 10−4.Source: Table A2 · p. 21, Appendix A · p. 21

Appendix B. Computational Effect of Pruning

Pruning substantially reduces model size and improves inference efficiency in the reported benchmark. The comparison uses synthetic inputs and isolates pruning’s computational effect rather than prediction accuracy or full-pipeline timing.

Appendix B. Computational Effect of Pruning

  • The timing comparison uses synthetic inputs and 512-pixel patches, excluding model loading, conditioning-image encoding, preprocessing, and storage.It measures 50 denoising steps followed by VAE decoding, isolating pruning’s computational effect rather than prediction accuracy or full-pipeline processing.Source: Appendix B · pp. 21–22
  • The benchmark reports runtime, throughput, and GPU memory on one NVIDIA A100 80 GB PCIe GPU using BF16 and 50 denoising steps.Times average two runs after one warm-up; memory is measured after warm-up.Source: Table A3 · p. 22
  • The transformer shrinks from 2.028B to 1.033B parameters and each ControlNet from 1.063B to 0.547B, with faster inference and lower memory at every tested batch size.The shared VAE contains 0.084B parameters.Source: Appendix B · p. 22

Appendix C. Dataset Splits, Filtering, and Normalization

The appendix describes patch assignment, filtering, normalization, and land-cover grouping, then evaluates excluded high-error patches, original voids, and variation across inference seeds.

Appendix C. Dataset Splits, Filtering, and Normalization

  • Each city is split into non-overlapping 512 × 512-pixel patches before assignment; eight in-context cities use approximately 70% training, 20% test, and 10% validation, while Bordeaux is reserved for testing.Adjacent patches may touch without a spatial buffer, and all RGB acquisitions for a patch stay in the same split.Source: Appendix C · p. 22, Table A4 · p. 23
  • The P90 filter removes DSM conditioning from 4,064 of 40,589 assessed patches while retaining LiDAR targets, and exclusions contain a larger share of Dense Urban and other Theia classes.The threshold is target-dependent because it is based on error against LiDAR; filtering changes land-cover composition as well as input quality.Source: Appendix C · p. 22, Table A5 · p. 23
  • Paired LiDAR and DSM patches have similar mean-elevation distributions but differ in local elevation variation; observed LiDAR patch scales include values below 0.1 m, mostly over water.No paired DSM patch fell below 0.1 m standard deviation, and the implementation provides neither scale clipping nor a zero-variance fallback.Source: Appendix C · p. 23, Figure A1 · p. 24
  • Land-cover comparisons use five Theia groups—Dense Urban, Croplands, Sparse Built-Up, Roads, and Vegetation—with 10 m labels resampled to 0.5 m.Each source label covers approximately 20 × 20 evaluation pixels; classes outside these groups are excluded from reported comparisons.Source: Appendix C · p. 23

Appendix D.1. Performance on Patches Excluded by the RMSE Filter

  • On patches excluded by the P90 filter, both models reduce RMSE versus the calibrated input in every group; RGB adds gains except in Croplands, where DSM-only performs better.The comparison uses the same valid LiDAR/DSM pixels and averages predictions over 20 inference seeds.Source: Appendix D.1 · p. 24, Table A6 · p. 24

Appendix D.2. Recovery of Original DSM Voids

  • DSM + RGB reduces mean void RMSE relative to DSM-only in every group in both test sets, while Vegetation remains among the most difficult groups.Evaluation includes pixels with valid LiDAR but no input DSM elevation, excluding locations outside the source DSM image.Source: Appendix D.2 · pp. 24–25, Table A7 · p. 25

Appendix D.3. Spatial Uncertainty and Inference Seeds

  • Across 20 inference seeds, the largest reported standard deviation for MAE or RMSE is 0.035 m; repeated training runs are not included.Variation includes random generation and, for DSM + RGB, selection among available RGB acquisitions.Source: Appendix D.3 · p. 25, Table A8 · p. 25

Appendix D.4. Bias and Error Distribution

The appendix supplements MAE and RMSE with bias and error-distribution measures, then tests how removing land-cover boundaries and patch edges affects RMSE.

Appendix D.4. Bias and Error Distribution

  • DSM + RGB lowers NMAD in every group in both tests; its P95 is below the calibrated input across in-context groups but remains higher for Sparse Built-Up and Vegetation in Bordeaux.NMAD measures error spread around the median with less outlier sensitivity; P95 is the 95th percentile of absolute error.Source: Appendix D.4 · p. 25, Figure A2 · p. 26
  • Bias, NMAD, and P95 are reported by land cover for the in-context test and Bordeaux, using only pixels with valid LiDAR and input DSM elevations.Bias uses all evaluated pixels; NMAD and P95 use sampled pixels with equal patch weighting before grouping.Source: Figure A2 · p. 26

Appendix D.5. Effect of Excluding Land-Cover Boundaries

  • Removing boundaries generally lowers RMSE for Croplands, Sparse Built-Up, and Roads, but effects are less uniform for Dense Urban and Vegetation.Only a small fraction of Roads and Bordeaux Dense Urban pixels remains, so the analysis describes class interiors rather than improved label resolution.Source: Appendix D.5 · p. 26, Figure A3 · p. 27

Appendix E. Selected Qualitative Cases

The qualitative appendix pairs random examples with selected cases to show both visual changes and remaining local errors in predictions.

Appendix E. Selected Qualitative Cases

  • Four excluded locations illustrate visible changes between RGB dates, spanning urban redevelopment, industrial construction, woodland conversion to orchard, and shoreline change.The figure compares calibrated DSM, conditioning and later RGB, LiDAR, and DSM error.Source: Appendix E · p. 27, Figure A4 · p. 28
  • Random in-context examples show smoother open surfaces and more distinct buildings and tree crowns, alongside remaining differences from LiDAR.The figure shows calibrated DSM, RGB, DSM + RGB prediction, and LiDAR rows across 24 randomly selected test patches.Source: Appendix E · p. 27, Figure A5 · p. 29
  • Selected Bordeaux cases show local DSM + RGB RMSE increases in vegetation and built areas despite aggregate improvements.The figure selects the largest whole-patch increases and the largest increases within Dense Urban pixels; these are failure cases, not average results.Source: Appendix E · p. 27, Figure A6 · p. 30

Appendix F. Access and Limitations

Data redistribution is restricted by IGN/CNES and commercial-imagery licenses, and the implementation is not publicly released. The reported results are limited to vertically co-registered, compatible DSM–LiDAR pairs in the evaluated French cities.

Appendix F. Access and Limitations

  • IGN/CNES and commercial-imagery licenses restrict redistribution of the data.The paper provides rendered examples and numerical summaries, but no source imagery, elevation rasters, or model weights.Source: Appendix F · p. 31
  • The implementation is not publicly released.Underlying products remain subject to their original licenses.Source: Appendix F · p. 31
  • Results apply to vertically co-registered, compatible DSM–LiDAR pairs in the evaluated French cities.Source: Appendix F · p. 31

Related papers

Loading 2609.31199v1…