PaperlayerView exact revision on arXiv

Source-linked AI summary

Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on Aerial Lidar

Jamie Tolan, Hung-I Yang, Ben Nosarzewski, Guillaume Couairon, Huy Vo, John Brandt, Justine Spore, Sayantan Majumdar, Daniel Haziza, Janaki Vamaraju, Theo Moutakanni, Piotr Bojanowski, Tracy Johns, Brian White, Tobias Tiecke, Camille Couprie

arXiv:2304.07213v3cs.CVcs.LG

TL;DR

Fine-scale vegetation structure maps are needed for carbon modeling and monitoring forest degradation, restoration, and tree-based land uses, but airborne-lidar supervision is geographically limited. The paper combines self-supervised ViT features from Maxar imagery with an aerial-lidar-trained decoder and GEDI-based correction to map canopy height across California and São Paulo. The resulting maps capture small-scale tree-structure variability, while their validation and transfer remain bounded by imagery and training-data limitations.

  • Problem

    Airborne-lidar scarcity limits the geographic generalizability of high-resolution canopy-height mapping, despite the importance of fine-scale vegetation structure for carbon and restoration monitoring.

  • Method

    The method extracts self-supervised ViT features from Maxar imagery, trains a dense convolutional decoder on aerial lidar, and rescales predictions using a globally trained GEDI model.

  • Results

    The maps better capture tree-structure variability at small spatial scales and support canopy-height mapping across California and São Paulo.

  • Takeaways & Limitations

    Very high resolution canopy-height maps can improve monitoring of forest degradation, restoration, and forest carbon dynamics.

  • Takeaways & Limitations

    Recent ALS training data remain geographically limited, and field data did not assess possible height saturation for very tall trees.

Abstract

from arXiv · show

Vegetation structure mapping is critical for understanding the global carbon cycle and monitoring nature-based approaches to climate adaptation and mitigation. Repeated measurements of these data allow for the observation of deforestation or degradation of existing forests, natural forest regeneration, and the implementation of sustainable agricultural practices like agroforestry. Assessments of tree canopy height and crown projected area at a high spatial resolution are also important for monitoring carbon fluxes and assessing tree-based land uses, since forest structures can be highly spatially heterogeneous, especially in agroforestry systems. Very high resolution satellite imagery (less than one meter (1m) Ground Sample Distance) makes it possible to extract information at the tree level while allowing monitoring at a very large scale. This paper presents the first high-resolution canopy height map concurrently produced for multiple sub-national jurisdictions. Specifically, we produce very high resolution canopy height maps for the states of California and Sao Paulo, a significant improvement in resolution over the ten meter (10m) resolution of previous Sentinel / GEDI based worldwide maps of canopy height. The maps are generated by the extraction of features from a self-supervised model trained on Maxar imagery from 2017 to 2020, and the training of a dense prediction decoder against aerial lidar maps. We also introduce a post-processing step using a convolutional network trained on GEDI observations. We evaluate the proposed maps with set-aside validation lidar data as well as by comparing with other remotely sensed maps and field-collected data, and find our model produces an average Mean Absolute Error (MAE) of 2.8 meters and Mean Error (ME) of 0.6 meters.

1. Introduction

High-resolution vegetation structure maps are needed to reduce uncertainty in carbon accounting and monitor distributed restoration and land-use practices. Because airborne lidar is geographically scarce, the paper develops a scalable canopy-height mapping approach using self-supervised learning and high-resolution imagery.

  • Tree height and crown area maps support forest degradation, restoration, and above-ground woody biomass assessment.
  • Coarse spatio-temporal resolution in global carbon models contributes to uncertainty in nation-wide and global forest carbon accounting.
  • Distributed practices such as agroforestry, reforestation, and natural regeneration are difficult to evaluate at large scale because they occur across individual land parcels.
  • Forest height and biomass measurements are critical for representing spatially variable carbon removal rates in heterogeneous landscapes.
  • Existing spaceborne-lidar approaches produced global or regional canopy-height maps at resolutions including 30 m and 10 m.
  • Airborne lidar enables high-resolution canopy-height mapping but its limited availability can reduce model generalizability to unseen, data-poor geographies.
  • The study uses DINOv2 self-supervised encodings, a dense vision-transformer decoder trained on aerial lidar, and GEDI-based refinement to produce scalable high-resolution maps.

2. Data

The study combines globally available Maxar imagery with airborne-lidar canopy-height data and manually annotated imagery for training and evaluation. The data are standardized into fixed-size image regions and carefully separated into training, calibration, and site-held-out validation sets.

  • Maxar Vivid2 mosaic imagery provided global RGB input from multiple satellite instruments and observation dates.
  • The imagery pipeline selected scenes with less than 2% cloud cover, mostly below-36-month revisit intervals, view angles below 30°, and sun angles below 60° from zenith.
  • Training thumbnails used a 152.7 m box resampled to 256×256 pixels in a local tangent-plane coordinate system.
  • The self-supervised stage randomly sampled 18 million 256×256 satellite thumbnails without labels.
  • Approximately 5,800 one-meter NEON canopy-height maps were paired with RGB imagery to train the supervised decoder.
  • The NEON pairs were split into 80% training, 10% calibration, and 10% site-held-out validation data called NEON test.
  • For São Paulo, canopy-height maps were generated from point clouds using the pit-free algorithm.
  • Training imagery was augmented with random 90° rotations, brightness changes, and contrast jittering to improve prediction stability.

3. Model and data generation methods

The method combines self-supervised ViT features, an aerial-lidar-trained dense decoder, and a globally trained GEDI model that rescales high-resolution predictions. This design uses aerial lidar for spatial detail and GEDI for broader geographic context.

  • Aerial lidar supplies high-resolution canopy estimates, while GEDI offers near-global transects but cannot identify individual trees because its beam width is approximately 25 m.
  • DINOv2 was self-supervised on 18 million globally distributed 256×256 Maxar images to produce vegetation-oriented image features.
  • A convolutional DPT decoder was trained on aerial-lidar canopy-height maps to convert SSL features into full-resolution canopy predictions.
  • The decoder replaced DPT’s transformer with the SSL encoder, fused outputs from multiple layers, and performed best when the SSL encoder layers were frozen.
  • The supervised model used Sigloss between predicted and true canopy-height maps, with δ_i defined as the difference between predicted and true log heights.
  • A five-convolutional-layer and five-fully-connected-layer GEDI network predicted lower-resolution canopy-height information from Maxar RGB imagery and metadata.
  • R2 = 0.88 measured the correlation between ALS 95th-percentile canopy height and simulated GEDI RH95.
  • The GEDI model produced a dense correction factor that was upsampled, smoothed, and clipped between 0.5 and 2 before multiplying the ALS canopy-height map.

4. Results

The study generates very high-resolution canopy height maps for California and São Paulo and evaluates them against lidar, GEDI, field data, and other remotely sensed maps. Results show strong performance, improved segmentation with self-supervised learning, and useful generalization across imagery and regions.

  • Map generation: 0.59 m GSD Maxar imagery was used to generate canopy height maps for California and São Paulo with the SSL + GEDI model trained on 1 m aerial lidar.The maps cover imagery acquired mainly between 2018–2020 in California and 2017–2019 in São Paulo.
  • Canopy height metrics: The SSL model trained on 18 million satellite images reduced Neon MAE from 3.1 to 2.6 meters and produced similar São Paulo results with substantially lower ME than the ResUNet baseline.The ablation also indicated that satellite-image pre-training and more self-supervised data improved performance.
  • Regional performance: The SSL model performed best across U.S. datasets, whereas SSL + GEDI performed best in São Paulo, possibly because of ecosystem domain shift and added geographic metadata.This regional difference indicates that the GEDI correction can improve generalization outside the U.S. evaluation settings.
  • GEDI correction: GEDI correction reduced the CHM model’s negative bias against GEDI RH95 and increased R2 in 42 of 45 subregion-model combinations.The SSL + GEDI model had the highest agreement with GEDI RH95 in 13 of 15 geographic regions.
  • Segmentation metrics: The SSL approach improved all segmentation metrics over ResUNet, while both approaches produced similarly sharp maps and GEDI correction slightly degraded segmentation results.Global validation reported user’s accuracy of 0.88 ± 0.006, producer’s accuracy of 0.82 ± 0.008, and IOU of 0.77 ± 0.006.
  • Generalization to aerial imagery: On out-of-domain NEON aerial imagery, the SSL model nearly doubled ResUNet’s R2, although its predictions were blurrier and showed higher bias than with satellite imagery.The reported MAE was 3.0 versus 2.7, while R2 was 0.55 versus 0.70 for aerial and satellite inputs, respectively.

5. Discussion

The proposed approach combines ALS-supervised high-resolution canopy-height modeling with GEDI-based rescaling, enabling sub-GEDI-scale variability mapping and generalization to aerial imagery. The authors also identify geographic, imagery, and tall-tree limitations.

  • 5. Discussion: The method directly models Maxar imagery against ALS data, then uses a GEDI-based model to rescale the high-resolution canopy-height map.This differs from approaches that use ALS only for validation.
  • 5. Discussion: Sub-GEDI-scale canopy-height variability can be mapped, sometimes at per-tree level outside dense, closed-canopy forests.
  • 5. Discussion: SSL representations generalize from satellite to aerial imagery, with aerial-image accuracy described as similar to in-domain satellite performance despite blurrier outputs.
  • 5. Discussion: Validation remains constrained by limited ALS geographic coverage and possible prediction saturation for very tall trees above 30 m.
  • 5. Discussion: Input variation in view angle, sun angle, acquisition timing, dates, and aerosols limits map consistency and currently constrains tree-height change detection.
  • 5. Discussion: The approach is compared with Wagner et al. (2023) using aerial imagery and obtains a similar result despite changed input scale and omission of infrared data.

6. Conclusion

The study presents jurisdiction-scale, high-resolution canopy-height maps from VHR Maxar imagery trained on aerial lidar and calibrated with GEDI. The maps capture small-scale tree-structure variability and are intended to support forest and carbon monitoring.

  • 6. Conclusion: The study produces jurisdiction-scale canopy-height maps from VHR Maxar imagery using aerial-lidar training, GEDI calibration, self-supervised learning, and vision transformers.
  • 6. Conclusion: The presented data better capture tree-structure variability at small spatial scales than existing canopy-height maps.
  • 6. Conclusion: The maps can improve monitoring of forest degradation, restoration, and forest carbon dynamics within the demonstrated scope.

Appendix A. Data used in training / calibration / validation

The appendix documents the ALS datasets and site partitions used for training, calibration, and set-aside validation, including non-overlapping validation regions and a tile-level split at one training site.

  • Appendix A. Data used in training / calibration / validation: Training and calibration used NEON sites including SJER, SOAP, TEAK, BART, DSNY, HARV, JERC, OSBS, DELA, and GRSM, among others.
  • Appendix A. Data used in training / calibration / validation: The NEON test set contains CUPE, REDB, WLOU, HOPB, and GUAN.
  • Appendix A. Data used in training / calibration / validation: The complete list of CHM files used during training and calibration is provided through an AWS Open Data CSV.
  • Appendix A. Data used in training / calibration / validation: The appendix distinguishes training, calibration, and set-aside validation ALS data, with validation datasets drawn from non-overlapping geographic regions.

Appendix B.1. GEDI dataset

The GEDI dataset supplies RH95 supervision paired with Maxar image patches, while simulation supports interpreting GEDI outputs as canopy-height-map percentiles. The data are split into training, calibration, and set-aside validation subsets.

  • Appendix B.1. GEDI dataset: GEDI is an ISS full-waveform lidar instrument with approximately 25 m ground footprints, and the filtered dataset contains 1.3 × 10^9 measurements.
  • Appendix B.1. GEDI dataset: GEDI RH95 measurements were paired with 128 × 128 pixel Maxar patches covering approximately 76 × 76 m.
  • Appendix B.1. GEDI dataset: The GEDI model output is assumed to approximate the sample canopy-height map's 95th percentile over each 128 × 128 pixel sample.
  • Appendix B.1. GEDI dataset: Simulated GEDI RH95 values correlate strongly with canopy-height-map p95, with R2 = 0.88 across the diverse ecosystems represented by 40 NEON sites.
  • Appendix B.1. GEDI dataset: GEDI measurements were split into 80/10/10% train, calibration, and set-aside validation subsets, using log-inverse weighting across 1 m RH95 bins.
  • Appendix B.1. GEDI dataset: Satellite metadata included location, off-nadir view angle, sun angle, and terrain slope; slope was set to zero during inference.

Appendix C. Details on different metrics

The appendix defines block R2, mean error, and an edge error score for evaluating canopy-height predictions, with additional figures illustrating GEDI-related comparisons and edge scoring.

  • Block R2: Block R2 averages ground-truth and predicted canopy heights within 50×50-pixel blocks before computing classical R2.The block averages form 5×5 arrays reshaped into 1×25 vectors.
  • GEDI Comparisons: Figure B.19 computes 95th-percentile canopy heights within weighted Gaussians with σ = 12.5m to approximate the GEDI beam width.It correlates these ALS-map percentiles with simulated GEDI RH95 values from the same maps.
  • Mean Error (ME): Mean error, also called bias, summarizes the average pixel-wise difference between predicted and ground-truth canopy height.The test-set size is represented by |D|.
  • Edge Error Score: The edge error score compares prediction and ground-truth image gradients to measure map sharpness beyond accuracy metrics that can reward blurry predictions.The metric uses Sobel edge detectors on predicted and ground-truth canopy-height maps.
  • GEDI Comparisons: Figure B.20 compares terrain slope with GEDI RH95 and indicates that GEDI height estimates do not fall below terrain change within the beam.The dashed line represents terrain height change based on GEDI beam radius and terrain slope.
  • Edge Error Score: A worked example reports edge error scores of 0.66 for ResUNet and 0.55 for the SSL model.The scores are computed from differences between predicted and ground-truth edge maps.

Appendix D. Architecture and Training Details

The appendix describes the ViT–DPT architecture, self-supervised pretraining, decoder training, implementation details, and the substantially higher training carbon footprint of the largest model.

  • Implementation: The implementation uses PyTorch 1.9.0 with CUDA 10.2.
  • Training: Self-supervised pretraining used dataset-specific normalization computed from 3.5M satellite images and took slightly less than three days on two 8-GPU Volta systems.The pretraining followed the referenced SSL procedure with modified image normalization parameters.
  • Training: The decoder was trained with the frozen SSL encoder for 140k steps using cosine learning-rate scheduling, 12k warmup iterations, and batch size 16.The DPT decoder contains 34.2M parameters.
  • Carbon Footprint: 1.81T of CO2 was estimated for ViT huge training, two orders of magnitude above ResUNet training, while inference energy and emissions were within the same order of magnitude.The ViT estimate used 16,896 GPU hours and 4,647 kWh; ResUNet training was estimated at 16.1 kg of CO2.
  • Architecture: The ViT encoder processes image patches with self-attention, providing a global receptive field while preserving fine-grained feature maps without intermediate downsampling.The DPT decoder reassembles multistage tokens, fuses feature maps, and produces dense canopy-height predictions.

Appendix E. Alternate Loss Function Ablation

The ablation compares SSL canopy-height models across pretraining-data sizes and loss functions, finding a benefit from more pretraining data and only modest loss-function differences.

  • Loss Functions: L1-loss results were slightly worse, while L2 and sigloss showed no strong performance difference.The comparison covers models trained with L1 or Sigloss and reports the L2 comparison in the discussion.
  • Evaluation: Table E.7 reports canopy-height prediction accuracy metrics across loss functions and architectural variants, including classification-output and linear-layer alternatives.The table omits CA Brande from display for visibility but includes it in the average.

Appendix F. Residuals with respect to the GEDI RH95

The appendix examines residuals between block canopy-height predictions and GEDI RH95 measurements, with residuals plotted relative to GEDI RH95.

  • Residual Analysis: Figure F.22 displays differences between the p95 of block-model canopy-height predictions and measured GEDI RH95 values.Residuals are shown with respect to the GEDI RH95 measurement.

Appendix G. Normalization for inference on aerial imagery

The method normalizes aerial imagery to improve inference by the self-supervised model trained on satellite imagery. This normalization is applied only to that SSL model because applying it to ResUNet deteriorated results.

  • Aerial-image histogram normalization improves inference performance for the SSL model trained only on satellite imagery.The RGB channels are normalized toward the satellite-image distribution, making color balance more similar.
  • The normalization is applied only to the SSL model trained on satellite imagery.
  • Applying the normalization to ResUNet deteriorated the results.
Loading 2304.07213v3…