Pixel-Level Transformers in Remote Sensing: A Canopy Height Case Study

1 Introduction

• Global-scale canopy-height mapping requires substantial inference resources, while medium-resolution satellite pixels can represent distinct local semantics.A Sentinel-2 pixel typically covers 10 × 10 meters.Source: §1 · p. 1
• The study investigates pixel-level ViTs for dense remote-sensing regression, with patch size affecting prediction quality, memory, and runtime.Smaller patches increase token count, making attention more expensive.Source: §1 · pp. 1–2
• The authors provide guidance on selecting ViT patch size and model dimension for dense regression on medium-resolution satellite imagery.The introduction names both as crucial hyperparameters.Source: §1 · p. 2
• The proposed Transformer configuration outperforms other well-known models on canopy-height prediction.The introduction identifies this as a contribution but does not provide the comparison values here.Source: §1 · p. 2
• The authors identify shortcomings in traditional quantitative evaluation of canopy-height predictions trained on noisy labels.The introduction lists this as a contribution without specifying the shortcomings here.Source: §1 · p. 2

2 Background

• The background section introduces related ViT work, attention-complexity variants, and the canopy-height prediction task.It frames these topics as context for the methods that follow.Source: §2 · p. 2

2.1 Vision Transformers

• Vision Transformers use hierarchical processing and reduced patch sizes; Swin and PVT differ in attention and in how they combine adjacent tokens between stages.Dense prediction commonly uses an encoder-decoder, with upsampling to restore spatial resolution.Source: §2.1 · pp. 2–3

2.2 Pixel-Level Vision Transformers

• Pixel-level ViTs treat each pixel as a token, avoiding patchification’s potential information-loss bottleneck, while smaller patches generally reduce test loss.Training on tens to hundreds of thousands of tokens per image remains challenging.Source: §2.2 · p. 3

2.3 Attention Mechanism Variants

• Attention variants reduce vanilla attention’s quadratic token complexity through approaches including sparse interactions and low-rank approximations.Examples include local or shifted windows, axis-based attention, clustering, and low-rank methods.Source: §2.3 · p. 3

2.4 Canopy Height Prediction

• Canopy-height prediction estimates the tallest tree’s height per image pixel, using satellite imagery and labels such as GEDI measurements or ALS data.GEDI provides global coverage between 51.6°N and 51.6°S, while ALS offers denser, higher-resolution measurements but limited spatial and temporal coverage; the experiments use OpenCanopy ALS data.Source: §2.4 · p. 3

3.1 Notation

• The notation defines Q, K, and V as learned linear projections of the input token embeddings.Source: §3.1 · p. 4

3.2 Vanilla Attention

• Vanilla attention compares every query with every key, forming an N×N attention matrix whose rows weight values; its runtime is O(N^2 d_m) and memory is O(N^2).Source: §3.2 · p. 4

3.3 Efficient Attention

• Efficient attention computes K^T V before combining it with queries, adjusts softmax, and approximates vanilla attention with lower runtime and memory complexity.Source: §3.3 · p. 4
• Its global context aggregates values by key feature maps, while each query weights those maps to produce its output.Source: §3.3 · pp. 4–5

3.4 Windowed Attention

• Windowed attention limits each token to its local window, achieving linear complexity in token count; alternating shifted windows allow information to cross window boundaries.Source: §3.4 · p. 5

3.5 Flash Attention

• FlashAttention preserves exact attention while hardware tiling reduces data copying between GPU memory levels; FlashAttention-2 reports a 2× speedup over FlashAttention.Source: §3.5 · p. 5

4.1 Mix Transformer

• MiT uses four stages of multiscale features, Mix-FFN depth-wise convolution, and overlapped PatchEmbed layers; token interaction with neighbors removes the need for positional encoding.Source: §4.1 · p. 5

4.2 U-MixFormer

• U-MixFormer replaces decoder self-attention with mix-attention: a single-scale feature supplies queries, while a multiscale feature map supplies keys and values.Source: §4.2 · p. 5
• The decoder builds keys and values from a pyramid of encoder features, substituting available decoder features, then pools, aligns, and concatenates the maps before projection.Source: §4.2 · pp. 5–6

4 Model Architecture

• The architecture pairs an adapted four-stage MiT encoder with a four-block U-MixFormer decoder, aligning and concatenating decoder features before an FFN generates the prediction.Source: §4 · p. 5, Figure 5 · p. 6

4.3 Architectural Adaptations

• Experiments vary patch size across {1, 2, 4, 8}, modify the first PatchEmbed accordingly, add PatchExpand for resolution recovery, and use efficient attention by default.Source: §4.3 · p. 6

5.1 Datasets

• Training used Sentinel-2 inputs from Europe and France; Europe supplied training and validation data, while non-overlapping France patches were reserved for final evaluation.Models were trained five times per configuration; accuracy was assessed with MAE>5m and R2, and cost with training-step runtime and peak VRAM.Source: §5.1 · p. 7, §5.2 · p. 7

5.3.1 Impact of Patch Size & Model Dimension.

• Patch sizes above 1 produced grid-like artifacts whose intensity increased with patch size; smaller patches also reduced error variance for small trees and absolute error for taller trees.Fine details such as narrow streams and forest paths showed especially pronounced perceptual differences.Source: §5.3.1 · p. 8

5.3.4 Discussion.

• Training time and required memory scale roughly by a factor of 1/P^2; when training is highly resource-constrained and a small performance loss is acceptable, P=2 Transformer-based or CNN-based models remain strong alternatives.These are relative comparisons within a controlled experimental setting, and reported metrics should not be directly compared with existing canopy height products.Source: §5.3.1 · p. 8, §5.3.4 · p. 10

5.3.1 Impact of Patch Size & Model Dimension.

• On France ALS labels, MAE>5m improved 25.1%–37.4% relative to GEDI-label evaluation; smaller patches generally improved performance at fixed model dimension, with diminishing gains from larger dimensions.The best configurations were (P=1, dm=96), (P=2, dm=192), (P=4, dm=192), and (P=8, dm=192).Source: Figure 7 · p. 7, §5.3.1 · p. 8, Figure 9 · p. 8

5.3.2 Impact of Attention Mechanism.

• Efficient attention in both encoder and decoder offered the best reported performance-efficiency trade-off: ALS MAE>5m averaged 3.84 m at about 78 ms runtime.Global attention outperformed shifted-window attention; FlashAttention took more than twice as long as efficient attention, while vanilla attention ran out of memory at this input size.Source: §5.3.2 · p. 9, Figure 11 · p. 9

5.3.3 Comparison with Baseline Models.

• The pixel-level Transformer achieved the strongest MAE>5m and R2, while U-Net-based models remained competitive and trained faster; high-patch Transformers were weaker.The comparison reports metrics in Table 1 and example predictions in Figure 12.Source: Table 1 · p. 10, §5.3.3 · p. 10, Figure 12 · p. 10

5.3.4 Discussion.

• The study varied patch size, model dimension, and attention, leaving other design choices such as model depth and decoder design unexplored.A stronger decoder might reconstruct spatial dimensions lost during patchification.Source: §5.3.4 · p. 10

• Appropriate model dimensions and efficient attention approximations yield strong performance from resource-hungry pixel-level Transformers.The experiments vary patch size, model dimension, and attention mechanism.Source: §6 · p. 10
• Models trained on noisy GEDI-derived labels learn fine details that can be evaluated against higher-quality ALS labels in selected regions.Source: §6 · p. 10
• Evaluation using GEDI labels alone gives an incomplete picture of model performance.Source: §6 · p. 10
• Pixel-level architectures are more intuitive because they avoid arbitrary image chunks and produce predictions at the input resolution.Source: §6 · p. 10
• The authors believe similar observations may apply to biomass estimation, soil moisture mapping, and crop yield forecasting.Source: §6 · p. 10

• The Europe dataset figure maps training patches in light grey and validation patches in dark grey, with France validation patches in black.Source: Figure 13 · p. 13
• Smaller-patch models underestimate taller trees less severely on both the Europe GEDI and France ALS labels.Source: Figure 14 · p. 13
• Figure 15 compares residual differences across versus within grid cells, using cell size P and a ratio of the two quantities.Source: Figure 15 · p. 13
• Table 2 describes feature-map height and width across patch sizes and stages for a 256 × 256 input, with channels doubling after each stage.Source: Table 2 · p. 13
• The P=8, dm=192 model has not learned to predict trees taller than 30 m.Source: Figure 17 · p. 14
• Smaller-patch models handle missing-data vertical lines better and produce more accurate predictions in the qualitative comparison.Source: Figure 18 · p. 14