PaperlayerView exact revision on arXiv

Source-linked AI summary

From Sentinel-2 to 1-m Imagery: Real-World $\times 10$ Cross-Sensor Super-Resolution via Visual Autoregressive Modeling

Tailai Chen, Yuluxin Fu, Jie Deng, Shu Luo, Yanan Wu, Danfeng Hong

arXiv:2610.34742v1cs.CV

TL;DR

Real-world ×10 reconstruction must learn from independent cross-sensor observations without a known degradation operator, unlike common synthetic SR settings. CrossVAR predicts the 1-m latent coarse-to-fine from Sentinel-2, refines the quantization residual without injected noise, and reconstructs 1600 × 1600 scenes by aligned windows at decoder training size. 27.6008 dB PSNR, 0.7725 SSIM, 0.1451 LPIPS, and 7.1486 FID are CrossVAR's best values among compared methods on 3,015 test scenes.

  • Problem

    Real-world ×10 reconstruction must learn from independent cross-sensor observations without a known degradation operator, unlike common synthetic SR settings.

    Source: §I · p. 1, C. Geospatial and Radiometric… · p. 5

  • Method

    CrossVAR predicts the 1-m latent coarse-to-fine from Sentinel-2, refines the quantization residual without injected noise, and reconstructs 1600 × 1600 scenes by aligned windows at decoder training size.

    Source: §I · p. 2, A. Preliminaries · p. 5

  • Results

    27.6008 dB PSNR, 0.7725 SSIM, 0.1451 LPIPS, and 7.1486 FID are CrossVAR's best values among compared methods on 3,015 test scenes.

    Source: C. Quantitative Results · p. 8

  • Takeaways & Limitations

    CrossVAR provides a strong autoregressive baseline for large-scale cross-sensor reconstruction.

    Source: §VII · p. 12

    YREB-SR covers the Yangtze River Economic Belt and uses a random, non-geographically blocked split, leaving transfer to other climates, landscapes, and spectral bands untested.

    Source: C. Limitations · p. 12

Abstract

from arXiv · show

Reconstructing 1-m imagery from Sentinel-2 combines two difficulties. The input and the reference are observed by different satellites, which differ in spatial resolution, spectral response, radiometry, acquisition time, and local geometry, so no synthetic degradation operator links them. The required magnification of ten in each spatial dimension also far exceeds the factors of two to four at which remote sensing super-resolution is usually developed. We study this real-world $\times 10$ problem using YREB-SR, which pairs 2022 Sentinel-2 RGB composites with 2022 Jilin-1 RGB references resampled to a 1-m grid over the Yangtze River Economic Belt. The benchmark contains 11,559 geographically matched scene pairs, with 6,409, 2,135, and 3,015 assigned to training, validation, and testing, respectively. We develop CrossVAR, a visual autoregressive model that predicts the 1-m latent as a nine-scale hierarchy conditioned on the Sentinel-2 input and estimates the remaining quantization residual with a diffusion refiner run without injected noise. Complete 1600 $\times$ 1600 scenes are reconstructed in overlapping windows that start on the Sentinel-2 pixel grid, as every training patch does, and every window is decoded at the size on which the decoder was trained before the windows are blended with Gaussian weights. On all 3,015 test scenes, CrossVAR achieves 27.60 dB PSNR, 0.7725 SSIM, 0.1451 LPIPS, and 7.15 FID, the best value of each metric among the compared methods. These results establish a strong autoregressive baseline for large-scale cross-sensor reconstruction. To facilitate reproducibility, the source code and implementation are publicly available at https://github.com/SHANOAL/RSSR.

I. INTRODUCTION

Sentinel-2-to-1-m reconstruction combines independent cross-sensor observations with a ×10 magnification, while complete scenes exceed training crop size. CrossVAR addresses these challenges with coarse-to-fine latent prediction, deterministic residual refinement, and aligned window decoding.

I. INTRODUCTION

  • Independent satellites differ in sensor, radiometry, and acquisition conditions, so the Sentinel-2 input is not a degraded copy of its geographically matched reference.Exact pixel-wise correspondence is not assumed.Source: §I · p. 1
  • At ×10, each Sentinel-2 pixel expands to 100 output pixels, and much 1-m structure lies below the input sampling.The learned priors used to add detail may produce sharp structure unsupported by the observation, while cross-sensor differences make the reference imperfect.Source: §I · p. 2
  • CrossVAR predicts the 1-m latent as nine coarse-to-fine code maps conditioned on Sentinel-2, then refines the residual without injected noise.Aligned inference windows start on the input grid and are decoded at the decoder's training size.Source: §I · p. 2
  • YREB-SR contains 11,559 real cross-sensor ×10 Sentinel-2/Jilin-1 scene pairs with a fixed scene-level partition; five model families are compared using common RGB metrics on 3,015 complete scenes.The comparison reports fidelity, perceptual quality, and efficiency for every method.Source: §I · p. 2

II. RELATED WORK

Real-world SR datasets pair independent acquisitions, while many methods target synthetic degradation or moderate magnification. CrossVAR applies visual autoregressive prediction to real Sentinel-2/Jilin-1 pairs at ×10.

A. Super-Resolution Datasets

  • YREB-SR pairs a single Sentinel-2 composite with a 1-m Jilin-1 reference at ×10 and evaluates complete scenes.Earlier real-world benchmarks include cross-sensor and multi-acquisition pairs, while L8S2PS uses a factor of three.Source: A. Super-Resolution Datasets · p. 2

B. Super-Resolution

  • Reconstruction networks, arbitrary-scale continuous representations, and generative models address fidelity, output-scale flexibility, and perceptually plausible detail, respectively.Examples include EDSR, SwinIR, TTST, LIIF, Thera, adversarial models, and diffusion methods.Source: B. Super-Resolution · pp. 2–3

C. Visual Autoregressive Modeling

  • CrossVAR applies coarse-to-fine next-scale prediction to real Sentinel-2/Jilin-1 pairs at ×10, with residual refinement without injected noise and input-aligned complete-scene windows.Earlier VAR-based approaches include VARSR, which conditions on LR prefix tokens and estimates quantization residuals with a diffusion refiner.Source: C. Visual Autoregressive Modeling · p. 3

III. YREB-SR DATASET CONSTRUCTION

YREB-SR samples geographically diverse locations across the Yangtze River Economic Belt and pairs Sentinel-2 RGB composites with Jilin-1 RGB references. The observations are matched by footprint but retain real cross-sensor and temporal differences.

A. Study Area and Sampling Design

  • Stratified sampling across the YREB's upstream, midstream, and downstream regions yields scenes spanning built-up areas as well as rural and natural land covers.A 3-km minimum distance between sampling centers prevents direct scene-footprint overlap.Source: A. Study Area and Sampling Design · p. 3

1) Sentinel-2 Low-Resolution Images:

  • Each input is a single 160 × 160 Sentinel-2 natural-color image at native 10-m spacing, covering a nominal 1.6 × 1.6 km2 footprint.Other bands, temporal sequences, Sentinel-1 data, labels, and geographic metadata are not supplied to reconstruction models.Source: 1) Sentinel-2 Low-Resolution Images · p. 3

2) Jilin-1 High-Resolution Reference Images:

  • Jilin-1 references originate from 0.75-m RGB imagery acquired in 2022 and are resampled to a 1-m grid over the corresponding footprint.They are cross-sensor references rather than strictly simultaneous ground truth; acquisition dates and sensor and atmospheric conditions differ.Source: 2) Jilin-1 High-Resolution Reference… · p. 3

C. Geospatial and Radiometric Standardization

YREB-SR standardizes geographically matched Sentinel-2/Jilin-1 scenes while retaining their real sensor and radiometric differences. Scene-level partitions and complete-scene testing provide a shared evaluation protocol.

C. Geospatial and Radiometric Standardization

  • YREB-SR pairs independent satellite observations rather than synthetically degraded images, so sensor, radiometric, temporal, atmospheric, and local alignment differences cannot be represented by one deterministic degradation operator.Pairs match geographically at scene level, but exact pixel-wise correspondence is not assumed.Source: C. Geospatial and Radiometric… · p. 5

D. Dataset Partitioning and Patch Generation

  • The 11,559 scenes are partitioned into 6,409 training, 2,135 validation, and 3,015 test pairs, with mutually exclusive scene identities and no geographically blocked holdout.Test evaluation uses all 3,015 complete scenes without border removal or favorable-crop selection, under common spatial support across methods.Source: D. Dataset Partitioning and Patch… · p. 5
  • Training and validation scenes are split into aligned 40 × 40 input and 400 × 400 reference patches, while standardized pairs retain matching geographic bounds and transforms.The resulting benchmark differs from synthetic SR through its ×10 scale, independent cross-sensor inputs, and complete-scene evaluation.Source: C. Geospatial and Radiometric… · p. 3, D. Dataset Partitioning and Patch… · p. 5

IV. METHOD

CrossVAR learns to reconstruct a 1-m Jilin-1 reference from geographically matched Sentinel-2 windows rather than from a known synthetic degradation. It predicts a quantized latent representation conditioned on the Sentinel-2 window and decodes it to the output image.

A. Preliminaries

  • CrossVAR learns a 400 × 400 reconstruction from each 40 × 40 Sentinel-2 window, using a geographically matched Jilin-1 reference rather than a known degradation operator.The reference is not an exactly registered observation, so the mapping is learned from paired data.Source: A. Preliminaries · p. 5
  • A frozen VQ autoencoder represents each reference window with nine coarse-to-fine code maps on grids from 1 × 1 to 25 × 25.The encoder reduces each spatial dimension by 16, and all scales share one codebook.Source: A. Preliminaries · pp. 5–6

B. Condition Encoding

  • The condition encoder resizes the Sentinel-2 window to 400 × 400 and produces one token at each of the 625 positions on the 25 × 25 latent grid.Each token represents Sentinel-2 content around the corresponding latent-grid location.Source: B. Condition Encoding · p. 6

C. Next-Scale Prediction

CrossVAR predicts latent code maps in an ordered scale hierarchy, with each scale conditioned on the input and coarser maps. A deterministic residual refiner estimates the quantization error left after the final scale.

C. Next-Scale Prediction

  • The transformer input combines a prefix with scale-specific maps formed by downsampling accumulated coarser latents, projecting them, and adding learned scale embeddings.The prefix receives learned position embeddings, and layer-normalization scale and shift come from the start token.Source: C. Next-Scale Prediction · p. 6
  • Two-dimensional rotary position encoding assigns tokens coordinates on a shared 25 × 25 grid, preserving relative positions across scales and condition tokens.For an s × s map, position (i, j) receives coordinates (25i/s, 25j/s).Source: C. Next-Scale Prediction · p. 6
  • Each scale’s code map depends on the input and coarser maps, while all positions within that scale are predicted in parallel.At inference, the transformer selects the most probable code at each position and uses the accumulated latent for the next scale.Source: C. Next-Scale Prediction · p. 6

D. Deterministic Residual Refinement

  • Because finite codebook embeddings cannot exactly match the continuous latent, a per-position diffusion refiner estimates the residual left by the selected final-scale codes.Its denoiser uses residual MLP blocks modulated by diffusion step and condition; the estimated residual completes the predicted latent.Source: D. Deterministic Residual Refinement · p. 6
  • Setting the reverse-diffusion noise scale τ to zero makes each step follow its predicted mean, avoiding sampling variance without extra training or computation.The initial state remains random, but its effect on the output is reported as negligible.Source: D. Deterministic Residual Refinement · p. 6

E. Training Objective

The condition encoder, next-scale transformer, and residual refiner are trained jointly while the VQ autoencoder remains frozen. The objective combines code prediction and decoded-image losses.

E. Training Objective

  • Joint training uses L = LCE + λd Ldiff + L1 + λs LSSIM, while keeping the VQ autoencoder frozen.The code cross-entropy is averaged over positions within each scale, with scales weighted in proportion to their side length.Source: E. Training Objective · p. 6
  • Weighting code cross-entropy by scale side length prevents the final code map, which holds about half of 1,265 positions, from outweighing coarser maps.The coarser maps set the tone and layout of the window.Source: E. Training Objective · p. 6
  • L1 and SSIM compare the reference with a decoded latent using reference codes at the first K − 1 scales and a straight-through estimate at the last scale.The forward pass uses the most probable final-scale code, while its gradient follows the probability-weighted mean of the most probable codes’ embeddings, training last-scale logits through the frozen decoder.Source: E. Training Objective · p. 7

F. Aligned Window Decoding of Complete Scenes

Aligned window decoding reconstructs complete scenes by keeping each window on the input grid and decoding it at the decoder’s training size before blending. Evaluation uses the fixed 3,015-scene test set and common RGB metrics.

F. Aligned Window Decoding of Complete Scenes

  • Windows start on the input grid at 80-pixel intervals, avoiding input-to-latent offsets absent from training.For the complete scene, starts are 0, 320, 640, 960, and 1200 pixels along each axis.Source: F. Aligned Window Decoding of Complete… · p. 7
  • Each window is decoded separately at its training size, then blended using positive Gaussian weights.This avoids applying decoder normalization and attention across a larger latent field than the 25 × 25 latents used in training.Source: F. Aligned Window Decoding of Complete… · p. 7

A. Implementation and Evaluation

B. Comparison Methods

CrossVAR leads the compared methods on the full-scene ×10 benchmark and retains advantages across scene complexity and regions. The section also compares baseline families and traces reconstruction quality across scales and qualitative examples.

B. Comparison Methods

  • The comparison covers reconstruction networks, a continuous image field, adversarial and diffusion models, and a released natural-image autoregressive model.EDSR-baseline, SwinIR, TTST, Real-ESRGAN+, and EDiffSR are trained on the training partition with outputs adapted to ×10; TexADiff and VARSR use released weights.Source: B. Comparison Methods · p. 8

C. Quantitative Results

  • CrossVAR has higher PSNR and SSIM and lower LPIPS than every retrained baseline on all 3,015 test scenes.Against Thera, the median PSNR gain is 2.30 dB, with a 5th percentile of 1.39 dB and a minimum of 0.09 dB.Source: C. Quantitative Results · p. 8, Table IV · p. 9

D. Spectral, Complexity, and Regional Analysis

  • Reconstruction networks and Thera retain 10.6 to 14.0 dB less power than the reference at 0.05–0.15 cycles per meter, while CrossVAR retains 1.8 dB less.CrossVAR’s mean gradient magnitude reaches 0.74 of the reference, versus 0.77 for Satlas-SR and 0.28 to 0.34 for reconstruction networks and Thera.Source: D. Spectral, Complexity, and Regional… · p. 8, Figure 4 · p. 9
  • CrossVAR stays most accurate across reference-texture quartiles, with its PSNR gain over Thera narrowing from 2.52 dB in the smoothest to 2.00 dB in the most textured quartile.Its gain over Thera is positive across the basin, averaging 1.91 dB in the upper reaches, 2.36 dB in the middle reaches, and 2.49 dB in the lower reaches.Source: D. Spectral, Complexity, and Regional… · p. 8, Figure 4 · p. 9

E. Coarse-to-Fine Reconstruction

  • The first six code maps raise PSNR from 18.55 to 21.54 dB; the 18×18 and 25 × 25 maps add fine structure and raise it to 25.63 dB.Deterministic residual refinement adds a further 1.78 dB and sharpens textures in all four examples.Source: E. Coarse-to-Fine Reconstruction · p. 8

F. Qualitative Results

  • In the urban example, CrossVAR separates buildings and keeps narrow field roads continuous; in the water example, it renders pond levees as thin continuous lines close to the reference.Across four scenes, reconstruction networks produce smooth textures, while adversarial and autoregressive models restore sharper local structure, consistent with spectra and lower LPIPS.Source: F. Qualitative Results · pp. 8–9

C. Quantitative Results

  • 27.6008 dB PSNR, 0.7725 SSIM, 0.1451 LPIPS, and 7.1486 FID: CrossVAR records the best value on all four metrics.It exceeds Thera by 2.30 dB PSNR and 0.087 SSIM, and lowers Satlas-SR’s LPIPS and FID by 33% and 57%, respectively.Source: C. Quantitative Results · p. 8, Table III · p. 9

G. Computational Cost

Across 3,015 test scenes, aligned window decoding and deterministic residual refinement together improve CrossVAR’s PSNR and FID. The ablations also quantify component effects and the cost-memory tradeoff of complete-scene inference.

G. Computational Cost

  • 2.77 s per scene is slower than one-pass networks but faster than VARSR and iterative diffusion; window-wise decoding cuts peak memory from 15,109 to 9,219 MiB at 6% more model time.The reported times are per complete scene.Source: G. Computational Cost · p. 9, Table V · p. 10

H. Ablation Study

  • On 3,015 test scenes, deterministic residual refinement and aligned window decoding together raise PSNR from 25.93 to 27.60 dB and lower FID from 10.29 to 7.15.Source: H. Ablation Study · p. 9, Table VI · p. 10
  • Different seeds agree to 52.5 dB PSNR with τ = 0, compared with 31.5 dB with τ = 1.The comparison is on 64 validation patches.Source: H. Ablation Study · pp. 9–10
  • Aligned starts add 0.99–1.15 dB across fusion rules, training-size decoding adds 0.31 dB, and RGB fusion is best on all four metrics.On 427 validation scenes, the selected assembly reaches 27.64 dB versus 26.32 dB for latent fusion with one-pass decoding.Source: H. Ablation Study · p. 10
  • A 0.2-input-pixel offset costs 0.6–0.7 dB in one axis, while offsets near pixel middles in both axes cost up to 2.2 dB; aligned starts are best in 87% of scenes.The comparison evaluates single windows on 427 validation scenes.Source: H. Ablation Study · p. 10

VI. DISCUSSION

At ×10, pixel-loss and adversarial approaches face a fidelity-perception tradeoff in unobserved frequencies. CrossVAR’s coarse-to-fine codes place fine structure relative to an input-constrained layout, while residual temperature controls the remaining balance.

A. Fidelity and Perceptual Quality at a Large Magnification

  • Pixel-loss models predict the conditional mean of unobserved frequencies, which carries little power, while adversarial models restore power but may misplace structures and lower PSNR.The passage illustrates the low-power conditional mean in Fig. 4(a).Source: A. Fidelity and Perceptual Quality at… · p. 11
  • CrossVAR selects fine structure as discrete codes conditioned on coarser maps, which fix the layout constrained by the input; this is a likely explanation for gains in both fidelity and perceptual quality.The authors frame the layout explanation as likely, not established causation.Source: A. Fidelity and Perceptual Quality at… · p. 11
  • With aligned windows, τ = 1 gives the lowest FID, whereas τ = 0 gives the best PSNR, SSIM, and LPIPS; the study uses τ = 0 because the benchmark measures reference agreement.Source: A. Fidelity and Perceptual Quality at… · p. 11

B. Window Alignment in Tiled Reconstruction

At ×10, tiled reconstruction benefits from input-grid-aligned windows and decoding at the training size. The discussion also identifies limits in reference-based evaluation, sub-pixel object interpretation, and the benchmark’s geographic scope.

B. Window Alignment in Tiled Reconstruction

  • At ×10, latent and input grids coincide every 80 output pixels; a 0.2-input-pixel offset already costs 0.6–0.7 dB, motivating grid-aligned starts and training-size decoding.At ×3 and ×6 the grids coincide every 48 output pixels; common factors ×2, ×4, ×8, and ×16 align at every latent-grid window.Source: B. Window Alignment in Tiled… · pp. 11–12, Figure 7 · p. 12

C. Limitations

  • Reference metrics include error from sensor and temporal differences that reconstruction cannot remove, because Sentinel-2 and Jilin-1 footprints are geographically matched rather than pixel-registered.Source: C. Limitations · p. 12
  • Structures below a Sentinel-2 pixel are prior-predicted, so outputs suit visual interpretation and layout-level mapping rather than measuring individual objects below 10 m.Source: C. Limitations · p. 12
  • YREB-SR covers the Yangtze River Economic Belt with a random scene-level split, not a geographically blocked holdout; transfer to other climates, landscapes, and spectral bands remains untested.The conclusion describes the evaluation as a fixed 3,015-scene test split.Source: C. Limitations · p. 12, §VII · p. 12

VII. CONCLUSION

  • CrossVAR’s advantage holds across scene complexity and region, and its fine structure retains power above Sentinel-2’s Nyquist frequency while placing it more accurately than adversarial models.Source: §VII · p. 12

Related papers

Loading 2610.34742v1…