PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-Efficient ConvNeXt Pretraining for High-Resolution Sentinel-1 GRD SAR Representation Learning

Yuki Natsume, Sang-Ho Yun, Chai Kiat Yeo

arXiv:2610.21993v1cs.CV

TL;DR

Labeled S1GRD benchmarks remain scarce, and existing masked-pretraining approaches often demand substantial data and compute. SparK-SAR adapts sparse convolutional masked pretraining to S1GRD; its finetuned encoder improves over non-pretrained baselines across four tasks and exceeds a frozen ViT baseline on three.

  • Problem

    Labeled S1GRD benchmarks remain scarce, while existing ViT-based masked-image-modeling approaches typically require large pretraining corpora and heavy compute.

    Source: §1 · p. 2

  • Method

    SparK-SAR combines SparK and ConvNeXt with lightweight reconstruction decoders to pretrain S1GRD representations under constrained compute.

    Source: §1 · p. 3, §6 · p. 24

  • Results

    The 27.8M-parameter encoder improves over a non-pretrained baseline by +4.7 pp, +5.0 pp, +5.8 pp, and +1.4 pp on EuroSAT-S1, BigEarthNet-S1, DFC2020-S1, and Flood-S1, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Classification performance largely saturates after 10–25% of the pretraining corpus, whereas dense prediction continues improving significantly up to 25% and more gradually through 50%.

    Source: §6 · p. 24

    The reported frozen Copernicus-FM score on BigEarthNet-S1 may partly reflect geographic-coordinate inputs rather than purely learned SAR representations.

    Source: §4.3 · p. 12

Abstract

from arXiv · show

Labeled Sentinel-1 Ground Range Detected (S1GRD) datasets remain scarce, limiting supervised deep learning for land-surface remote sensing, while self-supervised pretraining for this domain has been dominated by Vision Transformer (ViT)-based masked image modeling requiring substantial data and compute. We propose SparK-SAR, a sparse convolutional masked pretraining framework built on SparK and ConvNeXt, designed to exploit convolutional inductive biases for S1GRD imagery under constrained compute. SparK-SAR makes three core technical contributions: (i) ConvNeXt-RS, a domain-adapted encoder that halves the stem stride (4x to 2x) to produce 14x14 feature maps at stride 16x, providing 4x more spatial tokens at identical parameter count; (ii) a lightweight ConvNeXt-512 reconstruction decoder with PixelShuffle upsampling that reduces pretraining VRAM by 7.5-8.5% per GPU relative to a UNet-128 baseline while matching its accuracy; and (iii) an empirical characterization of FP16 mixed-precision pretraining instability in the standard SparK UNet-768 decoder on S1GRD data, complemented by a finding that finetuning with layer-wise learning rate decay (LLRD) improves downstream performance of pretrained encoders. On four Copernicus-Bench S1GRD tasks, our finetuned 27.8M-parameter ConvNeXt-RS encoder improves over a non-pretrained baseline by +4.7, +5.0, +5.8, and +1.4 pp on EuroSAT-S1, BigEarthNet-S1, DFC2020-S1, and Flood-S1 respectively, and outperforms the Copernicus-FM ViT (139M parameters, frozen) on three of four tasks at roughly 5x fewer parameters. A pretraining data-scaling study reveals task-dependent data efficiency: classification saturates with as little as 10-25% of the SSL4EO-S12 v1.1 corpus, while dense prediction continues to improve significantly up to 25-50%. The study also includes a CKA representation analysis and a geographic de-collision experiment.

1. Introduction

Labeled S1GRD benchmarks are scarce, while existing masked-pretraining approaches often rely on compute-intensive ViT encoders. SparK-SAR adapts sparse convolutional masked pretraining to S1GRD and evaluates its transfer, data efficiency, label efficiency, and geographic robustness.

1. Introduction

  • The standard SparK UNet-768 decoder produces NaN losses under FP16 mixed-precision pretraining on S1GRD, motivating decoder redesign.ConvNeXt-512 reduces pretraining VRAM by 7.5–8.5% per GPU relative to UNet-128 while matching its accuracy.Source: §1 · pp. 2–3
  • SparK-SAR combines SparK and ConvNeXt for sparse masked pretraining on S1GRD imagery under constrained compute.The study asks whether this approach transfers, how much pretraining and labeled data it requires, and whether gains persist after geographic overlap is removed.Source: §1 · pp. 2–3
  • Layer-wise learning rate decay improves downstream performance and reduces across-seed variance for the UNet-128-pretrained encoder, but not for a non-pretrained encoder.This finetuning recipe is presented as a contribution alongside the pretraining adaptations.Source: §1 · p. 3
  • +4.7 pp, +5.0 pp, +5.8 pp, and +1.4 pp over a non-pretrained baseline on EuroSAT-S1, BigEarthNet-S1, DFC2020-S1, and Flood-S1, respectively.These results come from the 27.8M-parameter encoder evaluated across four Copernicus-Bench S1GRD tasks.Source: §1 · p. 3
  • The pretraining data-scaling study evaluates eight fractions of the SSL4EO-S12 v1.1 corpus with formal statistical analysis.It examines task-dependent saturation in SAR amplitude representation learning.Source: §1 · p. 3
  • The CKA analysis tracks how encoder representations evolve across pretraining-data fractions.The analysis complements the data-scaling study by examining representation changes as data quantity varies.Source: §1 · p. 3
  • Pretraining yields statistically significant gains over random initialization at every tested label fraction from 1–50%, with 10–50% of labeled data matching full-label random-initialization performance across all four tasks.The comparison spans the four downstream tasks evaluated in the paper.Source: §1 · p. 3
  • After all overlapping pretraining tiles were removed, downstream accuracy changed by less than 0.1 percentage points on every task.The geographic de-collision experiment re-pretrained SparK-SAR from scratch on the de-collided corpus.Source: §1 · p. 4

2. Related Work

Prior masked-image modeling work includes both transformer-based reconstruction methods and sparse convolutional approaches, while remote-sensing SSL spans contrastive, generative, and multimodal methods. SAR-specific studies have explored land-cover and target-recognition tasks, but the cited work identifies a gap in sparse convolutional masked pretraining for S1GRD across diverse downstream SAR tasks.

2.1. Masked Image Modeling

  • MAE reconstructs pixels from visible ViT patches, SimMIM applies reconstruction to hierarchical Swin Transformers, and I-JEPA predicts latent representations of masked regions.I-JEPA's latent prediction objective emphasizes semantic structure over low-level texture.Source: §2.1 · p. 4
  • SparK demonstrates sparse masked modeling on hierarchical CNNs without transformers, while the cited MIM methods are described as data- and compute-hungry.MCMAE is a hybrid convolution-transformer approach, and MMEarth applies CNN-based multimodal MIM to electro-optical Sentinel-2 imagery.Source: §2.1 · p. 4

2.2. Self-Supervised Learning for Remote Sensing

  • Remote-sensing SSL includes contrastive and generative paradigms, but optical-statistics augmentation pipelines used by SeCo and GASSL do not transfer cleanly to SAR backscatter.Other cited methods include electro-optical MIM adaptations and multimodal Sentinel-1/electro-optical models, often evaluated with frozen encoders.Source: §2.2 · pp. 4–5

2.3. SAR-Specific Self-Supervised Learning

  • SAR-specific SSL includes Sentinel-1 MAE for vegetation-cover and land-cover tasks, while SAR-JEPA and SARATR-X focus on automatic target recognition.The cited target-recognition models are trained and evaluated exclusively on ATR tasks, limiting their generalizability to land-cover and change detection.Source: §2.3 · p. 5
  • The paper positions SparK-SAR among the first sparse convolutional masked-pretraining methods for S1GRD with transfer demonstrated across diverse SAR tasks under constrained compute.Earlier cited SAR self-distillation methods showed modest gains and poor geographic generalization without further domain-specific adaptation.Source: §2.3 · p. 5

3. Materials and Methods

SparK-SAR adapts sparse masked pretraining to S1GRD imagery with a higher-resolution ConvNeXt-RS encoder and lightweight reconstruction decoders. Pretraining uses seasonal SSL4EO-S12 imagery, while downstream evaluation finetunes the encoder end-to-end under task-specific protocols.

3.1. ConvNeXt-RS Encoder

  • ConvNeXt-RS uses a 2×2 stem stride to produce 14×14 feature maps and four times as many spatial tokens as standard ConvNeXt-Tiny, at the same 27.8M parameters.Its 16×16-pixel token coverage retains recognizable Stage 4 scene structure, unlike the standard encoder’s 7×7 output.Source: §3.1 · p. 6

3.2. Reconstruction Decoders

  • UNet-768 produces FP16 NaN losses at epochs 5–12, whereas UNet-128 and ConvNeXt-512 remain stable at lr=10−3.ConvNeXt-512 uses 7.5–8.5% less VRAM per GPU than UNet-128 without sacrificing downstream accuracy; UNet-768 is stable at lr=10−4 but performs worse than UNet-128 at that rate.Source: §3.2 · pp. 7–8

3.3. Pretraining Setup

  • Pretraining uses 243,968 globally distributed SSL4EO-S12 v1.1 locations, randomly sampling one of four seasonal acquisitions and applying 60% masking with an L2 reconstruction objective.On 4×V100 GPUs, UNet-128 and ConvNeXt-512 pretraining take 8.9 and 7.9 hours, respectively.Source: §3.3 · pp. 8–9

3.4. Finetuning Protocol

  • The encoder is finetuned end-to-end for each task, using benchmark-specific heads and resolutions rather than a single unified input size.DFC2020-S1 remains at 256×256, while EuroSAT-S1 and BigEarthNet-S1 are upsampled to 224×224; the pretraining and finetuning runs use different framework versions, whose effect was not isolated.Source: §3.4 · pp. 9–10
  • LLRD improves mean accuracy on all four tasks and reduces across-seed variance for the UNet-128-pretrained encoder, but does not improve the non-pretrained encoder.The recipe uses a 0.75 layer-decay factor and lowers non-pretrained Flood-S1 mIoU by 0.8 pp.Source: §3.4 · p. 10

4. Results

Across four Copernicus-Bench S1GRD tasks, end-to-end-finetuned SparK-SAR improves over a non-pretrained baseline and beats the frozen Copernicus-FM ViT on three tasks. The section also describes the evaluation datasets, comparison baselines, and the distinct outcomes under frozen evaluation.

4.1. Datasets

  • The four evaluation tasks use EuroSAT-S1, BigEarthNet-S1, DFC2020-S1, and Flood-S1, each derived from an established source dataset.They span classification, multi-label classification, semantic segmentation, and change detection.Source: §4.1 · p. 12

4.2. Baselines

  • The comparisons include a matched non-pretrained ConvNeXt-RS control, a standard-resolution ConvNeXt, and four published frozen ViT results.The random-initialization control isolates the SAR-specific pretraining contribution because no ImageNet-pretrained weights are available for the modified ConvNeXt-RS input and stride.Source: §4.2 · p. 12

4.3. Main Results

  • Under frozen evaluation, ConvNeXt-RS trails frozen ViT baselines on classification but remains competitive on DFC2020-S1 and exceeds them on Flood-S1.The table caption cautions that frozen and end-to-end-finetuned protocols are not directly comparable; linear probing also disadvantages convolutional architectures.Source: §4.3 · pp. 12–13, Table 6 · p. 13
  • +4.7 pp, +5.0 pp, +5.8 pp, and +1.4 pp over the non-pretrained baseline on EuroSAT-S1, BigEarthNet-S1, DFC2020-S1, and Flood-S1, respectively.The 27.8M-parameter finetuned encoder also beats the 139M-parameter frozen Copernicus-FM ViT on EuroSAT-S1, DFC2020-S1, and Flood-S1.Source: §4.3 · p. 12, Table 5 · p. 13
  • SparK-SAR beats CROMA and DOFA on every task and SoftCon on three tasks; BigEarthNet-S1 is the exception for SoftCon.On BigEarthNet-S1, SparK-SAR scores 77.03±0.40 versus 78.7±0.0% for SoftCon.Source: §4.3 · pp. 12–13

4.4. Ablation Studies

The ablations identify higher output resolution as especially useful for dense prediction, find comparable accuracy with a lighter decoder, and report qualified gains from LLRD. Flood-S1 task-head changes yield small gains, while possible ceiling factors remain unisolated.

4.4. Ablation Studies

  • +1.4 pp on DFC2020-S1 and +0.8 pp on Flood-S1 from the higher-resolution encoder, compared with +0.3 pp on EuroSAT-S1.The matched-learning-rate ablation supports resolution, rather than learning rate, as the primary contributor; the table compares 7×7 and 14×14 encoders.Source: §4.4 · p. 14, Table 7 · p. 14

4.4.2. Decoder Design and E

  • 7.5–8.5% less pretraining VRAM per GPU with ConvNeXt-512 than UNet-128, with comparable accuracy across all four tasks.The paired comparison found statistically indistinguishable performance (p=0.17); it is controlled at matched learning rate and precision, not across the full decoder-capacity range.Source: §4.4.2 · p. 14

4.4.3. LLRD Finetuning

  • LLRD improves mean performance and reduces variance across all four tasks, with corrected statistical significance on BigEarthNet-S1 and DFC2020-S1 only.For ConvNeXt-512, the observed LLRD direction was similar but not statistically significant (p=0.21); tested decay factors support retaining λ = 0.75, which was indistinguishable from the best tested value.Source: §4.4.3 · pp. 14–15, Table 8 · p. 15, Table 9 · p. 15

4.4.4. Flood-S1 Ceiling Analysis

  • At most about 0.4 pp separates Flood-S1 task-head configurations, small beside the +1.4 pp pretraining gain.The ablation does not isolate suspected ceiling factors: 3,000 samples, 85.3% non-flood pixels, and a mismatch between single-image MIM and temporal reasoning.Source: §4.4.4 · pp. 15–16, Table 10 · p. 16

4.5. Pretraining Data Scaling

The pretraining data-scaling study tests subsets of SSL4EO-S12 v1.1 and finds task-dependent saturation: classification saturates at lower data fractions than semantic segmentation. Non-significant differences are not proof of equivalence.

4.5. Pretraining Data Scaling

  • The study pretrains on random SSL4EO-S12 v1.1 subsets from 1% to 100% (2,439 to 243,968 tiles), with a fixed subset seed and five finetuning seeds.The 0% random-initialization condition is also finetuned with LLRD, unlike the primary no-pretraining baseline in Table 5.Source: §4.5 · p. 16
  • EuroSAT-S1 and Flood-S1 show no significant difference from the 100% baseline at fractions ≥10%, BigEarthNet-S1 is effectively consistent with saturation at ≥25%, and DFC2020-S1 improves significantly up to 25%.DFC2020-S1 saturates between 25–50%; BigEarthNet-S1 has a significant non-monotonic dip at 50%, amid high inter-seed variance.Source: §4.5 · p. 17, Table 11 · p. 17

4.6. Finetuning Label Efficiency

Pretraining yields statistically significant gains over random initialization across tested label fractions, and pretrained models match full-label random-initialization results with fewer labels. The representation analysis relates CKA similarity to pretraining data, while noting an unexplained Flood-S1 dip.

4.6. Finetuning Label Efficiency

  • Pretraining significantly outperforms random initialization at every tested label fraction and task, and matches the 100%-label random-initialization result using 10–25% of labels on EuroSAT-S1 and DFC2020-S1, and 25–50% on BigEarthNet-S1 and Flood-S1.Gains were significant after correction across 20 comparisons, including at 30 labeled Flood-S1 samples.Source: §4.6 · p. 18, Table 12 · p. 18

4.7. Representation Analysis

  • CKA compares encoder representations on shared inputs; the analysis uses standardized Stage 3 post-GAP features across pretraining fractions and test sets.Stage 3 avoids collapsing the spatial structure of the 14×14 Stage 4 output into a scalar summary.Source: §4.7 · p. 18
  • 0.64–0.70 to 0.83–0.88: CKA rises near-monotonically for EuroSAT-S1, BigEarthNet-S1, and DFC2020-S1 as pretraining data increases.This broadly follows the downstream performance saturation pattern.Source: §4.7 · p. 18, Figure 7 · p. 19
  • Flood-S1 CKA dips to 0.733 and 0.726 at 25% and 50% pretraining data, while downstream accuracy remains monotonic within noise across those fractions.The analysis reports no confident explanation for the dip.Source: §4.7 · pp. 18–19
  • ≈0.65–0.70 CKA at 0% pretraining reflects ConvNeXt's spatially correlated features even with random initialization.Source: §4.7 · p. 19

4.8. Geographic Data Overlap Analysis

The geographic de-collision experiment tests whether proximity between pretraining tiles and evaluation samples inflates reported gains. Removing all identified overlapping tiles changed every task’s results by less than 0.1 percentage points, though DFC2020-S1 could not be directly assessed for overlap.

4.8. Geographic Data Overlap Analysis

  • The proximity screen identified 10.9% of EuroSAT-S1, 4.3% of BigEarthNet-S1, and 10.1% of Flood-S1 test samples, while implicating 0.58% of pretraining tiles overall.DFC2020-S1 was excluded from the direct screen because its release lacks coordinate metadata.Source: §4.8 · p. 19
  • Removing 1,406 geographically proximate pretraining tiles changed results on all four tasks by less than 0.1 percentage points, within one standard deviation.The de-collided corpus retained 242,562 tiles; the experiment used five seeds per task.Source: §4.8 · p. 19, Table 13 · p. 20
  • DFC2020-S1 remains unassessed for direct geographic overlap because its release lacks coordinate metadata; unchanged performance after de-collision provides indirect reassurance only.Source: §4.8 · pp. 19–20

4.9. Qualitative Results

Across representative DFC2020-S1 scenes, pretrained configurations generally preserve class boundaries and avoid some errors made by the non-pretrained encoder, although all models show limitations. Flood-S1 scenes likewise show advantages for pretrained configurations in some cases, while all three recover the main permanent-water channel.

4.9. Qualitative Results

  • In the DFC2020-S1 shrubland-and-cropland scene, pretrained ConvNeXt-RS recovers more shrubland and barren land than the standard or non-pretrained configurations.The non-pretrained encoder collapses much of the shrubland into cropland and suppresses barren regions.Source: §4.9 · p. 20, Figure 8 · p. 21
  • In the DFC2020-S1 cropland scene, both the standard and non-pretrained encoders make spurious predictions over barren land, while pretrained ConvNeXt-RS instead extends cropland into that region.Source: §4.9 · p. 20, Figure 8 · p. 21
  • In the first Flood-S1 scene, the pretrained ConvNeXt-RS avoids spurious top-right flood predictions made by the other configurations and better separates permanent water from flood water.Source: §4.9 · pp. 20–21
  • For scattered Flood-S1 patches, the non-pretrained encoder misses more flooded area, while all configurations recover the meandering permanent-water channel at comparable accuracy.Source: §4.9 · p. 21
  • Across DFC2020-S1 scenes, the non-pretrained encoder more often confuses flood water with permanent water and dry land, while pretrained configurations preserve some class boundaries more closely.All configurations recover the overall layout in the heterogeneous scene, but predictions smooth fine-grained boundaries.Source: §4.9 · pp. 20–21, Figure 8 · p. 21

5. Discussion

The discussion argues that SAR-specific architecture and training choices can yield useful representations with modest compute and data, while benefits vary by task. It also identifies limits of single-image pretraining for temporal change detection and reports mixed results for alternative objectives and inputs.

5. Discussion

  • The higher-resolution ConvNeXt-RS encoder is the most consequential of the three design changes for dense prediction, with its largest gains on segmentation and change detection.Halving the stem stride increases spatial tokens without adding parameters.Source: §5 · pp. 21–22
  • UNet-128 and ConvNeXt-512 have statistically indistinguishable downstream accuracy (p=0.17), while ConvNeXt-512 saves 7.5–8.5% VRAM per GPU.LLRD improves mean performance and reduces variance across four tasks for pretrained encoders, with significance on two; it did not help a non-pretrained encoder.Source: §5 · p. 22
  • Classification and change detection saturate with 10–25% of the corpus, while DFC2020-S1 segmentation improves through 50%.The study reports significant segmentation gains up to 25% and more gradual gains through 50%.Source: §5 · pp. 22–23
  • CKA representations approach the fully pretrained configuration as data increases, broadly mirroring the downstream performance plateau; random initialization also has a non-trivial CKA baseline.Source: §5 · p. 23
  • The single-image pretraining objective provides little temporal structure for change detection, while tested task-head choices alter Flood-S1 mIoU by at most about 0.4 pp.The authors therefore point to multi-temporal pretraining as a more promising direction than further task-head engineering.Source: §5 · p. 23
  • On BigEarthNet-S1, SparK-SAR scores 77.03±0.40 versus 77.9±0.0% for Copernicus-FM and 78.7±0.0% for SoftCon under frozen evaluation.The authors offer metadata access and geographically clustered labels as a plausible, not demonstrated, explanation for Copernicus-FM’s advantage.Source: §5 · pp. 23–24
  • At 1% labeled Flood-S1 samples, SparK-SAR gains +6.26 pp mIoU over random initialization, a setting the authors relate to rapid-response mapping with scarce event annotations.Source: §5 · p. 24
  • A 27.8M-parameter encoder, pretrained on four V100 GPUs in under nine hours, beats frozen 139M-parameter Copernicus-FM on three of four tasks.The authors frame this as a compute-efficient alternative to larger multimodal ViT foundation models.Source: §5 · p. 24

6. Conclusions

The paper adapts masked self-supervised pretraining to Sentinel-1 GRD with a high-resolution ConvNeXt-RS encoder and lightweight decoders, then evaluates transfer across four downstream tasks. Results show task-dependent data scaling and competitive performance for a compact encoder, while identifying temporal structure as a boundary of single-image pretraining.

6. Conclusions

  • SparK-SAR combines a 14×14-feature-map ConvNeXt-RS encoder with lightweight reconstruction decoders to address mixed-precision pretraining difficulties on SAR data.The conclusion describes the framework as designed for a limited computational budget.Source: §6 · p. 24
  • The 27.8M-parameter encoder exceeds frozen ViT-based encoders with up to 5× more parameters on three of four evaluated tasks.Source: §6 · p. 24
  • Classification performance largely saturates after 10–25% of the pretraining corpus, whereas dense prediction improves significantly up to 25% and more gradually through 50%.As data increases, learned representations progressively approach those from full pretraining, following trends consistent with downstream results.Source: §6 · pp. 24–25
  • The authors conclude that compact CNNs adapted to SAR can provide useful representations with lower computational requirements, while single-image pretraining is limited for temporal change analysis.Future work will investigate multi-temporal objectives and extensions incorporating interferometric SAR phase and coherence.Source: §6 · p. 25

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

Claude assisted with drafting and language editing across several manuscript sections, while the authors reviewed the output and retain responsibility for the published content.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

Related papers

Loading 2610.21993v1…