PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Sensitivity Analysis of Tessera Embeddings

Julia Guerrero-Viu, Alex López-Cifuentes, Ignacio Pérez-Villar, Fabio Pacifici

arXiv:2608.27175v1cs.CV

TL;DR

LULC mapping needs timely updates, but Earth Observation foundation models commonly rely on full-year observations, leaving reduced temporal support insufficiently studied. The paper freezes Tessera, recomputes embeddings over windows from one year to one day, and evaluates them across three benchmarks against from-scratch models. Embedding value depends on task and label availability, while shorter windows cause gradual, class-dependent degradation and preserve useful information even at a single day.

  • Problem

    Operational LULC mapping needs frequent updates, while Tessera and related models assume full-year temporal support whose systematic reduction has not been analyzed.

  • Method

    The study freezes Tessera, recomputes embeddings over one-year-to-single-day windows, and evaluates linear probes and UNet heads on LUCAS, DynamicEarthNet, and PASTIS-R.

  • Results

    Embedding value is task-dependent and label-efficient: it is strongest for phenology-driven mapping, while shorter-window degradation is gradual and class-dependent.

  • Takeaways & Limitations

    Temporal coverage can be treated as a tunable cost rather than a fixed prerequisite for operational LULC mapping and near-real-time applications.

  • Takeaways & Limitations

    The evaluation changes inference-time windows for a frozen encoder pretrained on full-year sequences with fixed-length, repetition-padded inputs, so short-window pretraining may behave differently.

Abstract

from arXiv · show

Many Earth Observation applications need land-use/land-cover maps that are both precise and frequently updated, yet the strongest Earth Observation foundation models build their embeddings from a full year of observations. We present a controlled study of the temporal sensitivity of Tessera, one of these leading foundation models, for land-use/land-cover mapping. Keeping the encoder frozen, we recompute its embeddings over varying observation windows, from a full year down to a single day. We use them as inputs to a linear probe and a UNet segmentation head, benchmarking both of them against from-scratch networks on LUCAS, DynamicEarthNet, and PASTIS-R datasets. We show that the value of the embeddings is task-dependent. Where classes are separated by phenology, as for the crop types of PASTIS-R, they reach a mean Intersection-over-Union of $58.3$, about $46\%$ above the best from-scratch model. Where classes are temporally stable (e.g., forests in DynamicEarthNet and LUCAS), embedding-based and from-scratch models match only under full supervision. On both datasets, Tessera embeddings remain markedly more label-efficient. Degradation under shorter temporal windows is gradual and class-dependent. Contracting the window from one year to one month costs $39\%$ of the segmentation accuracy on PASTIS-R but only $5\%$ on DynamicEarthNet. Single-day embeddings still classify land cover in LUCAS at $3.4$ times the chance level. Our study shows that temporal coverage is therefore a tunable cost rather than a fixed prerequisite, opening regimes such as near-real-time mapping and faster land-use/land-cover refresh cycles.

I. Introduction

Earth Observation needs LULC maps that are both accurate and refreshed at application-appropriate frequencies, while existing foundation models generally rely on full-year satellite observations. This study tests whether Tessera embeddings retain value with reduced temporal coverage and limited labels.

  • Annual global land-cover products are refreshed at most yearly and released months after the mapped period, limiting timely LULC updates.
  • Tessera encodes full-year Sentinel-1/2 time series into dense 10-meter per-pixel embeddings suited to phenological land-cover differences.Its per-pixel design applies temporal attention to each pixel’s radar and spectral series.
  • The study asks how robust Tessera embeddings are when the inference-time temporal observation window is reduced.
  • The controlled framework freezes the encoder and recomputes embeddings from full-year to single-day windows across LUCAS, PASTIS-R, and DynamicEarthNet.Linear classification probes and UNet segmentation heads evaluate the resulting embeddings.
  • Embedding value is task-dependent: it is decisive for phenology-separated classes but marginal for temporally stable classes under full supervision.
  • Tessera embeddings are markedly more label-efficient than from-scratch models, while single-day windows retain useful semantic information above chance.The reported findings motivate reduced-coverage regimes such as near-real-time mapping and faster LULC refresh cycles.

B. Temporal Modeling of Satellite Image Time Series

Temporal information is especially important for crop-type mapping because classes may differ by phenology rather than single-date spectra. The study evaluates Tessera across complementary sparse, dense, general land-cover, and crop-type benchmarks on a common 10-meter embedding grid.

  • Crop types can remain spectrally similar for much of the year and become distinguishable through phenological patterns.
  • Most land-cover benchmarks omit either the temporal axis or full multispectral inputs, whereas only a few expose genuine Sentinel-1/2 time series.
  • The analysis uses LUCAS for sparse in-situ polygons, DynamicEarthNet for dense general land-cover segmentation, and PASTIS-R for phenologically driven crop-type mapping.
  • LUCAS Dataset: LUCAS provides 15,000 georeferenced polygons across eight top-level classes, with precise survey dates but sparse, imbalanced, and potentially mixed 10-meter pixels.
  • DynamicEarthNet Dataset: DynamicEarthNet contains 75 worldwide areas with daily 3-meter Planet mosaics over two years and monthly per-pixel annotations for seven classes.
  • PASTIS-R Dataset: PASTIS-R contains 2,433 128×128 patches observed over one agricultural year, annotated for 18 crop types plus background.

B. Temporal Sensitivity Evaluation Framework

The framework varies only the temporal support used to construct Tessera embeddings while keeping the encoder and downstream evaluation fixed. It restricts Sentinel acquisitions around a reference date, adapts them to Tessera’s fixed input length, and evaluates each window with classification and segmentation heads.

  • The framework varies the observation window while keeping the encoder, downstream probe, and data splits fixed.The window ranges from one year to one day across three LULC benchmarks.
  • Calendar time is measured in days, and each pixel has Sentinel-1 and Sentinel-2 acquisition pairs with observations and timestamps.
  • The frozen Tessera encoder maps the Sentinel time series to a 128-dimensional embedding and requires a fixed-length input distinct from the observation window.
  • Temporal Window Construction: The window is parameterized by duration W in days and centered on a reference date t.For example, W=30 represents one month and W=365 represents one year.
  • Temporal Window Construction: Only acquisitions whose timestamps fall inside the selected window are supplied to the encoder.
  • Temporal Window Construction: When the window contains too many acquisitions, the method subsamples; when it contains too few, it repeats acquisitions to meet Tessera’s fixed input length.
  • Temporal Window Construction: Figure 1 evaluates each temporal window by encoding its observations into dense per-pixel maps, then training a linear probe and UNet segmentation head.
  • Temporal Window Construction: Changing W alters both the number of distinct acquisitions and their temporal span, while W=1Y recovers the original model.The model always ingests n=40 observations and produces d=128-dimensional embeddings.

2) Reference Dates

Reference dates are aligned to annotation timestamps, and observation windows are centered on the nearest acquisition. The evaluation varies temporal support across benchmarks while keeping other factors controlled.

  • Reference dates use the acquisition closest to each annotation timestamp and center each observation window on that date.
  • LUCAS and DynamicEarthNet provide one label per sample, with daily and monthly annotation timestamps respectively.
  • PASTIS-R uses one annual label per patch, supervising non-overlapping temporal composites tiled across the observation period.
  • Dense benchmarks use 1M, 3M, 6M, and 1Y windows, while LUCAS additionally includes 1W and 1D windows.
  • The study decodes frozen Tessera embeddings with both logistic-regression probes and a compact UNet segmentation head, while reproducing the reported Tessera training recipe for dense datasets.

A. Implementation Details

Implementation details standardize training, preprocessing, optimization, and baseline inputs so comparisons primarily reflect the model inputs and temporal representations.

  • Training setup: All segmentation models use the same training recipe, including Adam, a 10^-4 learning rate, 10^-4 weight decay, cosine annealing, and 80 epochs.
  • Training setup: The segmentation objective is class-weighted pixel-wise cross-entropy over labeled pixels, with void or unlabeled pixels ignored.
  • Training setup: Class weights use the inverse square root of class frequency and are normalized so present classes have mean weight one.
  • Preprocessing and initialization: Inputs are standardized using training statistics, trainable layers use Kaiming uniform initialization, and training and evaluation use mixed precision.
  • Seeds and reporting: UNet results average five random seeds, whereas the deterministic logistic-regression probe is reported from one run without stochastic initialization.
  • Baseline models: From-scratch baselines use the same UNet architecture and temporal windows as Tessera-UNet, but receive Sentinel composites rather than frozen embeddings.

3) Evaluation Metrics

Evaluation uses pixel-level classification and segmentation metrics across temporal windows to test how Tessera embeddings retain information relative to from-scratch models. Results show strong task- and class-dependent temporal sensitivity, with phenology-driven classes benefiting most from long temporal context.

  • 3) Evaluation Metrics: The benchmarks use overall accuracy, balanced accuracy, macro-F1, and mean Intersection-over-Union, all reported as percentages.
  • 3) Evaluation Metrics: Table I reports DEN and PASTIS-R segmentation performance for 1M, 3M, 6M, and 1Y windows, with five-run means and standard deviations for UNets.
  • Main Results: 58.3 mIoU at 1Y makes TESSERA-UNet 46% better than the best from-scratch model on PASTIS-R.
  • Main Results: At full supervision on DEN, embeddings remain within 2.5% of the best from-scratch UNet, with 43.1 versus 44.2 mIoU at 6M.
  • Temporal Sensitivity: Shrinking from 1Y to 1M reduces TESSERA-UNet mIoU by 39% on PASTIS-R, but only 5% on DEN.
  • Per-Class Temporal Sensitivity: Phenology-driven classes degrade steeply with shorter windows, whereas structurally or spectrally distinct classes remain comparatively stable.
  • Per-Class Temporal Sensitivity: Some DEN and LUCAS classes remain weak at every window because under-representation and sparse polygons, rather than temporal coverage, limit performance.
  • Temporal Resolution on LUCAS: LUCAS balanced accuracy falls from 50.8% at 1Y to 36.8% at 1D, while 6M remains near yearly performance at 49.8%.

2) Comparison With the State of the Art

On PASTIS-R, TESSERA-UNet reaches 58.3 mIoU and exceeds published Tessera and AlphaEarth results, while no directly comparable published DEN results are available.

  • 58.3 mIoU is achieved by TESSERA-UNet on PASTIS-R, improving over published Tessera and AlphaEarth results.The reported gains are 15% over Tessera’s 50.68 and 14% over AlphaEarth’s 51.08.
  • 48.5 mIoU from TESSERA-LP is competitive with published head-based results despite using a simpler decoder.
  • Published Tessera results are not reproduced exactly because the original implementation details are not fully specified.
  • DEN has no directly comparable published results because this study reformulates it as six-class land-cover segmentation with labels resampled from 3 m to 10 m.

3) Label Efficiency

At a fixed one-year window, Tessera embeddings are substantially more label-efficient, especially on PASTIS-R, while V1.1 provides no meaningful downstream advantage over V1.

  • Label efficiency: At 1% labels, TESSERA-UNet reaches approximately 24 mIoU on PASTIS-R, roughly 50% above the best from-scratch UNet.
  • Label efficiency: TESSERA-LP matches or exceeds all fully supervised from-scratch UNets across the entire PASTIS-R label range.
  • Label efficiency: On DEN, Tessera models reach approximately 38–42 mIoU at 1% labels, while from-scratch UNets begin around 26–35 and catch up only near 30–100%.
  • Embedding versions: V1.1 and V1 track each other across the PASTIS-R label-efficiency curve, so the update adds no label-efficiency advantage.
  • Ablation studies: On PASTIS-R, the baseline configuration reaches 57.2 mIoU, while additional ingredients raise it to 58.3, a 1.9% gain.
  • Embedding versions: V1.1 differs from V1 by at most 0.3 points for both the TESSERA-UNet head and the linear probe.
  • Embedding versions: Linear class separability is essentially unchanged at 6M and 1Y, with V1.1 changes of −8.8% and −1.9%.

C. Qualitative Results

Qualitative predictions mirror the quantitative contrast: Tessera produces smoother DEN maps and identifies crop types on PASTIS-R, while from-scratch models better preserve parcel geometry but confuse phenology.

  • Figure setup: Figures 4 and 5 show randomly selected test samples with RGB composites, ground truth, and predictions from all compared models.
  • DEN: On DEN, Tessera predictions are smoother and less speckled than those of from-scratch UNets.From-scratch RGB variants show water–forest/soil and bare-soil–agriculture confusions, while Tessera can occasionally over-generalize.
  • PASTIS-R: On PASTIS-R, from-scratch UNets recover parcel geometry but frequently assign incorrect crop types.TESSERA-LP more often identifies the correct crop while exhibiting salt-and-pepper noise.
  • Temporal evolution: As the PASTIS-R window grows from 1M to 1Y, crop assignments refine and linear-probe noise fades.
  • Temporal evolution: On DEN, predictions are already close to their final form at 1M and change little thereafter.

V. Discussion

Tessera’s value depends on whether classes are separated by phenology or spectral appearance, while shorter temporal windows impose gradual, class-dependent costs. The study supports treating temporal coverage as a tunable operational cost, but its evaluation remains bounded by a frozen full-year-pretrained encoder and three benchmarks.

  • Task dependence: PASTIS-R reaches 58.3 mIoU at 1Y, 46% above the best from-scratch model, whereas DynamicEarthNet approaches from-scratch performance because its classes are spectrally distinct.TESSERA-UNet wins at every PASTIS-R window and metric; on DEN it is within approximately 2.5% of the strongest from-scratch UNet at comparable windows.
  • Label efficiency: Embedding advantages are strongest when labels are scarce, while DynamicEarthNet models reach parity with from-scratch UNets only at label saturation.With 1% of labels, TESSERA-UNet achieves approximately 24 mIoU on PASTIS-R, about 50% above the best from-scratch UNet.
  • Temporal sensitivity: A one-year-to-one-month window reduction costs 39% mIoU on PASTIS-R but only 5% on DynamicEarthNet.LUCAS declines from 50.8% balanced accuracy at 1Y to 44.6% at 1M and 36.8% from a single day, still 3.4 times its 12.5% chance level.
  • Class dependence: Temporal degradation is localized: DEN agriculture loses about half its 1Y F1 by 1M, while forest and water are nearly window-invariant.On PASTIS-R, potatoes, sorghum, and winter triticale retain only 13–22% of their 1Y F1 at 1M, compared with 84% for meadow.
  • Class dependence: Weak classes across all windows are limited by under-representation and small polygons near Sentinel’s 10 m resolution rather than temporal coverage.This pattern is reported for DEN wetland, LUCAS water, and wetlands.
  • Limitations: The framework varies only inference-time windows for a frozen encoder pretrained on full-year sequences, so its degradation does not establish how short-window pretraining would behave.The study also covers three benchmarks and one foundation model, although the framework is model-agnostic.

Appendix Reproducibility Details

The appendix standardizes Sentinel-1/2 acquisition, preprocessing, temporal compositing, model inputs, and evaluation across datasets. From-scratch UNets use the same acquisitions and windows as TESSERA-UNet, isolating the input representation.

  • Data preparation: Sentinel-1 and Sentinel-2 observations are downloaded through Planetary Computer, harmonized to a common 10 m grid, quality-filtered, and stored in standardized per-region stacks.The same preprocessing is applied across LUCAS 2022, DynamicEarthNet, and PASTIS-R.
  • Baselines: From-scratch UNets consume per-window composites built from the same acquisitions and temporal windows as Tessera embeddings, so only the input representation differs.This construction is intended to isolate the contribution of the input representation.
  • Modalities: Sentinel-2 retains ten bands at 10 m after cloud filtering and resampling, while Sentinel-1 RTC provides VV and VH channels on the same grid.Sentinel-2 scenes with cloud cover below 15% are retained; Sentinel-1 channels are converted to decibels.
  • Temporal compositing: Temporal windows of 1M, 3M, 6M, and 1Y are centered on the embedding reference date, with Sentinel-2 composites formed by per-pixel temporal medians after SCL masking.The windows correspond to 30, 91, 182, and 365 days.
  • Model inputs: UNet-S2 uses ten Sentinel-2 bands, UNet-S1S2 adds two Sentinel-1 channels, and both are standardized using train-split statistics.Training uses random crops and flips; evaluation runs on full images padded to a multiple of 32.
  • Evaluation: Performance is evaluated with overall accuracy, balanced accuracy, macro F1, and mIoU using pixel-level confusion matrices accumulated over test-set classes.All metrics are reported as percentages.

2) Balanced Accuracy

Balanced accuracy averages per-class recall so that each class contributes equally despite imbalanced pixel frequencies. It complements macro F1 and mIoU by focusing on recall rather than overlap or precision–recall balance.

  • 2) Balanced Accuracy: Balanced accuracy averages per-class recall, giving each class equal weight regardless of pixel frequency.It is designed to compensate for class imbalance in land-cover and crop-type maps.
  • 2) Balanced Accuracy: Macro F1 also treats classes equally but combines precision and recall, thereby penalizing over-prediction of majority classes.This distinguishes it from balanced accuracy’s recall-only averaging.
  • 2) Balanced Accuracy: mIoU averages per-class Jaccard overlap and jointly captures false positives and false negatives for semantic segmentation.The average is computed over the benchmark-defined class set.
Loading 2608.27175v1…