TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysis

1. Introduction

• Cloud obstruction and irregular satellite revisits complicate EO time series, and compositing can flatten phenological dynamics and transient events needed by downstream models.Source: §1 · p. 1
• TESSERA uses Barlow Twins to make pixel-wise Sentinel-1/2 embeddings invariant to temporal sampling, with global shuffling and mix-based regulation addressing spatial correlation and severe sparsity.Ablations confirm that temporal sampling invariance, global shuffling, and mix-based regulation each contribute to robustness and generalization.Source: §1 · pp. 1–2
• The release includes global, annual, 10 m pixel-wise int8 embeddings, open weights and code, and lightweight adaptation heads for analysis-ready use.The embeddings can be paired with small MLPs or shallow convolutional models for high performance with minimal computation.Source: §1 · pp. 1–2
• Across diverse classification, segmentation, and regression benchmarks, TESSERA achieves state-of-the-art accuracy with strong label efficiency, including under low-label conditions.Its embeddings also degrade gracefully under heavy cloud cover.Source: Figure 1 · p. 2, §1 · p. 2

2.1. Foundation Models for Earth Observation

• Remote-sensing foundation models use contrastive, masked-image, global–local alignment, distilled, and multisensor-fusion approaches, yet most assume preprocessed inputs and task-specific backbone fine-tuning.These assumptions discard partly cloudy observations and can make downstream use compute- and label-intensive.Source: §2.1 · p. 2

2.2. Embedding-Based Foundation Models

• Embedding-as-data models offer task-agnostic features adapted with lightweight heads, but existing releases often lack full openness, global pixel-level granularity, or explicit handling of irregular sampling.TESSERA addresses these gaps with temporal sampling invariance, global shuffling and mix-based regulation, and FAIR global annual 10 m pixel-wise embeddings with open weights, code, and GeoTessera.Source: §2.2 · pp. 2–3, Figure 2 · p. 3

3.1. d-pixel: A Representation for Temporal Data

• A d-pixel collects all spectral channels across timesteps at one location, while a mask marks valid observations and the representation preserves temporal phenology under irregular sampling.Source: §3.1 · p. 3

3.2. Model Architecture

• TESSERA encodes optical and radar modalities separately; sequences are padded to L = 40 via sampling with replacement if needed, temporally pooled, fused into a 128-dimensional embedding, and quantized to 8-bit integers.The Sentinel-1 branch uses a 4-block Transformer followed by GRU pooling; QAT reduces storage by ∼4× with negligible performance loss.Source: §3.2 · p. 3, Figure 3 · p. 4

3.3. Pretraining

• Barlow Twins training aligns embeddings from two augmented views of the same input while reducing correlation between different embedding dimensions.The invariance term encourages diagonal cross-correlations toward 1, while redundancy reduction encourages off-diagonal terms toward 0.Source: §3.3 · p. 4
• Mix-based regularization blends one view with a shuffled view and penalizes deviations from linear interpolation between input and embedding spaces.The shuffled view is created by permuting examples along the batch dimension.Source: §3.3 · p. 4
• Global shuffling randomizes d-pixels across geographic tiles before batching to prevent spatial autocorrelation and improve generalization.Source: §3.3 · p. 5

3.4. Inference

• TESSERA generates annual, global 128-dimensional embeddings for every 10 m pixel using frozen dual encoders, with replacement sampling for pixels having fewer than 40 non-cloudy observations.The released maps can be used directly by downstream models without raw satellite data processing or running TESSERA.Source: §3.4 · p. 5

4.1. Experimental Setup

• The evaluation spans six classification, segmentation, and regression benchmarks, with two datasets per task whenever possible to test geographic transfer and fine-detail modeling.The authors add Austrian Crop parcel-level mapping and Borneo Canopy Height benchmarks; comparisons include remote-sensing foundation models and standard UNet and ViT baselines, evaluated at 1%, 30%, and 100% label ratios.Source: §4.1 · p. 5

4.2. Classification Results

• 77.96 F1 on TreeSatAI-TS with full supervision led the classification results; at 1% labels, TESSERA scored 60.58 versus AlphaEarth’s 52.79.On Austrian Crop, it scored 66.15 F1 at 1% labels, versus 37.22 for AlphaEarth and 32.74 for Presto.Source: §4.2 · p. 5, Table 1 · p. 6

4.3. Segmentation Results

• 53.12 mIoU on Austrian Crop segmentation established a new state of the art, exceeding AlphaEarth by 27.4 points and Presto by 19.0.At 1% labels, TESSERA scored 28.20 mIoU versus AlphaEarth’s 14.64.Source: Table 1 · p. 6, §4.3 · pp. 6–7
• On PASTIS-R, TESSERA scored 50.68 mIoU with full labels, ranking second, and 27.54 at 1% labels, ahead of AlphaEarth’s 27.12.A compact 30M-parameter UNet sufficed for the task, unlike some RSFMs requiring decoders above 100M parameters.Source: Table 1 · p. 6, §4.3 · p. 6

4.4. Regression Results

• TESSERA recorded the lowest Borneo Canopy Height RMSE at 12.2 m, outperforming AlphaEarth (16.1 m), Presto (17.9 m), and Skysense (15.6 m).On Biomassters, full-label RMSE was 27.43 t/ha, below AlphaEarth’s 29.59 t/ha and Skysense’s 30.78 t/ha.Source: Table 1 · p. 6, §4.4 · p. 7

• Removing global shuffling or mixup drops validation F1 by −9.2/−11.1 and RankMe by −0.116/−0.106; removing both causes the largest F1 fall, −14.7.Source: §5 · p. 7, Table 3 · p. 7
• TESSERA closely tracks the task-specific winner in Biomassters AGB regression while using significantly fewer labels.Source: Figure 5 · p. 7
• At a global scale, embeddings are robust to cloudiness with more than 70 observations per year for each orbit, but Macro-F1 drops sharply at ≤10 valid Sentinel-2 observations.Source: §5 · p. 7
• Pretraining improves Austrian Crop F1 by +33.5 over random initialization, while encoder finetuning brings no benefit and region-specific retraining offers negligible gains.Source: §5 · p. 7
• Validation F1 scales predictably with projector capacity, FLOPs, and parameters; TESSERA v1.0 sits near the compute–accuracy frontier.Source: Figure 7 · p. 8, §5 · p. 8
• Adding a spatial head such as a UNet consistently improves segmentation and regression, with larger gains at higher label budgets.Source: §5 · p. 8
• With L=40 temporal samples, repeated random draws produce tightly clustered results, and L=40 is the reported accuracy–efficiency sweet spot.Source: §5 · p. 8
• Shorter seasonal inference windows remain effective, while int8 embeddings retain performance with only a small drop and use ∼25% of fp32 storage.Source: §5 · p. 8

• TESSERA is a pixel-wise Earth-observation foundation model that learns robust representations from sparse, multimodal time series through temporal sampling invariance.Source: §6 · p. 8
• Extensive experiments report state-of-the-art accuracy across diverse tasks with strong label efficiency.Source: §6 · p. 8
• The paper releases analysis-ready global embeddings through GeoTessera as an Embeddings-as-Data resource intended to lower barriers for geospatial analysis.Source: §6 · p. 8

• The appendix organizes remote-sensing foundation models into visual foundation models and visual language models.Source: §A.1 · p. 15
• Visual foundation models include masked-autoencoder, scale-aware, multimodal-fusion, and temporal-analysis approaches [17, 35, 73, 67, 66, 92, 91].Source: §A.1 · p. 15
• Most visual foundation models require substantial preprocessing, computationally intensive finetuning, GPU resources, and technical expertise; AlphaEarth provides pre-generated maps but is not fully open-source.Source: §A.1 · p. 15
• Visual language models align satellite imagery with text for zero-shot classification and retrieval [54] [123] [71], while newer systems support conversational and multisensor comprehension [121] [57] [55].Source: §A.1 · p. 15
• The identified gap is a user-centric framework that provides powerful representations in an accessible, ready-to-use format, rather than simply more powerful models.Source: §A.1 · p. 15

• Separate Sentinel-1 and Sentinel-2 transformer encoders process their respective time series, using day-of-year positional encodings and four transformer blocks each.Source: §A.2 · p. 15
• Attention pooling aggregates each modality’s time steps, and an MLP fuses the resulting Sentinel-1 and Sentinel-2 embeddings.Source: §A.2 · p. 15
• A six-layer projector expands the fused representation for self-supervised loss computation; downstream tasks use the fusion MLP’s 128-dimensional output as the pixel embedding.Source: §A.2 · pp. 15–16
• The Barlow Twins training objective compares two views produced by independently sampling valid Sentinel-1 and Sentinel-2 observation dates for the same pixel.Source: §A.2 · p. 16
• The sampled views are valid but incomplete snapshots of a pixel’s temporal-spectral evolution, and training reconciles them rather than artificially distorting complete inputs.Source: §A.2 · p. 16

A.3. Pre-training Details

• The training set comprises approximately 800 million d-pixels from 3,012 globally distributed MGRS tiles selected from 2017–2024.Each d-pixel contains annual Sentinel-2 time series for 10 bands and Sentinel-1 VV and VH polarization series; spatial downsampling is by a factor of 400.Source: §A.3 · p. 17
• For each location, sparse sampling selects 40 valid dates per year; replacement sampling fills sequences when fewer than 40 cloud-free dates are available.This standardizes both modalities to 40 timesteps and serves as augmentation intended to build invariance to data gaps.Source: §A.3 · p. 17
• Global shuffling yields a markedly smoother, more stable loss curve than localized shuffling, which produces volatile loss and risks regional overfitting.The authors describe stability as fundamental for robust convergence and preventing overfitting to regional characteristics.Source: §A.3 · p. 17
• Global shuffling pools d-pixels across tiles to break spatial contiguity and mix geographic and environmental contexts within each batch.Tile-level spatial autocorrelation makes sequential or locally shuffled loading prone to geographic bias.Source: §A.3 · p. 17
• The pipeline globally shuffles the multi-terabyte pool, applies Barlow Twins augmentations, then serializes and streams chunks to assemble training batches.The custom pipeline was designed to manage the I/O demands of processing the data volume.Source: §A.3 · p. 17

A.4. Global Embedding Map Generation

• For every land pixel and year from 2017–2024, 40 sampled valid timesteps per modality are encoded and fused into a 128-dimensional embedding.The pretrained, frozen dual encoder is used without its projector; no spatial downsampling is performed when generating the maps.Source: §A.4 · p. 17
• The training set spans over 3,000 MGRS tiles distributed globally from 2017 to 2024, with hexagon colors encoding average valid observation days for Sentinel-1 and Sentinel-2.The bivariate map shows the combined observation availability across regions.Source: Figure 9 · p. 18
• During pretraining, the 16,384-dimensional projector output RankMe score quickly saturates, while the fused 128-dimensional representation becomes progressively richer and less redundant.The figure also tracks loss, learning-rate schedule, and Austrian crop-classification accuracy and F1 during pretraining.Source: Figure 10 · p. 18
• Each annual output has shape (H, W, 128), with H and W denoting dimensions of the global 10-meter grid.Source: §A.4 · p. 19

A.5. Spatial Context: Pixel-wise vs. Patch-based Approaches

• Pixel-wise independence avoids fixed spatial priors, preserving raw spectral signatures and leaving downstream performance driven by signal characteristics rather than preset structure.Source: §A.5 · p. 19
• Patch-based spatial priors can reduce representation robustness, while requiring clear patches can create large spatial voids in cloud-prone Borneo.The figure provides a quantitative comparison and a Borneo embedding visualization; the passage reports no numeric values.Source: §A.5 · p. 19, Figure 12 · p. 19
• TESSERA allows domain-specific spatial constraints to be incorporated downstream without compromising the foundational embeddings.Source: §A.5 · p. 19

A.6. Geographic Generalization

• Despite using only continuous day-of-year sinusoidal encoding as a weak temporal prior, TESSERA handles local seasonality driven primarily by spectral trends.The evaluation uses the CropHarvest Togo and Ethiopia Crop classification datasets.Source: §A.6 · p. 19
• TESSERA significantly outperforms CROMA and AlphaEarth on both the Togo and Ethiopia benchmarks in challenging zero-shot and low-data transfer scenarios.The table reports F1 scores for Togo and accuracy for Ethiopia.Source: §A.6 · p. 19, Table 5 · p. 20

A.7. Scaling with data and network size

• TESSERA performance improved predictably with model size and training computation, while v1.0 was positioned as a cost-effective balance.For Austrian crop classification, encoder performance was relatively insensitive to size; the selected encoder used 4 transformer layers and 4 attention heads. Deeper and wider projectors improved validation F1, but v1.0 used four hidden layers of width 16,384; the largest tested projector required halving the batch size to 16,384 after an out-of-memory error.Source: §A.7 · p. 20

A.7.1. Ablation study details

• Global shuffling and mixup were jointly needed to prevent performance collapse and support generalizable representations; removing either degraded results.Removing Sentinel-1 caused a moderate performance decrease, while Sentinel-2-only performance remained relatively good. Using QAT for int8 embeddings gave nearly four-fold lower storage and bandwidth requirements with negligible embedding-quality impact, compared with float32.Source: §A.7.1 · pp. 20–21

A.7.2. Justification for the Number of Sampled Timesteps (L)

• The study selected L = 40 sampled timesteps for both Sentinel-2 and Sentinel-1, balancing crop-classification performance, inference cost, and global observation availability.Sentinel-2 averages approximately 37.6 valid observations per pixel globally; performance gains beyond 40 were marginal or negligible relative to added inference cost, and performance was largely insensitive to Sentinel-1 sample count.Source: §A.7.2 · p. 21, Table 6 · p. 21

A.8.1. TESSERA

• TESSERA embeddings are used as fixed features with lightweight task-specific heads trained on labeled downstream data.The described heads include shallow MLPs for pixel-wise classification or regression and spatial models such as UNet for patch-based dense prediction.Source: §A.8.1 · pp. 21–22

A.8.2. Presto & AlphaEarth

• Presto and AlphaEarth evaluation used their publicly available embeddings as input features with lightweight task-specific heads.Presto embeddings were generated using the authors’ official GEE scripts and released weights; AlphaEarth’s public Satellite Embedding V1 dataset was used.Source: §A.8.2 · p. 22

A.8.3. Other RSFMs

• Other remote-sensing foundation models were benchmarked with the Pangaea protocol across Austrian Crop, PASTIS-R, Biomassters, and Borneo CHM.The protocol attaches a LinearClassifier for classification and heavier decoders such as UperNet+LTAE or baseline Unet for regression and segmentation.Source: §A.8.3 · p. 22

A.9. Downstream Task: Austrian Crop Classification

• Excluding winter observations raised Austrian crop-classification F1 from 78.94% to 79.83% in this regional dataset.The authors attribute the gain to winter data adding non-discriminative noise associated with dormancy, cloud cover, snow, and low solar elevation.Source: Table 9 · p. 25, §A.9 · p. 25

A.10. Downstream Task: Canopy height estimation

• Canopy-height evaluation compared embeddings and baseline products using spatially disjoint train, validation, and test data in Danum Valley, Borneo.The study trained three models per spatial fold and compared TESSERA, Presto, and AlphaEarth embeddings, a global canopy-height product, and Sentinel-2 annual median composites; the supplied passages describe setup but not comparative results.Source: §A.10 · pp. 25–26

A.11. Downstream Task: Above-ground biomass estimation

• TESSERA’s task-specific AGB training took five hours, versus eight days for the pretrained BioMassters winner’s original training.The authors report that retraining the winner from scratch performed worse than using its pretrained weights; TESSERA’s five-hour figure is for task-specific training.Source: §A.11 · p. 26

A.12. Interactive Habitat Mapping Tool

• The open habitat-mapping tool lets users label map points, train a kNN classifier on embeddings, and iteratively refine predicted habitat maps.It displays a PCA-derived false-colour view, supports immediate feedback after adding labels, and includes batch processing for larger regions.Source: §A.12 · p. 27, Figure 18 · p. 27

A.13. Pseudo Code

• Mixup consistency regularization adds a loss that matches correlations from a mixed multimodal view to targets blended from original-view correlations.The pseudocode adds this consistency loss to the standard Barlow Twins loss.Source: §A.13 · p. 28
• Global shuffling samples sparse temporal views per valid pixel across tiles, then permutes the pooled pixels to decorrelate spatial neighborhoods.The shuffled paired views are saved in chunks for the pretraining dataloader.Source: §A.13 · p. 28
• Table 10 characterizes TESSERA as open, gap-free, global annual 10 m pixel-level embeddings with cloud-invariant compressed spectral-temporal features combining multispectral and radar data.The table also describes the embeddings as oriented to land features.Source: Table 10 · p. 29