Lightweight, Pre-trained Transformers for Remote Sensing Timeseries

• Prior remote-sensing self-supervised models treat data as single-timestep images, missing temporal patterns, typically using one product and facing compute and label-shape constraints.Some datasets use points or irregular polygons, requiring additional methods to handle those labels.Source: §1 · pp. 1–2
• Presto uses self-supervised masked autoencoding to reconstruct unobserved timepoints and modalities from pixel-timeseries, accommodating diverse inputs and missing data.At fine-tuning, the decoder is discarded and the encoder output is used for downstream tasks.Source: §1 · p. 2, Figure 1 · p. 2
• Up to 1000× fewer trainable parameters and orders of magnitude fewer inference FLOPs let Presto remain competitive with ViT- or ResNet-based models.The passage frames computational performance as a primary consideration when making contiguous predictions over millions or billions of samples.Source: §1 · p. 2
• When one input is missing, Presto can leverage alternative remote-sensing sources, such as Sentinel-1 when Sentinel-2 imagery is cloudy.Source: §1 · p. 3
• Pixel-based Presto can in some cases match or outperform sophisticated computer-vision approaches; the authors hypothesize that dense data sources and smaller task receptive fields help explain this result.The authors note that many remote-sensing tasks may need smaller receptive fields than computer-vision models provide.Source: §1 · p. 3

• Remote-sensing time-series work has investigated transformers, task-specific transformer architectures, and recurrent networks.Source: §2 · p. 3
• For single- or few-timestep images, remote-sensing studies commonly use computer-vision architectures ranging from ResNets to Vision Transformers.Source: §2 · p. 3
• Remote-sensing masked-autoencoder work generally uses raw products and few timesteps, expects fixed input shapes, and yields models ranging from 2.5 million to over 300 million parameters.CROMA is noted as an exception that ingests Sentinel-1 and Sentinel-2; other approaches typically ingest a single sensor.Source: §2 · p. 3

3.3 Pre-training via Structured Masking

• Presto uses masked autoencoding: an encoder processes unmasked inputs, a decoder reconstructs masked values during pre-training, and the encoder is retained for downstream tasks.Training masks may cover random values, channel groups, contiguous timesteps, or timesteps; the loss balances continuous-value MSE and categorical cross-entropy per reconstructed value.Source: §3 · p. 3, §3.3 · pp. 5–6

3.1 Pre-training Data

• Presto was trained on 21.5M globally sampled pixel-timeseries, each covering 12 contiguous monthly timesteps with 15 dynamic channels and static topography and location data.The sample combines Sentinel-1, Sentinel-2, ERA5, NDVI, and Dynamic World data; Figure 2 illustrates reconstruction of masked Sentinel-2 RGB channels from single-pixel inputs.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 Encoding and tokenization

• Presto tokenizes each timestep by sensor or source, projects real-valued channel groups into a shared latent space, and embeds categorical Dynamic World classes by lookup.Tokens also receive positional, month, and learned channel-group encodings; static topography and coordinates are represented separately.Source: §3.2 · pp. 4–5

3.3 Pre-training via Structured Masking

• On CropHarvest, Presto outperformed TIML and MOSAIKS-1D while adapting fewer parameters, and PrestoR recovered their full-12-month performance with a subset of months.The CropHarvest comparison measures mean F1 across tasks; the partial-month comparison is reported for AUC ROC without Dynamic World input.Source: Table 2 · p. 6, Figure 3 · p. 6

Model configuration

• The standard Presto configuration uses two attention layers, eight heads, 128-dimensional embeddings, and an MLP ratio of four.The paper also investigated alternative encoder configurations.Source: §4 · p. 6

Image-level predictions

• For EuroSat image-level predictions, Presto’s per-pixel outputs were summarized by their mean and standard deviation, concatenated, and passed to a downstream classifier.Source: Figure 4 · p. 7

Downstream evaluation approach

• Presto’s encoder is used either as a feature extractor with a regression or random-forest model, or fine-tuned with a linear output layer.Feature-extraction variants train only the downstream model; fine-tuning updates the encoder and linear transformation.Source: §4 · pp. 6–7

Pre-training validation

• Pre-training validation classified CropHarvest points by FAO indicative crop class, excluding points reserved for evaluation.Source: §4 · p. 7

Baseline comparisons

• Evaluation compared Presto with task-specific baselines and adapted MOSAIKS to timeseries by convolving over time.The adapted features were evaluated with random forests and regression models.Source: §4 · p. 7

5 Evaluation Tasks & Results

• Across six tasks spanning four continents and 38 countries, Presto was benchmarked against state-of-the-art models where possible and could be fine-tuned on a single GPU or CPU.Fuel-moisture fine-tuning took under six minutes on a 2017 MacBook Pro CPU.Source: §5 · p. 7

5.1 Timeseries Tasks

• The timeseries evaluations include CropHarvest crop classification, fuel-moisture regression, and algae-bloom severity prediction.Fuel moisture used a geographically separated test set; algae-bloom comparisons also used a geographically separated test set because competition test data were unavailable.Source: §5.1 · pp. 7–8

5.1.1 Timeseries Results

• Presto significantly outperformed CropHarvest state of the art and beat regression-task baselines, while remaining performant with fewer timesteps or without Dynamic World.The regression table notes that literature baselines are not directly comparable because of differing inputs or private test data.Source: §5.1.1 · pp. 8–9, Table 3 · p. 8

5.2.1 Image-based Results

• Up to four orders of magnitude fewer FLOPs to encode an image, with competitive EuroSAT accuracy versus image-based models; multispectral results significantly outperformed those models.Presto’s average accuracy with RGB was comparable to larger ViT-based models across resolutions.Source: Figure 5 · p. 9, §5.2.1 · pp. 9–10, Table 5 · p. 10

5.2 Image-based Tasks

• TreeSatAI detects tree species from forestry images, while EuroSAT classifies Sentinel-2 images into ten land-cover classes; both were compared with task-specific methods.The EuroSAT comparisons included RGB and multispectral inputs and multiple image resolutions.Source: §5.2 · p. 9

5.3 Image-Timeseries Tasks

• S2-Agri100 evaluates crop-type classification using 5 × 5 pixel patches and compares Presto with pre-trained and randomly initialized SITS-Former.Source: §5.3 · p. 10

5.3.1 Image-Timeseries Results

• Presto achieved results comparable to SITS-Former on 24-timestep S2-Agri100 despite having six times fewer parameters, and remained performant with less accurate location metadata.The task’s intervals range from 10 to 30 days, unlike Presto’s 12-month pre-training timeseries.Source: §5.3.1 · p. 10

5.4 Ablations

• Combining structured and random masking outperformed random masking alone; pre-training improved results on most reported comparisons, and larger Presto variants performed better.Exceptions and specific reported increases include a slight scratch-model advantage on TreeSatAI with Sentinel-1, a 50% accuracy increase on S2-Agri100, and EuroSAT accuracy increases of 14% with RGB and up to 22% at lower resolutions.Source: §5.4 · p. 11, Table 7 · p. 11

6 Discussion & Conclusion

• Presto is pre-trained for 10 m/px imagery, does not natively process imagery below 1 m/px, and image-based models may suit whole-image scene classification better.On EuroSAT, performance plateaued as input resolution increased when relevant class pixels were a minority, as with highways.Source: §6 · p. 11
• Presto achieves state-of-the-art results across globally distributed evaluation tasks while meeting remote sensing’s computational-efficiency constraint.The conclusion also states that self-supervised learning can benefit small models.Source: §6 · p. 11
• Table 8 examines the effect of model size on PrestoR’s CropHarvest validation performance and reports parameter counts and FLOPs for a full input.The passage describes the table’s scope but provides no cell values.Source: Table 8 · p. 12

Impact statement

• Presto’s model size and use of publicly available training data are intended to make it accessible in compute-constrained environments and reduce its carbon footprint.The carbon-footprint claim is linked to the model’s small size.Source: Impact statement · p. 12
• The authors encourage deployment in collaboration with local communities and stakeholders because people may be monitored without knowing or consenting to data collection.The stated concern is the collection of information about individuals through remote sensing.Source: Impact statement · p. 12

A.1 Pre-training details

• The model was trained for 20 epochs with batch size 4096, yielding 5950 batches per epoch, on a single NVIDIA V100 GPU.Source: §A.1 · p. 16
• Training used AdamW with weight decay 0.05, βs of (0.9, 0.95), and cosine annealing to a maximum learning rate of 0.001 at epoch 2.Source: §A.1 · p. 16
• The masking ratio was 0.75, with a masking strategy randomly selected for each instance and additional random masking used when needed to reach the ratio.Source: §A.1 · p. 16

A.1.1 Pre-training data

• Sampling divided Earth into three regions, stratified samples by land-cover classes within ecoregions, and produced the distribution shown in Figure 6.Source: §A.1.1 · p. 16, Figure 6 · p. 16
• Each sample was a 510 × 510-pixel tile at 10 meters per pixel, with 2,500 pixels grid-sampled from each tile.Source: §A.1.1 · p. 16
• The resulting dataset contained 21,535,000 pixel samples, each with 24 one-month timesteps.Source: §A.1.1 · p. 16

A.1.2 Input data

• Table 9 describes model sizes and FLOPs for encoding a EuroSat image or, for Presto, a pixel; image-level Presto FLOPs scale by the number of pixels encoded.At EuroSAT’s highest resolution of 64 × 64, the table caption says to multiply per-pixel FLOPs by 4,096.Source: Table 9 · p. 17
• Dynamic inputs include Sentinel-1 and Sentinel-2 observations, ERA5 precipitation and temperature, NDVI, and monthly Dynamic World land-cover classes.Sentinel-2 contributes visible, near-infrared, and short-wave infrared bands at 10 m and 20 m resolution; the 60 m bands were removed.Source: §A.1.2 · p. 17
• Topography supplies static elevation and slope, while location supplies static 3D Cartesian coordinates computed from each pixel’s latitude and longitude.Source: §A.1.2 · p. 17

A.1.3 Channel Groups

• Channels are grouped by shared sensor or product, equivalent native spatial resolution, and similar spectral content for Sentinel-2 groups.Each group is transformed into a token.Source: §A.1.3 · p. 17
• The satellite groups separate Sentinel-1 radar bands from Sentinel-2 RGB, red-edge, near-infrared, and short-wave-infrared bands.Sentinel-2 near-infrared is split into 10 m and 20 m groups.Source: §A.1.3 · p. 17
• Additional groups represent NDVI, ERA5 precipitation and temperature, SRTM-derived elevation and slope, and pixel coordinates.Source: §A.1.3 · p. 17

A.2 FLOP calculations

• Presto’s EuroSAT encoding FLOPs are estimated per pixel and scaled by image pixel count, while the comparison models use fixed 224 × 224 inputs.The FLOPs are calculated with thop and recorded in Table 9.Source: §A.2 · p. 18

A.3 Baselines

• The comparison baselines include random forests and MOSAIKS-1D, an adapted timeseries version evaluated with random forests or regressions.MOSAIKS-1D is excluded from Dynamic World because it does not work with categorical inputs.Source: §A.3 · p. 18

A.4 Downstream Results

• Downstream results cover CropHarvest, regression, EuroSAT, TreeSatAI, and Sen2-Agri100; most classifiers use three seeds, while kNN and linear regression are deterministic.Main-paper tables report run averages, with standard errors provided in supplementary tables; feature-extractor predictions use mean and standard deviation of pixel embeddings, while fine-tuning uses AdamW.Source: §A.4 · p. 18

A.5 Disentangling the effect of pre-training

• EuroSat pre-training increased accuracy by >0.1, representing gains of up to 25%; pre-training also helped TreeSatAI Sentinel-2 and regression tasks, but penalized TreeSatAI Sentinel-1.The authors hypothesize that the Sentinel-1 exception reflects the mismatch between single-timestep, single-channel-group input and the pre-training data; CropHarvest was omitted because its small size makes validation-set construction challenging.Source: §A.5 · p. 19
• Presto’s per-pixel processing omits spatial context, and pooling pixel representations by their mean and standard deviation loses information that affects image-based prediction.The authors omit CropHarvest from the pre-training comparison because its small size makes validation-set construction for fine-tuning difficult.Source: §A.6 · p. 19, §A.5 · p. 19

A.6 Presto’s failure modes

• On EuroSAT, accuracy plateaus as input resolution increases, with Highway and River classes identified as difficult cases.Figure 7 plots RGB kNN@5 accuracy against resolution by category; the authors hypothesize that simple token aggregation is insufficient when only a small part of an image contains the labelled class.Source: §A.6 · p. 21, Figure 7 · p. 21, Figure 8 · p. 21
• For image-based tasks dependent on objects in image subregions, models that natively process spatial information may be better suited than Presto.The authors plan to explore techniques to mitigate this difficulty.Source: §A.6 · p. 21