PaperlayerView exact revision on arXiv

Source-linked AI summary

SPEAR NeXT Causal Latent Forecasting Across Multiple Horizons for Spectral Temporal Earth Representation Learning

Rajiv Ranjan, Udaiveer Singh, Shashank Tamaskar, Dharmendra Saraswat

arXiv:2609.16871v1cs.CV

TL;DR

EO foundation models often learn temporal information through reconstruction, alignment, or summarization rather than directional prediction of unseen future states. SPEAR-NeXT encodes pixel spectral states and causally predicts multiple future latent states with relative and calendar-time conditioning. Its representations transfer strongly across land-cover and temporally structured agricultural tasks, while remaining non-uniform across downstream objectives.

  • Problem

    Existing EO temporal objectives mainly reconstruct, align, or summarize observed data instead of requiring representations to predict several unseen future latent states from preceding context.

  • Method

    SPEAR-NeXT separates wavelength-aware spectral encoding from a causal temporal model that predicts multiple future latent states using RoPE plus month and year embeddings.

  • Results

    94.81% accuracy in India and 88.78% in CONUS exceeded the strongest corresponding baselines by 9.61 and 2.38 percentage points, while SICKLE showed lowest errors for sowing-date, harvest-date, and yield prediction.

  • Takeaways & Limitations

    Causal multi-horizon latent forecasting produces reusable representations that transfer effectively to land-cover mapping, crop monitoring, phenological estimation, and in-season yield forecasting.

  • Takeaways & Limitations

    The method inherits the information limits and biases of frozen SPEAR embeddings, and monthly complete-coverage regional training does not establish cross-region generalization.

Abstract

from arXiv · show

Earth observation is inherently dynamic, yet temporal information in many foundation models is learned through reconstruction, invariance, or retrospective sequence summarization. SPEAR NeXT is introduced as a compact pixel-wise multimodal spectral temporal foundation model in which temporal self supervision is formulated as past only, multi horizon latent Earth state prediction. Instantaneous states are first encoded by the pretrained SPEAR model from optical, radar, and environmental observations into compact 32 dimensional embeddings. Their temporal evolution is then modeled by a causally masked Trans former that predicts multiple future latent states from pre ceding observations. Relative temporal order is represented using Rotary Position Embeddings, while month and year embeddings encode seasonal phase and interannual con text.

1. Introduction

SPEAR-NeXT addresses the limited use of directional predictive supervision in EO by learning compact pixel representations that predict several future latent Earth states from preceding observations. It separates instantaneous spectral encoding from causal temporal evolution and evaluates whether the resulting representations transfer across mapping, agricultural, phenological, and forecasting tasks.

  • Most temporal EO objectives reconstruct missing content, align views, or summarize observed intervals rather than predict several unseen future states from preceding context.
  • SPEAR-NeXT formulates temporal pretraining as causal latent Earth-state prediction, where each prediction uses states observed at or before the prediction time.Here, causal prediction does not mean intervention-based causal inference, and an Earth state denotes a learned pixel spectral representation.
  • The model first encodes instantaneous spectral states with SPEAR, then learns a separate transition operator over the resulting state sequence.This modular decomposition allows spectral states to be cached and reused when training the temporal model.
  • RoPE encodes sequence order and relative displacement, while month and year embeddings represent seasonal phase and interannual context.The two conditioning mechanisms distinguish sequence offsets from calendar position.
  • Horizon-weighted supervision uses multiple future horizons to constrain local continuity, intermediate transitions, and longer-term seasonal structure.Cosine alignment preserves latent-space direction, while latent regression constrains coordinate scale and limits magnitude drift.

2. Related Work

Related EO foundation models establish the value of temporal context but mainly learn through masked recovery, cross-view alignment, or conditional summarization. SPEAR-NeXT instead integrates spectral encoding with causal multi-horizon latent prediction and calendar-aware temporal conditioning to learn reusable predictive representations.

  • Existing EO temporal objectives primarily use masked recovery, cross-view alignment, or conditional summarization rather than explicit causal prediction of future latent Earth states.
  • Its spectral-to-temporal factorization uses wavelength-aware pixel encoding for the instantaneous state and a separate temporal model for state evolution.The approach treats wavelength metadata as a physical measurement index without claiming an explicit physical-process model.
  • Predictive representation methods motivate learning by forecasting latent variables or embeddings instead of reconstructing pixels, while EarthNet2021 forecasts future imagery in observation space.Latent forecasting avoids allocating capacity to radiometric detail and other measurement-specific effects when future imagery itself is not the target.
  • SPEAR-NeXT uses multi-horizon prediction as a representation constraint, exposing the model to short-range continuity, intermediate transitions, and longer-range seasonal recurrence.Short horizons receive stronger optimization weights, while longer horizons prevent the representation from relying only on adjacent-state interpolation.
  • SPEAR-NeXT combines RoPE for relative sequence progression with month and year embeddings for calendar phase and interannual context.
  • The central contribution is the integration of these components into a self-supervised objective for compact predictive Earth representations reusable through frozen or lightweight adaptation.The intended downstream scope includes land-cover mapping and temporally demanding agricultural tasks.

3. Methodology

SPEAR-NeXT separates instantaneous spectral-state encoding from causal temporal evolution, using a frozen SPEAR encoder and a Transformer to predict multiple future latent states from preceding observations. Calendar embeddings, relative-position encoding, horizon-specific heads, and weighted trajectory supervision support temporally informed representations while preserving strict causality.

  • Problem formulation: SPEAR-NeXT maps chronologically ordered multispectral pixel observations into spectral states, then predicts each future state from observations available only up to the current time.The paper uses causal information flow rather than intervention-based causal inference, and defines an Earth state as a learned compact spectral representation.
  • Instantaneous spectral states: The frozen pretrained SPEAR encoder produces compact 32-dimensional instantaneous states whose parameters remain fixed during temporal training.The decoder is discarded, the encoder output associated with the spectral class token is retained, and states can be computed and cached once.
  • Temporal token construction: Month and year embeddings encode absolute seasonal and interannual context, while RoPE represents relative temporal order and displacement within attention.Separating these signals distinguishes sequence offsets from calendar phase and year-specific conditions.
  • Causal temporal dynamics encoder: The causal temporal Transformer uses masked self-attention, QK normalization, residual connections, and gated SwiGLU feed-forward transformations to form contextual states.The attention mask ensures that the representation at time t uses only observations at times 1 through t.
  • Causal multi-horizon latent forecasting: Separate prediction heads directly forecast each future horizon, with horizon k representing a future sequence step that corresponds to k months under monthly compositing.Direct horizon-specific prediction avoids recursive error accumulation and allows the mapping from context to target to vary with forecast distance.
  • Horizon-weighted latent trajectory objective: Horizon-weighted supervision combines cosine alignment with latent regression, strengthening near-term optimization while requiring longer horizons to retain predictive temporal information.The regression term constrains both direction and scale, limiting magnitude drift that cosine alignment alone would not constrain.

4. Experimental Analysis

The experimental design evaluates SPEAR-NeXT through independent regional pipelines, controlled preprocessing, leakage safeguards, and downstream transfer tasks. India and CONUS models use monthly multimodal observations with geographically separate training, validation, and test data.

  • Preprocessing: Frozen SPEAR modality encoders produce cached monthly pixel states before temporal training, while invalid and cloud-contaminated observations are removed during preprocessing.The modular pipeline uses optical, radar, and environmental SPEAR components with regional fusion before temporal modeling.
  • Regional pipelines: Two independent SPEAR-NeXT models are pretrained and evaluated separately for India and CONUS, without sharing samples or parameters.This preserves distinct geographic, climatic, agricultural, and land-cover distributions across the regional evaluations.
  • Leakage control: Pixel-level 60:20:20 partitioning, geographic separation, regional preprocessing, and temporal holdouts are used to prevent spatial and temporal leakage.Normalization is fitted only on each regional training partition and applied unchanged to validation and test data.
  • Evaluation scope: The evaluation spans forecasting diagnostics, land-cover and crop classification, flood and fire-related classification, and CONUS crop-yield prediction.India uses SICKLE, Sen1Floods11, VIIRS fire-related classification, and Dynamic World; CONUS uses USDA NASS crop yield, CropHarvest, Sen1Floods11, and Dynamic World.

4.2. Training Configuration

SPEAR-NeXT trains a compact causal temporal predictor on frozen SPEAR embeddings using direct multi-horizon supervision. The configuration combines stable Transformer components, horizon-weighted cosine and regression losses, and controlled temporal and architectural ablations.

  • Architecture: The temporal encoder uses four causal Transformer layers, dimension 128, four attention heads, and direct prediction across 46 future sequence steps.The frozen SPEAR encoder prevents the target embedding space from drifting during temporal pretraining.
  • Optimization: RoPE, QK-Norm, SwiGLU layers, AdamW optimization, warmup, gradient clipping, mixed precision, and EMA validation stabilize and streamline temporal training.The EMA copy is used for validation and final evaluation rather than as a separate target network.
  • Objective: Multi-Horizon Prediction attaches independent lightweight MLP heads to future horizons, combining cosine directional alignment with MSE magnitude control.The composite objective weights the two terms equally with α = 0.5 and uses inverse-horizon weighting with γ = 1 unless otherwise stated.
  • Baselines: Forecasts are compared with persistence, linear latent extrapolation, and seasonal persistence baselines to test gains beyond continuity and annual-cycle priors.For monthly composites, the seasonal period is P = 12.
  • Ablations: Ablations vary loss composition, horizon weighting, single-step versus multi-horizon supervision, temporal conditioning, and QK normalization under matched data and capacity.These comparisons isolate whether temporal design choices improve forecasting and transferable representations.

4.4. Horizon-Wise Evaluation Metrics

Horizon-wise evaluation measures directional accuracy, latent reconstruction error, explained variance, and magnitude drift across held-out forecast horizons. Per-horizon curves are retained alongside weighted aggregates to expose long-range degradation and distinguish continuity from learned temporal evolution.

  • Horizon analysis: Per-horizon results are reported alongside aggregate scores because weighted averages can conceal long-horizon failure.Longer-horizon degradation reflects increasing uncertainty, weaker context–target dependence, and fewer valid pairs rather than recursive error accumulation.
  • Forecast metrics: Forecasting quality is measured with cosine similarity, MSE, MAE, and macro-averaged coordinate-wise R2 over valid held-out context–target pairs.The metrics are computed separately for each horizon k and aggregated using the same normalized horizon weights as pretraining.
  • Baseline interpretation: Persistence, linear extrapolation, and seasonal persistence provide continuity and seasonality baselines for interpreting forecast improvements.Improvement over persistence indicates state-evolution learning beyond local continuity, while improvement over seasonal persistence indicates context-dependent departures from a fixed annual cycle.
  • Magnitude stability: Relative norm bias and normalized absolute norm error complement cosine similarity by detecting incorrect embedding scale.Post-hoc affine calibration is diagnostic only and does not replace the primary unmodified-model results.

4.7. Temporal Self-Similarity Preservation

The analysis tests whether forecasts preserve seasonal and recurrent geometry and align with the correct future temporal position. It also connects these diagnostics to controlled conditioning, multi-horizon, and downstream-transfer comparisons.

  • Temporal geometry: Self-similarity preservation compares target and predicted temporal geometry using matrices, Frobenius discrepancy, and heatmap patterns such as diagonal continuity and recurrent bands.These patterns indicate latent temporal regularity, not a specific biological process unless supported by labels or phenological observations.
  • Prediction–target alignment: Prediction–target alignment evaluates whether each horizon forecast matches the correct future position rather than a generic or identity-like embedding.A positive, increasing diagonal concentration relative to baselines indicates temporally specific progression rather than constant or persistence behavior.
  • Loss ablations: The composite objective is compared with cosine-only and regression-only variants using horizon metrics, norm drift, and self-similarity discrepancy.These ablations test whether directional alignment or Euclidean reconstruction alone preserves latent geometry.
  • Temporal conditioning: RoPE encodes relative ordering and displacement, whereas month–year embeddings encode seasonal phase and interannual context; the combined model addresses their distinct roles.The combination is most relevant when similar relative offsets occur in different seasonal regimes.
  • Downstream transfer: Forecasting diagnostics are paired with capacity-matched downstream transfer experiments to test whether causal multi-horizon prediction yields more useful Earth-observation representations.The downstream evaluation covers land-cover mapping and temporally demanding agricultural tasks using common lightweight architectures and data splits.

5. Results and Discussion

SPEAR-NeXT transfers effectively across regional land-cover, crop-classification, agricultural regression, and yield-forecasting tasks, with strongest gains on temporally structured objectives. Its yield forecasts improve over baselines across all crops, update through the season, and remain subject to regional and task-specific uncertainty.

  • Regional Land-Cover Classification: 94.81% India accuracy and 88.78% CONUS accuracy exceed the strongest regional baselines by 9.61 and 2.38 percentage points, respectively.SPEAR-NeXT remains strongest under both regional evaluation settings, though the CONUS margin is smaller.
  • Agricultural Transfer: SPEAR-NeXT achieves the best performance on all three SICKLE agricultural regression tasks, reducing MAPE to 1.25% for sowing date, 3.03% for harvest date, and 21.70% for yield.These correspond to relative error reductions of approximately 20.9%, 36.3%, and 4.8%, respectively.
  • USDA-NASS Crop-Yield Prediction: 0.721 mean crop-wise R2 exceeds 0.644 for TESSERA and 0.613 for Presto, with SPEAR-NeXT achieving the highest R2 for corn, soybean, wheat, and cotton.Absolute improvements over the strongest crop-specific baselines are 0.129 for corn, 0.085 for soybean, 0.053 for wheat, and 0.025 for cotton.
  • USDA-NASS Crop-Yield Prediction: 0.437 R2 improvement for corn shows the complete model outperforming the prediction-head-only control, while gains remain positive for soybean, wheat, and cotton.For corn, R2 increases from 0.289 to 0.726; identical heads and protocols attribute the difference primarily to pretrained temporal representations.
  • USDA-NASS Crop-Yield Prediction: Kriging adds modest or negligible improvement to the complete model but raises head-only R2 by 0.190 on April 15 and 0.232 on August 5.Even after kriging, the head-only configuration remains below complete SPEAR-NeXT, indicating spatial interpolation complements rather than replaces temporal pretraining.
  • Summary of Downstream Transfer: State-wise forecasts update as observations accumulate and often converge toward final USDA NASS yields, although early-season deviations and regional variation remain.Early observations contain limited information about late-season weather, stress, and management effects; the NASA Acres comparison is an external operational reference rather than a controlled benchmark.

A. Dataset Information

The SPEAR-NeXT pipeline uses two geographically distinct pretraining corpora: India for SPEAR spectral and multimodal pretraining, and CONUS for temporally ordered SPEAR-NeXT sequences.

  • The India corpus supports SPEAR spectral and multimodal pretraining, whereas the CONUS corpus provides temporally ordered pixel sequences for SPEAR-NeXT pretraining.
  • Figure 7 summarizes the two pretraining regions and their associated downstream evaluation datasets.The downstream suite includes SICKLE, Sen1Floods11, VIIRS active-fire observations, USDA NASS Quick Stats, CropHarvest, and Dynamic World.

A.1. India Corpus

The India corpus is a diverse, stratified pixel-level dataset supporting SPEAR pretraining and evaluation across agriculture, flooding, fire, and general land-cover mapping.

  • The India corpus contains approximately 2.7 million pixel-level samples spanning agricultural, forested, urban, barren, wetland, and water-covered environments.Sampling uses Dynamic World land-cover classes to reduce domination by prevalent surface types and increase coverage of less frequent classes.
  • Its observation sources include Sentinel-2 optical imagery, PlanetScope imagery, and Sentinel-1 radar imagery for multimodal SPEAR pretraining.
  • India-derived representations are evaluated on complementary Earth-observation tasks including crop classification and flood detection.
  • The India downstream suite spans agriculture, flooding, fire, and general land-cover mapping rather than a single semantic domain.

A.2. CONUS Corpus

The CONUS corpus supplies geographically diverse temporal sequences for SPEAR-NeXT pretraining and emphasizes temporally demanding agricultural transfer alongside general environmental evaluation.

  • The CONUS corpus contains approximately 4.5 million pixel-level samples across the contiguous United States, covering varied vegetation, climates, agricultural systems, management practices, and seasonal timing.Samples are stratified using Dynamic World land-cover classes.
  • Retained Sentinel-2 observations are organized into pixel-level sequences of T = 61 timesteps, with each observation converted into a frozen 32-dimensional SPEAR embedding.
  • The broader CONUS data include Sentinel-2 optical imagery, Sentinel-1 radar imagery, and matched climate and environmental variables.
  • The temporal model is pretrained on frozen Sentinel-2 SPEAR embeddings; radar and climate variables enter only where explicitly specified by downstream experiments.
  • CONUS downstream evaluation emphasizes county-level crop-yield prediction and geographically distributed crop-presence classification while retaining general environmental tasks.

A.3. Role of the Datasets in the Experimental Design

The experimental design separates self-supervised pretraining data from downstream evaluation labels and combines general mapping with agriculture-focused tasks to assess representation reuse across settings.

  • No downstream labels are used during SPEAR-NeXT temporal pretraining; Table 9 distinguishes pretraining datasets from downstream-only datasets.
  • The combination of general mapping and agriculture-focused datasets tests whether SPEAR-NeXT learns reusable temporal representations rather than features specialized to one region or application.
  • The India and CONUS corpora provide an initial basis for assessing transfer across contrasting land-cover distributions, agro-climatic conditions, and seasonal regimes.
Loading 2609.16871v1…