Source-linked AI summary
Foundation Models Meet Agriculture: Challenges Beyond Pretraining
Vishal Nedungadi, Xingguo Xiong, Marc Rußwurm, Ioannis N. Athanasiadis
TL;DR
Agricultural foundation-model transfer remains unreliable because downstream tasks combine heterogeneous modalities and structural settings that fixed EO architectures may not accommodate. The paper benchmarks EO, tabular, and supervised models across seven agricultural datasets, finding modality penalties and unstable rankings, while motivating domain-aware model design and evaluation.
Problem
Agricultural monitoring requires models that handle diverse modalities and task structures, but current EO foundation models are architecturally constrained and transfer unreliably.
Method
The study benchmarks two EO foundation models, a tabular foundation model, and supervised baselines across seven datasets spanning three agricultural tasks and five structural axes.
Results
EO foundation models incur penalties when tasks depend on modalities outside their pretraining sets, and no single architecture ranks consistently across datasets.
Takeaways & Limitations
Agricultural foundation models should ingest varied modalities and be evaluated across task heterogeneity rather than through averaged benchmark performance alone.
Takeaways & Limitations
The evaluation is limited by the small number of EO models applicable to these datasets and does not cover truly global regional, farming, or task diversity.
Abstract
from arXiv · showhide
Global food security and sustainable climate action increasingly rely on robust, scalable agricultural monitoring. Earth observation foundation models have emerged as powerful, label-efficient tools across general remote sensing domains, yet early attempts to deploy them for agricultural applications have yielded surprisingly poor results. We hypothesize that this performance gap stems from the extreme heterogeneity of agricultural landscapes and the inherent inability of current earth observation foundation models to adapt to task-specific nuances. In this work, we systematically evaluate two critical bottlenecks hindering the deployment of foundation models in agricultural tasks, benchmarking two earth observation foundation models, a foundation model designed for tabular data, and conventional supervised baselines across seven real-world agricultural datasets spanning yield prediction, phenology estimation, and crop classification. First, we identify a pretraining-deployment modality gap: agricultural downstream tasks frequently require diverse, non-imagery data modalities that earth observation foundation models are architecturally unequipped to ingest, while a foundation model built for tabular data handles this heterogeneity more naturally. Second, we formalize the agricultural task space across five structural axes to demonstrate why current models fail to generalize reliably, resulting in highly unstable model rankings across evaluation settings. By characterizing these structural and modal gaps, our insights highlight the friction between general-purpose architectures and specialized agricultural downstream data, providing a strategic roadmap for developing the next generation of domain-aware foundation models.
1 Introduction
Agricultural monitoring is essential for food security, but transferring general-purpose foundation models to agriculture is fragile because tasks require heterogeneous modalities and structures. The study benchmarks these mismatches across seven real-world datasets and model families to identify why performance and rankings remain unstable.
- Agricultural monitoring must answer where crops are grown, when they develop, and how much they produce across diverse geographies.These questions support scalable, timely, and precise monitoring as climate change destabilizes historical production baselines.
- EO foundation models learn generalized representations from massive unlabeled satellite archives and are expected to transfer with limited labeled data.Their expected advantage comes from shared satellite modalities between pretraining and agricultural tasks.
- Agricultural tasks require non-imagery inputs such as soil properties, agro-meteorological indices, and daily weather that fixed EO architectures may not ingest.Because datasets use different modality subsets, no fixed pretraining modality set covers every task.
- The study compares EO foundation models, supervised baselines, and tabular foundation models across seven datasets to separate modality mismatch from the value of pretraining.The model families differ in pretrained priors and their ability to ingest arbitrary modalities.
- Across the benchmark, EO models incur penalties when critical modalities are unavailable, while no architecture ranks consistently across datasets.The evaluation treats this instability as evidence of structural and modal mismatch rather than averaging it away.
2 Experimental Setup
The experimental setup spans seven diverse agricultural datasets, three model families, task-specific metrics, tailored splits, and standardized evaluation procedures. It compares modality-flexible supervised and tabular models with EO foundation models under native input and sequence constraints.
- Seven datasets cover crop classification, phenology estimation, and yield estimation across varied regions, farming systems, and structural axes.The axes include spatial unit, farming system, split design, target type, and dataset size.
- CropHarvest uses a Kenya pixel-level binary classification subset, while BreizhCrop uses parcel-level nine-class classification in Brittany with 45 temporal steps.These datasets stress contrasting smallholder and large-scale farming systems.
- The benchmark includes supervised baselines, EO foundation models, and tabular foundation models.Supervised models include Random Forest, LSTM, and Transformer; EO models include Galileo and CropFM; TabPFN represents tabular foundation models.
- Classification uses Macro F1-score, phenology regression uses MAE, and yield regression uses R2.The metrics are selected by dataset task type.
- Evaluation uses dataset-specific splits, three deterministic random seeds, and mean performance reporting.Splits include randomized, spatial, and chronological holdouts tailored to generalization dimensions.
- EO models are evaluated with full fine-tuning and frozen feature extraction, while temporal aggregation standardizes inputs across models.Galileo additionally uses monthly composites for BreizhCrop, CY-Bench, and Phenology because its sequence limit is exceeded at weekly resolution.
- The modality coverage matrix separates dataset-curated inputs from the physical modalities each model architecture natively supports.It lists imagery, vegetation, elevation, weather, agro-meteorological, and soil-moisture modalities.
- A fixed hyperparameter policy prioritizes realistic robustness comparisons over extensive per-dataset optimization.The policy addresses limited labeled samples and reduces overfitting risk during benchmarking.
3 Results and Empirical Analysis
The evaluation exposes two linked challenges for agricultural foundation models: limited native support for task modalities and structural heterogeneity that produces unstable performance rankings. Modality alignment helps in some settings, but it does not guarantee reliable transfer, while unconstrained tabular modeling can remain competitive.
- Evaluation framework: Agricultural evaluation is structured around modality mismatch and task heterogeneity, spanning seven datasets across five structural axes.The axes include spatial unit, farming system, split design, task type, and dataset size.
- Pretraining-downstream modality gap: Modality coverage measures the percentage of each dataset’s curated modalities that an architecture can natively ingest, and varies substantially by dataset.Coverage is near-complete for Sentinel-2 crop classification but drops for yield datasets requiring soil and agro-meteorological signals.
- Pretraining-downstream modality gap: Incomplete modality coverage imposes a context-dependent penalty: both Galileo and CropFM underperform the all-modality baseline on German and US yield estimation.In Argentina, Galileo reaches ΔR2 ≈+0.05 while CropFM reaches ΔR2 ≈−0.21, showing volatile behavior under low coverage.
- Pretraining-downstream modality gap: Near-complete modality coverage does not ensure transfer, as Galileo incurs ΔF1 ≈−0.16 in both Kenya and France, whereas German phenology shows ΔMAE > 0 for frozen FM features.The results indicate that resolving modality mismatch is necessary but not sufficient for robust agricultural modeling.
- Unconstrained feature access: TabPFN matches or outperforms other models across several datasets by ingesting all available task features without modality constraints; on Senegal, ΔR2 ≈+0.22.Its prior is learned from synthetic tabular data rather than geospatial or agricultural pretraining.
- Task heterogeneity: Model rankings fluctuate across datasets: TabPFN reaches R2 of 0.65 on CY-US, CropFM reaches R2 = 0.33 on YS-AR, and LSTMs reach R2 = −0.15 there.The same instability appears in crop mapping, where the sequence Transformer leads with F1-scores of 0.89 in CH-KE and 0.60 in BC-FR.
- Fine-tuning behavior: Galileo fine-tuning performs poorly on CY-DE and PH-DE, with R2 = −0.85 and MAE = 53.42, respectively.The authors hypothesize catastrophic forgetting after fine-tuning an overparameterized model on localized small-scale data under shifts.
4 Reflections on Agricultural FM Development
Agricultural monitoring presents modality and structural heterogeneity that general-purpose foundation models do not reliably accommodate. These mismatches create unstable performance and a practical deployment gap relative to simpler baselines.
- The Modality Superset Problem: Agricultural datasets provide different modality subsets, while pretrained foundation models rely on fixed multimodal input spaces.
- The Modality Superset Problem: Learned representations sometimes compensate for missing modalities, but the benefit is unreliable and dataset-dependent.
- Task Heterogeneity as a Fundamental Barrier: Agricultural monitoring spans pixel-level and sub-national yield estimation, point-observation phenology, and field- and parcel-level crop classification.
- Task Heterogeneity as a Fundamental Barrier: No single pretraining objective suits all agricultural settings, particularly when targets range from pixel time series to county-level annual scalars.
- Ease of Use and the Practical Deployment Gap: Foundation-model deployment requires adaptation choices and regularization, whereas tabular baselines ingest available modalities with minimal preprocessing and are competitive in multiple settings.
5 Conclusion
The evaluation tests foundation models and standard baselines across seven diverse agricultural datasets and identifies modality mismatch, task heterogeneity, and deployment friction as central challenges. The findings support domain-aware architectures and evaluations that preserve task diversity rather than averaging it away.
- The evaluation spans seven datasets across crop mapping, phenology estimation, and yield estimation, varying in spatial unit, farming system, split design, target type, and dataset size.
- Foundation models incur a substantial penalty when datasets depend on variables outside their pretraining set, while frozen representations do not consistently outperform unconstrained baselines even with full modality coverage.
- No single model ranks consistently across datasets; the best architecture changes with geography, spatial aggregation, and label regime.
- The authors recommend architectures that ingest varied modalities, benchmarks treating task heterogeneity as a first-class axis, and candid use of supervised baselines when appropriate.
6 Limitations and Ethical Considerations
The study’s comparisons are constrained by model applicability, geographic and task coverage, baseline selection, and possible unexamined sources of ranking instability. Ethical concerns also arise if pretraining data overrepresent industrial farming systems and affect high-stakes decisions.
- Limitations: Few EO models are structurally applicable because they are not designed for long temporal sequences or varied modalities, limiting cross-model comparisons.
- Limitations: The evaluation does not cover a truly global range of regions, farming practices, or agricultural tasks.
- Limitations: The study uses traditional simple baselines rather than complex domain-specific models developed for individual tasks.
- Limitations: Pretraining biases and downstream information leakage may be additional unknown sources of model-ranking instability.
- Ethical Considerations: Bias toward data-rich industrial farming systems could distort market prices and disadvantage vulnerable farming communities in high-stakes applications.
7 Generative AI Usage
Generative AI assisted code development and manuscript drafting, while the resulting code and text were checked, validated, and modified for technical accuracy and scientific integrity.
- Generative AI tools supported code development and manuscript drafting, followed by double-checking, validation, and modification.
- The authors state that they retain responsibility for the final research content.
B Full set of results
The paper reports the complete benchmark results in Table 3 and adds a modality-controlled experiment restricting supervised baselines to CropFM’s available modalities.
- Table 3 reports results across all datasets and models using each model’s native modality access.Table 4 additionally evaluates supervised baselines under CropFM’s modality constraints.
C.1 Pretraining Data
CropFM pretraining samples temporary-crop regions worldwide on a regular grid and combines nine temporal and static Google Earth Engine modalities on a standardized weekly timeline.
- Pretraining retains 5km × 5km grid cells containing at least 5km2 of temporary cropland and samples 10 points per retained grid.Sampling is restricted to regions identified by the ESA WorldCereal GEE product.
- Each sampled point uses a fixed one-year window from 2021-01-01 to 2022-01-01 with nine temporal and static modalities.The modalities are summarized in Table 6.
- Temporal inputs are resampled to a common weekly grid, truncated to 52 weeks, with coordinates and calendar position encoded.Mean aggregation is used by default.
- Normalization statistics are computed separately for every variable and modality using only non-NaN entries.
- CropFM projects each modality-timestep pair into a token using modality-specific linear layers, then processes shared tokens with one transformer encoder/decoder.Static modalities contribute a single token, while invalid timesteps use learned modality-specific missing tokens.
C.2.1 Masking Strategy.
CropFM uses structured masking to expose the model to missing modalities, missing timesteps, and longer-range temporal reconstruction rather than relying only on random token masking.
- Each forward pass uniformly samples one of four masking strategies, assigning each a 25% probability.
- Random: Random masking independently masks tokens across the full sequence.
- Modality: Modality masking removes entire modalities, approximately 45% of the nine modalities per sample, requiring reconstruction from the others.
- Random timesteps: Random-timestep masking removes approximately 75% of timesteps within each modality while preserving cross-modality structure at each timestep.
- Contiguous / seasonal: Contiguous or seasonal masking retains calendar-selected timesteps, forcing longer-range temporal interpolation or extrapolation.Examples include one sample per month or only the first or second half of the year.
C.2.2 Optimization and Training Configuration.
The paper documents CropFM’s feature support, training configuration, and comparison tables, including native-modality results and a CropFM-matched modality evaluation.
- Table 2 lists granular task features and which foundation models natively support them.It covers satellite, meteorological, terrain, and soil variables using acronyms such as DEM, ndvi, and soc.
- Table 3 reports native-access results using R2 for yield, F1-score for classification, and MAE in days for phenology, averaged over three seeds.EO foundation models include frozen-embedding and fully fine-tuned variants, with standard deviations reported.
- Table 4 isolates modality-access effects by restricting supervised baselines to CropFM’s modality subset and comparing them with CropFM variants.
- Tables 5 and 6 summarize CropFM’s optimization configuration and pretraining data modalities.