PaperlayerView exact revision on arXiv

Source-linked AI summary

EarthJev: Calibrated, Non-Generative Answers to Natural-Language Questions about Sentinel-2 Land Cover

Domenico Benfenati, Antonio Maria Rinaldi, Cristian Tommasino

arXiv:2611.02096v1cs.CVcs.AI

TL;DR

Remote-sensing vision-language models generate text for closed-option satellite-image questions, incurring decoding and answer-parsing costs without directly usable probabilities. EarthJev instead reads option logits in one pass over 12-band Sentinel-2 input. It is substantially faster than generative baselines, with accuracy close to the 2B models and calibrated probabilities.

  • Problem

    Generative answers to closed-option satellite questions require decoding and mapping text back to options, and do not directly provide answer probabilities.

    Source: §1 · p. 1

  • Method

    EarthJev is a 0.8B-parameter model that answers typed questions about 12-band Sentinel-2 patches by reading option-letter logits without generating text.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    6.8 times faster than its generative backbone at 0.8 points of accuracy, EarthJev is 0.6–1.2 points below the 2B models and returns calibrated probabilities.

    Source: §1 · p. 2

  • Takeaways & Limitations

    EarthJev provides valid-by-construction answers and calibrated probabilities while accepting declared option sets, with speed advantages over multimodal language models.

    Source: §1 · p. 2, §7 · p. 12

    EarthJev answers only questions with declared options, and is less accurate than the 2B multimodal models and EarthDial-4B.

    Source: §6 · p. 11

Abstract

from arXiv · show

Vision-language models for remote sensing answer questions about satellite images by generating text: every answer costs a decoding loop, has to be parsed back to an option, and comes without a usable probability. We present EarthJev, a 0.8B-parameter model that answers typed questions about multispectral Sentinel-2 patches without generating any token. All 12 bands enter the vision encoder of Qwen3.5-0.8B through a widened input layer initialised from the RGB kernels, and, following the open OneJev recipe, the answer is read from the logits of the option letters at a single position and trained with cross-entropy plus Brier score. Under a pre-specified analysis plan, we compare it with the same backbone fine-tuned to generate the answer and with three multimodal language models fine-tuned with LoRA, all trained on the same 70,890 questions from 5% of BigEarthNet v2.0 and timed in one job on the same GPU; VQA models that classify over a fixed answer vocabulary are reported as a reference for the cost of answering only the questions seen in training. EarthJev answers a question in 19.9 ms, 4.3 to 8.7 times faster than the multimodal language models and 6.8 times faster than its own backbone used generatively; when all the questions about an image are asked together, it answers them in 42.8 ms, 3.7 to 10.1 times faster than the batched generative models. On 36,028 test questions it reaches 86.0% accuracy (three seeds), 0.6 and 1.2 points below the two 2B multimodal models and 3.5 points below the remote-sensing EarthDial-4B, with answers that are valid by construction and calibrated probabilities (ECE 0.006). Replacing the nine non-RGB bands with their mean costs 4.5 points, so the model uses the multispectral input, and in an ablation training the whole vision encoder raises accuracy to 88.4% at the same cost. Code and model: https://github.com/CosmoIknosLab/EarthJev

1 Introduction

Remote-sensing vision-language models answer closed-option questions by generating text, which requires decoding, parsing, and recovering probabilities. EarthJev applies a typed, single-pass decision to multispectral Sentinel-2 questions and is substantially faster than generative baselines with a small accuracy gap to the 2B models.

Motivation

  • Generating free text incurs an autoregressive decoding loop, requires mapping text back to admissible options, and does not directly provide an answer probability.The cited remote-sensing VLMs generate free text [13] [17] [22] [25].Source: §1 · p. 1
  • OneJev provides an open typed-decision interface that returns option probabilities from a forward pass, but it has no remote-sensing training data and reads only three-channel images.EarthJev tests whether that recipe can answer multispectral satellite questions.Source: §1 · pp. 1–2

Contributions

  • EarthJev is a 0.8B-parameter model that answers typed questions about Sentinel-2 patches from all 12 bands without generating text.Its widened vision-encoder input layer is the only change to the OneJev recipe.Source: §1 · p. 2
  • The benchmark contains 36,028 test questions across five types on 5,991 test patches, with a second rendering using templates held out from training.Questions are generated from BigEarthNet v2.0 reference masks [3].Source: §1 · p. 2
  • The controlled comparison uses the same training questions for EarthJev, its generative backbone, and three LoRA-fine-tuned multimodal language models, with VQA classifiers as fixed-vocabulary references.Models are tested on the CORINE benchmark and two benchmarks authored by others, and timed in one job on one GPU.Source: §1 · p. 2

Findings

  • 6.8 times faster than the same backbone used generatively, EarthJev is 0.8 points less accurate, with valid-by-construction answers and calibrated probabilities.It is also 4.3 to 8.7 times faster per question than multimodal language models, at 0.6–1.2 points of accuracy from the 2B models.Source: §1 · p. 2

2 Related work

Prior remote-sensing VQA systems commonly classify over fixed answer vocabularies, while remote-sensing vision-language models generate free text. EarthJev adapts a typed one-pass decision approach to multispectral Sentinel-2 questions, where calibration is also part of the design.

Visual question answering in remote sensing

  • Remote-sensing VQA models combine image and question encoders and classify over fixed answer vocabularies, so they do not accept arbitrary questions or option sets.Examples include VBFusion [24] and LiT-4-RSVQA [7], trained on benchmark question distributions.Source: §2 · p. 2

Remote-sensing vision-language models

  • Remote-sensing VLMs such as GeoChat [13], LHRS-Bot [17], EarthDial [25], and EarthMind [22] answer in free text.BigEarthNet.txt [11] provides a manually verified benchmark split used here for transfer testing, not training.Source: §2 · p. 3

Discriminative decisions with multimodal language models

  • OneJev reads option-letter logits at the last position and trains with cross-entropy plus Brier loss, while its training data contain no remote-sensing source.EarthJev differs in task, 12-band Sentinel-2 input, and calibration evaluation.Source: §2 · pp. 3–4

Calibration

  • The Brier score and log-loss are strictly proper scoring rules, and temperature scaling is described as a post-hoc calibration fix [6].OneJev and EarthJev train using both proper scoring rules.Source: §2 · p. 4

3 Method

EarthJev handles typed questions by reading option-letter logits at one position, producing calibrated option probabilities without generating text. It widens Qwen3.5’s vision input for Sentinel-2’s 12 bands and follows OneJev’s training recipe.

3.1 Typed questions and single-position readout

  • The model reads logits for the declared option letters at the last input position, applies a validation-fitted temperature, and returns probabilities without generating text.Questions about one image share its encoded prefix and are evaluated in a batched forward pass; the returned answer is a deterministic function of the probabilities.Source: §3.1 · p. 4

3.2 Twelve Sentinel-2 bands in a vision-language model

  • The vision input layer widens from three to 12 channels, with RGB kernels copied to Sentinel-2’s B04, B03, and B02 channels and the other nine band kernels initialized to zero.The RGB pipeline is thereby preserved before training; bands are resampled and processed into the same number of visual tokens as an RGB image.Source: §3.2 · pp. 4–5
  • Only the widened input layer of the otherwise frozen vision encoder is trained, using the learning rate selected by a validation pilot.The reported pilot favored 5 × 10−6 over 10−4 and 10−3; this is the only change to the OneJev recipe.Source: §3.2 · p. 5

3.3 Model and training

  • EarthJev follows the Qwen3.5-0.8B OneJev configuration and trains on option-letter distributions with cross-entropy plus Brier score, randomly permuting option order.Only the last position is supervised; a temperature is fitted for each question type and option count using held-out validation questions.Source: §3.3 · p. 5

4 Data and benchmarks

The benchmark uses a stratified 5% subset of BigEarthNet v2.0, mask-derived CORINE questions, and two external transfer tests. A separate EuroSAT-MS collection is used only in an ablation, and the comparison follows a pre-specified protocol.

4.1 Images

  • The image set is a stratified 5% subset of BigEarthNet v2.0 [3]: 24,000 Sentinel-2 L2A patches with 12 bands and dense CORINE level-3 masks, split into 11,893 training, 6,116 validation, and 5,991 test patches.The test patches are 45.1% forest and semi-natural areas, 38.9% agricultural areas, 12.9% water bodies, 2.4% artificial surfaces, and 0.7% wetlands.Source: §4.1 · p. 5

4.2 Typed CORINE questions

  • Five question types are generated from CORINE masks, with cases near decision boundaries skipped: dominant class, presence, class-share interval, location, and pairwise area comparison.The rules include thresholds such as 5 points between dominant and second-largest classes, 5% presence, and 15 points for compare questions.Source: §4.2 · pp. 5–6
  • Test questions are rendered with both seen and held-out templates; the benchmark has 70,890 training, 36,698 validation, and 36,028 test questions.Applying the same rules to reference masks yields 100% test accuracy, checking the question generator.Source: §4.2 · p. 6

4.3 Transfer tests

  • Transfer testing uses RSVQAxBEN presence questions and BigEarthNet.txt binary and multiple-choice questions on the reBEN test split, without training on either benchmark.BigEarthNet.txt includes question kinds absent from training, including adjacency, counting, relative position, and season.Source: §4.3 · p. 6

4.4 EuroSAT-MS (ablation only)

  • EuroSAT-MS contributes 81,000 questions only to an ablation that adds a second collection; it is an L1C product, unlike reBEN.Its questions cover land-cover class and four class-group presence questions using a spatial split.Source: §4.4 · p. 6

4.5 Protocol

  • All models use the 70,890 CORINE training questions, benchmark images are disjoint from training images, and the hypotheses and claim rule were fixed before training.Validation questions are reserved for temperatures, checkpoint selection where applicable, and the learning-rate pilot.Source: §4.5 · p. 6

5 Experiments

In a controlled comparison, EarthJev answers faster than generative models while remaining close in accuracy to the two 2B multimodal models; its accuracy varies by question type and transfer setting. The experiments also test band use, calibration, and performance on questions written by other authors.

5.1 Setup

  • All models train on the same 70,890 CORINE questions with three seeds; a pre-specified analysis plan sets the hypotheses, tests, and support rule before training.The comparison includes a same-backbone generative model, three LoRA-tuned multimodal LLMs, and fixed-vocabulary VQA references.Source: §5.1 · p. 7

5.2 Speed and quality

  • EarthJev is 0.6 points below Qwen3-VL-2B and 1.2 below InternVL3-2B, with ECE 0.006; EarthDial-4B is 3.5 points more accurate, while multimodal LLMs are 4.3–8.7 times slower.The comparison uses CORINE accuracy and ECE alongside batch-1 latency.Source: Table 1 · p. 8, §5.2 · p. 8
  • When all questions about an image are asked together, EarthJev answers them in 42.8 ms and is 3.7 to 10.1 times faster per image than generative models.EarthJev encodes the image once; fixed-vocabulary models are faster but answer only questions and options from their training vocabulary.Source: §5.2 · p. 8, Table 2 · p. 9

5.3 Accuracy, calibration and transfer

  • Replacing the nine non-RGB bands with their training mean lowers CORINE accuracy by 4.5 points, while the preregistered H1 and H2 claims are not supported.H1 tests equivalence to multimodal and same-backbone generative models; H2 tests superiority to fixed-vocabulary models.Source: §5.3 · p. 9
  • Fixed-vocabulary models reach 87.8–88.8% on dominant-class questions versus EarthJev’s 83.6%, while EarthJev reaches 66.1% on location questions and retains 86.0% accuracy with held-out templates.The fixed-vocabulary models are weaker on location questions; two answer barely above the 25% chance rate for four options.Source: §5.3 · p. 9
  • On RSVQAxBEN, EarthJev reaches 81.0%, within the multimodal LLMs’ 80.6–82.4% range; on BigEarthNet.txt, all models remain well below training accuracy.The generative same-backbone model matches none of the four options on 45.9% of multiple-choice questions, whereas EarthJev returns a distribution over declared options.Source: §5.3 · p. 10, Table 3 · p. 10

5.4 Ablations

Ablations show that training the vision encoder can improve accuracy without changing inference cost, while initialization, data, and recipe choices have benchmark-dependent effects.

5.4 Ablations

  • 2.5 points: training the whole vision encoder raises CORINE accuracy to 88.4% and RSVQAxBEN accuracy to 83.0%, with ECE 0.006 and unchanged inference cost.With one seed, the authors describe this as an indication, not a tested result; the OneJev recipe keeps the encoder frozen.Source: §5.4 · p. 10
  • Starting new bands from RGB-kernel means is 1.6 points worse on CORINE and 3.3 points worse on BigEarthNet.txt than zero initialization.With zero kernels, the model starts exactly from the RGB model.Source: §5.4 · pp. 10–11
  • Using 50% and 25% of the training patches lowers CORINE accuracy by 1.4 and 2.7 points, respectively.Source: §5.4 · p. 11
  • Adding acquisition date and country raises accuracy by 0.5 points on CORINE and 1.5 on RSVQAxBEN, but the authors leave them out to keep the model image-only.They note that other collections, such as EuroSAT, carry no date.Source: §5.4 · p. 11
  • Adding EuroSAT-MS questions raises CORINE accuracy by 0.6 points but lowers RSVQAxBEN accuracy by 1.5 points.The same model answers the EuroSAT-MS land-cover question for 93.5% of test images in the spatial split.Source: §5.4 · p. 11
  • Removing the Brier term or state-serialization augmentation leaves CORINE accuracy within 0.2 points but raises RSVQAxBEN accuracy by 1.6–1.7 points.The authors conclude these recipe components do not help transfer in this setting.Source: §5.4 · p. 11

6 Limitations

EarthJev’s accuracy remains below the strongest comparison models, and its measured latency depends partly on the inference setup. Its scope is also bounded by closed-option questions, limited training coverage, and benchmark labels and phrasing.

Closed answers only

  • EarthJev handles only questions with declared options; open-ended image descriptions are outside its scope.Applications must map user questions to typed options, using rules or a separate language model that only parses the question.Source: §6 · p. 11

Accuracy

  • EarthJev is 0.6–1.2 points less accurate than the 2B multimodal models and 3.5 points less than EarthDial-4B; its case rests on speed at a small accuracy cost.On trained questions, fixed-vocabulary ResNet-50 is both more accurate and faster.Source: §6 · p. 11

Frozen vision encoder

  • Training freezes the OneJev vision encoder, and the whole-encoder ablation gains 2.5 points with one seed.Training the encoder and testing larger backbones require pre-specified tests with several seeds before any claim.Source: §6 · p. 11

Latency

  • Latency measurements depend on inference software and node load; the generative models’ unoptimized serving engine could narrow the measured gap.Generative-model timings varied by up to 26% between jobs, and the reported job was less favourable to EarthJev.Source: §6 · p. 12

Inputs of the baselines

  • Multimodal language-model baselines see only true-colour composites, so some accuracy differences in either direction may reflect unequal spectral input.This is how those models are used, but it limits attribution of differences to the models alone.Source: §6 · p. 12

Scale and coverage of the data

  • Training uses 5% of reBEN—11,893 images from ten European countries in 2017–2018—and calibration beyond that training distribution is untested.Accuracy still grows with data in the reported ablations.Source: §6 · p. 12

Labels and questions

  • CORINE’s 25 ha minimum mapping unit makes reference masks approximate at 10 m, and templated questions do not cover real-user phrasing; transfer to BigEarthNet.txt is poor for every model.Held-out templates test new phrasings, but not the variety of real user questions.Source: §6 · p. 12

Contamination

  • EarthDial may have seen 3,951 of the 5,991 test images through its BigEarthNet training data, while Qwen and InternVL pre-training data are undisclosed.Possible exposure would favour EarthDial.Source: §6 · p. 12

7 Conclusion

EarthJev reads typed answers from all 12 Sentinel-2 bands without generating text. It is substantially faster than generative models with a modest accuracy gap to the 2B models, while accepting declared option sets and producing calibrated probabilities.

7 Conclusion

  • EarthJev is 6.8 times faster than the same generative backbone and 4.3 to 8.7 times faster than larger multimodal models, at 0.6–1.2 points below the 2B models.Its answers are valid by construction and its probabilities are calibrated without decoding; one-question latency is 1.2–1.6 times that of fixed-vocabulary VQA models.Source: §7 · p. 12
  • The extra spectral bands are used, while the frozen vision encoder is identified as EarthJev’s main accuracy limit.Next steps include training the encoder, evaluating larger backbones, using all of BigEarthNet v2.0, and testing outside the training distribution.Source: §7 · p. 12
  • Because answers cost milliseconds and require no text, the authors propose monitoring with change questions over many Sentinel-2 images and dates as a potential application.The passage frames this as a possible use, not an evaluated result.Source: §7 · p. 12

A Example state and question

The example inference state describes the sensor rather than image content and is shared across patches. It represents a multispectral Sentinel-2 patch, while RGB-only models receive a true-colour sensor description.

A Example state and question

  • The state is identical across patches and contains sensor information, not information derived from the image.The image placeholder is replaced by 64 visual tokens for the 12-band patch.Source: A Example state and question · p. 15
  • Models that accept only RGB receive the same state with the sensor described as a true-colour composite (B4, B3, B2).Source: A Example state and question · p. 15
  • The example specifies 12 Sentinel-2 bands, with 20 m and 60 m bands resampled to 10 m, in a 120 x 120 pixel patch.The patch covers 1.2 km x 1.2 km and is north up.Source: A Example state and question · p. 15

B Question templates

The question templates use CORINE class names at level 1 or 2. Seen templates serve training, validation, and the main test set; unseen templates are reserved for a second test rendering.

B Question templates

  • The templates use CORINE class names at level 1 or 2 for their class placeholders.Source: B Question templates · p. 15
  • Seen templates are used for training, validation, and the main test set, while unseen templates are used only for the second rendering of each test question type.Source: Table 5 · p. 16
Loading 2611.02096v1…