PaperlayerView exact revision on arXiv

Source-linked AI summary

RSJEV: Discriminative Remote Sensing Scene Classification with Multimodal Large Language Models

Dongchen Si, Di Wang, Mingzhen Xu, Jing Zhang, Bo Du, Liangpei Zhang

arXiv:2610.08539v1cs.CV

TL;DR

Remote sensing scene classification faces limitations in fixed-label visual classifiers, static image-text similarity, and autoregressive MLLM generation for explicit candidate sets. RSJEV uses a candidate-conditioned one-pass multimodal decision process, and reports the highest precision, recall, F1, and overall accuracy across three benchmarks alongside lower inference costs than generative MLLMs.

  • Problem

    Visual classifiers use predefined label spaces, CLIP-based methods rely on image-text similarity, and generative MLLMs can require multiple decoding steps for explicit-category classification.

    Source: §I · p. 1

  • Method

    RSJEV jointly models image representations, task instructions, and candidate semantics, using an answer-slot decision representation to score categories without autoregressive generation.

    Source: §I · pp. 1–2

  • Results

    RSJEV achieves the highest precision, recall, F1-score, and overall accuracy across UCM, AID, and NWPU, with F1-scores of 97.72%, 96.30%, and 94.58%, respectively.

    Source: 1) Quantitative Analyses · p. 4

  • Takeaways & Limitations

    The results highlight the potential of JEV-style multimodal discriminative modeling as a decision paradigm for remote sensing and a direction toward more efficient, scalable multimodal foundation models.

    Source: §V · p. 6

Abstract

from arXiv · show

Remote sensing scene classification is a fundamental task in Earth observation and geospatial analysis. Existing approaches mainly follow three paradigms: task-specific visual classification, vision-language similarity matching, and autoregressive multimodal generation. However, visual classifiers rely on predefined label spaces, CLIP-based methods perform recognition through static image-text alignment, and multimodal large language models (MLLMs) introduce unnecessary token-level generation for classification tasks with explicit candidate categories. To address these limitations, we propose RSJEV, a one-pass multimodal decision framework for remote sensing scene classification. Unlike conventional MLLMs that formulate classification as autoregressive text generation, RSJEV reformulates scene classification as a candidate-conditioned multimodal discriminative decision process, where visual representations, task instructions, and candidate category semantics are jointly modeled. Specifically, we introduce a OnePass Decider that extracts multimodal decision states and directly estimates category probabilities within the candidate category space, eliminating autoregressive decoding while preserving vision-language interactions. Extensive experiments on three widely used remote sensing scene classification benchmarks, including UC Merced, AID, and NWPU-RESISC45, demonstrate that RSJEV achieves superior classification performance compared with representative CNN-, Transformer-, Mamba-, CLIP-, and MLLM-based methods. Moreover, RSJEV significantly reduces inference costs and achieves a better accuracy-efficiency trade-off with only a compact 0.8B-parameter model. These results demonstrate the effectiveness of state-conditioned multimodal decision making for efficient remote sensing image understanding. The code will be available at https://github.com/Dongtcs/RSJEV.

I. INTRODUCTION

RSJEV replaces iterative text generation for remote sensing scene classification with candidate-conditioned, one-pass multimodal decisions. The paper reports competitive accuracy with lower inference latency than generative MLLMs across three benchmarks.

I. INTRODUCTION

  • Remote sensing scene classification is challenged by intra-class variability, inter-class visual similarity, and differences in spatial resolution.The task provides scene-level semantic information for geospatial applications.Source: §I · p. 1
  • Task-specific visual classifiers use predefined label spaces and do not explicitly incorporate natural-language category semantics into prediction.The passage notes visually confusing categories such as desert versus barren land and railway versus freeway.Source: §I · p. 1
  • CLIP-style methods use image-text embedding similarity, while generative MLLMs can require multiple decoding steps to produce category answers.The passages identify limited direct interaction in similarity-based prediction and added latency and computational cost from iterative generation.Source: §I · p. 1
  • RSJEV jointly models visual representations, task instructions, and candidate category semantics to make one-pass discriminative decisions.Its OnePass Decider extracts an answer-slot representation and scores candidate options without autoregressive answer generation.Source: §I · pp. 1–2
  • Across three benchmarks, RSJEV achieves competitive accuracy with significantly lower inference latency than generative MLLMs.The introduction reports comprehensive ablations alongside the benchmark experiments.Source: §I · p. 2

II. RELATED WORK

Related work spans task-specific visual models, contrastive vision-language approaches, and generative MLLMs. RSJEV instead conditions a discriminative decision representation on image content, instructions, and candidate categories.

II. RELATED WORK

  • Task-specific CNN, Transformer, and state-space models generally use classification heads over predefined category spaces, limiting adaptability to dynamic and open-vocabulary scenarios.Examples include ResNet, DenseNet, EfficientNet, Vision Transformer, Swin Transformer, and RSMamba.Source: §II · p. 2
  • Contrastive remote-sensing models classify by comparing image and candidate-category text embeddings, rather than jointly conditioning a decision representation on image, instructions, and candidates.RemoteCLIP and SkyScript are cited as domain-oriented image-text alignment efforts.Source: §II · pp. 2–3
  • Generative MLLMs produce autoregressive responses, whereas RSJEV extracts an answer-slot representation and scores candidate options without iterative answer generation.RSJEV conditions this decision on image content, task instructions, and candidate category semantics.Source: §II · pp. 2–3

III. METHODOLOGY

The formulation treats scene classification as choosing among an ordered set of candidate categories, with the image and a question-and-options prompt as inputs. RSJEV estimates probabilities over those candidates and selects the highest-probability option in one forward pass.

III. METHODOLOGY

  • Each labeled sample is formulated from a remote sensing image, an ordered candidate-category list, and the ground-truth category index.The task prompt consists of a classification question and its candidate options.Source: A. Problem Formulation · p. 3
  • Candidate descriptions act as semantic conditions processed jointly with the image, and RSJEV estimates a probability distribution over the candidate options.The probabilities are defined conditional on the image, prompt, and candidate set.Source: A. Problem Formulation · p. 3
  • The predicted category is the option with the highest probability, enabling classification within the supplied candidate space in one forward pass without autoregressive answer generation.Source: A. Problem Formulation · p. 3

B. RSJEV

RSJEV addresses the repeated input paths used by the original JEV for candidate options by making the decision through one input path and one forward pass.

B. RSJEV

  • In the original JEV, the question is expanded into multiple input paths according to the number of candidate options.Source: B. RSJEV · p. 3
  • Separately processing those paths repeats image-feature and textual-task-prompt computation, adding computational overhead.Source: B. RSJEV · p. 3
  • RSJEV completes the task decision with a single input path in one forward pass.Source: B. RSJEV · p. 3

1) Overall Workflow:

RSJEV processes the image alongside an instruction and candidate categories, then predicts directly from a multimodal answer-slot representation. Its OnePass Decider scores candidate options without autoregressive answer generation.

2) Model Components:

  • The multimodal backbone produces contextualized states, and OPD uses the final-layer [RSSC] state conditioned on the image, instruction, and candidate descriptions.Source: 2) Model Components · p. 3
  • OPD selects pretrained language-model-head weights for candidate token IDs and normalizes only those candidate scores into probabilities.Source: 2) Model Components · p. 3
  • The highest-probability option becomes the predicted category, completing classification in one forward pass without autoregressive answer generation.Source: 2) Model Components · p. 3

1) Overall Workflow:

  • Fig. 2 summarizes the one-pass scene-classification workflow, in which the backbone processes the image and a prompt containing the instruction and candidate categories.Source: 1) Overall Workflow · p. 3, Figure 2 · p. 4

IV. EXPERIMENTAL RESULTS AND ANALYSIS

The experiments cover three remote sensing scene-classification benchmarks and compare RSJEV with representative baselines. They also examine OPD across MLLM backbones and compare inference costs with generative MLLMs.

  • The evaluation covers UCM, AID, and NWPU-RESISC45, comparing RSJEV's classification performance with representative baselines.Source: §IV · p. 4
  • The experiments assess OPD's applicability across different MLLM backbones.Source: §IV · p. 4
  • RSJEV is compared with generative MLLM baselines on inference latency, GPU memory consumption, and throughput.Source: §IV · p. 4

A. Dataset Description

The evaluation uses three remote sensing scene-classification datasets under UCM-55, AID-28, and NWPU-28 split settings. The datasets differ in image counts, category counts, and spatial-resolution information reported for UCM and AID.

  • UCM-55, AID-28, and NWPU-28 use 50%, 20%, and 20% of images for training, respectively, with the remainder reserved for testing.Source: A. Dataset Description · p. 4
  • UCM contains 2,100 aerial images across 21 categories, with 100 images per category and approximately 0.3 m-per-pixel resolution.Source: A. Dataset Description · p. 4
  • AID contains 10,000 Google Earth images across 30 categories, with 220–420 images per category and spatial resolutions of 0.5–8 m per pixel.Source: A. Dataset Description · p. 4
  • NWPU contains 31,500 RGB images from Google Earth across 45 categories, with 700 images per category.Source: A. Dataset Description · p. 4

B. Implementation Details

Experiments use Qwen3.5-0.8B and train for 10 epochs with AdamW, cross-entropy loss, and Brier score regularization. Evaluation uses precision, recall, F1-score, and overall accuracy.

  • Qwen3.5-0.8B is trained for 10 epochs on eight NVIDIA RTX A40 GPUs, using AdamW with a 1 × 10−4 learning rate and cross-entropy loss plus Brier score regularization.Source: B. Implementation Details · p. 4
  • Training uses a total batch size of 128, a maximum input length of 1536 tokens, random image flipping, label smoothing of 0.05, and FP32 precision.Source: B. Implementation Details · p. 4
  • Evaluation metrics are precision (P), recall (R), F1-score (F1), and overall accuracy (OA).Source: B. Implementation Details · p. 4

C. Comparison Experiments

RSJEV leads the reported comparisons on three remote-sensing scene-classification benchmarks, improves performance across two MLLM backbones, and reduces inference costs relative to evaluated generative MLLMs.

C. Comparison Experiments

  • The comparison covers CNN-, Transformer-, Mamba-, vision-language, and multimodal large language model approaches, using pretrained weights or checkpoints fine-tuned for scene classification.The listed baselines include ResNet-101, Swin Transformer, RSMamba, Open-CLIP, and Qwen3-VL.Source: C. Comparison Experiments · p. 4

1) Quantitative Analyses:

  • RSJEV posts F1-scores of 97.72% (UCM), 96.30% (AID), and 94.58% (NWPU), beating OSCLIP, Qwen3-VL, and VMamba by 0.39, 0.78, and 0.41 percentage points, respectively.It also achieves the highest precision, recall, and overall accuracy across all three benchmarks.Source: 1) Quantitative Analyses · p. 4, Table I · p. 5

D. Generalizability of OnePass Decider

E. Inference Efficiency Analysis

  • On one A40, RSJEV uses 1.68 GB and 102.18 ms per image, with 9.79 FPS versus generative baselines’ 1.31–4.09 FPS.The reported comparison measures GPU memory and latency alongside throughput; RSJEV has the lowest memory use and latency among evaluated methods.Source: E. Inference Efficiency Analysis · p. 5

V. CONCLUSION

The conclusion presents RSJEV as a candidate-conditioned, one-pass framework for remote sensing scene classification. It reports strong classification performance and improved inference efficiency over generative MLLM baselines, with cross-backbone experiments supporting OnePass Decider’s generalizability.

V. CONCLUSION

  • RSJEV jointly models visual representations, task instructions, and candidate category semantics for remote sensing scene classification.OnePass Decider estimates candidate probabilities from the final-layer hidden state at the classification answer slot, avoiding autoregressive answer decoding.Source: §V · pp. 5–6
  • Experiments report strong classification performance and substantially improved inference efficiency over generative MLLM baselines.Cross-backbone experiments further validate OnePass Decider’s effectiveness and generalizability across MLLM architectures.Source: §V · p. 6
  • The conclusion presents JEV-style multimodal discriminative modeling as a potential direction for more efficient and scalable multimodal foundation models.This is framed as a promising direction for remote sensing.Source: §V · p. 6

Related papers

Loading 2610.08539v1…