Source-linked AI summary
Lightweight Adaptation of General-Purpose VLMs for Multispectral and SAR Image Understanding
Shanji Liu, Kelu Yao, Junxiao Xue, Chenghui Lv, Xiangyang Miao, Yekai Huang, Yaying Chen, Chao Li
TL;DR
General-purpose VLMs are not directly suited to native multispectral and SAR inputs, while conventional adaptation often requires dedicated encoders and domain pretraining. The paper renders observations as named optical and SAR views, adapts language and selected visual blocks with LoRA, and adds evidence-aware supervision and preference pairs. On the primary benchmark, Qwen3-VL reaches 0.8275 micro F1, with improvements also reported across four architectures and two transferred tasks.
Problem
General-purpose VLMs built for three-channel natural images do not directly accommodate native multispectral and SAR observations, and adapting them commonly requires dedicated encoders and domain pretraining.
Method
The paper renders each observation as five named optical views and one named SAR view, then applies LoRA to the language network and selected visual transformer blocks with structured evidence supervision and retained-evidence preference pairs.
Results
0.8275 micro F1 is achieved by adapted Qwen3-VL on the six-class benchmark, while the same protocol improves all four tested VLM architectures and transfers to flood verification and captioning.
Takeaways & Limitations
Rendered sensor views and compact adapters provide a route for repurposing general-purpose VLMs for multispectral and SAR tasks without training a new foundation model.
Abstract
from arXiv · showhide
General-purpose vision-language models (VLMs) now support strong visual recognition, instruction following, and generation. However, most pretrained visual encoders are built around three-channel natural images and do not directly accommodate observations such as native multispectral measurements or synthetic aperture radar (SAR). Adapting VLMs to these sensors typically requires dedicated encoders and domain pretraining, slowing the reuse of stronger general-purpose checkpoints. We show that the multi-image interface of general-purpose VLMs offers a lightweight alternative. Our protocol renders each observation as five optical views and one SAR view, names them in the prompt, and adapts the language network and selected visual transformer blocks with LoRA. This exposes band composites, spectral indices, and radar backscatter through an existing visual interface. For land-cover recognition, structured supervision couples predicted classes with sensor evidence. We further construct preference pairs in which a true label is omitted while its supporting evidence is retained, encouraging complete predictions that remain consistent with the observations. On a balanced six-class land-cover benchmark derived from BigEarthNet-v2, the adapted Qwen3-VL reaches 0.8275 micro F1. The same input and adaptation protocol improves all four tested VLM architectures and transfers to Sen1Floods11 flood verification and BigEarthNet.txt captioning. Image removal and mismatch controls show that the adapted models use the supplied sensor observations. Together, these results demonstrate that VLMs can be repurposed for multispectral and SAR tasks through rendered inputs and compact LoRA adaptation, without training a new foundation model.
1 Introduction
The paper addresses the sensor input gap between general-purpose VLMs and multispectral or SAR observations by rendering sensor measurements as named images in an existing multi-image interface. Compact LoRA adaptation, structured evidence supervision, and preference construction improve recognition while supporting transfer across models and tasks.
- Motivation: General-purpose VLMs cannot directly receive native MSI and SAR channel structures through standard RGB image interfaces.Native MSI measures reflectance across visible, near-infrared, and shortwave-infrared bands, while SAR measures microwave backscatter.
- Motivation: Native multispectral and SAR support often requires new encoders or projections, substantial pretraining, specialized data, and considerable computation.This limits reuse of rapidly improving general-purpose VLM checkpoints.
- Approach: The protocol supplies five named optical renderings and one named SAR rendering through the existing multi-image interface, updating language and selected visual blocks with LoRA.Renderings include interpretable composites and spectral index maps, while names connect each view to its measurement.
- Approach: Structured responses separate class predictions from optical and radar evidence, enabling multilabel evaluation and textual consistency inspection.Supervised fine-tuning teaches label mappings, sensor vocabulary, and response format; preference pairs target omitted labels while retaining supporting evidence.
- Results: 0.8275 Qwen3-VL micro F1 is achieved with retained-evidence DPO after named-view SFT raises micro F1 from 0.5921 to 0.8242.The same adaptation route improves all four tested VLM architectures and transfers to Sen1Floods11 verification and BigEarthNet.txt captioning.
2 Related Work
Prior remote-sensing research includes specialist and foundation models, remote-sensing VLMs, guided multispectral prompting, and parameter-efficient preference optimization. These lines of work motivate adapting general-purpose models while highlighting the limited isolation of lightweight MSI-and-SAR adaptation.
- VLMs for remote sensing: Remote-sensing VLMs extend models to retrieval, dialogue, grounding, instruction following, multisensor tasks, and pixel-grounded tasks.Their heterogeneous inputs, tasks, and response formats do not isolate lightweight adaptation.
- Foundation models for remote sensing: Remote-sensing foundation models use contrastive learning, masked image modeling, flexible modality inputs, and multimodal or multitemporal Sentinel data.Examples include Prithvi, SkySense, CROMA, and TerraFM.
- Guided multispectral inputs for generalist models: Guided multispectral prompting converts Sentinel-2 bands into true-color, false-color, and spectral-index images for generalist multimodal models.BigEarthNet and related archives support large-scale multilabel recognition with Sentinel-1 and Sentinel-2 imagery.
- Preference optimization and evidence supervision: LoRA, supervised post-training, and DPO provide parameter-efficient adaptation routes, while existing preference objectives generally omit structured outputs linking labels to sensor evidence.Multimodal DPO work also emphasizes conditioning preference learning on the image.
3 Method
The method converts aligned Sentinel-2 and Sentinel-1 observations into named three-channel views and adapts a VLM with LoRA for structured classification and evidence generation. Supervised and preference training target complete labels whose textual evidence remains consistent with optical and radar inputs.
- Task Formulation: The task merges 18 BigEarthNet-v2 labels into six land-cover groups spanning built, vegetated, agricultural, aquatic, and mixed regimes.Samples without mapped labels and the “Beaches, dunes, sands” category are excluded before prompt and evidence construction.
- Named Sensor Views: Each input uses aligned Sentinel-2 MSI and Sentinel-1 SAR rendered as true color, false color, SWIR, NDVI, NDBI, and SAR images.Every rendering is a three-channel 224×224 image compatible with the pretrained image interface.
- Named Sensor Views: Fixed view order and adjacent sensor names identify each measurement, while identical pixels across naming controls isolate the effect of textual view labels.Classification and captioning share the renderings but differ in instruction and target format.
- Instruction Construction: SFT trains a two-line response containing the complete class set and corresponding optical and radar evidence.The model input combines the task instruction with the ordered multimodal sequence, and parsing uses canonical class names plus aliases.
- Parameter-Efficient Adaptation: Rank 16 LoRA modules adapt the language network and selected visual transformer blocks while the original VLM parameters remain fixed.Visual LoRA parameters use 0.3 times the language learning rate, with 51.35M trainable parameters including 7.70M in the visual transformer.
- Preference Data for Underprediction Errors: Retained-evidence preference pairs remove one true label from the rejected response while preserving its optical and radar cues, creating a class–evidence disagreement for DPO.Matched controls remove the label and associated cues, while pair sampling and recognition error are held fixed.
- Evaluation Metrics: The textual evidence audit requires both optical/MSI and SAR terms, source–cue compatibility, and at least one matching cue for every predicted class.A separate blinded assessment rates source compatibility, physical plausibility, cue coverage, unsupported cues, and judgeability.
4 Experiments
The experiments evaluate named rendered MSI and SAR views with LoRA adaptation across benchmarks, architectures, and controlled image interventions. Results show strong land-cover performance, transfer to other tasks and models, and measurable dependence on supplied imagery.
- Input Controls: 0.8048 micro F1 is obtained with individually named views, compared with 0.7877 for generic indices and 0.7863 for sensor-family names.The matched comparison uses the same six images, sample order, and targets.
- Main Results: 0.0121 exact-match improvement and 0.0034 micro F1 improvement result from retained-evidence DPO over its SFT initialization.The evidence audit remains statistically unchanged, with a 95% CI for the difference of [−0.0031, 0.0025].
- Human Evidence Assessment: 0.893 physical-plausibility, visible-cue-coverage, and unsupported-cue absence rates with images fall to 0.321 without images in blinded assessment.The paired plausibility gap is 0.571 with a 95% CI of [0.357, 0.769].
- Transfer Across Architectures: 0.1761 to 0.2529 micro F1 gains occur across all four tested architectures under one-epoch SFT with named views, with audit rates above 0.93.The fixed protocol separates architecture transfer from the two-epoch Qwen3-VL main model.
- Transfer Across Tasks and Datasets: 0.7714 flood-verification F1 and 0.8202 accuracy are reached on Sen1Floods11 after LoRA adaptation, versus 0.5294 and 0.5506 before adaptation.Complete MSI+SAR input improves class F1 by 6.4 points and accuracy by 11.8 points over MSI alone.
- Transfer Across Tasks and Datasets: 0.5654 BigEarthNet-19 concept F1, 42.07 SacreBLEU, and 0.5337 ROUGE-L are achieved for BigEarthNet.txt captioning with the same language and visual modules.Removing images lowers these scores to 0.2753, 31.88, and 0.4421, respectively.
- Image Dependence: 0.2439 micro F1 is lost when all images are removed, while replacing optical views with a donor patch reduces micro F1 by 0.3328.Replacing SAR changes 5.8% of predictions, whereas optical content carries most recognition signal for the six-class taxonomy.
5 Conclusion
The paper presents a lightweight route for adapting VLMs to multispectral and SAR land-cover recognition. Explicit view names and compact adapters support reuse across architectures and tasks while retaining structured evidence evaluation.
- Named renderings with language and visual LoRA raise Qwen3-VL micro F1 from 0.5921 to 0.8242.
- Retained-evidence DPO reaches 0.8275 micro F1 while maintaining the structured evidence audit.
- The same adaptation protocol improves four VLM architectures and transfers to flood verification and land-cover captioning.
- Explicit view names connect each rendering to its measurement, while compact adapters preserve the reusable base checkpoint.
- The approach is positioned as a practical route for bringing VLM advances to MSI and SAR applications.
A Additional Experimental Details
The appendix provides detailed BigEarthNet-v2 diagnostics, transfer protocols, and results omitted from the main paper for space.
- The appendix contains detailed BigEarthNet-v2 diagnostics, transfer protocols, and results omitted from the main paper for space.
Dataset and Training Details
The experimental details use BigEarthNet-v2 data with aligned MSI and SAR products, alongside a fixed LoRA adapter configuration for efficient adaptation.
- Training uses 12,900 examples from the official train assignment, while 3,234 test examples support configuration selection and exploratory comparisons.
- The fixed adapter updates language-network attention and feedforward projections plus selected visual transformer blocks while keeping original VLM weights fixed.
- The adapter contains 51.35M trainable parameters, including 7.70M in the visual transformer.
Target-Label Mapping
The target-label setup maps BigEarthNet-v2 labels into six coarse classes and evaluates selected checkpoints on an official validation sample.
- Table 6 maps 18 BigEarthNet-v2 labels to six target classes.
- Beach, dune, and sand samples without another mapped label are removed from the target taxonomy.
- Selected zero-shot, SFT, and DPO checkpoints are evaluated once on the official validation sample.
- Counts exceed the number of examples because the task is multilabel.
Class Parsing
Recognition scoring extracts the class line, normalizes common label variants, and applies a fallback matcher when the requested format is incomplete.
- The parser lowercases the class line and maps common label variants to six canonical land-cover labels.
- If the class line is absent, alias matching over the full response supports recognition scoring while leaving the response format invalid.
Rendering Protocol
The rendering protocol converts multispectral and SAR measurements into standardized RGB views that preserve sensor-specific visual encodings.
- All Sentinel-2 bands are resampled to a common grid before rendering.
- Channels are clipped to scene-level 2nd–98th percentile ranges and linearly mapped to 8-bit intensity.
- NDVI and NDBI become three-channel pseudocolor maps using a fixed diverging color map centered at zero.
- The SAR view combines log-scaled VV, VH, and VV−VH backscatter into a three-channel false-color composite.
- Every VLM input is therefore a standard RGB image whose colors encode indices or radar backscatter values.
Instruction and Evidence Templates
The protocol asks for a fixed two-line multilabel response, pairing predicted classes with evidence tied to named optical and radar views.
- Training prompts request the same multilabel question without naming a target class.
- The six aligned views are named as true color, false color, SWIR, NDVI, NDBI, and SAR radar.
- Responses must contain exactly two lines: class: ... and evidence: ... .
- Evidence templates connect land-cover classes to sensor cues such as built-up texture, vegetation indices, field parcels, and radar texture.
- The evidence pool is grammar-cleaned while preserving class lists, source names, and cue terms, then manually audited.
Evidence Dictionaries
Evidence evaluation combines fixed source-and-cue dictionaries, blinded plausibility ratings, and controlled comparisons of naming, images, adaptation, and task transfer.
- Evidence Dictionaries: The audit uses fixed case-insensitive dictionaries for optical/MSI terms, SAR terms, and class-specific cues.
- Evidence Audit: Evidence passes source coverage only when it contains at least one optical/MSI form and one SAR form.
- Blinded Human Evidence Assessment: Annotators rate sensor compatibility, physical plausibility, class-cue coverage, unsupported cues, and scene ambiguity without seeing model identity or labels.
- Blinded Human Evidence Assessment: 0.893 physical plausibility, complete coverage, and absence of unsupported cues are obtained with images, versus 0.321 without images.
- Preference Optimization: 0.0034 micro F1 and 0.0121 exact match are DPO improvements under paired bootstrap estimates, while evidence audit changes by −0.0003.
- View Naming Controls: Individual view names reach 0.8048 micro F1, versus 0.7877 with generic indices and 0.7863 with family names.
- Image Dependence Controls: 0.3373 micro F1 is lost when all six images are replaced, with 93.6% of predicted label sets changing.
- Preference Optimization: 0.8275 micro F1 is achieved by DPO, compared with 0.8242 for its SFT initialization on the official validation sample.