PaperlayerView exact revision on arXiv

Source-linked AI summary

Restrict, Don't Retrain: Inference-Time VLM Guidance for Zero-Shot Aerial Segmentation

Teresa DiMeola, Charles Walter, Hong Xiao

arXiv:2609.00628v1cs.CVcs.AI

TL;DR

Aerial segmentation requires reliable pixel-level localization, but general foundation models may miss relevant classes and small objects. The paper uses a frozen dense-labeling model guided at inference time by two VLM queries for class selection and minority-object localization, yielding positive and highly significant ΔmIoU where the backbone is competent across four datasets without domain-specific training.

  • Problem

    General foundation models can miss scene-relevant classes and small important objects, while domain-specific alternatives require costly training and become stale when class needs change.

  • Method

    A frozen foundation model labels pixels while a VLM supplies structured, inspectable guidance on which classes to weight and where small rare objects are located.

  • Results

    ΔmIoU is positive and highly significant across datasets where the backbone is competent, with the workflow producing better zero-shot segmentation without domain-specific training.

  • Takeaways & Limitations

    Structured VLM guidance can steer and patch a frozen segmentation model while keeping each judgment traceable through readable lists or boxes.

  • Takeaways & Limitations

    Spatial grounding in commercial VLMs on aerial imagery remains an open problem, and viewpoint variation can leave MCI with little to add.

Abstract

from arXiv · show

Global welfare often depends on the correct interpretation of aerial and satellite imagery. Acting on such imagery (mapping flooded ground, crop extent, or damaged infrastructure) demands pixel-level segmentation to ensure perfect class localization. Pretrained general foundation models, when applied directly, often miss important features and cannot always find all the classes belonging to a given scene, overlooking smaller objects that matter most. We use a single consumer-grade GPU running a vision-language model (VLM) to supply this missing guidance, improving segmentation while producing structured, auditable evidence that drives the result and can be inspected on its own. We fuse three approaches: the frozen foundation model that labels every pixel, and two queries to a VLM, one to choose the classes that matter, and one to locate the small objects the base model misses. Evaluating across four aerial datasets, we see consistent gains at each stage where the base model is competent.

1. Introduction

Aerial imagery must be converted into pixel-level structured information, but general models need inference-time guidance to select relevant classes and recover small objects. The proposed workflow combines a frozen pixel-labeling model with VLM-based class weighting and minority-object localization.

  • Semantic segmentation assigns every aerial-image pixel an object class for decisions such as flood mapping, crop mapping, and infrastructure inspection.
  • Domain-specific foundation models require labeled data, compute, expertise, and separate retraining when the relevant class set changes.
  • Inference-time guidance addresses a selection problem: general models lack knowledge of which vocabulary applies and where small important objects are.
  • The workflow uses a frozen foundation model for dense pixel labeling and a VLM that returns structured scene cues without painting pixels.
  • Automated Class Weighting boosts scores for image-present classes, while Minority Class Identification boxes and fills small objects the weighted classifier misses.
  • Across four aerial datasets, the study derives principles favoring vocabulary restriction, majority-class weighting, and VLM recovery of dropped small classes.

2. Related Work

Related work increasingly uses large domain-specific models, open-vocabulary methods, and multimodal models for remote-sensing interpretation. This paper situates its workflow within inference-time adaptation and structured, inspectable decision support rather than weight changes or opaque reasoning narratives.

  • Remote-sensing VLM research has shifted toward large models trained from scratch with substantial data, energy, and hardware costs.
  • Zero-shot segmentation requires assigning a class label to every pixel, including classes absent from training, while transfer accuracy can fall without fine-tuning or retraining.
  • Open-vocabulary methods accept user-named classes, whereas vocabulary-free methods invent names but can produce inconsistent labels and unpredictable specialized-domain outputs.
  • Inference-time adaptation changes model outputs rather than weights; prior work, for example, adjusts per-pixel logits to address class imbalance.
  • Multimodal language models have been used for fine-grained scene description and image-aware annotation.
  • Workflow-oriented decision support fuses heterogeneous model outputs into structured features that downstream tools can use, while design-science framing emphasizes reusable principles and inspectable outputs.

3. Design and Methods

The method uses two frozen models in a three-stage workflow: a foundation model provides dense labeling over a curated 150-class vocabulary, while VLM queries select relevant classes and locate missed small objects. Inference-time weighting and mechanical mask merging provide inspectable guidance without changing model weights.

  • Design and Methods: The workflow combines a frozen foundation model for dense labeling with a VLM that supplies class-selection and small-object localization guidance.The VLM acts as an interpretive layer rather than painting pixels itself.
  • Stage 1: Vocabulary Restriction: Stage 1 restricts inference to a hand-adjusted 150-class vocabulary and fine-tunes on each dataset’s annotated classes while finding remaining classes zero-shot.The vocabulary is adapted toward infrastructure, natural features, vehicles, land use, and specialized aerial objects.
  • Stage 2: Automated Class Weighting (ACW): Automated Class Weighting selects image-relevant classes from the 150-class vocabulary and boosts their per-pixel logits before softmax.The VLM returns a structured class list and weight dictionary; no model weights are changed.
  • Stage 2: Automated Class Weighting (ACW): ACW produces complete per-pixel segmentation, with localization remaining entirely in the base model until bounding boxes enter in Stage 3.The VLM supplies class sets and weights but no spatial information during this stage.
  • Evaluation: Class-selection quality is evaluated with precision, recall, and F1 against the ground-truth class set present in each image.The VLM prompt requests reasoning over global and local scene detail and a parseable list and weight dictionary.
  • Stage 3: Minority Class Identification (MCI): Minority Class Identification asks the VLM for image-coordinate boxes around small objects that Stage 2 still misses, then fills and merges those boxes into the segmentation.Existing classes are updated only when the new mask scores better by per-class IoU, while absent classes are painted in directly.

4. Experimental Setup

The study evaluates zero-shot transfer across four aerial benchmarks using cross-dataset testing and consumer-hardware constraints. It also screens several VLMs for spatially grounded bounding-box generation, selecting Qwen-VL for MCI without a quantitative comparison.

  • Datasets: The evaluation covers UAVid, Aeroscapes, DroneSeg, and UDD6, spanning 6 to 24 annotated classes across urban, campus, rooftop, and diverse aerial imagery.The datasets contain 8, 11, 24, and 6 classes, respectively.
  • Evaluation Protocol: For each dataset pair, the model is fine-tuned on dataset A and tested zero-shot on dataset B, with the 150-class vocabulary mapped to the target ground-truth classes for mIoU.Ground truth is used only for evaluation metrics, not for weighting, prompt construction, or MCI.
  • Evaluation Protocol: The primary evidence is consistency of ΔmIoU across stage transitions and datasets rather than absolute mIoU on any single dataset.Absolute mIoU is treated as a conservative lower bound because catch-all annotations penalize finer predictions.
  • Implementation: Experiments use one NVIDIA RTX A6000 with memory capped at 16 GB to simulate consumer-hardware constraints.Stage 1 fine-tuning takes 1 to 2 hours per dataset on a single GPU, and Stage 2 uses three seeds per dataset and configuration.
  • VLM Screening: Qwen-VL is the only screened model that produced spatially grounded bounding boxes suitable for MCI; GPT-4o, Claude Sonnet, and Gemini produced displaced boxes.The authors did not perform quantitative ranking because the screening gap was qualitatively decisive.

5. Results

Across four aerial datasets, the staged workflow improves segmentation most when the backbone is competent, while viewpoint and annotation granularity constrain later-stage gains. ACW and MCI address complementary failure modes, with MCI recovering minority classes that dense segmentation misses.

  • Viewpoint effects: UDD6 benefits from nearly nadir imagery, whereas UAVid’s varying field of view and oblique angles cause systematic difficulty grounding objects.The UAVid figure shows predicted boxes drifting from targets, leaving Stage 3 unable to improve Stage 2.
  • Minority-class recovery: Stage 3 exceeds the per-image Oracle because MCI recovers classes visible to the VLM but not pixel-localized by the segmentation algorithm.The VLM returns boxes for such classes, which are filled and merged with Stage 2.
  • UDD6: 37.12% mIoU on UDD6 follows 21.64% unweighted and 17.99% weighted results, showing MCI supplied almost all of the improvement.The scene is dominated by small classes, so minority-class recovery outweighed the lack of ACW benefit.
  • Cross-dataset results: The pipeline’s gains are large and significant on Aeroscapes and DroneSeg, while UDD6 and UAVid remain near flat for distinct reasons.The evaluation reports cumulative mIoU across stages and emphasizes consistency of stage-transition changes across datasets.
  • Cross-dataset results: +15.07 mIoU on Aeroscapes and +14.68 on DroneSeg mark highly significant Stage 1→3 gains (p < 0.0001).The corresponding Stage 1→3 changes on UAVid (+1.18) and UDD6 (−1.89) are not significant.
  • Complementary stages: ACW and MCI target complementary failure modes: weighting helps early on Aeroscapes and DroneSeg, while MCI can carry performance when weighting does not.On UDD6, ACW produces no gain and the full Stage 1→3 difference is −1.89, attributed to its six coarse annotation classes.

6. Ablation Studies

The ablations show that ACW’s benefit comes primarily from selecting the right classes, not from increasing the boost magnitude, while automated VLM selection is stable and competitive with oracle selection.

  • 6.1. ACW Weight Value: ACW performance is stable from moderate weighting through w = 500, so the study adopts w = 100 for subsequent experiments.Nearly all gains appear at the first nonzero weight; UDD6 is the exception, where the unweighted baseline reaches 27.52 mIoU versus 26.7–26.9 for weighted settings.
  • 6.1. ACW Weight Value: On UDD6, ACW is inert because the unweighted baseline of 27.52 exceeds every weighted value.This indicates that restriction does not universally improve performance across datasets.
  • 6.2. Class Selection Strategy: The advantage over random selection widens on weaker-backbone and smaller-vocabulary datasets but shrinks when the backbone already predicts common aerial classes well.This makes class-selection precision especially important where the base model is less capable.
  • 6.2. Class Selection Strategy: Selection quality, rather than boosting alone, determines whether ACW helps: wrong-class boosting can suppress correct predictions on UAVid and UDD6.Random selection is worse than no weighting on those datasets and varies substantially across seeds.
  • 6.2. Class Selection Strategy: The VLM matches oracle selection to within about 0.6 mIoU on every dataset and exceeds it on UAVid, reaching 27.14 versus 25.40.The authors associate this with the VLM sometimes naming useful classes omitted by ground-truth class sets; its low variance also indicates stable automated selection.

7. Conclusion

The paper concludes that a frozen foundation model paired with a VLM can improve scene-coherent aerial segmentation without domain-specific training. The workflow remains lightweight and auditable, but VLM localization accuracy is still its principal obstacle.

  • 7. Conclusion: Pairing a frozen foundation model with a scene-reading VLM produces more coherent segmentation without domain-specific training.The VLM supplies structured guidance while the foundation model performs the pixel labeling.
  • 7. Conclusion: ∆mIoU is positive and highly significant across datasets where the backbone is competent, while flat cases are attributed to benchmark or viewpoint limits.The conclusion frames the gains as consistent across the relevant datasets rather than tied to one benchmark choice.
  • 7. Conclusion: The workflow can run with any local general foundation model, without domain-specific pretraining or more than one consumer-grade GPU.If pretraining missed rare or new classes, the authors state that simple fine-tuning on those classes is sufficient.
  • 7. Conclusion: Automated class weighting and minority-class identification provide structured cues that guide the frozen model without performing pixel labeling themselves.MCI pairs VLM-derived class and bounding-box information with a class-agnostic fill step to address minority-class under-representation.
  • 7. Conclusion: VLM localization remains the principal obstacle because the model can name scene classes and objects but cannot yet localize them reliably.Improving localization is identified as future work, while transfer to medical and industrial imaging remains a longer-term aim.
Loading 2609.00628v1…