PaperlayerView exact revision on arXiv

Source-linked AI summary

Aperture: Training-Free Multiscale Concept Bottlenecks for Remote Sensing

Rishabh Mondal, Nipun Batra, Utkarsh Mall

arXiv:2609.38603v1cs.CV

TL;DR

Existing training-free CBMs struggle with satellite image recognition, while remote-sensing models remain short on interpretability for expert users. APERTURE combines MLLM-derived continuous concept evidence with greedy multiscale quadtree search and blends global and home-scale scores. On SiFC All, combining home- and global-scale results yields the best F1-score, 80.19%, beating the best-performing baseline LaZSL by more than 10 pp.

  • Problem

    Existing training-free CBMs struggle with satellite image recognition, while remote-sensing models remain short on interpretability for expert users.

    Source: §1 · p. 1

  • Method

    APERTURE combines MLLM-derived continuous concept evidence with greedy multiscale quadtree search and blends global and home-scale scores.

    Source: §1 · p. 2, §4.3 · p. 5, §4.5 · p. 6

  • Results

    On SiFC All, combining home- and global-scale results yields the best F1-score, 80.19%, beating the best-performing baseline LaZSL by more than 10 pp.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    APERTURE achieves state-of-the-art training-free performance on SiFC and FMoW and outperforms supervised CBMs on out-of-region images.

    Source: §1 · p. 2, §6 · p. 9

    Strong performance still requires blending home-scale predictions with global scores; home-scale evidence alone is not yet sufficient.

    Source: §6 · p. 9

Abstract

from arXiv · show

While earth observation models have advanced substantially, they still lack interpretability. While concept-bottleneck models provide interpretability and expert interaction, they are either too expensive to train for the remote sensing domain or perform poorly without annotation. We posit that in expert domains like remote sensing, such training-free models require both fine details in both image and concept space. In image space, we propose a multiscale concept bottleneck using greedy quadtree routing to locate small concepts. In concept space, we replace contrastive vision language models with pre-trained MLLMs and present a way to get reliable concept scores from them. We introduce APERTURE that blends concept scores at the global image and native concept-scale level to give state-ofthe-art training-free model performance. To test these models, introduce SiFC, a fine-grained concept-centric dataset across three countries, with human-reviewed class-level concept maps. On SiFC, APERTURE outperforms the best training-free baselines by more than 10 percentage points in macro F1-score, and notably also outperforms supervised concept bottleneck models. Targeted component-removal tests examine whether concept scores respond to changes in visual evidence, while temporal experiments show that descriptor updates improve recognition of technological changes without retraining.

1 INTRODUCTION

Training-free concept bottlenecks suit expert domains with limited labels, but existing methods struggle with fine-grained satellite recognition. APERTURE combines MLLM-based concept scoring with multiscale image search, and SiFC benchmarks fine-grained facility recognition.

1 INTRODUCTION

  • Training-free CBMs avoid expensive concept annotations and support adding classes and concepts at deployment, but existing methods struggle on satellite imagery.Remote-sensing experts can understand and intervene in ante-hoc model decisions.Source: §1 · pp. 1–2
  • MLLMs replace contrastive VLMs for domain-specific concepts, with normalized Yes/No next-token probabilities providing continuous evidence instead of unreliable reported confidence.CBMs require continuous concept scores, while generative MLLMs naturally produce discrete presence/absence decisions.Source: §1 · p. 2
  • Greedy quadtree routing searches one crop at each zoom level, reducing search time from exponential to linear in depth.Exhaustively splitting into four quadrants across four levels would require 341 times as many region evaluations as global evaluation.Source: §1 · p. 2
  • Known ground sampling distance associates concepts with a characteristic physical home scale, allowing search to stop at an appropriate resolution.The figure illustrates how resizing a 4096-pixel coal-plant image shrinks its yard and chimney to 112 and 14 pixels.Source: Figure 1 · p. 1, §1 · p. 2
  • SiFC and a fine-grained FMoW subset support evaluation across world regions, where relevant concepts may occupy very different spatial extents.SiFC contains 800 images from six industrial classes; the coal-plant example shows a chimney occupying less than 0.4% of the image.Source: §1 · p. 2
  • APERTURE achieves state-of-the-art performance among training-free CBMs on SiFC and FMoW, and outperforms supervised CBMs on out-of-region images.SiFC is a concept-centric benchmark for fine-grained recognition and geographic shifts.Source: §1 · p. 2

2 RELATED WORKS

Prior work spans post-hoc explanations and ante-hoc concept models, including label-free and training-free CBMs. For satellite imagery, existing interpretability approaches do not expose how intermediate concepts contribute, while SiFC targets fine-grained compositional facility recognition.

Interpretable recognition

  • Post-hoc methods explain predictions without guaranteeing faithful insight, motivating ante-hoc models that incorporate human-readable concepts into prediction.Standard CBMs use an intermediate concept layer but require class labels and costly image-level concept annotations.Source: §2 · p. 3
  • Label-free CBMs remove concept annotations but still learn a class predictor, whereas training-free CBMs remove target-specific fitting and labeled-image requirements.The passage identifies label-free methods as Oikarinen et al. (2023) and Yang et al. (2023), Srivastava et al. (2024), and training-free approaches as classification-by-description methods.Source: §2 · p. 3

Interpretable recognition in satellite imagery

  • Satellite interpretability methods highlight regions or generate textual reasoning, but do not explicitly expose intermediate concept contributions.PCINet uses LLM-generated concepts but evaluates broad scene categories; SiFC instead targets facilities distinguished by combinations of shared parts and attributes across regions.Source: §2 · p. 3

3 SATELLITE IMAGERY FACILITY CONCEPTS (SIFC)

SiFC is designed for concept-based classification of industrial facilities, whose classes depend on combinations of constituent parts and attributes. Its geographically diverse images and class-level concept maps support training-free and out-of-region evaluation.

3 SATELLITE IMAGERY FACILITY CONCEPTS (SIFC)

  • Industrial facilities suit compositional recognition because their classes can be characterized by the presence, arrangement, and attributes of recurring functional components.Existing remote-sensing datasets largely focus on coarse or relatively atomic scene and land-cover categories.Source: §3 · p. 3
  • SiFC contains 800 samples from the USA, India, and China across six facility classes, with human-annotated class concept maps and country-specific component descriptions and layout.Images are 4096 × 4096 pixels at zoom level 18, covering approximately 4 km2; coordinates and retrieval scripts are released instead of imagery.Source: §3 · pp. 3–4
  • SiFC supports direct zero-shot classification, training-free and concept label-free CBMs, and evaluation of out-of-region generalization.The geographic diversity captures variation in facility appearance, surrounding land use, and component layout.Source: §3 · p. 4

4 METHOD

APERTURE builds a training-free concept bottleneck for remote sensing by adapting concept recognition to domain-specific language and spatial scale. It combines scores from the full image and concept-specific crops.

4 METHOD

  • APERTURE routes each class- and country-conditioned descriptor to its home scale and blends global and home-scale scores into a concept score.The class score averages the resulting concept scores.Source: Figure 3 · p. 4, §4 · p. 4

4.1 BACKGROUND

  • Training-free CBMs avoid target-image labels, concept annotations, and target-data optimization, while supervised CBMs rely on costly concept-annotated examples.The paper selects training-free CBMs because data collection is expensive and notes their capacity for concept-level correction without retraining.Source: §4.1 · pp. 4–5

4.2 PROBLEM SETUP AND OVERVIEW

  • Satellite facilities contain discriminative structures at different scales, while contrastive models struggle with local attributes and expert-domain concepts.The method replaces CLIP-like VLMs with MLLMs and recognizes concepts at multiple scales.Source: §4.2 · p. 5

4.3 FROM CLIP TO MLLM

  • Temperature-scaled probabilities for yes and no answers provide continuous concept scores, unlike unreliable verbalized confidence estimates.The method uses normalized next-token probabilities to estimate concept evidence.Source: §4.3 · pp. 5–6

4.4 CHANGES TO THE CONCEPT DEFINITIONS

  • Descriptors are expanded into detailed visual sentences, conditioned on class and country, and assigned concept-specific home scales.The structured descriptions add appearance, placement, and likely confusers; class and country conditioning avoids prepending the class label.Source: §4.4 · p. 6

4.5 CONCEPT-DEPENDENT SPATIAL SCALE.

  • Greedy quadtree routing follows one quadrant per level to a concept’s home scale, reducing calls from 341 to 5 at 256-pixel crops.Exhaustive search requires O(4^ℓk) calls, whereas greedy routing requires O(ℓk + 1).Source: §4.5 · p. 6
  • The final concept score linearly blends global-image and home-scale crop scores, preserving facility layout alongside small components.The home-scale contribution is controlled by α, and class scores average the blended concept scores.Source: §4.5 · p. 6

5 RESULTS

APERTURE combines global and home-scale concept predictions with MLLM-based scoring; results compare it with training-free and supervised baselines across datasets, regions, and ablations.

5.2 DISCUSSION

  • 80.19% macro-F1 on SiFC and 76.55% on the fMoW facility subset; APERTURE beats LaZSL by more than 10 pp on SiFC and VLG-CBM by more than 3 pp on fMoW.On SiFC All, global-scale scoring reaches 78.62% F1 and home-scale scoring 55.09%, while blending them reaches 80.19%.Source: Table 1 · p. 7, §5.2 · pp. 7–8
  • Class- and country-conditioned descriptors improve scores by more than 8 pp over general descriptors, while document-grounded RAG conditioning yields little improvement.Decoding probabilities usually outperform MLLM-reported confidence scores.Source: §5.2 · p. 8
  • APERTURE exceeds the best supervised CBM in four of six country comparisons, but supervised CBMs are slightly stronger on fMoW-FC USA and Russia.APERTURE performs better in India, the USA, and China on SiFC and France on fMoW-FC.Source: §5.2 · p. 8
  • Blending global and home-scale views improves results across Gemma 4 31B, GLM-4.6V, Qwen3.5 122B-A10B, and Gemini 3.1 Pro.Each blended result exceeds its own best global-only concept setting wherever both were tested.Source: §5.2 · p. 8
  • After three part removals, concept-based scores drop 0.43–0.49 on average versus 0.36 for zero-shot scoring; after the first removal, home-scale drops 0.10 versus global 0.04 and zero-shot 0.03.The intervention study covers 90 images across six classes; Table 3 reports mean true-class score drops over 15 images per class.Source: §5.2 · p. 8, Table 3 · p. 9
  • Across 100 kiln-conversion sites, adding Zigzag-specific cues raises the conversion score from 0.343 (D1) to 0.727 (D2) with Gemma 4.This is a descriptor-only update for the change from FCBK to Zigzag.Source: §5.2 · p. 8, Figure 4 · p. 8
  • On a 600-image, 20-class CUB subset, APERTURE outperforms CbD by 5 pp but remains below LaZSL (68.68%).The paper attributes LaZSL’s advantage to random crops being better suited to datasets without a meaningful home scale.Source: §5.2 · p. 8

5.3 ABLATION

  • On SiFC, fixed 2048-pixel crops reach 54.9% macro-F1, close to routed home scale at 55.1%, while global scoring reaches 78.6%; in-domain CCCD and GCD reach 84.53% and 83.98% versus APERTURE’s 80.19%.The in-domain gains require labels from the evaluated country; the in-domain setup uses within-country out-of-fold evaluation, while LF-CBM tests cross-country.Source: §5.3 · p. 9, Table 6 · p. 9

6 CONCLUSIONS

APERTURE combines global and home-scale concept evidence with concept-level recognition to improve training-free recognition in satellite imagery. Its results support expert updates, while strong performance still depends on global scores alongside home-scale predictions.

6 CONCLUSIONS

  • APERTURE combines global and characteristic home-scale recognition, and the paper reports significant improvements from both MLLMs and multiscale recognition.The conclusion also frames effective concept recognition in expert domains as requiring attention to what visual evidence defines a concept and where to look for it.Source: §6 · p. 9
  • APERTURE outperforms prior training-free CBMs on SiFC and has better out-of-region generalization than supervised concept bottlenecks.The paper also introduces SiFC and reports that the method enables easy expert intervention and concept-level updates.Source: §6 · p. 9
  • Strong performance still requires blending home-scale predictions with global scores; home-scale evidence alone is the stated goal for component-level interpretability.The paper identifies reducing reliance on global context as future work.Source: §6 · p. 9

A.1 CHANGES TO CONCEPT QUESTIONING

The descriptor changes make concept verification more explicit about visual evidence and confusers, while conditioning descriptions on facility class without naming that class in the verification question.

A.1 CHANGES TO CONCEPT QUESTIONING

  • Descriptors expand a concept into appearance, spatial placement, and likely confusers, giving the model supporting and contrasting visual evidence.The paper says this structure makes verification less ambiguous.Source: §A.1 · p. 15

A. Descriptor-generation template (reconstructed)

  • Class-conditioned descriptors specialize a concept to its facility class while keeping the class name out of the verification question.The paper contrasts this with prepending class names, which it says reduces performance and weakens interpretability by introducing an indirect class signal.Source: A. Descriptor-generation template… · p. 15
  • The verification template asks whether a concept is visible, supplies its appearance cues, and names confusers before requiring a yes-or-no answer.The reconstructed template instructs the generator to focus on the object itself without naming the facility class.Source: A. Descriptor-generation template… · p. 15

B. Worked example: Chimney stack

The chimney-stack example illustrates how appearance and confuser cues can support object verification, while the class-conditioned template specializes descriptions without naming the facility class in returned fields.

B. Worked example: Chimney stack

  • The chimney-stack example uses a small circular appearance and a long, needle-like shadow, distinguishing it from silos, water towers, and flare stacks.It also cites a possible smoke or steam plume as a cue.Source: B. Worked example: Chimney stack · p. 15

C. Class-conditioned generation template (condensed)

  • The class-conditioned description template organizes concept cues into appearance, confusers, and placement, and directs the generator to adapt these fields to the facility class.It asks for class-general descriptions and excludes the facility class name from the returned fields.Source: C. Class-conditioned generation… · p. 16

D. Worked example: Chimney stack (adapted question)

The adapted chimney-stack question uses class-specific visual cues without explicitly naming the class. The examples contrast country-conditioned descriptions and specify how home-scale crop widths are selected.

D. Worked example: Chimney stack (adapted question)

  • For a coal or thermal power plant, the adapted chimney-stack question uses size, shadow, plume, confuser, and placement cues without naming the facility class.The example places the stack beside the boiler or emissions-control equipment and describes visible flue-gas ducts connecting them.Source: D. Worked example: Chimney stack… · p. 16

F. Worked example: The same concept across three countries

  • India's description allows one to six chimney dots with 150 to 300 m shadows, versus one to four in China and one to three in the USA.These are differences in generated appearance descriptions, not measured country-level statistics.Source: F. Worked example: The same concept… · p. 16

G. Home-scale assignment template (reconstructed)

H. Worked example: Chimney stack

The chimney-stack example uses a country- and class-specific descriptor, then routes toward a saved 512-pixel home scale for verification. Its prompts distinguish global presence detection, quadrant selection, and final verification on a clean crop.

H. Worked example: Chimney stack

  • The India chimney-stack descriptor pairs appearance and placement cues with confusers, and assigns the concept a 512-pixel home scale.The cues include circular bases, long thin shadows, nearby boiler buildings, and flue gas ducts.Source: H. Worked example: Chimney stack · p. 17

K. Home-scale verification: object-level concepts

I. Global verification

J. Quadrant routing

  • Quadrant routing selects the quadrant with the clearest candidate evidence, even when evidence is weak; it is a zoom-location decision, not a final presence judgment.The prompt uses a numbered four-quadrant grid and descriptor cues.Source: J. Quadrant routing · p. 17

L. Saved example: Chimney stack in India

The saved chimney-stack example uses a 512-pixel home scale and routes through progressively smaller crops before verification. In a wider-search test on SiFC, beam search improved home-only scoring, but produced little or no gain for the global–home blend.

L. Saved example: Chimney stack in India

Loading 2609.38603v1…