PaperlayerView exact revision on arXiv

Source-linked AI summary

MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs

Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, Filip Ilievski

arXiv:2502.17422v1cs.CVcs.AIcs.CL

TL;DR

MLLMs struggle more with small visual details than large ones, and the paper shows this size effect is causal. It uses internal attention and gradient maps for training-free cropping, which significantly improves accuracy across detail-sensitive VQA settings.

  • Problem

    MLLM accuracy declines as the queried visual concept becomes smaller, motivating study of this perception bias.

  • Method

    The paper proposes training-free visual cropping methods based on MLLMs’ internal attention and gradient maps.

  • Results

    The methods significantly improve MLLM accuracy on detail-sensitive benchmarks without requiring training, while maintaining performance on larger visual concepts.

  • Takeaways & Limitations

    MLLMs can often identify where to look despite incorrect answers, supporting visual intervention through their internal states.

  • Takeaways & Limitations

    ViCrop is particularly difficult to apply to relation and counting questions because it focuses on one image region.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have experienced rapid progress in visual recognition tasks in recent years. Given their potential integration into many critical applications, it is important to understand the limitations of their visual perception. In this work, we study whether MLLMs can perceive small visual details as effectively as large ones when answering questions about images. We observe that their performance is very sensitive to the size of the visual subject of the question, and further show that this effect is in fact causal by conducting an intervention study. Next, we study the attention patterns of MLLMs when answering visual questions, and intriguingly find that they consistently know where to look, even when they provide the wrong answer. Based on these findings, we then propose training-free visual intervention methods that leverage the internal knowledge of any MLLM itself, in the form of attention and gradient maps, to enhance its perception of small visual details. We evaluate our proposed methods on two widely-used MLLMs and seven visual question answering benchmarks and show that they can significantly improve MLLMs' accuracy without requiring any training. Our results elucidate the risk of applying MLLMs to visual recognition tasks concerning small details and indicate that visual intervention using the model's internal state is a promising direction to mitigate this risk.

1 INTRODUCTION

The paper examines MLLMs’ difficulty perceiving small visual details and proposes training-free visual interventions based on their internal attention and gradients.

  • MLLMs are increasingly used in critical applications, making limitations in visual perception important to study.
  • BLIP-2’s answers can improve as visual cropping enlarges the questioned object, even when the question remains unchanged.
  • Partially relevant incorrect answers suggest that MLLMs may locate the queried region while failing to perceive its visual details accurately.
  • The study establishes the size effect causally, tests whether models know where to look, and develops scalable training-free cropping methods using attention maps and gradients.

2 RELATED WORKS

Related work covers MLLM architectures, visual localization methods, and prior evidence that small visual concepts challenge MLLM perception.

  • MLLMs: MLLMs include end-to-end and modular pretrained models supporting diverse language and vision tasks.
  • Visual Localization Methods: Dedicated localization methods such as YOLO, SAM, and GLIP rely heavily on dense spatial annotations, while Grad-CAM localizes regions from classifier gradients without spatial supervision.
  • Visual Perception Limitations in MLLMs: Prior studies report difficulty answering questions about small image objects and explore high-resolution fine-tuning, multi-agent pipelines, and visual cropping.
  • Visual Perception Limitations in MLLMs: Related evidence also concerns degraded perception at specific locations, sensitivity to visual quality and distractors, and local object-location perturbations.

3 MLLMS’ SENSITIVITY TO THE SIZE OF VISUAL CONCEPTS

The study quantifies MLLMs’ sensitivity to visual-concept size in TextVQA and shows that ground-truth cropping improves performance, especially for smaller concepts.

  • Dataset Partition: TextVQA validation examples are partitioned by bounding-box area into small (773), medium (2411), and large (1186) question-image pairs.Small: S < 0.005; medium: 0.005 ≤ S < 0.05; large: S ≥ 0.05.
  • Sensitivity Study: Accuracy declines for every evaluated MLLM as the ground-truth visual concept becomes relatively smaller.
  • Sensitivity Study: 24 and 23 absolute percentage points separate large from small partitions for zero-shot BLIP-2 and InstructBLIP, respectively.
  • Sensitivity Study: 11 and 12 accuracy points separate large from small partitions for LLaVA-1.5 and Qwen-VL, while GPT-4o declines 7 points between small and medium partitions.
  • Intervention Study: Human-CROP significantly improves accuracy on small and medium partitions, showing that visual concept size causally contributes to the perceptual limitation.

4 DO MLLMS KNOW WHERE TO LOOK?

The paper measures MLLM attention over image regions and finds that models often attend to the correct location even when answering incorrectly.

  • The analysis distinguishes localization difficulty from detail-perception difficulty as possible causes of poor performance on small concepts.
  • MLLMs’ Setup: MLLMs process image patches into visual tokens, transform them into the LLM input space, prepend them to question tokens, and generate answers autoregressively.
  • Quantifying Spatial Attention: Answer-to-token attention and token-to-image attention are combined to produce answer-to-image attention across model layers.
  • Figure 2 illustrates cases where MLLMs answer incorrectly while their relative attention remains directed toward relevant image content.
  • Attention Ratio: Attention ratios exceed 1 in most layers, and remain similarly strong for correct and incorrect answers, indicating that MLLMs know where to look.

5 AUTOMATIC VISUAL CROPPING (VICROP)

ViCrop uses an MLLM’s internal attention or gradient information to locate a question-relevant image region, then crops and zooms that region without training. The methods retain the original image tokens alongside cropped tokens and use specialized handling for very high-resolution images.

  • Method overview: ViCrop constructs three automatic cropping methods that use attention maps or gradients to estimate the image region relevant to a question.The methods are designed to use each MLLM’s internal information rather than external spatial supervision.
  • Cropping design: The cropped image tokens are concatenated with the original image tokens, preserving the original image context while adding a zoomed view.The concatenation approach is used for all proposed methods.
  • Importance maps: Rel-att selects a target layer using held-out TextVQA samples and uses relative attention as the visual-cropping importance map.The target layer may be in the LLM or connector, and its choice is ablated later.
  • Importance maps: Grad-att weights answer-to-token and token-to-image attention by positive gradients of the model’s decision, avoiding a second generic-instruction forward pass.Negative gradients are removed because they correspond to tokens that decrease the decision score.
  • Importance maps: Pure-grad localizes relevant image regions directly from the input-image gradient, suppressing constant-color regions with edge filtering before patch pooling.This method does not depend on a Transformer-based architecture.
  • Cropping design: Sliding windows sized from 1 to 2 times the MLLM input resolution convert each importance map into a bounding box by selecting the best-scoring position.Windows are tested with stride 1 across the image.
  • High-resolution handling: For very high-resolution V* images, ViCrop first computes importance over smaller blocks before producing the final crop for the MLLM.This two-stage strategy addresses visual concepts that may disappear when the full image is resized.

6 VICROP METHOD ANALYSIS

ViCrop methods are evaluated across detail-sensitive and general-purpose VQA benchmarks, with ablations examining layer selection, high-resolution processing, external localization tools, and inference-time cost. The methods improve perception of small visual concepts without training, while their effectiveness and overhead vary by model and configuration.

  • Evaluation setup: The evaluation covers two open-source MLLMs, four detail-sensitive datasets, and three general-purpose benchmarks.The study tests InstructBLIP and LLaVA-1.5, including TextVQA, V*, POPE, and DocVQA among the detail-sensitive datasets.
  • Main results: All ViCrop methods significantly improve accuracy on detail-sensitive benchmarks without training while maintaining performance on larger visual concepts.The largest gains occur on TextVQA and V*, and the improvement does not appear to reduce accuracy on larger details and relations.
  • Evaluation setup: TextVQA evaluation excludes externally extracted OCR tokens to measure the MLLMs’ visual perception directly.This differs from the setup used in the original TextVQA paper.
  • Ablation studies: Layer averaging preserves improvements for both rel-att and grad-att, although grad-att loses about 3.5 percentage points in accuracy.The result suggests averaging is suitable when no data are available for selecting an informative layer.
  • Ablation studies: The high-resolution strategy benefits LLaVA-1.5 but reduces grad-att and pure-grad performance for InstructBLIP, while all variants still improve accuracy.The ablation compares ViCrop with and without the two-stage high-resolution strategy on V*.
  • External tools: External SAM, YOLO, and CLIP localization improves MLLM accuracy but remains weaker than every proposed internal ViCrop method.The comparison is conducted on TextVQA, after which the external methods are not explored further.
  • Inference cost: All proposed methods except SAM add approximately 1 to 2 seconds on GPU, and their overhead remains constant regardless of answer length.The rel-att computation takes about as long as generating five answer tokens, whereas MLLM inference scales approximately linearly with answer-token count.

7 CONCLUSION

The paper identifies a perception bias against small visual details in MLLMs and proposes training-free visual localization methods using models’ internal dynamics. These methods improve accuracy, but remain limited for multi-region questions and add inference overhead.

  • MLLMs exhibit a perception bias against small visual details, rooted in perception rather than localization limitations.The paper reports that MLLMs often know where to look even when answering incorrectly.
  • Training-free visual localization methods based on models’ internal dynamics significantly improve accuracy, especially in detail-sensitive scenarios.
  • Limitations and Future Work: ViCrop methods do not enhance all question types equally, particularly struggling with relation and counting questions.The method focuses on one image region, which limits its usefulness for questions requiring multiple regions.
  • Limitations and Future Work: ViCrop adds a few seconds of time overhead and additional visual tokens during inference.The paper suggests lower precision and weight quantization as possible optimizations.

A IMPLEMENTATION DETAILS

Experiments use specified software, hardware, model implementations, and recommended hyperparameters, with GPT-4o accessed through its public API. TextVQA is evaluated without externally extracted OCR tokens to measure the MLLM’s own perception ability.

  • Experiments use Python 3.10.6, Transformers 4.29.1, Torch 2.1.2, NVIDIA RTX A6000 GPUs, and recommended model hyperparameters.The environment also includes an Intel Gold 5317 CPU with 48 cores and 756 GB of RAM.
  • GPT-4o is evaluated through the official public API available at the time of submission.
  • TextVQA Evaluation: TextVQA evaluation provides only the image and question, without OCR-extracted tokens, to measure LLaVA-1.5’s own perception ability.This setting yields slightly lower accuracy than the original paper’s OCR-enriched evaluation.

B DATASET STATISTICS

The paper reports dataset sizes and uses a 50K validation subset for VQAv2, while using complete validation sets elsewhere. TextVQA analysis relies on manually annotated answer-region boxes, and models receive standardized zero-shot prompts.

  • Evaluation Datasets: VQAv2 experiments use a random 50K subset of the official validation set, while all other datasets use their entire validation sets.
  • Evaluation Datasets: Dataset statistics include average image height and width, image counts, and question counts.
  • TextVQA Analysis: TextVQA contains 5,000 questions and 3,166 images, with manually annotated bounding boxes covering answer-relevant regions.The annotations address the dataset’s lack of answer-specific bounding-box labels.
  • Prompt Formats: LLaVA and InstructBLIP use different zero-shot prompt formats adapted from their original papers.The prompts request a short answer to the visual question.

D ORTHOGONAL BENEFITS TO LLAVA-NEXT

The rel-att visual cropping method also improves LLaVA-NeXT, a newer MLLM supporting higher-resolution images. This indicates benefits that are orthogonal to training for higher image-patch resolution.

  • Rel-att visual cropping boosts LLaVA-NeXT’s performance without requiring any training.LLaVA-NeXT supports higher resolution than LLaVA-1.5 and is a current state-of-the-art model on several VQA benchmarks.
  • The LLaVA-NeXT result provides further evidence that visual cropping generalizes across models with different image-resolution capabilities.

E COMPARISON WITH THE V* METHOD (SEAL)

The paper compares its training-free internal ViCrop methods with SEAL and with external-tool localization methods. External approaches use CLIP, YOLO, or SAM to identify question-relevant regions before visual cropping.

  • Comparison with SEAL: SEAL requires substantial training and fine-tuning of several neural networks, whereas the paper’s methods are completely training-free.The paper therefore does not consider a direct comparison with SEAL fair, but reports an “as-is” comparison for practical context.
  • External-tool ViCrop methods: CLIP ViCrop progressively crops the image toward the region most semantically similar to the question.It evaluates four overlapping crops per iteration for 20 iterations and selects the crop with the highest CLIP similarity.
  • External-tool ViCrop methods: YOLO ViCrop filters candidate regions using YOLOv8 detections, retaining predicted objects with confidence higher than 0.25.The filtering step removes regions without salient detected objects before subsequent question-guided processing.
  • External-tool ViCrop methods: SAM ViCrop supplies a more granular set of candidate regions than YOLO by generating extensive zero-shot segmentation masks.This addresses YOLO’s restriction to bounding boxes for a fixed number of object classes.
  • Visual cropping: Each localization method produces the smallest covering square and provides it alongside the original image-question pair to the MLLM.The square preserves the crop’s proportions when resized to the MLLM’s input resolution.

G ADDITIONAL EXAMPLES ON MODEL’S PREDICTIONS

The appendix provides additional success and failure examples for LLaVA-1.5 and InstructBLIP after rel-att localization on V* and TextVQA. The examples include object-breed, spatial-relation, and color questions, with cyan boxes marking cropped regions.

  • Additional prediction examples: For a dog-breed question, LLaVA-1.5 with ViCrop predicts option D, golden retriever.The displayed question offers Husky, corgi, Dalmatian, and golden retriever as the four choices.
  • Additional prediction examples: For the truck-position question, LLaVA-1.5 with ViCrop predicts option A, right.The question asks whether the blue truck is on the left or right side of the white vehicle.
  • Additional prediction examples: For the handbag-color question, LLaVA-1.5 with ViCrop predicts option D, yellow.The answer choices are white, red, black, and yellow.
  • Additional prediction examples: For the parachute-color question, LLaVA-1.5 with ViCrop predicts option A, blue.The answer choices are blue, yellow, green, and red.
  • Figure examples: Figures 6–8 show success and failure examples for rel-att on V* and TextVQA, with cyan boxes indicating the cropped regions.Figure 6 covers LLaVA-1.5 on V*, while Figures 7 and 8 cover LLaVA-1.5 and InstructBLIP on TextVQA.
Loading 2502.17422v1…