MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs

Introduction

• Because MLLMs are rapidly entering applications including biomedicine and security, the authors argue that understanding their visual-perception limitations is crucial.Source: §1 · p. 1
• In examples, focused cropping raises answer probability for a small street sign, helps distinguish an egret from other white birds, and improves reading of a word the model initially misidentifies.The bird example suggests the error reflects insufficient visual detail rather than misunderstanding the question; the word example suggests the model knows where to look but cannot accurately perceive the word.Source: §1 · pp. 1–2
• The authors propose three training-free visual cropping methods using MLLM attention maps and gradients, and report improved accuracy across seven VQA benchmarks on two MLLMs, especially on detail-sensitive benchmarks.The methods are presented as scalable solutions that use the MLLM’s own internal state.Source: §1 · p. 2
• The study establishes a causal link between visual-concept size and perception, and finds that MLLMs consistently know where to look even when they answer incorrectly.The authors distinguish difficulty perceiving details from difficulty locating the concept.Source: §1 · p. 2

Multimodal Large Language Models (MLLMs)

• MLLMs include end-to-end and modular pretrained families, with modular approaches dominating recent state-of-the-art methods.The paper presents its training-free perception improvements as orthogonal to existing approaches.Source: §2 · p. 2

Visual Localization Methods

• Dedicated methods such as YOLO (Redmon et al., 2016), SAM (Kirillov et al., 2023), and GLIP (Li et al., 2022b) use dense spatial annotations for localization.Grad-CAM (Selvaraju et al., 2017) localizes via classifier gradients without spatial supervision; prior adaptations target BLIP, whereas this work proposes a more general MLLM localization method and training-free use of internal states.Source: §2 · p. 3

Visual Perception Limitations in MLLMs

• Prior work reports difficulty answering questions about small objects and explores fine-tuning, multi-agent pipelines, or cropping; this work attributes the difficulty to observing, not locating, small details.Related work also reports visual blind spots and sensitivity to image quality, distractors, and local object-position perturbations (Zhang et al., 2024a).Source: §2 · p. 3

Sensitivity Study

• The study partitions TextVQA validation questions by the ground-truth answer box’s share of image area into small, medium, and large concepts.The groups are S < 0.005 (773 pairs), 0.005 ≤ S < 0.05 (2,411 pairs), and S ≥ 0.05 (1,186 pairs); TextVQA offers many small-concept questions with textual answers that are harder to guess from other image context.Source: §3 · pp. 3–4

Intervention Study

• Human-CROP significantly improves all models’ accuracy on small and medium concepts, with smaller gains on large concepts, supporting a causal role for visual size.For each image-question pair, the smallest square containing the ground-truth box is resized to the model’s input resolution and provided alongside the original image-question pair.Source: Table 1 · p. 3, §3 · p. 4

Sensitivity Study

• Accuracy falls as concepts shrink: BLIP-2 and InstructBLIP drop 24 and 23 absolute percentage points from large to small, while LLaVA-1.5 and Qwen-VL drop 11 and 12.GPT-4o also declines by 7 percentage points from small to medium, despite its training set being unknown and potentially including all of TextVQA.Source: §3 · p. 4

Method

• The analysis measures attention to the ground-truth bounding box against same-sized alternative boxes to assess whether errors reflect localization or perception limits.It combines attention from the starting answer token to image tokens with attention from image tokens to image regions.Source: §4 · pp. 4–5

Results

• Figure 2 illustrates examples where MLLMs answer incorrectly despite relative attention being directed to the relevant image region.The examples display relative image attention for one MLLM layer.Source: Figure 2 · p. 5
• In most layers, all considered MLLMs show attention ratios significantly greater than 1 for the ground-truth region, with similarly strong attention whether answers are correct or incorrect.The ratio is averaged over TextVQA’s validation set and compared across layers.Source: Figure 3 · p. 5, §4 · p. 6

Automatic visual cropping

• ViCrop estimates a question-relevant region from an MLLM’s attention or gradients, then adds its resized crop to the original image tokens to retain global context.The methods use the model’s internal information to approximate the subject’s region of interest; concatenating crop tokens with original image tokens is used for all methods.Source: §5 · pp. 6–7

Relative Attention ViCrop (rel-att)

• Relative Attention ViCrop (rel-att) uses answer-to-image relative attention as its crop-importance map, selecting a target layer on held-out TextVQA samples.The target layer is chosen from the LLM and connector; the layer choice is ablated in Section 6.Source: §5 · p. 7

Gradient-Weighted Attention ViCrop (grad-att)

• Gradient-Weighted Attention ViCrop (grad-att) weights answer-to-image attention by positive decision gradients, avoiding rel-att’s additional generic-instruction forward pass.It forms the decision score from the log probability of the most likely starting answer token and removes negative gradients, which can indicate distracting locations (Selvaraju et al., 2017). It uses the same target layer as rel-att.Source: §5 · p. 7

Input Gradient ViCrop (pure-grad)

• Input Gradient ViCrop (pure-grad) derives its crop-importance map from the decision gradient with respect to image pixels, making it independent of Transformer architecture.It suppresses high-gradient constant-color regions using high-pass and median filtering, thresholds the filtered image at its spatial median, then pools the edge-emphasized gradients into image patches.Source: §5 · p. 7

Bounding Box Selection for Visual Cropping

• Bounding boxes are selected by maximizing importance within sliding windows, then choosing the window whose sum differs most from those of adjacent positions.Window sizes range from 1 to 2 times the MLLM input resolution in increments of 0.2; the selected crop is resized to that resolution and supplied with the image-question pair.Source: §5 · p. 7

High-Resolution Visual Cropping

• For V*, ViCrop first computes importance maps on near-square image blocks smaller than 1024 × 1024, reassembles them, and selects a crop from the combined map.The original image-question pair is provided with the resized crop.Source: §5 · pp. 7–8

ViCrop Improves VQA Accuracy

• LLaVA-1.5 gains more from ViCrop than InstructBLIP, whose connector-only tuning leaves its backbone less able to use additional cropped image tokens.Despite this difference, the results show ViCrop can be applied to different MLLMs as an inference-time solution.Source: §6 · pp. 8–9
• ViCrop significantly improves both MLLMs on detail-sensitive benchmarks without training while maintaining performance on benchmarks with larger visual concepts.On TextVQA, evaluation excludes externally extracted OCR tokens to measure the MLLMs’ own perception.Source: §6 · p. 8, Table 2 · p. 9

Ablation Study on the Choice of Layer

• Averaging across layers leaves rel-att robust and reduces grad-att accuracy by about 3.5 percentage points, while both still improve TextVQA accuracy.Layer averaging is therefore a suitable choice when no data are available for selecting a layer.Source: §6 · p. 9, Table 3 · p. 9

Ablation Study on the High-Resolution ViCrop

• The high-resolution strategy benefits LLaVA-1.5 on V∗ but reduces grad-att and pure-grad performance for InstructBLIP; all methods still improve accuracy with or without it.Source: §6 · p. 9, Table 3 · p. 9

ViCrop with External Tools

• SAM, YOLO, and CLIP improve TextVQA accuracy, but each is weaker than the proposed internal ViCrop methods.Source: §6 · p. 9, Table 4 · p. 10

Inference Time Overhead

• All proposed methods except SAM add a reasonably fast 1–2 seconds on GPU; rel-att’s overhead equals generating five tokens and stays constant with answer length.By contrast, MLLM inference time scales approximately linearly with the number of answer tokens.Source: Table 4 · p. 10, §6 · p. 10

7 CONCLUSION

• Training-free visual localization significantly improves MLLM accuracy, especially in detail-sensitive scenarios, addressing a small-detail perception bias rather than a localization failure.The authors caution against using MLLMs in detail-sensitive applications and identify localization with the model’s own knowledge as a promising direction.Source: §7 · p. 10

Limitations and Future Work

• ViCrop is less helpful for relation and counting questions because it focuses on one image region; focusing on multiple regions is future work.Source: §7 · p. 10
• ViCrop adds inference time and visual tokens, but the authors see potential to reduce costs through lower precision, weight quantization, or MQT-enabled rescaling of crops.The methods worked with two MLLMs with distinct visual context sizes; compatibility with varying context sizes under MQT is presented as a possibility.Source: §7 · p. 10
• Because ViCrop methods can offer complementary benefits, combining them, for example based on prediction uncertainty, is proposed as future work.Source: §7 · p. 10

A IMPLEMENTATION DETAILS

• Experiments used Python 3.10.6, Transformers 4.29.1, PyTorch 2.1.2, an Intel Gold 5317 CPU, and NVIDIA RTX A6000 GPUs.The environment had 48 CPU cores and 756 GB RAM; models used Hugging Face implementations with paper-recommended hyperparameters, except GPT-4o, accessed through its official public API.Source: A IMPLEMENTATION DETAILS · p. 15
• TextVQA evaluation supplied only the image and question, unlike LLaVA-1.5’s original setting, which used externally extracted OCR tokens (Liu et al., 2023a).The authors describe OCR tokens as outsourcing text perception to an external model and say their no-OCR setting better reflects LLaVA-1.5’s perception ability.Source: A IMPLEMENTATION DETAILS · p. 15
• 59.8 accuracy for LLaVA-1.5 without visual cropping and 63.95 with rel-att were measured when OCR tokens were provided.The authors report these results for completeness, noting that visual cropping can remain beneficial when OCR tokens are supplied.Source: A IMPLEMENTATION DETAILS · p. 15

B DATASET STATISTICS

• VQAv2 uses a random 50K subset of its official validation set, while every other dataset uses its entire validation set; Table 5 reports image dimensions and dataset counts.Source: B DATASET STATISTICS · p. 15, Table 5 · p. 15
• TextVQA has 5,000 questions across 3,166 images, and the analysis uses manually annotated question-image pairs with ground-truth boxes over answer regions.Source: B DATASET STATISTICS · p. 15
• Filtering out pairs with multiple instances of the question subject leaves 4,370 TextVQA question-image pairs, each with one subject instance.Source: B DATASET STATISTICS · p. 15

C PROMPT FORMAT FOR ZERO-SHOT INFERENCE

• The paper adapts distinct zero-shot prompt formats for LLaVA and InstructBLIP from their original papers.Source: C PROMPT FORMAT FOR ZERO-SHOT INFERENCE · p. 16

LLaVA-1.5

• The first shown format presents the image and question, then asks for an answer using a single word or phrase.Source: LLaVA-1.5 · p. 16
• A second shown format presents the image and question followed by “Short Answer:”.Source: LLaVA-1.5 · p. 16

D ORTHOGONAL BENEFITS TO LLAVA-NEXT

• Rel-att visual cropping boosts higher-resolution LLaVA-NeXT without training, showing orthogonal benefits to higher-resolution patch training.The authors present this as further evidence for the generalizability of visual cropping.Source: Table 6 · p. 16, D ORTHOGONAL BENEFITS TO LLAVA-NEXT · p. 16

E COMPARISON WITH THE V* METHOD (SEAL)

• LLaVA-1.5+rel-att outperforms SEAL except on the V* benchmark in the reported as-is comparison.The authors suggest SEAL’s V* exception may reflect its tuning for high-resolution V* images; they caution that the comparison is not fair because SEAL requires substantial training and fine-tuning.Source: E COMPARISON WITH THE V* METHOD (SEAL) · p. 16, Table 7 · p. 16
• SEAL (Wu and Xie, 2023) uses multi-agent fine-tuning, whereas the proposed methods are training-free; SEAL also takes longer per question in the reported test.On 100 random TextVQA samples with one A6000 GPU, average inference time was 4.44s for SEAL and 1.88s for the proposed method; the methods could also be integrated into SEAL or other multi-agent pipelines.Source: E COMPARISON WITH THE V* METHOD (SEAL) · p. 16

F EXTERNAL TOOLS VICROP

• External ViCrop uses CLIP, YOLO, and SAM for question-guided localization via multimodal encoding, object detection, and semantic segmentation, respectively.The section compares these external-tool methods with internal ViCrop methods in Table 4.Source: F EXTERNAL TOOLS VICROP · p. 17
• CLIP ViCrop repeatedly crops each image from four sides at a 0.9 ratio, selecting the crop with highest question similarity for the next of 20 iterations.The crop with the highest CLIP similarity after the iterations is selected for visual cropping.Source: F EXTERNAL TOOLS VICROP · p. 17
• YOLO ViCrop proposes YOLOv8-detected object boxes above 0.25 confidence, then uses CLIP question similarity to select the region of interest.YOLO filtering excludes regions without salient detected objects, which could otherwise receive high CLIP similarity.Source: F EXTERNAL TOOLS VICROP · p. 17
• SAM ViCrop converts SAM’s segmentation masks for objects and object parts into bounding boxes, selecting the box with highest CLIP question similarity.This supplies more granular candidate regions than YOLO, whose boxes cover a fixed set of object classes.Source: F EXTERNAL TOOLS VICROP · p. 17
• Each external ViCrop method supplies the MLLM with the smallest covering square crop alongside the original image-question pair.The square crop avoids deformation when resized to the MLLM’s input resolution.Source: F EXTERNAL TOOLS VICROP · p. 17

• On V∗, Figure 6 shows three LLaVA-1.5 (rel-att) success examples and one failure, with zoomed insets and cyan boxes marking cropped regions.Source: Figure 6 · p. 18
• On TextVQA, Figure 7 shows nine LLaVA-1.5 (rel-att) success examples and six failures, with cyan boxes marking cropped regions.Source: Figure 7 · p. 19
• On TextVQA, Figure 8 shows nine InstructBLIP (rel-att) success examples and six failures, with cyan boxes marking cropped regions.Source: Figure 8 · p. 20