PaperlayerView exact revision on arXiv

Source-linked AI summary

VisLens: Single-Pass Interpretable Visual Search for Multimodal LLMs

Jingyi He, Sanghwan Kim, Zeynep Akata

arXiv:2608.30705v1cs.CV

TL;DR

MLLMs struggle to locate tiny or rare targets in high-resolution images, while existing search methods trade repeated queries or opaque tool-use policies for accuracy. VisLens uses tuned-lens decoding and semantic matching to create explicit crops in a single forward pass, and it improves visual-search QA across evaluated benchmarks. Its local vocabulary-level matching is less reliable for symbolic, textual, or globally distributed evidence.

  • Problem

    MLLMs struggle with fine-grained Visual Search because small or rare targets occupy little of a high-resolution image, while existing remedies require repeated queries or opaque tool-use policies.

  • Method

    VisLens uses a tuned-lens translator to decode early visual-token semantics, match them to query targets, and construct explicit crops for crop-guided answering in one forward pass.

  • Results

    VisLens consistently improves visual-search QA across V∗Bench and HR-Bench, including +10.5 points on Direct Attribute for InternVL3 and +12.8/+12.0/+11.0 points on HR-Bench-8K Single for LLaVA-OneVision, Qwen2.5-VL, and InternVL3 respectively.

  • Takeaways & Limitations

    VisLens offers an accuracy–latency Pareto-frontier approach whose advantage over multi-pass search widens as image resolution grows.

  • Takeaways & Limitations

    VisLens is less reliable for symbolic, textual, or globally distributed evidence and is less suited to OCR, mathematical notation, chart understanding, and global diagram layout reasoning.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) struggle with fine-grained Visual Search, the task of locating small or rare objects in high-resolution images. Existing remedies fall into two families: (1) Training-free methods based on attention or confidence scores are accurate but slow, since they require multiple MLLM queries per example. (2) Reinforcement Learning (RL) trained tool-use models are faster at inference but opaque, since their tool calls remain uncontrollable and hard to interpret. To overcome this, we propose \emph{VisLens} (Visual Focus via Logit Lens), a Visual Search method built on the logit lens, which decodes the semantics held in a hidden state by projecting it through the LLM head. VisLens further uses a lightweight tuned-lens that maps early hidden states into the final hidden state space, so visual tokens can be read out from early layers. These tokens are matched to target words in the query to generate a crop of the relevant region, which is fed back in alongside the original image to produce the final answer. The whole process, from decoding to the final answer, completes in a single forward pass without repeated queries. VisLens matches or exceeds prior baselines while delivering a substantial latency advantage, running $8.5$--$9.9\times$ faster than Thyme and up to $22.2\times$ faster than training-free multi-pass search methods.

1 Introduction

VisLens addresses fine-grained visual search by decoding early visual-token semantics into controllable crops, avoiding repeated queries and reinforcement-learning tool policies. It combines logit- and tuned-lens projections with semantic matching in a single-pass, interpretable pipeline.

  • Motivation: Fine-grained Visual Search is difficult because small targets in high-resolution images can be lost in irrelevant background content.This limitation matters for applications including document analysis, surveillance, and autonomous driving.
  • Existing Remedies: Training-free search methods use attention or uncertainty to propose regions but require multiple MLLM queries per example.Repeated querying multiplies inference cost and limits practicality in time-sensitive settings.
  • Existing Remedies: RL-trained tool-use methods can be accurate but require costly task-specific training and produce crop policies that are difficult to control or interpret.Their opaque decisions make failures hard to diagnose.
  • VisLens: VisLens decodes early visual-token states, matches their semantics to query target words, and feeds the resulting crop alongside the original image without extra queries or reinforcement learning.The method relies on information already encoded in spatially corresponding visual tokens.
  • VisLens: A lightweight tuned-lens maps early hidden states into the final representation space before logit-lens decoding, improving early-layer localization without changing the underlying MLLM.Only the translator is trained.
  • Contributions: VisLens provides a single-pass, controllable, and interpretable Visual Search pipeline that converts decoded tokens into crop regions through semantic matching, clustering, and box merging.The method is designed to expose how crop regions are selected.
  • Results: VisLens shows a favorable accuracy–latency trade-off across V∗Bench and HR-Bench, supported by interpretability case studies.The supplied passage states the cross-benchmark trade-off but does not provide its numerical magnitude.

2 Related Work

Visual Search evaluates whether MLLMs can locate small or rare targets in high-resolution images, but existing approaches trade extra computation or specialized training for improved localization.

  • Visual Search in MLLMs: Visual Search asks an MLLM to locate a small or rare target in a high-resolution image.The task is formalized by V∗Bench and its SEAL framework.
  • Training-Free Methods: Training-free approaches use attention or confidence signals to propose and zoom into candidate regions, requiring several forward passes per example.Each candidate region must be re-encoded and re-queried.
  • RL-Tuned Methods: RL-tuned approaches train models to use visual tools such as cropping or zooming through specialized instruction data.The supplied passage introduces this family but does not provide further comparison details.

3 Visual Focus via Logit Lens (VisLens)

VisLens localizes visual-search targets by decoding spatially aligned visual tokens from early MLLM layers, matching them to query phrases, and constructing crops for crop-guided answering. A tuned translator makes early representations readable by the frozen output head, while clustering and area-capped box merging produce target crops.

  • Task Definition: Visual Search takes a high-resolution image and natural-language query, requiring an answer from the MLLM.VisLens adds localization and cropping to address the small-target bottleneck before answering.
  • Task Definition: Small-target visual search is bottlenecked by localization because fixed resolution and token budgets dilute the target signal.The intended crop approximates the relevant region before reasoning over it.
  • Logit Lens: The MLLM represents the image as spatially arranged visual tokens, each carrying a hidden state at an intermediate layer.The one-to-one correspondence between visual tokens and grid cells enables spatial semantic maps.
  • Logit Lens: The logit lens applies the frozen language-model head to an intermediate hidden state to produce a vocabulary distribution approximating its prediction at that layer.At visual positions, decoded top tokens can name objects or attributes represented by image patches.
  • Tuned Lens: Early-layer logit-lens predictions are unreliable because the output head is tuned for final-layer representations with a different basis and scale.An early state may encode the eventual prediction without being directly readable by the final head.
  • Tuned Lens: A lightweight residual-MLP translator maps source-layer states into final-layer space before decoding.It is initialized near identity and trained separately for each source layer.
  • Tuned Lens: The translator is trained by vocabulary-space knowledge distillation against the model’s own final-layer distribution, using visual-token positions and no external labels.The base MLLM remains frozen while only a small translator is updated.
  • Full Pipeline: A single forward pass decodes every visual cell into top-K token–probability pairs, forming a semantic map used to generate target crops.The map is combined with the query as VisLens(TL, q).

4 Experiments

VisLens improves fine-grained visual-search accuracy across backbones and benchmarks while preserving standard VQA performance and substantially reducing inference latency. Ablations show that early visual-token states support effective localization, with crop construction depending on resolution and target spread.

  • 4.2 Main Results: VisLens improves every backbone on every visual-search benchmark, with the largest gains in single-object settings that directly test small-target localization.Reported gains include +10.5 points on Direct Attribute for InternVL3 and +12.8/+12.0/+11.0 points on HRBench-8K Single for LLaVA-OneVision, Qwen2.5-VL, and InternVL3.
  • 4.2 Main Results: VisLens also improves relational questions, although gains are generally smaller because tight crops can remove context needed for comparing multiple targets.The original image supplies global context alongside the crop, supporting many relational queries.
  • 4.3 Accuracy–Latency Trade-off: 8.5–9.9× lower latency than Thyme accompanies accuracy that matches or exceeds it on V*Bench, HRBench-4K, and HRBench-8K in a single pass.Against ZoomEye at matched accuracy, VisLens is 4.8×, 12.5×, and 22.2× faster on V*Bench, HRBench-4K, and HRBench-8K respectively.
  • 4.4 Effect on General VQA: VisLens preserves standard VQA performance, staying within about one point of each frozen baseline across A-OKVQA, POPE, and GQA.This supports using VisLens on regular-size images as well as fine-grained visual-search inputs.
  • 4.5 Ablation: Source Layer for the Tuned-Lens: Early source layers provide nearly flat localization quality, with translated readouts varying by at most 0.010 absolute per metric and costing at most ∼0.02 relative to the final-layer baseline.The pre-layer 1 representation is already competitive with deeper source layers and is used as the default.
  • 4.6 Ablation: Crop Construction: Union crops work best on V∗Bench, whereas separate crops become stronger on HR-Bench at 8K when targets are distant.Minimal cell padding is sufficient, favoring tight object-centered crops over broad scene crops.
  • 4.7 Detection Ablation: Replacing VisLens matching with SAM 3 improves over the vanilla backbone but yields much smaller gains because SAM 3 often fails to return crops for small or ambiguous targets.The stated main failure mode is recall.

5 Analysis

VisLens makes visual-search localization inspectable by decoding early visual-token semantics into target-matching crops, while its failures reveal when local evidence is absent, mismatched, or dominated by context. The method is strongest for object-centric targets and less reliable when relevant evidence is symbolic, textual, or distributed across global structure.

  • Interpreting decoded tokens: VisLens uses tuned-lens readouts from early visual tokens to produce local vocabulary-level evidence and explicit crops for answering.The tuned-lens maps early states into final-layer decoding space before semantic matching and crop formation.
  • Interpreting decoded tokens: A pants-color example shows locally recoverable object semantics, including pants, shorts, trousers, and black, despite the baseline missing the person.The readout demonstrates that object-level semantics can be recovered around a relevant visual region.
  • Interpreting failures: VisLens failures are inspectable because decoded tokens show whether target evidence is absent, expressed through a related concept, or dominated by surrounding scene context.This makes a missed answer also a readable localization failure.
  • Interpreting failures: In the broom case, tuned-lens and final-layer readouts emphasize related gardening context rather than a reliable broom token, preventing safe target matching.The readouts include branches, trees, leaves, shovel, gardening, and handle.
  • Interpreting failures: In the person–building case, VisLens recovers the building but not the person because both readouts represent nearby evidence as background-like concepts.The unresolved person target leaves the relational query unanswered.
  • Limitations and remaining challenges: VisLens is less reliable for symbolic, textual, chart-like, and diagrammatic evidence because independent patches may expose fragments rather than the queried higher-level concept.The current local matching step is best suited to visually localizable objects or object parts and would need structural or relation-aware extensions for these settings.

6 Conclusion

VisLens is a single-pass, interpretable visual-search method that decodes local visual-token semantics, matches them to query targets, and forms explicit crops while keeping the base MLLM frozen. Across V*Bench and HR-Bench, it improves visual-search QA and lies on the accuracy–latency Pareto frontier, while remaining less suited to structurally distributed evidence.

  • Conclusion: VisLens decodes local visual-token semantics with a tuned-lens, matches tokens to query targets, and forms explicit crops for crop-guided answering.The base MLLM remains frozen; only a lightweight translator is trained.
  • Conclusion: Across V*Bench and HR-Bench, VisLens consistently improves visual-search QA, especially on single-object questions where small-target localization is the bottleneck.The largest gains occur on these single-object questions.
  • Conclusion: VisLens sits on the accuracy–latency Pareto frontier across evaluated benchmarks, with its advantage over multi-pass search widening as image resolution grows.This advantage occurs in the high-resolution regime targeted by visual search.
  • Conclusion: Localization decisions are inspectable because each crop is grounded in decoded vocabulary tokens that also expose related surrounding semantics when target evidence is unavailable.The same readouts indicate whether target evidence is locally recoverable.
  • Conclusion: The current design is strongest for object-centric visual search and less suited to symbolic, textual, chart-like, or diagrammatic inputs with structurally distributed evidence.Future extensions include OCR, structural grouping, and relation-aware matching.
Loading 2608.30705v1…