PaperlayerView exact revision on arXiv

Source-linked AI summary

ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration

Haozhan Shen, Kangjia Zhao, Tiancheng Zhao, Ruochen Xu, Zilun Zhang, Mingwei Zhu, Jianwei Yin

arXiv:2411.16044v4cs.CV

TL;DR

Text-level visual reasoning leaves the image unchanged, limiting access to fine-grained information in high-resolution scenes. Zoom Eye uses tree-based visual search to guide MLLMs through image regions; evaluated models improve, and performance rises with more search steps.

  • Problem

    Text-level reasoning explores language-model tokens while leaving visual input static, limiting fine-grained visual processing in high-resolution images.

    Source: §1 · p. 2

  • Method

    Zoom Eye organizes an image as a tree of zoomed subregions and uses MLLM-guided ranking and stopping to search for task-relevant visual evidence.

    Source: §1 · p. 2

  • Results

    All evaluated models improve with Zoom Eye; LLaVA-ov-7B improves by 14.19% on V ∗Bench.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    Performance consistently improves with more search steps, suggesting vision-level reasoning benefits from deeper exploratory search.

    Source: §1 · p. 2

    Zoom Eye relies on heuristic rankings and stopping criteria, and fixed-size patches may fragment cues or fit semantic regions poorly.

    Source: §6 · p. 9

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in vision-language understanding. Recently, with the integration of test-time scaling techniques, these models have also shown strong potential in visual reasoning. However, most existing reasoning approaches remain text-level in nature: MLLMs are prompted to explore various combinations of textual tokens via their underlying language model, while the visual input remains fixed throughout the reasoning process. This paradigm limits the model's ability to fully exploit rich visual information, particularly when dealing with images containing numerous fine-grained elements. In such cases, vision-level reasoning becomes crucial - where models dynamically zoom into specific regions of the image to gather detailed visual cues necessary for accurate decision-making. In this paper, we propose Zoom Eye, a training-free, model-agnostic tree search algorithm tailored for vision-level reasoning. Zoom Eye treats an image as a hierarchical tree structure, where each child node represents a zoomed-in sub-region of its parent, and the root corresponds to the full image. The algorithm enables MLLMs to simulate human-like zooming behavior by navigating from root to leaf nodes in search of task-relevant visual evidence. We experiment on a series of high-resolution benchmarks and the results demonstrate that Zoom Eye consistently improves the performance of multiple MLLMs by a large margin (e.g., InternVL2.5-8B increases by 15.71% and 17.69% on HR-Bench) and also enables small 3-8B MLLMs to outperform strong large models such as GPT-4o. Code: https://github.com/om-ai-lab/ZoomEye

1 Introduction

Text-level reasoning leaves the image unchanged, limiting access to fine-grained details in high-resolution scenes. Zoom Eye adds tree-based visual search, and the paper reports improvements across evaluated MLLMs and a benefit from deeper search.

1 Introduction

  • Text-level reasoning keeps visual input static, restricting fine-grained perception in element-rich high-resolution images.The paper motivates vision-level reasoning as active zooming into informative regions.Source: §1 · p. 2
  • Zoom Eye uses a hierarchical image tree for MLLM search, with the full image at the root and zoomed subregions as child nodes.The tree supports looking ahead into finer regions and backtracking to inspect alternatives.Source: §1 · p. 2
  • The MLLM identifies relevant objects, scores candidate nodes with two confidence values, and stops when it can confidently answer.The final response uses visual information gathered during the search.Source: §1 · p. 2
  • All evaluated MLLMs improve substantially over baseline when equipped with Zoom Eye on high-resolution visual-understanding benchmarks.The evaluated models include Qwen2.5VL, LLaVA-v1.5, LLaVA-OneVision, and InternVL2.5.Source: §1 · p. 2
  • Performance consistently improves as the number of search steps increases, indicating a vision-level test-time scaling phenomenon.The paper connects deeper exploratory search with inference beyond static image perception.Source: §1 · p. 2

2 Preliminary

The section describes fixed-resolution image preprocessing and AnyRes, then explains how visual representations and text are jointly input to an MLLM.

2 Preliminary

  • Naive preprocessing resizes an image to a preset resolution before a vision encoder produces visual representations.The fixed, limited resolution motivates flexible processing methods.Source: §2 · p. 3
  • AnyRes divides an image into equal-area blocks up to a maximum count, encodes the blocks and whole image, and integrates their representations.The naive method is a special case when there are no blocks.Source: §2 · p. 3
  • An MLLM feeds encoded visual representations together with text into its language model to generate an output sequence.The visual representations come from naive resizing or AnyRes preprocessing.Source: §2 · p. 3

3 Methodology

Zoom Eye instantiates image exploration as tree search: it represents crops hierarchically, prioritizes nodes using MLLM confidence, and stops when a view suffices to answer the question.

3.1 Abstraction of Tree Search

  • The abstract search loop maintains a tree T, candidate-node container Q, ranking function R, and stopping criterion S to select and visit nodes.It starts from the root, expands visited nodes, sorts candidates, and stops when S is met.Source: §3.1 · p. 3

3.2 Tree Representation for Image

  • Each image-tree node represents a crop with normalized bounding-box coordinates, and oversized patches are recursively divided into four children until they meet the encoder’s resolution limit.The root represents the full image; MLLMs can receive either only a local patch or the global image plus a highlighted local patch.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 Ranking Function

  • Zoom Eye ranks nodes using an MLLM’s estimates of whether a cue is visible now and whether further zooming may reveal it, weighting these confidence values by node depth.Shallow nodes give more weight to latent confidence; deeper nodes shift weight toward existing confidence.Source: §3.3 · pp. 4–5

3.4 Stopping Criterion

  • Search stops when the MLLM’s answering confidence that the current view suffices to answer the question exceeds a predefined threshold τ.Source: §3.4 · p. 5

3.5 Overall Search Algorithm

  • Zoom Eye generates question-relevant visual cues, searches for single-instance cues until the stopping criterion is met, and collects sufficiently confident nodes across the tree for all-instance cues.It unions the selected nodes’ bounding boxes and uses the resulting focused region with the question to generate the final response.Source: §3.5 · p. 5

4 Experiments

The experiments evaluate Zoom Eye across high-resolution and real-world benchmarks, then examine its scaling behavior, zooming choices, comparisons, and case studies. Results include improvements across evaluated models, alongside performance declines on some real-world subtasks.

4.1 Implementation Details

  • Implementation settings vary by input method: local input uses LLaVA-v1.5-7B, while global-plus-local input uses Qwen2.5VL-3B, LLaVA-OneVision-7B, and InternVL2.5-8B.The specified maximum search depth for type 2 cues is 2; the local-input setting uses τ = 0.8 and b = 0.2, while global-plus-local uses τ = 0.6 and b = 0.6.Source: §4.1 · p. 5

4.2 Results on High-Resolution Benchmark

  • 14.19%, 6.63%, and 10.00% improvement for LLaVA-ov-7B on V∗Bench, HR-Bench 4K, and HR-Bench 8K, respectively, with gains across all evaluated models.The benchmarks test high-resolution visual understanding; Table 2 reports results for multiple models and includes FSP and FCP subtasks.Source: Table 2 · p. 6, §4.2 · p. 6

4.3 Results on Real-World Benchmark

  • LLaVA-ov-7B improves on most reported MME-RealWorld subtasks, but declines on some, including RS/Position where it locates the target yet gives an incorrect answer.The reported gains include MO/Intention (+11.23%), MO/Color (+12.9%), and AD/Motion (+12.1%). The authors suggest difficulty linking positional relationships between the full image and sub-images may explain the RS/Position decline.Source: §4.3 · pp. 6–7

4.4.1 Vision-level test-time scaling

  • As search steps increase, LLaVA-ov-7B’s performance on V∗Bench improves and eventually stabilizes, supporting vision-level test-time scaling.The authors suggest adjusting the confidence threshold or maximum search steps to balance performance and efficiency.Source: §4.4.1 · p. 7, Figure 4 · p. 7

4.4.2 Does the Zoom operation contribute to the improvement of the MLLM?

  • 54.55% to 93.45% accuracy when Zoom is successful rather than unsuccessful, with success defined as the searched box covering at least 50% of the target object.The comparison uses V∗Bench results.Source: §4.4.2 · p. 7, Table 4 · p. 7

4.4.3 Impact of the various number of the split sub-regions

  • More split sub-regions slightly improve performance while reducing average search steps, and results remain stable across zooming granularities.Table 5 reports performance conditioned on the number of split sub-regions; Figure 5 marks searched patches and gives image resolution.Source: §4.4.3 · p. 7, Table 5 · p. 7, Figure 5 · p. 8

4.5.1 Zoom Eye vs. V∗; 4.5.2 Zoom Eye vs. Others

  • Compared with V∗, Zoom Eye uses native MLLM capabilities without additional training or a separate search model, and the authors report superior performance; it also outperforms DC2, VisCrop, and Pixel Reasoner.The V∗ comparison uses a specifically trained 224px LLaVA-v1.5 model for input-resolution matching.Source: §4.5.1 · p. 8, §4.5.2 · p. 8, Table 6 · p. 8

4.6 Case Study

  • Case studies cover single and multiple type 1 cues and type 2 cues, showing Zoom Eye seeking relevant cues so the MLLM can focus on crucial visual information.The examples include error cases discussed in the real-world benchmark results.Source: §4.3 · p. 7, §4.6 · pp. 8–9

5 Related Work

Related work covers multimodal language models and tree-based search. Zoom Eye positions image exploration as a search over hierarchical visual features.

Multimodal LLMs

  • Prior MLLM work explores visual instruction data, encoders, AnyRes, and high-resolution encoders, while the paper argues that image perception still remains tied to the original image.The authors aim to let MLLMs explore hierarchical image features to capture key information.Source: §5 · p. 9

Tree-based search

  • Text-only tree-search work expands reasoning beyond chain reasoning, and Zoom Eye instead models an image as a tree for searching crucial visual information.The paper identifies V∗ as a closely related work and compares methods in §4.5.1.Source: §5 · p. 9

6 Limitations

Zoom Eye has limitations in its heuristic search, fixed-size patching, and focus on natural images. These constrain its fit across image types and tasks.

6 Limitations

  • Heuristic ranking functions and stopping criteria may not generalize optimally across all image types or task conditions.The current search procedure relies on manually defined heuristics.Source: §6 · p. 9
  • Fixed-size patches may not align with semantic regions, so visual cues can be fragmented or overlooked during traversal.Source: §6 · p. 9
  • Zoom Eye is primarily tailored to natural images with spatially distributed elements and is less applicable to document tasks centered on layout, reading order, or structured information.The authors identify learnable search strategies and adaptive patch partitioning as future directions.Source: §6 · p. 9

7 Conclusion

The conclusion presents Zoom Eye as a vision-level tree search method for finding detailed image information. Experiments report substantial performance improvements and a test-time scaling phenomenon, while failure-case analysis identifies limitations in current MLLMs’ visual reasoning.

7 Conclusion

  • Zoom Eye uses a tree search to navigate image structure and capture detailed, crucial information.The conclusion describes it as a vision-level reasoning method.Source: §7 · p. 9
  • Experiments show broad applicability and substantial performance improvement, alongside a test-time scaling phenomenon analogous to text-level reasoning.Source: §7 · p. 9
  • Failure-case analysis identifies inherent limitations in current MLLMs’ visual reasoning capabilities as a target for future work.Source: §7 · p. 9

A Results of More MLLMs on High-Resolution Benchmark

Additional evaluations on high-resolution benchmarks report improved performance after adapting all evaluated models to Zoom Eye, including models of smaller and larger scale.

A Results of More MLLMs on High-Resolution Benchmark

  • All evaluated models improve after adaptation to Zoom Eye on the high-resolution benchmarks in Table 8.The additional models include smaller or larger scales; the passage gives no specific scores.Source: A Results of More MLLMs on… · p. 13
  • Table 8 reports results for additional MLLMs on high-resolution benchmarks, including models of smaller or larger scale.Source: A Results of More MLLMs on… · p. 13
  • The reported improvement is described as consistent with findings in the main paper and as further demonstrating effectiveness in complex visual scenarios.Source: A Results of More MLLMs on… · p. 13

B Compared with Other HR Processing Methods

Zoom Eye differs from DC2 by using question-driven simulated zooming rather than question-agnostic textual descriptions. The passage reports better performance for Zoom Eye in Table 6.

B.1 Zoom Eye vs. DC2

  • Zoom Eye uses simulated zooming to let the MLLM actively discover missing visual details, unlike DC2’s textual supplementation.Source: §B.1 · p. 13
  • Zoom Eye searches for visual cues based on the question, whereas DC2 generates descriptions consistently across questions, which may produce unfocused text.Source: §B.1 · p. 13
  • Table 6 shows better performance for Zoom Eye than DC2.Source: §B.1 · p. 13

B.2 Zoom Eye vs. Pixel Reasoner

Zoom Eye is compared with Pixel Reasoner and VisCrop/MKWTL as a training-free, structured alternative for visual zooming. Its multi-step search is reported to remain stable as image resolution increases, unlike a single-look-again approach.

B.2 Zoom Eye vs. Pixel Reasoner

  • Zoom Eye outperforms Pixel Reasoner on both benchmarks even with a smaller Qwen2.5VL-3B backbone than Qwen2.5VL-7B.Pixel Reasoner requires a supervised fine-tuning data pipeline and resource-intensive reinforcement learning; Zoom Eye relies solely on prompting.Source: §B.2 · p. 13
  • Zoom Eye's comparison with Pixel Reasoner emphasizes competitive or superior performance without fine-tuning or task-specific training.The paper describes this as making Zoom Eye a more adaptable solution for vision-level reasoning.Source: §B.2 · p. 13

B.3 Zoom Eye vs. VisCrop

  • Zoom Eye guides the MLLM through confidence-driven tree search until it finds a high-confidence answer node, rather than revisiting one attention-focused region.The paper characterizes this as “look multiple times” in a structured and semantically informed way.Source: §B.3 · p. 13
  • Zoom Eye maintains stable performance from HR-4K to HR-8K, while MKWTL degrades significantly as resolution increases.The paper contrasts Zoom Eye's structured, repeated search with MKWTL's single “look again” operation.Source: §B.3 · p. 13
  • The paper frames Zoom Eye's repeated search as multi-step visual reasoning that becomes increasingly beneficial as visual complexity grows.This contrasts with MKWTL's “second glance” approach.Source: §B.3 · p. 13

C Complete Results on MME-RealWorld Benchmark

On MME-RealWorld, Zoom Eye improves LLaVA-ov-7B on most reported sub-tasks, with particularly large gains on three tasks. The full results also include declines on some sub-tasks.

C Complete Results on MME-RealWorld Benchmark

  • 20.22% on Personcolor, 29.11% on Motionvehicle, and 12.93% on Visualtrafficsignal are reported improvements for LLaVA-ov-7B with Zoom Eye.The complete results cover MME-RealWorld categories Monitoring, Autonomous Driving, OCR, and Remote Sensing; Zoom Eye was not tested on Diagram and Table.Source: C Complete Results on MME-RealWorld… · p. 13, Table 9 · p. 14
  • Zoom Eye improves LLaVA-ov-7B across most MME-RealWorld sub-tasks, but performance declines on some sub-tasks.The paper analyzes these declines in the main paper and identifies limitations of the employed MLLM.Source: C Complete Results on MME-RealWorld… · pp. 13–14

D Implementation Details

The appendix specifies Local Input and Global + Local Input configurations, shared search settings, and the complete Zoom Eye workflow. The two input modes differ in what visual context is supplied and how large unions of searched patches are handled.

D Implementation Details

D.1 Local Input

  • For Local Input, the appendix specifies LLaVA-v1.5-7B/13B at 336px, a stopping threshold τ = 0.8, and bias b = 0.2.When the searched-patch union is large, patches are pasted onto a blank image at their original relative positions instead of forming one union; this handling applies only to Local Input.Source: §D.1 · pp. 14–15, Figure 6 · p. 15

D.2 Global + Local Input

  • Global + Local Input uses LLaVA-OneVision-0.5B/7B at 384px with AnyRes processing, a maximum of 12 AnyRes blocks, τ = 0.6, and bias b = 0.6.Unlike Local Input's large-union fallback, Global + Local consistently supplies the full union patch.Source: §D.2 · p. 15, Figure 6 · p. 15

D.3 Additional Settings

  • For type 2 cues, both implementations cap search depth at 2 and use τ2 = 0.8; for type 1 search, τ decreases by 0.1 after the step threshold and every δ steps.The shared settings are δ = 2, τmin = 0, and C = D × 3.Source: §D.3 · pp. 15–16

D.4 Complete Algorithm Workflow

  • The workflow generates visual cues, builds an image tree, searches for cue-relevant nodes, unions their bounding boxes, and answers using the resulting visual input.The search procedure distinguishes type 1 from type 2 cues; type 1 ranks and visits nodes, while type 2 collects sufficiently confident nodes up to depth 2.Source: §D.4 · p. 16

Related papers

Loading 2411.16044v4…