PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolScope: An Agentic Framework for Vision-Guided and Long-Horizon Tool Use

Mengjie Deng, Guanting Dong, Zhicheng Dou

arXiv:2510.27363v1cs.AI

TL;DR

Multimodal agents still face challenges integrating external tools with global planning and preserving visual information across multi-step reasoning. ToolScope combines strategic planning, iterative tool use, and a Perceive tool for local visual grounding. Across four VQA benchmarks, it outperforms baselines, with average accuracy improvement up to +6.69%.

  • Problem

    Multimodal agents face challenges with global planning and retaining, referencing, and reusing visual information across multi-step inference.

    Source: §1 · p. 2

  • Method

    ToolScope combines a Global Navigator, Agentic Executor, and Response Synthesizer with a Perceive tool for on-demand visual grounding during iterative reasoning.

    Source: §1 · p. 2

  • Results

    Across four VQA benchmarks, ToolScope outperforms all baselines, with average accuracy improvement up to +6.69%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support training-free generalization across four diverse VQA benchmarks against prompt-based and tool-based baselines.

    Source: §6 · p. 8

    Evaluation covers four research-oriented benchmarks and may not capture open-world or safety-critical scenarios; real-user settings are left for future work.

    Source: §7 · p. 9

Abstract

from arXiv · show

Recently, large language models (LLMs) have demonstrated remarkable problem-solving capabilities by autonomously integrating with external tools for collaborative reasoning. However, due to the inherently complex and diverse nature of multimodal information, enabling multimodal large language models (MLLMs) to flexibly and efficiently utilize external tools during reasoning remains an underexplored challenge. In this work, we introduce ToolScope, an agentic framework designed to unify global planning with local multimodal perception, adopting a specialized Perceive tool to mitigates visual context degradation in long-horizon VQA task. ToolScope comprises three primary components: the Global Navigator, the Agentic Executor, and the Response Synthesizer. The Global Navigator functions as a "telescope", offering high-level strategic guidance. The Agentic Executor operates iteratively to augment MLLM with local perception through the integration of external tools-Search, Code, and Perceive. Finally, the Response Synthesizer consolidates and organizes the reasoning process into a coherent, user-friendly output. We evaluate ToolScope on four VQA benchmarks across diverse domains, including VQA 2.0, ScienceQA, MAT-Search and MathVista. It demonstrates strong generalization capabilities, achieving an average performance improvement of up to +6.69% across all datasets.

1 Introduction

ToolScope addresses gaps in global planning and visual-context retention with a multimodal agent that combines strategic guidance, iterative tool use, and local perception. Across four VQA benchmarks, it outperforms baselines, with an average accuracy improvement of up to +6.69%.

1 Introduction

  • Existing agent approaches lack global planning and can lose visual context over extended inference, motivating guidance that supports coherent multimodal decisions.The introduction frames global planning and retaining, referencing, and reusing visual information across multi-step inference as open challenges.Source: §1 · pp. 1–2
  • ToolScope combines a Global Navigator, Agentic Executor, and Response Synthesizer to integrate high-level planning with local multimodal perception.The Navigator provides strategic guidance before tool invocation, while the Executor reasons iteratively over decomposed subproblems.Source: §1 · p. 2
  • The Perceive tool supports focused visual subquestions throughout reasoning, treating the image as queryable memory rather than relying only on static encoding.This is intended to improve fine-grained grounding and visual consistency during long-horizon decisions.Source: §1 · p. 2
  • Across four VQA benchmarks, ToolScope outperforms all baselines, with average accuracy improvement up to +6.69%.It also achieves the highest average performance across three MLLM series.Source: §1 · p. 2
  • ToolScope autonomously uses code execution, multimodal retrieval, and image re-attention to mitigate visual-context degradation in long-horizon VQA.The perception module lets the agent revisit images during reasoning.Source: §1 · pp. 2–3
  • The paper also reports ablations, qualitative analyses, and case studies examining tool use in agentic multimodal reasoning.The experiments cover four VQA datasets spanning diverse domains.Source: §1 · p. 3

2 Related Work

Prior multimodal tool-use work ranges from trained agents to tool-augmented reasoning, planning-and-execution, and fixed workflows. The paper identifies gaps in global context integration and visual-context preservation.

2 Related Work

  • Training-based agent methods use supervised fine-tuning or reinforcement learning, but can incur high resource costs, especially in multimodal domains.The cited approaches include Liu et al. (2024), Liu et al. (2025), and Wu et al. (2025).Source: §2 · p. 3
  • Multimodal tool-use methods interleave reasoning with external visual tools, coordinate image operations, or invoke tools through predefined workflows.Examples include ReAct extensions, planning-and-execution approaches, and workflow-based approaches.Source: §2 · p. 3
  • These methods can overlook global context or lack coherent integration of global and local perception, while visual-context degradation remains underexplored.The paper positions its training-free framework and dedicated perceive module as addressing these gaps.Source: §2 · p. 3

3 Preliminary

The preliminary formulation treats tool-augmented VQA as generating a grounded answer from an image and question, potentially using external tools. It divides reasoning into global planning, iterative execution, and response synthesis.

3.1 Problem Formulation

  • Tool-augmented VQA takes an image and natural-language question and seeks an accurate, grounded answer, potentially using external tools.Examples include vision experts, OCR engines, knowledge retrievers, and code executors.Source: §3.1 · p. 3
  • The framework decomposes reasoning into global planning, local tool-augmented reasoning, and final response synthesis.The global plan selects relevant tools and provides high-level guidance; the reasoning chain proceeds through tool calls or internal deductions.Source: §3.1 · p. 3
  • The final answer is synthesized from the reasoning trace produced iteratively from the image, question, global plan, and prior reasoning steps.Each step is conditioned on the current image and question, the plan, and earlier steps.Source: §3.1 · p. 3

4 Methodology

ToolScope organizes VQA reasoning into global planning, iterative multimodal tool use, and final response synthesis. Its modules select and guide tools, execute reasoning with tool results, and condense the trajectory into an answer.

4.1 Method Overview

  • ToolScope processes VQA through global planning, iterative multimodal tool use, and response synthesis.The three components are the Global Navigator, Agentic Executor, and Response Synthesizer.Source: §4.1 · p. 3, Figure 2 · p. 4

4.2 Global Navigator

  • The Global Navigator analyzes task intent and complexity, selects a tool subset, and passes a plan and selected tools to the executor.It may select no tools for simple tasks or multiple tools for more complex ones.Source: §4.2 · p. 4

4.3 Agentic Executor

  • The Agentic Executor iteratively reasons and invokes tools, with Perceive enabling targeted visual re-attention and Search and Code supporting knowledge retrieval and computation.Perceive uses the backbone MLLM without external detectors, OCR engines, or specialized perception models; Code execution errors can be fed back for revision.Source: §4.3 · p. 5

4.4 Response Synthesizer

  • The Response Synthesizer aggregates the reasoning trajectory, filters redundant attempts, failed tool invocations, and irrelevant detours, then produces the final answer.It aims for coherence, completeness, and alignment with the original question and image.Source: §4.4 · p. 5

5 Experiment

Across four VQA benchmarks, ToolScope improves on baselines, while ablations and scaling analyses examine its components and performance across operating settings.

5.1 Experimental Setups

  • The evaluation samples 300 VQA 2.0 validation questions, 300 image-based ScienceQA test questions, 75 hard-split MAT-Search questions, and 1,000 MathVista testmini examples.Accuracy is used for MathVista and VQA 2.0; Exact Match accuracy is used for the other three datasets.Source: §5.1 · p. 6

5.2 Main Experimental Results

  • MiMo-VL-7B-RL ToolScope gains up to +9.12% on MAT-Search, +4.67% on ScienceQA, and +1.4% on both VQA 2.0 and MathVista.The paper reports that ToolScope consistently outperforms baselines and achieves these gains without task-specific fine-tuning.Source: §5.2 · pp. 6–7
  • Average improvements across all datasets are +6.69% on MiMo-VL-7B-RL, +6.33% on InternVL3-78B, +5.50% on InternVL3-8B, and +4.81% on Qwen2.5-VL-7B.The reported advantages extend across backbone families and model sizes.Source: §5.2 · p. 7

5.3 Ablation Study

  • Removing Search lowers MAT-Search accuracy from 39.40% to 33.10%; removing Code lowers MathVista from 65.3% to 64.0%, and Perceive removal also reduces performance.Search is reported as the most impactful tool; Code produces the largest drop on MathVista.Source: Table 2 · p. 7, §5.3 · p. 7

5.4 Scaling with Top-k Retrieval Documents.

  • ToolScope beats both baselines at every k and peaks at k = 8; Naïve RAG degrades monotonically as k increases on ScienceQA.The analysis describes a trade-off: small k may provide insufficient context, while large k may introduce irrelevant or noisy content.Source: Figure 3 · p. 7, §5.4 · p. 7

5.5 Scaling with Reasoning Max Turns

  • More reasoning turns improve ToolScope on MathVista and VQA 2.0, but gains taper beyond five turns and excessive depth may hurt VQA 2.0.The largest improvements occur when increasing the maximum turns from 1 to 4.Source: Figure 4 · p. 8, §5.5 · p. 8

5.6 Scaling with Model Size

  • On ScienceQA, ToolScope outperforms baselines across the tested InternVL3 model scales, while accuracy generally improves as model size increases.The analysis tests 2B, 8B, 9B, 14B, and 78B variants.Source: Figure 5 · p. 8, §5.6 · p. 8

6 Conclusion

ToolScope combines global planning, iterative multimodal tool use, and response synthesis, with Perceive enabling on-demand visual subquestions. The paper reports strong generalization across four VQA benchmarks without fine-tuning.

6 Conclusion

  • ToolScope integrates global task planning with local multimodal perception through the Global Navigator, Agentic Executor, and Response Synthesizer.These components provide strategic planning, iterative tool-augmented reasoning, and final response refinement.Source: §6 · p. 8
  • Perceive treats the image as dynamic perceptual memory, allowing the agent to formulate and resolve visual subquestions on demand.The paper presents it as a response to visual context degradation during extended reasoning.Source: §6 · p. 8
  • Without fine-tuning, ToolScope outperforms prompt-based and tool-based baselines across four diverse VQA benchmarks.The conclusion describes the framework as a general and extensible blueprint for multimodal agents.Source: §6 · p. 8

7 Limitations

ToolScope’s evaluation and deployment claims are bounded by its benchmark coverage, baseline selection, retrieval corpus, and tool-runtime scope. The authors also flag safety considerations requiring stronger governance beyond research settings.

7 Limitations

  • Evaluation covers four research-oriented benchmarks and may not fully capture open-world or safety-critical scenarios; real-user settings remain future work.The authors identify this as a scope boundary for the evaluation.Source: §7 · p. 9
  • The comparison does not exhaustively cover prior agentic frameworks because of reproducibility and engineering constraints.Source: §7 · p. 9
  • Retrieval experiments primarily use Wikipedia, while heterogeneous or domain-specific corpora may add integration costs and failure modes.Source: §7 · p. 9
  • The Code tool targets short-running, stateless snippets and lacks support for long-running jobs, heavy dependencies, or file I/O without additional governance.Source: §7 · p. 9
  • Search and code execution raise risks including hallucinated citations, unsafe code suggestions, and inadvertent leakage; deployment beyond research settings requires stronger policy, auditing, and sandboxing.Source: §7 · p. 9

A.1 Implementation Details

Experiments use off-the-shelf multimodal LLMs without training or attached perception modules, with a capped reasoning budget and fixed decoding settings. Retrieval uses an English Wikipedia corpus and CLIP ViT-B/16.

A.1 Implementation Details

  • Experiments use off-the-shelf MLLMs in a strictly training-free setting, with Perceive supplied by the backbone rather than external perception modules.The authors cap reasoning at 10 turns; increasing the cap monotonically improves accuracy with moderate latency growth, and 10 turns offers a favorable accuracy–latency trade-off.Source: §A.1 · p. 11
  • Decoding hyperparameters are fixed across experiments, with distinct temperature, top-p, and top-k settings for Qwen2.5-VL and MiMo-VL versus InternVL3.All models use repetition penalty 1.05, and outputs retain tool-invocation stop strings for auditing.Source: §A.1 · p. 11
  • Retrieval uses English Wikipedia pages segmented into 256-word chunks with 32-word overlap, and CLIP ViT-B/16 as the retriever.Pages under 32 words are removed; the resulting corpus is reported as approximately 15.8 billion passages.Source: §A.1 · p. 11

A.2 Case Study

The case study contrasts two examples in which ToolScope selects tools suited to different task demands: Search for factual retrieval and Code for mathematical computation.

A.2 Case Study

  • In the MAT-Search example, ToolScope selects Search to retrieve the exhibition locations of two paintings, while in MathVista it selects Code to compute an unknown using the Pythagorean theorem.The authors present the examples as evidence of task-appropriate tool selection and goal-directed use.Source: §A.2 · p. 11

A.3 Latency Analysis

The latency analysis reports modest reasoning depth and identifies model time as the largest share of end-to-end latency. The appendix prompts describe how the Navigator plans tool use and how the Executor, Search refiner, and Synthesizer support execution and output formation.

A.3 Latency Analysis

  • End-to-end latency is measured from question receipt to final answer, using median p50 across each full benchmark evaluation set.The analysis separates MLLM time from overall latency and uses Qwen2.5-VL-7B on one NVIDIA A800 with batch size 1 and a 10-turn cap.Source: §A.3 · p. 11
  • 3.29–4.21 turns is the observed reasoning depth under a 10-turn cap; LLM time accounts for approximately 65–71% of p50 latency, and median latency is 2.3–5.9s.The authors identify LLM time as the principal bottleneck and note longer latency for harder multi-step tasks; reported medians are acceptable for research evaluation and interactive analysis, with optimization headroom.Source: §A.3 · p. 11, Table 3 · p. 11

A.4 Prompt

  • The Global Navigator prompt requests tool selection and a concise high-level plan, allowing no tools for simple tasks and modular tool combinations for complex ones.Available tools are Search for background knowledge, Code for computation or logic, and Perceive for fine-grained visual information.Source: §A.4 · p. 12
  • The Agentic Executor prompt directs step-by-step reasoning with Search, Perceive, or Code requests before returning a final answer.Source: §A.4 · p. 12
  • The Response Synthesizer prompt forms a clear reasoning path, extracts the final answer, and reformats it for evaluation.Source: §A.4 · p. 13
  • The Search refiner prompt integrates information relevant to the current query from retrieved documents into the existing reasoning chain.Its example asks the refiner to extract directly relevant content and rephrase it fluently rather than quote the documents.Source: §A.4 · p. 13
Loading 2510.27363v1…