PaperlayerView exact revision on arXiv

Source-linked AI summary

OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning

Pan Lu, Bowen Chen, Sheng Liu, Rahul Thapa, Joseph Boen, James Zou

arXiv:2502.11271v2cs.LGcs.CLcs.CVcs.MA

TL;DR

Complex reasoning requires coordinated multi-step decomposition, specialized knowledge, computation, and visual understanding beyond what existing restricted or training-dependent tool methods provide. OctoTools introduces a training-free planner-executor framework with standardized tool cards and task-specific tool selection. Across 16 benchmarks, it reports broad accuracy gains over GPT-4o and other agentic frameworks, while remaining robust across models and noisy tools.

  • Problem

    Complex reasoning spans visual understanding, specialized knowledge, computation, and multiple steps, while existing tool-augmented methods can be domain-limited or require additional training.

  • Method

    OctoTools uses standardized tool cards, a planner-executor workflow for multi-step tool calls, and lightweight task-specific toolset optimization without updating model weights.

  • Results

    Across 16 diverse benchmarks, OctoTools achieves average accuracy gains of 9.3% over GPT-4o and up to 10.6% over strong agentic frameworks using the same tools.

  • Takeaways & Limitations

    The analyses attribute improvements to dynamic task planning, effective tool usage, and multi-step problem decomposition, with robustness under weaker backbones and noisy tool environments.

  • Takeaways & Limitations

    Reproducibility depends on snapshotting third-party tool versions or containerized environments because interfaces, quotas, and model versions may evolve.

Abstract

from arXiv · show

Solving complex reasoning tasks may involve visual understanding, domain knowledge retrieval, numerical calculation, and multi-step reasoning. Existing methods augment large language models (LLMs) with external tools but are restricted to specialized domains, limited tool types, or require additional training data. In this paper, we introduce OctoTools, a training-free, user-friendly, and easily extensible multi-agent framework designed to tackle complex reasoning across diverse domains. OctoTools introduces standardized tool cards to encapsulate tool functionality, a planner for both high-level and low-level planning, and an executor to carry out tool usage. We validate OctoTools' generality across 16 diverse tasks (including MathVista, MMLU-Pro, MedQA, and GAIA-Text), achieving substantial average accuracy gains of 9.3% over GPT-4o. Furthermore, OctoTools also outperforms AutoGen, GPT-Functions, and LangChain by up to 10.6% when given the same set of tools. Through comprehensive analysi, ablations, and robustness tests with compact backbones and noisy tool environments, OctoTools demonstrates advantages in task planning, effective tool usage, and multi-step problem solving. Code, demos, and visualization are publicly available at https://octotools.github.io/.

1 Introduction

Complex reasoning tasks require coordinated visual understanding, specialized knowledge, computation, and multi-step decomposition, while existing tool-augmented methods remain restricted in coverage or training requirements. OctoTools addresses this with a training-free, extensible planner-executor framework and shows broad gains across 16 benchmarks.

  • Complex reasoning often combines multiple steps, logical decomposition, visual understanding, computation, and specialized domain knowledge.
  • Existing tool-augmented approaches may require supervised training, lack refinement, or focus on specialized tool domains.
  • OctoTools uses standardized tool cards to encapsulate heterogeneous tools and support integration without retraining or framework changes.
  • A planner handles global and step-level decisions, while an executor converts actions into commands and updates structured context for final summarization.
  • 9.3% average accuracy gain over zero-shot GPT-4o and 7.7% over chain-of-thought was achieved across 16 reasoning benchmarks.
  • OctoTools combines multi-step planning and specialized tool usage, with planning aiding decomposition and tools aiding intricate calculations or specialized knowledge.

2 The OctoTools Framework

OctoTools structures tool use as an iterative planner-executor-verifier process around modular tool cards. A validation-based optimizer can select beneficial tools for each task, while the framework records intermediate results and refines actions until solving or reaching limits.

  • 2.1 Preliminary: OctoTools replaces direct single-step responses with a structured, multi-step process that leverages external tools.
  • 2.3 Planner: The planner proposes a global plan, selects tools and sub-goals, and iteratively refines actions using evolving context.
  • 2.4 Executor: The executor converts planner actions into machine-executable commands, runs them, and records intermediate outputs in structured context.
  • 2.2 Tool Cards: Tool cards package each tool with metadata such as input-output types, constraints, versions, and command demonstrations, enabling low-effort integration or replacement.
  • 2.4 Executor: A context verifier checks completeness and ambiguity after each execution, continuing the loop or triggering final solution summarization.
  • 2.5 Task-specific Toolset Optimization: The optimization algorithm evaluates candidate tools alongside a base set and retains those producing positive validation-accuracy improvements.
  • 2.5 Task-specific Toolset Optimization: The selected subset does not guarantee a global optimum, although overall improvements over using all tools are observed.

3 Experiments

OctoTools is evaluated across 16 diverse benchmarks and compared with standard prompting, agent frameworks, and controlled tool-use settings. The experiments examine benchmark coverage, tool usage, multi-step reasoning, and performance gains.

  • Experimental Setups: The evaluation spans 16 benchmarks covering visual, mathematical, scientific, medical, and agentic reasoning tasks.The benchmarks include diverse reasoning skills such as visual understanding, numerical calculation, knowledge retrieval, and multi-step reasoning.
  • Experimental Setups: The experiments use validation examples for toolset optimization and ablations, held-out test examples for final evaluation, and three-trial averages with standard deviations.Each benchmark generally uses 100 validation examples and 200 held-out test examples, subject to dataset size.
  • Main Results: OctoTools achieves 58.5% average accuracy, improving over zero-shot prompting by 9.3% and chain-of-thought by 7.7%.OctoToolsbase improves over zero-shot by 4.2% and CoT by 2.6%, indicating a contribution from step-by-step reasoning independent of external tools.
  • Comparisons with Other Agent Frameworks: With the same model, tools, reasoning budget, and time budget, OctoTools improves average accuracy over AutoGen by 10.6%, GPT-Functions by 7.5%, and LangChain by 7.3%.The controlled setup uses GPT-4o, a 10-step maximum reasoning budget, and a 300-second time budget.
  • Analysis of Performance Gains: OctoTools uses both a base decomposition tool and specialized external tools, with external tools accounting for 67.8% of usage versus 10.6% for AutoGen.It selects task-specific tools, such as five tools for GAIA-Text and vision, search, and calculation tools for MathVista.
  • Analysis of Performance Gains: Tasks requiring both many steps and substantial external tool usage include Game of 24 and GAIA-Text.Game of 24 frequently uses Python_Code_Generator to explore and verify arithmetic expressions, while GAIA-Text requires multiple tools for varied skills.

4 Ablation Study

The ablation study examines reasoning-step budgets, toolset strategies, backbone strength, and noisy-tool robustness. Performance generally improves with longer reasoning chains, optimized toolsets, and diverse backbones, while remaining resilient to substantial tool failures.

  • Reasoning-step budget: Performance tends to improve as the maximum allowed reasoning steps increases, highlighting the benefit of longer multi-step chains.The analysis averages results over three trials across 16 benchmarks.
  • Toolset optimization: 58.9% accuracy with the optimized toolset exceeds the 53.9% base-tool setup by 5.0%.Enabling all tools reaches 57.4% accuracy, 3.5% above the base setup.
  • Toolset optimization: 57.4% accuracy from the full toolset exceeds the base setup by 3.5%, while optimization raises accuracy to 58.9%.The results indicate that broader tool availability helps, but task-specific selection performs better overall.
  • Backbone generalization: OctoTools yields consistent accuracy gains of 5.8–8.4% across four additional backbone LLMs.The evaluated models are Claude 3.5 Haiku, Gemini 2.5 Pro, Grok-2 Vision, and Qwen2.5-VL-72B.
  • Compact backbones and noisy tools: With compact backbones, average accuracy improves by 13.6% with Qwen2.5-3B and 6.3% with Qwen2.5-7B.The compact-backbone evaluation covers four representative tasks.
  • Compact backbones and noisy tools: When tool-call failures occur with probability up to p = 0.4, average accuracy drops by at most 1.6% on 10 tasks.The failures return explicit error messages during inference with Qwen2.5-7B.

5 In-depth Analysis

The in-depth analysis measures resource use, user experience, and failure sources. It reports low average execution counts and costs, positive feedback on interpretability, and failures concentrated in tools and planning rather than command generation.

  • Cost analysis: GPT-4o costs about 0.05 per query, while GPT-4o-mini costs under 0.01 per query.Using Qwen2.5-VL-72B locally eliminates query cost; average executed steps are 2.56 and 3.49, respectively.
  • Human study: The demo processed 3.6K user queries, and 69% of 100 sampled feedback entries received positive upvotes.Users praised the visualized reasoning trajectory and its interpretability.
  • Failure case study: Among 200 failure cases, weak tools accounted for 65.0% and suboptimal planning for 77.5% combined, while invalid command generation accounted for 1.5%.The analysis identifies tool quality and planner reasoning as key areas for improvement.

6 Conclusion

OctoTools is presented as a training-free, extensible multi-agent framework that combines standardized tool cards with planner–executor workflows for complex reasoning. Across 16 diverse benchmarks, it consistently outperforms prompting and agentic-framework baselines, with improvements attributed to dynamic planning, tool usage, and multi-step decomposition.

  • Conclusion: OctoTools is a training-free, extensible multi-agent framework for complex reasoning.Its workflow uses standardized tool cards and separates high-level planning from low-level command generation.
  • Conclusion: Across 16 diverse benchmarks, OctoTools achieves average accuracy gains of up to 9.3% over GPT-4o and up to 10.6% over strong agentic frameworks.The conclusion attributes improvements to dynamic task planning, effective tool usage, and multi-step problem decomposition.

7 Limitations

OctoTools has limitations tied to its base model, external tools, optimization strategy, computational demands, deployment risks, reproducibility, and misuse.

  • Base LLM quality constrains planning, command prediction, and summarization; hallucinated or misinterpreted tool metadata can degrade downstream performance.Stronger or more specialized LLMs can mitigate this issue.
  • Tool benefits may diminish when tools are inaccurate, inaccessible, rate-limited, or deprecated, and robustness to degraded or missing tools remains open.The framework permits tool replacement or upgrades, but does not resolve this robustness challenge.
  • Greedy task-specific toolset optimization is efficient but not globally optimal and may select suboptimal subsets with scarce or biased validation data.The current optimization is static per task rather than query-adaptive.
  • Multi-step planning increases latency, cost, and memory use as step budgets and tool counts grow; some queries can exceed 90 seconds.Large-scale or latency-sensitive applications may require batching, pruning, or model distillation.
  • API calls and generated code can expose user data, execute unsafe code, and propagate tool or model biases.The paper identifies systematic auditing as necessary for fairness, privacy, and security.
  • Third-party interface, quota, and model-version changes can hinder exact reproduction without snapshotting tool versions or containerizing environments.The authors release code and tool cards but identify environment snapshotting as future work.
  • Modularity lowers barriers to building agentic systems while also creating misuse risks, including large-scale disinformation generation.Governance and safeguards are identified as critical but outside this work’s scope.

Broader Impacts and Ethical Considerations

OctoTools may support scientific and healthcare applications while producing reusable reasoning trajectories, but its capabilities also create privacy, security, bias, and misuse risks requiring safeguards and oversight.

  • OctoTools could support scientific discovery through literature search, database access, and computational tools, and assist healthcare report generation.
  • Automated reasoning and tool use could enable disinformation, deceptive profiles, unauthorized surveillance, privacy violations, and unfair outcomes.Risks may arise from malicious use, unintended consequences, or incorrect outputs in critical domains.
  • Recommended mitigations include sandboxing executable tools, human oversight, auditing and replacing biased or insecure tools, and gated access to sensitive tools.The generated trajectories may also support AI safety, interpretability, and robustness research.
  • The authors used LLMs only for writing assistance, with outputs critically assessed, revised, and approved by the authors.LLMs were not used to formulate ideas, design experiments, analyze data, or generate the core scientific arguments.

Contribution Statement

The paper situates OctoTools among tool-augmented LLMs, multi-agent systems, and complex-task reasoning methods, emphasizing training-free integration and structured coordination.

  • Tool-Augmented LLMs: Tool-augmented LLMs offload specialized subtasks to search engines, browsers, calculators, translation systems, or Python interpreters.Earlier approaches often required large-scale fine-tuning, whereas OctoTools uses standardized tool cards and a deterministic planner-executor paradigm.
  • LLM Agents and Multi-Agent Systems: OctoTools differs from flexible peer-to-peer multi-agent systems by assigning strategy and decomposition to a planner and concrete actions to an executor.This fixed hierarchical interaction is described as simplifying coordination, enhancing transparency, and improving reliability.
  • Complex Task Reasoning: Complex-task reasoning commonly decomposes multi-step problems into simpler sub-questions solved step by step.Prior work includes unsupervised or weakly supervised decomposition and prompting methods such as Chain-of-Thought, Least-to-Most, and ReAct.
  • The evaluation uses 16 benchmarks, generally sampling 100 validation examples and 200 test examples per dataset unless otherwise specified.Open-ended questions received no additional preprocessing; multiple-choice options were capitalized and appended on new lines.

B.1.1 General Domain Benchmarks

The general-domain benchmark suite spans visual, mathematical, scientific, medical, and agentic reasoning tasks, with 16 benchmarks covering diverse reasoning requirements. The experiments use specialized tools alongside a base tool to address these tasks.

  • Benchmark coverage: The benchmark suite covers visual understanding, mathematical reasoning, scientific reasoning, medical reasoning, and general AI assistant capabilities.Examples include visual question answering, arithmetic puzzles, expert scientific questions, pathology and radiology tasks, and tool-use-intensive agentic questions.
  • Visual benchmarks: The visual benchmarks require fine-grained perception, language understanding, geometric reasoning, or visual context to determine answers.AlgoPuzzleVQA, Hallusion-VD, PuzzleVQA, and VQA 2.0 target complementary visual reasoning abilities.
  • Mathematical benchmarks: The mathematical benchmarks combine numerical calculation with iterative verification, multimodal reasoning, or advanced competition-level problem solving.Game of 24 requires constructing and checking arithmetic expressions, while other tasks involve mathematical and visual reasoning or Olympiad-level problems.
  • Scientific and medical benchmarks: The scientific and medical benchmarks test expert knowledge, complex reasoning, scientific figure interpretation, diagnosis, and visual pathology or radiology understanding.Medical evaluations include text-only board questions, pathology classification, pathology visual question answering, and radiology VQA.
  • Tool coverage: The toolbox includes general reasoning, search, extraction, image-processing, calculation, and pathology-classification tools for these heterogeneous tasks.The implemented tools include 11 toolbox components, such as Python Code Generator, image tools, web and literature searchers, and Path Generalist Classifier.

C More Experimental Results

Additional experiments examine toolset composition, task-adaptive tool usage, reasoning-chain length, and the prompting components that plan and execute tool calls. The results indicate that optimized toolsets and longer multi-step reasoning are generally beneficial.

  • Toolset optimization: Optimized toolsets select diverse tool combinations across the 16 tasks, with vision, mathematical, and scientific benchmarks favoring different specialized tools.Image_Captioner and Relevant_Patch_Zoomer are common in vision tasks, Python_Code_Generator appears in mathematical tasks, and Wikipedia_Knowledge_Searcher appears in scientific tasks.
  • Task adaptation: OctoTools adapts its number of reasoning steps and tool chains to each task rather than using a fixed procedure.The step distribution shows task-dependent chains formed from different tool sets.
  • Reasoning depth: Most benchmarks improve when the maximum number of allowed reasoning steps increases relative to one step.Figure 18 plots accuracy changes from a maximum step count of 1.
  • Planner and executor prompts: The framework separately prompts components to analyze queries, choose one next tool, and generate executable commands from the selected action.The query analyzer identifies skills and relevant tools, the action predictor selects a single next step, and the command predictor constructs tool-execution code.

E.10 URL Text Extractor Tool

The URL Text Extractor retrieves text from a supplied webpage and returns it in a structured result. Examples demonstrate extraction from both a simple domain and an arXiv paper, while the Wikipedia tool separately searches and extracts article content.

  • Tool interface: The URL Text Extractor accepts a URL and returns extracted page text together with any error messages in a dictionary.Its documented command invokes tool.execute(url=...).
  • Example 1: An example using https://example.com returns the page’s illustrative-domain text.The example shows both unstructured text output and a structured result containing the URL and extracted text.
  • Example 2: An arXiv URL example returns bibliographic and abstract text for Attention Is All You Need.The extracted content includes the title, authors, submission information, abstract, and reported translation results.
  • Related search tool: The Wikipedia Knowledge Searcher instead searches Wikipedia using a query and returns search results plus extracted article text.Its interface takes a string query, and the examples use queries such as kidney, Python programming language, Artificial Intelligence, and Theory of Relativity.

F.1 Object Detector Tool

The Object Detector identifies labeled objects in an image using Grounding DINO and returns detections with confidence scores, bounding boxes, and saved image paths. Its documented parameters support threshold, model size, and padding controls, while accuracy varies across inputs.

  • Tool interface: The Object Detector takes an image, object labels, and optional detection settings, then returns detected objects with scores, boxes, and saved image paths.Supported settings include a confidence threshold, tiny or base model size, and pixel padding around detected objects.
  • Limitations: Detection performance can vary with the image and labels, especially for small, uncommon, or narrowly specified objects.The tool documentation recommends supplementary tools or image processing for verification and notes that the detector is not always reliable.
  • Example 1: An example detects baseball objects in example.png using the label baseball and reports confidence scores, bounding boxes, and output filenames.The example includes detections with confidence scores such as 0.69 and saved paths such as example_baseball_1.png.

F.2 Advanced Object Detector Tool

The Advanced Object Detector Tool detects labeled objects in images and returns structured detections with confidence scores, bounding boxes, and saved object-image paths. Its metadata also describes configurable thresholds, model size, and padding, while examples show related Nature news-fetching outputs.

  • Tool description: The Advanced Object Detector Tool detects labeled objects in an image and saves individual object images with empty padding.Its output includes detected-object lists with scores, bounding boxes, and saved image paths.
  • Inputs and parameters: Inputs include an image path, object-label list, confidence threshold, model size, and padding.The default threshold is 0.35 and default padding is 20 pixels.
  • Limitations: The tool metadata cautions that detection accuracy varies with images and labels, especially for small, uncommon, or narrowly attributed objects.Supplementary tools or image-processing methods may improve verification.
  • Detection output: Example detections identify baseballs with confidence scores, bounding boxes, and saved paths across multiple instances.Reported confidence scores include 0.73 and 0.66 for separate baseball detections.
  • Related tool examples: The supplied Nature News Fetcher examples return article titles, URLs, descriptions, authors, dates, and image URLs.The tool supports configurable article counts and page limits.

G.1 Demonstration Example

The demonstration shows OctoTools combining image captioning with object detection to answer a baseball-counting question. The workflow illustrates iterative verification: an initial caption suggests 20 baseballs, while later tool outputs contain inconsistent counts before the final answer returns 20.

  • Final determination: The detector outputs and intermediate arithmetic contain conflicting totals, including 19, while later highlighted conclusions report 20 baseballs.The trajectory includes a 19-ball bucket breakdown but ultimately states that both tools confirm 20.
  • Task setup: The task asks for the total number of baseballs in an image containing four buckets.The workflow requires image analysis, tool operation, and critical evaluation of tool outputs.
  • Initial perception: OctoTools first uses an Image_Captioner_Tool to describe the image and identify relevant objects before specialized counting.The planner selects captioning as the initial context-building step.
  • Initial result: The caption reports four buckets containing five baseballs each, implying 20 baseballs.The description states that the buckets are arranged in a grid and each contains five baseballs.
  • Verification: The verification trajectory identifies object detection as necessary because the caption-based count has not yet been independently checked.The verifier explicitly marks the memory incomplete until the Object_Detector_Tool is used.

G.2 Example for Multi-step Mathematical Reasoning

The mathematical demonstration uses iterative tool calls to solve a 24-game expression task with numbers [1, 1, 6, 9]. Early generated expressions fail, but systematic enumeration eventually finds valid expressions using all four numbers.

  • Problem setup: The task requires constructing an expression equal to 24 from [1, 1, 6, 9] using basic arithmetic and parentheses.The numbers must be combined into a valid target expression.
  • Planning: The planner selects Python_Code_Generator_Tool to automate testing of arithmetic combinations and parenthesized expressions.The tool is chosen for systematic calculation rather than relying only on general reasoning.
  • Failed attempt: The first generated expression, 6 / (1 - (9 / 1)), evaluates to -0.75 rather than 24.The verifier therefore requires additional exploration.
  • Iterative search: Repeated tool use expands the search over number permutations, operations, and parenthesizations.The code evaluates candidate expressions and prints those matching the target.
  • Successful solution: The search finds valid solutions including ((1 + 1) * 9) + 6, which equals 24.The verifier confirms that the expression uses all four supplied numbers and basic operations.

G.4 Example for Medical Reasoning

The medical examples illustrate OctoTools using specialized perception and reasoning tools to resolve ambiguous image interpretations. In the MRI case, early tool outputs conflict, but later reasoning corrects the interpretation and recommends medical verification; a histopathology example shows direct use of a specialized classifier.

  • MRI task: The MRI task asks which organ appears on the left side of a cross-sectional image.The workflow initially focuses on a zoomed patch of the image.
  • Agentic workflow: OctoTools decomposes the query into patch selection, image captioning, and separate verification, allowing it to correct subpar earlier tool usage.The example explicitly says the first two steps are unhelpful but a separate tool corrects the reasoning trajectory.
  • Initial interpretation: Initial analysis identifies the relevant top-left patch but does not clearly identify the organ.The verifier marks the result ambiguous and recommends further captioning.
  • Conflicting outputs: Image captioning then describes the patch as a cross-sectional view of the brain, creating an inconsistency with the thoracic interpretation.The inconsistency prompts another reasoning step.
  • Correction and verification: A later generalist analysis corrects the context to a thoracic MRI showing lungs and heart, while the trajectory also records a final organ answer and recommends expert verification.The supplied trajectory contains multiple competing organ interpretations, including heart and right lung.
  • Specialized medical tools: In histopathology, OctoTools leverages a specialized tool directly, with the example’s ground truth labeled necrotic tumor.The paper presents this as an instance of using domain-specific tooling.
Loading 2502.11271v2…