OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning

• Across 16 reasoning benchmarks, OctoTools gains 9.3% over zero-shot GPT-4o and 7.7% over chain-of-thought, and improves up to 10.6% over agentic frameworks using the same tools.The evaluation spans vision, mathematics, science, medicine, and agentic tasks.Source: Figure 1 · p. 1, §1 · p. 3
• OctoTools combines standardized tool cards with a planner that handles high- and low-level decisions and an executor that runs tool commands and updates context.Tool cards specify metadata, input-output formats, constraints, and best practices, supporting tool integration or replacement without retraining or framework changes.Source: §1 · pp. 1–3
• Toolset optimization selects task-specific tools using validation performance; the full toolset reaches 57.4% accuracy, 3.5% above the base tool, and optimization raises accuracy to 58.9%.Source: §1 · pp. 2–3
• Analyses find distinct benefits from multi-step planning for reasoning decomposition and specialized tool use for intricate calculations or domain knowledge; demo users praised transparency and interpretability.Controlled comparisons hold the backbone, tool implementations, and reasoning budgets constant to isolate architectural benefits.Source: §1 · pp. 2–3
• OctoTools gains an average 7.1% over GPT-4o-mini across 16 tasks and improves compact Qwen2.5 3B/7B models by 6.3–13.6% on representative tasks, while degrading gracefully under random tool failures.Accuracy also increases with larger step budgets; a failure case study attributes errors mainly to tool limitations rather than the core system.Source: §1 · p. 3

2.1 Preliminary

• The planner decomposes queries into sub-goals and iteratively selects tools, while the executor runs commands and records results for the final solution.The planner refines actions using the evolving context until it finds a complete solution or reaches inference limits, then summarizes the stored trajectory.Source: §2.1 · p. 3

2.2 Tool Cards

• Tool cards standardize each tool’s metadata, command demonstrations, and optional usage constraints, and can be added, replaced, or updated without changing framework logic.Adding a card requires lightweight natural-language metadata and a few demonstration commands; authoring can be partially automated with LLM assistance.Source: §2.2 · p. 4

2.3 Planner Agent

• The planner creates a global plan, selects tools and sub-goals for stepwise actions, then verifies completeness and ambiguity before summarizing the trajectory.If the verifier finds the problem incomplete, the planner predicts another action; the summarizer integrates intermediate outputs and reasoning traces into a concise answer.Source: §2.3 · p. 4

2.4 Executor Agent

• A command generator translates planner actions into executable Python tool calls, and a separate executor runs them and records outputs, logs, or errors in the trajectory.Separating command generation and execution from planning addresses the challenges of executable code and environment dependencies while preserving the action, code, and result history.Source: §2.4 · p. 4

2.5 Task-specific Toolset Optimization

• With validation examples, greedy O(n) toolset optimization adds candidates that individually improve accuracy over a base set, though it does not guarantee a global optimum.The resulting subset is selected once per task; actual tool use remains query-dependent, and the authors report overall improvements over using all tools.Source: §2.5 · p. 5

3.1 Experimental Setups

• Experiments span 16 benchmarks across two modalities, five domains, and four reasoning types, using held-out tests and three-trial average accuracy with standard deviations.Each benchmark used 100 validation examples for toolset optimization and ablations, plus 200 held-out test examples where available; for benchmarks with fewer than 300 samples, tests used all remaining examples.Source: §3.1 · p. 5

3.2 Main Results

• OctoTools reaches 58.5% average accuracy across 16 benchmarks, beating zero-shot by 9.3% and CoT by 7.7%.OctoToolsbase improves over zero-shot by 4.2% and CoT by 2.6%, indicating gains from step-by-step reasoning even without external tools.Source: Table 1 · p. 6, §3.1 · p. 6, §3.2 · p. 6

3.3 Comparisons with Other Agent Frameworks

• With the same toolbox, OctoTools gains 10.6% over AutoGen, 7.5% over GPT-Functions, and 7.3% over LangChain in average accuracy.Frameworks used GPT-4o, identical tool implementations, a 10-step reasoning budget, and a 300-second time budget.Source: Table 2 · p. 6, §3.3 · pp. 6–7

3.4 Analysis of Performance Gains

• OctoTools uses external specialized tools for 67.8% of tool usage versus 10.6% for AutoGen, 23.3% for GPT-Functions, and 10.7% for LangChain.It also uses its base tool for 32.2% of tool usage and adapts tool selection to each task, using five tools on GAIA-Text and specialized perception, search, and calculation tools on MathVista.Source: §3.4 · p. 7, Figure 4 · p. 7
• Game of 24 and GAIA-Text combine high average step counts with high external-tool use, with Game of 24 using Python_Code_Generator to explore and verify arithmetic expressions.GAIA-Text requires multiple tools to address varied underlying skills.Source: §3.4 · p. 7, Figure 5 · p. 7

Number of maximum allowed steps

• Performance tends to improve as the maximum allowed reasoning steps increase, highlighting the benefit of longer multi-step reasoning chains.Results are averaged over three trials across 16 benchmarks.Source: §4 · pp. 7–8, Figure 6 · p. 8

Toolset optimization

• 58.9% average accuracy with the optimized toolset exceeds OctoToolsbase at 53.9% by 5.0%, while enabling all tools reaches 57.4%, 3.5% above the base.The paper notes that enabling all tools still offers substantial improvement as the toolset scales.Source: Figure 7 · p. 8, §4 · p. 8

Using a weaker LLM

• With GPT-4o-mini, OctoTools generally outperforms zero-shot and CoT baselines across all 16 tasks and various categories.Task-specific analyses are provided in §C.Source: Figure 8 · p. 8, §4 · p. 8

Generalization to backbone LLMs

• Across four additional backbones, OctoTools gains 5.8–8.4%, which the paper attributes to its planner-executor design and modular tool integration rather than a specific LLM.The evaluated backbones include Claude 3.5 Haiku, Gemini 2.5 Pro, Grok-2 Vision, and Qwen2.5-VL-72B.Source: Figure 9 · p. 8, §4 · p. 8

Compact backbones and noisy tool environments

• With Qwen2.5-3B and Qwen2.5-7B, OctoTools improves average accuracy by 13.6% and 6.3%; at p = 0.4 tool failures, the 10-task average drop is at most 1.6%.Under injected failures, each tool call returns an error message with probability p, and performance degrades gracefully.Source: §4 · pp. 8–9, Table 3 · p. 9, Table 4 · p. 9

Cost analysis

• GPT-4o costs about 0.05 per query, while GPT-4o-mini costs under 0.01 and local Qwen2.5-VL-72B eliminates cost.Average executed steps were 2.56 with GPT-4o and 3.49 with GPT-4o-mini despite a 10-step limit; the verifier often stops early on simpler queries.Source: Table 5 · p. 9, §5 · p. 9

Human study

• In 100 sampled demo feedback entries, 69% were positive upvotes, and the demo had attracted over 22.9K visits and processed 3.6K queries.The online demo visualizes the initial plan, each tool call, and intermediate results.Source: §5 · p. 9

Failure case study

• 77.5% of failures involved suboptimal planning and 65.0% involved weak tools, while invalid command generation accounted for 1.5%.The authors attribute most errors to tool limitations and imperfect LLM planning, rather than the framework itself.Source: Table 6 · p. 9, §5 · p. 9

6 Conclusion

• Up to 9.3% average accuracy gains over GPT-4o and up to 10.6% over strong agentic frameworks were achieved across 16 diverse benchmarks.The conclusion reports consistent outperformance of baselines.Source: §6 · p. 9
• Standardized tool cards integrate diverse tools, while a planner-executor workflow separates multi-step high-level planning from low-level planning and command generation within each step.OctoTools is described as a training-free, extensible multi-agent framework for complex reasoning.Source: §6 · p. 9
• Dynamic task planning, effective tool use, and multi-step problem decomposition account for OctoTools’s improvements, and the framework supports future work on automated tool discovery and advanced long-horizon planning.The paper presents OctoTools as a robust, practical foundation for more capable, reliable, and transparent AI agents.Source: §6 · p. 9

7 Limitations

• Base-LLM reasoning and code-generation limits can degrade planning, command prediction, and summarization when the model hallucinates or misreads tool metadata.The framework remains robust with weaker models, but stronger or more specialized LLMs can mitigate these issues.Source: §7 · p. 10
• External-tool benefits may diminish when tools are inaccurate, inaccessible, quota-limited, or deprecated, and robustness to degraded or missing tools remains an open challenge.OctoTools’s modular design allows tools to be replaced or upgraded.Source: §7 · p. 10
• The efficient greedy toolset search is not globally optimal and may select a suboptimal subset when validation examples are scarce or biased.Optimization is static per task; query-level dynamic adaptation is proposed as future work.Source: §7 · p. 10
• Multi-step planning adds latency, cost, and memory use as step budgets and tool counts grow, with some queries exceeding 90 seconds.Large-scale or latency-sensitive deployments may require batching, pruning, or model distillation.Source: §7 · p. 10
• API calls and generated code can expose user data or execute unsafe code, while external tools may propagate bias; systematic auditing remains necessary.Sandboxed execution is possible, but risks persist, including with the Python_Code_Generator.Source: §7 · p. 10
• Third-party tool interfaces, quotas, and model versions can change, so exact reproduction requires snapshotting tool versions or containerized environments.The authors release code and tool cards but identify environment snapshotting as necessary future work.Source: §7 · p. 10
• OctoTools’s modularity lowers barriers to building agentic systems but also enables potential misuse, including large-scale disinformation generation.Governance and safeguards are critical but outside the scope of this work.Source: §7 · p. 10

Broader impacts

• OctoTools could accelerate scientific discovery and clinical report generation, while its planning and tool-use trajectories may support development of more capable, aligned agents.Potential applications combine literature search, database access, and computational tools; trajectories may also inform AI safety, interpretability, and robustness research.Source: Broader Impacts and Ethical… · p. 10

Ethical risks

• Tool-enabled complex reasoning could enable disinformation, deceptive profiles, or unauthorized surveillance, while privacy breaches, amplified bias, and harmful errors pose additional risks.The paper notes potential harms from malicious use, unintended consequences of otherwise correct tool use, and incorrect outputs in critical domains.Source: Broader Impacts and Ethical… · p. 10

Mitigations

• Proposed safeguards include sandboxing executable tools, human oversight, modular auditing and replacement, gated access to sensitive tools, and active misuse monitoring.The authors also encourage community best practices for responsible deployment.Source: Broader Impacts and Ethical… · p. 10

A Related Work

• OctoTools integrates diverse tools through standardized tool cards and a deterministic planner-executor paradigm, avoiding the training complexities of learned tool-use policies.The paper contrasts this design with fine-tuning and outcome-reward reinforcement learning, which can face long-horizon credit assignment, instability, and brittle generalization challenges.Source: A Related Work · p. 15
• Specialized single-agent systems target narrow domains, while multi-agent systems decompose complex tasks but may remain constrained by static designs.The passage cites chemistry and vision agents alongside AutoGen and MetaGPT as examples of these approaches.Source: A Related Work · p. 15
• OctoTools assigns high-level strategy and decomposition to a planner and concrete actions to an executor, creating a fixed hierarchical interaction.The authors state that this division simplifies coordination, enhances transparency, and supports reliability.Source: A Related Work · p. 15
• OctoTools combines iterative multi-step decomposition with specialized tools and context-aware function calling, unlike reasoning methods that rely mainly on an LLM’s latent capacity.Tool cards make domain-specific functions incorporable, while external modules can check intermediate steps.Source: A Related Work · p. 15

B.1 Benchmark Datasets

• For each benchmark, the study generally sampled 100 validation examples and 200 test examples, with exceptions noted separately.Open-ended questions received no additional preprocessing; multiple-choice options were lettered and appended to each question on a new line.Source: §B.1 · p. 15

B.1.1 General Domain Benchmarks

• General-domain evaluation uses visual puzzles (AlgoPuzzleVQA, PuzzleVQA), illusion-dependent questions (Hallusion-VD), and open-ended image questions (VQA 2.0).AlgoPuzzleVQA tests visual perception, language understanding, and algorithmic reasoning; PuzzleVQA tests reasoning over geometric-pattern attributes.Source: §B.1.1 · pp. 15–16

B.1.2 Mathematical Benchmarks

• Math benchmarks range from arithmetic-puzzle verification (Game of 24) and Olympiad-level text problems (Omni-MATH) to multimodal addition/subtraction (CLEVR-Math) and visual multi-step tasks (MathVista).MathVista includes both multiple-choice and open-ended questions.Source: §B.1.2 · p. 16

B.1.3 Scientific Benchmarks

• Scientific benchmarks span expert-authored graduate biology, physics, and chemistry Q&A (GPQA), reasoning-focused science questions (MMLU-Pro), and scientific-figure interpretation (SciFIBench).Source: §B.1.3 · p. 16

B.1.4 Medical Benchmarks

• Medical evaluation spans board-exam medicine (MedQA), H&E pathology diagnosis (PathCLS), open-ended pathology image QA (PathVQA), and open-ended radiology QA without labels or masks (SLAKE).SLAKE images include X-rays, CT scans, and MRI; object-detection labels and segmentation masks were excluded to increase difficulty.Source: §B.1.4 · p. 16

B.1.5 Agentic Benchmark

• GAIA-Text evaluates general AI assistants on difficult text-only questions requiring multi-step reasoning, web browsing, and tool-use proficiency.The experiments use the benchmark’s text-only subset.Source: §B.1.5 · p. 16

B.2 Tools Used in Our Experiments

• The experimental toolbox comprises 11 tools, including web and PubMed search, LLM fallback reasoning, pathology classification, and Python code generation and execution.Source: §B.2 · pp. 16–17
• Other experimental tools caption and zoom images, detect image text, extract webpage text, and search Wikipedia.Source: §B.2 · pp. 16–17

B.3 Additional Tools for Exploration Study

• Exploration tools add label-conditioned object detection, an upgrade from Grounding DINO to API-powered DINO-X, and retrieval of the latest Nature science news.Source: §B.3 · p. 17

C More Experimental Results

• Optimized tool sets vary by task: image captioning is used in all vision benchmarks, relevant-patch zooming in 6 of 10, Python generation in 3 of 4 math benchmarks, and Wikipedia search in all science benchmarks.Domain-specific tools also appear in corresponding uses, including Pubmed_Search in general medicine and Path_Generalist_Classifier in pathology classification.Source: C More Experimental Results · p. 17
• OctoTools adapts its tool chains and reasoning depth to each task, while most benchmarks improve as the maximum allowed steps increase.Figure 17 shows the distribution of steps taken, and Figure 18 plots accuracy changes from a maximum of one step.Source: C More Experimental Results · p. 17, Figure 17 · p. 21, Figure 18 · p. 22
• OctoTools outperforms agent baselines across all benchmarks; toolset optimization improves most tasks over using the full toolset, with similar trends on GPT-4o-mini.The supplied captions provide no exact accuracy values.Source: Figure 10 · p. 18, Figure 11 · p. 19, Figure 12 · p. 19

D.1 Query Analyzer

• The Query Analyzer identifies a query’s objectives, required skills, relevant tools, and tool limitations, then returns a structured analysis with additional considerations.It is instructed to account for accompanying inputs and explain each skill’s and tool’s contribution.Source: §D.1 · p. 23

D.2 Action Predictor

• The Action Predictor selects one suitable next tool and defines an achievable sub-goal using the query analysis, tool metadata, prior results, and remaining steps.Its output must include the exact tool name and all relevant data, files, and variables needed in context.Source: §D.2 · p. 24

D.3 Command Predictor

• The Command Predictor constructs valid Python tool calls from tool input requirements and context, including necessary preparation and required parameters.Commands must call tool.execute(), assign executions to the execution variable, and capture the final output there.Source: §D.3 · pp. 25, 27

D.4 Context Verifier

• The Context Verifier assesses whether tool memory sufficiently and accurately addresses the query, checking completeness, unused tools, inconsistencies, verification needs, and ambiguities.It decides whether to stop or use additional tools.Source: §D.4 · p. 28

D.5 Solution Summarizer

• The Solution Summarizer uses the query, image, actions, and tool results to produce a coherent final answer with a summary, step-by-step analysis, and key findings.The analysis explains each tool’s purpose, results, and contribution to answering the query.Source: §D.5 · p. 29
• The Solution Summarizer’s guidance calls for directly answering each query part, adding relevant insights or uncertainties, and concluding with a summary and possible next steps.Source: §D.5 · p. 29

E.1 ArXiv Paper Searcher Tool

• The ArXiv Paper Searcher searches for papers using a query and configurable result limits, and returns paper information as a list of dictionaries.The documented page-size options are 25, 50, 100, or 200; the maximum number of papers is separately configurable.Source: §E.1 · p. 30

E.2 Generalist Solution Generator Tool

• The Generalist Solution Generator answers prompts step by step and can accept images, but may produce hallucinated or incorrect responses.Examples include image captioning and answering a question about objects in an image.Source: §E.2 · p. 33

E.3 Google Search Tool

• The Google Search Tool performs Google searches from a text query and returns a list of dictionaries with search-result information; its example uses a 2024 Chemistry Nobel query.The example’s first result is titled “Nobel Prize in Chemistry Laureates” and is from NobelPrize.org.Source: §E.3 · pp. 35–36

E.4 Image Captioner Tool

• The Image Captioner generates image descriptions but may miss details or misread complex scenes, object relationships, counts, attributes, or culturally specific content.Its example describes four buckets containing five baseballs each; the paper advises using other tools for context or verification and checking critical information with specialists or human expertise.Source: §E.4 · pp. 37–38

E.5 Path Generalist Classifier Tool

• The Path Generalist Classifier answers multiple-choice questions on H&E-stained microscopy images, but is unsuitable for open-ended queries and other image types.The tool is not always reliable and should be cross-referenced; its example classifies a tissue image as “Viable tumor.”Source: §E.5 · pp. 39–40

E.6 Pubmed Search Tool

• PubMed Search retrieves article titles, abstracts, keywords, and URLs for text queries, and should be used only when more specific ontology tools are unavailable.The tool recommends shorter, more general queries.Source: §E.6 · p. 41

E.7 Python Code Generator Tool

• The Python Code Generator creates and executes simple arithmetic or math code in a restricted environment without imports, external libraries, file access, or complex data processing.Its example sums [1, 2, 3, 4, 5] and returns 15.Source: §E.7 · pp. 43–44

E.8 Relevant Patch Zoomer Tool

• The Relevant Patch Zoomer divides an image into four quarters and a center, selects question-relevant regions, and returns patches zoomed in by a factor of two.The guidance positions it for high-level analysis before other tools perform fine-grained analysis; in the car example, region D contains the full car view, so region E is unnecessary.Source: §E.8 · pp. 45–46
• The Relevant Patch Zoomer selects center region (E) as the clearest view for identifying the red car’s color.Its output names the selected region and provides a zoomed patch image.Source: §E.8 · p. 47

E.9 Text Detector Tool

• Text Detector uses EasyOCR to detect image text, with configurable languages and detail, and returns text with coordinates and confidence scores.Examples show English poster text and Chinese and English street-sign text.Source: §E.9 · pp. 48–50

E.10 URL Text Extractor Tool

• The URL Text Extractor accepts a URL and returns a dictionary containing extracted text and any error messages.Its description specifies extracting all text from the given URL.Source: §E.10 · p. 51
• For https://example.com, the extractor returned the page’s “Example Domain” text, including its illustrative-use notice.The structured result includes the URL and extracted text.Source: §E.10 · p. 51
• For an arXiv page, the extractor output included the paper title, authors, abstract, and reported Transformer results.The example page is Attention Is All You Need; its abstract text includes 28.4 BLEU for WMT 2014 English-to-German and 41.8 BLEU for English-to-French.Source: §E.10 · p. 52

E.11 Wikipedia Knowledge Searcher Tool

• The Wikipedia Knowledge Searcher takes a query and returns a dictionary containing search results, extracted text, and any error messages.Its description says it searches Wikipedia and returns web text based on the query.Source: §E.11 · p. 53
• The card’s example commands search Wikipedia for Python programming language, Artificial Intelligence, and Theory of Relativity.Source: §E.11 · p. 53
• For a kidney query, the searcher returned ten related results and extracted article text describing human kidneys and their anatomy.The listed results include Kidney, Kidney disease, and Kidney failure; the extracted text describes kidneys as blood-filtering organs and notes their location and connections to renal vessels and ureters.Source: §E.11 · p. 53

F.1 Object Detector Tool

• The Object Detector uses Grounding DINO to detect labeled objects, save padded object images, and return scores, bounding boxes, and image paths.Its inputs include an image, labels, confidence threshold (default 0.35), model size (tiny or base), and padding (default 20 pixels).Source: §F.1 · p. 54
• The Object Detector may miss objects, particularly small, uncommon, or specifically attributed ones, and the appendix recommends supplementary tools or image processing for verification.Detection performance can vary with the input image and labels.Source: §F.1 · p. 54

F.2 Advanced Object Detector Tool

• The Advanced Object Detector uses Grounding DINO-X to detect labeled objects, save padded crops, and return scores, bounding boxes, and image paths.It accepts an image, labels, confidence threshold (default 0.35), and padding (default 20 pixels).Source: §F.2 · p. 57
• The Advanced Object Detector has similar stated accuracy limitations, with small, uncommon, or specifically attributed objects posing particular challenges.The appendix suggests supplementary tools or image processing for verification.Source: §F.2 · p. 57
• The advanced detector’s baseball example returns detections with confidence scores of 0.73 and 0.66 and saves cropped images.The listed results include bounding boxes and saved-image paths.Source: §F.2 · p. 58

F.3 Nature News Fetcher Tool

• The Nature News Fetcher retrieves recent Nature articles as dictionaries, with defaults of 100 articles and up to five pages.The example output includes article titles, descriptions, authors, dates, URLs, and image URLs.Source: §F.3 · p. 59
• The Nature fetcher example returns two articles dated Jan 14, 2025, including stories about Bluesky use by scientists and testing AI intelligence.The structured results include article metadata and image URLs.Source: §F.3 · p. 60

• In a MathVista abstract-scene example, base GPT-4o fails to answer correctly; the four buckets contain 4, 5, 4, and 6 baseballs, totaling 19.Source: §G.1 · p. 61
• OctoTools first captions the image, then detects and counts baseballs, using the tools’ consistent outputs to produce its answer.The example presents the two-step process as overcoming GPT-4o’s visual-perception limitation.Source: §G.1 · pp. 61, 64, 65
• After captioning, the context verifier identifies that object detection is still needed to complete and verify the count.The action predictor then selects the Object_Detector_Tool because it is designed to detect and quantify objects.Source: §G.1 · pp. 62–63
• 20 baseballs are confirmed by both the image caption and object detector, yielding the correct answer.The captioner describes four buckets with five baseballs each, and the detector also counts 20.Source: §G.1 · pp. 64–65

G.2 Example for Multi-step Mathematical Reasoning

• G.2’s initial tool expression evaluated to -0.75 rather than 24, so verification prompted further exploration.The tool evaluated 6 / (1 - (9 / 1)); the verifier identified the mismatch and recommended trying other expressions.Source: §G.2 · pp. 67–68
• G.2’s tool search found valid expressions, including (1 + 1) × 9 + 6 = 24, using all four numbers.The Python-based search explored number permutations, arithmetic operations, and parenthesized forms.Source: §G.2 · p. 69

G.3 Example for Agentic Reasoning

• In G.3, GPT-4o’s prediction was incorrect: the cuneiform numeral’s ground truth is 536, computed as 8 × 60 + 56.The example requires interpreting Babylonian base-60 notation and recognizing the cuneiform symbols.Source: §G.3 · p. 70
• G.3’s initial Wikipedia search returned no results, after which OctoTools broadened its search and visited a returned URL for useful information.The supplied passage describes this search progression but does not yet give OctoTools’ final conversion answer.Source: §G.3 · p. 70
• After searches failed to find direct conversion information, the agent extracted a Babylonian numerals page that supplied the needed base-60 explanation.The example first tries Google and Wikipedia, then uses a URL text extractor on a converter page.Source: §G.3 · pp. 74–75

1. Identify the Symbols: -

• 536 results from interpreting the groups as 8 × 60 and 50 + 6, then adding 480 + 56.Babylonian numerals use a positional system based on 60.Source: §1 · p. 75

G.4 Example for Medical Reasoning

• After an ambiguous patch analysis and a captioner’s brain identification, OctoTools recognized the conflict and invoked a generalist tool to re-evaluate the image.The patch analysis did not clearly identify the organ; the generalist tool concluded the image showed a thoracic MRI rather than a brain scan.Source: §G.4 · pp. 76–77
• OctoTools ultimately identifies the image’s left-side organ as likely the patient’s right lung after correcting a captioner result that misidentified the thoracic MRI as a brain.The final answer attributes the initial error to the captioner and says the third step corrected the earlier tool usage.Source: §G.4 · p. 78

G.5 Example for Pathology Diagnosis

• The specialized Path_Generalist_Classifier_Tool classified the osteosarcoma H&E image as B) Necrotic tumor, unlike GPT-4o’s viable-tumor prediction.The example presents this as a case where OctoTools directly leverages a domain-specific classification tool to reach the answer.Source: §G.5 · pp. 79–80

G.6 Example for Scientific Reasoning

• For the appeal-to-indignation question, OctoTools retrieves knowledge and applies generalist reasoning, describing the fallacy as provoking anger or outrage instead of presenting logical evidence.The verifier says the reasoning tool’s output aligns with an understanding of the fallacy as involving emotional reactions rather than logical reasoning.Source: §G.6 · pp. 81–82
• OctoTools resolves an initially unanswered fallacy question with a generalist tool, identifying option C as the appeal to indignation.The example also states that OctoTools can use retrieved knowledge directly when a knowledge-retrieval tool is available.Source: §G.6 · p. 83

G.7 Example for Fine-grained Visual Reasoning

• Image captioning helps OctoTools overcome GPT-4o’s visual-perception error and determine that all fresh kiwis become rotten after 4 minutes.The captioner describes the grid’s initial state, which supports the subsequent simulation of the rotting process.Source: §G.7 · pp. 84, 86

G.8 Exploration Example for Literature Review

• The literature-review example finds LLM tool-agent research spanning multi-agent systems, healthcare, education, UAVs, and reasoning models.Its searches draw on ArXiv, PubMed, and Nature news, with examples including mental-health support, vaccination information, and reasoning-language-model development.Source: §G.8 · pp. 87–88
• The review notes that healthcare applications require attention to accuracy and equity, alongside broader concerns about computational overhead, bias, domain adaptation, and ethics.It concludes that future work should optimize models for specific applications and ensure ethical use.Source: §G.8 · p. 88