Source-linked AI summary
ART: Automatic multi-step reasoning and tool-use for large language models
Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, Marco Tulio Ribeiro
TL;DR
ART addresses the need for more extensible multi-step reasoning and tool use without task-specific prompt engineering. It retrieves related-task demonstrations, represents reasoning as programs with tool calls, and integrates tool outputs during generation. ART improves over direct few-shot prompting and automatic CoT on unseen BigBench and MMLU tasks, while human feedback further improves selected-task performance.
Problem
Existing multi-step reasoning and tool-use methods often require task- or tool-specific fine-tuning, handcrafted demonstrations, or carefully constructed prompts.
Method
ART uses a structured program representation, retrieves related-task demonstrations, and pauses generation to call tools before resuming with their outputs.
Results
ART matches or outperforms automatic CoT on 32 / 34 BigBench tasks and all MMLU tasks, averaging over 22 percentage points of improvement, while human feedback surpasses best-known GPT3 results by over 20% points on 12 tasks.
Takeaways & Limitations
ART provides an extensible reasoning framework that humans can improve by correcting task programs or adding tools through library updates.
Takeaways & Limitations
String-manipulation tasks can suffer from code-generation errors, and the overview assumes only input-output pairs without decomposition or tool-use supervision.
Abstract
from arXiv · showhide
Large language models (LLMs) can perform complex reasoning in few- and zero-shot settings by generating intermediate chain of thought (CoT) reasoning steps. Further, each reasoning step can rely on external tools to support computation beyond the core LLM capabilities (e.g. search/running code). Prior work on CoT prompting and tool use typically requires hand-crafting task-specific demonstrations and carefully scripted interleaving of model generations with tool use. We introduce Automatic Reasoning and Tool-use (ART), a framework that uses frozen LLMs to automatically generate intermediate reasoning steps as a program. Given a new task to solve, ART selects demonstrations of multi-step reasoning and tool use from a task library. At test time, ART seamlessly pauses generation whenever external tools are called, and integrates their output before resuming generation. ART achieves a substantial improvement over few-shot prompting and automatic CoT on unseen tasks in the BigBench and MMLU benchmarks, and matches performance of hand-crafted CoT prompts on a majority of these tasks. ART is also extensible, and makes it easy for humans to improve performance by correcting errors in task-specific programs or incorporating new tools, which we demonstrate by drastically improving performance on select tasks with minimal human intervention.
1 Introduction
ART addresses the difficulty of extending multi-step reasoning with tool use by automatically selecting task demonstrations, generating decompositions, and integrating tool calls. On unseen benchmarks, it improves over few-shot prompting and automatic CoT, with further gains from human feedback.
- Existing chained-reasoning methods with tool use are difficult to extend because they require task- or tool-specific fine-tuning and prompt engineering.
- ART retrieves demonstrations of related tasks from a task library to automatically decompose new instances and select appropriate tools.Its structured programs support parsing intermediate steps, pausing for external tools, and resuming generation with tool outputs.
- ART evaluates cross-task transfer on unseen BigBench and MMLU tasks while also measuring performance on arithmetic and algorithmic reasoning.
- ART matches or outperforms automatically generated CoT on 32 / 34 BigBench tasks and all MMLU tasks, by an average of over 22 percentage points.
- 10.8% percentage points is ART’s average improvement over direct few-shot prompting across unseen BigBench and MMLU tasks.
- Human feedback through task- and tool-library updates improves ART substantially, surpassing the best-known GPT3 results by over 20% points on 12 test tasks.
2 Related Work
Related work develops in-context reasoning and external tool use, but often depends on handcrafted, task-specific supervision or prompts. ART contributes a shared representation for cross-task demonstrations and extensible tool use.
- Scaled finetuning across broad task mixtures improves zero-shot and few-shot cross-task generalization, including on BigBench and MMLU.ART instead uses API access to InstructGPT and Codex to leverage their in-context learning abilities.
- Chain-of-thought prompting encourages LLMs to generate intermediate reasoning steps before final answers through task-specific variants.
- Zero-shot CoT and AutoCoT reduce reliance on handcrafted prompts, with AutoCoT generating CoT-style prompts automatically.
- ART extends prior reasoning prompts with a common language for cross-task demonstrations and flexible, extensible tool use.The framework is designed to improve the accuracy of intermediate reasoning steps.
- Tool-use approaches commonly employ search, browsers, calculators, translation systems, or Python interpreters, but many require substantial supervision or task-specific prompts.
- ART’s feedback loop allows extending tool use and improving performance by updating task and tool libraries.
3 ART
ART uses a task library of structured reasoning programs to help a frozen LLM decompose new tasks and invoke tools during generation. Human edits to programs or tools can further improve task performance without finetuning.
- 3 ART: ART decomposes new-task instances into sequential reasoning steps and uses external tools when appropriate, despite lacking explicit decomposition or tool-use supervision.
- Prompt building: ART retrieves related-task demonstrations and inserts them into a dynamic prompt for cross-task transfer.The demonstrations contain related sub-steps and tool calls.
- Generation: ART parses generation online, pauses at tool calls, integrates tool outputs, and resumes generation to complete the program.In the Physics QA example, search finds a formula while code generation and execution compute the answer.
- Program grammar: Programs encode task inputs, sub-step query-answer pairs, symbolic tool calls, and a final answer in a structured query language.The grammar supports sequential decompositions and external function calls.
- Human feedback: Users can improve ART by adding corrected reasoning demonstrations or new tools to the libraries without additional finetuning.Examples include adding unit-handling steps and implementing dictionary lookup.
4 Experimental Setup
ART is evaluated on unseen BigBench and MMLU tasks, plus tasks used in prior tool-use work, using frozen-model baselines and tool-disabled comparisons.
- Evaluation datasets: ART is evaluated on 19 unseen BigBench tasks, a random subset of MMLU tasks, and tasks previously used to evaluate Toolformer.
- Details: The main setup uses InstructGPT as the frozen LLM and Codex as the code-generation tool, with performance averaged over 5 runs.The prompt uses N = 3 seed tasks and 2 demonstration programs from each task.
- Baselines: The baselines include direct few-shot prompting, automatically generated CoT without tools, tool-disabled ART, and the best published GPT-3/Codex result.
5 Results
ART improves multi-step reasoning across task-library and unseen benchmark tasks, with tool use, self-consistency, and limited human feedback providing additional gains.
- 5.1 Results on the task library: 14.9 percentage points: ART improves over few-shot learning on average for task-library tasks using only two decomposition demonstrations.Performance is weaker on language games, code description, and auto-debugging, where code-generation errors can cascade through reasoning.
- 5.1 Results on the task library: 8% points: ART outperforms AutoCoT on most task-library tasks even without tool use.The authors hypothesize that ART’s structured program format elicits multi-step reasoning better than free-form chain of thought.
- 5.1 Results on the task library: 17.7 percentage points: ART with tools outperforms AutoCoT on all but one task-library task.Tools are called in approximately 95% of test instances, with especially large arithmetic gains averaging 21.85 points.
- 5.2 Test tasks (cross-task transfer): 8.5 points: ART outperforms the few-shot baseline on average across 5 of 6 MMLU tasks without supervision for demonstrations or tool use.Most MMLU tasks benefit particularly from the search tool because they require extensive world knowledge.
- 5.3 Improving ART: 15 outputs: self-consistency consistently improves ART performance, while five-example human feedback produces significant task-specific gains and can surpass published GPT-3 results.Corrected programs and tools can be added to the libraries, extending improvements beyond the directly edited tasks.
6 Conclusion
ART combines automatic multi-step reasoning generation with automatic tool use for large black-box language models. Its interpretable, extensible programs support human improvement and strong performance on unseen tasks.
- ART is a gradient-free approach for automatically generating multi-step reasoning and using tools with large black-box language models.
- ART represents reasoning as programs with a lightweight grammar, tool calls and arguments, supported by seed-task and tool libraries.
- The interpretable framework lets humans improve task decomposition and tool use, including through feedback and library updates.
- ART improves over few-shot prompting and automatic CoT on unseen BigBench and MMLU tasks, while human feedback substantially exceeds hand-crafted CoT performance.
A.1 Task Library
The task library organizes diverse BigBench tasks around recurring reasoning skills and supplies decomposed programs with suitable tools. ART represents these programs in a structured grammar for parsing and execution.
- Library Design: The library analyzes BigBench tasks to identify recurring skills, while excluding long-text understanding, long-text generation and multilingual tasks.
- Library Design: The selected skill areas include arithmetic, code generation and editing, specialized search, string operations, recursive operations, classification, evidence extraction and reasoning.
- Library Design: 15 tasks are selected from five frequently used skill clusters to form the task library.
- Program Format: A parsing expression grammar describes full reasoning programs containing an input, numbered substeps, tool calls, and a final answer.
- Cluster Programs: Programs decompose task instances into sequential subtasks and can invoke symbols corresponding to tools in the tool library.
A.2 Task Selection
ART retrieves task-library demonstrations to construct dynamic multi-task prompts for new tasks. It compares cluster-based selection with LLM-similarity selection, with held-out tuning performing better on average.
- Task Selection: For a new task instance, ART retrieves N library tasks to construct a dynamic multi-task prompt.
- Task-Cluster based: Cluster-based selection evaluates demonstrations from each task cluster on held-out examples and chooses the cluster with the highest performance.
- LLM-Similarity based: LLM-similarity selection ranks library tasks by the relative log probabilities of “Similar” and “Not similar” and chooses the top-N tasks.
- Comparison: LLM-based similarity performs worse on average than held-out cluster tuning and has high variance across runs.
A.3 Tool Use
ART integrates code generation, code execution, lookup, and logic tools into its program-based reasoning pipeline. Tool calls are parsed during generation, paused for execution, and replaced with returned outputs; lookup remains restricted but can be extended with human feedback.
- Code Generation: Code generation uses the previous answer or input together with a generated prompt to produce or edit executable Python code.
- Generation: ART parses programs during generation, pauses at tool calls, and resumes after incorporating tool outputs.
- Code Execution: Code execution runs generated Python in a virtual environment with arithmetic, symbolic and scientific packages, then integrates the execution result into the program.
- Knowledge Base lookup: Knowledge-base lookup parses generated keys and prompts into a lookup function that returns matching entries, such as valid English words.
- Extensibility: Human feedback can add custom lookup tools, while the paper characterizes the built-in general knowledge-base lookup as restricted.
A.4 Baselines
The evaluation compares ART with direct prompting and free-form automatic CoT, alongside selected human-supervised GPT-3 approaches. The baselines differ in whether they provide intermediate reasoning and tool use, while some prior results required format adjustments for fair comparison.
- Few-shot baseline: Direct prompting uses three randomly chosen input-output instances without intermediate reasoning steps.Both InstructGPT and Codex are evaluated, with the higher performance reported.
- Auto CoT: Automatic CoT elicits free-form natural-language, multi-step reasoning from randomly selected dataset examples.The generated reasoning is parsed to extract an answer string for evaluation.
- Best GPT-3 approaches: The GPT-3 comparison includes human-supervised approaches using multi-step reasoning and tool use.These results are described as the best GPT-3 results reported for such approaches.
- Best GPT-3 approaches: Suzgun et al.'s hand-crafted BigBench prompts were modified because they converted tasks into multiple-choice format, unlike the original format used here.The modification was intended to provide a fair comparison with ART, so the reported results may differ from the original study.
- Best GPT-3 approaches: Table 15 reports ART and free-form CoT improvements using self-consistency and human-in-the-loop feedback on library tasks.An asterisk marks improvements applied to automatically generated CoT reasoning.
- Best GPT-3 approaches: Human feedback examples include editing code and adding a dictionary-lookup tool for English-word unscrambling.The described changes target the reasoning program and tool set for the task.
A.5 Human Feedback
The paper presents human feedback for five task-library settings. The feedback introduces task-specific reasoning structures or code changes, including representations for inputs, explicit substeps, and tool-assisted word unscrambling.
- Human feedback: Human feedback is presented for five task-library settings.The listed settings include colored objects, physics, temporal sequences, tracking shuffled objects, and word unscrambling.
- Reasoning and representation: For colored-object reasoning, code editing introduces a consistent data structure for representing the input.The intervention changes how the task input is structured for subsequent reasoning.
- Reasoning and representation: For physics, human feedback adds substeps that identify given quantities, the required quantity, and a connecting physics formula.The prompt structure explicitly links the quantities through the relevant formula.
- Reasoning and representation: For temporal sequences, added substeps examine each option to determine whether the subject is busy during the specified times.The intervention provides an option-by-option procedure for the temporal reasoning task.
- Additional tasks: Human-feedback tables also cover tracking shuffled objects and word unscrambling.The word-unscrambling setting is separately identified as Table 20, where code and dictionary lookup are used in the described intervention.