ART: Automatic multi-step reasoning and tool-use for large language models

• Existing chained-reasoning methods with tool use are difficult to extend because they require task- or tool-specific fine-tuning and prompt engineering.Source: §1 · p. 2
• ART retrieves demonstrations of related tasks from a task library to automatically decompose new instances and select appropriate tools.Its structured programs support parsing intermediate steps, pausing for external tools, and resuming generation with tool outputs.Source: §1 · p. 2
• ART evaluates cross-task transfer on unseen BigBench and MMLU tasks while also measuring performance on arithmetic and algorithmic reasoning.Source: §1 · p. 2
• ART matches or outperforms automatically generated CoT on 32 / 34 BigBench tasks and all MMLU tasks, by an average of over 22 percentage points.Source: §1 · p. 2
• 10.8% percentage points is ART’s average improvement over direct few-shot prompting across unseen BigBench and MMLU tasks.Source: §1 · p. 2
• Human feedback through task- and tool-library updates improves ART substantially, surpassing the best-known GPT3 results by over 20% points on 12 test tasks.Source: §1 · p. 2

• Scaled finetuning across broad task mixtures improves zero-shot and few-shot cross-task generalization, including on BigBench and MMLU.ART instead uses API access to InstructGPT and Codex to leverage their in-context learning abilities.Source: §2 · p. 2
• Chain-of-thought prompting encourages LLMs to generate intermediate reasoning steps before final answers through task-specific variants.Source: §2 · p. 2
• Zero-shot CoT and AutoCoT reduce reliance on handcrafted prompts, with AutoCoT generating CoT-style prompts automatically.Source: §2 · pp. 2–3
• ART extends prior reasoning prompts with a common language for cross-task demonstrations and flexible, extensible tool use.The framework is designed to improve the accuracy of intermediate reasoning steps.Source: §2 · p. 3
• Tool-use approaches commonly employ search, browsers, calculators, translation systems, or Python interpreters, but many require substantial supervision or task-specific prompts.Source: §2 · p. 3
• ART’s feedback loop allows extending tool use and improving performance by updating task and tool libraries.Source: §2 · p. 3

3 ART

• ART decomposes new-task instances into sequential reasoning steps and uses external tools when appropriate, despite lacking explicit decomposition or tool-use supervision.Source: §3 · p. 3

Prompt building

• ART retrieves related-task demonstrations and inserts them into a dynamic prompt for cross-task transfer.The demonstrations contain related sub-steps and tool calls.Source: §3.1 · p. 3

Generation

• ART parses generation online, pauses at tool calls, integrates tool outputs, and resumes generation to complete the program.In the Physics QA example, search finds a formula while code generation and execution compute the answer.Source: §3.1 · p. 3, Figure 2 · p. 4, TOOL LIBRARY · p. 4

Program grammar

• Programs encode task inputs, sub-step query-answer pairs, symbolic tool calls, and a final answer in a structured query language.The grammar supports sequential decompositions and external function calls.Source: §3.1 · p. 3, §3.2 · pp. 4–5

Human feedback

• Users can improve ART by adding corrected reasoning demonstrations or new tools to the libraries without additional finetuning.Examples include adding unit-handling steps and implementing dictionary lookup.Source: TOOL LIBRARY · p. 4, Figure 3 · p. 5, §3.4 · p. 6

Evaluation datasets

• ART is evaluated on 19 unseen BigBench tasks, a random subset of MMLU tasks, and tasks previously used to evaluate Toolformer.Source: §4 · p. 6

Details

• The main setup uses InstructGPT as the frozen LLM and Codex as the code-generation tool, with performance averaged over 5 runs.The prompt uses N = 3 seed tasks and 2 demonstration programs from each task.Source: §4 · p. 6

Baselines

• The baselines include direct few-shot prompting, automatically generated CoT without tools, tool-disabled ART, and the best published GPT-3/Codex result.Source: §4 · p. 6

5.1 Results on the task library

• 14.9 percentage points: ART improves over few-shot learning on average for task-library tasks using only two decomposition demonstrations.Performance is weaker on language games, code description, and auto-debugging, where code-generation errors can cascade through reasoning.Source: §5.1 · p. 7
• 8% points: ART outperforms AutoCoT on most task-library tasks even without tool use.The authors hypothesize that ART’s structured program format elicits multi-step reasoning better than free-form chain of thought.Source: §5.1 · p. 7
• 17.7 percentage points: ART with tools outperforms AutoCoT on all but one task-library task.Tools are called in approximately 95% of test instances, with especially large arithmetic gains averaging 21.85 points.Source: §5.1 · p. 7

5.2 Test tasks (cross-task transfer)

• 8.5 points: ART outperforms the few-shot baseline on average across 5 of 6 MMLU tasks without supervision for demonstrations or tool use.Most MMLU tasks benefit particularly from the search tool because they require extensive world knowledge.Source: §5.2 · p. 9

5.3 Improving ART

• 15 outputs: self-consistency consistently improves ART performance, while five-example human feedback produces significant task-specific gains and can surpass published GPT-3 results.Corrected programs and tools can be added to the libraries, extending improvements beyond the directly edited tasks.Source: Table 6 · p. 9, §5.3 · p. 10

• ART is a gradient-free approach for automatically generating multi-step reasoning and using tools with large black-box language models.Source: §6 · p. 10
• ART represents reasoning as programs with a lightweight grammar, tool calls and arguments, supported by seed-task and tool libraries.Source: §6 · p. 10
• The interpretable framework lets humans improve task decomposition and tool use, including through feedback and library updates.Source: §6 · p. 10
• ART improves over few-shot prompting and automatic CoT on unseen BigBench and MMLU tasks, while human feedback substantially exceeds hand-crafted CoT performance.Source: §6 · p. 10

Library Design

• The library analyzes BigBench tasks to identify recurring skills, while excluding long-text understanding, long-text generation and multilingual tasks.Source: §A.1 · p. 12
• The selected skill areas include arithmetic, code generation and editing, specialized search, string operations, recursive operations, classification, evidence extraction and reasoning.Source: §A.1 · p. 12
• 15 tasks are selected from five frequently used skill clusters to form the task library.Source: §A.1 · p. 12

Program Format

• A parsing expression grammar describes full reasoning programs containing an input, numbered substeps, tool calls, and a final answer.Source: §A.1 · p. 12

Cluster Programs

• Programs decompose task instances into sequential subtasks and can invoke symbols corresponding to tools in the tool library.Source: §A.1 · p. 16

Task Selection

• For a new task instance, ART retrieves N library tasks to construct a dynamic multi-task prompt.Source: §A.2 · p. 18

Task-Cluster based

• Cluster-based selection evaluates demonstrations from each task cluster on held-out examples and chooses the cluster with the highest performance.Source: §A.2 · p. 18

LLM-Similarity based

• LLM-similarity selection ranks library tasks by the relative log probabilities of “Similar” and “Not similar” and chooses the top-N tasks.Source: §A.2 · p. 18

Comparison

• LLM-based similarity performs worse on average than held-out cluster tuning and has high variance across runs.Source: §A.2 · p. 18

Code Generation

• Code generation uses the previous answer or input together with a generated prompt to produce or edit executable Python code.Source: §A.3 · pp. 18, 20

Generation

• ART parses programs during generation, pauses at tool calls, and resumes after incorporating tool outputs.Source: Figure 4 · p. 20

Code Execution

• Code execution runs generated Python in a virtual environment with arithmetic, symbolic and scientific packages, then integrates the execution result into the program.Source: §A.3 · pp. 18, 20

Knowledge Base lookup

• Knowledge-base lookup parses generated keys and prompts into a lookup function that returns matching entries, such as valid English words.Source: §A.3 · pp. 20–21

Extensibility

• Human feedback can add custom lookup tools, while the paper characterizes the built-in general knowledge-base lookup as restricted.Source: §A.3 · p. 21

Few-shot baseline

• Direct prompting uses three randomly chosen input-output instances without intermediate reasoning steps.Both InstructGPT and Codex are evaluated, with the higher performance reported.Source: §A.4 · p. 21

Auto CoT

• Automatic CoT elicits free-form natural-language, multi-step reasoning from randomly selected dataset examples.The generated reasoning is parsed to extract an answer string for evaluation.Source: §A.4 · p. 21

Best GPT-3 approaches

• The GPT-3 comparison includes human-supervised approaches using multi-step reasoning and tool use.These results are described as the best GPT-3 results reported for such approaches.Source: §A.4 · p. 21
• Suzgun et al.'s hand-crafted BigBench prompts were modified because they converted tasks into multiple-choice format, unlike the original format used here.The modification was intended to provide a fair comparison with ART, so the reported results may differ from the original study.Source: §A.4 · pp. 21–22
• Table 15 reports ART and free-form CoT improvements using self-consistency and human-in-the-loop feedback on library tasks.An asterisk marks improvements applied to automatically generated CoT reasoning.Source: Table 15 · p. 22
• Human feedback examples include editing code and adding a dictionary-lookup tool for English-word unscrambling.The described changes target the reasoning program and tool set for the task.Source: §A.4 · p. 22

Human feedback

• Human feedback is presented for five task-library settings.The listed settings include colored objects, physics, temporal sequences, tracking shuffled objects, and word unscrambling.Source: §A.5 · p. 22

Reasoning and representation

• For colored-object reasoning, code editing introduces a consistent data structure for representing the input.The intervention changes how the task input is structured for subsequent reasoning.Source: §A.5 · p. 22, Table 16 · p. 23
• For physics, human feedback adds substeps that identify given quantities, the required quantity, and a connecting physics formula.The prompt structure explicitly links the quantities through the relevant formula.Source: §A.5 · p. 22, Table 17 · p. 24
• For temporal sequences, added substeps examine each option to determine whether the subject is busy during the specified times.The intervention provides an option-by-option procedure for the temporal reasoning task.Source: §A.5 · p. 22, Table 18 · p. 25

Additional tasks

• Human-feedback tables also cover tracking shuffled objects and word unscrambling.The word-unscrambling setting is separately identified as Table 20, where code and dictionary lookup are used in the described intervention.Source: §A.5 · p. 22, Table 19 · p. 26, Table 20 · p. 26