Source-linked AI summary
CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models
Cheng Qian, Chi Han, Yi R. Fung, Yujia Qin, Zhiyuan Liu, Heng Ji
TL;DR
Tool-augmented LLMs are constrained by limited tool scope, fragile reasoning, and insufficient error handling. CREATOR has LLMs create tools and separates that abstract work from concrete decisions, with execution feedback used for rectification. It outperforms CoT, PoT, and tool-using baselines on the reported benchmarks and introduces a dataset for evaluating tool creation.
Problem
Tool-augmented LLMs face limited tool scope, fragile case-by-case reasoning, and insufficient automatic error handling.
Source: §1 · p. 1Method
CREATOR separates abstract tool creation from concrete decisions and uses execution feedback to rectify tools or decisions.
Source: §1 · p. 2Results
CREATOR surpasses CoT, PoT, and tool-using baselines, reporting 59.7% average accuracy on MATH and 94.7% on TabMWP.
Source: §1 · p. 2Takeaways & Limitations
The Creation Challenge dataset addresses the lack of benchmarks specifically evaluating tool creation, and experiments indicate knowledge transfer across similar queries.
Source: §1 · p. 2Evaluation is limited to MATH, TabMWP, and Creation Challenge, while the execution ability and correctness of tools for larger project pipelines remain insufficiently evaluated.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability and the instability of implicit reasoning, particularly when both planning and execution are involved. To overcome these limitations, we propose CREATOR, a novel framework that enables LLMs to create their own tools using documentation and code realization. CREATOR disentangles abstract tool creation and concrete decision execution, resulting in improved performance. We evaluate CREATOR on MATH and TabMWP benchmarks, respectively consisting of challenging math competition problems and diverse tabular contents. Remarkably, CREATOR outperforms existing chain-of-thought, program-of-thought, and tool-using baselines. Additionally, we introduce the Creation Challenge dataset, featuring 2K diverse questions, to emphasize the necessity and benefits of LLMs' tool creation ability. Further research demonstrates that leveraging LLMs as tool creators facilitates knowledge transfer, and LLMs exhibit varying levels of tool creation abilities, enabling them to adapt to diverse situations. The tool creation ability revolutionizes the LLM's problem-solving paradigm, driving us closer to the next frontier of artificial intelligence. All the codes and data are released.
1 Introduction
Tool-augmented LLMs face limited tool scope, fragile case-by-case reasoning, and insufficient error handling. CREATOR lets LLMs create tools and separates abstract creation from concrete decisions, outperforming established baselines on MATH and TabMWP.
1 Introduction
- Tool-augmented LLMs are described as limited in available tool scope, vulnerable to fragile reasoning, and lacking automatic, specific error handling.The paper contrasts human use of commonalities across similar questions with case-by-case reasoning by LLMs.Source: §1 · p. 1
- CREATOR enables LLMs to create and modify tools for a problem, rather than only selecting and planning API use.Its four stages are Creation, Decision, Execution, and Rectification; code serves as the medium for tool creation.Source: §1 · p. 2
- On MATH and TabMWP, ChatGPT built on CREATOR achieves average accuracies of 59.7% and 94.7%, respectively, surpassing standard CoT, PoT, and tool-using baselines by significant margins.Source: §1 · p. 2
- The Creation Challenge dataset targets novel problems inadequately solved by existing tools or code packages and supports experiments on knowledge transfer across similar queries.The paper also reports that LLMs show varying tool-creation abilities across problem settings.Source: §1 · p. 2
2 Related Work
Related work covers prompting and verification for LLM problem-solving, external tools and tool-use pipelines, and program-based reasoning. CREATOR differs by using code to create tools rather than relying solely on existing tools or using code only for reasoning.
Large Language Models
- Prompting methods and verification mechanisms guide LLM problem-solving and align model behavior with human expectations.The paper situates its work in these areas and uses them as baselines for complex problem-solving.Source: §2 · pp. 2–3
Tool Use and Tool Creation
- Tool-use research equips LLMs with external resources and pipelines for planning, tool calling, and result synthesis, while CREATOR enables tool creation.The related work names LATM and LLM customization as concurrent investigations of tool-creation ability.Source: §2 · p. 3
Reasoning and Execution with Program
- Program-based reasoning uses code for computational reasoning, but the cited approaches do not produce reusable or testable enhanced tools.CREATOR uses code as the medium for tool creation and adds tool creation, code rectification, and disentangled reasoning relative to PoT.Source: §2 · p. 3
3 Design of CREATOR
CREATOR uses four stages to make LLM-created tools reusable across queries and to separate abstract tool creation from query-specific decisions. Execution runs the resulting code, while rectification uses execution feedback to revise tools or decisions.
3 Design of CREATOR
- CREATOR’s four stages—creation, decision, execution, and rectification—distinguish it from prior tool-using frameworks and make created tools transferable to other queries.Unlike mainly linear CoT and PoT processes, CREATOR’s rectification makes reasoning non-linear.Source: §3 · p. 3, Figure 2 · p. 4
3.1 Creation Stage
- The creation stage uses abstract reasoning to build generally applicable tools from problem characteristics rather than numerical details.Prompts include demonstrations and tool documentation; demonstrations need not be semantically similar to test queries.Source: §3.1 · pp. 3–4
3.2 Decision Stage
- The decision stage uses concrete reasoning to select tool calls and relate their outputs to the specific query after creation is separated from decision-making.The paper says this separation facilitates eliciting different aspects of knowledge and improves task performance.Source: §3.2 · p. 4
3.3 Execution Stage
- Execution combines the created tool and the LLM’s decision in a code block and runs it with a code interpreter.Captured outputs and intercepted errors inform whether an answer is available or rectification is needed.Source: §3.3 · pp. 4–5
3.4 Rectification Stage
- Rectification uses execution error tracebacks to guide revisions to the tool or decision, while successful outputs are extracted and checked against the standard answer.The paper states that traceback guidance helps identify error locations and causes, and that rectification can improve performance.Source: §3.4 · p. 5
4 Experiments
CREATOR is evaluated on MATH, TabMWP, and the Creation Challenge dataset against several reasoning and tool-use baselines. It achieves the best reported results and is analyzed through tests of hints, reasoning separation, difficulty, and rectification.
4.1 Experimental Setup
- Experiments use ChatGPT on MATH, TabMWP, and a 2K-example Creation Challenge dataset, comparing CREATOR with vanilla LLM, CoT, PoT, Tool Use, and CREATOR-Entangled baselines.The baselines include PoT with rectification and Tool Use with WolframAlpha for calculations.Source: §4.1 · p. 5, §4 · p. 5
4.2 Creation Challenge
- Creation Challenge targets novel problems inadequately solved by existing tools or code packages, and its tool hints vary in detail, including possible inputs and outputs.The created-tool components in each data point can serve as hints, and the experiments vary hint utilization.Source: §4.2 · pp. 5–6, Table 4 · p. 7
4.3 Experimental Results
- CREATOR reaches 59.7% on MATH, 94.7% on TabMWP, and 75.5% on Creation Challenge, surpassing the best baseline performance on all three tasks.A case study also shows Tool Use failing where CREATOR derives a tool and solves the novel question.Source: §4.3 · p. 6, Table 2 · p. 6, Table 3 · p. 6
4.4 Results Analysis
- CoT lowers performance on MATH under PoT and CREATOR but improves TabMWP, which the authors attribute to conflicts on intricate math problems and alignment on simpler tabular reasoning.Without CoT, brute-force algorithms and straightforward calculations yield higher MATH accuracy.Source: §4.4 · p. 6
- CREATOR outperforms baselines on both tasks, particularly on difficult problems, and rectification raises accuracy by approximately 10% of the original value.Figure 4 compares accuracy by problem difficulty; Figure 5 presents results for different numbers of rectification rounds.Source: §4.4 · pp. 6–7, Figure 4 · p. 7, Figure 5 · p. 8
- Separating creation from decision generally improves performance over CREATOR-Entangled, while more detailed tool-creation hints can significantly improve performance.The authors say hints make desired tools easier to implement and reduce uncertainty and misdirections in CoT or tool documentation.Source: §4.3 · p. 6, §4.4 · p. 7
5 Further Analysis
Further analyses examine whether created tools support transfer across related questions and characterize different levels of tool creation. Transferred tools improve accuracy in the tested query sets, while observed creations range from repurposing APIs to hierarchical tools.
5.1 Facilitation of Knowledge Transfer
- The transfer test uses 300 questions in 100 three-query sets, each sharing a core knowledge concept but differing in scenario.The paper frames tool reusability as abstraction of knowledge that can support problems in different scenarios.Source: §5.1 · p. 8
- Transferred tools raise accuracy by 15.3%, with 39 query sets positively influenced among questions sharing core concepts across different scenarios.The experiment applies a correctly created tool from one scenario to the other two queries in its set.Source: §5.1 · p. 8
5.2 Different Levels of LLM’s Tool Creation
- Observed tool creations fall into three levels: enhancing an existing tool, concatenating multiple tools, and creating a hierarchy of caller-callee tools.The authors report that LLMs created tools at different levels without special guidance.Source: §5.2 · p. 8
6 Conclusion
The conclusion presents CREATOR as a framework that uses LLMs to create tools and separates abstract from concrete reasoning. Its evaluations on established benchmarks and Creation Challenge report advantages over CoT, PoT, and tool-using approaches.
6 Conclusion
- CREATOR uses LLMs as tool creators and disentangles abstract and concrete reasoning, with evaluations reporting advantages over CoT, PoT, and tool-using approaches.The authors anticipate further development of AI systems leveraging LLM tool-creation potential.Source: §6 · p. 9
Limitations
The experiments cover MATH, TabMWP, and Creation Challenge, leaving generalizability beyond those tasks unassessed. The scope of the tool-creation demonstration is also limited, with execution ability and correctness not properly evaluated.
Limitations
- The experiments cover only MATH, TabMWP, and Creation Challenge, so broader-task generalizability remains for future assessment.The authors call for expanding the framework to a broader array of tasks.Source: Limitations · p. 9
- The tool-creation demonstration leaves execution ability and correctness insufficiently evaluated, including for tools that build a full project pipeline.The authors say these capabilities remain questionable and call for further study of the boundaries of tool creation.Source: Limitations · p. 9
Ethics Statement
The ethics statement identifies privacy, fairness, and transparency as concerns across CREATOR’s stages. It calls for protecting sensitive information, mitigating bias, and documenting how created tools work and where they are limited.
Ethics Statement
- Demonstrations or instructions may contain sensitive information, so privacy safeguards should prevent disclosure to closed-source LLMs and protect private tool-creation knowledge.The concern applies when stages require demonstration examples and clear instructions.Source: Ethics Statement · p. 9
- Tool creation calls for mitigating biases in demonstrations and instructions and monitoring performance across stages for disparities.Source: Ethics Statement · p. 9
- Developers should document created tools’ biases and limitations and make their development and decision processes understandable.The statement also calls for understanding tools’ strengths and weaknesses before application.Source: Ethics Statement · p. 9
Appendices
The experiments use GPT-turbo-3.5 with a maximum generation length of 512 and temperature 0.3.
Appendices
- GPT-turbo-3.5 is used throughout, with generation capped at 512 tokens and temperature set to 0.3.The temperature is chosen to encourage deterministic generations while retaining some diversity, particularly during tool creation.Source: §A.1 · p. 12
A.2 Dataset Details.
The appendices describe evaluation coverage and dataset construction, then illustrate cases where natural-language and program reasoning suit different problems. The tool-transfer data groups three scenarios around shared core knowledge, while the evaluation datasets use numerical-answer questions for MATH and TabMWP.
A.2 Dataset Details.
- In the MATH and TabMWP evaluations, numerical-answer questions cover approximately 80% of each dataset; MATH scores are weighted across seven domains, and TabMWP spans grades one through eight.The authors exclude other answer formats because matching problems, such as matrix answers, may introduce bias.Source: §A.2 · p. 12
B.1 Creation Challenge Details
- Creation Challenge problems are seeded with novel settings and unconventional reasoning, then expanded iteratively with Text-Davinci-003 using random seed-data sampling.Each entry includes a problem, a standard created tool with its utility, input, output, and realization, a tool-calling decision, and a final answer.Source: §B.1 · p. 12, Figure 6 · p. 12
B.2 Tool Transfer Dataset Details
- The tool-transfer dataset is organized into sets of three queries, each with three corresponding answers, one standard tool applicable across all three scenarios, and three separate decisions about tool use.Seed data were manually written, including five query sets used as format examples; Text-Davinci-003 was used to iteratively create more data.Source: §B.2 · p. 12
C.1 CoT Incompatible with Code
- Two cases show natural-language and program thoughts can conflict: natural-language ambiguity leads to hard-coded digit-finding in MATH, while CoT avoids needless complexity in a simple TabMWP multiplication.The cases illustrate that reasoning paths may differ in suitability for code and natural language on challenging problems.Source: §C.1 · p. 13
C.2 Different Levels of Tool Creation
The cases distinguish three levels of tool creation: enhancing an existing tool, concatenating tools into a query-specific pipeline, and building a hierarchy. Prompting details describe demonstrations for CREATOR and its baselines.
C.2 Different Levels of Tool Creation
- The first case enhances an existing API: the LLM understands the tool and transfers that knowledge to a new problem scenario.The example learns to average temperatures across several days, then creates a tool to solve the problem.Source: §C.2 · p. 13
- The second case concatenates multiple tools into a logically planned, query-specific pipeline rather than merely calling tools for separate purposes.The tools are provided, and their individual uses are simple to follow.Source: §C.2 · p. 13
- The third case hierarchically organizes tools, with a main tool calling a sub-tool; the authors say this offloads reasoning and improves accuracy.The authors describe these case studies as only a glimpse of LLM tool-creation potential.Source: §C.2 · pp. 13–14
D Prompting Details
- CREATOR uses demonstrations for creation, decision, and rectification; execution does not require prompting, and the format also applies to TabMWP and Creation Challenge.A MATH query is shown as the demonstration example.Source: D Prompting Details · p. 15, Figure 14 · p. 17, Figure 15 · p. 17
- CoT, PoT, and Tool Use baselines also use demonstrations across tasks; Tool Use separates WolframAlpha inquiry from final-answer retrieval.The same baseline prompt formats apply to MATH, TabMWP, and Creation Challenge.Source: D Prompting Details · p. 15