PaperlayerView exact revision on arXiv

Source-linked AI summary

CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets

Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R. Fung, Hao Peng, Heng Ji

arXiv:2309.17428v2cs.CLcs.AIcs.LG

TL;DR

General-purpose tool collections often inadequately represent APIs tailored to specialized tasks. CRAFT creates curated, reusable toolsets and retrieves relevant snippets at inference; it reports improved performance across challenging tasks, including vision-language gains over strong baselines.

  • Problem

    General-purpose tool collections often inadequately represent APIs tailored to specialized tasks and challenges.

    Source: §1 · p. 1

  • Method

    CRAFT generates task-specific code tools, abstracts, validates, and deduplicates them, then retrieves relevant tools using the target problem, function names, and docstrings.

    Source: §1 · p. 2

  • Results

    CRAFT improves performance on challenging tasks; in vision-language tasks, it reports an average 43.16% relative F1 improvement over the best baselines.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    CRAFT provides a general tool creation and retrieval framework, with analyses reporting contributions from its components and low-complexity, atomic, reliable tools.

    Source: §6 · p. 9

    CRAFT currently creates tools through code generation, restricting it to tasks solvable with code, and its effectiveness is greatly affected by backbone tool-using ability.

    Source: A LIMITATIONS AND FUTURE WORK · p. 15

Abstract

from arXiv · show

Large language models (LLMs) are often augmented with tools to solve complex tasks. By generating code snippets and executing them through task-specific Application Programming Interfaces (APIs), they can offload certain functions to dedicated external modules, such as image encoding and performing calculations. However, most existing approaches to augment LLMs with tools are constrained by general-purpose APIs and lack the flexibility for tailoring them to specific tasks. In this work, we present CRAFT, a general tool creation and retrieval framework for LLMs. It creates toolsets specifically curated for the tasks and equips LLMs with a component that retrieves tools from these sets to enhance their capability to solve complex tasks. For each task, we collect specific code solutions by prompting GPT-4 to solve the training examples. Following a validation step ensuring the correctness, these solutions are abstracted into code snippets to enhance reusability, and deduplicated for higher quality. At inference time, the language model retrieves snippets from the toolsets and then executes them or generates the output conditioning on the retrieved snippets. Our method is designed to be flexible and offers a plug-and-play approach to adapt off-the-shelf LLMs to unseen domains and modalities, without any finetuning. Experiments on vision-language, tabular processing, and mathematical reasoning tasks show that our approach achieves substantial improvements compared to strong baselines. In addition, our in-depth analysis reveals that: (1) consistent performance improvement can be achieved by scaling up the number of tools and the capability of the backbone models; (2) each component of our approach contributes to the performance gains; (3) the created tools are well-structured and reliable with low complexity and atomicity. The code is available at https://github.com/lifan-yuan/CRAFT.

1 INTRODUCTION

CRAFT addresses the mismatch between general-purpose tool collections and specialized tasks by creating reusable task-specific tools and retrieving relevant ones at inference. Experiments and analyses report improved performance, including gains on vision-language tasks, and describe the created tools as low-complexity and reliable.

1 INTRODUCTION

  • Specialized APIs tailored to specific challenges are often inadequately represented in general-purpose toolsets.The paper proposes customizable toolsets curated for specific problems of interest.Source: §1 · p. 1
  • CRAFT generates solutions for training problems, abstracts them into reusable code snippets, and validates and deduplicates tools for quality.Its toolsets are designed to contain diverse, reusable, and correct APIs.Source: §1 · p. 2
  • At inference, CRAFT matches the target problem, tool names, and docstrings to retrieve relevant tools for generated code solutions.The retrieved snippets are added to the LLM prompt.Source: §1 · p. 2
  • 43.16% average relative F1 improvement over the best baselines is reported for vision-language tasks.The evaluation also covers visual question answering, tabular processing, and mathematical reasoning.Source: §1 · p. 2
  • The analysis reports that created tools exhibit atomicity and low complexity, supporting their reliable structure.Performance also rises with more tools and more capable backbone models, and each CRAFT component contributes to gains.Source: §1 · pp. 2–3
  • CRAFT is presented as a broadly applicable framework, and its released toolsets are estimated to cost around $2,500 to construct.The tools are intended for downstream tasks.Source: §1 · p. 3

CRAFT

CRAFT creates task-specific toolsets to address tool creation and retrieval challenges, then retrieves relevant tools for LLMs to use at inference. Its pipeline aims to make tools diverse, reusable, correct, and less redundant.

CRAFT

  • CRAFT addresses tool creation by building extensive, diverse, reusable, and correct toolsets, and tool retrieval by finding relevant tools tailored to each question.Retrieved code is instantiated and added to the prompt so LLMs can call functions for complex operations.Source: CRAFT · p. 3

2.1 TOOL CREATION

  • The creation pipeline iteratively samples less-similar problems, then prompts GPT-4 to generate executable Python solutions and keeps correct ones.The sampling strategy adds problems least similar to those already sampled to promote tool diversity.Source: §2.1 · pp. 3–4
  • Abstraction generalizes code names and inputs for reuse, validation discards tools that fail on original problems, and deduplication selects comprehensive tools within groups.Deduplication groups tools by function name and number of input arguments.Source: §2.1 · p. 4

2.2 TOOL RETRIEVAL

  • At inference, CRAFT retrieves tools by matching the target problem, generated function names, and docstrings, then ranks tools by frequency across the three lists.It selects the three most frequent tools and filters out tools appearing only once; if none remain, the LLM generates code without task-specific tools.Source: §2.2 · p. 4
  • Retrieved snippets are added to LLM prompts, invoked in generated code, and executed to produce predictions.CRAFT is training-free and plug-and-play, and its atomic code snippets are explainable and can be examined and fixed by humans.Source: §2.2 · pp. 4–5

3 EXPERIMENT

The experiments evaluate CRAFT across vision-language, tabular-processing, and mathematical-reasoning tasks against several baseline families. CRAFT outperforms baselines on all datasets, with especially notable gains on VQA.

3.1 EXPERIMENTAL SETTING

  • Evaluation covers VQA on GQA, OK-VQA, and A-OKVQA using SAcc and F1, TabMWP using exact-match accuracy, and MATH algebra using accuracy.The paper notes that descriptive LLM responses can lead exact-match evaluation to underestimate VQA performance, motivating the additional F1 metric.Source: §3.1 · p. 5
  • Baselines include tool-free chain-of-thought, existing tools, other LLM-created tools, and alternative retrieval methods; experiments use GPT-3.5-Turbo.The comparisons include LATM and CREATOR for tool creation, and SimCSE and BM25 for retrieval.Source: §3.1 · pp. 5–6

3.2 EXPERIMENTAL RESULTS

  • External Python libraries did not improve performance and sometimes hurt mathematical reasoning; LATM had negative effects, while CREATOR notably helped math but had marginal effects elsewhere.Retrieval baselines generally did not substantially enhance performance except on TabMWP.Source: §3.2 · p. 6
  • Further analysis examines abstraction and retrieval ablations and compares CRAFT with ViperGPT across backbone models.The supplied results passage identifies this analysis's scope but does not provide its outcomes.Source: Table 3 · p. 7
  • CRAFT outperforms baselines on all datasets; against ViperGPT, its absolute SAcc improvements are 10.4 on GQA, 18.0 on OK-VQA, and 15.2 on A-OKVQA.The authors describe the gains as especially notable on challenging VQA tasks.Source: Table 2 · p. 6, §3.2 · p. 7
  • Using the same created toolset, CRAFT's retrieval approach generally performs better than alternatives, with BM25 on TabMWP as an exception.The authors attribute the TabMWP exception to relatively straightforward patterns that do not sufficiently showcase CRAFT's retrieval advantages.Source: §3.2 · p. 7

4 FURTHER ANALYSIS.

This analysis uses VQA to examine external tool augmentation, then tests CRAFT’s abstraction and retrieval components, backbone capability, toolset scaling, and toolset structure. The results show gains across these analyses, with the toolsets’ complexity and diversity varying by task.

4.1 DOES ABSTRACTION FACILITATE TOOL USE?

  • Ablating abstraction reduces performance, although abstraction-ablated CRAFT still improves over ViperGPT on all three datasets, especially OK-VQA and A-OKVQA.The authors suggest the large, diverse toolset and correct retrieved problem-solving strategies may help LLMs adapt even without abstraction.Source: §4.1 · p. 7

4.2 IS EVERY MATCHING IN THE RETRIEVAL TRIPLET EQUALLY IMPORTANT?

  • Removing any of the three multi-view similarity measures lowers performance, and ablating function-name matching causes a greater than 6.6-point absolute SAcc drop.The three views match against problems, function names, and docstrings.Source: §4.2 · pp. 7–8

4.3 DOES CRAFT STILL WORK FOR MORE POWERFUL BACKBONE MODELS?

  • With GPT-4, CRAFT performs consistently better than vanilla ViperGPT, though the improvement is less pronounced than with GPT-3.5-Turbo.The comparison is limited to vanilla ViperGPT because of budget constraints; the authors hypothesize GPT-4-created tools may offer GPT-4 fewer insights.Source: §4.3 · p. 8

4.4 CAN CRAFT IMPROVE PERFORMANCE AS THE TOOLSET GETS LARGER?

  • CRAFT’s soft accuracy consistently rises across three datasets as the toolset grows, with the largest improvement when moving from no created tools to 261 tools.The analysis compares toolsets of 0, 261, 337, and 525 tools.Source: §4.4 · p. 8, Figure 3 · p. 8

4.5 WHAT IS INSIDE THE TOOLSET?

  • Created toolsets have relatively low complexity; MATH has the most distinct tool classes (234 of 282 tools), while TabMWP is more homogeneous (23 of 181).VQA has 195 distinct classes among 525 tools.Source: §4.5 · pp. 8–9, Table 4 · p. 8

5 RELATED WORK

Related work connects LLMs to external tools and explores both tool learning and tool creation and retrieval. CRAFT extends this line of work by creating customized tools and retrieving relevant ones from a large toolbase.

5.1 TOOL LEARNING WITH LLMS

  • Prior work connects LLMs to external systems and expands tool repertoires to calculators, calendars, interpreters, simulators, and maps (Paranjape et al., 2023; Liu et al., 2023c; Drori et al., 2022; Pan et al., 2023).Other approaches distill tool-use examples from ChatGPT into open-source LLMs (Qin et al., 2023b; Tang et al., 2023).Source: §5.1 · p. 9

5.2 TOOL CREATION & RETRIEVAL

  • Prior creation work uses few examples (Cai et al., 2023) or query-specific generation that limits reuse, while retrieval has used curated tools, heuristics, or query-name similarity (Qin et al., 2023b; Patil et al., 2023).CRAFT targets a large toolbase for related downstream tasks and retrieval of relevant tools from that base.Source: §5.2 · p. 9

6 CONCLUSION

CRAFT presents a general framework for tool creation and retrieval to generalize LLMs across domains and tasks, and reports improved performance alongside analyses of component contributions, toolsets, and scalability.

6 CONCLUSION

  • CRAFT combines tool creation and retrieval to generalize LLMs across diverse domains and tasks, with improved performance on challenging tasks.The paper also reports analyses of component contributions, constructed toolsets, and scalability.Source: §6 · p. 9

A LIMITATIONS AND FUTURE WORK

The paper identifies constraints on CRAFT’s task scope and its dependence on the backbone model’s tool-use ability.

A LIMITATIONS AND FUTURE WORK

  • Because current CRAFT creates tools through code generation, it suits only tasks solvable with code; effectiveness also depends greatly on the backbone’s tool-use ability.The authors report that some open-source models achieve near-random performance in challenging tool manipulation and propose exploring pseudocode and ways to elicit tool-use ability.Source: A LIMITATIONS AND FUTURE WORK · p. 15

B CRAFT IMPLEMENTATION DETAILS

CRAFT’s implementation builds task-specific toolsets from sampled problems and sets retrieval counts separately for each task.

Tool Creation

  • VQA tools use 2,000 sampled problems from LLaVA and COCO-based instruction data, filtered to retain questions requiring visual information.ViperGPT generates the code solutions.Source: B CRAFT IMPLEMENTATION DETAILS · p. 15
  • Tabular and mathematics toolsets use 500 sampled training problems each from TabMWP and MATH algebra, respectively.For each task, 200 come from the first sampling epoch and 300 from the next three, at 100 per epoch.Source: B CRAFT IMPLEMENTATION DETAILS · p. 15

Tool Retrieval

C PROMPT

Appendix C presents prompts for evaluating VQA answers and turning task-specific solutions and queries into reusable tools, with instructions for generalizing their reasoning, inputs, functions, and docstrings while preserving their application. Examples cover VQA, tabular processing, and mathematical reasoning.

C.1 PROMPT TO FILTER IMAGE-DEPENDENT QUESTIONS IN VQA DATASET

  • The VQA filter asks whether GPT-3 can directly answer a query without visual information: image-dependent questions receive “no,” while image-independent inference or search questions receive “yes.”The prompt frames this as a yes-or-no decision about whether the function can solve the query directly.Source: §C.1 · p. 15

C.2 EVALUATE THE SPECIFIC GENERATION FOR VQA

  • The VQA evaluator asks whether a prediction has the same meaning as the reference in the question’s context, judging understanding rather than superficial word overlap.It returns a direct yes-or-no answer.Source: §C.2 · p. 16

C.3.1 VQA

  • For VQA, code is abstracted into a higher-level tool for similar queries, using common reasoning rather than specific object types and avoiding unsupported ImagePatch assumptions.The instructions also call for honest function names, purposeful code, generalized variables, a docstring, and a usage example.Source: §C.3.1 · p. 16
  • The VQA example turns a backpack-relative-to-man query into a generic horizontal-existence checker with object, reference-object, direction, and query arguments.The helper returns “yes” or “no” and falls back to a direct image query if the reference object is not found.Source: §C.3.1 · p. 17

C.3.2 TABULAR PROCESSING

  • Tabular tools should capture a shared reasoning pattern and data organization, generalize specific names and values, accept column names as arguments, and be called on the original query.The instructions require a docstring and an example call.Source: §C.3.2 · p. 18
  • For stem-and-leaf tables, the example generalizes a candy-counting solution into a tool parameterized by data frame, stem and leaf columns, and a threshold.The tool combines stem and leaf values, filters items meeting the threshold, and counts them.Source: §C.3.2 · pp. 19–20
  • For tabular processing, the prompt abstracts item-by-item price and quantity calculations into a reusable function with table, column-name, and quantity inputs.The function returns total item cost; its specific item names and columns are replaced with generic variables.Source: §C.3.2 · p. 21

C.3.3 MATHEMATICS REASONING

  • Generic tool functions should solve a query type through shared reasoning, use generalized names and variables, document data organization, and be called on the original query.The mathematics prompt also specifies a final response format containing the generic function and an example call.Source: §C.3.3 · p. 22
  • For continuity problems, the example generalizes solving an unknown constant by equating two piecewise-function values at a specified point.The generic tool accepts the variable, constant, two function pieces, and continuity point, then solves the equality.Source: §C.3.3 · p. 23
  • For inverse-function problems, the example generalizes iterating a common multiplier rule from a base input-output pair to a target output.The abstracted tool replaces specific values and multipliers with generic parameters.Source: §C.3.3 · pp. 24–25

C.4 TOOL DEDUPLICATION

  • Tool deduplication asks the model to choose the most general, reusable tool based on specificity and functional versatility rather than tool-name frequency.The prompt frames the choice around typical VQA use cases and tool capabilities.Source: §C.4 · p. 25

C.5.1 VQA

  • The VQA prompt converts each query into a declarative command and concise imperative instruction, then infers useful tool functions and their docstrings.Instructions should prioritize the action over query-specific nouns, and docstrings should remain general.Source: §C.5.1 · p. 26
  • VQA function names should follow a short verb_noun format, with abstract wording illustrated by changing a desk-specific query into identifying objects on a specified surface.The example ends by listing useful functions and returning the generalized instruction.Source: §C.5.1 · p. 26
  • The VQA example abstracts a query about people associated with an object into a reusable search function and docstring, then returns the answer about a specific object.The proposed function is named 'search_people_associated_with_object'.Source: §C.5.1 · p. 27

C.5.2 TABULAR PROCESSING

  • Tabular prompts ask for the task goal and table structure, then generic tool functions and docstrings that are abstract, concise, and informed by frequently used terms.Function names should be under eight words, and responses should list useful functions and give a final answer.Source: §C.5.2 · p. 27
  • For a stem-and-leaf query, the example tool combines stems and leaves into values, filters at a threshold, and counts qualifying groups.The suggested function names include 'count_groups_above_threshold_in_stem_leaf'.Source: §C.5.2 · p. 28
  • For item-price queries, the example tool calculates total cost by multiplying each unit price by its quantity and summing across items.Its docstring describes a price-table DataFrame and a quantity dictionary as inputs.Source: §C.5.2 · p. 28

C.5.3 MATHEMATICS REASONING

  • Mathematics prompts first infer the core skill, then generic mathematical functions and task-goal docstrings, with short names and a required useful-functions/final-answer format.The prompt also supplies frequently used terms for function names and docstrings.Source: §C.5.3 · pp. 28–29
  • For continuity problems, the example tool equates the two piecewise expressions at the boundary and solves for the unknown constant.Suggested names include 'solve_continuous_piecewise_function' and 'find_constant_for_continuity'.Source: §C.5.3 · p. 29
  • For an inverse-function query, the example tool uses the function's stated properties to establish a relationship and evaluate the inverse at the requested input.The proposed docstring describes an inverse value based on a linear combination rule for the original function.Source: §C.5.3 · p. 29
Loading 2309.17428v2…