PaperlayerView exact revision on arXiv

Source-linked AI summary

Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models

Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Jianfeng Gao

arXiv:2304.09842v3cs.CLcs.AIcs.CVcs.LG

TL;DR

LLMs struggle with current information, external tools, and precise mathematical or logical reasoning. Chameleon addresses these gaps by using an LLM planner to compose diverse plug-and-play tools into executable programs, achieving strong results on ScienceQA and TabMWP.

  • Problem

    LLMs have inherent limitations in accessing up-to-date information, using external tools, and performing precise mathematical and logical reasoning.

  • Method

    Chameleon uses an LLM-based planner to synthesize natural-language-like programs that compose LLMs, vision models, web search, Python functions, and heuristic modules.

  • Results

    86.54% overall accuracy on ScienceQA and 98.78% accuracy on TabMWP establish strong performance across both evaluated tasks.

  • Takeaways & Limitations

    Chameleon demonstrates adaptable and effective compositional reasoning across multimodal, knowledge-intensive ScienceQA and tabular mathematical TabMWP.

  • Takeaways & Limitations

    Failures can arise from module limitations, suboptimal planner-generated programs, or missing tools for specific abilities.

Abstract

from arXiv · show

Large language models (LLMs) have achieved remarkable progress in solving various natural language processing tasks due to emergent reasoning abilities. However, LLMs have inherent limitations as they are incapable of accessing up-to-date information (stored on the Web or in task-specific knowledge bases), using external tools, and performing precise mathematical and logical reasoning. In this paper, we present Chameleon, an AI system that mitigates these limitations by augmenting LLMs with plug-and-play modules for compositional reasoning. Chameleon synthesizes programs by composing various tools (e.g., LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristic-based modules) for accomplishing complex reasoning tasks. At the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response. We showcase the effectiveness of Chameleon on two multi-modal knowledge-intensive reasoning tasks: ScienceQA and TabMWP. Chameleon, powered by GPT-4, achieves an 86.54% overall accuracy on ScienceQA, improving the best published few-shot result by 11.37%. On TabMWP, GPT-4-powered Chameleon improves the accuracy by 17.0%, lifting the state of the art to 98.78%. Our analysis also shows that the GPT-4-powered planner exhibits more consistent and rational tool selection via inferring potential constraints from instructions, compared to a ChatGPT-powered planner. The project is available at https://chameleon-llm.github.io.

1 Introduction

Chameleon addresses LLM limitations by composing diverse external tools through an LLM-generated natural-language program. It demonstrates strong performance on ScienceQA and TabMWP, while GPT-4 improves tool-selection consistency.

  • LLMs show emergent reasoning abilities but cannot reliably access current information, external tools, or precise mathematical reasoning.
  • Chameleon synthesizes natural-language-like programs that compose LLMs, vision models, web search, Python functions, and heuristic modules.Its planner uses tool descriptions and examples to select an executable sequence without additional training or carefully curated rules.
  • 86.54% accuracy on ScienceQA improves the best published few-shot result by 11.37%.
  • 98.78% state-of-the-art accuracy on TabMWP represents a 17.0% increase over the best-published model.
  • GPT-4 as planner produces more consistent and rational tool selection than ChatGPT by inferring potential constraints from instructions.

2 Related Work

Prior work augments LLMs with compositional or external tools, but often limits tool breadth, requires domain-specific designs, or depends on tailored supervision. Chameleon emphasizes flexible composition and extension across tasks.

  • Compositional Reasoning: Neural modular approaches decompose complex reasoning into subtasks but can depend on brittle parsers and fixed module configurations.
  • Tool-Augmented Language Models: LLMs remain limited in accessing current information, using external tools, and performing precise mathematical reasoning.
  • Tool-Augmented Language Models: Visual tool-augmentation methods incorporate models such as Hugging Face, Azure, and visual foundation models for visual tasks.
  • Tool-Augmented Language Models: Many tool-augmented approaches use small or task-specific tool sets, limiting capabilities across skills and generalization to new tasks.
  • Tool-Augmented Language Models: Chameleon uses natural-language module descriptions and calling examples, avoiding additional training or tool-specific prompts while supporting new tools and tasks.

3 General Framework: Chameleon

Chameleon uses an LLM planner to generate a sequential natural-language program from a query, module inventory, constraints, and demonstrations. Modules execute in order while updating inputs and cached intermediate information.

  • Chameleon combines a module inventory of tools with an LLM-based planner that decomposes problems into sub-tasks.
  • The planner selects a sequence of modules using the input query, inventory M, constraints G, instructions, and demonstrations D.
  • Each generated natural-language program contains T modules, with every M_t drawn from the module inventory M.
  • Modules execute sequentially, passing the current input to each step and binding modules through string matching.
  • Execution updates the next input and cache, preserving information such as image semantics, retrieved knowledge, and generated programs.

4 Applications of Chameleon

Chameleon applies a rich inventory of external tools to ScienceQA and TabMWP, with task-specific modules and constrained program search for multimodal and tabular reasoning.

  • 4.1 Module Inventory: Chameleon uses a rich module inventory of external tools to support diverse reasoning capabilities across queries.The inventory is presented in Table 2 and includes multiple tool types.
  • 4.1 Module Inventory: Knowledge Retrieval supplies specialized background context, while Bing Search targets broader or up-to-date information from multiple sources.Query Generator typically precedes Bing Search to create more targeted queries, improving retrieved information's recall and precision.
  • 4.1 Module Inventory: Image Captioner and Text Detector convert visual content into language or extract image text for semantic understanding of diagrams, charts, tables, maps, and scenes.These modules support image-based reasoning by supplying supplementary context or detected textual information.
  • 4.1 Module Inventory: Program Generator, Verifier, Executor, Solution Generator, and Answer Generator respectively support computation, validation, execution, solution construction, and concise task-specific output.The planner may directly use Solution Generator for simpler queries, while Answer Generator normalizes final results.
  • 4.2 Science Question Answering: ScienceQA requires multimodal tools and skills, while Chameleon constrains its program search to a relevant inventory subset and defaults invalid programs to a Solution Generator–Answer Generator sequence.The benchmark involves image captioning, text detection, knowledge retrieval, online search, and multi-clue visual reasoning.
  • 4.3 TabMWP: TabMWP tests understanding of diverse tabular contexts and precise numerical or symbolic computation, using tools for table digestion and mathematical reasoning.Its contexts include schedules, prices, tax forms, plots, and function relations.

5 Experiments

Chameleon is evaluated on ScienceQA and TabMWP, combining diverse tools through generated programs. It improves accuracy, supports adaptive tool selection, and reduces several error categories, while failures remain tied to module and planning limitations.

  • Experimental Results: 86.54% accuracy on ScienceQA with GPT-4 surpasses GPT-4 CoT by 2.55% and GPT-3 CoT by 11.37%.With ChatGPT, Chameleon reaches 79.93% accuracy, improving over ChatGPT CoT by 1.62%.
  • Experimental Results: 98.78% accuracy on TabMWP with GPT-4 surpasses Codex PoT-SC by 17.0% and human performance by 8.56%.Chameleon with ChatGPT outperforms ChatGPT CoT and ChatGPT PoT by 11.25% and 3.79%, respectively.
  • Qualitative Analysis: GPT-4 selects tools more consistently than ChatGPT, including simultaneous Query Generator and Bing Search use on ScienceQA.ChatGPT calls Knowledge Retrieval in 72% and Bing Search in 3% of ScienceQA queries; GPT-4 uses them in 81% and 11%, respectively.
  • Ablation Study: Knowledge Retrieval is vital in both tasks; ScienceQA also depends on search and vision modules, while TabMWP depends on program tools.The ablation disables key modules using ChatGPT and 500 test examples.
  • Qualitative Analysis: GPT-4 generates distinct tool-sequencing modes: Knowledge Retriever or Bing Search on ScienceQA, and solution-generation or program-execution routes on TabMWP.The transition probabilities are computed from tool transitions observed on the test sets.
  • Case Study: ScienceQA examples show sequential combinations of image captioning, knowledge retrieval, text detection, and Bing Search for different query types.The planner adapts generated programs to image semantics, advertising text, and domain-specific animal-survival knowledge.
  • Case Study: TabMWP examples combine knowledge retrieval, table description, row lookup, language solutions, and program-aided tools for mathematical table reasoning.One example uses row lookup and an LLM-generated solution instead of program-based tools.
  • Error Analysis: Chameleon reduces image-understanding mistakes from 32 to 10 with ChatGPT and 19 with GPT-4, while knowledge-understanding mistakes fall from 37 to 6 and 3.The analysis uses 50 ChatGPT baseline mistakes from ScienceQA and also reports reductions in solution-generation errors.

6 Conclusion

The paper concludes that Chameleon augments LLMs with diverse external tools through plug-and-play composition. Results on ScienceQA and TabMWP indicate adaptability and higher accuracy across challenging reasoning tasks.

  • 6 Conclusion: Chameleon augments LLMs with diverse external tools in a plug-and-play compositional reasoning framework.The framework demonstrates adaptability and effectiveness on ScienceQA and TabMWP.

Supplementary Materials for Chameleon: Plug-and-Play Compositional Reasoning

The supplementary-material passage consists only of the paper title fragment.

  • Supplementary Materials: The supplementary-material passage contains the title fragment “with Large Language Models.”

A Appendix

The appendix passage frames external tools and modular approaches as responses to limitations in LLM access, tool use, and mathematical reasoning.

  • Compositional Reasoning: External tools and plug-and-play modular approaches are studied to address LLM limitations in accessing information, using tools, and performing precise reasoning.Web search can provide real-time and domain-specific knowledge, while other work targets mathematical reasoning with LLMs.

A.2 Experimental Details

The experiments define task-specific tool inventories and use deterministic LLM-based planners to generate compositional programs. Modules update inputs and cache newly generated elements so later tools can use them.

  • Module inventory: ScienceQA and TabMWP use task-specific subsets of the available module inventory.The inventory subsets are listed in Table 6.
  • Planner: The planner uses ChatGPT or GPT-4, generates programs up to 128 tokens, and runs at temperature 0 for deterministic generation.Separate prompts are provided for ScienceQA and TabMWP.
  • ScienceQA modules: ScienceQA LLM modules generally use four in-context demonstrations, temperature 0, and up to 512 completion tokens.Knowledge Retrieval, Query Generator, and Solution Generator have module-specific prompts and token limits.
  • ScienceQA modules: ScienceQA includes image captioning, text detection, Bing search, and LLM-based retrieval, generation, and answering modules.The image captioner uses beam search, while Bing Search returns the top three responses.
  • TabMWP modules: TabMWP LLM modules generally use four demonstrations, temperature 0, and up to 512 completion tokens, with lookup modules enabled under table-size conditions.Row Lookup requires more than three rows and 18 cells; Column Lookup requires at least two columns and 18 cells.
  • Execution state: update_input changes the execution state for tools such as Row Lookup, while update_cache stores generated elements for later tool execution.Examples of generating tools include Image Captioner, Text Detector, Knowledge Retrieval, Web Search, and Program Generation.

A.3 Experimental Results

GPT-4 produces fewer distinct Chameleon programs than ChatGPT on both ScienceQA and TabMWP, indicating more consistent tool-sequence generation under the same prompt.

  • Generated program statistics: GPT-4 generates fewer distinct programs than ChatGPT on both ScienceQA and TabMWP.The comparison uses the exact same planning prompt for both base models.
  • Generated program statistics: The paper relates GPT-4’s program consistency to stronger long-context understanding, instruction alignment, and high-level reasoning than ChatGPT.This interpretation is presented as consistent with prior findings cited by the authors.

B Limitations

The paper identifies scope, planning, and scalability boundaries for Chameleon. Its evaluations cover limited benchmarks, while one-shot planning and context-window assumptions may constrain larger or more complex deployments.

  • Scope: Chameleon’s adaptability has been demonstrated on the presented benchmarks, leaving expansion to wider tasks and domains as future work.The limitation concerns generalization beyond the benchmark settings.
  • Planning: The one-step LLM planner may affect overall performance because Chameleon does not re-plan while executing generated modules.The paper identifies tool-selection optimization and sequence planning as open research questions.
  • Scalability: Chameleon assumes that module lists and descriptions fit within the LLM context window, which may become difficult as complexity and inventory size increase.The paper also anticipates increased computational demands under those conditions.

C Broader Impacts

The paper describes potential benefits of Chameleon for complex multimodal applications while emphasizing risks from uncurated tools, data sources, and excessive reliance on system autonomy.

  • Potential benefits: Chameleon could support applications in education, finance, and decision-support systems by providing responses to complex multimodal tasks.The paper presents these as potential applications rather than evaluated deployments.
  • Risks: Uncurated data sources and external tools may create misinformation and privacy risks.The paper states that these risks can be managed through careful curation and data protection.
  • Human oversight: The paper warns that excessive reliance on Chameleon’s autonomy could undermine critical-thinking skills or job functions.It recommends treating the system as augmenting rather than replacing human capabilities.
Loading 2304.09842v3…