PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models as Analogical Reasoners

Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H. Chi, Denny Zhou

arXiv:2310.01714v3cs.LG

TL;DR

CoT prompting often depends on labeled reasoning exemplars, motivating a method that can provide relevant guidance without manual exemplar construction. Analogical prompting has LLMs self-generate exemplars or knowledge before solving problems, and it outperforms 0-shot and few-shot CoT across diverse reasoning tasks. Its main limitations are increased inference computation and dependence on sufficiently capable models with relevant learned knowledge.

  • Problem

    Existing CoT methods either provide generic guidance or require labeled reasoning exemplars, creating a need to automate relevant exemplar generation.

  • Method

    Analogical prompting instructs LLMs to self-generate relevant exemplars and optionally high-level knowledge in context before solving the target problem.

  • Results

    Analogical prompting outperforms 0-shot CoT and few-shot CoT across math, code-generation, and other reasoning tasks.

  • Takeaways & Limitations

    Self-generated exemplars provide customized reasoning guidance without labeled data, while knowledge generation supports generalization on challenging tasks such as code generation.

  • Takeaways & Limitations

    The approach increases inference computation and works less reliably when models lack sufficient strength or relevant learned knowledge.

Abstract

from arXiv · show

Chain-of-thought (CoT) prompting for language models demonstrates impressive performance across reasoning tasks, but typically needs labeled exemplars of the reasoning process. In this work, we introduce a new prompting approach, analogical prompting, designed to automatically guide the reasoning process of large language models. Inspired by analogical reasoning, a cognitive process in which humans draw from relevant past experiences to tackle new problems, our approach prompts language models to self-generate relevant exemplars or knowledge in the context, before proceeding to solve the given problem. This method presents several advantages: it obviates the need for labeling or retrieving exemplars, offering generality and convenience; it can also tailor the generated exemplars and knowledge to each problem, offering adaptability. Experimental results show that our approach outperforms 0-shot CoT and manual few-shot CoT in a variety of reasoning tasks, including math problem solving in GSM8K and MATH, code generation in Codeforces, and other reasoning tasks in BIG-Bench.

1 INTRODUCTION

Analogical prompting addresses CoT’s tension between generic guidance and costly labeled exemplars by having LLMs generate relevant reasoning support before solving each problem. Across math, code, and other reasoning tasks, it outperforms 0-shot and few-shot CoT.

  • Motivation: CoT prompting improves complex-task reasoning, but 0-shot guidance may be too generic while few-shot CoT requires costly labeled reasoning exemplars.The paper frames automating relevant exemplar generation as a way to combine detailed guidance with less manual labeling.
  • Approach: Analogical prompting asks LLMs to self-generate relevant problems, solutions, and optionally high-level knowledge before solving the original problem.The process can generate knowledge, exemplars, and the final solution within one prompt.
  • Advantages: Self-generated exemplars eliminate manual labeling and tailor reasoning guidance to each problem rather than relying on generic exemplars.The approach is presented as an alternative to retrieving relevant exemplars from external labeled data.
  • Results: +4% average accuracy gain over 0-shot CoT and few-shot CoT across reasoning tasks and multiple base LLMs.Evaluation covers GSM8K, MATH, Codeforces, and BIG-Bench using GPT-3.5, GPT-4, and PaLM2.

2 RELATED WORKS

The related work situates analogical prompting within LLM in-context learning, self-generation, and chain-of-thought methods. Its distinguishing choice is to self-generate reasoning exemplars instead of retrieving them from external labeled data.

  • LLM Prompting: Large language models use prompts, instructions, and exemplars to guide generated responses through in-context and few-shot learning.Analogical prompting harnesses these abilities to guide reasoning.
  • Self-Generation: Self-generated reasoning exemplars differ from factual recitation because they recall problem-solving and reasoning processes rather than only relevant facts.The distinction is made relative to prior self-generation work for open-domain question answering.
  • Chain-of-Thought Prompting: Chain-of-thought prompting guides LLMs to produce intermediate reasoning steps toward a final answer.The section identifies 0-shot CoT and few-shot CoT as common instances.
  • Chain-of-Thought Prompting: Few-shot CoT uses labeled question–rationale–answer exemplars, whereas 0-shot CoT uses a general instruction without exemplars.Few-shot CoT can be stronger but requires labeled exemplars.
  • Related CoT Methods: Retrieval-based CoT obtains problem-relevant exemplars from external data, while analogical prompting makes the LLM self-generate them.Self-generation is described as simpler because it does not require external data retrieval.

3 PRELIMINARIES

The preliminaries formalize problem solving and prompting as mapping a problem statement into an LLM input whose generated output is a solution. The goal is to design this mapping to improve solution accuracy.

  • Problem-Solving Setup: Problem-solving tasks map a problem statement x to a solution y, which may contain reasoning steps r and a final answer a.The framework covers mathematical questions and code-generation specifications.
  • Prompting Formalization: A prompting method ϕ maps problem statement x to textual input ϕ(x), and the LLM generates solution ŷ=LLM(ϕ(x)).This notation defines prompting as the construction of the model’s textual input.
  • Prompting Variants: 0-shot prompting supplies x directly, 0-shot CoT adds a general instruction, and few-shot CoT adds labeled exemplars {(x_i,r_i,a_i)}^K_i=1.The three methods differ in what the prompting function appends to the target problem.
  • Objective: The preliminaries define the design objective as creating a prompting method ϕ that improves the accuracy of generated solutions.This objective motivates the analogical prompting method introduced next.

4 APPROACH

Analogical prompting augments a target problem with self-generated relevant exemplars and, for difficult tasks, high-level knowledge before producing the solution. The design emphasizes tailored guidance, exemplar diversity, and knowledge-first generation.

  • Overview: Analogical prompting has LLMs self-generate relevant exemplars or knowledge in context before solving the target problem.The approach includes separate techniques for self-generated exemplars and self-generated knowledge plus exemplars.
  • Overview: The method assumes LLMs can recall training-acquired problem-solving knowledge to perform in-context learning on new problems.The prompt explicitly asks the model to recall or generate relevant problems and solutions.
  • Self-Generated Exemplars: A single prompt asks the model to recall distinct relevant problems, explain their solutions, and then solve the initial problem.The generated question–rationale–answer sequences remain in context for the final solution.
  • Self-Generated Exemplars: Generating diverse exemplars is important because repetitive exemplars can mislead the model when solving the target problem.The prompt explicitly instructs the model to generate problems distinct from one another.
  • Advantages: Tailored exemplars provide detailed reasoning guidance without manual labeling and are more problem-specific than fixed few-shot exemplars.Examples include adapting guidance to geometry or probability rather than general mathematics.
  • Self-Generated Knowledge + Exemplars: For complex tasks, self-generated knowledge complements low-level exemplars because models may otherwise over-rely on exemplars and fail to generalize.The knowledge instruction asks the model to identify core concepts and provide a tutorial.
  • Self-Generated Knowledge + Exemplars: Generating knowledge before exemplars yields superior results by aligning exemplars with fundamental problem-solving approaches rather than surface-level lexical similarities.The paper reports this ordering as a technical design choice supported by Table 7.

5 EXPERIMENTAL SETUP

The evaluation covers mathematical reasoning, code generation, and diverse BIG-Bench reasoning tasks across several base LLMs. It compares analogical prompting with zero-shot, few-shot, and retrieved CoT baselines using task-specific evaluation procedures.

  • Tasks: Experiments cover mathematical problem solving, competitive-programming code generation, and diverse BIG-Bench reasoning tasks.The mathematical benchmarks are GSM8K and MATH; BIG-Bench includes word sorting, logical deduction, temporal sequences, colored objects, and formal fallacies.
  • Tasks: Codeforces evaluation uses 2023 level-A problems, with generated-code correctness determined by passing all test cases.Each problem supplies a statement to the LLM and test cases for assessment.
  • Methods to compare: The proposed method self-generates K = 5 exemplars for GSM8K and K = 3 for MATH and BIG-Bench, while Codeforces also receives self-generated knowledge.The math prompt asks the model to recall relevant problems before solving; the code prompt asks it to explain core concepts, provide relevant problems, and solve the original problem.
  • Evaluation: Mathematical performance is reported as accuracy, while Codeforces performance uses Acc@1 and Acc@10.Acc@k measures whether at least one of k sampled outputs is correct; ten outputs are sampled at temperature 0.7 for code generation.
  • Base LLMs: The study evaluates GPT-3.5-turbo, GPT-4, and PaLM 2-L as base language models.The reported access period for GPT-3.5-turbo and GPT-4 is June–September 2023.
  • Methods to compare: Baselines include 0-shot CoT, fixed few-shot CoT, and dynamically retrieved few-shot CoT using labeled problem-solution pairs.Retrieved exemplars are selected with Sentence-BERT embeddings and cosine similarity; fixed few-shot CoT uses K = 5 or K = 3 exemplars depending on the dataset.

6 RESULTS

Across math, code-generation, and BIG-Bench reasoning tasks, analogical prompting generally outperforms standard CoT baselines by generating problem-relevant exemplars and, when useful, high-level knowledge. Results also show scale- and task-dependent limits: self-generation is stronger with larger models, while generated exemplars can fail to generalize or be valid.

  • Main results: Analogical prompting outperforms 0-shot and few-shot CoT across GSM8K, MATH, Codeforces, and BIG-Bench reasoning tasks.The method is evaluated against these baselines across multiple tasks and model scales.
  • Mathematical problem solving: Self-generated exemplars match test problems more closely than fixed few-shot exemplars, supporting tailored reasoning for diverse mathematical tasks.Qualitative examples show geometry and probability exemplars preceding correct solutions, whereas fixed exemplars may mismatch the test problem.
  • Code generation: 15% Acc@1 for GPT3.5-turbo versus 16% Acc@1 for GPT4 on Codeforces shows near-competitive code-generation performance.Self-generated knowledge provides an additional boost over self-generated exemplars on this challenging task.
  • Knowledge and exemplars: Generating knowledge complements exemplars by encouraging high-level generalization and closer alignment with core algorithms, especially for Codeforces.The reported benefit is less significant for GSM8K and BIG-Bench, which the authors describe as less complex.
  • Scale and retrieval: Self-generated exemplars outperform retrieved exemplars with larger-scale models, whereas retrieval performs better with smaller-scale models.The comparison spans text-curie-001 through text-davinci-003 and links the difference to the models’ ability to generate useful related exemplars.
  • Qualitative analysis: Among incorrectly solved problems, 12/50 involved a generalization gap and 8/50 involved overreliance on specific exemplars.Other failures included irrelevant or incorrect exemplars and calculation errors, motivating exemplars that better facilitate generalization.

7 CONCLUSION

The paper introduces analogical prompting, which self-generates relevant reasoning exemplars for solving problems without labeled data. Across diverse reasoning tasks, it outperforms both 0-shot and few-shot CoT.

  • Analogical prompting self-generates relevant reasoning exemplars for solving problems without requiring labeled data.
  • The approach provides detailed, customized exemplars for individual problems, addressing limitations of existing 0-shot and few-shot CoT methods.
  • Analogical prompting outperforms 0-shot CoT and few-shot CoT across math, code-generation, and logical or temporal reasoning tasks.

8 LIMITATIONS AND FUTURE RESEARCH

The approach has limitations involving inference cost, model capability, and prompt sensitivity. Its self-generation may fail when models lack sufficient strength or relevant knowledge, while stronger models can handle somewhat more complex problems.

  • Analogical prompting increases inference computation because it generates more tokens than vanilla 0-shot and 0-shot CoT prompting.
  • Compared with few-shot CoT, the approach uses fewer input tokens but more output tokens because generated exemplars are counted as output.
  • Self-generation can fail when an LLM lacks sufficient strength or has not learned relevant knowledge for the new problem.
  • The approach is better suited for stronger or larger-scale LLMs, which can draw on relevant prior knowledge for somewhat more complex problems.
  • Performance remains subject to the specific prompt phrases used to query the model.

A.1 LANGUAGE MODELS AND REASONING

The paper situates its work within research on teaching and evaluating reasoning in language models. It complements prompting, training, retrieval, and tool-based approaches by using analogical reasoning to improve inference.

  • Reasoning applies knowledge through a series of steps to derive solutions for new problems.
  • Reasoning capabilities are evaluated with datasets spanning multi-step question answering, mathematical problem-solving, and code generation.
  • Existing approaches teach reasoning through training or fine-tuning, structured knowledge retrieval, and external reasoning modules.
  • Chain-of-thought prompting guides language models to generate intermediate reasoning steps, while related work adds structured algorithms, search, tools, and programs.
  • This work complements these efforts by drawing inspiration from human analogical reasoning to improve language-model prompting and inference.

A.2 ANALOGICAL REASONING

Analogical reasoning involves recalling relevant past experiences for new challenges and identifying structural or relational similarities. The paper applies these principles to language-model inference as a pioneering effort.

  • Analogical reasoning is a cognitive process in which people recall relevant past experiences when facing new challenges.
  • It relies on identifying structural and relational similarities between past and current situations.
  • Analogical reasoning has influenced artificial intelligence and machine-learning algorithms and has been used as a reasoning benchmark.
  • This work makes a pioneering effort to apply analogical reasoning principles to improve language-model inference.

B CODEFORCES DATA COLLECTION

The Codeforces dataset was scraped from codeforces.com using procedures from prior work and restricted to Level-A problems published between January and August 2023.

  • Data collection: The dataset contains Level-A Codeforces problems published between January 2023 and August 2023.Each problem includes its full description and test cases, including public and post-contest hidden tests.

C ADDITIONAL RESULTS

Additional results examine prompt construction and show that analogical prompting can improve reasoning outputs by recalling relevant examples, while baseline prompts may produce incorrect solutions. Ablations also consider exemplar diversity, knowledge-generation order, and self-consistency.

  • Ablations: The ablations examine exemplar diversity, the order of self-generated knowledge and exemplars, and whether self-consistency further boosts analogical prompting.The cited tables report these as studies of prompting design and performance enhancement.
  • Prompt construction: Analogical prompting instructs models to recall relevant mathematical examples before solving the initial problem.The prompt requests distinct examples, worked solutions, and then a solution to the original problem.
  • Prompt construction: The prompt can generate several analogous problems with worked solutions before solving an initial discounted-price problem.The examples vary across shirts, shoes, meals, laptops, and toys, using both forward and reverse discount calculations.
  • Codeforces: The baseline GPT3.5-turbo-16k vanilla 0-shot output fails to use the prefix product algorithm and produces an incorrect solution.The cited baseline is explicitly contrasted with an approach that recalls relevant exemplars.
  • BIG-Bench: The baseline GPT3.5-turbo 0-shot CoT output does not recall relevant exemplars and adopts an incorrect approach to a deductive reasoning problem.This comparison indicates that generic 0-shot chain-of-thought did not supply the task-specific guidance used by analogical prompting.
Loading 2310.01714v3…