PaperlayerView exact revision on arXiv

Source-linked AI summary

Iteratively Prompt Pre-trained Language Models for Chain of Thought

Boshi Wang, Xiang Deng, Huan Sun

arXiv:2203.08383v3cs.CL

TL;DR

PLMs store factual knowledge but have difficulty recalling it for complex multi-step reasoning, and existing prompting approaches have limitations in handling changing step contexts. The paper introduces iterative prompting with a context-aware prompter that dynamically synthesizes prompts from the current context. Experiments across three datasets report effectiveness for the iterative scheme and the prompter design.

  • Problem

    PLMs may internalize factual knowledge yet struggle to recall the sequence of facts needed for complex multi-step inference.

    Source: §1 · p. 1

  • Method

    The method elicits knowledge step by step and dynamically synthesizes prompts conditioned on the query and knowledge gathered so far.

    Source: §1 · p. 2

  • Results

    Experiments on three multi-step reasoning datasets show the iterative scheme is effective and the context-aware prompter outperforms existing prompting methods by notable margins.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that iterative prompting is superior to baselines and that its context-aware prompter addresses key limitations of earlier prompting methods in this setting.

    Source: §6 · p. 9

    Experiments use BART-large; larger models remain untested because their computational cost was a main obstacle.

    Source: Limitations · p. 9

Abstract

from arXiv · show

While Pre-trained Language Models (PLMs) internalize a great amount of world knowledge, they have been shown incapable of recalling these knowledge to solve tasks requiring complex & multi-step reasoning. Similar to how humans develop a "chain of thought" for these tasks, how can we equip PLMs with such abilities? In this work, we explore an iterative prompting framework, a new prompting paradigm which progressively elicits relevant knowledge from PLMs for multi-step inference. We identify key limitations of existing prompting methods, namely they are either restricted to queries with a single identifiable relation/predicate, or being agnostic to input contexts, which makes it difficult to capture variabilities across different inference steps. We propose an iterative context-aware prompter, which addresses these limitations by learning to dynamically synthesize prompts conditioned on the current step's contexts. Experiments on three datasets involving multi-step reasoning show the effectiveness of the iterative scheme and the context-aware prompter design.

1 Introduction

Pre-trained language models can store factual knowledge yet struggle to recall it in sequence for complex, multi-step inference. The paper proposes iterative, context-aware prompting and reports benefits across three reasoning datasets.

1 Introduction

  • PLMs can internalize simple facts but struggle with complex multi-step questions even when the needed facts are memorized.The paper frames the challenge as eliciting a sequence of stored knowledge needed to answer a query.Source: §1 · p. 1
  • Fine-tuning all PLM parameters becomes costly as models scale, while fine-tuning or prompting for one-shot knowledge generation can produce irrelevant facts or hallucinations.The paper also notes concern that fine-tuning may interfere with implicit knowledge storage.Source: §1 · pp. 1–2
  • Existing prompts are either restricted to single identifiable relations or insensitive to step-wise input contexts, limiting their fit for iterative reasoning.Source: §1 · p. 2
  • The proposed iterative context-aware prompter dynamically synthesizes prompts from the query and previously gathered evidence to elicit the next knowledge piece.At each step, prompts should focus on the relevant query component and integrate knowledge from earlier steps.Source: §1 · p. 2
  • Experiments across three reasoning datasets show the iterative scheme is effective, the context-aware prompter outperforms existing prompting methods by notable margins, and analyses examine faithfulness.The datasets cover multi-hop question answering and scientific reasoning over taxonomic relations.Source: §1 · p. 2

2 Methodology

The methodology formalizes multi-step knowledge recall with fixed PLM parameters and develops iterative prompting with a context-aware prompter. It trains the prompter on gold step contexts and uses recalled knowledge as context during inference.

2.1 Problem Setup

  • Given a query, the setup seeks a sequence of knowledge statements sufficient to decide the response, learning prompts that maximize log-likelihood while keeping PLM parameters fixed.The training data consists of query and knowledge-sequence pairs.Source: §2.1 · p. 2

2.2 Iterative Prompting Framework

  • At each inference step, the PLM recalls the next knowledge piece conditioned on the query and all previously gathered knowledge.Source: §2.2 · p. 3

2.3 Context-Aware Prompter

  • A learnable prompter dynamically generates prompt T from the current query and prior knowledge, addressing the limits of single-relation and context-agnostic prompts.Source: §2.3 · p. 3
  • The prompter contextualizes special tokens with the current step input and projects them into the PLM embedding space; the implementation uses an encoder-decoder PLM with prefix prompts.Its trainable parameter count scales comparably to Prompt-Tuning and more slowly than Prefix-Tuning.Source: §2.3 · p. 3, Figure 2 · p. 3

2.4 Learning and Inference

  • Training uses teacher-forced gold contexts and sequence-to-sequence objectives; inference feeds each recalled knowledge piece into the next step autoregressively.Inference runs for a predefined number of steps, with an optional jointly trained stopper described in the appendix.Source: §2.4 · pp. 3–4

3 Experimental Setup

The experiments test iterative knowledge recall on three multi-step reasoning datasets, comparing prompting and fine-tuning methods with measures of recalled evidence and final answer quality.

3 Experimental Setup

  • The study asks how to guide a PLM to recall knowledge and derive a chain of thought for multi-step reasoning, comparing prompting, fine-tuning, and prompter design on complex-reasoning datasets.Inference uses four steps for 2Wiki and R4C because over 99% of samples require at most four, and two for LoT. The final training set randomly samples 10% of the training data.Source: §3 · p. 4, §3.1 · p. 4

3.1 Datasets & Preprocessing

  • Across 2Wiki, R4C, and LoT, the evaluation covers multi-hop question answering and taxonomic reasoning using annotated knowledge statements or evidence paths.2Wiki has over 192k samples; R4C has 4.6k examples, and LoT tests reasoning over taxonomic relations.Source: §3.1 · p. 4

3.2 Compared Methods

  • The comparison includes iCAP, Prompt-Tuning, Prefix-Tuning, and PLM fine-tuning in iterative and non-iterative settings, plus PLM-QA and Oracle-RD for extrinsic evaluation.Dataset-specific methods using gold or distractor evidence paragraphs are excluded because they use attributes outside this work’s focus.Source: §3.2 · p. 5

3.3 Evaluation Metric

  • Intrinsic evaluation measures answer-entity recall and evidence coverage, while extrinsic evaluation measures answer quality using Exact Match and Answer F1.For LoT, extrinsic evaluation reports Exact Match for yes/no answers.Source: §3.3 · p. 5

3.4 Implementation Details

  • The experiments use BART-large as the PLM and RoBERTa-base as the prompter, with most experiments run using three random seeds and mean scores reported.The PLM is knowledge-enhanced by recovering masked elements in knowledge triplets, and the same enhanced PLM is used across compared methods.Source: §3.4 · pp. 5–6

4 Results

Across datasets, iterative prompting generally improves over non-iterative prompting, and iCAP outperforms Prompt-Tuning and Prefix-Tuning; analyses examine dataset-pattern exploitation and stepwise attention behavior.

4.1 Effectiveness of iCAP

  • On 2Wiki, iCAP gains more than 15% and 10% absolute F1 over Prompt-Tuning and Prefix-Tuning, respectively, while approaching PLM fine-tuning; Oracle-RD remains substantially better.Most compared methods benefit from iteration, and iCAP’s advantage appears across datasets and metrics.Source: §4.1 · p. 6
  • Knowledge-recall methods outperform PLM-QA in almost all comparisons, supporting the value of deriving intermediate knowledge for multi-step inference.The paper reports this as evidence that PLMs struggle to use stored knowledge directly for these tasks.Source: §4.1 · p. 6

4.2 Faithfulness of Prompting

  • Random controls indicate that prompting methods capture only mild spurious dataset regularities, while PLM fine-tuning captures substantially more.The authors report that iCAP captures a bit more regularity than Prompt-Tuning and Prefix-Tuning, but at a very small level.Source: §4.2 · p. 7, Table 3 · p. 7
  • In the illustrated example, attention focuses on the queried relation at step one and integrates the prior answer with the query at step two.The authors describe this visualization as qualitative and note that systematic examination of transformer internals remains an open challenge.Source: §4.2 · pp. 7–8, Figure 3 · p. 8

5 Related Work

Related work situates the paper between PLMs’ stored knowledge and their multi-step recall difficulty, and among prompting approaches and iterative retrieval methods.

Memorization and Reasoning in PLMs

  • PLMs can memorize world knowledge and perform on knowledge-intensive tasks, yet prior studies report difficulty recalling it for multi-step inference [Rae et al., 2021].The paper reports that its own experiments also verify this difficulty.Source: §5 · p. 8

Prompt Learning

  • Prior prompting work includes manually designed task-specific templates, while this paper seeks automatically learned prompts for recalling knowledge step by step.Chain-of-Thought prompting provides intermediate reasoning steps but is reported to work well only with extremely large PLMs, typically over 100B parameters.Source: §5 · p. 8
  • Learned prompting spans discrete prompt-token search and continuous prompt optimization, but existing methods may be relation-specific or insensitive to sample-level inputs.Prior work also raises the concern that learned prompts could exploit spurious dataset regularities.Source: §5 · pp. 8–9

Iterative Knowledge Retrieval

  • Methods that iteratively retrieve from explicit knowledge sources inspire viewing this task as iterative retrieval over knowledge implicit in PLMs [Xiong et al., 2021; Qi et al., 2019; Khattab et al., 2021; Mo et al., 2022b].Source: §5 · p. 9

6 Conclusion & Future Work

The paper concludes that iterative prompting and a context-aware prompter are effective for multi-step reasoning, with analyses examining learned prompting behavior.

6 Conclusion & Future Work

  • The paper presents iterative prompting as a way to elicit a chain of thought from PLMs for multi-step reasoning.Source: §6 · p. 9
  • Quantitative and qualitative analyses examine the faithfulness of learned prompting behaviors.Source: §6 · p. 9
  • Future work includes extending the approach to pretraining and investigating human intervention during inference to edit intermediate steps.The authors suggest such intervention could improve transparency and trustworthiness and reduce error propagation.Source: §6 · p. 9

Limitations

The authors report limitations in model scale, knowledge-statement quality, and the limited range of context-aware prompter designs explored.

Experiments with larger-scale models

  • The iterative prompting approach outperforms baselines by a large margin, but the authors say there is still room to improve.Experiments used BART-large; larger models may have greater capacity for internalizing knowledge, but their computational cost was an obstacle.Source: Limitations · p. 9

Datasets with noisy knowledge statements

  • Processed HotpotQA and StrategyQA knowledge statements remained noisy, and all methods could not be trained well with them.The source statements are raw evidence-paragraph sentences with irrelevant information; preprocessing included coreference resolution and sentence simplification and decomposition.Source: Limitations · p. 9

Exploring alternative architectural designs

  • The context-aware prompter was implemented as a simple, intuitive instantiation, leaving alternative iterative-prompting designs unexplored.Suggested directions include alternative Prompter-PLM interfaces and prompt lengths that vary across inference steps.Source: Limitations · p. 9

A.1 Additional Details on Experiments

The experiments used dataset-specific batch sizes and epoch counts, with other hyperparameters summarized in Table 4.

A.1 Additional Details on Experiments

  • Batch sizes were 32, 128, and 32 for 2Wiki, LoT, and R4C, respectively.Source: §A.1 · p. 13
  • Training ran for 70, 50, and 40 epochs for 2Wiki, LoT, and R4C, respectively.Source: §A.1 · p. 13
  • Table 4 summarizes the other hyperparameters used in the experiments.Source: §A.1 · p. 13

A.2 More details on PLM Knowledge Enhancement

The knowledge-enhancement setup adds masked-element recovery pre-training and, for 2Wiki and LoT, distractor statements; the authors report near-perfect recovery accuracy.

A.2 More details on PLM Knowledge Enhancement

  • For 2Wiki, the authors added 383k extra knowledge statements to 240k original ones, filtering processed Wikidata triples by subjects appearing in original statements; LoT used provided distractors, while R4C received none.Although the training set was downsampled in the main experiment, knowledge enhancement used the full dataset.Source: §A.2 · p. 13
  • Table 6 presents intrinsic evaluation results from jointly training the Prompter and a Stopper that learns when recalled knowledge is adequate for answering a query.Source: Table 6 · p. 13
  • The enhanced PLM recovered masked elements in knowledge statements with near-perfect accuracy.Source: §A.2 · p. 13

A.3 Standard Metrics for Intrinsic Evaluation

The authors compare standard text-generation metrics with their proposed intrinsic metrics for 2Wiki, finding similar trends but closer grouping with standard metrics.

A.3 Standard Metrics for Intrinsic Evaluation

  • Standard ROUGE 1/2/L and BLEU results for 2Wiki show the same overall trend as the proposed metrics.Source: §A.3 · p. 13
  • Standard metrics group results more closely than the proposed metrics.Source: §A.3 · p. 13
  • The authors attribute the closer grouping to standard metrics' weaker focus on important recalled-evidence elements such as entities.Source: §A.3 · p. 13

A.4 Prompter with Automatic Stopping

The appendix examines automatic stopping, training-test overlap, prompter scale, and additional attention visualizations. Automatic stopping usually matches the ground-truth step count, with small intrinsic-performance drops after truncation.

A.4 Prompter with Automatic Stopping

  • The Stopper is a two-hidden-layer feed-forward network on the Prompter’s [CLS] embedding, jointly trained with the Prompter to predict when to stop recall.Its hidden dimensions are 500 and 100, and its binary classification loss is weighted by 0.1 in the combined objective.Source: §A.4 · p. 13
  • Truncating recall with the Stopper causes small intrinsic-performance drops; its 98.5% ground-truth step-count match is not a perfect measure.The authors note that recalled evidence differs from the ground-truth knowledge pieces.Source: §A.4 · pp. 13–14

A.5 Test-Train Overlap

  • 2Wiki inference is mostly two or four steps, making 50% training-overlap coverage common; Table 7 bins development and test examples by overlap percentage.The table distinguishes full-training-set statistics from the down-sampled set used in the experiment.Source: §A.5 · p. 14, Table 7 · p. 14

A.6 Examples of processed data samples & failure cases of iCAP

  • Table 9 presents processed examples across datasets and subcategories, alongside failure cases of the proposed method..Source: §A.6 · p. 14, Table 9 · p. 17

A.7 Variants of Prompter Scales

  • Prompter performance grows with scale: BERT-small (28.8 million parameters) is only slightly behind RoBERTa-base, while BERT-tiny (4.4 million parameters) basically fails.The authors conclude that the prompter must exceed a certain scale to work well; Table 8 reports the intrinsic evaluation.Source: §A.7 · p. 14, Table 8 · p. 14

A.8 More Examples on Prompter Attention Visualizations

  • Additional attention visualizations cover Composition, Comparison, Inference, and Bridge-comparison reasoning types.The examples are from 2Wiki, with each figure’s caption identifying its reasoning type.Source: §A.8 · p. 14, Figure 4 · p. 14, Figure 5 · p. 15
Loading 2203.08383v3…