PaperlayerView exact revision on arXiv

Source-linked AI summary

Analyzing Chain-of-Thought Prompting in Large Language Models via Gradient-based Feature Attributions

Skyler Wu, Eric Meng Shen, Charumathi Badrinath, Jiaqi Ma, Himabindu Lakkaraju

arXiv:2307.13339v1cs.CLcs.AI

TL;DR

Why CoT prompting works remains poorly understood, despite the importance of understanding its behavioral effects for responsible deployment. The paper uses gradient-based saliency to examine token influence in open-source LLMs and finds that CoT does not increase relevant-token saliency magnitudes, but makes those scores more robust to question and output variation.

  • Problem

    Why CoT prompting improves question answering and how it changes model behavior remain poorly understood, a gap the authors regard as important for responsible deployment.

    Source: §1 · p. 1

  • Method

    The authors use gradient-based feature attribution to examine input-token saliency while comparing CoT and standard prompting in open-source LLMs.

    Source: §1 · p. 1

  • Results

    CoT does not increase relevant-token saliency magnitudes, but relevant-token scores vary less under question rewording and stochastic output than with standard prompting.

    Source: §1 · p. 2, §4 · p. 4

  • Takeaways & Limitations

    The findings suggest that simple interpretability tools can provide useful insights into LLM capabilities, limitations, and behavior, while further mechanistic study of CoT is warranted.

    Source: §5 · p. 5

    The study could not evaluate larger models because of computational limits, and gradient-based attribution methods produce significantly different values.

    Source: §5 · p. 5

Abstract

from arXiv · show

Chain-of-thought (CoT) prompting has been shown to empirically improve the accuracy of large language models (LLMs) on various question answering tasks. While understanding why CoT prompting is effective is crucial to ensuring that this phenomenon is a consequence of desired model behavior, little work has addressed this; nonetheless, such an understanding is a critical prerequisite for responsible model deployment. We address this question by leveraging gradient-based feature attribution methods which produce saliency scores that capture the influence of input tokens on model output. Specifically, we probe several open-source LLMs to investigate whether CoT prompting affects the relative importances they assign to particular input tokens. Our results indicate that while CoT prompting does not increase the magnitude of saliency scores attributed to semantically relevant tokens in the prompt compared to standard few-shot prompting, it increases the robustness of saliency scores to question perturbations and variations in model output.

1. Introduction

Understanding why chain-of-thought (CoT) prompting works is important for responsible deployment, yet its effects on model behavior remain poorly understood. The paper uses gradient-based saliency to probe CoT and finds more stable relevance scores under question and output variation, without increased score magnitudes for relevant tokens.

1. Introduction

  • The reasons CoT prompting improves question-answering performance remain poorly understood, despite the need to understand how it changes model behavior for responsible deployment.The introduction also situates this gap amid broader concerns about understanding LLM internal mechanisms and ensuring robust, safe, aligned behavior.Source: §1 · p. 1
  • The authors probe CoT with gradient-based feature attribution, whose saliency scores represent input tokens’ influence on the model’s output.The scores are derived from the gradient of the predicted next-token logit with respect to each input token.Source: §1 · p. 1
  • The paper examines open-source models with one to six billion parameters, where CoT accuracy gains have not been observed, and reports quantitative saliency differences.It calls for further work on the relationship between CoT’s empirical and mechanistic characteristics.Source: §1 · pp. 1–2
  • Across question-answering datasets, relevant-token saliency scores vary less under question phrasing changes and output stochasticity with CoT than with standard prompting.The authors suggest this stability may reflect more consistent consideration of question tokens and may be associated with more consistently accurate answers.Source: §1 · p. 2

2. Related Work

Prior work studies CoT’s performance effects and gradient-based attribution, but offers little gradient-based analysis of how CoT functions. The paper motivates inspecting token attributions rather than generated explanations, which may not faithfully reflect model behavior.

Feature Attribution Methods

  • Gradient-based feature attribution has a history in NLP, but its applications to LLMs have been comparatively limited.Prior work includes saliency for tokens, Integrated Gradients, transformer prediction analysis, and contrastive explanations for text classification and generation.Source: §2 · p. 2

Chain of Thought Prompting

  • Prior CoT studies found that incorrect intermediate reasoning can still accompany much of CoT’s performance gains, while relevance and correct ordering of reasoning steps matter in ablations.Counterfactual-prompting work also argued that correct factual content in intermediate steps may be less important than producing answer-like output structure.Source: §2 · p. 2

Understanding Chain-of-Thought Prompting

  • The paper identifies little prior use of gradient-based methods to probe CoT, motivating attention to token attributions rather than generated explanations.This choice is motivated by findings that incorrect CoT explanations can still accompany performance gains and may mislead human auditors about model functioning.Source: §2 · p. 2

3. Our Framework

The framework compares token saliency under standard and CoT few-shot prompting using gradient-based attribution methods. Experiments probe relevant-token saliency magnitude, robustness to question rewording, and stability across stochastic outputs.

3. Our Framework

  • The framework applies regular and contrastive gradient × input and L1-norm methods to score prompt and generated-output tokens with respect to the final answer token.Larger saliency magnitude is treated as greater influence on generation of the final answer.Source: §3 · p. 2
  • The main experiments use GPT-J, a 6-billion-parameter model, with additional experiments on GPT-2 XL and GPT-Neo; computational limits precluded applying the method to larger models.Findings were described as fairly consistent across the tested models.Source: §3 · pp. 2–3
  • The evaluation uses 50 questions each from SST, CoinFlip, GSM8K, and CSQA, with manually reworded CoinFlip and GSM8K versions.The datasets cover sentiment, coin-flip state, grade-school math, and commonsense multiple-choice questions; prompts use eight exemplars with direct answers or brief explanations.Source: §3 · p. 3

3.1. Analyzing the Impact of CoT

  • For relevant-token magnitude, the first experiment manually labels answer-relevant question tokens and compares their saliency under standard and CoT prompting.The authors hypothesize that CoT increases saliency magnitude for these tokens; labeling is more subjective for CSQA and SST than for GSM8K and CoinFlip.Source: §3.1 · p. 3
  • The rewording experiment compares fixed relevant-token saliency distributions across original and manually rewritten, semantically equivalent CoinFlip and GSM8K questions.The hypothesis is that CoT decreases variation, indicating more consistent focus despite rewording.Source: §3.1 · p. 3
  • The output-variation experiment reruns five questions per dataset 20 times to test whether CoT stabilizes saliency scores across stochastic outputs.The study uses GPT-J and calculates relevant-token saliency with all four methods; Figure 1 presents mean absolute scores for relevant tokens on the four original datasets.Source: §3.1 · pp. 3–4, Figure 1 · p. 4

4. Experimental Results

Across experiments, CoT lowered relevant-token saliency magnitudes rather than increasing them, while making scores more robust to question rewordings and variation in model outputs.

4. Experimental Results

  • For GPT-J, CoT did not increase relevant-token saliency magnitudes but made scores more robust to prompt perturbations and output stochasticity.The subsections report robustness to question rewordings and variation in model outputs.Source: §4 · p. 4

4.1. Impact of CoT Prompting on Model Accuracy

  • Accuracy rose from 0.5 to 0.64 on SST with CoT, the exception to otherwise absent gains at the tested model scale.The largest model tested, GPT-J, has 6B parameters.Source: §4.1 · p. 4

4.2. Impact of CoT Prompting on Relevant Tokens’ Saliencies

  • CoT uniformly decreased relevant-token saliency magnitudes across datasets and saliency methods, including on SST, where accuracy improved.The paper notes this SST result indicates model sensitivity to CoT prompting.Source: §4.2 · p. 4
  • Where cumulative absolute-saliency gaps were largest, standard prompts had more right-skewed token-score distributions; authors hypothesize longer CoT inputs dilute scores across more tokens.The proposed explanation is that gradients span more tokens because CoT uses longer exemplar prompts and explanations.Source: §4.2 · p. 4

4.3. Impact of CoT Prompting on Robustness to Question Rewordings

  • CoT made relevant-token saliency scores more constant across original and reworded prompts on both CoinFlip and GSM8K.Figure 2 presents GPT-J mean absolute scores for relevant tokens on original and reworded GSM8K, with and without CoT.Source: Figure 2 · p. 4, §4.3 · p. 4

4.4. Impact of CoT Prompting on Saliency Score Stability Across Variation in Model Outputs

  • 20 varied GPT-J generation runs show tighter relevant-token saliency clusters under CoT, indicating lower score variance across model outputs.Figure 3 plots manually labeled relevant-token scores for one GSM8K question; points sharing a color correspond to runs with the same answer.Source: §4.4 · pp. 4–5, Figure 3 · p. 5

5. Conclusion

Even at model scales where CoT accuracy gains do not materialize, CoT changes input-token saliency; the authors present focus and robustness analysis as an early step toward characterizing model responsiveness. They also identify limits in scale, attribution methods, and evaluation rigor.

5. Conclusion

  • CoT changes input-token saliency even at scales where its accuracy improvements do not appear.The authors say the observed changes highlight the need for further mechanistic investigation.Source: §5 · p. 5
  • The analysis examines models’ focus on input tokens and the robustness of that focus, without interpreting generated explanations themselves.Saliency scores are described as agnostic to output semantics, which is useful at the tested scales where explanations may be misleading or nonsensical.Source: §5 · p. 5
  • The authors characterize their approach as an extensible first step and argue that simple interpretability tools can yield useful insights at this scale.They connect those insights to understanding LLM capabilities, limitations, and behavior.Source: §5 · p. 5
  • Different gradient-based attribution methods produce significantly different values, motivating tests with other interpretability tools.The authors propose checking whether the observations hold across such tools.Source: §5 · p. 5
  • The evaluation should extend to larger models, more questions and datasets, and more procedural identification of relevant question tokens.The authors also suggest crowdsourcing relevant-token identification and question rewording.Source: §5 · p. 5

A. Appendix

The appendix gathers supplementary method and question-selection details, experimental results, prompts, and question-level materials.

A. Appendix

  • The appendix includes additional details on saliency methods and question selection, plus overall accuracy and error breakdowns.Source: A. Appendix · p. 8
  • It collects the full results and plots for Experiments 1–3.Source: A. Appendix · p. 8
  • It also provides few-shot exemplar prompts, full dataset questions, selected Experiment 3 repetition questions, and hand-annotated relevant tokens.Source: A. Appendix · p. 8

A.1. Additional Details on Saliency Methods

The appendix describes how questions and prompts become model input and how saliency is assigned to tokens preceding the final answer. It also specifies contrastive foil choices and the resulting score variants.

A.1. Additional Details on Saliency Methods

  • For each dataset question, the procedure tokenizes the prompt and question, feeds the sequence to the model, and identifies the final-answer token in its output.If an answer spans multiple tokens, the first is used; questions without a generated answer are excluded.Source: §A.1 · p. 9
  • Saliency methods are applied to input tokens preceding the final-answer token, with each token scored from the gradient of the model’s final-answer prediction.Source: §A.1 · p. 9
  • Contrastive scores use a foil token chosen by dataset: the correct answer when CSQA or GSM8K predictions are wrong, or the opposite choice for SST and CoinFlip.For incorrect CSQA or GSM8K predictions, correct-answer foils are used; correct predictions do not receive contrastive explanations.Source: §A.1 · p. 9
  • The procedure produces four score sets per question by pairing regular and contrastive versions of L1 and gradient×input attribution.Source: §A.1 · p. 9

A.2. Additional Details on Question Selection

Question selection was adapted to dataset structure and the relatively small models, using shorter, simpler, or more obvious items where appropriate.

A.2. Additional Details on Question Selection

  • Experiment 1 used 50 GSM8K questions of at most 200 characters, 50 randomly sampled CSQA questions, and 25 positive plus 25 negative SST questions with at least 10 tokens.CoinFlip questions followed a shared template and were not curated for difficulty.Source: §A.2 · p. 9
  • Experiment 3 repetition trials used five questions per dataset, selected randomly for CoinFlip and by relative simplicity or obviousness for the other datasets.GSM8K questions were relatively short with fewer reasoning steps; CSQA and SST items were judged relatively obvious to a human observer.Source: §A.2 · p. 9
  • The selection choices aimed to help smaller models parse questions and demonstrate their reasoning abilities rather than fail at parsing.Source: §A.2 · p. 9

A.3. Overall Model Accuracies and Error Breakdowns

CoT prompting is associated with lower saliency magnitudes and variability in several analyses, while example error breakdowns distinguish reasoning from final-answer correctness. The appendix also compares relevant-token saliency across original and reworded questions and notes limits in assessing reasoning quality.

A.3. Overall Model Accuracies and Error Breakdowns

  • The CSQA and SST breakdowns examine cases where reasoning quality and final-answer correctness diverge; the examples include an incorrect answer despite reasoning that mentions relevant content.The authors did not conduct this reasoning analysis on GSM8K or CoinFlip because sound reasoning was not immediately clear to define.Source: §A.3 · p. 10
  • In the illustrated SST example, CoT saliency magnitudes decreased while earlier tokens became relatively more significant; L1 methods also highlighted “positively.”The example uses GPT-J and contrasts CoT with standard prompting.Source: Figure 4 · p. 13
  • CoT lowered relevant-token saliency magnitudes and variance, especially on CoinFlip and GSM8K with GPT-Neo; the same pattern is reported for GPT-2 XL.The reported decreases concern manually labeled relevant tokens.Source: Figure 5 · p. 14, Figure 6 · p. 14
  • Across datasets and models, CoT saliency magnitudes for all tokens were less right-skewed, while standard prompting produced larger magnitudes regardless of token relevance.The discussion hypothesizes that CoT dilutes scores because gradients span more tokens.Source: Figure 7 · p. 15
  • Figures 19–23 compare relevant-token mean saliency between original and reworded CoinFlip and GSM8K questions, across three models, four methods, and both prompting styles.The GPT-J GSM8K comparison is included in the main text.Source: §A.5 · p. 21

A.5.1. GPT-J

For GPT-J, CoT prompting is associated with smaller differences in relevant-token mean saliency between original and reworded CoinFlip questions than standard prompting.

A.5.1. GPT-J

  • With GPT-J, CoT prompting yields much smaller original-versus-reworded CoinFlip differences in relevant-token mean saliency across all four methods.The figure cautions that its two subplots use differently scaled y-axes.Source: Figure 19 · p. 21
  • The comparison covers manually labeled relevant tokens from original and reworded CoinFlip questions under both standard and CoT prompting.The figure reports distributions of saliency-score magnitudes.Source: Figure 19 · p. 21
  • Because the two subplots use different y-axis scales, compare the relative original-versus-reworded differences rather than their absolute plotted magnitudes.Source: Figure 19 · p. 21

A.5.2. GPT-NEO

On GPT-Neo, the GPT-J pattern does not clearly hold for CoinFlip, whereas CoT is associated with smaller original-versus-reworded relevant-token mean-saliency differences on GSM8K.

A.5.2. GPT-NEO

  • For GPT-Neo on CoinFlip, the GPT-J trend of smaller original-versus-reworded differences with CoT does not seem to hold consistently.The figure compares relevant-token saliency distributions under standard and CoT prompting.Source: Figure 20 · p. 22
  • For GPT-Neo on GSM8K, CoT yields much smaller original-versus-reworded differences in relevant-token mean saliency across all four methods.The figure cautions that its subplots use different y-axis scales.Source: Figure 21 · p. 22

A.5.3. GPT-2 XL

For GPT-2 XL, CoT prompting is associated with smaller original-versus-reworded differences in relevant-token mean saliency on both CoinFlip and GSM8K.

A.5.3. GPT-2 XL

  • For GPT-2 XL on CoinFlip, CoT yields much smaller original-versus-reworded differences in relevant-token mean saliency across all four methods.The figure cautions that the two subplots use different y-axis scales.Source: Figure 22 · p. 23
  • For GPT-2 XL on GSM8K, CoT yields much smaller original-versus-reworded differences in relevant-token mean saliency across all four methods.The comparison uses distributions from original and reworded questions.Source: Figure 23 · p. 23

A.6.1. GPT-J

For GPT-J, CoT prompting produced tighter relevant-token saliency groupings than standard prompting on GSM8K and tightly grouped scores on CoinFlip despite seven unique answers. The appendix also reports answer-format deviations in CSQA and SST.

A.6.1. GPT-J

  • The figures plot manually labeled relevant-token saliency scores across 20 reruns of selected questions from CoinFlip, CSQA, GSM8K, and SST.The selected questions were described as representative of the general trend across each dataset.Source: §A.6.1 · p. 24
  • CSQA sometimes received bare-letter answers, while SST sometimes received an unprompted “neutral” answer despite that option being absent from the prompt.These deviations produced more final answers than expected for some questions.Source: §A.6.1 · p. 24
  • In Experiment 3, CoT unique-answer counts combine explanations and final answers, whereas standard-prompt counts distinguish final answers only.Source: Table 6 · p. 24, Table 7 · p. 24
  • GPT-J's CoT runs showed tighter relevant-token saliency groupings than standard prompting on GSM8K, and tight groupings on CoinFlip despite seven unique answers.The GSM8K comparison is reported across all four saliency methods; the CoinFlip figure also notes the binary-answer setting.Source: Figure 24 · p. 25, Figure 26 · p. 26

A.6.2. GPT-NEO

For GPT-Neo, CoT saliency groupings remained tight across CoinFlip, CSQA, and GSM8K, especially under regular and contrastive L1 norm methods. The contrast with standard prompting was less distinct on GSM8K than it was for GPT-J.

A.6.2. GPT-NEO

  • The figures plot manually labeled relevant-token saliency scores across 20 reruns of selected questions from CoinFlip, CSQA, GSM8K, and SST.The same selected questions were used as for GPT-J, for consistency.Source: §A.6.2 · p. 28
  • In Experiment 3, CoT unique-answer counts combine explanations and final answers, whereas standard-prompt counts distinguish final answers only.Source: Table 8 · p. 28, Table 9 · p. 28
  • CoinFlip and CSQA also showed remarkably tight CoT saliency groupings, despite 10 and 16 unique answers, respectively.The tightness was especially apparent under regular and contrastive L1 norm methods.Source: Figure 28 · p. 29, Figure 29 · p. 30
  • GPT-Neo showed tighter CoT relevant-token saliency groupings than standard prompting on GSM8K, with the trend especially clear under regular and contrastive L1 norm methods.The difference was less distinct than the corresponding GPT-J plots.Source: Figure 30 · p. 30

A.6.3. GPT-2 XL

For GPT-2 XL, CoT prompting yielded tighter relevant-token saliency groupings on GSM8K and CSQA, with tight groupings also reported on CoinFlip and SST despite many unique answers.

A.6.3. GPT-2 XL

  • The figures plot manually labeled relevant-token saliency scores across 20 reruns of selected questions from CoinFlip, CSQA, GSM8K, and SST.The same selected questions were used as for GPT-J and GPT-Neo, for consistency.Source: §A.6.3 · p. 32
  • In Experiment 3, CoT unique-answer counts combine explanations and final answers, whereas standard-prompt counts distinguish final answers only.Source: Table 10 · p. 32, Table 11 · p. 32
  • CoT saliency groupings remained tight on CoinFlip and SST despite 14 and 13 unique answers, respectively.Source: Figure 32 · p. 33, Figure 35 · p. 35
  • GPT-2 XL's CoT saliency groupings were noticeably tighter than standard prompting on GSM8K, with few outliers, and were especially tight under regular and contrastive L1 norm methods on CSQA.Source: Figure 34 · p. 34, Figure 33 · p. 34

A.7. Few-Shot Exemplar Prompts for All Datasets

This appendix section documents few-shot prompt exemplars and hand-annotated relevant-token lists across CSQA, SST, GSM8K, and CoinFlip, including standard and chain-of-thought prompts and original or reworded forms. The examples span sentiment classification, commonsense questions, arithmetic word problems, and coin-state questions.

A.7.1. SST WITH STANDARD FEW-SHOT PROMPTING (ORIGINAL)

  • Standard SST prompts pair movie-review sentences with positive or negative labels, without a rationale.The examples include both positive and negative reviews.Source: §A.7.1 · p. 36

A.7.2. SST WITH COT FEW-SHOT PROMPTING (ORIGINAL)

  • CoT SST prompts explain each sentiment label by identifying positively or negatively connotated words or phrases in the review.Examples cite terms such as “loves” and “beautiful” for positive sentiment and “complain” for negative sentiment.Source: §A.7.2 · p. 37

A.7.3. COINFLIP WITH STANDARD FEW-SHOT PROMPTING (ORIGINAL)

  • Standard CoinFlip prompts ask whether the coin remains heads up after named people flip or do not flip it, then give a yes-or-no answer.The examples cover two flips, one flip, and no flips.Source: §A.7.3 · p. 38

A.7.4. COINFLIP WITH COT FEW-SHOT PROMPTING (ORIGINAL)

  • CoT CoinFlip prompts count the flips and use odd or even parity to explain whether the initially heads-up coin remains heads up.The exemplars show two flips preserving heads and one flip changing it to tails.Source: §A.7.4 · p. 39

A.7.5. COINFLIP WITH STANDARD FEW-SHOT PROMPTING (REWORDED)

  • Reworded Standard CoinFlip prompts express the flip events in varied wording while retaining yes-or-no answers about the coin’s final orientation.Examples use forms such as “both flip,” “does not flip it, but” another person does, and “both do not flip.”Source: §A.7.5 · p. 40

A.7.6. COINFLIP WITH COT FEW-SHOT PROMPTING (REWORDED)

  • Reworded CoT CoinFlip prompts retain the parity-based explanation, counting flips in the rephrased scenarios before giving the answer.The examples explain both two-flip and one-flip outcomes.Source: §A.7.6 · p. 41

A.7.7. CSQA WITH STANDARD FEW-SHOT PROMPTING (ORIGINAL)

  • Standard CSQA prompts present multiple-choice questions and provide the answer as a letter, without an explanatory rationale.Examples include questions about absorbing fountain-pen ink and cable-requiring home entertainment equipment.Source: §A.7.7 · p. 42

A.7.8. CSQA WITH COT FEW-SHOT PROMPTING (ORIGINAL)

  • CoT CSQA prompts justify the selected multiple-choice letter with a brief explanation connecting the question to an answer choice.Rationales identify relevant properties, such as blotters absorbing ink or televisions requiring cable.Source: §A.7.8 · p. 43

A.7.9. GSM8K WITH STANDARD FEW-SHOT PROMPTING (ORIGINAL)

  • The original standard GSM8K exemplars state answers directly, without showing intermediate calculations, for arithmetic questions such as the trees and chocolates problems.The supplied examples give answers of 6 and 39, respectively.Source: §A.7.9 · p. 44

A.7.10. GSM8K WITH COT FEW-SHOT PROMPTING (ORIGINAL)

  • The original CoT GSM8K exemplars work through arithmetic step by step before stating answers to the same types of questions used in the standard prompts.Examples include combining quantities, repeated additions, and successive subtractions.Source: §A.7.10 · p. 45

A.7.12. GSM8K WITH COT FEW-SHOT PROMPTING (REWORDED)

  • The cited GSM8K exemplars show worked solutions: Leah’s chocolates total 32 + 42 = 74, then decrease by 35 to 39; Olivia’s five bagels cost 5 * 3 = 15, leaving 8 dollars.Source: §A.7.12 · p. 47

A.8. Full Dataset Questions

  • The full-dataset appendix describes selections of 50 questions each from SST, CoinFlip, CSQA, and GSM8K, plus semantically preserved reworded GSM8K and CoinFlip versions.The passage introduces the full datasets used for the experiments.Source: §A.8 · p. 48

A.8.1. SST (ORIGINAL)

  • The SST original examples are numbered movie-review snippets paired with an instruction to classify each sentence as positive or negative.The supplied examples include both favorable and critical review language.Source: §A.8.1 · p. 49
  • SST exemplars frame each movie-review sentence as a positive-or-negative classification question.The supplied examples include sentences about film quality, humor, warmth, and enjoyment.Source: §A.8.1 · p. 51

A.8.2. COINFLIP (ORIGINAL)

  • Original CoinFlip prompts state the starting face, describe whether named people flip the coin, and ask whether it remains on that face.Examples include both heads-up and tails-up starting states.Source: §A.8.2 · p. 52
  • Original CoinFlip examples include cases where neither person flips, so the prompt may describe no change to the coin.For example, one item starts heads up and says neither Matthew nor Lynn flips.Source: §A.8.2 · p. 52

A.8.3. COINFLIP (REWORDED)

  • Reworded CoinFlip examples preserve the initial-face and action structure while varying phrasing such as “starts,” “begins,” and “chooses not to flip.”One example says the coin “starts heads up,” while another says a person “begins by flipping” it.Source: §A.8.3 · p. 54
  • CoinFlip examples connect contrasting action statements with “however” when one person does not flip and another does, and with “Both” when two people flip.Source: §A.8.3 · p. 54
  • Reworded CoinFlip exemplars ask whether the starting heads-or-tails state remains after people flip or decline to flip the coin.The examples vary the initial state, number of people, and whether each person flips.Source: §A.8.3 · p. 55

A.8.4. CSQA (ORIGINAL)

  • Other original CSQA prompts ask about contextual associations, such as what a vacation massage brought or what a soldier saw when fighting began.Each prompt is accompanied by five candidate answers.Source: §A.8.4 · p. 56
  • The original CSQA examples also cover practical and general-knowledge questions, including finding out whether a restaurant serves milk and the effect of a virus on a person.Both are presented as multiple-choice questions.Source: §A.8.4 · p. 56, §30 · p. 58
  • Original CSQA exemplars pair everyday questions with five answer choices, including where children might play and where canned goods are kept.The listed choices include “family room” for the game question and “cabinet,” “pantry,” “kitchen,” “cupboard,” and “shelf” for the storage question.Source: §A.8.4 · p. 56
  • A CSQA exemplar asks where a note might indicate a melody and offers five choices, including “sheet music.”The prompt is “Where is a note likely to indicate a melody?”Source: §30 · p. 58

A.8.5. GSM8K (ORIGINAL)

  • Original GSM8K exemplars are arithmetic word problems about sales ratios, camel and dromedary counts, and daily free time.The prompts provide numerical conditions, such as combined record sales of 88,000 copies and a herd with 180 heads and 304 bumps.Source: §A.8.5 · p. 59
  • Original prompts also ask about rates and distances, including comparing how far two hikers travel at different speeds.Source: §A.8.5 · p. 59
  • The original GSM8K examples include multi-step cost questions, such as finding change after buying a pen and paper.Source: §A.8.5 · p. 60

A.8.6. GSM8K (REWORDED)

  • The reworded Cho and Chloe question preserves the original speeds, durations, and distance comparison while changing the phrasing.Source: §A.8.5 · p. 59, §A.8.6 · p. 62
  • The reworded couch problem presents the couch and rug relationships before stating the chair width, unlike the original ordering.Source: §A.8.5 · p. 59, §A.8.6 · p. 62
  • Other reworded examples cover multi-step arithmetic, including a car’s depreciation over time and the cost of filling a pool.Source: §A.8.6 · pp. 62–63
  • The reworded pen-and-paper prompt states each item’s price in separate sentences but retains the original values and change question.Source: §A.8.5 · p. 60, §A.8.6 · p. 63
  • The reworded GSM8K examples include arithmetic problems about prices, earnings, ages, quantities, and totals.Examples include discount calculations and questions about combined ages or amounts.Source: §A.8.6 · pp. 63–64

A.9. Selected Questions For Experiment 3 Repetition Trials

  • The Experiment 3 repetition trials used five questions each from CSQA, SST, GSM8K, and CoinFlip.The appendix also lists the selected prompts for each dataset.Source: §A.9 · p. 65

A.9.1. SST (ORIGINAL)

  • The five original SST examples are prompts asking for positive-or-negative sentiment classifications.The sentences range from praise to criticism and include descriptions of films and performances.Source: §A.9.1 · p. 65

A.9.2. COINFLIP (ORIGINAL)

  • The five original CoinFlip prompts ask whether a coin remains heads-up or tails-up after named people do or do not flip it.The examples vary the initial side and the sequence of flips.Source: §A.9.2 · p. 65

A.9.3. CSQA (ORIGINAL)

  • The original CSQA examples pair everyday commonsense questions with multiple-choice answer options.The supplied examples ask about interpreting a vivid dream and what a waiter does at the end of a meal.Source: §A.9.3 · p. 65

A.9.4. GSM8K (ORIGINAL)

  • The five original GSM8K examples are word problems involving ages, dental work, lemons, pool volume, and cookies.Each asks for a numerical answer based on the stated quantities or relationships.Source: §A.9.4 · p. 65

A.10.1. EXPERIMENTS 1 AND 2

  • The Experiment 1 and 2 annotations cover 50 questions each from SST, CoinFlip, CSQA, and GSM8K, including reworded CoinFlip and GSM8K sets.The tokens were hand-encoded as most relevant to each question’s final model output.Source: §A.10.1 · p. 66

A.10.2. EXPERIMENT 3

Loading 2307.13339v1…