PaperlayerView exact revision on arXiv

Source-linked AI summary

Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango

Aman Madaan, Amir Yazdanbakhsh

arXiv:2209.07686v2cs.CLcs.AIcs.LG

TL;DR

Although chain-of-thought prompting benefits few-shot LLM performance, why it works remains unclear. The paper uses controlled counterfactual prompts that alter prompt components to study this question, finding that text and patterns act symbiotically in CoT.

  • Problem

    Despite chain-of-thought prompting’s demonstrated benefits for few-shot reasoning, the rationale for its success remains unclear.

    Source: §1 · p. 2

  • Method

    The authors use controlled counterfactual experiments to examine how prompt components affect chain-of-thought performance.

    Source: §3.1 · p. 4, §10 · p. 14

  • Results

    The authors find that text and patterns have a symbiotic relationship in chain-of-thought reasoning, while the correctness of symbols and patterns is immaterial to task solve rate.

    Source: §1 · p. 2, §10 · p. 15

  • Takeaways & Limitations

    The authors conclude that text helps extract semantic patterns, while patterns reinforce task understanding and direct text generation.

    Source: §1 · p. 2, §10 · p. 15

    The authors caution that counterfactual examples can be misleading and precarious.

    Source: prompts · p. 3

Abstract

from arXiv · show

The past decade has witnessed dramatic gains in natural language processing and an unprecedented scaling of large language models. These developments have been accelerated by the advent of few-shot techniques such as chain of thought (CoT) prompting. Specifically, CoT pushes the performance of large language models in a few-shot setup by augmenting the prompts with intermediate steps. Despite impressive results across various tasks, the reasons behind their success have not been explored. This work uses counterfactual prompting to develop a deeper understanding of CoT-based few-shot prompting mechanisms in large language models. We first systematically identify and define the key components of a prompt: symbols, patterns, and text. Then, we devise and conduct an exhaustive set of experiments across four different tasks, by querying the model with counterfactual prompts where only one of these components is altered. Our experiments across three models (PaLM, GPT-3, and CODEX) reveal several surprising findings and brings into question the conventional wisdom around few-shot prompting. First, the presence of factual patterns in a prompt is practically immaterial to the success of CoT. Second, our results conclude that the primary role of intermediate steps may not be to facilitate learning how to solve a task. The intermediate steps are rather a beacon for the model to realize what symbols to replicate in the output to form a factual answer. Further, text imbues patterns with commonsense knowledge and meaning. Our empirical and qualitative analysis reveals that a symbiotic relationship between text and patterns explains the success of few-shot prompting: text helps extract commonsense from the question to help patterns, and patterns enforce task understanding and direct text generation.

1. INTRODUCTION

Large language models can perform few-shot tasks through prompts, and chain-of-thought prompting improves performance with intermediate rationales, but the mechanism behind its success remains unclear (Wei et al., 2022). The paper investigates that mechanism through controlled prompt alterations and reports findings about symbols, patterns, and text.

  • The study uses controlled counterfactual prompts that alter one of three components—symbols, patterns, or text—across four reasoning tasks and PaLM, GPT-3, and CODEX.It analyzes performance divergence systematically and qualitatively.Source: §1 · p. 2
  • The exact symbols used and the correctness of symbols and patterns are virtually immaterial to task solve rate.The paper reports this counterintuitive result across its experiments.Source: §1 · p. 2
  • Text and patterns work symbiotically: text supplies commonsense knowledge, while patterns reinforce task understanding and guide useful output generation.The authors argue that chain-of-thought success may depend more on imitating prompts and generating the right tokens than on reasoning ability.Source: §1 · p. 2
  • CCOT prunes prompts by 20% while retaining indispensable tokens without reducing task solve rate.The authors propose concise prompting, or CCOT, based on their findings.Source: §1 · p. 2

2. COUNTERFACTUAL EXPLANATION FOR CHAIN OF THOUGHT

The section uses prompts that alter one aspect of in-context examples to investigate what supports chain-of-thought performance. On GSM-8K, numbers and thought credibility were not paramount, while removing either component nullified CoT efficacy; counterfactual explanations are treated cautiously.

2. COUNTERFACTUAL EXPLANATION FOR CHAIN OF THOUGHT

  • Each counterfactual prompt changes one aspect of the in-context examples, enabling performance comparisons that probe that aspect’s role in task success.For example, replacing numbers in math thoughts with symbols tests the role of actual numbers; the study makes no assumptions about model architecture.Source: §2 · pp. 2–3
  • On GSM-8K, neither actual numbers nor thought credibility was paramount to CoT success; changing thought style or wording had modest impact, but removing either component nullified CoT efficacy.Source: §2 · p. 3
  • When a model performed similarly under a counterfactual prompt and the original, its attention patterns were comparable.Source: prompts · p. 3

prompts

  • Because counterfactual examples can mislead, consistent and systematic performance divergences are treated as more credible interpretations, while contradictory observations are not rejected.The authors also note that prompts critical to outputs can be identified and collected, unlike fine-tuned methods.Source: prompts · p. 3

3. EXPERIMENTAL SETUP

The setup centers on PaLM-62B experiments across reasoning tasks, then analyzes prompts through three semantic components: symbols, patterns, and text.

3. EXPERIMENTAL SETUP

  • Experiments center on PaLM-62B, with GPT-3 and CODEX added for reproducibility; findings concur across the studied models.Results from PaLM-62B appear in the main body, while results from GPT-3 and CODEX are deferred to Appendix E.Source: §3 · p. 3
  • Tasks target settings where CoT improves over direct prompting: mathematical (GSM-8K; Cobbe et al. (2021)), commonsense (date and sports; BIG-bench Collaboration (2022)), and symbolic (sorting) reasoning (Wei et al., 2022).Source: §3 · p. 3

3.1. SEMANTIC COMPONENTS OF PROMPTS

  • Symbols are token sequences the model reasons about, including numerical quantities, sports players and activities, and date/time expressions.Source: §3.1 · p. 3
  • Patterns combine symbols and operators or prompt structure; examples include equations, sports relations, sorting comparisons, and DATE’s implicit but consistent calculation and answer-generation steps.Source: §3.1 · pp. 3–4
  • Text comprises tokens outside symbols and patterns, helping outline tasks, connect patterns to symbols, and contextualize symbols as the prompt’s conceptual glue.Source: §3.1 · p. 4

4. ROLE OF SYMBOLS

Symbol identities and types are generally immaterial to CoT performance, but removing symbols nullifies its gains. Vanilla CoT and abstract-symbol prompts also show near-identical attention patterns.

4. ROLE OF SYMBOLS

  • Abstract placeholders and out-of-distribution symbols had little to no performance impact, but deleting all symbols nullified CoT gains.The results support the finding that symbol type is primarily immaterial to model performance.Source: §4 · pp. 4–5
  • 60.6% to 80.0% was the SORTING solve-rate increase with integers ≥10 over vanilla CoT, while artificial SPORTS names and activities closed the DIRECT–CoT gap yet remained marginally better than DIRECT.The paper proposes that larger integers made the modified thoughts more informative about the sorting task.Source: §4 · p. 5
  • Cohen’s κ agreement between CoT and symbolic counterfactual prompts ranged from moderate (>0.4) to substantial (>0.61), whereas agreement between DIRECT and counterfactual prompts was meager (<0.2).These agreement results support the observation that model behavior may be similar across symbol types and values.Source: §4 · p. 5

4.1. ATTENTION ANALYSIS

  • Near-identical attention patterns for vanilla CoT and abstract-symbol prompts indicate that the model presumably uses a similar inference mechanism for both.Attention is used as a proxy for understanding inference, and the analysis covers randomly sampled questions across the studied datasets.Source: §4.1 · p. 6

5. ROLE OF PATTERNS

Patterns are necessary but not sufficient for CoT prompting to succeed. Their absence undermines performance, while the effects of wrong patterns depend on the task.

No patterns

  • Without patterns, CoT consistently underperforms and approaches DIRECT, while pattern-only prompts erase CoT’s gains.Even without patterns, prompts still outperform DIRECT across datasets, suggesting that semantically correct statements in thoughts also matter.Source: §5 · p. 7, Table 3 · p. 7

Pattern-only prompts

  • Pattern-only prompts are futile and annul CoT’s gains.Removing the rest of the thoughts while retaining only patterns does not preserve CoT performance.Source: §5 · p. 7, Table 3 · p. 7

Wrong pattern

  • Wrong patterns leave performance on par with vanilla CoT for GSM-8K, DATE, and SORTING, but SPORTS task solve rate falls from 93.67% to 46.02%.The passage describes the model as robust to specific mistakes in GSM-8K (24.39%), DATE (44.84%), and SORTING (64.80%).Source: §5 · pp. 6–8

Qualitative Analysis of the Role of Patterns

  • In SPORTS, wrong sport activities can disorient the model, which fails to make factual player–activity connections and scores slightly below random.The analysis suggests the model may actively generate incorrect answers in this setting.Source: §5 · p. 8

6. ROLE OF TEXT

Text is necessary for correct conclusions in chain-of-thought prompting, and performance depends on the text’s entities and grammatical form. The experiments show that altering text can impair performance differently across tasks, even when symbols and patterns remain unchanged.

Role of text

  • Text is necessary for correct conclusions, and performance is proportionally sensitive to textual variation; concordant entities in questions and thoughts are crucial to CoT success.The study’s counterfactual prompts retained symbols and patterns while altering text or grammar.Source: §6 · p. 8

Experimental method

  • The text and grammar counterfactuals preserve symbols and patterns, isolating the effects of textual changes.The paper presents a subset of results in Table 4 and defers further analysis and results to Appendix H.Source: §6 · p. 8

Discrepant entities

  • Changing thought entities makes SPORTS perform similarly to DIRECT, while GSM-8K improves over DIRECT but remains below CoT.Semantically similar substitutions can preserve meaningful prompts and elicit analogical thoughts in GSM-8K; in SPORTS, substitutions undermine factual correctness and may confuse the model.Source: §6 · p. 8

Altered grammatical style

  • Yodish thought prompts moderately harm GSM-8K solve rates and drastically harm SPORTS and DATE; greater text randomness also worsens performance.The paper attributes the task differences to how tightly text and patterns are intertwined: answers derive from text in SPORTS and DATE, whereas GSM-8K patterns are more explicit.Source: §6 · p. 9

7. SYMBIOSIS BETWEEN TEXT AND PATTERNS

The section argues that effective chain-of-thought prompting depends on a symbiosis: patterns guide meaningful intermediate text, while text supplies patterns with commonsense knowledge and task meaning.

7. SYMBIOSIS BETWEEN TEXT AND PATTERNS

  • For GSM-8K, questions solved exclusively by CoT averaged 3.98 entities versus 3.62 overall, a significant difference (p = 0.04).The authors suggest CoT helps more when questions contain more patterns and require more intermediate steps; patterns also help prevent premature conclusions and omission of symbols.Source: §7 · p. 9
  • CoT thoughts can elicit commonsense knowledge, including animal-leg counts and the date of Christmas Eve, that supports the answer.These examples illustrate text supplying semantic knowledge for GSM-8K and DATE tasks.Source: §7 · p. 9
  • In SPORTS, activities for which CoT alone was correct averaged 68k Google results versus 399k across the dataset, while patterns-only correctness required 3,575k-result activities.The authors conjecture that intertwined text and patterns help CoT recover commonsense about rarer activities, whereas patterns-only prompting succeeds when activity popularity is high.Source: Table 7 · p. 12, §7 · p. 12
  • Patterns prompt and structure meaningful intermediate text, while text imbues patterns with knowledge and meaning that help solve tasks.Thoughts glue the two together: patterns guide connections between clauses, and text helps elicit commonsense, such as distinguishing four-legged from two-legged animals.Source: §7 · p. 12

8. CCOT: CONCISE CHAIN OF THOUGHT

CCOT is designed to retain the information needed for factual answers while removing ineffectual tokens; it outperforms COT with fewer prompt tokens.

8. CCOT: CONCISE CHAIN OF THOUGHT

  • For all tasks except GSM-8K, CCOT keeps the examples identical and rephrases thoughts to use fewer tokens; for GSM-8K, it uses randomly selected training questions with shorter thoughts.The authors report no systematic mechanism for shortening GSM-8K thoughts.Source: §8 · p. 12
  • CCOT aims to reduce prompt noise and generated-token counts, potentially improving task solve rates while lowering inference runtime and carbon footprint.The proposed prompts distill the hypothesized minimum information needed to drive the model to factual conclusions.Source: §8 · p. 12
  • CCOT outperforms COT while using fewer prompt tokens, and its task solve rate remains relatively high at PaLM-540B.Table 8 compares task solve rates for PaLM-62B and PaLM-540B.Source: Table 8 · p. 13, §8 · p. 13
  • On average, CCOT reduces input tokens (1.39ˆ) and output tokens (1.58ˆ).Source: §8 · p. 13

9. RELATED WORK

The related work places this study alongside research on few-shot examples, chain-of-thought prompting, prompt design, and model explanations. It emphasizes the study’s analysis of prompt components and attention, alongside its focus on standard single-output CoT.

Exploring the role of examples in few-shot setup

  • Building on Min et al. (2022) and Kim et al. (2022), this study extends label-correctness analyses by separating placeholder symbols, patterns, and text and examining attention alongside outcomes.Label correctness is immaterial for GSM-8K but matters more for SPORTS, and wrong examples can sometimes improve performance by better indicating the task.Source: §9 · p. 13
  • SPORTS results align with Razeghi et al. (2022) on DIRECT suiting easy web-grounded questions, while SORTING results support Xie et al. (2021)’s shared-concept account; text-pattern synergy aids instruction inference.Source: §9 · p. 13

Least to most prompting

  • Zhou et al. (2022) generate subquestions, answer them, then combine them, whereas Kojima et al. (2022) generate rationales directly; the paper links least-to-most’s strength to meaningful CoT substeps.Source: §9 · pp. 13–14

Rationale generation as an intermediate step

Prompt selection

  • Unlike prompt-selection methods, which alter prompt construction or example order, CoT is relatively robust to minor prompt-design changes, making this study’s findings potentially more general.Prompt-selection approaches include question-conditioned prompts, list or question formats, reordered examples, and task instructions.Source: §9 · p. 14

Explaining model behavior using counterfactual prompts and attention

  • The study uses counterfactual prompts and attention to interpret outputs and gain insight into model behavior, while acknowledging counterfactual artifacts and debate over attention as an explanation.Counterfactual inputs with consistent, systematic performance changes are more likely to reflect model behavior; attention adds intuition and insight to the empirical findings.Source: §9 · p. 14

10. CONCLUSIONS

The study uses controlled counterfactual experiments to evaluate chain-of-thought reasoning in PaLM, GPT-3, and CODEX. Its conclusions emphasize the symbiosis between text and patterns, with text helping models extract semantic patterns to shape correct answers.

  • The study evaluates chain of thought’s capacity to elevate complex reasoning in PaLM, GPT-3, and CODEX using controlled counterfactual experiments.Source: §10 · p. 14
  • The results suggest that the symbiosis between text and patterns bears substantial weight in chain-of-thought reasoning.Source: §10 · p. 15
  • Text serves as a channel for extracting semantic patterns that help models form correct answers.Source: §10 · p. 15

REPRODUCIBILITY STATEMENT

The paper supports reproducibility through controls for experimental variability, released code and materials, and documentation of model access and generated outputs.

  • Experiments used multiple random seeds, reported average and standard deviation, tested prompt differences with McNemar’s test, and measured model-output agreement with Cohen’s kappa.The tests account for example-order randomness and compare outputs from different models.Source: REPRODUCIBILITY STATEMENT · p. 16
  • The authors released code and provided one-click scripts to reproduce results for the paper’s publicly available models.The code is available in the project’s Google Research repository.Source: REPRODUCIBILITY STATEMENT · p. 16
  • PaLM was unavailable at submission, but the code was compatible with OpenAI API v0.23.0; CODEX was free to use, supporting reproduction with publicly available models.The experiments used PaLM, GPT-3 (text-davinci-002), and CODEX (code-davinci-002).Source: REPRODUCIBILITY STATEMENT · p. 16
  • Prompts and GPT-3 and CODEX outputs were provided, with each output recording the question, generated answer, and true answer.Output files use the naming convention task_name_model_name_sseed.jsonl.Source: REPRODUCIBILITY STATEMENT · p. 16

ETHICS STATEMENT … F.3. ATTENTION ANALYSIS

This appendix contextualizes the counterfactual study’s scope and limitations, extends the symbol experiments with prompt and output comparisons, and reports attention analyses, cross-model robustness, and PaLM-62B statistical tests. Together, these analyses examine how symbols relate to chain-of-thought gains and support the study’s findings.

ETHICS STATEMENT

  • The study identifies systematic few-shot model behaviors, while acknowledging potential societal harms and the environmental cost of its compute.The authors argue few-shot prompting may reduce environmental impact by avoiding additional training and dataset collection, but uncontrolled advances in machine reasoning could aid bad actors; they hope the compute used will promote positive environmental outcomes.Source: ETHICS STATEMENT · p. 17

A. EXTENDED BACKGROUND

A.1. LIMITATIONS

  • Counterfactual findings may be misleading because of artifacts or spurious correlations, and may not generalize beyond the studied tasks, datasets, or models.The study covers a subset of math, commonsense, and symbolic tasks; its models are not fully documented, including on pretraining data, size, and biases.Source: §A.1 · pp. 25–26

B. DETAILS ON STUDIED REASONING TASKS; C. COMPUTATIONAL RESOURCES AND MODELS

D. ATTENTION ANALYSIS; D.1. PER-LAYER ATTENTION ANALYSIS

  • PaLM attention spectra for standard and symbolic COT prompts were identical across layers, indicating the averaged result was not a spurious correlation.The analysis uses attention from inference-question tokens to prompt tokens and zeros values below the 10th-largest attention value; GPT-3 and CODEX attention ablations were unavailable through their APIs.Source: D. ATTENTION ANALYSIS · pp. 27–28, §D.1 · p. 28, Figure 5 · p. 29

D.2. SPECIALIZED ATTENTION HEADS

  • Some PaLM head-layer pairs favored past- or future-tense tokens, while attention to patterns and text was approximately equal overall.The figure reports patterns receiving slightly higher attention across most layers, consistent with the prose finding that both contribute similarly to COT success.Source: §D.2 · pp. 29–30, Figure 11 · p. 32

E.1. RESULTS ON CODEX, GPT-3, PALM-540B

  • Across CODEX, GPT-3, PaLM-62B, and PaLM-540B, findings hold across tasks, and CCOT matches or outperforms COT while being 20% shorter.The reported findings are that pattern correctness is immaterial, abstract and out-of-distribution symbols remain helpful, and sensitivity to text increases with its randomness; some GPT-3 and CODEX variations were limited by usage or rate limits.Source: Table 11 · p. 33, Table 12 · p. 33, §E.1 · p. 33

E.2. SIGNIFICANCE TESTS FOR PALM-62B

  • PaLM-62B experiments were repeated with three seeds; McNemar’s test assessed performance differences against COT(p), and Cohen’s kappa measured output agreement.The tables report p-values and κ for GSM-8K, DATE, SPORTS, and SORTING; their captions interpret p > 0.01 as not rejecting the null and κ above 0.4 or 0.6 as moderate or substantial agreement, respectively.Source: §E.2 · p. 35, Table 15 · p. 35, Table 16 · p. 36

F.1. COUNTERFACTUAL PROMPTS FOR SYMBOLS

  • Abstract placeholders have little performance impact, while out-of-distribution symbols usually preserve results; larger SORTING integers raise task solve rate from 60.6% to 80.0%.For SPORTS, artificial names and activities close the gap with DIRECT, though performance remains marginally better; vague, ungrammatical substitutions reduce solve rate to 52.96%.Source: §F.1 · pp. 39–40
  • Removing symbols drops GSM-8K solve rates to 11.97 (DIRECT = 9.6) and DATE to 29.95 (DIRECT = 31.6).The authors conclude that without symbols, the model effectively neglects the semantic value of prompts.Source: §F.1 · p. 40

F.2. ROLE OF SYMBOLS: OUTPUT ANALYSIS

  • Cohen’s κ agreement between COT and symbolic counterfactual prompts is moderate (>0.4) to substantial (>0.61), versus meager (<0.2) agreement between DIRECT and those prompts.This supports the finding that model behavior can remain similar despite changes to symbol type or value.Source: §F.2 · p. 40
  • For GSM-8K questions containing decimals, p and Csymb_oodppq each solve 50 of 124 questions, but p solves 18 of the 24 exclusive cases versus six for Csymb_oodppq.The prompts did not restore correct answers preferentially on decimal questions.Source: §F.2 · p. 40

F.3. ATTENTION ANALYSIS

  • Vanilla COT and abstract-symbol prompts show near-identical attention patterns, with similar tokens receiving similar attention scores.The appendix also reports no-symbol accuracy drops to 29.94% for DATE and 11.98% for GSM-8K.Source: §F.3 · pp. 40–41

F.4. ROLE OF DATASET FOR EFFECTIVE CHAIN OF THOUGHT · F.5. ADDITIONAL EXPERIMENTS FOR SPORTS

The appendix shows that CoT’s math performance is sensitive to number magnitude, while sports prompts work best when the model can retain or infer the needed patterns and text. The findings highlight limits of number-based reasoning and the importance of extracting information absent from the question.

F.4. ROLE OF DATASET FOR EFFECTIVE CHAIN OF THOUGHT

  • As Cobbe et al. (2021) observed, math ability is largely limited to simple operations on small numbers, and single-digit numbers make up 50% of GSM-8K digits.The authors suggest that commonly occurring digit values help explain models’ success on math word problems.Source: §F.4 · pp. 41, 43, Figure 14 · p. 43
  • For a candle-arithmetic question, vanilla CoT and an abstract-symbol prompt both answer correctly and attend to relatively similar tokens, suggesting the vanilla prompt supplies effective patterns.The attention comparison uses the 0th layer averaged over heads; the authors report the same phenomenon in higher layers.Source: Figure 13 · p. 42
  • As magnitude increases, accuracy drops from 100% to 12% (a 10x loss); even PaLM-540B reaches only 16%, while calculator-evaluated final outputs still reach 30% (a 3x performance reduction).Source: §F.4 · pp. 42–43

F.5. ADDITIONAL EXPERIMENTS FOR SPORTS

  • In SPORTS, CoT substantially beats no-thought and direct baselines, relation-extraction thoughts outperform CoT, and linguistic patterns matter.Table 20 also reports that abstracting PERSON or ACTIVITY does not reduce performance, abstracting sports substantially decreases it, and abstracting all information performs worst.Source: Table 20 · p. 43

F.4. ROLE OF DATASET FOR EFFECTIVE CHAIN OF THOUGHT

  • Changing the number distribution in the prompt has no bearing on performance, despite models’ limited ability to do math.The authors refer to results in Section 4 for this finding.Source: §F.4 · p. 43

F.5. ADDITIONAL EXPERIMENTS FOR SPORTS

  • Replacing PERSON or ACTIVITY has negligible impact, but abstracting sports drops performance below DIRECT levels.PERSON and ACTIVITY are present in the question and copied in the thought, whereas SPORT is not present and must be extracted.Source: §F.5 · pp. 43–44
  • The sports examples show that effective CoT requires both the right pattern and text: abstracting a SPORT can produce an incorrect answer, while abstracting PERSON or ACTIVITY preserves correct inference.For the blocking-foul example, the abstract-sports prompt generates conflicting SPORT1 and SPORT2 labels and answers no; person/activity prompts infer basketball and answer yes.Source: §F.5 · p. 44
  • In a few cases, forcing the model to generate text leads to errors caused primarily by faulty connections.The paper points to Table 37 for examples.Source: §F.5 · p. 44

G. ADDITIONAL EXPERIMENTS ON ROLE OF PATTERNS IN COT · G.1. OUTPUT ANALYSIS

The experiments find that patterns in thoughts are important to CoT, but their factual accuracy is usually not; removing them or using patterns alone undermines performance. Output analyses show that wrong or diverse patterns can affect task understanding differently across tasks.

G. ADDITIONAL EXPERIMENTS ON ROLE OF PATTERNS IN COT

G.1. OUTPUT ANALYSIS

  • In SPORTS, wrong thoughts can drive accuracy below 50%, versus 72.1% with DIRECT, and the model may generate incorrect answers.Its thoughts remain structurally valid despite factual errors: among discordant samples, six misassociate either the player or activity, one misassociates both, and only two associate all correctly.Source: §G.1 · p. 47, Table 23 · p. 49
  • In SORTING, wrong-pattern prompts score 46.0% vs. 64.8% for DIRECT, and planned noise can help the model recognize the sorting task rather than generate a familiar integer sequence.The passage describes the wrong-pattern variant as outperforming DIRECT, despite listing the values in that order.Source: §G.1 · pp. 47–48
  • In the SORTING case study, mixing thoughts whose number sequences start with “1” and “2” achieved an 87.78% task solve rate, while restricting thoughts to either start yielded at most 40%.Source: §G.1 · pp. 47–48

H. ADDITIONAL EXPERIMENTS ON ROLE OF TEXT … K. COMPLETE LIST OF COUNTERFACTUAL PROMPTS

This section presents additional analyses of how text and patterns support chain-of-thought prompting, including prompt-length findings, and inventories counterfactual prompt examples across GSM-8K, SPORTS, DATE, and SORTING. The prompt examples show variations in content and form without reporting experimental outcomes.

H. ADDITIONAL EXPERIMENTS ON ROLE OF TEXT

  • Text is necessary for correct conclusions, and performance worsens as text changes, with effects depending on whether question entities and thoughts remain compatible.Replacing thought entities yields better GSM-8K solve rates than DIRECT but does not match COT; semantically meaningful substitutions can preserve useful abstract reasoning, whereas random thoughts and shuffled text harm performance. No text counterfactuals were run for SORTING.Source: H. ADDITIONAL EXPERIMENTS ON ROLE OF… · pp. 51, 53

H.1. TEXT WITH ALTERED GRAMMATICAL STYLE

  • Changing only question grammar has little impact, while changing thought grammar can hurt performance, especially when text and patterns are intertwined in SPORTS and DATE.The authors report moderate effects on GSM-8K and drastic negative effects on SPORTS and DATE when thoughts use Yodish; DATE’s complex time-and-date relations make factual thought generation especially difficult under altered structure.Source: §H.1 · p. 53

I. SYMBIOSIS BETWEEN TEXT AND PATTERNS: ADDITIONAL EXAMPLES AND QUALITATIVE ANALYSIS

  • The qualitative analysis frames text and patterns as symbiotic: thoughts purposefully glue them together, while either component alone is insufficient for CoT to thrive.The section presents examples to clarify how this relationship supports effective intermediate thoughts.Source: §I · p. 61

I.1. CONSTRUCTING EFFECTIVE INTERMEDIATE THOUGHTS

  • On GSM-8K, COT(p) exclusively solves 140 questions versus 71 for Cpat_none and 33 for Cpat_only, and those questions average 3.98 numerical entities versus 3.62 overall (p = 0.04).The authors attribute the advantage on questions with more patterns to longer, more detailed intermediate steps; patterns help sustain meaningful structures and prevent premature conclusions or missed symbols.Source: §I.1 · pp. 61–62

I.2. COMMONSENSE EXTRACTION

  • Text helps CoT elicit commonsense knowledge, while patterns help the model connect generated clauses into factually meaningful conclusions.Examples include deriving animal-leg counts and dates from commonsense, and using patterns to connect a player’s sport with an activity in SPORTS.Source: §I.2 · pp. 62–63

I.3. TEXT AND PATTERNS: IT TAKES TWO TO TANGO

  • Patterns guide meaningful intermediate text and factual connections, while explicit text gives patterns the knowledge and meaning needed to solve tasks.The paper illustrates text’s role with animal properties in GSM-8K, sports identity in SPORTS, and date knowledge in DATE.Source: §I.3 · p. 63

J.2. REDUCTION IN LENGTH BY CCOT

  • CCOT outperforms COT overall while generally using fewer prompt tokens, reducing average input and output tokens by 1.39× and 1.58×, respectively.For GSM-8K, the authors could not systematically shorten thoughts and instead sampled questions with shorter thoughts; they report that CCOT remains relatively effective at PaLM-540B scale.Source: §J.2 · pp. 63, 67, Table 36 · p. 67

K. COMPLETE LIST OF COUNTERFACTUAL PROMPTS

  • The appendix compiles the counterfactual prompts used across GSM-8K, DATE, SPORTS, and SORTING, including variants with abstract values and altered or incorrect content.The listed prompt sets include abstract dates, replaced numbers, abstract sports entities, and wrong-answer variants.Source: K. COMPLETE LIST OF COUNTERFACTUAL… · p. 68, Table 38 · p. 69, Table 39 · p. 70
  • The appendix lists task-specific prompts that remove or restrict patterns, equations, or thoughts, and prompts with wrong or random thoughts.The listed examples cover GSM-8K, SPORTS, DATE, and SORTING.Source: Table 57 · p. 88, Table 58 · p. 89, Table 59 · p. 90
  • Other listed variants alter entities, language, or numerical expression, including same-type replacements, YodaSpeak, and verbalized numbers.These examples span GSM-8K, SPORTS, and DATE prompts.Source: Table 68 · p. 99, Table 69 · p. 100, Table 70 · p. 101
  • The inventory also includes shuffled tokens or sentences, reordered clauses, and verbose or abstract prompt variants.The captions specify token shuffling within sentences, sentence shuffling within thoughts, and fixed number locations for some GSM-8K variants.Source: Table 79 · p. 110, Table 80 · p. 111, Table 82 · p. 113

Related papers

Loading 2209.07686v2…