Chain of Thought Still Thinks Fast: APriCoT Helps with Thinking Slow

• In LLaMa 3.1 8B, CF prompting combined with CoT exacerbates BRP-driven bias rather than mitigating it.The authors report consistent BRP bias in this model, in line with findings for other models.Source: Introduction · p. 1
• APriCoT extends CoT prompting by forcing the model to reason down multiple semantic directions.The approach addresses the concern that CoT may traverse semantic space without sufficient priming.Source: Introduction · p. 1
• APriCoT’s preferred-response distribution is nearly identical to the ground-truth answer distribution, and its accuracy exceeds both CF and CoT.The authors treat evaluation across semantic paths as an estimate of the model’s preferred response.Source: Introduction · p. 1
• The paper extends CF prompting to accommodate CoT, enabling analysis of BRP effects under that reasoning strategy.Source: Introduction · p. 1

• Earlier BRP studies found effects in multiple-choice question answering and tested prompt reformulation or calibration of output probabilities as mitigation strategies.Moore et al. (2024) proposed reformulation; S.-L. Wei et al. (2024) and Zheng et al. (2023) calibrated output probabilities using measured BRPs.Source: Prior Work · p. 1
• The review notes similar undesirable behaviors with CoT; Shaikh et al. (2023) found CoT prone to reveal and exacerbate social biases.Chua et al. (2024) used fine-tuning to mitigate user-induced, rather than inherent, initial behavior biases in a CoT context.Source: Prior Work · p. 1
• Saparov & He (2023) found that CoT models can make strong instantaneous deductive steps but rarely explore multiple semantic paths.The passage describes models as greedy reasoners; APriCoT explicitly attempts to induce multiple pathways.Source: Prior Work · p. 2
• Prior LLM cognition research also examined fan effects, typicality effects, and theory of mind.The cited examples include Roberts, Moore, Fisher, et al. (2024), Misra et al. (2021), and Ullman (2023).Source: Prior Work · p. 2

Background

• MMLU contains 14,042 questions across 57 subjects, and its flexible prompting practices make accuracy comparisons across models difficult.The benchmark allows optional in-context learning up to five shots and CoT reasoning.Source: Background · p. 2
• Cloze testing selects the answer choice whose signifier has the highest probability given the shared task context.The option probabilities are computed independently.Source: Background · p. 2
• CF prompting uses a separate query for each option and selects the option with the highest probability for a shared canary completion.For the multiple-choice example, the canary completion is “correct.”Source: Background · p. 2
• CoT encourages a model to generate freely before answering and is intended to approximate System-2-like processing.Models may still reason greedily, traversing toward pre-existing or induced biases rather than exploring multiple pathways.Source: Background · p. 2

Base-Rate Probability Effects

• BRP is a token’s probability given minimal context, and its effects can make measured behavior sensitive to word preferences rather than task meaning.The passage illustrates that reversing question polarity can reverse results without changing semantic context.Source: Base-Rate Probability Effects · pp. 2–3
• The BRP measurement prompt iterates through every ordering of the answer-choice labels and averages each choice’s measured probability.Source: Table 1 · p. 3

LLaMa-3.1 Token Base-Rate Probability

• The contextual BRP preference ordering for LLaMa 3.1 was A ⪰D ≻B ⪰C.BRPs were measured with cloze prompting adapted to the MMLU task context.Source: LLaMa-3.1 Token Base-Rate Probability · p. 3

Methods: CF+CoT

• CF+CoT and CF used a second-person conversational question-and-choice prompt; CoT used “Let’s think step by step.” and generated up to 100 tokens before the model was queried for an answer using CF prompting.Four copies of the context, one per answer choice, were scored by mean “Yes” probability over 10 trials per question.Source: Methods: CF+CoT · p. 3

Results: CF+CoT

• r2 = 0.93 for CF+CoT, versus r2 = 0.68 for CF, with the stronger BRP correlation under CF+CoT indicating greater influence from prior expectations.Correlations were calculated per subject and aggregated via Fisher-z transformation.Source: Results: CF+CoT · p. 4
• CF+CoT magnified existing answer-choice preferences, diverging from MMLU’s balanced ground-truth distribution; Figure 1 contrasts this skew with APriCoT’s flattening toward ground truth.The model strongly favored C and disfavored A under CF+CoT, and never selected D as its preferred answer.Source: Results: CF+CoT · pp. 3–4, Figure 1 · p. 4

Discussion: CF+CoT

• The authors suggest CF+CoT may produce confirmation bias, while APriCoT’s agnostic traversal directions and path-coherence commentary are associated with decreased bias and increased accuracy.They offer no mechanistic explanation for the negative BRP-choice correlations observed under CF+CoT.Source: Discussion: CF+CoT · p. 4

Counterfactual APriCoT

• APriCoT primes CoT reasoning with a candidate answer and evaluates that same answer independently for each choice.The method was evaluated on the same MMLU question set as CF+CoT, with a 100-token maximum generation limit and 10 iterations.Source: Counterfactual APriCoT · pp. 4–5, Methods: APriCoT · p. 5

Results: APriCoT

• APriCoT’s answer distribution is balanced and close to ground truth; BRP explains r2 = 35% of answer-choice variance, versus r2 = 93% for CF+CoT and r2 = 0.08 for ground truth.Its correlations are also reported as closer to ground truth than CF+CoT’s.Source: Results: APriCoT · p. 5, Figure 2 · p. 5
• The reported MMLU accuracies are substantially below LLaMa 3.1 release results, and the authors lack access to that release’s evaluation methodology.The paper notes that evaluation-method differences may account for the discrepancy.Source: Results: APriCoT · p. 5
• APriCoT reaches 42.79% accuracy, above CF at 37.96% and CF+CoT at 35.56%.Figure 3 reports the same ranking with rounded accuracies: 43%, 38%, and 36%, respectively.Source: Results: APriCoT · p. 5, Figure 3 · p. 6
• For questions answered incorrectly by CF, APriCoT is more likely than CF+CoT to produce a correct answer and less likely to remain incorrect.The same direction holds for questions answered correctly with CF: APriCoT is more likely to stay correct and less likely to become incorrect than CF+CoT.Source: Results: APriCoT · p. 6

Conclusion

• APriCoT significantly reduces, but does not fully eliminate, BRP effects and noticeably improves MMLU performance; CoT prompting exacerbates BRP bias, which the authors link to confirmation-bias-like fast thinking.The authors argue that the CoT effect approximates thinking fast.Source: Conclusion · p. 6

Future Work

• Whether APriCoT transfers to search, planning, cloze testing, or other evaluation methods remains for future work; it is currently defined only for closed-option tasks.The authors note that APriCoT makes no assumptions about evaluation method despite its influence from counterfactual prompting.Source: Future Work · p. 6
• Open questions for future study include mechanistic explanations of confirmation bias, CF overt behavior’s dependence on cloze BRPs, and the nature of LLM reasoning.Source: Future Work · p. 6

Potential Experimental Bias

• The authors caution that biases and assumptions may limit generalizability; experiments were limited to Meta LLaMa 3.1 8B and MMLU, with resources restricting tests to the 8B variant.Open-weight models were chosen to maximize reproducibility, and the authors note results may not generalize to larger variants.Source: Potential Experimental Bias · p. 6

Limitations

• APriCoT increases inference costs by the number of answer choices—four on MMLU—and is intended for model evaluation, not end-user applications.The authors consider a one-time evaluation cost increase justified for a more trustworthy and transparent gauge of model capabilities.Source: Limitations · p. 6

• Experiments used approximately 100 hours on a Google Colab NVIDIA A100 GPU, and the code and data are available on GitHub.Source: Technical Details · p. 6