Over-Reasoning and Redundant Calculation of Large Language Models

1 Introduction

• Although CoT can improve performance, lengthy answers increase inference latency and commercial costs and may make simple questions seem difficult to users.The paper argues that LLMs should use CoT when it is necessary, while noting that whether they do so remains unclear.Source: §1 · p. 1
• Prior CoT research examined reasoning correctness and faithfulness, but the paper identifies redundancy as an unstudied question.It asks whether LLMs generate unnecessary reasoning when a question clearly needs none.Source: §1 · p. 1
• LLMs generate redundant calculations on math questions answerable without calculation, sometimes complicating responses and leading to wrong answers.The study evaluates seven LLMs trained with reinforcement learning from human feedback.Source: §1 · pp. 1–2
• GPT-4 and ChatGPT show a strong preference for long answers containing redundant calculations, even when those answers are incorrect.The paper proposes that imperfect reward models preferring longer answers regardless of correctness may contribute to LLMs’ tendency toward lengthy outputs.Source: §1 · p. 2
• GSM8K-Zero is introduced as a dataset for studying redundant reasoning, with the authors releasing it to reveal this tendency in LLM outputs.The paper describes its work as an initial study of output redundancy.Source: §1 · p. 2

2.1 Construction of GSM8K-Zero

• The construction removes GSM8K’s final sentence asking for an unknown quantity and replaces it with a question about a known variable.The new question is generated using few-shot prompting with ChatGPT, and GPT-4 is used to check the answer against the selected ground truth.Source: §2.1 · p. 2
• From 3,500 randomly selected GSM8K training questions, the procedure yields 2,978 question-answer pairs.Questions whose GPT-4 answer differs from the selected ground truth are discarded.Source: §2.1 · p. 2
• A manual inspection of 250 random pairs estimates that about 85% of GSM8K-Zero question-answer pairs are valid.The authors describe this as an estimate based on their inspection.Source: §2.1 · p. 2

2.2 Evaluating Redundancy

• Redundant output is superfluous response information not required for an accurate answer; on GSM8K-Zero, calculations are flagged when mathematical operators ×, +, or = appear.The dataset makes this test possible because answers are explicitly stated in the questions and require no calculations.Source: §2.2 · p. 2

3 Experiments

• The experiments use zero-shot prompts across proprietary and open-source models, sampling one response per question; preliminary results were robust to sampling hyperparameters.Source: §3 · p. 2

3.1 Main Results

• Almost half of the tested LLMs achieve below 50% accuracy on GSM8K-Zero, whose answers can be extracted directly from the questions.Source: §3.1 · p. 3
• ChatGPT includes unnecessary calculations in almost half its answers, while every tested Llama-2 model does so in more than 80% of responses without being prompted.Source: §3.1 · p. 3
• For almost all models except PaLM, answers containing calculations have significantly lower accuracy than answers without calculations.Source: §3.1 · p. 3
• PaLM often states the numeric answer before its calculations, so later incorrect calculations do not affect the answer and accuracy is similar with or without them.Source: §3.1 · p. 3

3.2 Do LLMs Know When to Use CoT?

• When explicitly allowed to omit step-by-step reasoning, models reduce redundancy and generally improve accuracy, but redundancy remains above 25%.Source: §3.2 · p. 4

• The authors use ChatGPT and GPT-4 as proxy reward models because they cannot access the reward models used to train ChatGPT or Llama models.The proxy models compare long and short answers for accuracy and helpfulness, with answer order inverted to counter position bias.Source: §4 · p. 5
• Long answers are ChatGPT responses with redundant calculations, paired with template-based short answers containing the ground-truth answer.The authors review approximately 100 correct and 100 incorrect long answers.Source: §4 · p. 5
• GPT-4 and ChatGPT prefer long answers when both answers are correct; when the long answer is wrong, ChatGPT consistently prefers it, and GPT-4 chooses it in 34% of cases.GPT-4 prefers the short, correct answer in 61% of cases.Source: §4 · p. 5, Figure 2 · p. 5

• The authors construct GSM8K-Zero to illustrate redundant LLM output and report that unnecessary reasoning can sometimes lead to wrong answers.They suggest future work on training techniques to teach models when to think step-by-step.Source: §5 · p. 5

Limitations

• The paper studies redundancy only on the manually constructed GSM8K-Zero dataset, leaving evaluation on other existing datasets to future work.The authors chose GSM8K-Zero because redundancy is easier to define and calculate on it.Source: Limitations · p. 6
• About 85% of GSM8K-Zero question-answer pairs are estimated valid based on review of 250 random pairs, and the authors say their main observations still hold when accounting for noise.The authors advise caution when interpreting results on GSM8K-Zero.Source: Limitations · p. 6
• The authors identify the paper’s short length as a constraint that leaves further questions unexplored.They describe the main content as self-contained and point curious readers to potential questions in Appendix A.Source: Limitations · p. 6

Ethical Statements

• The authors do not see possible harmful outcomes from the work.Source: Ethical Statements · p. 6
• The authors state that they followed the ACL ethical guidelines while conducting the research.Source: Ethical Statements · p. 6

A FAQs

• Alpaca and Vicuna, which are not RLHF-trained, also produce redundant outputs in 40% of cases on GSM8K-Zero.Their outputs are described as messy, making accuracy difficult to calculate with regular expressions, so these results are not reported in the main paper.Source: A FAQs · p. 7
• The FAQ raises whether regular-expression accuracy errors could make proxy reward models prefer long answers, and answers that the reviewed long-answer labels were assured correct or wrong.Source: A FAQs · p. 8
• Authors manually reviewed 100 correct and 100 wrong long answers used in Section 4 to check their labels.They also randomly sampled approximately 100 correct and 100 wrong answers containing calculations for Figure 2 because they could not review all such answers.Source: A FAQs · p. 8

B.1 Dataset Cards

• GSM8K-Zero is constructed from GSM8K, but the source dataset does not include a license, leaving the release license uncertain.Source: §B.1 · p. 8

B.2 Manual Review by the Authors

• One author assigned each reviewed sample to one of four categories covering correct or incorrect ground truths and answers, or an invalid question.Invalidity included an incorrect ground truth, a question requiring calculation, or ambiguity.Source: §B.2 · p. 8
• The authors estimate that 14.8% of GSM8K-Zero questions are invalid, accounting for ChatGPT’s roughly 80% accuracy.The estimate combines the reviewed correct and incorrect answer groups.Source: §B.2 · p. 8
• Among questions ChatGPT answered correctly, 89% were valid and all had correct ground truths; among inaccurate answers, about 70% were valid and 2% had wrong ground truths.Source: §B.2 · p. 8
• Regular-expression parsing falsely counted wrong ChatGPT answers as correct in 7% of the correct-answer group, while it falsely counted correct answers as wrong in 4% of the inaccurate-answer group.Source: §B.2 · p. 8

B.2.1 Does Invalid Questions Affect the Results?

• If models generated CoT reasoning and calculations for invalid questions, redundancy should be around 15%; all models except GPT-4 exceed that, so they also generate redundant calculations for valid questions.GPT-4 also generated redundant calculations in some cases; Table 3, item 2 is one example.Source: §B.2.1 · p. 8
• If all valid questions were answered correctly, accuracy would be about 85%; all models except Claude-2 fall short of that level.Source: §B.2.1 · p. 8
• Even if all invalid samples contained calculations, answers with calculations would still be less accurate than answers without them.Source: §B.2.1 · p. 9

C Prompts

• The generation prompt requires questions to ask for a number already stated in the source sentence, disallowing even simple calculation.The template fills in the known information and ground-truth answer from the original question.Source: C Prompts · p. 9

Prompts for using ChatGPT and GPT-4 as the proxy in Section 4

• The proxy-judging prompt asks ChatGPT or GPT-4 to compare two assistants on response quality and return a prescribed A, B, or tie verdict.Source: Prompts for using ChatGPT and GPT-4 as… · p. 9

D Sampling parameters of LLMs

• For LLM answer generation, the authors used temperature 0.7 and default settings for other parameters; Llama-2 ran through Huggingface Transformers.Source: D Sampling parameters of LLMs · p. 9