Source-linked AI summary
Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost
Sania Nayab, Giulio Rossolini, Marco Simoni, Andrea Saracino, Giorgio Buttazzo, Nicolamaria Manes, Fabrizio Giacomelli
TL;DR
Long CoT outputs can increase generation time and raise concerns about conciseness, motivating length-aware evaluation and ways to control reasoning length. The paper proposes correctness-and-conciseness metrics, CCoT, and scores for redundancy and information flow; experiments report an accuracy-efficiency trade-off, especially for large models.
Problem
Method
The paper introduces HCA, SCA, and CCA to account for output length in correctness evaluation, CCoT to constrain reasoning length, and scores for redundancy and information flow.
Source: §1 · p. 1, §7.4 · p. 10Results
Across experiments, CCoT highlights the accuracy-efficiency trade-off, particularly for Falcon-40b and Llama2-70b; on three LLaMA2 datasets, CCoT30 raises average accuracy 4.41% and reduces computational costs 5.12s.
Source: §1 · p. 2, §8 · p. 12Takeaways & Limitations
The paper presents CCoT as a simple strategy to enhance conciseness and potentially support more predictable and efficient LLMs.
Source: §8 · p. 12Smaller models may struggle to improve the accuracy-efficiency trade-off and may produce incorrect answers when reasoning length is limited.
Source: §8 · p. 11
Abstract
from arXiv · showhide
Today's large language models (LLMs) can solve challenging question-answering tasks, and prompt engineering techniques, such as chain-of-thought (CoT), have gained attention for enhancing the explanation and correctness of outputs. However, many models and techniques tend to produce excessively verbose and lengthy answers, leading to issues with both conciseness and generation time. To address this, this paper analyzes the impact of output lengths on LLM inference pipelines by introducing and proposing novel metrics to evaluate the \textit{correct conciseness} of a model and related prompting techniques. Then, we examine the impact of controlling output length through a refined prompt engineering strategy, Constrained-CoT (CCoT), which encourages the model to produce more concise outputs. To better understand the effects of such a prompt, we also introduce two additional scores for analyzing the conciseness, measured in terms of redundancy and information flow in generated answers. Experiments on pretrained LLMs and multiple datasets demonstrate the benefits of the proposed metrics and the effectiveness of CCoT across different models.
1 Introduction
The introduction argues that CoT can improve answers while producing lengthy outputs that raise generation-time and conciseness concerns. It proposes length-aware correctness metrics, Constrained-CoT (CCoT), and redundancy and information-flow scores; experiments report accuracy and cost benefits for CCoT.
1 Introduction
- The paper introduces HCA, SCA, and CCA to assess answer correctness while accounting for output length as an indicator of computational cost.The metrics reweight accuracy with output-length considerations that affect inference time and its predictability.Source: §1 · pp. 1–2
- CCoT explicitly limits reasoning length and offers a flexible constraint that lets users tune the strictness of brevity.Its stated objective is concise, computationally efficient output without sacrificing accuracy.Source: §1 · pp. 1–2
- On three datasets, LLaMA2 with CCoT30 raises average accuracy 4.41% and reduces computational costs by 5.12s; the metrics improve 10% for HCA, 9% for SCA, and 9% for CCA.The reasoning-length constraint is 30 words.Source: §1 · p. 2
- Additional scores analyze redundancy and information flow, while experiments across models and datasets report CCoT benefits in inference time, accuracy, and conciseness relative to CoT.The paper describes these scores as helping assess conciseness and the effects of constraining output length.Source: §1 · p. 2
2 Related work
The related-work discussion identifies limited attention to conciseness, inference cost, and response-time predictability in metrics and benchmarks focused on correctness. The paper responds with length-aware correctness metrics, redundancy and information-flow scores, and CCoT.
2 Related work
- Earlier work discussed longer explanations and possible hallucinations, while prompting approaches aimed at accuracy could add context and detail that make concise, factual responses challenging.The related-work section also describes multi-hop processing that extracts a rationale from an input prompt.Source: §2 · p. 2
- Prior metrics and benchmarks largely focus on correctness, while cost-focused work does not address conciseness; the paper identifies gaps in cost, predictability, unified evaluation, and conciseness analysis.It also notes accuracy-oriented prompting can produce longer outputs, and CoT responses tend to be lengthy and increase generation time.Source: §2 · pp. 2–3
- The paper proposes metrics that jointly account for conciseness and correctness, plus embedding-based scores that examine reasoning-step redundancy and information flow.It also introduces CCoT to guide reasoning length while aiming to preserve answer quality and improve inference time.Source: §2 · p. 3
3 Motivations
The motivations section links output length to generation time through autoregressive decoding, where each token is generated using earlier tokens. Preliminary Falcon-40B tests on GSM8K show longer outputs and generation times with CoT.
3 Motivations
- Each generated token depends on previously generated tokens, so longer answers require more decoder invocations and take longer to generate.The paper models the output as a token list generated from the input representation and preceding output tokens.Source: §3 · p. 3
- In preliminary Falcon-40B tests, CoT significantly increased output length and generation time on 100 random GSM8K questions.Figure 1’s top plot relates response time to output length with and without CoT; its bottom plot compares output words for 50 random questions.Source: §3 · p. 4, Figure 1 · p. 4
- The paper motivates attention to length because longer CoT outputs can raise response time and reduce conciseness and precision, concerns for interactive use.It describes the preliminary comparison as evidence that the time cost of CoT merits attention alongside its correctness benefits.Source: §3 · p. 4
4 Metrics for Correct Conciseness
The section defines three metrics that extend accuracy evaluation with output-length penalties and, for CCA, response-length variability. Together, they allow evaluation of correctness, conciseness, and consistency of response lengths.
4 Metrics for Correct Conciseness
- The metrics begin with correctness determined by whether a post-processed answer matches the ground truth, then multiply correctness by a penalty that decreases for long outputs.The post-processing function is user-defined, and the penalty lies between 0 and 1.Source: §4 · p. 4
- HCA measures the fraction of correct outputs within a user-specified length k and excludes answers that exceed the maximum.The paper describes it as useful where strict length adherence matters, such as real-time systems or limited-resource environments.Source: §4 · p. 4
- SCA extends HCA by exponentially penalizing correct answers longer than k, with user-defined decay α controlling the tolerance.At α = 0, SCA reduces to HCA.Source: §4 · p. 5
- CCA further accounts for variation across output lengths; its tolerance parameter β controls the penalty based on standard deviation σ.The metric targets consistent response lengths, which the paper relates to inference-time predictability.Source: §4 · p. 5
5 CCoT Prompting
Constrained-CoT (CCoT) adds a maximum-word constraint to chain-of-thought prompting, aiming to preserve reasoning benefits while reducing answer length and generation time.
5 CCoT Prompting
- CCoT adds an explicit maximum-word constraint to the reasoning prompt, encouraging the model to compress its reasoning and answer more quickly.The constraint is a tunable sentence appended to the user question and reasoning prompt.Source: §5 · p. 5
- 67 words in 17.64s for CoT versus 34 words in 11.65s for CCoT with a 45-word constraint; both answers are correct.Source: §5 · p. 5
- The paper evaluates CCoT with various metrics and reports benefits for answer conciseness, redundancy, and information-flow scores.Source: §5 · p. 5
6 Analysis of the conciseness
The section assesses CoT conciseness through reasoning-step count and repeated content, defining redundancy and information-flow scores for comparing answers.
6 Analysis of the conciseness
- The section treats fewer reasoning steps and less repeated information as two ways to assess CoT answer conciseness.Figure 2 contrasts CoT and CCoT answers, showing fewer output words with CCoT while retaining a correct answer; the titles include word counts and inference times.Source: §6 · p. 5, Figure 2 · p. 6
6.1 Redundancy Score
- Redundancy measures syntactical overlap between answer steps; the score averages each step’s overlap with the others using matching-subsequence ratios.The analysis assumes high redundancy indicates repeated information rather than progress in the reasoning.Source: §6 · p. 6, §6.1 · p. 6
6.2 Information Flow
- Information Flow measures how much semantic content persists between consecutive steps, using BERTScore for similarity and assigning the final step a score of zero.When answers have similar step counts, the paper assumes higher information flow tends to accompany less concise answers because steps reuse details and context.Source: §6.2 · p. 6
7 Experiments
The experiments compare CCoT with base and CoT prompts on three arithmetic-reasoning datasets, testing accuracy, generation time, correct conciseness, reasoning-step redundancy and information flow, and length control. Across the reported analyses, CCoT generally improves the accuracy–efficiency trade-off and produces more concise outputs, though results depend on model, dataset, and length constraint.
7.1 Experimental setup
- Both models were tested on GSM8K, SVAMP, and ASDIV using base, CoT, and CCoT prompts; the setup used Falcon-40b-instruct and Llama2-70b-chat.GSM8K had over 1.3k test problems, SVAMP 300, and ASDIV 1.22k; experiments ran with TGI on 8 NVIDIA A100 GPUs.Source: §7.1 · p. 7
7.2 Impact of CCoT on Accuracy and Efficiency
- Across datasets, CCoT generally reduced generation time while improving or minimally impacting accuracy; Falcon-40b accuracy gains were absent on complex GSM8K.For Llama2-70b on GSM8K, generation time fell from 30.09 seconds with CoT to 23.86 seconds with CCoT-100, while accuracy rose from 36% to 41.77%.Source: §7.2 · p. 7, Figure 3 · p. 8
7.3 Analysis of the correct conciseness
- For both models, CCoT outperformed base and CoT on HCA across k values, and strict constraints mitigated CoT’s performance drop at low k.SCA also improved CCoT scores over base and CoT at smaller k, where tolerance α = 10 accepts some correct answers longer than k.Source: §7.3 · p. 8, Table 1 · p. 9, Table 2 · p. 9
- CCA improved over base and CoT for both models, but scores tended to decrease under high CCoT length constraints such as 100.The authors attribute this decrease to greater freedom to generate outputs with higher length variation.Source: §7.3 · pp. 8–9, Table 1 · p. 9, Table 2 · p. 9
7.4 Understanding the effect of CCoT
- CCoT produced fewer reasoning steps than CoT for Falcon-40b across datasets and for Llama2-70b on ASDIV; Llama2-70b step counts were similar on GSM8K and SVAMP.The comparison plots answer percentages by reasoning-step count for each model and dataset.Source: §7.4 · p. 9, Figure 4 · p. 10
- For Llama2-70b on GSM8K and SVAMP, CoT redundancy was higher than CCoT within the answer-distribution interquartile ranges, despite similar step-count ranges.Those ranges were approximately 5–14 steps on GSM8K and 4–12 on SVAMP.Source: Figure 5 · p. 10, §7.4 · p. 10
- CCoT reduced redundancy on both datasets: ORR ranged from 19.77% to 22.72% on SVAMP and from 12.64% to 24.74% on GSM8K.For eight-step answers, CCoT-15 reduced information flow by 26%–41% versus CoT on GSM8K, while CCoT-100 reduced it by 4%–20%.Source: §7.4 · pp. 10–11, Table 3 · p. 11, Table 4 · p. 11
7.5 Ability to control the output length
- CCoT constraints affected output length, but models did not always stay within the requested limit, especially at smaller values such as 15, 30, or 40.Figure 6 summarizes lengths from the 5th to 95th percentiles, with the requested constraint and median and mean lengths marked.Source: §7.5 · p. 11
8 Final Remark and Conclusion
The conclusion presents CCoT and length-sensitive correctness metrics as a unified approach to evaluating and controlling answer conciseness. Experiments support an efficiency–accuracy trade-off for larger models, while smaller models may struggle with constrained reasoning.
8 Final Remark and Conclusion
- Smaller models may produce incorrect answers when reasoning length is constrained, unlike the sufficient-scale models discussed in the conclusion.The paper suggests exploring fine-tuning as a way to address this constraint for smaller models.Source: §8 · pp. 11–12
- The study’s main scope is efficiency; possible conciseness benefits for hallucinations and error propagation remain future research directions.The authors suggest that lower information flow may indicate retention of logically necessary information, but do not establish those broader benefits here.Source: §8 · p. 12
- The paper introduces three metrics for evaluating conciseness and correctness, proposes Constrained Chain-of-Thought to control answer length, and analyzes redundancy and information flow.The authors frame these contributions as a way to assess conciseness and its role in inference time.Source: §8 · p. 12
- Experiments show CCoT’s trade-off between output-length control and accuracy particularly for Falcon-40b and Llama2-70b.The conclusion also reports analysis of output-length control and information within reasoning steps.Source: §8 · pp. 11–12
A.1 Analysis of the RMS
The RMS analysis compares CCoT and CoT answers in cases where CCoT succeeds and CoT fails. Its redundancy trend resembles the broader analysis, including for complex or ambiguous cases.
A.1 Analysis of the RMS
- The RMS plots examine cases where CCoT answers correctly but CoT does not, with Llama2-70b on the left and Falcon-40b on the right.They show answer-distribution intervals from Q1 to Q3 and their overlap for CoT and CCoT.Source: §A.1 · p. 16
- The plots test whether CCoT’s redundancy behavior matches the all-question trend and explore its potential relationship with response accuracy.The passage describes these as purposes of the analysis, rather than reported causal findings.Source: §A.1 · p. 16
- CCoT’s reduction in redundancy for Llama2-70b and reduction in steps for Falcon-40b are consistent with the broader trend and accompany more accurate, concise answers than CoT.The authors say this pattern persists in cases where CoT fails, including complex or ambiguous scenarios.Source: §A.1 · p. 16
A.2 Information Flow
The information-flow analysis finds reduced semantic repetition under CCoT in Llama2-70b answers, with exceptions that vary by dataset and step. Falcon-40b shows contrasting information-flow patterns, and its fixed-step comparison is unbalanced.
A.2 Information Flow
- For GSM8K answers from Llama2-70b with 9 steps, CCoT shows reduced repetition of semantic information at all steps; Table 5 reports information-flow mean values.Source: §A.2 · p. 16, Table 5 · p. 17
- For SVAMP, all CCoTs retain semantic information throughout the steps, while a different behavior is observed at the final step for all CCoTs, as shown in Table 6.Source: §A.2 · p. 16
- Falcon-40b often has higher information-flow scores, suggesting greater semantic repetition; its 8- and 9-step comparison is unbalanced because fewer CCoT answers fall in those groups.Table 8 has no CCoT-15 scores because Falcon-40b produced no answers with 8 or 9 steps under that constraint.Source: §A.2 · pp. 16–17
B Testing CCoT with smaller LLMs.
Tests of CCoT across five LLMs show that constrained prompting is harder for smaller models. Outcomes vary by model and training context, and the paper identifies fine-tuning as a possible direction for improving smaller-model support.
B Testing CCoT with smaller LLMs
- For the model associated with RefinedWeb training, CCoT does not improve accuracy over CoT but remains more accurate than plain prompting while reducing generation time.The authors describe the differing outcomes as potentially related to training data, training approach, model size, and training technique.Source: B Testing CCoT with smaller LLMs · pp. 17–18
- The authors identify training or fine-tuning for better length handling as a future direction that could benefit smaller models.The proposed direction is not presented as an established result.Source: B Testing CCoT with smaller LLMs · p. 18
- The five-model evaluation reports difficulties with CCoT for smaller models, including higher generation times or incorrect answers under some constraints.The evaluation includes Falcon-7b, Llama2-7b, Falcon-40b, Llama2-70b, and Vicuna-13b.Source: B Testing CCoT with smaller LLMs · pp. 17–18, Figure 10 · p. 18