PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learning

Chenwei Lou, Zewei Sun, Xinnian Liang, Meng Qu, Wei Shen, Wenqi Wang, Yuntao Li, Qingping Yang, Shuangzhi Wu

arXiv:2505.11896v2cs.LGcs.AI

TL;DR

LLMs benefit from CoT on difficult reasoning tasks, but always generating lengthy reasoning wastes tokens on simple queries. AdaCoT uses Pareto-based RL control and Selective Loss Masking to adapt CoT invocation, reducing usage and response length while retaining strong performance, though a small performance gap remains versus always-on CoT.

  • Problem

    CoT improves sophisticated reasoning but generates unnecessary tokens for simple queries, creating deployment-cost and inference-efficiency concerns.

  • Method

    AdaCoT frames CoT triggering as Pareto optimization and uses RL penalty control with Selective Loss Masking to adapt triggering while preventing decision boundary collapse.

  • Results

    AdaCoT reduces CoT triggering to as low as 3.18% and average response tokens by 69.1% in production settings while maintaining strong performance across 15 benchmarks.

  • Takeaways & Limitations

    AdaCoT navigates the Pareto frontier by reducing CoT usage and computational costs for simpler queries while preserving high performance on complex reasoning tasks.

  • Takeaways & Limitations

    A small performance gap persists relative to specialized always-on CoT models, and the binary on/off invocation limits reasoning-depth nuance.

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated remarkable capabilities but often face challenges with tasks requiring sophisticated reasoning. While Chain-of-Thought (CoT) prompting significantly enhances reasoning, it indiscriminately generates lengthy reasoning steps for all queries, leading to substantial computational costs and inefficiency, especially for simpler inputs. To address this critical issue, we introduce AdaCoT (Adaptive Chain-of-Thought), a novel framework enabling LLMs to adaptively decide when to invoke CoT. AdaCoT framed adaptive reasoning as a Pareto optimization problem that seeks to balance model performance with the costs associated with CoT invocation (both frequency and computational overhead). We propose a reinforcement learning (RL) based method, specifically utilizing Proximal Policy Optimization (PPO), to dynamically control the CoT triggering decision boundary by adjusting penalty coefficients, thereby allowing the model to determine CoT necessity based on implicit query complexity. A key technical contribution is Selective Loss Masking (SLM), designed to counteract decision boundary collapse during multi-stage RL training, ensuring robust and stable adaptive triggering. Experimental results demonstrate that AdaCoT successfully navigates the Pareto frontier, achieving substantial reductions in CoT usage for queries not requiring elaborate reasoning. For instance, on our production traffic testset, AdaCoT reduced CoT triggering rates to as low as 3.18\% and decreased average response tokens by 69.06%, while maintaining high performance on complex tasks.

1 Introduction

CoT improves sophisticated reasoning but imposes unnecessary token and deployment costs on simple queries. AdaCoT frames adaptive CoT triggering as Pareto optimization and uses RL to balance response accuracy with inference efficiency.

  • CoT prompting enhances reasoning by having models generate explicit step-by-step reasoning before final answers.
  • Indiscriminate CoT generation increases tokens and deployment costs, even for simple queries that do not benefit from elaborate reasoning.
  • Prior approaches reduce reasoning length or use handcrafted controls but do not adapt adequately to query-complexity variation or formally balance quality against deployment cost.
  • AdaCoT formulates adaptive reasoning as Pareto optimization over response accuracy and deployment cost.
  • AdaCoT uses RL with penalty coefficients to control CoT triggering and explore trade-offs between response accuracy and token expenditure.
  • 3.18% CoT triggering and 69.1% lower average response tokens were achieved in production settings while maintaining strong performance across 15 benchmarks.

2 The AdaCoT Framework

AdaCoT treats CoT invocation as a performance–efficiency trade-off and trains an adaptive policy through SFT followed by multi-stage RL. Its formulation measures performance and CoT usage, while SLM protects adaptive triggering from collapse during biased RL stages.

  • AdaCoT allocates reasoning to complex queries while avoiding unnecessary token overhead for simple queries.
  • The framework casts CoT invocation as Pareto optimization that maximizes response accuracy while minimizing deployment costs.
  • An RL policy assesses query complexity and decides whether to invoke CoT using rewards that combine task performance with token efficiency.
  • CoT triggering rate T(θ) measures the proportion of responses containing explicit reasoning, while P(θ) averages evaluation scores.
  • Training begins with SFT initialization and continues through multi-stage RL to refine adaptive CoT triggering behavior.
  • The RL reward combines response quality with penalties for reasoning omission, reasoning overuse, and format errors; α1 and α2 steer decision boundaries.
  • 2.2.3 Addressing Decision Boundary Collapse with Selective Loss Masking: Decision boundary collapse can arise when specialized RL data has skewed CoT distributions, causing homogeneous always-on or never-on behavior.
  • 2.2.3 Addressing Decision Boundary Collapse with Selective Loss Masking: SLM preserves earlier CoT-triggering distributions by masking the loss contribution of the decision token during distribution-biased RL phases.

3 Experiments

AdaCoT is evaluated across diverse benchmarks, daily-use queries, and production traffic for performance-cost trade-offs, adaptive triggering, and inference efficiency. Results show Pareto improvements, stable triggering with SLM, and substantial production reductions in CoT use.

  • Experimental Setup: Experiments evaluate AdaCoT across 15 diverse open-source benchmarks, daily-use queries, and natural, unfiltered production traffic.The study compares adaptive models with Full CoT and No CoT baselines and evaluates triggering performance across training stages.
  • Pareto Frontier Analysis: 62.8% average score at a 53.3% CoT rate approaches the Full CoT RL baseline’s 65.0% score at 100% CoT usage.AdaCoT RL Exp2 operates with nearly half the CoT usage of the always-on baseline.
  • Pareto Frontier Analysis: AdaCoT RL Exp3 and Exp4 define high-performing adaptive points at 64.3% and 64.4% scores, but remain below the Full CoT RL baseline’s 65.0%.The results show a superior score-to-cost trade-off alongside a slight performance bottleneck at higher triggering rates.
  • Adaptive CoT Triggering Performance and Ablation Studies: SLM preserves adaptive triggering during RL-Math, raising precision to 0.938 and accuracy to 0.813 instead of the no-SLM collapse to Recall=1.0, precision 0.503, and accuracy 0.506.The final RL models then fine-tune the decision boundary through penalty coefficients; Exp2 achieves an F1-score of 0.814.
  • Response Length Reduction and Efficiency Gains: 3.18% CoT triggering on mobile production traffic and 12.50% on PCs substantially reduce usage from the non-adaptive model’s 100%.The reduction translates into substantial computational savings in production settings.

4 Discussion and Future Work

The discussion examines AdaCoT’s design trade-offs, meta-reasoning, and peak-performance behavior. It reports improved triggering and preserved forced-reasoning capability while identifying limits in granularity, generalization, and token efficiency.

  • Design Considerations and Limitations: AdaCoT combines principle-guided initial labeling with RL-based optimization because autonomous trigger learning faces counterfactual and quality-assessment challenges.These challenges are especially relevant when judging omitted CoT benefits and quality degradation on subjective tasks.
  • Design Considerations and Limitations: The framework currently uses binary CoT invocation, requires recalibration across base models, and lacks domain generalization and personalization for verbosity preferences.These scope boundaries limit nuance across reasoning depths, knowledge areas, and user preferences.
  • Meta-Reasoning for Reasoning Decisions: Meta-reasoning raises the daily-use triggering F1-score from 0.750 to 0.840 by having the model assess query complexity before deciding on full CoT.The approach also produced emergent, imperfect user-prompt controllability over CoT invocation.
  • Meta-Reasoning for Reasoning Decisions: Explicit meta-reasoning was not adopted as the default because its assessment step generates additional tokens for every query, including queries that do not trigger full CoT.The paper identifies token-efficient implicit or parallel alternatives as future directions.

5 Related Work

Prior CoT-efficiency work mainly reduces reasoning length or manually controls invocation, whereas AdaCoT targets adaptive, query-dependent triggering within a single model. It frames triggering as Pareto optimization and uses RL to control the decision boundary.

  • Existing CoT-Efficiency Approaches: Existing approaches reduce reasoning length through RL penalties or brevity rewards, post-processing and compression, or explicit instructions and selection mechanisms.These directions address the cost of CoT outputs but primarily focus on shortening reasoning.
  • Gap Addressed by AdaCoT: These methods generally do not equip one model to decide dynamically whether CoT should be invoked based on query nature.Alternative strategies include model merging or manual CoT toggling.
  • Gap Addressed by AdaCoT: AdaCoT fills this gap by framing adaptive triggering as Pareto optimization and using RL to control a single model’s CoT decision boundary.The framework therefore supports context-dependent invocation rather than monotonic reasoning reduction.

6 Conclusion

AdaCoT adaptively controls CoT triggering using Pareto optimization and reinforcement learning, reducing reasoning costs for simpler queries while preserving performance on complex tasks.

  • AdaCoT dynamically controls CoT triggering according to implicit query complexity through an RL-based method with adjustable penalty coefficients.
  • AdaCoT navigates the Pareto frontier by balancing reduced CoT usage and computational costs against performance on complex reasoning tasks.
  • Selective Loss Masking ensures robust adaptive triggering throughout multi-stage RL training.

A Benchmark Dataset Details and Analysis

Across diverse benchmarks, AdaCoT generally increases CoT triggering for reasoning-intensive tasks and reduces it for simpler factual questions. Several adaptive variants approach or exceed always-on CoT performance, although isolated datasets show counter-intuitive results and peak average performance can remain slightly lower.

  • Reasoning-intensive benchmarks: 84.1 score at 39.6% TR on MMLU-Pro shows AdaCoT Exp3 surpassing FullCoT SFT while using substantially less CoT.Exp2 also achieves 83.2 at 27.3% TR; higher triggering generally benefits this more challenging reasoning benchmark.
  • Reasoning-intensive benchmarks: 59.2 score at 61.0% TR on KORBENCH approaches FullCoT SFT with significantly less CoT, showing adaptive behavior on rule-driven reasoning tasks.Exp3 similarly reaches 57.4 at 62.2% TR, while peak performance remains slightly below FullCoT RL.
  • Reasoning-intensive benchmarks: 97.2 score at 68.8% TR on MATH nearly matches the FullCoT RL baseline with about 30% less CoT.Exp2 reaches 95.9 at 52.8% TR, while Exp1 scores 91.7 at 40.9% TR, indicating greater CoT is generally beneficial for MATH.
  • Coding and software benchmarks: 55.9 score at 91.4% TR on LiveCodeBench matches FullCoT RL with slightly less CoT, while Exp2 reaches 50.6 at 83.9% TR.The high trigger rates indicate that AdaCoT applies CoT extensively on coding tasks in this benchmark.
  • Simple factual benchmarks: 0.2%–6.0% TR on Chinese SimpleQA demonstrates infrequent CoT use for simple factual questions, though some AdaCoT RL scores fall slightly below NoCoT SFT.SimpleQA likewise shows little CoT benefit, with Exp4 scoring 9.6 at 20.0% TR and performing worse despite higher triggering.
  • Reasoning-intensive benchmarks: 72.4 score at 93.1% TR on ProcBench significantly surpasses FullCoT RL, indicating effective identification of tasks requiring procedural reasoning.Exp2 and Exp3 also match or exceed the FullCoT RL baseline with high trigger rates.

D Meta-Reasoning Showcase: User Prompt Control

The appendix illustrates how meta-reasoning adapts CoT triggering to user instructions and query complexity, while noting that some historical answers remain speculative. It contrasts elaborate reasoning for complex creative questions with direct responses to simple factual questions.

  • D Meta-Reasoning Showcase: User Prompt Control: Meta-reasoning enables the SFT model to adapt CoT triggering based on explicit user instructions about the desired reasoning level.The appendix presents this as user-prompt controllability over CoT.
  • D.1 Case 1: Historical Creative Question: The historical question is treated as complex because answering it may require reconstructing Li Shimin’s thoughts, future plans, and reflections after the coup.The model first identifies the question’s core and its possible interpretive dimensions.
  • D.1 Case 1: Historical Creative Question: Historical context shapes the response: the 626 coup killed Li Jiancheng and Li Yuanji, led to Li Yuan’s abdication, and created political and personal tensions.The proposed answer addresses legitimacy, Li Yuan, consolidation of power, and the aftermath for the brothers’ families.
  • D.1 Case 1: Historical Creative Question: Because no precise historical record exists of what Li Shimin wrote that night, the memoir is explicitly presented as reasonable speculation grounded in historical facts.The proposed style balances classical historical atmosphere with accessibility.
  • D.1 Case 1: Historical Creative Question: The generated memoir presents the coup as self-defense while combining political justification, filial appeals, governance intentions, and guilt over fratricide.Its structure includes a title, date, justification of the coup, appeals to Li Yuan, treatment of the families, and personal unease.
  • D.1 Case 1: Historical Creative Question: The response also promises humane treatment of the brothers’ families and frames future prosperity as justification for the coup’s unavoidable actions.The closing passage emphasizes ambition, uncertainty, and hopes for the Tang Empire.
  • D.2 Case 2: Factual Question: For the factual mountain question, the model identifies it as simple and answers directly with Mount Everest under the usual altitude convention.The appendix contrasts this direct response with reasoning that considers alternative definitions of “tallest.”

E Training and Testing Details

The appendix documents the experimental and testing configurations used for SFT, reward-model training, RL training, and evaluation. It complements the main paper’s setup description.

  • E Training and Testing Details: The appendix provides further details on the study’s experimental and testing configurations.These details complement the setup described in Section 3.1 of the main paper.
  • E Training and Testing Details: The documented training stages include Supervised Fine-Tuning, Reward Model training, and Reinforcement Learning training.Evaluation settings are also included in the configuration overview.
  • E Training and Testing Details: The appendix covers both training and evaluation settings rather than only one component of the pipeline.

SFT Training

The SFT setup uses a 15B/150B Mixture-of-Experts base model, truncates training cases at 32,000 tokens, and applies cosine learning-rate decay.

  • SFT Training: The base model uses a 15B/150B Mixture-of-Experts architecture.
  • SFT Training: Training cases are truncated to a maximum of 32,000 tokens.
  • SFT Training: The learning rate follows cosine decay from 2 × 10−5 to 2 × 10−6.The first value is the peak learning rate and the second is the decayed rate.

RM Training

The reward model is initialized from the SFT model and trained using diverse internally human-annotated data.

  • RM Training: The Reward Model is initialized using the SFT model.
  • RM Training: Reward-model training uses internally annotated data.
  • RM Training: The annotation data is human-annotated and diverse.

RL Training

RL training used both verifier-based and reward-model-scored data, while evaluations averaged results across repeated stochastic inferences.

  • RL Training: RL training combined verifiable data validated against known criteria with general data scored by a reward model.The reward model scored how well responses aligned with human preferences.
  • RL Training: Verifiable data received feedback from a verifier, enabling direct validation of model outputs.
  • RL Training: General data was scored according to response alignment with human preferences.
  • RL Training: Evaluations used temperature 1.0, top-p 0.7, and at least five inferences per test case before averaging scores.

Note on Data and Setup Disclosure

The authors provide limited methodological disclosure because proprietary considerations and confidentiality policies restrict details about the training data and setup.

  • Note on Data and Setup Disclosure: Proprietary considerations and company confidentiality policies prevent disclosure of further training-dataset composition details.
  • Note on Data and Setup Disclosure: The authors also withhold more granular details about the training setup.
  • Note on Data and Setup Disclosure: The authors state that the available information is intended to contextualize findings and support reproducibility of the core concepts within these constraints.
Loading 2505.11896v2…