PaperlayerView exact revision on arXiv

Source-linked AI summary

When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs

Xiaomin Li, Zhou Yu, Zhiwei Zhang, Xupeng Chen, Ziji Zhang, Yingying Zhuang, Narayanan Sadagopan, Anurag Beniwal

arXiv:2505.11423v3cs.CL

TL;DR

The paper investigates whether explicit reasoning improves instruction following, a question left open by reasoning models’ success on complex tasks. It evaluates models and analyzes failures and attention, then tests four mitigation strategies. CoT consistently reduces instruction-following accuracy, while selective approaches—especially classifier-selective reasoning—substantially recover performance across benchmarks.

  • Problem

    The paper asks whether explicit reasoning improves instruction-following accuracy, an issue not established by reasoning models’ success on complex tasks.

  • Method

    The study evaluates 15 models on IFEval and ComplexBench, analyzes reasoning-induced failures and constraint attention, and evaluates four mitigation strategies.

  • Results

    CoT consistently reduces instruction-following accuracy across IFEval and ComplexBench, while classifier-selective reasoning provides substantial gains across benchmarks.

  • Takeaways & Limitations

    Selective reasoning, particularly classifier-selective reasoning, can recover lost instruction-following performance, with strategy choice depending on model capacity and instruction complexity.

  • Takeaways & Limitations

    The study focuses exclusively on instruction-following tasks, leaving possible effects in other domains for future work.

Abstract

from arXiv · show

Reasoning-enhanced large language models (RLLMs), whether explicitly trained for reasoning or prompted via chain-of-thought (CoT), have achieved state-of-the-art performance on many complex reasoning tasks. However, we uncover a surprising and previously overlooked phenomenon: explicit CoT reasoning can significantly degrade instruction-following accuracy. Evaluating 15 models on two benchmarks: IFEval (with simple, rule-verifiable constraints) and ComplexBench (with complex, compositional constraints), we consistently observe performance drops when CoT prompting is applied. Through large-scale case studies and an attention-based analysis, we identify common patterns where reasoning either helps (e.g., with formatting or lexical precision) or hurts (e.g., by neglecting simple constraints or introducing unnecessary content). We propose a metric, constraint attention, to quantify model focus during generation and show that CoT reasoning often diverts attention away from instruction-relevant tokens. To mitigate these effects, we introduce and evaluate four strategies: in-context learning, self-reflection, self-selective reasoning, and classifier-selective reasoning. Our results demonstrate that selective reasoning strategies, particularly classifier-selective reasoning, can substantially recover lost performance. To our knowledge, this is the first work to systematically expose reasoning-induced failures in instruction-following and offer practical mitigation strategies.

1 Introduction

The paper asks whether explicit reasoning improves instruction following and finds that CoT can instead degrade accuracy. It analyzes failure patterns, attention shifts, and mitigation strategies, especially selective reasoning.

  • Empirical finding: 15 language models evaluated across IFEval and ComplexBench show consistent accuracy drops when prompted with CoT.IFEval contains simple verifiable constraints, whereas ComplexBench uses compositional dependent constraints.
  • Failure analysis: Reasoning helps with formatting and structural requirements, as well as lexical constraints that override default tendencies.These patterns emerge from a large-scale manual case study of samples where reasoning changes performance.
  • Failure analysis: CoT often neglects simple constraints by over-focusing on high-level content or violates them by adding redundant content.The paper provides detailed examples of both reasoning-induced failure modes.
  • Attention analysis: Constraint attention measures attention directed toward instruction constraint tokens, and visualizations show reduced constraint awareness with CoT across datasets, models, and layers.The attention shift may partially explain diminished instruction adherence.
  • Mitigation: The paper evaluates in-context learning, self-reflection, self-selective reasoning, and classifier-selective reasoning to mitigate reasoning-related accuracy losses.Classifier-selective reasoning provides substantial gains, while self-reflection is particularly helpful for larger models and simple instructions.
  • Contributions: The study evaluates 15 models and systematically examines a previously identified phenomenon in instruction following.The authors describe this as the first work to discover and systematically explore the phenomenon.

2 Related Work

Prior work shows that CoT and reasoning-focused LLMs improve complex reasoning tasks, while instruction following remains central to aligning outputs with user expectations. This paper examines the less studied possibility that reasoning can harm instruction adherence.

  • Chain-of-Thought and Reasoning LLMs: Reasoning-focused LLMs and CoT prompting have achieved strong results on complex tasks, particularly mathematics.The related work also notes that CoT can add computational costs and yield limited or no improvement in some contexts.
  • Research gap: The paper extends prior discussion of reasoning’s mixed effects by examining its negative impact on instruction-following performance.This focus connects the success of reasoning methods with a less explored failure mode.
  • Instruction Following: Instruction following aligns language-model outputs with user expectations and enables reliable execution of user-specified tasks.Instruction tuning on instruction-response pairs has helped foster this capability.

3 Experiments

The experiments evaluate instruction following on two complementary benchmarks with different constraint structures and compare models with and without CoT. CoT generally reduces performance, including across reasoning-enhanced model pairs.

  • Datasets and metrics: IFEval contains 541 prompts with one to three verifiable constraints drawn from 25 types, including word count, formatting, and keyword usage.The study uses instruction-level loose accuracy to tolerate minor formatting deviations.
  • Datasets and metrics: ComplexBench contains 1,150 compositional instructions and over 5,300 scoring questions spanning four operations, four constraint types, and 19 dimensions.Its evaluation combines rule-based and LLM-based assessments, with dependent constraints failing when prerequisites fail.
  • Experimental setup: The evaluation compares answer quality with and without step-by-step CoT reasoning across a diverse set of closed- and open-source models.Open-source models range from 1B to 70B parameters, and inference uses temperature 0.
  • Results: 13 of 14 models decline on IFEval with CoT, while all models decline on ComplexBench; Llama3-8B-Instruct falls from 75.2% to 59.0%.The Llama3-8B-Instruct change is a reduction of over 16 percentage points.
  • Results: Reasoning-enabled models consistently underperform their corresponding non-reasoning variants, although these comparisons are not fully controlled for additional training stages.Examples include Claude3.7-Sonnet versus Claude3.7-Sonnet-Think and DeepSeek-V3 versus DeepSeek-R1.
  • Results: Explicit reasoning can increase instruction-constraint violations and impair instruction-following reliability despite often helping on complex tasks.This finding is presented as an underexplored vulnerability of reasoning-enhanced models.

4 Analysis

The analysis examines when CoT helps or harms constraint satisfaction and uses constraint attention to explain reasoning-induced instruction-following failures.

  • Manual Case Study: Manual analysis covered 541 IFEval samples and over 1,000 ComplexBench samples where CoT changed constraint satisfaction.The cases were analyzed for affected constraints and reasons performance improved or degraded.
  • When Reasoning Helps: Reasoning improves formatting and structural adherence, including valid JSON, double quotes, and markdown syntax.
  • When Reasoning Helps: Reasoning enhances lexical precision for requirements such as rare-character counts, omitted punctuation, and exact numbers of capitalized words.
  • When Reasoning Hurts: Reasoning often neglects simple mechanical constraints when it prioritizes high-level content planning.Observed violations include word-count limits, exact repetition, lowercase-only requirements, and unwanted additions.
  • When Reasoning Hurts: Reasoning can introduce redundant explanations, translations, emphasis, or commentary that violate output constraints.Examples include foreign-language-only, no-commas, quote-only, and capitalization-limit tasks.
  • Constraint-Aware Attention: Constraint attention measures attention directed toward constraint tokens during generation, comparing base and CoT runs over answer segments.Constraint tokens are extracted from each instruction, mapped to prompt indices, and tracked during reasoning and answering.
  • Constraint-Aware Attention: CoT generally flattens constraint-attention traces; degraded cases show lower answer-phase attention, while improved cases often show an answer-aligned bump.The traces were visualized across IFEval and ComplexBench using Qwen2.5-1.5B-Instruct.
  • Constraint-Aware Attention: ∆β > 0 for most cases, with larger drops in LOSE cases, particularly in early-to-middle layers.The analysis suggests that lower constraint attention predicts reasoning-induced failures.

5 Mitigating Reasoning-Induced Failures in Instruction Following

The paper evaluates four mitigation strategies for reasoning-induced instruction-following degradation, finding distinct trade-offs across models and benchmark complexity. Classifier-selective reasoning is broadly effective but requires model-specific training.

  • 5 Mitigating Reasoning-Induced Failures in Instruction Following: Four strategies are evaluated to mitigate CoT-related performance degradation: in-context learning, self-reflection, self-selective reasoning, and classifier-selective reasoning.The methods are compared against the CoT baseline in Table 1 and Figure 3.
  • 5.1 Method 1: Few-Shot In-Context Learning: Few-shot in-context learning prepends manually revised failure-case examples containing instructions, corrected thinking, and answers.
  • 5.1 Method 1: Few-Shot In-Context Learning: Few-shot in-context learning yields only modest improvements because token limits restrict examples to four for IFEval and three for ComplexBench, with possible source-model bias.
  • 5.2 Method 2: Self-Reflection: Self-reflection generates an initial thinking-based response, then retains or revises it after a second inference evaluates the reasoning and answer.
  • 5.2 Method 2: Self-Reflection: Self-reflection improves IFEval performance in 11 of 14 models and is best among the four methods for 7 models, but degrades weaker models.
  • 5.3 Method 3: Self-Selective Reasoning: Self-selective reasoning lets the model decide from the instruction whether to use CoT or answer directly.
  • 5.3 Method 3: Self-Selective Reasoning: Self-selective reasoning improves 10 of 14 models on IFEval and all models on ComplexBench, leading among methods for 6 models on ComplexBench.
  • 5.4 Method 4: Classifier-Selective Reasoning: Classifier-selective reasoning trains a separate external classifier for each target model to predict whether CoT improves per-sample constraint satisfaction.

6 Conclusion

The study finds that explicit Chain-of-Thought reasoning can impair instruction-following by diverting attention from constraints, while selective reasoning—especially classifier-selective reasoning—substantially recovers lost performance. The work focuses exclusively on instruction-following tasks and leaves effects in other domains for future study.

  • Explicit reasoning through Chain-of-Thought can negatively impact instruction-following abilities across IFEval and ComplexBench.
  • Reasoning can divert model focus from constraint-related tokens, causing instructions to be overlooked or violated.
  • The study is limited to instruction-following tasks, leaving possible effects in other domains for future work.
  • Four mitigation strategies are evaluated: in-context learning, self-reflection, self-selective reasoning, and classifier-selective reasoning.
  • Classifier-selective reasoning achieves the most consistent improvements across both datasets while substantially recovering lost performance.

A Experiments: Additional Results

The additional results present visual comparisons and detailed IFEval examples showing how reasoning can either satisfy formatting and lexical constraints or improve constraint satisfaction through explicit planning.

  • Figure 3 visualizes instruction-following accuracy across models and methods on IFEval and ComplexBench.
  • The examples compare responses with and without reasoning, their constraint-satisfaction scores, and analyses of performance differences.
  • Reasoning Helps: Reasoning helps satisfy formatting or structural requirements by acting as a structured checklist that prevents requirements from being overlooked.
  • Reasoning Helps: In one example, reasoning produced a name satisfying both the repetition and double-angular-bracket formatting requirements.
  • Reasoning Helps: Reasoning helps satisfy exact lexical constraints by explicitly recognizing that required text must end without additional punctuation or words.

Reasoning Hurts

Reasoning can hurt instruction-following when it prioritizes high-level content over simple mechanical constraints or adds helpful material that itself violates the constraints.

  • Reasoning Hurts: Reasoning can over-focus on high-level content and neglect simple mechanical constraints.
  • Reasoning Hurts: Reasoning can add helpful elements that break constraints even when the final requested content satisfies them.
  • Reasoning Hurts: In the hotel-ad example, the reasoning response planned the structure but inserted a comma, violating the no-comma requirement.
  • Reasoning Hurts: The non-reasoning hotel-ad response satisfied both exact-prompt repetition and comma avoidance.
  • Reasoning Hurts: In the haiku example, reasoning planned vocabulary and letter frequency but introduced commas in its meta-commentary, violating the no-comma constraint.

C Attention Trace

The attention-trace figures show examples from IFEval and ComplexBench for Llama3.2-1B-Instruct, comparing cases where reasoning ties, loses, or wins against no reasoning.

  • IFEval: Figure 4 presents IFEval constraint-attention traces for Llama3.2-1B-Instruct across TIE, LOSE, and WIN cases.
  • In both figures, the red dashed line marks the start of Thinking and the green dashed line marks the start of the Answer.
  • ComplexBench: Figure 5 presents ComplexBench constraint-attention traces for Llama3.2-1B-Instruct across TIE, LOSE, and WIN cases.

E Few-Shot Examples

The few-shot materials provide constrained instruction, thinking, and answer examples across IFEval and ComplexBench, alongside prompt templates for reasoning and example-based generation.

  • Example Structure: Few-shot examples pair each instruction with internal thinking and a constraint-adhering final answer.The materials explicitly describe this three-part structure for both IFEval and ComplexBench.
  • IFEval Examples: IFEval examples emphasize independently verifiable constraints such as sentence limits, comma avoidance, required repetition, headings, and calls to action.The examples include a blog post, a repeated hotel-ad request, and a formatted print advertisement.
  • ComplexBench Examples: ComplexBench examples combine multiple requirements, including tone, character limits, allegorical endings, kaomoji placement, language exclusivity, and lexical exclusions.Examples cover a humorous food response, a Marathi-only haiku, and a dessert name unrelated to Oreo.
  • Reasoning Example: A pet-count example demonstrates constraint-preserving arithmetic: 20 members each have one pet, leaving 12 extra pets, so at most 6 members can have three pets.Each three-pet member contributes two extra pets beyond the one-pet minimum.
  • Prompt Templates: The reasoning prompts require step-by-step thinking before an official answer and can incorporate provided examples through in-context learning.The templates distinguish THINK and ANSWER sections and instruct generation based on example strings.

G Self-Selective Reasoning

Self-selective reasoning compares the model’s decision to use CoT against sample-level performance labels on IFEval and ComplexBench. The method tends to identify helpful cases but overuse reasoning when it is not beneficial.

  • Decision Evaluation: Self-selective reasoning decisions are compared with binary ground-truth labels indicating whether CoT improves sample-level constraint satisfaction.Figure 9 reports decision accuracy on IFEval and ComplexBench.
  • Decision Evaluation: The model’s decisions show high recall but low precision, indicating that it often selects reasoning even when reasoning does not improve outcomes.The reported pattern characterizes self-selective reasoning as prone to overuse.

H Training Classifier for Selective-Reasoning

Classifier-selective reasoning trains a target-model-specific classifier to predict when CoT improves constraint satisfaction, while a separate analysis tests whether reasoning length predicts instruction-following effectiveness. The classifier setup uses labeled response comparisons, whereas reasoning length shows weak and inconsistent predictive value.

  • Classifier-Selective Reasoning: Classifier-selective reasoning trains a separate binary classifier for each target model to predict whether CoT yields higher constraint satisfaction than a base response.Label 1 denotes higher constraint satisfaction with CoT; label 0 denotes otherwise.
  • Classifier Training: 50% of labeled data trains each classifier and the remaining 50% evaluates downstream mitigation effectiveness.Training uses full fine-tuning for three epochs on one NVIDIA-H100-80GB GPU at a learning rate of 1e-5.
  • Classifier Training: The classifier search spans five Llama and Qwen backbone models, learning rates from 5e-5 to 5e-6, and one-to-eight epochs, with 10% of training data reserved for validation.Selected configurations are based on validation accuracy, typically ranging from 0.75 to 0.92 across models.
  • Reasoning-Length Analysis: Reasoning effectiveness is defined as score_diff = score(CoT) - score(Base), and its Pearson correlation with THINK-token length is computed separately for IFEval and ComplexBench.This analysis directly tests whether longer reasoning improves instruction-following.
  • Reasoning-Length Analysis: Most Pearson correlations are close to zero, so reasoning length is not a reliable predictor of improved instruction adherence across models or benchmarks.A few smaller models show slightly stronger correlations, but no consistent or interpretable trend emerges.
Loading 2505.11423v3…