PaperlayerView exact revision on arXiv

Source-linked AI summary

The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning

Yubo Li, Lu Zhang, Tianchong Jiang, Ramayya Krishnan, Rema Padman

arXiv:2603.29025v3cs.CLcs.AI

TL;DR

This paper studies how salient surface cues override unstated feasibility constraints in LLM reasoning. It introduces the Heuristic Override Benchmark and a behavioral analysis, finding systematic heuristic dominance alongside partial recovery from explicit constraint prompting.

  • Problem

    Reasoning errors from correctly perceived premises are difficult to detect, motivating study of failures when salient cues conflict with unstated feasibility constraints.

  • Method

    The paper combines a 500-instance benchmark with minimal pairs and explicitness gradients, causal behavioral characterization, and input-side interventions across models.

  • Results

    No model among 14 frontier systems exceeds 75% strict accuracy, while heuristic cues exert 8.7–38× more influence than goals and explicit hints recover 15.3 pp.

  • Takeaways & Limitations

    Heuristic override is a systematic behavioral vulnerability, and explicit constraint enumeration provides a tested partial mitigation.

  • Takeaways & Limitations

    The account is behavioral rather than implementational, and the benchmark is English-only and focused on everyday feasibility constraints.

Abstract

from arXiv · show

Large language models fail when a salient surface cue conflicts with an unstated feasibility constraint. We introduce the Heuristic Override Benchmark (HOB): 500 instances spanning 4 heuristic families and 5 constraint families, with minimal pairs and explicitness gradients. We pair HOB with a falsifiable behavioral characterization following a diagnose-measure-bridge-treat arc. Causal-behavioral analysis of the car wash problem across six models reveals context-independent sigmoid heuristics: the distance cue has 8.7 to 38 times more influence than the goal, and attribution better matches keyword association than compositional inference. Across 14 models, strict 10/10 evaluation shows that no model exceeds 75%, and presence constraints are hardest at 44%. A minimal hint improves performance by 15 pp, suggesting a constraint-inference failure rather than missing knowledge. However, 12 of 14 models perform worse when the constraint is removed, by up to 39 pp, revealing conservative bias. A thinking-mode ablation on Gemini 3.1 Pro drops performance from 74.6% with thinking on to 58.4% with thinking off, while explicit goal decomposition recovers it to 71.2%. Thus, internal deliberation does useful work, and explicit prompting can partially substitute for it. Reasoning models do not categorically outperform non-reasoning peers: after controlling for capability rank, the residual reasoning-mode effect is 1.8 pp and is not significant. Parametric probes show that the sigmoid pattern generalizes to cost, efficiency, and semantic-similarity heuristics. Goal-decomposition prompting improves performance by 5.0 pp, compared with 3.1 pp for generic chain-of-thought, isolating constraint enumeration as the active ingredient. Overall, heuristic override is a systematic reasoning vulnerability with a quantified locus in inference order, not knowledge, and a tested intervention.

1 Introduction

LLMs can produce plausible reasoning errors when salient surface cues conflict with unstated feasibility constraints. The paper introduces HOB and a behavioral analysis to diagnose, measure, and address this vulnerability.

  • Reasoning errors are difficult to detect because outputs can sound plausible and internally consistent despite incorrect conclusions.
  • Prior work documents shortcut learning across language, question-answering, mathematical, and arithmetic tasks, alongside cognitive-bias analogues.
  • 42 of 53 models recommended walking on a single pass, while only 5 answered correctly across ten trials in a subsequent car-wash evaluation.
  • The car-wash problem pits a short-distance heuristic against the implicit constraint that the car must be at the wash.
  • HOB contains 500 instances spanning 4 heuristic families and 5 constraint families, with minimal pairs and explicitness gradients.Its 15 populated cells are evaluated under a diagnose–measure–bridge–treat framework.

2 Method

The method combines a behavioral car-wash case study, causal occlusion, monotonicity sweeps, and the HOB benchmark. Controlled perturbations, minimal pairs, and prompting comparisons test heuristic reliance, constraint sensitivity, and mitigation.

  • The analysis is behavioral rather than implementational, characterizing frozen models’ input–output functions through input-side interventions.Representational validation is left for future work.
  • Task formulation: The car-wash task decomposes into a goal, a distance heuristic, and two options; DRIVE is correct because the car must be physically present.
  • Task formulation: The scalar score s(x) = log p(WALK | x) − log p(DRIVE | x) is obtained with anchored teacher-forced scoring for deterministic, reproducible comparison.
  • Behavioral analysis: Causal occlusion perturbs sentences, semantic spans, and tokens using mask, neutral, and contradict operators, requiring agreement across operators.
  • Behavioral analysis: A 14-point distance sweep compares conflict and control conditions to distinguish flat constraint-sensitive behavior from context-independent sigmoid heuristics.
  • HOB benchmark: HOB crosses 4 heuristic families with 5 constraint families, using 15 populated cells, minimal pairs, and heuristic-strength and constraint-explicitness gradients.
  • Experimental setup: Study 1 evaluates six models with six paraphrases, while Study 2 evaluates 14 models under strict correctness requiring all 10 trials to be correct.
  • Experimental setup: Parametric sweeps test cost, efficiency, proximity, and semantic-similarity heuristics across capability, scope, and related constraints.

3 Results

Across the car-wash case and HOB benchmark, models systematically overweight salient heuristics, while constraint explicitness and targeted decomposition improve strict accuracy.

  • 3.1 Behavioral Characterization: 0% accuracy: all six evaluated models chose Walk on every car-wash paraphrase, with wrong-answer confidence varying non-monotonically by model size.Qwen3-14B was more confident in Walk than the larger Qwen3-32B.
  • 3.1 Behavioral Characterization: 8.7× to 38.0×: distance perturbations influenced decisions at least an order of magnitude more than goal perturbations across six models.Distance perturbations shifted every model toward Drive, whereas goal effects were near-zero or sometimes increased Walk preference.
  • 3.1 Behavioral Characterization: 30.3 versus 5.8: the largest token-level goal effect was five times smaller than the distance effect, consistent with keyword associations rather than compositional inference.Within the goal span, washing-action tokens favored Drive while “car” and “vehicle” favored Walk, producing cancellation.
  • 3.1 Behavioral Characterization: Sigmoid conflict curves: all six models tracked the control across 14 distances, with goals shifting amplitude but never changing the curve’s shape.Even the strongest goal modulation merely shifted the sigmoid downward rather than gating the decision.
  • 3.2 HOB Benchmark: 49.6% to 74.6%: strict HOB accuracy stayed below 75%, with presence constraints hardest at 44.4% and capability constraints easiest at 71.6%.The benchmark used approximately 500 instances, 14 models, and a strict criterion requiring all 10 trials to be correct.
  • 3.2 HOB Benchmark: +15.3 pp: a single hint raised average accuracy from 59.2% to 74.5%, while removing constraints hurt 12 of 14 models by up to −38.5 pp.These paired effects expose both an inference bottleneck and conservative bias.
  • 3.3 Parametric Sweeps: Does the Signature Generalise?: Efficiency produced the most sigmoid failures, whereas cost and proximity-capability sweeps showed the most correct reasoning across six models.Concrete capability constraints were easier to maintain than abstract scope constraints.
  • 3.4 Mitigation: Goal-Decomposition Prompting: +9.0 pp and +6.6 pp: goal decomposition improved GPT-5.4 and Llama 4 Scout, while Gemini 3.1 Pro changed by −0.6 pp.The intervention asks models to enumerate necessary conditions before answering.

4 Discussion

The investigation identifies an inference bottleneck in which context-independent heuristics override implicit constraints, while deliberation or explicit decomposition can activate constraint reasoning. Minimal-pair results also reveal conservative bias, and the paper’s behavioral scope limits architectural conclusions and mitigation claims.

  • Unified account: Context-independent heuristic mappings override implicit goal constraints across distance, efficiency, and semantic similarity.The authors characterize the failure as an inference bottleneck rather than a categorical absence of knowledge.
  • Interventions: 74.6% with thinking on falls to 58.4% with thinking off, recovering to 71.2% with explicit decomposition.The ablation suggests internal deliberation and precondition-enumeration prompting can activate the same operation.
  • Conservative bias: 12/14 models perform worse when the constraint is removed, with drops up to −38.5 pp, exposing conservative bias.This asymmetry means constraint-active accuracy alone can overestimate genuine reasoning ability.
  • Deployment implications: The failure matters for decision-support applications because fluent, confident outputs can still be systematically wrong when unstated constraints compete with salient features.The paper specifically identifies medical triage, legal reasoning, and financial planning as relevant domains.
  • Limitations: The account is behavioral rather than implementational, so representational validation and broader architectural claims remain future work.The benchmark and interventions are English-only, target everyday feasibility constraints, and provide a proof-of-concept mitigation rather than a comprehensive solution.

5 Related Work

The paper distinguishes its target from shortcut learning, distractor filtering, commonsense knowledge tests, and mechanistic interpretability. It studies competing integral cues and hidden constraints at the decision-policy level using behavioral, input–output analyses.

  • Shortcut Learning and Heuristic Reliance: The paper targets reasoning-level compositional templates whose sigmoid signature differs from feature-level shortcut models.Its explicitness-gradient and goal-decomposition manipulations distinguish missing knowledge from misuse.
  • Novelty: Prior aggregate accuracy can miss conservative-bias confounds, whereas strict accuracy, minimal-pair asymmetry, and HDR expose them.The paper reports that 12/14 models drop on minimal pairs.
  • Distractibility and Constraint-Following: Unlike distractor benchmarks, HOB places both the heuristic cue and hidden constraint inside the prompt, requiring prioritization between competing signals.The setting differs from benchmarks that inject additive noise or test compliance with stated rules.
  • Commonsense Reasoning and the Frame Problem: The paper tests models that possess relevant commonsense knowledge yet err because a surface heuristic overpowers it.This complements commonsense benchmarks and connects the failure to the classical frame problem.
  • Diagnostic Methodology: The diagnostic method operates at the input–output level through perturbation-based attribution and counterfactual evaluation rather than mechanistic circuit analysis.Multiple replacement operators and agreement requirements are used to mitigate distribution-shift concerns.

6 Conclusion

The paper concludes that salient heuristics systematically override unstated feasibility constraints and presents HOB, behavioral diagnostics, and constraint-enumeration prompting as a quantified account and tested intervention. Its benchmark and experiments span controlled instances, model evaluations, and reproducible scoring procedures.

  • Conclusion: The paper’s central conclusion is that salient surface cues systematically override unstated feasibility constraints.The conclusion frames heuristic override as a reasoning vulnerability addressed through behavioral characterization and intervention.
  • Benchmark: HOB contains ∼500 instances spanning 15 H × C cells across seven domains, with base scenarios, minimal pairs, gradients, and controls.The benchmark includes 132 base scenarios, 132 minimal pairs, 64 heuristic-strength variants, 64 explicitness variants, and 30 controls.
  • Evaluation: Strict and trial-level evaluation, model tables, and anchored teacher-forced scoring support deterministic comparison across the reported studies.Study 1 uses anchored teacher-forced scoring, while Study 2 evaluates the HOB model set under the benchmark protocol.
  • Case study: The car-wash case study reports incorrect Walk preference across all six models on the evaluated item.The accompanying table reports accuracy and mean decision score, where positive mean score indicates incorrect Walk preference.

C.2 Full Occlusion Results

The occlusion and monotonicity analyses show that distance dominates goal evidence and produces context-independent sigmoid conflict curves, while benchmark results reveal uneven constraint and heuristic difficulty. Strict evaluation further exposes inconsistency and conservative bias.

  • Occlusion: HDR measures the relative influence of distance and goal spans as |∆s_dist|/|∆s_goal| under the contradict operator.The analysis uses span-level occlusion across six paraphrases.
  • Token attribution: Goal-span token effects weakly oppose one another, leaving near-zero net goal influence; no token matches the distance cue’s magnitude.The Qwen3-4B token-level profile provides no evidence of compositional constraint inference.
  • Monotonicity: Every model produces sigmoid conflict curves that track the control curve across distances.The monotonicity analysis covers all six models and compares conflict with control conditions.
  • Monotonicity: The monotonicity panels compare Qwen3-4B, Qwen3-32B, GPT-OSS-20B, Qwen3-14B, Qwen3-8B, and Qwen3.5-27B.The individual plots highlight Qwen3-14B as having the highest Walk bias at short distances.
  • Strict evaluation: Strict accuracy is correct performance on all 10 trials, revealing reliability beyond trial-level accuracy.DeepSeek R1 records 83.1% trial accuracy but 64.2% strict accuracy, while GPT-OSS-20B records 79.1% and 51.0%, respectively.
  • Constraint families: 44.4% mean strict accuracy makes presence constraints hardest, followed by procedural constraints at 52.9%, while capability constraints reach 71.6%.The constraint hierarchy is consistent across models; concrete, visualisable capability constraints are handled better.
  • Heuristic families: Cost heuristics are easiest to override, whereas proximity and semantic-match cues are hardest.The paper links proximity difficulty to frequent distance-to-decision mappings and semantic-match difficulty to category-level associations.

D.5 Heuristic Strength Analysis

Heuristic strength does not produce a monotonic accuracy pattern: even weak cues can override constraint inference. Parametric sweeps show that this failure depends on the heuristic–constraint pairing and generalizes beyond proximity.

  • Heuristic strength: 62.8% strict accuracy for strong cues, 56.2% for medium, and 59.6% for weak cues reveal a non-monotonic pattern.Weak heuristic cues can be sufficient to trigger override failures.
  • Heuristic strength: The failure appears tied to activating constraint reasoning rather than simply being overwhelmed by stronger heuristic signals.
  • Domain analysis: Travel and medical scenarios are substantially harder than everyday domains, with a 33-point gap between home at 74.5% and travel at 41.4%.The passage attributes the difference to specialised procedural constraints.
  • Parametric sweeps: Four parametric sweeps test cost, efficiency, proximity, and semantic-similarity heuristics across conflict and control conditions.The sweeps use four H × C combinations and repeated trials across grid points.
  • Parametric sweeps: H-eff × C-cap produces sigmoid failure in Qwen3-4B, whereas H-cost × C-scope and H-prox × C-cap show correct reasoning.
  • Parametric sweeps: Semantic similarity also produces a sigmoid: more car-related gas-station descriptions make most models incorrectly recommend tire repair there.Qwen3-4B shows the strongest semantic sigmoid, while larger Qwen models remain closer to the decision boundary.

F Inter-Rater Agreement on Cell Assignment

Independent annotators show substantial agreement on HOB cell assignments, which rises after calibration focused on the most contested constraint boundary. The resulting taxonomy is reproducible with brief calibration.

  • Aggregate agreement: κ = 0.71 across 50 instances, rising to κ = 0.84 after one calibration discussion on the C-scope versus C-cap boundary.
  • Aggregate agreement: 38 of 50 instances received unanimous cell agreement, 9 received majority agreement, and 3 were re-classified after calibration.
  • Per-cell agreement: Per-cell κ ranges from 0.55 for C-proc versus C-val to 0.88 for H-cost × C-cap.
  • Per-cell agreement: The C-scope versus C-cap boundary improved from κ = 0.62 before calibration to κ = 0.83 afterward.Disagreements fell from 6/50 to 1/50.
  • Operational distinction: C-scope denotes a service whose offering excludes the goal, whereas C-cap denotes means that cannot accomplish the goal.The gas-station tire-repair case illustrates C-scope; carrying a sofa on foot illustrates C-cap.
  • Implications: The taxonomy is reproducible by independent annotators with a brief calibration step.

G.1 Thinking-Mode Ablation on Gemini 3.1 Pro

Gemini 3.1 Pro’s internal thinking materially improves HOB performance, while explicit goal decomposition can substitute for much of that benefit. The two routes are not additive when used together.

  • Ablation result: 16.2 pp separates Gemini 3.1 Pro performance with thinking on from thinking off, according to the thinking-mode ablation.
  • Ablation result: Explicit goal decomposition recovers nearly all of the performance lost when internal thinking is disabled.
  • Substitutability: 74.6% with thinking on falls to 74.0% with extra goal decomposition, a non-significant difference showing no additive benefit.
  • Substitutability: Explicit prompting and internal thinking are substitutable routes to the same effect, and removing one while adding the other approximately preserves performance.
  • Trace audit: DeepSeek R1 traces show 88.5% correctness when constraints are mentioned and applied, versus 44.4% when they are never mentioned.The difference is significant by a two-sided Fisher’s exact test, p < 0.01.

I Temperature Ablation

Temperature ablation shows that strict-accuracy rankings are robust to decoding choices, while absolute values change only slightly across the tested temperatures.

  • Ranking stability: ρ > 0.97 for all pairwise model comparisons across temperatures, so strict-accuracy rankings remain invariant.
  • Absolute accuracy: Absolute strict accuracy shifts by at most 1.6 pp from T = 0.0 to T = 0.7.The evaluation uses three representative models on a 100-instance HOB sample with N=10 trials each.
  • Interpretation: The results indicate that the 10/10 strict criterion captures reasoning reliability rather than sampling-stability artifacts.

J Reasoning vs. Non-Reasoning Model Breakdown

The apparent advantage of reasoning models is largely explained by general capability, and controlled analyses do not establish a distinct reasoning-mode benefit. Minimal-pair behavior likewise does not separate cleanly by reasoning class, while the study uses outcome-matched statistical tests rather than relying on binary strict verdicts.

  • Capability-adjusted comparison: A 9.7 pp aggregate gap between reasoning and non-reasoning models is largely confounded with general capability.After controlling for Chatbot Arena Elo rank, the residual reasoning-mode effect is β = 1.8 pp (p = 0.31, not significant).
  • Capability-adjusted comparison: Two direct same-tier contrasts further refute a pure reasoning-mode explanation.
  • Minimal-pair asymmetry: −22.5 pp is the non-reasoning mean after excluding two outliers, versus −23.8 pp for reasoning models.The difference is statistically indistinguishable (Welch’s t-test, p = 0.78).
  • Statistical analysis: Matched-condition effects are significant at trial, per-instance pass-rate, and strict-verdict levels, so conclusions do not rely on binary collapse.The reported tests are trial-level logistic regression and paired Wilcoxon signed-rank tests; McNemar’s test is reference-only.
  • Statistical analysis: The DeepSeek R1 trace audit uses a two-sided Fisher’s exact test because its correctness groups are disjoint.
Loading 2603.29025v3…