PaperlayerView exact revision on arXiv

Source-linked AI summary

Recognition-Refusal Misalignment in LLMs: Why Models Answer Structurally Unanswerable Questions

Yucheng Du, Xiyang Hu

arXiv:2608.29109v1cs.CL

TL;DR

The paper asks whether confident answers to structurally impossible prompts reflect missing impossibility recognition or failed routing into abstention. It probes hidden-state geometry and intervenes during generation, finding a usable recognition signal that is poorly aligned with canonical safety refusal and causally changes invalidity-aware behavior.

  • Problem

    The paper investigates whether answering structurally impossible math and code questions reflects absent impossibility representation or failure to route an existing signal into abstention.

  • Method

    The study uses one-dimensional residual-stream probes, geometric comparisons with a verified safety-refusal direction, and generation-time steering across structural math and code tasks.

  • Results

    Across 22 model–dataset cells, impossibility recognition reaches mean AUC 0.939, while its cosine with the safety-refusal direction is 0.087; steering changes invalidity-aware behavior bidirectionally and dose-responsively.

  • Takeaways & Limitations

    The findings support a routing rather than encoding account: the model has a linearly accessible impossibility signal, but the trained safety-refusal pathway reads a different axis.

  • Takeaways & Limitations

    Causal evidence is cleanest for structural impossibility, and unmeasured directions could mediate other abstention routes.

Abstract

from arXiv · show

Large language models often answer structurally unanswerable questions, such as computing cot(-540°) or evaluating (1).startswith("1"), instead of abstaining. We ask whether this failure reflects missing recognition or failed routing from recognition to abstention. Across instruction-tuned models from 1.7B to 70B parameters, a single linear direction in the hidden state separates answerable from structurally impossible math and code prompts, showing that models represent impossibility before generation. Yet this recognition direction is nearly orthogonal to the canonical safety-refusal direction that mediates trained harmful-content refusal. An in-domain behavior-defined invalidity-aware direction is closer to recognition, but only partially aligned with it, and remains near-orthogonal to safety refusal. Generation-time steering along the recognition direction changes invalidity-aware behavior bidirectionally and dose-responsively on structural math and code cells, while random directions do not. Base/instruct comparisons further show that the low-cosine geometry is already present at the pretraining endpoint. The confident-on-impossible failure is therefore better explained as a routing failure than as an encoding failure: the model has a usable "no admissible answer" signal, but the safety-refusal pathway is not aligned to use it.

1 Introduction

LLMs can produce answer-like outputs for structurally impossible math and code despite representing impossibility before generation. The evidence supports a routing failure: the recognition signal is not aligned with the trained safety-refusal pathway.

  • cot(−540◦) is undefined and (1).startswith("1") raises an attribute error, yet clean-baseline generations can answer both prompts.
  • The failure could reflect missing impossibility representation or failure to route an existing signal into abstention.
  • Across 22 model–dataset cells, a one-dimensional null-space probe separates answerable from unanswerable prompts with mean AUC 0.939.
  • The impossibility direction has mean cosine 0.087 with the canonical safety-refusal direction, indicating near-orthogonality rather than reuse of that pathway.
  • Steering along the impossibility direction changes invalidity-aware behavior bidirectionally and dose-responsively, with signal-minus-random gated flip rates of +33 to +52pp.
  • The analysis is scoped to structural impossibility, while epistemic unanswerability and false-premise questions use different ground-truth definitions and behavioral signals.

2 Problem Setup

The paper distinguishes unanswerability by why a prompt lacks an acceptable answer. Its main setting is structural impossibility, where formal rules make an admissible answer impossible and matched pairs permit controlled analysis.

  • Unanswerability comprises structural impossibility, epistemic unanswerability, and false-premise questions, each requiring different mechanisms.
  • Structural impossibility violates a formal rule, as in division by zero, square roots of negative reals, singular-matrix inversion, or Python TypeError.
  • Structural-impossibility ground truth is verifiable from rules alone, and matched answerable/unanswerable pairs can differ only in a formally diagnosable feature.
  • Epistemic unanswerability means a correct answer could exist but is not determined by the provided evidence; fact800 operationalizes this with paragraph-matched SQuAD 2.0 pairs.
  • False-premise questions presuppose a false fact, so the desired behavior is to reject the premise rather than answer it.
  • Claims in the main analysis should be read as structural-impossibility claims unless fact800 or FalseQA is explicitly named.

3 Method Sketch

The method constructs a residual-stream impossibility direction, compares it with a verified safety-refusal direction, and tests causal effects through generation-time steering. It also documents labeling and evaluation controls for the intervention grid.

  • Impossibility direction: The CosNSRT probe removes the top k=100 principal components of train-answerable states before estimating a null-space mean-difference direction.
  • Impossibility direction: Test states are scored by cosine with the direction, using orientation-invariant AUC averaged over 5 seeds per cell.
  • Safety refusal and orthogonality: The safety-refusal comparator is constructed as dref,safety = µharmful −µharmless from 50 harmful and 50 harmless prompts, followed by behavior verification.
  • Steering and gated flip rate: Generation-time steering adds α · ˆd at a fixed layer, sweeps α ∈{5, 10, 20, 40} · σ in both signs, and compares against random directions using gated flip rate.
  • Steering and gated flip rate: The v2 grid covers four anchors crossed with math800, code800, and fact800, producing a 48-cell deterministic intervention design.
  • Labeling protocol and provenance: Labels were LLM-assisted rather than fully human-adjudicated, with a second pass and first-author review of uncertain cases but no independent review of every row.

4 Core Findings

The paper finds that structural impossibility is linearly represented before generation, but this recognition signal is not routed through the canonical safety-refusal pathway. Steering demonstrates that recognition is behaviorally active, especially in structural math and code, while transfer to fact unanswerability is limited.

  • Recognition Exists: Mean AUC 0.939 shows that a one-dimensional A-null direction separates answerable from unanswerable prompts across 22 model–dataset cells.The range is [0.841, 0.993], indicating recognition across the 1.7B–70B main grid.
  • Recognition Is Not the Safety-Refusal Axis: Mean cosine 0.087 shows that the recognition direction is near-orthogonal to the canonical safety-refusal direction across the 22-cell main grid.The range is [0.020, 0.130], and same-space A-null controls preserve the near-orthogonal angle.
  • Recognition Is Not the Safety-Refusal Axis: The in-domain invalidity-aware behavior direction is only partially aligned with recognition and remains near-orthogonal to safety refusal.Strict refusal-only generations are absent, and refusal-only gated effects are nearly zero, indicating that invalidity-aware behavior is not simply refusal vocabulary.
  • The Recognition Direction Causally Controls Invalidity-Aware Behavior: Mistral-7B shows bidirectional anchor-quality steering in both structural domains, with gated ∆G of +33pp to +44pp across math and code directions.The Mistral code A→U gated flip rate rises monotonically with steering magnitude, while the matched random-direction baseline is near zero.
  • The Recognition Direction Causally Controls Invalidity-Aware Behavior: Steering effects vary with model tolerance: all 10 anchor-quality structural directions reach +30pp with degeneration no higher than 16%, whereas most sub-threshold directions show much greater degeneration.The authors describe the recognition-to-behavior association as post hoc and correlational, not as a confirmed causal account of cross-model variation.
  • Scope and Limitations: Fact800 is a transfer boundary because clean unanswerable baselines rarely abstain, making U→A effects small-sample anecdotes rather than comparable structural evidence.The limitation reflects the absence of a passage-grounded abstention baseline, not necessarily the absence of fact-domain recognition.
  • Core Interpretation: Together, the geometry and intervention results support a routing account: recognition is a behaviorally active handle, but safety refusal does not read that axis.This conclusion is restricted to the formal structural-impossibility setting, chiefly math800 and code800.

5 Ruling Out Alternatives and Mapping Scope

Five controls support the recognition findings and establish a scope boundary: structural math and code are the main setting, while fact800 and FalseQA serve as transfer boundaries.

  • Five alternative explanations are ruled out: prompt length, answerable-class difficulty, a few easy categories, scale, and a generic unanswerability axis.
  • Structural math and code remain the paper’s main empirical setting.
  • Fact800 and FalseQA are treated as transfer boundaries rather than central evidence.

6 Related Work

Related work distinguishes epistemic unanswerability, false-premise questions, and structural impossibility, while this paper studies pre-generation recognition and its relation to refusal geometry.

  • Epistemic unanswerability concerns answers not recoverable from provided evidence, whereas false-premise questions presuppose false facts and should be challenged.
  • Structural impossibility is studied where ground truth is rule-verifiable, allowing the paper to test whether a no-answer signal exists before generation.
  • The paper asks whether impossibility recognition coincides with canonical safety refusal and whether it drives invalidity-aware abstention.
  • Prior probing work reads hidden states for truthfulness, hallucination, latent knowledge, or future non-factual responses, while also documenting generalization limits.
  • Prior refusal-geometry work studies safety directions, concept cones, harmful-intent geometry, and recognition-execution axes within refusal pathways.
  • Unlike safety-hardening or task-boosting steering, this paper uses steering to test whether the impossibility direction affects invalidity-aware behavior.

7 Discussion & Conclusion

The paper concludes that confident answers to structurally impossible math and code prompts reflect routing failure rather than missing impossibility encoding. Recognition is present before generation but is not reliably connected to abstention through the trained safety-refusal pathway.

  • Structural math and code failures are better explained as routing failures than encoding failures.
  • The model contains a pre-generation impossibility signal, but it is near-orthogonal to safety refusal and only partially coupled to behavior-defined invalidity awareness.
  • The low-cosine geometry predates instruction tuning, while steering changes invalidity-aware behavior on structural math and code cells.
  • Strict refusal-only language is essentially absent, and auxiliary AbstentionBench-GSM8K and FalseQA results are detection-only boundary tests.
  • The paper proposes testing whether similar recognition-action separations affect hedging, refusal, or format adherence in other settings.
  • Direct structural-impossibility supervision may be needed because controlled post-training left the Llama-70B angle essentially unchanged at ∆cos ≈−0.0001.

Limitations

The evidence is deliberately scoped to structural impossibility in open-weight transformer-family text decoders, with intervention and evaluation limitations affecting broader interpretation.

  • Causal evidence is cleanest on structural impossibility, while unanswerability more generally is not established by the measured interventions.
  • The analysis uses single directions, heuristic PCA k = 100, and labeled A-class prompts, leaving nonlinear, distributed, and alternative-direction mechanisms open.
  • The 11-model main grid covers open-weight transformer-family checkpoints in a text-only language-decoder setting, not all contemporary model families.
  • Whether the geometry and dose-responsive steering persist in other settings requires separate validation.
  • The high-rigor intervention grid contains four anchors, while broader behavioral evidence uses a coarser regex-based proxy across 16 models and three datasets.
  • The intervention labels are LLM-assisted rather than fully human-adjudicated, and the second-pass audit changed some provisional flips.

Ethical Considerations

The paper identifies practical and methodological boundaries around its structural-impossibility analysis, while emphasizing risks from manipulating refusal behavior and limitations in archival or externally sourced evidence.

  • Deployment should evaluate both erroneous answering and erroneous abstention, preserve user-visible uncertainty, and restrict intervention controls to authorized settings.The same steering diagnostics that may reduce confidently wrong outputs could also suppress warranted abstention without users’ knowledge.
  • FalseQA evidence is scope-defining rather than a clean external benchmark because the source lacks an explicit license and the cleaned file includes a documented local deduplication rewrite.The reported token-level length AUC is 0.526, and the artifact provides a fetch-and-clean script instead of redistributing the file.
  • +22.5pp MeanDiff AUC over a full-space baseline reflects improved accessibility for a one-dimensional cosine probe, not proof that impossibility exists only in the A-null subspace.Full-space linear SVMs match or exceed null-space counterparts in 3 of 4 representative cells, while MeanDiff Null exceeds Full in 3 of 4.
  • The legacy layer-emergence experiment is a heuristic, 3-model math-focused pilot and does not establish an 8-model-scale result or underpin the paper’s main conclusions.Layer selection was informed qualitatively by inverted-U curves, including a Qwen-7B AUC decline from 0.963 at L17–19 to 0.845 at the final layer.
  • The archival max-α steering envelope is an existence proof rather than an operating recommendation because stronger refusal can carry a proportional selectivity cost.For Qwen-14B×math800, max-α = 30 yields ∆refU = +1.00 and ∆wrongA = +1.00, meaning the model refuses both classes.

H Negative Results

Across several negative analyses, structural-impossibility recognition is concentrated in a coherent representation direction rather than internal component mismatches, output uncertainty, or unsupervised axes. Full-space and energy-decomposition checks further show that apparent alignment with safety refusal is largely shared answerable-structure variance.

  • Internal-mismatch hypotheses: None of 12 internal-mismatch hypotheses reached usable discriminative performance on structural-unanswerability prompts.The tested hypotheses included disagreements among attention, gradient, representation, and output-distribution signals.
  • Other negative findings: Replication changed the cross-dataset τ–AUC correlation from ρ = −0.950 to ρ = −0.147, p = 0.65, n = 12.Within-dataset NS-SNR versus detection AUC, with ρ = 0.76, replaced τ as the operative geometric summary.
  • Other negative findings: Output semantic entropy reached AUC = 0.625, versus AUC = 0.841 for representation geometry and mean AUC = 0.924 for dimp across instruct models.The comparison indicates that output-level uncertainty carries less impossibility information than residual-stream null-space geometry.
  • Unsupervised discovery: An unsupervised direction achieved AUC = 0.675, substantially below the labeled MeanDiff direction, indicating that A/U labels are needed to recover the impossibility axis.The unsupervised methods were applied to the same A-null residuals consumed by CosNSRT.
  • Geometry checks: The PC-PC component accounts for mean 0.813 of full-space cosine magnitude, while null-null share contributes only 0.187 across the 22 instruct cells.PC-PC share is ≥0.70 on 19/22 cells, and null-null share never exceeds 0.40.
  • Pipeline validation: The v2 audit reduced or invalidated many legacy steering effects because mixed-output, degenerate, lexical, and small-N artifacts inflated the v1 proxy.The mixed-output guard overrode 10 of 37 candidate flips on Mistral code, while 14 of 48 breadth-sweep cells became non-positive under v2.

J.4 Dose, Degeneration, and Tolerance-Window Diagnostics

Dose diagnostics show robust structural steering in anchor-quality directions, but degeneration and tolerance vary substantially across models and directions. Re-aggregated breadth results preserve a math > code > fact gradient while reducing legacy effect magnitudes under stricter invalidity-aware evaluation.

  • Dose and degeneration: All 10 anchor-quality directions reach ∆G ≥ +30pp with signal degeneration at most 16%.The dose tables report signal-minus-random gated effects together with signal and random degeneration rates.
  • Dose and degeneration: Five of six sub-threshold directions reach at least 40% signal degeneration, while Qwen3-8B code A→U peaks at +22.4pp with 20.4% degeneration.Restricting doses below 25% degeneration leaves a positive best effect in every sub-threshold direction, from +6.7 to +23.5pp.
  • Tolerance windows: At α=20, signal/random degeneration is 9.5/4.0% for Mistral-7B, 31.9/33.3% for Gemma-3-4B, 45.5/32.6% for Qwen3-14B, and 31.5/19.8% for Qwen3-8B.Matched-norm random branches reproduce the Mistral-versus-rest tolerance gap, although signal branches show additional direction-specific collapse.
  • Tolerance windows: The eight-cell exploratory correlation between behavior coupling and minimum U→A dose is ρ = −0.86, but it does not predict best effect size.Model tolerance, layer depth, and post-training design remain confounded, so the tolerance-window account does not identify a cause.
  • Breadth sweep: Under v2 re-aggregation, mean hallucination-rate reduction is +0.13 on math800, +0.09 on code800, and +0.004 on fact800.The resulting math > code > fact gradient is a hallucination-reduction metric, not a signal-minus-random gated effect size.

K Behavior-Defined Invalidity-Aware Direction

The behavior-defined invalidity-aware direction provides an in-domain comparison for recognition, while strict refusal language is absent on structurally impossible prompts. Base/instruct analyses show that recognition and low-cosine geometry are already present before instruction tuning.

  • Scope and measurement: Fact800 U→A rows with gated denominators ≤4 are anecdotal and excluded from headline tables; all 16 such rows fall in fact U→A.No math or code cell, and no A→U fact cell, is anecdotal under this rule.
  • Construction: The behavior-defined comparison uses prompt last-token hidden states and contrasts U prompts whose clean generations are invalidity-aware with those answered anyway.Cosines are evaluated both in the shared A-null subspace and in full space, with bootstrap uncertainty applied to the behavior labels.
  • Behavior-defined direction: Strict refusal-only behavior occurs in 0 of 50 U-class prompts in every one of the eight intervention cells.This sparsity motivates the broader invalidity-aware criterion used for the behavior-defined direction.
  • Base/instruct comparison: Base-model recognition reaches mean AUC 0.977 across six paired checkpoints, comparable to the instruct main grid.The base-model probe uses one A-null MeanDiff direction on math800 at each checkpoint’s matched layer.

L Multi-Dimensional Geometry Robustness

A multidimensional check confirms that impossibility recognition is concentrated in a low-dimensional subspace, while safety-refusal overlap remains small in the shared A-null space. The result supports low overlap without ruling out nonlinear or more distributed abstention representations.

  • Impossibility subspace: The impossibility subspace captures 0.89–0.99 of dimp’s energy, and its first component remains discriminative with held-out AUC 0.74–0.93 in all four cells.The subspace is constructed from top-k singular directions of U-class deviations from the A-class mean in A-null space.
  • Refusal overlap: Projecting dref,safety into the impossibility subspace captures only 1.2–2.1% of its energy at k=5 and 2.0–3.4% at k=10.Random-direction means are 0.1–0.25%, so refusal overlap is small but above chance.
  • Refusal overlap: The smallest behavior-verified principal angle is at least 74° in every cell, with first-principal cosine 0.14–0.27 versus random means 0.05–0.09.The projector-energy measurements are the primary same-space evidence because the angle compares an A-null impossibility subspace with a full-space refusal subspace.
  • Interpretation: Low overlap survives the 5–10-dimensional linear-subspace analysis, although nonlinear or more distributed abstention representations remain possible.The conclusion is therefore restricted to the measured linear subspaces and does not exclude other abstention mechanisms.

M Robustness Controls (Ruling Out Alternatives)

The robustness controls rule out length, category composition, and model-scale artifacts as explanations for impossibility detection and its orthogonality pattern. Additional transfer and subspace analyses argue that the signal is not merely a generic unanswerability axis.

  • Length controls: Length matching leaves CosNSRT detection at AUC 0.799 while a length-only classifier falls to AUC 0.500.This holds on the length-matched AbstentionBench-GSM8K subset, neutralizing length as a classifier.
  • Category controls: Within-cell NS_SNR and per-category CosNSRT AUC correlate at mean Spearman ρ 0.730, with 18/22 cells reaching p < 0.05.The global 22-cell mean therefore is not attributed only to a few easy categories.
  • Scale controls: Detection AUC ranges 0.841–0.993 across 22 cells, while orthogonality strengthens with scale rather than disappearing.The 24B Mistral-Small math800 cosine is 0.065 with bootstrap CI [0.059, 0.071], and 32B/70B cosines remain near 0.098.
  • Specificity controls: The mean within-minus-cross directional drop is 0.080 across 22 comparisons, arguing against a single generic unanswerability axis.The reported range is [−0.005, 0.207].
  • Subspace controls: The robustness analysis compares impossibility and refusal subspaces using energy capture, projected refusal energy, minimum angle, and random-subspace baselines.It evaluates subspace dimensionalities k ∈ {5, 10}.
  • Specificity controls: Math800-fitted impossibility directions transfer partially to AbstentionBench-GSM8K and FalseQA under dot-product NSRT scoring.The auxiliary earlier-grid check covers 8 models and reports ranges of 0.64–0.98 and 0.59–0.90, respectively.
Loading 2608.29109v1…