PaperlayerView exact revision on arXiv

Source-linked AI summary

When Decodability Is Not Enough: Logical Validity Representations, Behavioral Dissociation, and Causal Tests in Language Models

Smitha Muthya Sudheendra, Jaideep Srivastava

arXiv:2609.02438v1cs.CLcs.LG

TL;DR

The paper asks whether logical validity is represented internally when behavioral answers are unreliable, and whether that representation generalizes and causally affects decisions. Using controlled matched valid–invalid pairs across five open-weight transformers, it finds strong hidden-state decodability despite near-chance behavior, but limited uniformity across shifts and weak, nonspecific intervention effects. The findings support treating representation, behavioral expression, and causal use as distinct forms of evidence.

  • Problem

    Correct or incorrect answers alone provide limited evidence about what language models represent internally or how those representations are used.

  • Method

    The study evaluates five open-weight transformers on controlled matched valid–invalid premise–claim pairs while testing held-out generalization, error-conditioned decodability, and interventions on probe-derived validity directions.

  • Results

    Validity is strongly decodable from hidden states despite near-chance behavior, remains decodable on many held-out conditions and behavioral errors, but shows leave-one-out exceptions and weak, nonspecific causal effects.

  • Takeaways & Limitations

    Validity-related information can be recoverable from hidden states without being reliably expressed in output behavior or serving as a simple causal control mechanism.

  • Takeaways & Limitations

    The controlled synthetic dataset limits direct generalization to naturalistic reasoning, longer contexts, open-ended tasks, and the broader space of logical reasoning.

Abstract

from arXiv · show

Large language models can look capable of logical reasoning, but correct or incorrect answers alone tell us little about what the model represents internally. We study logical verification in five open-weight transformer models using matched valid--invalid premise--claim pairs that vary across inference families, semantic domains, templates, and difficulty levels. Despite near-chance behavioral performance, logical validity is often almost perfectly decodable from hidden states and remains strongly decodable under held-out templates, domains, and inference families. Validity also remains highly decodable on behaviorally incorrect examples in the conditions where correctness-conditioned evaluation is well defined. At the same time, exhaustive leave-one-out tests reveal clear limits to this generalization, and interventions along probe-derived validity directions have only weak, nonspecific effects compared with random controls. Our results suggest that representing validity, expressing it in behavior, and using it causally are distinct. Validity related information can be strongly decodable from a model's hidden states without being reliably expressed in its output.

1 Introduction

The paper separates behavioral success from what models represent internally and whether those representations causally influence decisions. In logical verification, validity is often strongly decodable despite near-chance behavior, but generalization and causal control remain limited.

  • Research questions: The study separates correct validity judgments, linear accessibility of validity in hidden states, and causal influence on final decisions.It evaluates five open-weight transformers on 800 matched examples spanning five inference families, five semantic domains, and three difficulty levels.
  • Main findings: Behavioral verification remains at or near chance across all five models, often with strong answer-label preferences.The evaluation uses behavioral likelihoods rather than accuracy alone to characterize these output patterns.
  • Main findings: Validity is almost perfectly linearly decodable in-distribution and remains strongly decodable across held-out templates, domains, and many inference-family shifts.Exhaustive leave-one-out tests nevertheless reveal recurring failures, including transfer to unseen syllogistic reasoning.
  • Main findings: Validity remains highly decodable on incorrectly answered examples where correctness-conditioned evaluation is well defined.Probes also preserve the ordering between valid and invalid members of matched pairs despite shifts in global calibration.
  • Main findings: Interventions along the probe-derived validity direction produce only small and inconsistent output changes comparable to or smaller than random controls.The results therefore distinguish decodability from a strong causal control variable for verification behavior.

2 Related Work

Prior work shows that reasoning benchmarks and internal probes provide useful but incomplete evidence about language-model reasoning. This paper responds by testing validity generalization, persistence on errors, and causal influence separately.

  • Reasoning evaluation: Behavioral reasoning benchmarks show progress while remaining sensitive to task formulation and prompting.More targeted benchmarks test deductive reasoning and systematic generalization across logical structures and distribution shifts.
  • Internal representations: Logical validity differs from factual truth because it depends on the relation between premises and a conclusion rather than the conclusion’s truth alone.The matched-pair construction is designed to isolate this relational property.
  • Probing and causality: High probe accuracy demonstrates linear accessibility, not necessarily that the model uses the decoded feature in its reasoning process.Probe performance can reflect nuisance variables or task-format artifacts unless controlled with matched pairs and held-out evaluations.
  • Probing and causality: The study asks whether validity representations generalize beyond probe-training conditions, persist on wrong answers, and causally affect final behavior.These questions distinguish internal representation from output expression and causal use.

3 Controlled Verification Dataset

The controlled verification dataset isolates logical validity as a relation between premises and candidate claims while varying inference structure, semantic content, templates, and difficulty. Matched valid–invalid pairs support within-context comparisons and distribution-shift tests.

  • Task and structure: Each example pairs premises with a candidate claim, and the task is to determine whether the claim follows from the premises.Gold validity labels are assigned deterministically from known inference rules.
  • Task and structure: The dataset spans five inference families and five semantic domains, using shared structures across domains to test content dependence.The families are syllogism, transitivity, set inclusion, causal chain, and permission logic.
  • Task and structure: Three difficulty levels vary relational integration while keeping the output task unchanged: one-step inference, two-step chains, and chains with an irrelevant distractor premise.The levels contain direct verification, multi-step reasoning, and distractor handling.
  • Matched pairs: Matched valid–invalid pairs hold premises, inference family, domain, template, and difficulty fixed while changing only the candidate claim.This reduces label-correlated context differences and enables a direct within-context representation test.
  • Dataset composition: The final dataset contains 800 examples, equally divided between valid and invalid inferences, with 160 examples per inference family and domain.The strict template split uses 600 training examples and 200 held-out examples.
  • Generalization design: Random, template-held-out, domain-held-out, and inference-family-held-out conditions test transfer across distribution, wording, semantic content, and reasoning structure.These conditions probe different kinds of generalization rather than forming a strict difficulty hierarchy.
  • Scope: The dataset is diagnostic rather than comprehensive, covering only a limited portion of logical reasoning while separating output expression, hidden-state accessibility, generalization, and error-conditioned accessibility.Its controlled design enables matched comparisons and distribution-shift analyses.

4 Experimental Setup

The experiments measure behavioral validity judgments, hidden-state linear accessibility, and causal effects of probe-derived directions. They combine held-out evaluations and controls designed to distinguish validity information from lexical, metadata, or arbitrary linear signals.

  • Models and behavior: Five open-weight transformers are evaluated on the same 800 examples and split definitions, with causal interventions run on three models.Behavior is measured from likelihoods assigned to VALID and INVALID.
  • Models and behavior: Behavioral evaluation reports accuracy, the fraction of VALID predictions, and margin AUROC from the continuous VALID–INVALID output margin.Answer-label tokenization is audited separately for each model.
  • Probing: Layer-wise ℓ2-regularized logistic probes use final prompt-token hidden states to predict gold validity, with preprocessing and layer selection restricted to probe-training data.The selected probe is evaluated once on the held-out test set.
  • Generalization: Probe evaluation includes random, template-held-out, domain-held-out, inference-family-held-out, and exhaustive leave-one-out splits.Matched-pair accuracy and correctness-conditioned AUROC test accessibility within pairs and on behavioral errors.
  • Controls: TF–IDF, metadata-only, and shuffled-label controls test whether probe performance reflects lexical regularities, construction metadata, or arbitrary separability.These controls complement the hidden-state probes.
  • Causal tests: Interventions modify the final prompt-token state along a normalized probe direction across α ∈{−4, −2, −1, 0, 1, 2, 4} and measure output-margin changes.The learned direction is compared with five norm-matched random orthogonal directions and matched-projection patching.

5 Results

Across five models, behavioral verification stays near chance while validity is strongly decodable from hidden states, including many held-out conditions. Generalization is broad but uneven, and probe-derived interventions have weak, nonspecific behavioral effects.

  • 5.1 Behavioral Verification Is Unreliable: Near-chance behavioral accuracy masks substantially different model-specific answer-label preferences.Pythia-1.4B and Pythia-2.8B mostly predict INVALID, SmolLM3-3B and Mistral-7B mostly predict VALID, while Llama-3.2 reaches 0.470 accuracy.
  • 5.2 Validity Is Strongly Decodable from Hidden States: Random-split validity-probe AUROC is essentially perfect, with first-block AUROC ranging from 0.935 to 0.995 across models.Validity-related information is linearly accessible very early in the network.
  • 5.2 Validity Is Strongly Decodable from Hidden States: Held-out-template AUROC ranges from 0.963 to 0.999, while most domain- and inference-family-held-out evaluations also remain strong.Selected probe layers vary from normalized depth 0.214 for Llama-3.2 to 0.556 for SmolLM3-3B.
  • 5.3 OOD Generalization Is Broad but Not Uniform: Exhaustive leave-one-out tests reveal nonuniform transfer, with syllogistic reasoning the weakest unseen inference family for every model.Domain failures are model-specific; syllogism AUROC falls to 0.525, 0.687, 0.544, 0.500, and 0.762 across the five models.
  • 5.4 Validity Remains Decodable When Behavior Fails: Matched-pair accuracy is 1.000 across all five models and four primary splits, even when global AUROC deteriorates.Mistral-7B ranks every valid member above its invalid counterpart despite 0.771 global AUROC under domain holdout.
  • 5.5 Controls Qualify the In-Distribution Result: Near-perfect in-distribution probe performance is not sufficient evidence for a general validity representation because TF–IDF reaches 0.970 full-prompt AUROC and 0.965 claim-only AUROC.Lexical baselines weaken under distribution shift, while hidden-state probes remain comparatively strong.
  • 5.5 Controls Qualify the In-Distribution Result: Within Pythia, scaling improves domain- and family-held-out performance but reduces template-held-out AUROC from 0.999 to 0.963.The study therefore finds neither a uniform within-family scaling effect nor a monotonic behavioral difficulty gradient.
  • 5.6 Strong Decodability Does Not Imply Causal Control: At α = +4, probe-direction margin changes are −0.0037, −0.0022, and +0.0023, with inconsistent signs and no binary prediction changes.Random orthogonal perturbations produce comparable or larger effects, indicating weak and nonspecific control at the tested site.

6 Discussion

The findings separate validity decodability from behavioral expression and causal control. Validity can remain accessible in hidden states despite incorrect decisions, while generalization and intervention effects remain limited.

  • Validity remains decodable from hidden states even when the model answers verification examples incorrectly.This persists in matched-pair and correctness-conditioned analyses.
  • Leave-one-out evaluation reveals model-specific domain failures and recurring weakness when syllogistic reasoning is excluded from probe training.Validity-related structure appears partly shared across settings but is not fully invariant.
  • Probe-derived validity directions produce small and inconsistent behavioral changes, often no larger than random-direction controls.High probe accuracy therefore does not establish causal control over output behavior.
  • Linear accessibility should be distinguished from behavioral expression and causal control.

7 Limitations

The study’s conclusions are bounded by its controlled synthetic dataset, restricted probe location and form, and relatively small quantized model set. These choices limit direct generalization to broader reasoning settings and architectures.

  • The controlled synthetic dataset limits direct generalization to naturalistic reasoning, longer contexts, and more open-ended tasks.Its five inference families and semantic domains cover only a small part of logical reasoning.
  • Linear probes at the final prompt-token representation may miss nonlinear, distributed, or differently located validity information.Weak steering effects rule out only a relatively simple causal interpretation of the probe direction.
  • The experiments use relatively small open-weight transformers under quantization, so replication with larger models and broader architectures is needed to establish generality.

8 Conclusion

The paper studies logical verification at behavioral, representational, and causal levels, using controlled matched examples and cross-condition evaluations. Its conclusion is that validity information can be recoverable without being reliably expressed or causally controlling decisions.

  • 8 Conclusion: The study evaluates behavioral expression, representational accessibility, and causal use as distinct levels of logical verification.
  • 8 Conclusion: Validity remains strongly decodable on incorrectly answered examples where correctness-conditioned evaluation is well defined.
  • 8 Conclusion: Generalization is not universal, and probe-derived validity directions have little specific causal effect on output behavior.
  • A.1 Task Representation: The dataset contains 800 examples organized into 400 matched valid–invalid pairs.
  • A.1 Task Representation: Each example records premises, a candidate claim, validity, inference family, semantic domain, template family, and difficulty.The five inference families span five semantic domains.
  • A.1 Task Representation: Matched pairs keep the premise context and construction variables fixed while changing the candidate claim.Pairs remain together during splitting, probe validation, and pair-level bootstrap resampling.
  • A.1 Task Representation: The evaluation uses random, template-held-out, domain-held-out, and inference-family-held-out probe splits.Exhaustive leave-one-domain-out and leave-one-inference-family-out evaluations are also performed.
  • B.1 Behavioral Label Scoring and Tokenization: Behavioral scoring compares conditional likelihoods for VALID and INVALID while auditing answer-prefix tokenization.Prefix selection prioritizes equal token counts, then the shortest encoding.

C Probe Training and Statistical Details

The probe analysis extracts final prompt-token hidden states, selects regularized logistic probes and layers using training data, and evaluates held-out validity prediction with pair-aware statistics.

  • Probe fitting: Final prompt-token hidden states are extracted for each example and transformer block.
  • Probe fitting: Layer-wise probes use ℓ2-regularized logistic regression to predict gold validity.Features are standardized using only the probe-training partition.
  • Probe fitting: The regularization coefficient is selected from C ∈{0.1, 1, 10} with group-aware cross-validation by pair_id.
  • Probe fitting: Layer selection is performed entirely within training data before refitting the probe and evaluating once on the held-out partition.
  • Pairwise evaluation: Pairwise accuracy tests whether the probe assigns a larger validity score to the valid member of each matched pair.The probe score q(·) determines the within-pair ordering.
  • Statistical details: Leave-one-domain-out and leave-one-inference-family-out evaluations provide the selected-layer validity-probe AUROC analyses.
  • Statistical details: Probe performance is reported with AUROC, while matched-pair analyses use pairwise accuracy and confidence intervals from 2000 pair-level bootstrap resamples.Pair_ids are treated as the independent resampling unit.

D Full Generalization Results

Leave-one-out tests show that validity decodability generalizes broadly but unevenly across semantic domains and inference families. Syllogistic reasoning is the most consistent unseen-family weakness across models.

  • Inference-family generalization: Syllogism is the weakest unseen inference family for all five models.This pattern is more systematic than the domain-specific weaknesses.
  • Domain generalization: Domain-held-out weaknesses are model-specific: Pythia models weaken on unseen biological content, while Mistral-7B weakens on legal-policy.Llama-3.2 is described as the most stable across domains.
  • Overall pattern: The exhaustive leave-one-domain-out and leave-one-family-out results support substantial shared validity-related structure rather than a single representational pattern.The conclusion is qualified by the observed heterogeneity across models and held-out conditions.

E Behavior–Representation Dissociation Details

Matched-pair and correctness-conditioned analyses show that validity remains accessible in hidden states even when behavior is unreliable. Controls and interventions then narrow the interpretation: decodability is not sufficient evidence of causal behavioral control.

  • Behavior–representation dissociation: 1.000 matched-pair accuracy is achieved by selected probes across all five models and four primary evaluation conditions.The pair-level bootstrap confidence intervals are [1.000, 1.000].
  • Correctness-conditioned validity: 0.965 versus 1.000 AUROC is obtained by Pythia-2.8B on correct versus incorrect examples under domain holdout.Correctness-conditioned AUROC is undefined when a subset contains only one gold class, so those conditions are omitted.
  • Lexical and metadata controls: Lexical controls perform strongly in-distribution, while premises-only classification remains at chance because matched examples share their premise context.These controls show why random-split probe performance alone cannot distinguish transferable validity information from dataset regularities.
  • Shuffled-label control: 200 shuffled-label permutations yield null AUROC means of 0.496–0.499 with standard deviations of approximately 0.031–0.039.Observed random-split AUROC exceeds all 200 shuffled-label results for every model, with p = 1/201 ≈ 0.005.
  • Within-family comparison: 0.968 to 0.989 domain-held-out AUROC and 0.950 to 0.976 family-held-out AUROC accompany only modest changes in mean leave-one-family-out performance.The two-model Pythia comparison shows that larger parameter count improves some transfer measures but not all.
  • Within-family comparison: 0.999 to 0.963 template transfer decreases between Pythia-1.4B and Pythia-2.8B, rejecting a simple monotonic parameter-count relationship.This is a limitation of the narrow two-model within-family comparison.
  • Nominal difficulty: Behavioral accuracy does not decline monotonically across the three nominal difficulty levels.Several models remain at 0.500 across levels, while Llama-3.2 rises from 0.435 to 0.495 across its listed conditions.
  • Full intervention sweep: At α = +4, no binary prediction flips occur, and at α = −4 only one Llama-3.2 decision changes among 160 examples.The sweep compares probe-derived interventions with norm-matched random orthogonal directions.
Loading 2609.02438v1…