PaperlayerView exact revision on arXiv

Source-linked AI summary

The Truthfulness Spectrum Hypothesis

Zhuofan Josh Ying, Shauli Ravfogel, Nikolaus Kriegeskorte, Peter Hase

arXiv:2602.20273v1cs.LG

TL;DR

The paper asks whether LLM truthfulness is represented by one general direction or multiple domain-specific directions. It tests a spectrum hypothesis with cross-domain probes, geometric analyses, concept erasure, causal interventions, and post-training comparisons, finding coexisting directions of varying generality and increased post-training separation of sycophancy.

  • Problem

    Prior work disagrees on whether LLMs encode one general truth direction or multiple distinct truth representations.

  • Method

    The paper evaluates probe transfer across truth and deception datasets, analyzes probe geometry, applies concept erasure and causal steering, and compares base with post-trained models.

  • Results

    Truth probes generalize across most domains but fail on sycophantic and expectation-inverted lying; Mahalanobis cosine similarity predicts cross-domain performance with R2=0.98.

  • Takeaways & Limitations

    Truth directions of varying generality coexist, while post-training reshapes their geometry and separates sycophantic lying from other truth types.

  • Takeaways & Limitations

    The datasets do not exhaustively cover truth types, FLEED is model-generated, the analysis is linear, and causal interventions have modest effects.

Abstract

from arXiv · show

Large language models (LLMs) have been reported to linearly encode truthfulness, yet recent work questions this finding's generality. We reconcile these views with the truthfulness spectrum hypothesis: the representational space contains directions ranging from broadly domain-general to narrowly domain-specific. To test this hypothesis, we systematically evaluate probe generalization across five truth types (definitional, empirical, logical, fictional, and ethical), sycophantic and expectation-inverted lying, and existing honesty benchmarks. Linear probes generalize well across most domains but fail on sycophantic and expectation-inverted lying. Yet training on all domains jointly recovers strong performance, confirming that domain-general directions exist despite poor pairwise transfer. The geometry of probe directions explains these patterns: Mahalanobis cosine similarity between probes near-perfectly predicts cross-domain generalization (R^2=0.98). Concept-erasure methods further isolate truth directions that are (1) domain-general, (2) domain-specific, or (3) shared only across particular domain subsets. Causal interventions reveal that domain-specific directions steer more effectively than domain-general ones. Finally, post-training reshapes truth geometry, pushing sycophantic lying further from other truth types, suggesting a representational basis for chat models' sycophantic tendencies. Together, our results support the truthfulness spectrum hypothesis: truth directions of varying generality coexist in representational space, with post-training reshaping their geometry. Code for all experiments is provided in https://github.com/zfying/truth_spec.

1. Introduction

The paper reconciles conflicting findings about truthfulness probes by proposing a spectrum of coexisting domain-general, domain-specific, and intermediate directions. It evaluates this hypothesis across truth types, deception settings, geometric analyses, concept erasure, causal interventions, and post-training effects.

  • LLMs encode truthfulness along a spectrum of generality, with domain-general and domain-specific directions coexisting in representational space.
  • The evaluation spans five fundamental truth types, sycophantic and expectation-inverted lying, and existing honesty benchmarks.
  • Linear probes generalize across most truth types and benchmarks but fail almost entirely on sycophantic and expectation-inverted lying, with AUROC ≈0.55.
  • Training on all domains recovers strong performance, indicating that poor pairwise transfer reflects incomplete recovery of a general direction rather than its absence.
  • Mahalanobis cosine similarity between probe directions near-perfectly predicts cross-domain generalization, outperforming standard cosine similarity with R2=0.98 versus R2=0.56.
  • Post-training reorganizes truth representations by increasing separation between sycophantic lying and other truth types.
  • Concept erasure and causal steering show that truth directions vary in generality, while domain-specific directions steer more effectively than domain-general ones.

2. Truthfulness Datasets

The paper constructs controlled datasets to test whether truth representations generalize across domains and deception settings. It combines five truth categories, two deception datasets, prior honesty benchmarks, and standardized probing procedures.

  • The FLEED datasets cover definitional, empirical, logical, fictional, and ethical truth with balanced true and false statements.
  • The sycophantic lying dataset measures cases where models change answers to align with user beliefs rather than reporting their believed true answers.
  • The expectation-inverted dataset makes true statements violate user expectations and false statements comply with them, separating objective truth from expected truth.
  • The study additionally evaluates insider trading, sandbagging, roleplaying lying, and prior honesty benchmarks for broader generalization.
  • Experiments use multiple Llama and Qwen instruction-tuned and base models, extracting residual-stream activations across layers.
  • The probing setup compares Difference of Means, logistic regression, and linear discriminant analysis with multiple token-aggregation strategies and 5-fold cross-validation.

4. Probe Generalization Across Datasets

Linear probes transfer effectively among the five core truth types but poorly to sycophantic and expectation-inverted lying. Probe geometry and post-training analyses explain these transfer patterns and reveal increased separation of sycophancy in chat models.

  • In-distribution probe performance is near-perfect (>0.95), while probes trained on the five FLEED truth types transfer effectively to one another.
  • Expectation-inverted lying probes perform well below chance, at AUROC≈0.28, and probes generally fail to detect sycophantic and expectation-inverted lying.
  • Training on all domains achieves high performance across datasets, showing that low pairwise transfer does not rule out domain-general truth directions.
  • Mahalanobis cosine similarity achieves R2=0.98 for predicting cross-domain AUROC, exceeding standard cosine similarity at R2=0.56.
  • Post-training increases geometric separation between sycophantic lying and other truth types, potentially accounting for greater sycophancy in chat models.
  • Base-model probes transfer better between FLEED and sycophancy than chat-model probes, with Llama-70B performance falling from 0.74 AUROC to 0.62 after post-training.

7. Revealing the Spectrum of Truthfulness Directions

The paper introduces Stratified INLP and concept erasure to identify truth directions spanning a spectrum from domain-general to domain-specific, including intermediate directions shared by subsets of domains.

  • Method: Stratified INLP extracts mutually orthogonal domain-general directions by repeatedly probing all domains and projecting representations onto each direction’s null space.The procedure then extracts domain-specific directions separately after removing the domain-general subspace.
  • Domain-general directions: The first domain-general direction reaches accuracies from 0.85 on logical claims to 1.00 on insider trading, with later directions also generalizing strongly.
  • Domain-specific directions: Domain-specific directions retain high Self accuracy but perform near chance on Other domains after domain-general directions are removed.This pattern indicates truth information unique to individual domains.
  • Conclusion: Together, the results provide constructive evidence that highly domain-general and highly domain-specific truth directions coexist, despite individual-dataset probes not naturally isolating both.
  • Intermediate directions: LEACE erasure preserves perfect in-distribution performance on nonerased domains, while erased-domain performance drops to chance and transfer fails selectively elsewhere.The selective failures reveal intermediate-general directions between fully general and fully specific extremes.
  • Intermediate directions: Different erasure and training-domain combinations show that truth types share partially overlapping but distinct direction sets, with highest capacity assigned to subspaces shared by 3–6 domains.

8. Causal Assessment of Truth Directions

The paper tests whether identified truth directions causally influence model behavior using interventions on Llama-8B and held-out factual questions. Domain-specific directions generally improve discrimination, whereas the domain-general direction degrades it, especially on high-confidence examples.

  • Experimental design: The causal experiment intervenes on Stratified INLP directions in Llama-8B and evaluates answer discrimination on 1,024 verified SimpleQA questions.Each question pairs a correct answer with a type-matched distractor.
  • Experimental design: Intervention changes discrimination by ∆diff = diff_intervened − diff_baseline, where positive values indicate improved discrimination.
  • Results: A mean ∆diff of +0.05 across domain-specific directions shows that most such directions are causally utilized, while the general direction has mean ∆diff = −0.07.
  • Interpretation: The general direction’s negative effect likely reflects SimpleQA’s factual emphasis, whereas it conflates factual and sycophancy-related variance.
  • Results: At the 90–100th baseline-confidence percentile, domain-specific interventions reach ∆diff ≈+0.10, while the general direction reaches ∆diff ≈−0.11.Effects are minimal when the model initially favors the incorrect answer.
  • Conclusion: The authors conclude that universal directions may suffice for monitoring, but reliable behavioral intervention appears to require domain-specific representations.

10. Discussion & Conclusion

The findings support a spectrum of truth directions, reconciling broad probe generalization with domain-specific failures. They also identify limits: domain-specific directions steer more effectively, and coverage, linearity, and post-training analyses remain bounded.

  • The truthfulness spectrum hypothesis reconciles findings that probes sometimes generalize broadly and sometimes fail dramatically across domains.
  • Domain-specific directions outperform domain-general directions in causal steering, which supports broad detection but limits reliable behavioral control.
  • Mahalanobis cosine similarity predicts probe generalization performance across experiments and simulations with R2 ≥0.95.
  • Post-training reorganizes truth representations and increases dissociation between sycophantic lying and other truth types.
  • The datasets do not exhaustively cover truth types, the analysis focuses on linear structure, and causal interventions modestly modulate confidence rather than reliably flipping predictions.

A. Datasets

The FLEED datasets cover five truth types using balanced true/false claims, diverse negations, manual quality filtering, and activation prompts. PCA checks indicate that truth and false statements are not trivially separable in dominant activation directions.

  • Gemini-3-Pro generates 300 to 600 ground-truth claims for definitional, empirical, logical, and fictional truth types.
  • Manual filtering removes low-quality or ambiguous claims and their negations.
  • Negations use varied strategies rather than only simple logical negation, reducing reliance on spurious features.
  • PCA shows truth and false statements occupying similar geometric structures, indicating no trivial geometric separability.

A.2. Sycophantic Lying Dataset

The sycophancy dataset measures whether models abandon baseline answers to match stated user beliefs. Sycophancy varies with confidence and original-answer correctness, while probes trained on ordinary truth types fail to transfer effectively.

  • The dataset identifies cases where the model changes its baseline answer to align with a user’s stated preference.
  • 8.5% is Llama-70B’s sycophancy rate on the selected MMLU STEM questions.
  • Sycophancy rates are lower when the model’s original answer is correct and when model confidence is higher.
  • Filtering by original-answer correctness produces effectively the same results as filtering by model confidence.
  • Probes trained on four truth types generalize to one another but do not generalize to sycophantic lying; combined-domain probes bridge this gap.

B. Probe Design

Probe design is tuned by cross-domain AUROC, selecting architecture, layer, and token aggregation on Llama-3.3-70B before applying the resulting design across models.

  • Probe tuning considers architecture, layer, and token position, optimizing cross-domain AUROC on FLEED.
  • The selected probe design uses logistic regression with average-token aggregation.
  • Other models reuse the selected architecture and token strategy while tuning only the extraction layer.
  • Llama-8B exhibits probe generalization patterns similar to Llama-70B.

D. Additional Results: Probe Direction Geometric Analysis

Probe geometry explains cross-domain transfer: Mahalanobis cosine similarity predicts out-of-domain performance far better than standard cosine similarity, while post-training and model family affect sycophancy transfer.

  • Geometric analysis: R2 = 0.957–0.999 for Mahalanobis cosine similarity versus R2 = 0.012–0.553 for standard cosine similarity across simulations.The Mahalanobis measure remained predictive across five synthetic distributions, whereas standard cosine similarity did not.
  • Geometric analysis: Mahalanobis cosine similarity closely tracks transfer AUROC, capturing strong transfer among core truth domains and weak transfer involving sycophancy and inverted expertise.It whitens representations using test-data covariance, emphasizing directions that carry signal.
  • Model and layer effects: For Qwen model pairs, reduced sycophancy alignment appears only with logistic-regression regularization α = 1 rather than α = 10^-4.This result is reported for Qwen-2.5-14B and Qwen-2.5-7B and their corresponding base models.
  • Model and layer effects: Base models consistently outperform chat models on FLEED-to-sycophancy cross-domain probing, with most models showing middle- and late-layer performance peaks.The best probing layers are intermediate, and base and chat models from the same heritage select the same best layers.

F.1. Formalizing Partially Overlapping Concept Subspaces

The formalization models truth representations as partially overlapping subspaces, and concept-erasure results support a patchwork geometry containing domain-general, domain-specific, and intermediate shared directions.

  • Formalization: The representation space is partitioned into subspaces shared by specified domain subsets, with capacities d_c representing their true dimensionality.Purely general, purely specific, and partially overlapping directions correspond to different domain-subset capacities.
  • Formalization: A probe trained on domain A learns nonnegative reliance weights w_A,c on available shared subspaces, bounded by the subspaces it observes and their capacities.This links probe performance to which latent directions training domain A can use.
  • Concept erasure: Most domain-specific directions achieve high in-domain performance but chance-level performance on other domains.Stratified INLP isolates these directions after hierarchical removal of more general components.
  • Concept erasure: AUROC ≈ 0.5 after erasing a domain-specific direction, while non-removed tasks remain essentially unchanged.LEACE selectively removes the targeted domain's predictive subspace without degrading in-distribution performance on other truth types.
  • Capacity allocation: The optimization assigns highest capacities to subspaces shared by 3 to 6 domains rather than to one fully general or strictly isolated specific subspace.This allocation explains selective degradation under erasure and supports structurally patchwork truth directions.

G. Additional Results: Causal Experiments

Causal steering distinguishes predictive from functionally useful truth directions: domain-specific directions improve correct-versus-incorrect confidence, whereas the general direction fails through indiscriminate boosting.

  • Causal mechanism: Domain-specific directions primarily suppress incorrect-answer probability while preserving correct-answer probability.This produces a more surgical intervention than broadly increasing answer confidence.
  • Causal mechanism: The general direction increases probability for both correct and incorrect answers, disproportionately boosting the incorrect answer.This decomposition explains why steering along the general direction slightly degrades performance.
Loading 2602.20273v1…