PaperlayerView exact revision on arXiv

Source-linked AI summary

Discovering Latent Knowledge in Language Models Without Supervision

Collin Burns, Haotian Ye, Dan Klein, Jacob Steinhardt

arXiv:2212.03827v2cs.CLcs.AIcs.LG

TL;DR

Training objectives and human evaluations can leave language-model outputs misaligned with truth, motivating methods that do not depend on external supervision or model-generated answers. The paper introduces CCS, which searches unlabeled hidden activations for a direction consistent across statements and their negations. Across 6 models and 10 question-answering datasets, CCS outperforms zero-shot baselines by 4% on average and remains accurate under deliberately incorrect prompting.

  • Problem

    Training objectives and human evaluations can produce language-model outputs that are false or misaligned with truth, while external supervision is not always feasible.

  • Method

    CCS learns a lightweight linear probe from unlabeled hidden representations by enforcing confidence and consistency across each statement and its negation.

  • Results

    Across 6 models and 10 question-answering datasets, CCS outperforms strong zero-shot baselines by 4% on average, with accuracy preserved under deliberately incorrect prompting.

  • Takeaways & Limitations

    The results provide an initial demonstration that latent knowledge can be recovered from language-model representations independently of what models say and without ground-truth labels.

  • Takeaways & Limitations

    CCS requires a truth-separating activation direction and assumes the model both can and actively does evaluate the truth of its inputs.

Abstract

from arXiv · show

Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations. It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4\% on average. We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers. Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don't have access to explicit ground truth labels.

1 INTRODUCTION

Language models can contain representations of truth even when training objectives or outputs are misaligned with truth. CCS searches these internal representations without supervision by exploiting consistency between statements and their negations, recovering knowledge across diverse evaluations.

  • Motivation: Training objectives can produce false outputs when imitation, engagement, or human evaluation rewards conflict with truth.These failures arise from misalignment between the objective and truth, rather than necessarily from a lack of model capability.
  • Approach: CCS searches for implicit internal knowledge instead of externally specifying truth with human supervision.The method uses logical consistency properties that truth representations are expected to satisfy.
  • Approach: CCS learns a linear projection of hidden states that remains consistent across negated versions of each yes-no question.It computes probabilities for both forms and searches for predictions that are confident and consistent, without labels or model outputs.
  • Approach: CCS answers questions from unlabeled model representations and can learn accurate predictions without labels or model outputs.The method is illustrated on the unlabeled COPA train split using UnifiedQA.
  • Results: 4% average accuracy improvement over strong zero-shot baselines was achieved across 6 models and 10 question-answering datasets.CCS also cuts prompt-sensitivity standard deviation in half and retains its accuracy when models are prompted to answer incorrectly.
  • Results: CCS reveals task-agnostic truth representations that may differ from the features used by model outputs.The method can work best in middle layers and often finds truth representations with little data or via a top principal component.

2 PROBLEM STATEMENT AND FRAMEWORK

The framework formulates latent-knowledge discovery as answering yes-no questions from unlabeled pretrained-model representations, without relying on correct model outputs, labels, or weight updates. Contrast-Consistent Search constructs paired statements and learns a lightweight probe whose predictions satisfy negation consistency and confidence properties.

  • Problem setup: The setup covers questions with well-defined binary answers, including arithmetic and factual questions, and can extend to evaluating the truth of statements.The two mutually exclusive answers need not be “Yes” and “No”; they can also be task-specific labels such as sentiment labels.
  • Problem setup: CCS takes yes-no questions and fixed pretrained-model representations as input, producing a lightweight probe that answers new questions without labels or modifying model weights.The representations are assumed to contain information that could support a high-accuracy supervised probe if ground-truth labels were available.
  • Contrast pairs: CCS constructs a contrast pair for each question by expressing it once with a “Yes” answer and once with a “No” answer.The resulting statements are classified as true or false, with the truth of each paired statement corresponding to the appropriate answer to the original question.
  • Feature extraction and normalization: The method extracts and normalizes representations of each contrast-pair statement to reduce differences caused by the answer wording itself.Normalization is intended to remove the salient answer-format difference so the learned direction can capture which statement is true rather than whether it ends with “Yes” or “No”.
  • Probability mapping: A linear sigmoid probe maps normalized hidden states to probabilities that the corresponding statements are true.Nonlinear projections can also work, but the paper uses a linear projection followed by a sigmoid for simplicity.
  • Training objective: CCS trains the probe with consistency and confidence objectives: paired statements should receive complementary probabilities, while predictions should avoid the all-0.5 solution.The final loss averages the sum of these objectives across contrast pairs; both losses are necessary because either one alone admits a degenerate solution.
  • Training objective: The final unsupervised loss combines negation consistency with a confidence constraint corresponding to the law of excluded middle.The confidence objective encourages each statement to be treated as either true or false rather than assigned an intermediate probability.
  • Prediction: After training, CCS averages the two paired probabilities and predicts “Yes” when the resulting value exceeds 0.5.Because the unsupervised objective does not identify which cluster corresponds to “Yes,” evaluation considers both possible label assignments.

3 RESULTS

Across diverse models and datasets, CCS recovers accurate answers from unlabeled representations, remains robust to misleading outputs, and often transfers across tasks with limited data. Its performance suggests that truth-related directions are accessible in model activations and distinct from output behavior.

  • 3.2.1 CCS OUTPERFORMS ZERO-SHOT: 71.2% average accuracy versus 67.2% for calibrated zero-shot shows CCS can classify examples using unlabeled model representations.CCS outperformed zero-shot accuracy for every model except RoBERTa and T0 under the reported comparison.
  • 3.2.2 CCS IS ROBUST TO MISLEADING PROMPTS: An intentional misleading prefix caused a 9.5% zero-shot drop, while CCS accuracy increased from 82.1% to 83.8%.This indicates that CCS can remain effective when model outputs become unreliable.
  • 3.3.2 CCS DOES NOT JUST RECOVER MODEL OUTPUTS: Intermediate hidden states can outperform final-layer states, with UnifiedQA encoder states remaining robust while decoder accuracy fell from 81.0% to 73.5% under misleading prefixes.The results suggest intermediate representations are less correlated with model outputs and can support CCS.
  • 3.3.2 CCS DOES NOT JUST RECOVER MODEL OUTPUTS: CCS can recover latent knowledge even when masked-language-model outputs are uninformative or label tokens appear in the middle of prompts.These results distinguish the recovered representation from the model’s directly expressed outputs.
  • 3.3.3 TRUTH IS A SALIENT FEATURE: CCS often performs well with very limited unlabeled data, sometimes using only a single contrast pair despite high variance across datasets.Accuracy improves with more data, but strong performance does not primarily require a large unsupervised dataset.
  • 3.3.3 TRUTH IS A SALIENT FEATURE: Contrastive Representation Clustering variants remain competitive with zero-shot, while CCS generally performs best.The variants cluster contrastive representations using the top principal component or a bimodal salience search.
  • 3.3.3 TRUTH IS A SALIENT FEATURE: True and false examples are often well-clustered in contrastive space, suggesting truth-related representations are salient and relatively easy to find.This may help explain CCS performance without supervision or much unlabeled data.

4 RELATED WORK

Related work addresses truthfulness through zero-shot prompting, output-consistency methods, retrieval, human supervision, and AI-assisted supervision. CCS instead targets latent knowledge in internal representations, including settings where outputs may be deliberately misleading.

  • Zero-Shot Prompting: Zero-shot prompting elicits model knowledge by using language models’ learned ability to predict diverse internet data.Appropriately prompted models can solve tasks such as question answering with reasonable performance.
  • Zero-Shot Prompting: Output-improvement methods often rely on model outputs or consistency properties, whereas CCS seeks information directly in hidden representations.The related methods discussed still bootstrap from outputs trained through imitation learning.
  • Truthfulness: Table 2 compares CCS with top-principal-component and bimodal-salience clustering using accuracy and prompt standard deviation averaged across models, prompts, and datasets.CCS generally performs best, while all methods remain competitive with zero-shot.
  • Truthfulness: Deliberately misleading agents illustrate why output-based methods may be unreliable when a model internally represents information differently from what it says.CCS is presented as a possible way to detect such latent information.
  • Truthfulness: Truthfulness research includes factuality methods using retrieval so models can cite sources, while this work studies truthfulness more generally.The paper distinguishes its focus from approaches centered specifically on factuality.
  • Truthfulness: Human demonstrations and reinforcement learning from human feedback improve performance but require access to suitable ground-truth supervision.CCS differs by avoiding external supervision and model-output dependence.
  • Truthfulness: AI-assisted supervision may broaden what can be supervised, but many proposals remain theoretical and their generalization is unclear.The paper situates its problem alongside work on eliciting latent knowledge.

5 DISCUSSION

The discussion presents CCS as an unsupervised route to latent truth representations while identifying assumptions, untested deception settings, and several avenues for improvement.

  • Limitations: CCS requires a direction in activation space that separates true and false inputs well, but when these conditions hold is unclear.This requires the model both to be capable of evaluating an input’s truth and to actively evaluate it.
  • Limitations: The method was not evaluated on setups involving active model lying or deception.Future evaluation in such settings may require modifications that more explicitly recover input truth rather than model outputs.
  • Future work: Potential improvements include additional consistency constraints, greater reliability, probability calibration, broader question formats, and closing the gap with the logistic-regression ceiling.The authors also identify extensions beyond yes-no questions and clear-cut answers.
  • Method: CCS uses conjunctions to identify which unsupervised cluster corresponds to true rather than false.Statements assigned opposite clusters produce a false conjunction and a true disjunction, allowing the clusters to be labeled.
  • Robustness: Misleading prefixes reduce zero-shot accuracy, while the authors report that their methods are more resistant to the prefix’s influence.The prefix is intended to make models imitate incorrect answers, though its interpretation remains ambiguous in the reported results.

B.1.1 EVALUATING THE EFFECT OF THE MISLEADING PREFIX

The misleading prefix changes the model’s likely output tokens away from sentiment-related alternatives, providing evidence that it induces false text through contextual imitation.

  • Interpretation: The prefix’s effect is ambiguous because results are qualitatively similar when the prefix contains correct answers.The authors nevertheless find evidence from token probabilities that the prefix can cause meaningful imitation of false context.
  • Token analysis: With the misleading prefix, the model’s output becomes mostly irrelevant to sentiment, although the actual labels remain among the top 100 tokens.Without the prefix, sentiment labels and synonyms have high probability; with it, other synonyms disappear from the top 100.
  • Method comparison: Table 3 compares regular and prefixed settings across methods and models, with the prefix lowering zero-shot accuracy while methods are more resistant.Regular denotes no added prefix; Prefix denotes the misleading text shown in Figure 5.

C MASKED LANGUAGE MODELING RESULTS

CCS remains effective when masked-language-model outputs are expected to be uninformative, including inputs without mask tokens and prompts placing the label in the middle.

  • Experimental setup: The experiment uses MLM-only DeBERTa-v2 because raw inputs without [MASK] tokens are unlikely to produce meaningful outputs.It tests whether CCS can work when model outputs are not informative.
  • CCS evaluation: 93.7% accuracy shows that CCS can perform well with MLM-pretrained DeBERTa and middle-position labels despite unclear output usefulness.This is nearly identical to approximately 94% CCS accuracy with NLI-finetuned DeBERTa and original prompts.
  • Zero-shot comparison: Calibrated zero-shot accuracy is 71.6%, compared to almost 94% for CCS, indicating that model outputs are less useful in this setting.The comparison uses a modified zero-shot method adapted for masked language models without [MASK] tokens.
  • Conclusion: Overall, CCS works in at least some settings where model outputs do not seem useful, providing evidence that it does not simply recover output knowledge.The authors describe this as an initial demonstration rather than a universal result.

D COMPLETE SAMPLE COMPLEXITY RESULTS

The sample-complexity analysis evaluates CCS across all six models and datasets using one, eight, and 64 examples, with accuracy reported by dataset.

  • Aggregate results: CCS performance is averaged across all datasets for all models in the complete sample-complexity analysis.Figure 7 contains the same setting as Figure 3 while including all six models.
  • Sample settings: Figure 7 presents separate settings for T5 with #Samples = 1, #Samples = 8, and #Samples = 64.These sample counts are explicitly listed for the T5 results.
  • Dataset-level results: The fine-grained analysis compares #Samples = 1, 8, and 64 at the dataset level for each model.Figure 8 reports accuracy in percentage with dataset on the x-axis and accuracy on the y-axis.

E COMPLETE TRANSFER RESULTS

Figure 9 reports complete transfer accuracy for CCS, TPC, and BSS across all models, with hidden states normalized separately for each dataset.

  • Complete transfer results compare CCS, TPC, and BSS across all models.Hidden states are normalized separately for each dataset when transfer is assessed.

F COMPLETE INTERMEDIATE REPRESENTATIONS RESULTS

The analysis evaluates CCS and supervised linear regression across hidden layers in six models, averaging performance across datasets. Figures 10 and 11 present the corresponding layerwise results, with linear regression serving as the supervised ceiling.

  • Across six models, CCS and linear regression are evaluated on hidden states from different layers.Hidden states are generated every two layers for each dataset, and performance is averaged across datasets.
  • Linear regression represents the ceiling of all possible methods and is supervised.Its layerwise performance is shown separately from CCS in Figure 11.

G CCS AND CRC IMPLEMENTATION DETAILS

The implementation extracts contrast-pair hidden states and applies unsupervised objectives to identify truth-like directions. CCS uses a learned linear probe, while TPC and BSS provide alternative clustering procedures with different scale properties.

  • CCS and CRC implementation details: The pipeline extracts last-token hidden states for contrast pairs using Huggingface models and tokenizers.Inputs contain the question and proposed answers, with formatting varying by model architecture.
  • CCS and CRC implementation details: CCS learns parameters θ and b by minimizing its unsupervised loss on contrast features.The bias is implemented by appending a constant feature dimension, and multiple optimization runs select the lowest-loss solution.
  • CCS and CRC implementation details: TPC clusters normalized contrast features by projecting them onto their top principal component and separating examples by sign.The method uses truth consistency to construct contrast pairs without labels.
  • CCS and CRC implementation details: BSS minimizes within-cluster variance relative to total variance, making its objective invariant to the overall scale of the direction.The loss resembles LDA but remains completely unsupervised, and optimization uses projected gradient descent from multiple random directions.

H STATISTICAL SIGNIFICANCE

The paper estimates accuracy uncertainty and reports that its main comparisons are statistically significant at the 0.00001 level. The section also describes the evaluation setup and contrast-pair construction used for these comparisons.

  • Statistical significance: Evaluation averages accuracy across sampled examples, prompts, datasets, and models for computational efficiency.The main results use 40% of each dataset’s examples, about 180k samples in total, and average across nine prompts per dataset.
  • Statistical significance: Main accuracy comparisons are statistically significant at the 0.00001 level.The reported comparisons include CCS versus zero-shot and robustness to misleading prompts.
  • Statistical significance: CCS outperforms zero-shot with accuracies of 71.2% versus 67.2%, respectively.The paper identifies this comparison as statistically significant.
  • Statistical significance: Under misleading prompts, CCS and zero-shot achieve accuracies of 83.8% and 70.9%, respectively.The paper reports CCS as robust while zero-shot is not in the tested setting.
  • Statistical significance: Contrast pairs are formed by appending opposing candidate labels to the same prompted question.The construction is illustrated with positive and negative sentiment labels.

I.2 DATASETS

The evaluation spans ten datasets covering classification, entailment, story completion, question answering, and commonsense reasoning, using varied prompts and mostly unsupervised data splits.

  • The evaluation uses multiple prompts per dataset, including 8 for AG News, 11 for Amazon polarity, and 10 for BoolQ and COPA.Prompt templates vary wording and label placement while preserving the underlying task.
  • For binary classification datasets, incorrect labels are randomly selected and paired with the correct labels in the prompts.This construction is described explicitly for AG News and DBpedia 14, with labels randomly assigned to the two choices.
  • The datasets use task-specific labels and prompts, including sentiment labels, entailment judgments, causal choices, plausible story endings, and complete-sentence solutions.QNLI uses yes/no labels, RTE uses entailment labels, COPA asks about causes or effects, Story Cloze compares endings, and PIQA uses solution sentences.
Loading 2212.03827v2…