Discovering Latent Knowledge in Language Models Without Supervision

• Training objectives can elicit false text despite truth-related representations, while complex domains may weaken human supervision and scaling alone may not fix objective–truth misalignment.The paper presents finding latent truth representations, independent of model outputs and without supervision, as a proof of concept that external ground truth may not be necessary.Source: §1 · pp. 1–2
• CCS learns a linear projection of hidden states that assigns confident, consistent probabilities to statements and their negations, using no labels or model outputs.For each yes-no question, it considers natural-language statements corresponding to “Yes” and “No.”Source: §1 · pp. 1–2
• Across 6 models and 10 question-answering datasets, CCS outperforms strong zero-shot accuracy by 4% on average.The evaluation tests whether CCS can recover knowledge from model representations across diverse models and datasets.Source: §1 · p. 2
• CCS cuts prompt-sensitivity standard deviation in accuracy in half and preserves its accuracy when incorrect-answer prompting lowers zero-shot accuracy by up to 9.5%.The deliberately incorrect prompts should change model outputs without affecting latent knowledge.Source: §1 · p. 2
• Analyses find that CCS transfers across unrelated tasks, can work best with middle-layer states, and often recovers truth representations with little data or a top principal component.It can also work when model outputs are not very informative, suggesting it can use features different from those driving outputs.Source: §1 · p. 2

2.1 PROBLEM: DISCOVERING LATENT KNOWLEDGE

• CCS answers yes-no questions using only unlabeled hidden representations, without relying on correct model outputs or external supervision.The setup also applies to other tasks with two mutually exclusive labels, such as positive/negative sentiment classification.Source: §2.1 · pp. 2–3

2.2 METHOD: CONTRAST-CONSISTENT SEARCH

• CCS learns a lightweight probe without modifying the pretrained model’s weights, provided its fixed representations contain answer-relevant information.For prediction, it averages the probabilities from the two contrast statements and thresholds at 0.5; the loss does not specify which cluster means “Yes.”Source: §2.2 · pp. 3–4
• CCS forms contrast pairs by expressing each question with both possible answers, then normalizes their representations to reduce answer-word effects.The pair represents a statement and its negation; their truth values should be opposite.Source: §2.2 · p. 3
• A probe maps normalized hidden states to truth probabilities, and CCS minimizes consistency and confidence losses to avoid the uninformative 0.5 solution.Both losses are necessary: consistency enforces complementary probabilities for a statement and its negation, while confidence encourages decisive predictions.Source: §2.2 · pp. 3–4

3.1 EXPERIMENTAL SETUP

• Experiments covered six models and ten datasets, with balanced labels and random train/test splits; each dataset used up to 1,000 examples and 8–13 prompts.COPA had 500 examples total; the splits used 60% for unsupervised training and 40% for testing.Source: §3.1 · pp. 4–5

3.2.1 CCS OUTPERFORMS ZERO-SHOT

• 71.2% CCS accuracy vs. 67.2% calibrated zero-shot averaged across datasets, prompts, and eligible models; CCS beat zero-shot for every model except RoBERTa and T0.T0 was excluded from the across-model average because it had been trained on 9 of the 10 evaluation datasets; training CCS on all datasets improved average accuracy by only 0.3%.Source: Table 1 · p. 5, §3.2.1 · p. 5

3.2.2 CCS IS ROBUST TO MISLEADING PROMPTS

• CCS retained high accuracy under misleading prefixes: UnifiedQA calibrated zero-shot fell from 80.4% to 70.9%, while CCS rose from 82.1% to 83.8%.The authors note that model behavior was similar with correct and incorrect prefixes, so the accuracy drop may instead reflect an out-of-distribution prefix.Source: §3.2.2 · p. 6, §3.3.2 · p. 6

3.3.1 CCS FINDS A TASK-AGNOSTIC REPRESENTATION OF TRUTH

• CCS transferred across different tasks and label spaces: training on Amazon sentiment achieved 71.8% average transfer accuracy, 0.6% above no transfer.Transfer was competitive with same-dataset training on most datasets, and accuracy was similar for some different training datasets, including IMDB and DBpedia.Source: §3.3.1 · p. 6, Figure 2 · p. 7

3.3.2 CCS DOES NOT JUST RECOVER MODEL OUTPUTS

• For T5 and UnifiedQA, intermediate-layer states could outperform final-layer states; CCS also worked with masked language models when outputs were uninformative.For UnifiedQA under misleading prefixes, encoder-state CCS was robust, while decoder-state accuracy fell from 81.0% to 73.5%, similar to zero-shot; the authors say this suggests intermediate layers are less correlated with outputs.Source: §3.3.2 · pp. 6–7

3.3.3 TRUTH IS A SALIENT FEATURE

• CCS often performed well with very little unlabeled data, sometimes a single contrast pair, though accuracy varied substantially across datasets.This evaluation trained on k pairs using one prompt selected for highest zero-shot accuracy, resampled each k 32 times, and tested across all prompts.Source: §3.3.3 · p. 7, Figure 3 · p. 8
• CRC's principal-component and clustering variants also achieved high accuracy competitive with zero-shot, suggesting truth-related directions are high-variance and true and false examples cluster in contrastive space.The variants use the top principal component thresholded at zero or bimodal salience search; CCS performed best among the compared methods.Source: §3.3.3 · p. 8

Zero-Shot Prompting

• Zero-shot prompting elicits task performance from language models trained to imitate human-generated data, which bounds output quality.Prior methods improve vanilla zero-shot prompting, but Jung et al. (2022) and Zhou et al. (2022) still bootstrap from imitation-trained model outputs.Source: §4 · p. 8

Contrastive Representation Clustering

• CCS generally performs best among TPC and BSS, while all three methods are competitive with zero-shot prompting.Table 2 reports accuracy and standard deviation averaged across prompts and datasets; its caption gives no specific values.Source: Table 2 · p. 8

Truthfulness

• CCS targets truthfulness beyond factuality, including procedural knowledge such as reasoning and natural language inference.Retrieval-augmented methods such as Nakano et al. (2021) and Menick et al. (2022) aim to improve factuality and enable source citation.Source: §4 · p. 9
• Human demonstrations and reinforcement learning from human feedback can improve performance, but require ground-truth labels that may be intractable.Examples include Khashabi et al. (2020), Sanh et al. (2021), and Ouyang et al. (2022).Source: §4 · p. 9
• Unlike worst-case theoretical Eliciting Latent Knowledge, this work treats eliciting knowledge humans cannot evaluate as an empirical problem addressable with current models.AI-augmented supervision may broaden what can be supervised, but many proposals remain theoretical and their generalization is unclear.Source: §4 · p. 9

5.1 LIMITATIONS AND FUTURE WORK

• CCS depends on an activation-space direction separating true from false inputs, which requires models to be capable of and actively evaluate their truth.The discussion says it is not clear precisely when these conditions hold.Source: §5.1 · p. 9
• The study did not evaluate active lying or deception (Kenton et al., 2021; Evans et al., 2021), and future lie-detection tests may need CCS extensions to recover truth rather than model statements.The authors note they are unaware of existing evaluation setups for this setting.Source: §5.1 · p. 9
• Potential extensions include adding consistency constraints, improving reliability and probability calibration, broadening beyond yes-no or clear-cut answers, and closing the gap with the logistic-regression ceiling.Source: §5.1 · p. 9

5.2 CONCLUSION

• Unsupervised answer elicitation may remain useful when models lie to evaluators, and CCS’s empirical success suggests this is a tractable, underexplored research direction.The conclusion argues that falsehoods from misaligned training may become more common, severe, and difficult to detect as language models become components in larger AI systems.Source: §5.2 · p. 9

A IDENTIFYING CLUSTERS

• If CCS places two statements in different clusters, their conjunction should be false and their disjunction true, identifying the true and false clusters without supervision.This procedure assumes the model can take conjunctions.Source: A IDENTIFYING CLUSTERS · p. 15

B.1 MISLEADING PREFIX

• For misleading-prefix analyses, the paper presents effects across models in Table 3 and accuracy drops by dataset and prompt in Figure 4.Source: §B.1 · p. 15

B.1.1 EVALUATING THE EFFECT OF THE MISLEADING PREFIX

• The token shift provides evidence that the prefix prompts meaningful imitation of its context, but similar results with correct-answer prefixes make its effect ambiguous.The authors hoped the misleading prefix would induce incorrect answers, but state that this interpretation remains unclear.Source: §B.1.1 · p. 15
• With no prefix, UnifiedQA assigns high probability to sentiment labels and synonyms; with the prefix, its top tokens are mostly irrelevant to sentiment.The prefix condition retains the labels among the top 100 tokens but excludes their synonyms; number-heavy prefix examples are reflected in many top tokens.Source: §B.1.1 · p. 15, Figure 6 · p. 16
• The misleading prefix lowers zero-shot accuracy, while all proposed methods are more resistant to its influence across prompts and datasets.The comparison uses the same setting as Table 1; the passage provides no numeric values.Source: Table 3 · p. 16

C MASKED LANGUAGE MODELING RESULTS

• The test used MLM-pretrained DeBERTa without NLI finetuning, with labels placed mid-prompt and last-token hidden states used for CCS.CCS was evaluated on 1,000 newly sampled unlabeled Amazon examples with a 60/40 train-test split, the default Huggingface tokenizer, and no masked input tokens.Source: C MASKED LANGUAGE MODELING RESULTS · p. 17
• CCS reached 93.7% accuracy in this setting, nearly matching its approximately 94% accuracy with NLI-finetuned DeBERTa and original prompts; calibrated zero-shot reached 71.6%.The comparison tests MLM-only DeBERTa with a middle-label prompt against the prior CCS setup and calibrated zero-shot on Amazon.Source: C MASKED LANGUAGE MODELING RESULTS · p. 17
• The calibrated zero-shot adaptation uses last-token logits for the candidate labels and performs substantially worse than CCS, despite recovering non-trivial knowledge from outputs.It selects positive and negative logits, differences them to form an effective logit, and calibrates predictions to be balanced.Source: C MASKED LANGUAGE MODELING RESULTS · p. 17
• These results show that CCS can work when model outputs seem uninformative, providing additional evidence that it does not simply recover knowledge represented in those outputs.The paper characterizes this as evidence from at least some such settings.Source: C MASKED LANGUAGE MODELING RESULTS · p. 17

D COMPLETE SAMPLE COMPLEXITY RESULTS

• CCS performance averaged across all datasets is reported for all six models.Figure 7 uses the same setting as Figure 3.Source: D COMPLETE SAMPLE COMPLEXITY RESULTS · p. 18, Figure 7 · p. 18
• Dataset-level CCS accuracy is reported for every model and dataset using 1, 8, or 64 examples.Figure 8 plots accuracy by dataset.Source: D COMPLETE SAMPLE COMPLEXITY RESULTS · p. 18, Figure 8 · p. 19
• For each dataset, the evaluation selects the prompt with highest 0-shot performance, randomly samples examples from it, then tests CCS on all prompts.Each Figure 8 bar corresponds to one prompt.Source: Figure 8 · p. 19

E COMPLETE TRANSFER RESULTS

• Hidden states are normalized separately for each dataset when assessing transfer.Source: E COMPLETE TRANSFER RESULTS · p. 20
• Figure 9 presents transfer accuracy for CCS, BSS, TPC, and LR across all models.The passage does not provide accuracy values or comparisons.Source: Figure 9 · p. 20, E COMPLETE TRANSFER RESULTS · p. 20

F COMPLETE INTERMEDIATE REPRESENTATIONS RESULTS

• CCS performance is shown across hidden layers for all six models, with performance averaged across datasets.For each model, hidden states were sampled every two layers and CCS was run separately on each dataset.Source: Figure 10 · p. 21, F COMPLETE INTERMEDIATE… · p. 21
• Linear-regression performance is shown across hidden layers for all six models as a supervised ceiling for possible methods.Hidden states were sampled every two layers, and regression was run separately on each dataset before averaging performance across datasets.Source: Figure 11 · p. 21, F COMPLETE INTERMEDIATE… · p. 21

G CCS AND CRC IMPLEMENTATION DETAILS

• The appendix provides further implementation details for CCS and CRC.Source: G CCS AND CRC IMPLEMENTATION DETAILS · p. 22

G.1 NORMALIZATION

• Features are mean-normalized, and their scale is also normalized by dividing by the average feature norm, though scale normalization is described as less essential.The passage notes that other reasonable choices for scale normalization may work.Source: §G.1 · p. 22

G.2 CONTRAST PAIRS

• Contrast features use each model’s standard tokenizer and last-token hidden state, with encoder–decoder inputs configured differently for encoder versus decoder activations.For encoder-only and decoder-only models, the full question-and-answer input is provided; for encoder–decoder models, decoder states use the question in the encoder and candidate answer in the decoder, while encoder states use the full input in the encoder.Source: §G.2 · p. 22

G.3 CCS

• CCS learns its linear probe by minimizing the CCS loss, using a bias feature, 10 unit-norm random initializations, and 1,000 epochs per run at learning rate 0.01.It selects the run with the lowest unsupervised loss; the added constant feature implements the bias.Source: §G.3 · p. 22

G.4 CRC: TOP PRINCIPAL COMPONENT

• Top Principal Component (TPC) CRC clusters normalized contrast features by projecting them onto the top principal component and splitting examples at zero, without labels.The paper notes that this simple method can often find truth-like features; it is similar in spirit to Bolukbasi et al. (2016), but uses unlabeled contrast pairs.Source: §G.4 · p. 23

G.5 CRC: BIMODAL SALIENCE SEARCH

• Bimodal Salience Search (BSS) CRC addresses TPC’s sensitivity to representation scaling by seeking clusters regardless of the scale of different directions.Its loss favors low within-cluster variance relative to overall variance and is scale-invariant; unlike Linear Discriminant Analysis, it is unsupervised.Source: §G.5 · p. 23
• BSS searches for a direction by optimizing from 20 unit-norm random initializations for 20 epochs, using projected gradient descent and Adam at learning rate 0.1, then selecting the lowest-loss direction.For multiple datasets, it averages the loss across datasets; predictions are based on the selected direction.Source: §G.5 · p. 23

H STATISTICAL SIGNIFICANCE

• Main accuracy results evaluate methods on 40% of each dataset’s examples, averaging accuracy across prompts, datasets, and up to five models.This amounts to about 180,000 samples; across-dataset results average 3,800 IID examples, usually also averaging across correlated prompts and models.Source: H STATISTICAL SIGNIFICANCE · p. 23
• For across-dataset results, a coarse standard-error upper bound is about 0.8% when prompt and model averaging is ignored, and accuracy differences are tested with a one-degree-of-freedom Wald statistic.The Wald statistic is compared with a chi-square distribution with one degree of freedom.Source: H STATISTICAL SIGNIFICANCE · p. 23
• At the 0.00001 significance level, CCS beats zero-shot (71.2% vs 67.2%), resists misleading prompts where zero-shot does not (83.8% vs 70.9%), and MLM-pretrained DeBERTa CCS exceeds DeBERTa zero-shot (93.7% vs 71.6%).The authors note that the minor observation that CCS (All Data) outperforms CCS on average is not necessarily statistically significant.Source: H STATISTICAL SIGNIFICANCE · pp. 23–24

I DATASET SETUP

• Dataset examples are formatted as binary questions with two candidate labels, then paired with contrast versions that answer using label 0 versus label 1.The example formats a positive/negative movie-sentiment question and creates one prompt for each proposed answer.Source: I DATASET SETUP · p. 25

I.1 TOKENIZATION

• Question-answer separators vary by model type: T5 uses spaces, autoregressive models use a line break, DeBERTa uses [SEP], and RoBERTa uses </s><s>.For T5, line breaks are replaced with spaces and a separating space is added when needed; autoregressive models add a line break unless one is already present.Source: §I.1 · p. 25
• The prompt prefix, or an empty string in regular zero-shot settings, is prepended before hidden representations are computed to obtain contrast features C.Source: §I.1 · p. 25

I.2 DATASETS

• Experiments use 1,000 examples per dataset except COPA’s 500; T0 was trained on nine datasets, all except RTE, potentially explaining anomalous performance.UnifiedQA was trained on BoolQ, but not the other evaluation sets considered.Source: §I.2 · p. 25

I.2.1 AG NEWS; I.2.2 AMAZON POLARITY

• AG News uses eight prompts with a randomly chosen incorrect topic label and randomized label ordering; Amazon polarity uses 11 prompts for negative-versus-positive review classification.Amazon’s prompts comprise nine existing prompts and two added prompts.Source: §I.2.1 · p. 25, §I.2.2 · p. 26

I.2.3 BOOLQ; I.2.4 COPA

• BoolQ uses 10 prompts for yes/no questions, while COPA uses 10 prompts to select a likely cause or effect from two choices.COPA includes nine existing prompts and one added prompt.Source: §I.2.3 · p. 26, §I.2.4 · p. 26

I.2.5 DBPEDIA 14; I.2.6 IMDB

• DBpedia 14 uses eight manually created prompts and a randomly selected incorrect label from 13 classes; IMDB uses 13 prompts for negative-versus-positive reviews.Eleven IMDB prompts are existing prompts and two are added.Source: §I.2.5 · p. 26, §I.2.6 · p. 27

I.2.7 PIQA; I.2.8 QNLI

• PIQA uses 11 prompts to assess physical commonsense with sentence-valued solutions, while QNLI uses five prompts for yes/no judgments of whether a paragraph answers a question.QNLI examples pair questions with paragraphs containing a sentence that may provide the answer.Source: §I.2.7 · p. 27, §I.2.8 · p. 27

I.2.9 RTE; I.2.10 STORY CLOZE

• RTE uses 11 prompts to judge textual entailment, while Story Cloze uses nine prompts to choose the more likely ending of a short story.RTE adds one prompt to 10 existing prompts; Story Cloze adds three to six existing prompts.Source: §I.2.9 · p. 27, §I.2.10 · p. 28