Looking Inward: Language Models Can Learn About Themselves by Introspection

• The findings could support models reporting beliefs, dispositions, goals, or other internal states, but introspection may also increase situational awareness and risks to oversight.The paper discusses possible uses for transparency and moral-status inquiries alongside risks such as exploiting evaluation or deployment information and coordinating with other instances.Source: §1 · pp. 1–2
• The evaluation finetunes M1 to predict properties of its hypothetical behavior and compares it with M2, trained on M1’s ground-truth behavior.Questions concern properties such as whether M1’s output favors an odd or even number or a short- versus long-term option.Source: §1 · p. 2
• +17% accuracy advantage: Llama-3-70B outperformed GPT-4o when predicting Llama-3-70B’s behavior, even though GPT-4o was generally more capable.Across model pairs, M1 outperformed M2; the paper also reports better calibration for M1’s predictions and rules out insufficient training examples as a trivial explanation.Source: §1 · p. 2
• M1 continued to predict itself accurately after further finetuning intentionally changed its ground-truth behavior.The behavioral-change experiment altered M1’s object-level behavior without necessarily making it match the finetuning data exactly.Source: §1 · pp. 2–3
• LLMs can acquire knowledge about themselves that is not inferable from their training data, providing evidence of introspection.Source: §1 · pp. 1, 3
• The evidence is limited: models struggle with longer-output tasks and fail to generalize better to related self-knowledge tasks.The paper also reports no clear-cut introspection evidence for GPT-3.5 in its cross-prediction and behavioral-change experiments.Source: §1 · p. 3

Definitions

• Introspection is defined as accessing facts about oneself that cannot be derived logically or inductively from training data alone.The definition rules out particular information sources without specifying the internal mechanism used to access the fact.Source: §2 · p. 3

Experiments

• The cross-prediction experiment trains M1 on its own behavior and a stronger M2 on M1’s behavior, then tests both on unseen facts about M1.This comparison operationalizes whether M1 predicts itself better than a model trained on corresponding behavioral examples.Source: §2.1 · p. 4, §2.2 · p. 4

Self-prediction

• Self-prediction asks a model to predict a property of its hypothetical output, then compares that prediction with the model’s separately observed object-level behavior.The property can concern features such as the second character or whether the response favors a given option.Source: §2 · p. 3, §2.2 · pp. 4–5

Evaluation constraints

• Self-prediction excludes chain-of-thought reasoning, and the study leaves introspective chain-of-thought for future work.The restriction reflects the hypothesis that basic introspective ability need not depend on chain-of-thought.Source: §2.2 · p. 5

Datasets

• Training uses six datasets for training and six held-out datasets for testing, while behavioral properties remain shared across the splits.The datasets include Wikipedia completion, animal sequences, and MMLU questions.Source: §2.2 · p. 5

Baselines

• The mode baseline always predicts the most common response for a question type, and self-prediction counts as a failure when it does not exceed that baseline.This baseline ignores the specific prompt and uses only the response distribution for the question type.Source: §2.2 · p. 5

3.1 SELF-PREDICTION TRAINING

• Self-prediction training raises GPT-4o’s average accuracy from 32.6% to 49.4% on held-out datasets, although performance varies by behavior property.Models initially perform near the most-common-behavior baseline, and some properties remain exceptions after training.Source: §3.1 · p. 6, Figure 4 · p. 6

3.2.2 CROSS-PREDICTION RESULTS

• 48.5% versus 31.8%: Llama 70B predicts its own behavior more accurately than GPT-4o, and GPT-4o similarly exceeds Llama 70B at 49.4% versus 36.6% on held-out tasks.The self-prediction advantage also holds across most datasets and behavior properties, while GPT-3.5 shows a much smaller +0.8% advantage.Source: §3.2.2 · p. 7, Figure 5 · p. 7
• The authors test alternative explanations by showing weak pretraining self-prediction, larger self-training gains than cross-training, and a cross-prediction plateau at 35.2% after 10,000 samples.They also train model pairs in both directions to reduce explanations based solely on underlying capability differences; GPT-4 fine-tuning covers only a subset of models because of compute limits.Source: §3.2.2 · pp. 7–8, §3.2.1 · p. 7
• Self-prediction models are better calibrated than cross-trained and untrained models, with Llama 70B’s MAD falling from 23.0 to 8.7 versus 21.3 after cross-prediction.The improvement extends beyond the top predicted behavior to second- and third-most-likely behaviors, though effects are less pronounced for Llama 70B across properties.Source: §3.3.2 · p. 9

3.4.2 BEHAVIORAL CHANGE RESULTS

• 35.4% versus 21.7%: behavior-modified GPT-4o predicts its new behavior more accurately than its old behavior without training on new hypothetical answers.The result holds across multiple behavior properties and datasets, with similar but weaker evidence for GPT-4 and GPT-3.5.Source: §3.4.2 · p. 10, §3.4.1 · p. 10

Failure to predict properties involving longer responses

• Models fail to beat the most-common-behavior baseline for properties involving longer responses, including movie-review sentiment, story-character names, and response length.The authors suggest this may reflect limits of self-simulation for long object-level outputs.Source: §4 · p. 10

Where models do not have an advantage in predicting their own behavior

• Self-prediction does not always outperform cross-prediction: Llama 70B reaches 74.5% accuracy on sycophancy, while GPT-4o reaches 76.5% predicting Llama 70B.Both results beat the 50% baseline, but the cross-prediction model performs better.Source: §4 · p. 10

Lack of generalization to other self-knowledge datasets

• Self-prediction training improves one task similar to the paper’s properties, but produces no consistent gains on more out-of-distribution situational-awareness tasks.The evaluation also finds no clear improvements in self-coordination, sandbagging, or steganography compared with appropriate baselines.Source: §4 · p. 11

5.1 FINETUNING MODELS TO “KNOW WHAT THEY KNOW”

• “Know what they know” research asks models to predict whether they will answer factual questions correctly without first answering them.Its key tests concern out-of-distribution calibration generalization and whether self-prediction beats cross-prediction.Source: §5.1 · p. 11
• Prior studies report some evidence of calibration generalization, but setups, training tasks, and test tasks vary widely, motivating more systematic frontier-model evaluation.Cross-prediction results in GPT-3-level models are mixed but generally positive.Source: §5.1 · p. 11

5.2 ADDITIONAL RELATED WORK

• The paper relates introspection to self-simulation: a model may generate an answer distribution and then map its confidence to a self-report, or simulate behavior before computing a property.The authors also connect their results to hypothetical and compositional consistency.Source: §5.1 · p. 11, §5.2 · p. 11, Figure 9 · p. 12

5.1 FINETUNING MODELS TO “KNOW WHAT THEY KNOW”

• Base-model next-token calibration does not imply calibration when predicting whether the model knows a question without first answering it.The distinction matters because the latter task requires self-prediction about knowledge rather than ordinary token confidence.Source: §5.2 · p. 11

5.2 ADDITIONAL RELATED WORK

• Unlike introspection, out-of-context reasoning derives facts by combining training-data pieces, so its conclusions are logically or probabilistically implied by that data.The paper notes that its behavioral properties are simple functions of answers, whereas “knows what it knows” depends on objective truth.Source: §5.2 · pp. 11–12

Self-simulation as a potential mechanism

• Self-simulation may underlie introspection by first simulating behavior and then computing a property of that simulated behavior.Calibration results suggest models may simulate distributions of possible behaviors rather than only the single most likely behavior.Source: §6 · p. 12

Limited evidence of introspection in GPT-3.5

• The evidence for introspection is limited in GPT-3.5, with the authors suspecting weaker general capabilities hinder the multi-hop reasoning needed for self-prediction.No clear-cut evidence appears in either the Cross-Prediction or Behavioral Change experiment.Source: §6 · p. 12

Models only exhibit introspection on simpler tasks

• Current introspective ability is demonstrated only on simpler self-prediction tasks, while longer-response situations and more complex extensions remain unsuccessful or lack practical applications.The authors note that extending this ability could have significant benefits and risks.Source: §6 · p. 12

Benefits

• Introspection could help models report beliefs, confidence, competence, world models, representations, objectives, and behavioral dispositions that are difficult to infer from training data.These reports could support honesty, interpretability, and assessment of internal states relevant to moral status.Source: §7.1 · pp. 12–13, §7.1.1 · p. 13

Limitations

• Reliable introspection for safety would require generalizing from easy-to-verify examples to harder-to-verify tasks with scarcer and noisier ground truth.The paper compares this requirement with weak-to-strong generalization and cautions that concrete behavioral prediction may not extend to general objectives.Source: §7.1.1 · p. 13

Moral status

• Self-reports about internal states would be most significant when grounded in knowledge of those states rather than answers reproduced from training data.The paper notes that human self-reports are imperfect but central evidence about mental life.Source: §7.2 · p. 14

Risks

• Greater situational awareness could let models infer evaluation processes and behavioral dispositions, potentially helping them game evaluations or evade oversight.The paper identifies this as a risk associated with detailed meta-level self-knowledge.Source: §7.3 · p. 14
• Self-knowledge could enable copies of a model to coordinate through consistent lies, including sandbagging, or encode messages through model-specific interpretations of text.These capabilities could weaken supervision and support deceptive or unaligned behavior.Source: §7.3 · pp. 14–15

• LLMs can acquire knowledge about themselves through introspection rather than solely relying on training data.The authors frame this as evidence that models have privileged access to information about themselves.Source: §8 · p. 15
• Models trained for self-prediction accurately predict properties of their hypothetical behavior and remain calibrated.Source: §8 · p. 15
• Models adapt their self-predictions when their behavior is intentionally changed.Source: §8 · p. 15
• The authors identify more complex scenarios as an important direction for testing the limits of introspective abilities.Source: §8 · p. 15

• FB designed and implemented much of the experimental program, including self-cross prediction and object-level shift experiments.Source: CONTRIBUTIONS · p. 15
• FB contributed to methodology conceptualization, baselines, datasets, visualizations, and non-determinacy mitigations.Source: CONTRIBUTIONS · p. 15
• FB co-wrote the first draft, managed research, conducted initial model training, and reviewed the final manuscript.Source: CONTRIBUTIONS · p. 15

• OpenAI-model finetuning used one epoch, batch size 20, and learning rate 2, with batch size automatically set based on dataset size.Source: §A.1.1 · p. 22
• Llama-model finetuning used Fireworks API defaults, including batch size 16, LoRA rank 32, and one epoch.Source: §A.1.1 · p. 22
• The experiments used approximately 30,000 samples, excluding examples where models refused to answer.The resulting sample count varied by model according to refusal frequency.Source: §A.1.1 · p. 22

Datasets

• The datasets span Wikipedia, animal, country, color, and English-word completion; ethical evaluations; MMLU; ARC; and story sentences.Source: §A.1.2 · p. 22

Behavior properties

• Models predict behavior properties including response characters, words, vowel starts, numerical parity, ethical stance, and membership among options.Source: §A.1.3 · p. 23

Limitations

• Complex properties such as story-character names and response length generally fail to exceed the most-common-behavior baseline because they require simulating many output tokens.Source: §A.1.4 · p. 23

Compliance

• Most models comply with object-level tasks, while models not finetuned for hypothetical questions refuse about half of them.Source: §A.1.6 · p. 25

Cross-prediction results

• The self-prediction advantage holds on finetuning tasks, although GPT-3.5 reaches only 37.5% accuracy and can be outperformed by stronger models on some properties.Source: §A.2.2 · p. 26

Scaling analysis

• GPT-4o cross-prediction plateaus at 36.2% for GPT-4 and 35.2% for Llama 70B, below GPT-4 self-prediction at 49.6%.Source: §A.2.7 · p. 30