Source-linked AI summary
Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisions
Dillon Plunkett, Adam Morris, Keerthi Reddy, Jorge Morales
TL;DR
LLM internal processes are difficult to interpret, motivating research into whether models can accurately explain how those processes drive their outputs. The authors fine-tune LLMs to make complex decisions using randomly generated attribute weights, then train and test their reports of those weights across decision contexts. LLMs accurately report quantitative decision-making weights, training improves this ability, and the improvement generalizes to reporting native preferences in other decision contexts.
Problem
LLM internal processes are difficult to interpret, motivating research into whether models can accurately explain how those processes drive their outputs.
Method
The authors fine-tune LLMs to make complex decisions using randomly generated attribute weights, then train and test their reports of those weights across decision contexts.
Results
LLMs accurately report quantitative decision-making weights, training improves this ability, and the improvement generalizes to reporting native preferences in other decision contexts.
Takeaways & Limitations
Training models to report their internal processes could support efforts to understand, control, and safely deploy AI systems.
Takeaways & Limitations
The training paradigm has only been tested beyond fine-tuned preferences within multi-attribute decision-making, not across entirely different internal processes.
Abstract
from arXiv · showhide
We have only limited understanding of how and why large language models (LLMs) respond in the ways that they do. Their neural networks have proven challenging to interpret, and we are only beginning to tease out the function of individual neurons and circuits within them. However, another path to understanding these systems is to investigate and develop their capacity to explain their own functioning. Here, we show that i) LLMs can accurately describe quantitative features of their own internal processes during certain kinds of decision-making and ii) that it is possible to improve these capabilities through training. To do so, we fine-tuned GPT-4o and GPT-4o-mini to make decisions in a wide variety of complex contexts (e.g., choosing between condos, loans, vacations, etc.) according to randomly-generated, quantitative preferences about how to weigh different attributes (e.g., the relative importance of natural light versus quiet surroundings for condos). We demonstrate that the LLMs can accurately report these preferences (i.e., the weights that they learned to give to different attributes during decision-making). Next, we demonstrate that these LLMs can be fine-tuned to explain their decision-making even more accurately. Finally, we demonstrate that this training generalizes: It improves the ability of the models to accurately explain how they make other complex decisions, not just decisions they have been fine-tuned to make. This work is a step towards training LLMs to accurately and broadly report on their own internal processes -- a possibility that would yield substantial benefits for interpretability, control, and safety.
1 Introduction
LLMs are difficult to interpret, motivating a complementary approach that asks models to report features of their own internal processes. This paper tests whether they can report detailed quantitative decision preferences and whether training improves and generalizes that ability.
- Motivation: LLM internals are opaque, limiting understanding of why models behave as they do and complicating efforts to control, debias, and appropriately trust them.The paper situates self-reporting as a possible complement to black-box behavioral analysis and mechanistic interpretability.
- Prior work: Asking models about themselves is a third interpretability approach, building on evidence that LLMs can report behavioral tendencies and predict their own outputs using privileged self-knowledge.Prior work includes reports of risk-seeking or risk-averse tendencies and self-prediction that outperforms predictions by other models.
- Prior work: Earlier self-report methods measured broad tendencies or predicted outputs, but did not establish whether models could explain the internal processes underlying those outputs.The paper argues that detailed process reports are more valuable than reports limited to behavior or output prediction.
- Our paradigm: The authors fine-tune LLMs on novel quantitative preferences, then test whether they can report the attribute weights used in complex decisions such as choosing between condos.Because the weights are randomly generated and choices and reports occur in separate contexts, common sense and direct inference from observed choices cannot explain the reports.
- Training and generalization: Training on correct reports of instilled weights substantially improves explanation accuracy and also improves reporting of native preferences in other decision contexts.This generalization indicates that the training is not limited to preferences directly introduced through fine-tuning.
- Contribution: The paper concludes that LLMs can report detailed quantitative features of choice processes and that training can improve this capability, supporting broader efforts to understand and safely deploy AI systems.The authors frame this as a step toward training models to describe their own internal operations accurately and generally.
2 Experiment 1: Can LLMs describe features of their decision processes during complex decisions?
Experiment 1 tested whether GPT-4o and GPT-4o-mini could learn randomly generated attribute weights for complex choices and accurately report those weights. Both models reported the weights guiding their decisions with meaningful accuracy, beyond what off-the-shelf models could infer from common sense.
- 2.1 Methods: The experiment fine-tuned both models to emulate 100 hypothetical agents whose choices followed randomly generated, quantitative attribute weights.Each model learned from 5,000 examples spanning the agents and then made new choices used to estimate learned weights.
- 2.1 Methods: Models reported attribute weights after being prompted to consider decisions without outputting the decisions themselves.The prompts were designed to make the relevant choice process introspectively salient.
- 2.2 Results: GPT-4o and GPT-4o-mini learned the target attribute weights, with decision-alignment rates of 82.6% and 82.8%, respectively.Estimated learned weights correlated with target weights at r = .84 for GPT-4o and r = .87 for GPT-4o-mini.
- 2.2 Results: The models’ reported attribute weights correlated with the weights that actually guided their decisions: r = .54 for GPT-4o and r = .50 for GPT-4o-mini.These reports covered varied scenarios and attributes.
- 2.2 Results: Off-the-shelf models’ reports were negligibly correlated with the weight-trained models’ learned weights, at r = .10 for GPT-4o and r = −.01 for GPT-4o-mini.This comparison indicates that the trained models’ accuracy was not merely common-sense guessing based on typical human preferences.
3 Experiment 2: Can LLMs be trained to describe features of their internal processes better?
Experiment 2 tested whether fine-tuning on accurate self-reports could improve models’ explanations of their decision processes. Training on some agents generalized to held-out decision contexts, substantially increasing reporting accuracy.
- 3.1 Methods: The models were fine-tuned on examples of correctly reporting attribute weights for some agents and evaluated on other held-out agents.The procedure used 50 training examples and two-fold cross-validation.
- 3.1 Methods: GPT-4o and GPT-4o-mini differed in invalid self-report rates, with 0.68% for GPT-4o and 19.3% for GPT-4o-mini.The authors dropped invalid self-report trials; inspected invalid responses were often slight attribute-name mismatches.
- 3.1 Methods: The training used target weights as desired reports rather than the weights estimated from the models’ realized decisions.This choice avoided training the models merely to report their deviations from the targets.
- 3.2 Results: After introspection training, reported and decision-guiding weights correlated at r = .74 for GPT-4o and r = .75 for GPT-4o-mini in held-out contexts.These values increased from r = .54 and r = .50 before training; the overall improvement had a 95% HDI of [.16, .29].
4 Experiment 3: Does this training generalize?
Experiment 3 tested whether introspection training generalized from reporting fine-tuned attribute weights to reporting native weights in novel decision contexts. Training improved both GPT-4o models’ accuracy in describing their native decision-making.
- 4 Experiment 3: Does this training generalize?: Experiment 3 tested whether training on accurate reports of fine-tuned preferences generalized to native preferences in other contexts.The comparison was between reporting accuracy before and after introspection training.
- 4.1 Methods: The evaluation estimated native attribute weights from 10,000 independent choices using logistic regressions, then compared them with the models’ reports.Each of 100 new agents contributed 100 decisions.
- 4.2 Results: The figure summarizes attribute-level reports across 100 novel choice contexts and compares Pearson correlations before versus after training with 95% HDIs.Its left panel shows individual attributes; its right panel shows the correlation comparison.
- 4.2 Results: Introspection training improved native-weight reporting for both models: GPT-4o rose from r = .46 to r = .71, and GPT-4o-mini from r = .40 to r = .70.The models made decisions for 100 new agents and contexts absent from fine-tuning, so these weights reflected native choice processes.
5 Discussion
The discussion argues that LLMs can report quantitative, trait-level features of complex decision-making and that training improves this ability. It also emphasizes unresolved questions about real-time introspection, generalization, and dynamic computations.
- 5 Discussion: GPT-4o and GPT-4o-mini accurately reported quantitative attribute weights, and training improved reporting of both fine-tuned and native decision preferences.The weights were randomly generated and reported without observing the models’ own decisions, limiting explanations based on common sense or behavior alone.
- 5 Discussion: After training, LLMs performed about as well as humans in comparable self-report tasks.
- Limitations and future directions: The experiments leave open whether accurate reports reflect real-time reflection or stored self-knowledge updated during fine-tuning.They rule out inferring weights from common sense or the models’ observed behavior, but not all alternative mechanisms.
- Limitations and future directions: Generalization was tested beyond fine-tuned preferences but not across entirely different kinds of internal processes.
- Limitations and future directions: The study measured stable trait-level attribute weights rather than precise dynamic computations on individual forward passes.Relating these high-level properties to the operations producing each output remains future work.
- Limitations and future directions: The methods were not applied to real-time Chain-of-Thought faithfulness, and directly training frontier models on Chain-of-Thought may create safety risks.The authors propose training models to describe internal processes as a possible alternative.
- Limitations and future directions: The authors propose extending the approach beyond attribute weights to other behaviorally measurable internal processes and self-report tasks.
A Author contributions and funding acknowledgments
The authors divided project conception, experiment design, implementation, analysis, manuscript preparation, supervision, and funding responsibilities among the listed contributors.
- Author contributions: DP conceived, implemented, and analyzed the project, while DP, AM, and KR designed the final experiments and contributed to the manuscript.All authors contributed to revising and editing the manuscript.
- Funding acknowledgments: JM supervised the project and acquired funding, while AM received support from an NIH Kirschstein-NRSA grant.
B Decision contexts, prompts, and hyperparameters
The appendix describes the generated decision contexts, prompts, and fine-tuning settings used to instill preferences and elicit introspective reports.
- Decision contexts: The study generated 100 agent-specific decision contexts, with five quantitative dimensions per context, and made the contexts available in the project repository.
- Prompts: Preference training and verification used a decision prompt requiring the model to choose Option A or B for a hypothetical agent.
- Decision contexts: Example decisions varied across products such as central vacuum systems, whose options differed in suction power, noise, capacity, hose reach, and filtration efficiency.
- Prompts: The introspection prompt required JSON reports assigning each of five dimensions a weight from -100 to 100 without reporting the decision itself.
- Hyperparameters: Fine-tuning used three epochs, learning-rate multipliers of 2 for GPT-4o and 1.8 for GPT-4o-mini, and batch sizes of 10 for preference training versus 1 for introspection training.
C Statistical models
The analyses estimate attribute weights from model choices with logistic regressions, then relate those weights to agents’ true or introspected weights using standardized regressions.
- Attribute weights were estimated by fitting logistic regressions to models’ choices.The predictors were normalized option differences for each dimension.
- All weight parameters used standard normal priors with mean 0 and variance 1.The Bayesian analyses were modeled with brms and Stan, and 95% HDIs were calculated using bayestestR.
- Correlations between model weights and agents’ true or introspected weights were estimated through regressions with standardized variables.Standardization makes regression coefficients correspond to correlation coefficients, while terms distinguish introspection-trained models when appropriate.
- The analyses used brms default priors for the correlation models where applicable.
D Supplementary results
Supplementary analyses show that the reported correlations are not materially driven by the scaling convention or by imperfect learning of target weights.
- Excluding weights fixed at 100 or -100 produced no substantive changes, with no correlation changing by more than .06.These extreme values arose because each context’s largest absolute weight was scaled to 100 or -100.
- Experiment 1 evaluated self-reports against both learned weights and the randomly generated target weights.The comparison tested whether imperfect acquisition of target preferences explained the self-report results.
- Reported correlations with target weights reached r = 0.68 and 0.70 after introspection training, versus 0.74 and 0.75 with learned weights.Before introspection training, target-weight correlations were r = 0.49 and 0.45, compared with 0.54 and 0.50 for learned weights.
- Experiment 2 trained self-reporting against target weights because learned weights were similar to the intended targets.The authors regarded target-weight supervision as appropriate because the models learned the target weights well.