Source-linked AI summary
Rethinking Interpretability in the Era of Large Language Models
Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, Jianfeng Gao
TL;DR
LLM interpretation seeks trustworthy ways to understand models and data, especially where limited interpretability constrains high-stakes and scientific applications. The paper reviews evaluation approaches and argues that LLMs can broaden interpretability through natural-language explanations, dataset analysis, and interactive use, while hallucination and computational scale remain central challenges.
Problem
Limited interpretability constrains LLM use in high-stakes applications and limits applications in science and data analysis.
Method
The paper reviews how LLMs are interpreted and used for explanation, evaluates emerging approaches, and organizes opportunities around model explanation and dataset explanation.
Results
The paper concludes that LLMs can broaden interpretability through more elaborate explanations and support practical applications including model auditing, model improvement, and human task performance.
Takeaways & Limitations
Dataset interpretation and interactive explanation are highlighted as priorities for extending interpretability toward knowledge discovery and accessible analysis.
Takeaways & Limitations
LLM interpretation remains constrained by hallucinated explanations and the immense size, computational cost, and opacity of modern LLMs.
Abstract
from arXiv · showhide
Interpretable machine learning has exploded as an area of interest over the last decade, sparked by the rise of increasingly large datasets and deep neural networks. Simultaneously, large language models (LLMs) have demonstrated remarkable capabilities across a wide array of tasks, offering a chance to rethink opportunities in interpretable machine learning. Notably, the capability to explain in natural language allows LLMs to expand the scale and complexity of patterns that can be given to a human. However, these new capabilities raise new challenges, such as hallucinated explanations and immense computational costs. In this position paper, we start by reviewing existing methods to evaluate the emerging field of LLM interpretation (both interpreting LLMs and using LLMs for explanation). We contend that, despite their limitations, LLMs hold the opportunity to redefine interpretability with a more ambitious scope across many applications, including in auditing LLMs themselves. We highlight two emerging research priorities for LLM interpretation: using LLMs to directly analyze new datasets and to generate interactive explanations.
1. Introduction
The paper positions LLMs as an opportunity to broaden interpretability through expressive natural-language explanations, while introducing hallucination, scale, cost, and opacity challenges. It organizes this research around explaining LLMs, explaining datasets, and interactive explanation as an emerging priority.
- 1. Introduction: LLM interpretability matters because poor understanding constrains high-stakes use, raises safety and regulatory concerns, and limits scientific and data-analysis applications.
- 1. Introduction: LLMs can provide more elaborate, targeted natural-language explanations than restricted interfaces such as saliency maps.Users can ask questions about model logic, alternative answers, or data patterns and receive immediate responses.
- 1. Introduction: The paper reviews LLM interpretation broadly, emphasizing aspects unique to LLMs and practically useful methods rather than exhaustively cataloging techniques.
- 1. Introduction: The paper distinguishes explaining an existing LLM for auditing from using an LLM to analyze a new text or tabular dataset.
- 1. Introduction: Interactive explanation and dataset explanation are highlighted as priorities with potential significance for science, statistics, and data analysis.
2. Background: definitions and evaluation
The paper defines LLM interpretation as extracting relevant knowledge about relationships in data or learned by a model, with relevance determined by the problem and audience. It argues that interpretations should be evaluated through real-world human use when possible, while automated and model-based evaluations require careful design against bias.
- Definitions: LLM interpretation means extracting relevant knowledge about relationships contained in data or learned by a model.The definition covers both interpreting an LLM and using an LLM to generate explanations.
- Definitions: An interpretation is relevant when it is useful for a particular problem and audience, such as judging prediction trustworthiness in medicine.
- Definitions: The paper defines LLMs as transformer-based language models with tens to hundreds of billions of parameters, pretrained on massive text data and often further aligned or adapted.
- Evaluation: The ideal evaluation tests whether people using an interpretation achieve a desired outcome in a real-world setting, rather than relying only on human judgments of explanation quality.
- Evaluation: Automated evaluation can improve scale, but LLM-based scoring requires structured procedures because models may systematically rate their own outputs too positively.
- Evaluation: Explanations can also be evaluated by whether they improve model performance, although this does not cover all interpretability uses involving human interaction.
3. Unique opportunities and challenges of LLM interpretation
LLMs expand interpretability through natural-language and interactive explanations, but their flexibility introduces hallucination, scale, opacity, and computational-cost challenges.
- Natural language lets LLMs explain complex patterns across modalities at different levels of granularity, potentially using evidence and counterfactuals.
- Interactive explanations let users tailor analyses through follow-up questions and enable independent auditing of decomposed LLM calls.
- Hallucinated explanations can be incorrect or baseless, making them unhelpful or misleading and motivating techniques to detect and reduce hallucination.
- LLMs’ immense size and opacity make human inspection infeasible and require efficient methods that may work without model weights or gradients.
4. Explaining an LLM
The paper surveys methods for explaining individual LLM generations and whole models, spanning attribution, natural-language, mechanistic, training-data, and interactive approaches.
- LLM explanation covers both individual generations and entire models, combining traditional interpretable-ML techniques with LLM-based methods.
- Local explanation: Feature-attribution methods assign relevance scores to input tokens using perturbation-based, gradient-based, or linear-approximation techniques.
- Local explanation: Natural-language explanations can elucidate predictions and simulate counterfactual scenarios, extending local explanation beyond token-level attributions.
- Global and mechanistic explanation: Mechanistic approaches probe representations, neurons, attention heads, circuits, and miniature models to understand model behaviors and capabilities.
- Global and mechanistic explanation: Training-data interpretation can use influence functions and known data distributions to study how patterns such as long-tail or repeated data relate to model behaviors.
- Global and mechanistic explanation: Interactive interfaces let users investigate model components through follow-up queries and altered prompts, while mechanistic insights increasingly inform model editing, instruction following, and compression.
5. Explaining a dataset
LLMs can explain datasets by interactively analyzing tabular data and constructing interpretable or partially interpretable models for text, while natural-language descriptions remain vulnerable to hallucination.
- LLMs can explain entire datasets to support data analysis, knowledge discovery, and scientific applications, across tabular and text data.
- Tabular data: Interactive interfaces let LLMs analyze tabular data by combining code, text, and numerical understanding to build visualizations.
- Tabular data: LLMs can analyze fitted generalized additive models to explain patterns in tabular data rather than explaining the predictive model itself.
- Text data: Because text is sparse, high-dimensional, and interaction-rich, conventional interpretable models have struggled to model it accurately.
- Text data: LLMs can build interpretable text models or decision trees whose important words or ngrams reveal patterns associated with different outcomes.
- Text data: Prompt chains partially expose dataset patterns by decomposing an LLM’s decision-making process into sequences of explanation-generating calls.
- Natural-language dataset descriptions can be rich and concise but remain prone to hallucination, motivating verification against patterns in the underlying data.
6. Future research priorities
The paper identifies explanation reliability, dataset explanation, and interactive explanations as priorities for advancing LLM interpretation. These directions target trustworthy explanations, knowledge discovery, improved human performance, and user-centric model auditing.
- Explanation reliability: LLM explanations remain bottlenecked by hallucinations, prompt sensitivity, and failures to use context, making reliability a central research priority.Studies find explanations can conflict with model outputs, fail to entail predictions, or lack factual grounding in inputs.
- Dataset explanation for knowledge discovery: Dataset explanation could support scientific hypothesis generation, human screening, and new data-driven insights beyond merely speeding up analysis or visualization.The paper frames dataset explanation as a route toward knowledge discovery from text or tabular data.
- Dataset explanation for knowledge discovery: LLM explanations can improve human task performance, including among expert chess players, and help people understand, audit, and improve expert behavior.The paper gives medical decision-making as an example of explaining expert human behavior.
- Interactive explanations: Interactive explanations can let users specify tasks, receive recommendations, converse with systems, and audit models through dialogue.The paper identifies conversational auditing as an application that can combine with other interpretation methods.
7. Conclusions
The paper argues that LLMs’ natural-language capabilities create more elaborate and accessible interpretations of complex data and model behavior. It positions reliable explanations and dataset interpretation for knowledge discovery as priorities for realizing this broader scope.
- 7. Conclusions: LLMs’ natural-language generation enables more elaborate and nuanced explanations, supporting deeper and more accessible understanding of complex data and model behaviors.The conclusion presents LLM integration into interpretability as a potentially transformative shift rather than a simple enhancement.
- 7. Conclusions: Future research should improve explanation reliability and advance dataset interpretation for knowledge discovery as LLM capabilities continue to develop.The paper links these directions to future applications and insights.