Interpreting Language Models Through Concept Descriptions: A Survey

• Earlier methods used probes or human-defined concepts to interpret components, but their fixed vocabularies can miss representations learned by the model (Conneau et al., 2018; Belinkov, 2021; Kim et al., 2018).Mechanistic interpretability seeks to reverse-engineer model algorithms by analyzing components such as neurons and attention heads (Ferrando et al., 2024).Source: §1 · p. 1
• LLMs can generate open-vocabulary descriptions of another model’s components from activation data, such as text fragments that maximally activate a neuron (Cunningham et al., 2024).The approach can describe concepts a component appears to detect, rather than testing only predefined concepts.Source: §1 · p. 1
• The description approach applies to native model components and learned abstractions such as Sparse Autoencoder features.Source: §1 · p. 2
• The survey structures its overview around methods, datasets, and evaluation techniques for generating concept descriptions of LLM internals.Source: §1 · p. 2
• It asks what methods and data sources are used for generation and what best practices can guide evaluation (§3–§4).Source: §1 · p. 2
• It also examines research trends and identifies gaps and promising future directions for concept-based descriptions of language models (§5–§6).Source: §1 · p. 2

2.1 Language Models

• Neurons are FFN intermediate dimensions that activate when their value is positive, but superposition can make them polysemantic and entangled rather than clean concepts (Geva et al., 2022; Elhage et al., 2022).Attention heads process different representation subspaces in parallel and can specialize in token relations such as syntax, coreference, and in-context retrieval (Clark et al., 2019; Kahardipraja et al., 2025).Source: §2.1 · p. 2

2.2 Concepts

• A concept is a human-understandable description of a learned pattern or property, ranging from lexical and syntactic features to high-level semantic categories (Dalvi et al., 2022).Ground-truth concepts may come from predefined taxonomies or classification datasets, human annotation, or post hoc validation; LLM topic-modeling approaches also assign human-readable semantics or explanations to text clusters (Schwettmann et al., 2023; Dumas et al., 2025; Antognini and Faltings, 2021; Abraham et al., 2022; Jourdan et al., 2023; Singh et al., 2023; Sun et al., 2025; Dalvi et al., 2022; Mousi et al., 2023; Yu et al., 2024; Rajani et al., 2022; Lam et al., 2024; Wang et al., 2023b).Source: §2.2 · pp. 2–3

2.3 Natural Language Explanations

• LLM-generated natural-language explanations can convey implicit reasoning and appear plausible, but lack faithfulness guarantees and vary with prompts and generator models (Camburu et al., 2018; Lanham et al., 2023; Chen et al., 2024; Bentham et al., 2024; Bartsch et al., 2023).They may also be overly broad or generic rather than capturing the nuances of a component’s represented concepts.Source: §2.3 · p. 3

2.4 Concept Datasets

• Concept descriptions are bounded by training data: models are unlikely to represent absent concepts, creating an interpretability illusion (Bolukbasi et al., 2021).Human-labeled datasets such as CEBaB and aspect-based sentiment resources enable controlled alignment tests, but restrict description vocabulary and may exclude concepts not obvious to human observers (Abraham et al., 2022; McAuley et al., 2012; Wang et al., 2010; Hewitt et al., 2025).Source: §2.4 · p. 3

3.1 Model Components

• GPT-4-generated explanations from strongly activating samples launched automated neuron labeling, while most such methods assign one description per neuron and struggle with polysemanticity (Gur-Arieh et al., 2025).The same top-activating-sample process can generate descriptions for neurons and SAE features.Source: §3.1 · p. 4, S SAE Features · p. 4, Figure 2 · p. 4

A Attention Heads

• MAPS infers attention-head functions from parameter matrices projected into vocabulary space, while Neo et al. (2024) use an LLM to generate and validate hypotheses about head functions.MAPS measures how strongly a head implements predefined relations, such as Country to capital or Word to antonym; Neo et al. examine attention-MLP interactions including copying from previous tokens.Source: A Attention Heads · p. 4, Figure 2 · p. 4

S SAE Features

• SAEs decompose activations into higher-dimensional, sparsely activated feature spaces that may capture more interpretable concepts, and neuron auto-interpretability methods have been extended to these features (Cunningham et al., 2024).SAEs have also been reported to underperform linear probes.Source: S SAE Features · pp. 4–5
• Circuit analysis represents interacting components as task-specific computational graphs; automated methods describe sparse-feature circuits, but natural-language descriptions of arbitrary higher-order circuits remain an open challenge.Earlier circuit studies used manual analysis and human labeling, including for indirect object identification; the IOI circuit is a benchmark task in the MIB suite.Source: Figure 2 · p. 4, S SAE Features · p. 5

4.1 P Predictive Simulation

• 1,000 of 307,200 neurons scored at least 0.8 on Bills et al.’s (2023) [0.0, 1.0] simulator-correlation scale, where an LLM predicts token activations from a description.The metric’s reliance on an opaque simulator can confound description quality with the simulator’s pattern-matching ability; classification variants include Detection and Fuzzing and Automated Simulatability.Source: §4.1 · pp. 6–7

4.2 I Input-based Evaluation

• Input-based metrics assess purity and coverage by comparing on-concept with distractor inputs; Specificity and Purity quantify selective activation, while AUROC and mean activation differences measure separation.The cited examples are Specificity, Purity, AUROC (Kopf et al., 2025), and mean activation differences (Gur-Arieh et al., 2025; Kopf et al., 2025).Source: §4.2 · p. 7

4.3 O Output-based Evaluation

• Output-based evaluation tests causal faithfulness through interventions or steering, checking whether feature effects on model outputs match the description; Paulo et al. (2024) include Intervention and Surprisal Scoring.Surprisal Scoring measures whether providing a description reduces model loss on relevant inputs as a proxy for causal impact.Source: §4.3 · p. 7

4.4 ≈Semantic Similarity

• When concepts are known, semantic similarity compares embedded descriptions with ground-truth labels; for polysemantic features, it can compare multiple descriptive annotations instead.The usual approach embeds the generated text and concept name with a sentence-embedding model and calculates cosine similarity; this is used as a sanity check or on features with known correlates. Kopf et al. (2025) compare multiple descriptions for a feature.Source: §4.4 · p. 7

4.5 H Human Evaluation

• Human evaluation assesses description plausibility (Elhelo and Geva, 2025), readability, ground-truth labels (Kopf et al., 2025), and causal faithfulness (Gur-Arieh et al., 2025).It is less common than in related NLP subfields because labeling polysemantic components is difficult, costly, and ambiguous, and the field traditionally favors expert-centric debugging explanations over layperson-understandable ones (Geva et al., 2022).Source: §4.5 · pp. 7–8

6 Recommendations for Future Work

• Scale automated descriptions from individual neurons or features to computational subgraphs, explaining how multiple features interact to implement complex functions.Circuit discovery is active, but describing entire circuits is identified as the next frontier.Source: §6 · p. 8

Scaling to New Data Domains and Modalities

• Extend concept description methods beyond general web text to specialized domains, multilingual models, and multimodal models, including investigation of unified descriptions across modalities.Examples of specialized domains include legal text, medical data, and source code.Source: Scaling to New Data Domains and… · p. 8
• Analyze LLM interpreters’ linguistic properties, biases, and conceptual limitations, including whether they favor simple atomic concepts over abstract or relational ones.The survey cautions that an interpreter’s biases could distort explanations of another model.Source: Scaling to New Data Domains and… · p. 8

A Finer-Grained View of Polysemanticity

• Develop description methods that capture fine-grained activation levels and ranges, since a feature may represent different concepts at different intensities (Haider et al., 2025).The passage identifies PRISM (Kopf et al., 2025) as an example of a method aimed at capturing this detail.Source: A Finer-Grained View of Polysemanticity · p. 8
• Scale intervention-based tests of feature effects on outputs and develop adversarial or out-of-distribution stress tests, because most automated metrics remain correlational.The passage notes that critiques of SAE feature steerability indicate their causal impact is not fully understood.Source: A Finer-Grained View of Polysemanticity · p. 8
• Create standardized benchmarks for natural-language descriptions, with components and agreed-upon gold descriptions to systematically compare description and evaluation methods.Source: A Finer-Grained View of Polysemanticity · p. 9

7 Conclusion

• The field has progressed from labeling individual, often polysemantic neurons to describing higher-level abstractions such as SAEs and circuits, while evaluation has expanded beyond predictive correlations.The toolkit now includes input-based purity, causal faithfulness, and nuanced human judgment.Source: §7 · p. 9
• Generated descriptions can support model analysis, debugging, and auditing, but high computational costs and debate over their faithfulness warrant critical use.A generated description does not yet provide a fully verifiable causal mechanism.Source: §7 · p. 9
• Refining generation methods and developing more rigorous, standardized evaluation benchmarks are key to making these techniques dependable tools for robust, transparent, and trustworthy NLP systems.Source: §7 · p. 9

Limitations

• The survey excludes mathematical analyses, non-text feature visualizations, and methods explaining final predictions rather than internal functions.Its scope is natural-language concept descriptions for internal model components and abstractions.Source: Limitations · p. 9
• Training high-quality SAEs and generating descriptions and automated evaluations require substantial GPU resources and often expensive proprietary-model API calls.These costs pose barriers to wider adoption and reproducibility, particularly in academic settings.Source: Limitations · p. 9
• Most surveyed research uses English-language models and text corpora, leaving generalization of its methods and discovered concepts to other languages an open question.The survey identifies this as an important area for future investigation.Source: Limitations · p. 9