Source-linked AI summary
Generative Interpretability via Scalable Neuro-Symbolic Models
Xiaocong Yang
TL;DR
Post-hoc interpretability explains completed model behavior but, the paper argues, does not provide reliable, scalable intervention for agentic LLMs. The authors propose generative interpretability—native, human-readable, causally intervenable checkpoints during inference—and argue it is a minimal architectural commitment for auditable LLMs, with Neuro-Symbolic Models as an instantiation.
Problem
Post-hoc methods explain trained models but cannot change their behavior systematically and scalably, a limitation relevant to auditing LLMs deployed in agentic settings.
Source: §1 · p. 1, §2.3 · p. 2Method
The paper proposes generative interpretability: an architecture exposes native, human-understandable checkpoints during inference that support causal intervention.
Source: §3 · p. 2, §3.1 · p. 2Results
The authors argue generative interpretability is the minimal architectural commitment sufficient for audit and intervention over LLM inference that is scalable to develop and tractable to deploy.
Source: §3.3 · p. 3Takeaways & Limitations
The authors propose Neuro-Symbolic Models as an instantiation, with checkpoint audits ideally marking loci of responsibility for undesirable outcomes.
Source: §4.2 · p. 3, §5 · p. 4Post-hoc tools face unreliable intervention and scalability or efficiency problems; lightweight tools are behavior-specific and depend on labeled data.
Source: §2.3 · p. 2
Abstract
from arXiv · showhide
As the use of Large Language Models moves from chatbots into agentic systems, where outputs become actions with irreversible consequences on reality, the existing paradigm on AI Interpretability research, post-hoc interpretability, is structurally inadequate for safe and trustworthy model deployment: it explains behavior after the fact but cannot audit or intervene in an inference computation before it commits to an output. We therefore argue for a shift toward \emph{generative interpretability}, an architectural property under which a model's inference pass natively exposes semantically meaningful checkpoints that are human-understandable and amenable to causal intervention. We show the merits of generative interpretability as comparison to other interpretability research paradigms, and propose Neuro-Symbolic Models as a concrete instantiation.
1 Introduction
The introduction argues that post-hoc interpretability is inadequate for auditing and intervening in LLM behavior as models move into agentic settings. It proposes generative interpretability as an alternative built into model computation.
1 Introduction
- Agentic systems can turn LLM outputs into real-world actions with irreversible consequences, including in healthcare and autonomous driving.Documented AI safety incidents surged 56.4% from 2023 to 2024 [25].Source: §1 · p. 1
- Post-hoc interpretability relies on external reverse-engineering tools and, the authors argue, cannot reliably translate into safer deployed behavior.The tools are confined to observation, can have approximation errors, and are usually discarded at deployment.Source: §1 · p. 1
- The authors propose generative interpretability as an inherent model property that supports audit and intervention during deployment.They frame the proposal as targeting research priorities for auditable autonomy and operational runtime control.Source: §1 · p. 1
2 Post-hoc Interpretability and Its Limitations
This section reviews post-hoc approaches to interpreting trained models, then identifies their shared difficulty: they do not provide systematic, scalable intervention in model behavior.
2 Post-hoc Interpretability and Its Limitations
- Post-hoc interpretability explains a trained model’s computations and behavior in human-understandable terms, with anatomical and physiological forms.Anatomical approaches study stable local structures, while physiological approaches recover information flow for a particular inference pass.Source: §2 · p. 1, §2.1 · p. 1, §2.2 · p. 1
2.1 Anatomical Interpretability
- Anatomical studies examine stable model structures, including Transformer attention heads and feed-forward networks associated with semantic processing and factual knowledge [9] [10] [15].The cited work also reports that attention heads in BERT specialize in semantic or syntactic information [9], and that feed-forward networks can be edited to change outputs on certain facts [26].Source: §2.1 · p. 1
2.2 Physiological Interpretability
- Physiological interpretability traces a specific inference pass as a causal graph, with Transformer Circuit Theory analyzing semantically meaningful features and their roles in generating outputs.Its cited tools include probing [4], saliency tracing [37] [38], and circuit tracing with attribution graphs [12].Source: §2.2 · pp. 1–2
2.3 The Shared Fundamental Limitation
- Post-hoc tools struggle to change behavior systematically: steering may not close the gap between representations and outputs, while other tools face deployment or scalability limits.Sparse Autoencoders and Cross-layer Transcoders have feature dimensions that impede efficient integration; model editing [43] and activation patching [52] are behavior-specific and depend on labeled data.Source: §2.3 · p. 2
3 Generative Interpretability
The paper defines generative interpretability as native, human-readable and causally intervenable checkpoints during inference. It compares this commitment with other interpretability paradigms and argues that multiple interior checkpoints offer a minimal sufficient basis for auditing LLM inference.
3.1 Definition
- Generative interpretability requires an inference pass to expose intermediate checkpoints that are natively human-meaningful and causally intervenable.The checkpoints must map to human-readable concepts and be variables on which the inference pass functionally depends, so interventions have well-defined counterfactuals by construction.Source: §3.1 · p. 2
3.2 Two Axes of Interpretability
- The paper compares interpretability approaches by when human-understandable states arise and where they sit along the inference path.It distinguishes white-box models readable before inference, post-hoc states reconstructed after inference, and during-inference approaches such as CBMs [19].Source: §3.2 · p. 2
- Generative interpretability occupies an intermediate position: checkpoint schemas may be predefined, but their values emerge as input is processed, at multiple interior locations.The authors argue that boundary readability and a single checkpoint do not suffice for effective causal intervention.Source: §3.2 · p. 3
3.3 Generative Interpretability as the Minimal Sufficient Commitment for Auditable LLMs
- For LLMs, the authors argue that generative interpretability is the minimal architectural commitment sufficient for inference-pass audit and intervention while remaining scalable to develop and tractable to deploy.They prescribe it as a property to build into models based on existing LLMs, rather than a property of LLMs as they currently are.Source: §3.3 · p. 3
- The paper orders architectural commitments by auditable content: throughout-readability subsumes multiple checkpoints, which subsume a single checkpoint and then input-output boundaries.Post-hoc interpretability lies outside this chain because its readable structure is reconstructed externally rather than residing in the model.Source: §3.3 · p. 3
- The authors argue that weaker commitments fall short: output-boundary text cannot audit computation before generation, and one interior checkpoint cannot attribute an undesired output to its computation.They also argue that compositional sparsity in efficiently computable functions calls for at least multiple checkpoints to model the relevant structures.Source: §3.3 · p. 3
- The paper argues that requiring semantic readability throughout a model would harm representational capacity, whereas limited checkpoints leave most of the model unconstrained.The authors describe superposition as packing exponentially many near-orthogonal directions into a d-dimensional space; requiring monosemantic directions would reduce meaningful concepts from exponential to linear in dimension.Source: §3.3 · p. 3
4 Neuro-Symbolic Model as an Instance with Generative Interpretability
Neuro-symbolic models are presented as a concrete instance of generative interpretability: symbolic states provide human-understandable checkpoints within an inference pass. Scaling this approach to LLMs faces challenges in building symbolic systems, integrating components, and deciding where neural and symbolic representations should meet.
4 Neuro-Symbolic Model as an Instance with Generative Interpretability
- Symbolic systems project onto predefined vocabularies to produce discrete, human-understandable states, whereas neural networks learn continuous, distributed representations.The paper contrasts symbolic algebraic generalization with neural interpolation.Source: §4 · p. 3
4.1 Symbolic Components as Checkpoints
- Type-3 models use predicate-symbol proof steps as auditable checkpoints, while similarity scoring inside each step remains opaque.In the refund example, audit and intervention are designed at the high-stakes refund-amount checkpoint.Source: §4.1 · p. 3, Figure 1 · p. 4
4.2 Challenges of Building Neuro-Symbolic Models with LLMs
- Most existing neuro-symbolic models use small neural components and remain confined to specific task domains, limiting their scalability.Source: §4.2 · p. 3
- Building a universal symbolic system from scratch is intractable; one proposed alternative is to align, merge, and extend existing domain-specific systems.The paper cites Cyc as a project that began in the 1980s and was paused in recent years amid information explosion; the alternative is under study.Source: §4.2 · pp. 3–4
4 Neuro-Symbolic Model as an Instance with Generative Interpretability
- Neuro-symbolic models place generative interpretability in a region that permits audit and intervention before undesired content while preserving distributed, high-capacity representations.The figure organizes interpretability paradigms along temporal and spatial axes.Source: Figure 2 · p. 4
4.2 Challenges of Building Neuro-Symbolic Models with LLMs
- Existing neural-symbolic integrations often reduce to fixed, task-specific workflows, while shared templates for task categories risk oversimplifying symbolic structure.Source: §4.2 · p. 4
- Neural components suit knowledge governed by graded similarity, while symbolic components capture discrete rules; locating their boundary depends on characterizing the knowledge involved.The paper uses near-synonyms as an example of graded similarity and hash correctness or formal syntax as rule-defined.Source: §4.2 · p. 4
5 Conclusion
The conclusion calls for shifting interpretability from explaining trained models to building models that can be audited and intervened on. It connects checkpoint audits to responsibility for outcomes and calls for institutional checks and sustained collaboration.
5 Conclusion
- Generative interpretability shifts the aim from observing trained models to building auditable models with human-understandable, causally intervenable checkpoints along inference.Source: §5 · p. 4
- In high-stakes agentic settings, checkpoint audits could identify loci of responsibility so an undesired outcome traces to a particular step rather than the whole system.Source: §5 · p. 4
- The mission also calls for checks and balances on AI audit power and sustained collaboration among universities, frontier AI labs, and governments.Source: §5 · p. 4