Source-linked AI summary
From Interpretability Methods to Interpretable Models
Julien Colin, Nuria Oliver, Thomas Serre
TL;DR
XAI has developed a mature toolkit, but has mostly evaluated explanation methods rather than the interpretability of models. This paper redirects attention toward comparing model representations and computations and directly measuring human understanding, arguing that the instruments for both are now available.
Problem
Despite a mature XAI toolbox, relatively little work determines how interpretable models are, whether they are improving, or which aspects remain unresolved.
Method
The paper reviews XAI methods, surveys model-comparison work, and proposes structural comparison plus psychophysical measurement of human understanding.
Results
Existing tools can characterize and compare model representations and computations, while psychophysics provides a direct way to measure whether people understand models.
Takeaways & Limitations
Interpretability should be treated as a measurable, comparable, and steerable property of models rather than methods alone.
Takeaways & Limitations
Large-scale human evaluations remain prohibitively expensive, motivating scalable proxies that are validated against direct evaluation without replacing it entirely.
Abstract
from arXiv · showhide
More than a decade in, explainable AI (XAI) for computer vision has assembled a mature toolbox: attribution, feature visualization, concept-based, and circuit-based methods. Yet almost all of the field's effort has gone into building and comparing these methods, and little into the question they were meant to answer---how interpretable are our models, and are we making progress as they evolve? We argue for shifting the field's focus from methods to models, along two complementary lines. One is already within reach: existing tools let us characterize and compare what different models represent and compute. The other is harder, and largely neglected: whether a model can actually be understood by the humans who rely on it---the independent evaluators on whom trust and certification depend, not the experts confirming what they already expect. It can only be measured, not inferred. We review why the toolbox is mature enough to support both, survey the thin body of work comparing models, draw a parallel to systems neuroscience, and close with a model-centric XAI agenda.
1 Introduction
XAI is increasingly needed as capable AI systems enter high-stakes domains while remaining opaque to the people who rely on them. Although the field has developed many explanation methods, it has devoted relatively little effort to determining how interpretable models themselves are.
- Motivation: AI systems now influence high-stakes medical, autonomous-driving, and robotic decisions while remaining largely opaque to users.Interpretability matters for catching failures, meeting emerging explanation expectations, and learning from models that make scientific discoveries.
- What interpretability means: Interpretability is fundamentally human-centric: an explanation counts only if a person can understand it.The term covers multiple distinct properties, including architectures designed around prototypes, concepts, or self-explanations.
- Field focus: Over roughly fifteen years, XAI has concentrated on developing and comparing attribution, visualization, concept-based, and circuit-based methods.Human evaluations have mostly tested how well methods help people understand models, rather than comparing the models being explained.
- Proposed shift: The paper argues that existing tools are mature enough to shift attention toward measuring model interpretability and tracking how it changes as models evolve.This involves both comparing model representations and computations and measuring whether independent evaluators can understand them.
2 The XAI toolbox has matured
Computer vision XAI now offers four complementary method families that address representations and computation, with enough maturity to support systematic model comparison. Their limitations remain important, especially because available explanations do not guarantee human understanding.
- Human understanding: Architectures designed to expose explanations guarantee that an explanation is available, not that humans can understand the model.The paper therefore treats “interpretable by design” methods as closer to explainable by design.
- Toolbox overview: The toolbox comprises attribution, feature visualization, concept-based, and circuit-based methods, each addressing a different interpretability question.Together they form a selected core rather than an exhaustive taxonomy, and none fully accounts for model reasoning.
- Attribution methods: Attribution methods identify image regions or internal activations most responsible for a model’s prediction.They include gradient-based, activation-based, and perturbation-based approaches.
- Attribution methods: Attribution methods remain imperfect after criticism concerning reliability, robustness, faithfulness, evaluation, and transfer to newer architectures.Recent progress has emphasized theoretically grounded or statistically principled methods rather than fundamentally new approaches.
- Evidence of maturity: Human studies show attribution maps can help users predict unseen model behavior and improve visual-search and human–AI teaming performance.This evidence supports using attribution as a reliable model-comparison instrument without treating it as a complete explanation of reasoning.
- Complementary methods: Feature visualization complements attribution by showing what a model attends to, while unconstrained synthesis can produce adversarial-looking or unfaithful stimuli.Concept-based methods instead use human-defined concept directions, but their explanations are limited to the tested vocabulary.
- Circuit-based methods: Circuit-based methods reverse-engineer connected features and weights implementing behavior, shifting analysis from representations to computation.They complete the toolbox but remain comparatively niche.
3 Why Now
The case for studying models now is supported by scientific precedent and a converging vision-model landscape. Existing tools can increasingly compare representations and computations across models rather than chasing every architectural variation.
- Scientific precedent: Neuroscience shows that imperfect instruments can support cumulative scientific understanding of complex systems.Early recordings and simple stimuli produced discoveries that were later refined by more powerful measurement and causal tools.
- Scientific precedent: Early single-electrode methods favored a serial feedforward account of vision, while later work revealed recurrent processing and feedback.The later findings refined rather than invalidated the reality of rapid feedforward processing.
- Model landscape: Vision models are becoming less architecturally diverse as dominant transformer backbones and training paradigms replace a fragmented convolutional landscape.These models increasingly produce broadly comparable visual representations despite differing supervision and objectives.
- Why now: Because models increasingly share comparable representations, existing XAI tools can systematically compare how training objectives and datasets shape representations and computations.This makes model comparison more feasible than continually developing methods for changing architectures.
- Comparison challenge: Model comparison seeks to characterize shared and differing representations and relate those differences to properties of interest, despite each model’s idiosyncratic basis.Prior work uses aggregate comparisons such as concept-detector counts and average human interpretability.
4 Interpretability methods to study models
A small but growing literature applies interpretability tools directly to models, finding comparative structure in representations while leaving human understanding largely unmeasured. Psychophysics offers a direct instrument for that missing evaluation, but systematic model studies remain uncommon.
- Existing model studies: Detailed circuit, feature-visualization, and concept-based studies have begun characterizing what individual vision models represent and compute.Examples include curve detectors in InceptionV1 and multimodal neurons in CLIP.
- Cross-model comparisons: Across models, interpretability can dissociate from both scale and capability, suggesting it is a measurable model property.How interpretability can be steered remains unclear.
- Cross-model comparisons: Models show mixed patterns of conceptual convergence, while representational comparison tools test shared structure across architectures and model zoos.Both convergence and divergence are informative: one identifies recurring concepts, while the other maps variation in interpretability.
- Human evaluation: Psychophysics can measure whether people identify preferred images, predict model behavior, improve teamwork, or understand generated explanations.Human understanding depends on the complexity, granularity, and retention demands of the information presented.
- Open gap: These human-evaluation methods have mostly compared XAI methods rather than asking which model people understand best.Systematic model analyses and head-to-head comparisons also face weak publication incentives.
5 A Research Agenda for Model-Centric XAI
The agenda shifts XAI from improving explanatory methods to measuring and comparing model interpretability, beginning with structural analysis and ultimately requiring validated measures of human understanding. It also treats model-centric study as a way to expose missing tools and guide the design of more understandable models.
- Defining human understanding is the prerequisite for stable interpretability comparisons, metrics, and automated proxies.
- Model-centric study can drive method progress by revealing phenomena such as polysemantic neurons that resist single-neuron characterization.
- Systematic model comparisons remain limited, leaving unresolved how far design-based and post hoc approaches have advanced toward human interpretability.
- Interpretability is dissociated from scale and task performance, while alignment, activation locality, and robustness remain candidate predictors rather than established explanations.
- Structural comparison is the most tractable starting point because crosscoders and universal sparse autoencoders enable direct comparison of representations and features across models.
- Human understanding remains the bottleneck: psychophysics is expensive, and automated proxies should follow principled human measurement rather than define interpretability prematurely.
6 Conclusion
The conclusion reframes interpretability as a measurable property of models rather than methods, enabled by a mature XAI toolbox. It presents the central challenge as bridging complete model access with genuine human understanding for trust, oversight, certification, and scientific learning.
- XAI should turn its mature tools from explaining individual models toward measuring, comparing, and steering model interpretability as models evolve.
- Model interpretability matters beyond research because it supports trust, oversight, certification, and learning from systems that may rival or surpass human performance.