Source-linked AI summary
ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing
Roshan Prakash Rane, Marco Simnacher, Manuel Pfeuffer, Marc-Andre Schulz, Nys Tjade Siegel, Maximilian Dreyer, Frederik Pahde, Wojciech Samek, Sonja Greven, Kerstin Ritter
TL;DR
Existing concept-based explanations can mistake correlated, decodable concepts for concepts the model uses. ICON instead performs multivariate variance decomposition across concepts and outcomes, yielding more accurate and sparse explanations that were supported by retraining and out-of-distribution tests. Its claims remain bounded by linear encoding, supplied-concept coverage, and an unexplained share that cannot distinguish missing from nonlinear concepts.
Problem
Univariate probes and CAVs can conflate a concept’s importance with correlated concepts or outcomes, while their scores are not consistently comparable across layers or concept types.
Method
ICON combines partial least squares with Type I sum-of-squares allocation to decompose layer variance jointly among correlated concepts and the outcome.
Results
ICON recovered simulated concept importance more accurately than seven baselines and isolated genuine shortcuts or sparse attributions in skin-lesion and brain-imaging models.
Takeaways & Limitations
Variance-based, layer-comparable scores can support sparse, testable model-auditing hypotheses and quantify representation variance unexplained by the supplied concepts.
Takeaways & Limitations
ICON assumes linear concept encoding, cannot separate missing from nonlinear concepts in the unexplained share, and does not discover concepts outside the supplied set.
Abstract
from arXiv · showhide
Deep neural networks often exploit spurious associations in their training data, a failure known as shortcut learning. Concept-based explainability methods screen for shortcuts by testing whether concepts such as a patient's sex or scanner settings can be decoded from a network layer. Because each concept is evaluated in isolation, these methods can mistake correlations between concepts as evidence that the model uses them. We introduce ICON decomposition, which instead quantifies how much of a layer's variance each concept explains after accounting for all other concepts and the outcome. On synthetic data with known ground truth, ICON recovers concept importance more accurately than seven alternative baseline methods. On skin-lesion and brain-imaging models, it isolates the concepts on which a model genuinely relies, quantifies the representation unexplained by any of the supplied concepts, and yields sparse explanations that we validate by retraining and out-of-distribution testing.
1 Results
Across simulations and biomedical applications, ICON more accurately identifies genuinely used concepts while suppressing correlation-driven false positives. It also tracks shortcut reliance, supports sparse explanations, and enables independent validation of model behavior.
- Simulated data: ICON was most accurate in hidden simulated layers, with TVD 0.33 at HL−2 and 0.23 at HL−1.At HL−2, the best baseline reached 0.52; at HL−1, the next best reached 0.38.
- Simulated data: At HL−1, ICON misallocated only 0.06 TVD to zero-importance concepts, versus 0.16 for the best baseline.Its spurious importance increased only 1.2–1.3× as correlation rose from r = 0 to r = 0.6, compared with 7–17× for key baselines.
- Simulated data: ICON’s advantage persisted as concept sets expanded and sample sizes shrank, while baselines degraded sharply with more correlated distractors.With q growing from 5 to 100, ICON’s HL−1 error never exceeded 0.22; beyond n ≈500, its lead remained stable.
- Skin-cancer models: In ISIC experiments, ICON avoided false positives under rising concept–outcome and concept–concept collinearity, assigning less than 0.01 importance to absent or unused artifacts.The baselines instead tracked test-set correlations or varied unstably across test sets.
- Skin-cancer models: ICON’s microscope importance rose from approximately 0.00 to 0.44 as shortcut reliance increased from Mmicro-00 to Mmicro-90, whereas baselines changed only 0.02–0.15.The artifact-related variance remained in final layers only for the shortcut-affected model, while ICON also reported unexplained representation variance.
- Biomedical models: In brain-imaging models, ICON’s sex attribution was supported by retraining: sex balancing reduced performance to 53.3±0.5%, close to chance.For brain-age prediction, ICON attributed most final-layer variance to age, while probe-flagged concepts showed no robust OOD error change, with |∆MAE| ≤0.069 years.
2 Discussion
ICON reframes concept-based explainability as multivariate variance decomposition, addressing false positives from correlated concepts while making scores comparable and exposing unexplained representation variance. Its sparse attributions support follow-up validation, but its claims remain bounded by linear encoding, supplied-concept coverage, and unresolved interpretation of unexplained variance.
- Method: ICON combines PLS with Type I sum-of-squares allocation to decompose high-dimensional representations across multiple correlated concepts.The method adapts variance-partitioning principles to concept-based explainability.
- Method: By assigning shared variance to the strongest contributor, ICON suppresses correlation-driven false positives and leaves fewer concepts requiring interpretation.This multivariate allocation explicitly accounts for correlations among concepts and between concepts and the outcome.
- Practical benefits: Because each score is a share of layer variance and all shares sum to one, ICON supports comparisons across layers and across categorical and continuous concepts.This enables tracking whether a concept remains represented through the network to the prediction.
- Practical benefits: Sparse ICON attributions generate a small number of testable candidates for retraining, out-of-distribution validation, or confounder analysis.The discussion reports supporting evidence for selected neuroimaging concepts, while additional probe-ranked concepts showed no evidence of use.
- Limitations: The unexplained share warns when supplied concepts account for little of the representation, but it conflates missing concepts with nonlinear encoding and cannot identify what is missing.ICON scores only supplied concepts and retains a linear-encoding assumption.
- Limitations: ICON does not discover new concepts, and its observational decomposition cannot generally be read causally without further conditions.The authors identify causal interpretation as an unresolved broader question.
3 Methods
ICON treats concept-based explainability as multivariate variance decomposition, estimating each concept’s contribution to a DNN layer after controlling for other concepts and the outcome. It uses PLS to derive orthogonal latent concepts, allocates their explained variance among original variables, and retains an unexplained remainder.
- Motivation and formal setup: ICON estimates concept importance in a latent representation after controlling jointly for all supplied concepts and the outcome.This addresses omitted-variable bias from correlations among concepts and with the outcome.
- ICON construction: PLS finds linear associations between the high-dimensional layer representation and concept matrix, producing latent concepts and corresponding activation vectors.Each ICON pair maximizes attainable covariance, while the activation vector scores the latent concept across samples.
- ICON construction: Subsequent PLS iterations operate on residuals, yielding mutually orthogonal ICON activations that span a low-rank subspace of the layer representation.Pairs are added until a new pair contributes less than 1% of total squared cross-covariance, with the number selected adaptively per layer.
- Variance decomposition: ICON importance is defined as the variance explained by each latent concept, normalized by the layer representation’s variance.The derivation uses an OLS inner relation and assumes the latent concept is uncorrelated with its residual.
- Variance decomposition: The unexplained term represents representation variance unassigned to supplied concepts or the outcome under the linearity assumption.This remainder marks what the provided concept set and linear model do not account for.
- Concept-level attribution: Within each ICON, Type I sums of squares allocate explained variance among original concepts and the outcome in descending squared-weight order.Sequential allocation gives higher-ranked variables the shared variance contributed by weaker correlated variables, reducing false positives during shortcut screening.
1. Bounded and sum-constrained: Imp(ck) ∈[0, 1] and P
The method constrains concept importance to a bounded, sum-compatible scale, enabling direct comparisons across concept types and layers.
- 1. Bounded and sum-constrained: Imp(ck) ∈[0, 1] and P: ICON uses l = 1 so concept importance is directly comparable across concept types and layers.The supplied passage states the normalization choice and its comparison benefit.
- 1. Bounded and sum-constrained: Imp(ck) ∈[0, 1] and P: The normalization places importance values on a common scale across layers.This follows directly from the stated purpose of setting l = 1.
- 1. Bounded and sum-constrained: Imp(ck) ∈[0, 1] and P: The same normalization supports comparisons between different concept types.The passage explicitly links l = 1 to direct comparison across concept types.
2. Transparent about unexplained variance: σ2
ICON makes unexplained representation variance explicit and displays concept contributions as a variance partition, unlike normalized linear-probe scores.
- 2. Transparent about unexplained variance: σ2: ICON quantifies representation variance outside the supplied concept set and linear assumptions, guarding against overconfidence in incomplete explanations.The unexplained share identifies what the provided concepts do not capture.
- 2. Transparent about unexplained variance: σ2: ICON bars form a variance partition whose lengths sum to one at each interpreted layer, with unexplained variance shown separately.Bars are stacked by concept category across layers from input representations to prediction logits.
- 2. Transparent about unexplained variance: σ2: Linear-probe bars are normalized cross-validated R2 or Cohen’s κ scores, not a variance partition and without an unexplained term.Only variables exceeding 5% normalized importance are named, although all contribute to the bars.
1. Linear probes
The paper evaluates statistical alternatives for concept importance, including marginal, coefficient-based, partial, and Shapley R2 measures. It also standardizes scores across methods, while testing them in controlled synthetic settings and applied models.
- Variable-importance baselines: Marginal R2 importance attributes shared variance to each concept separately, inflating scores when concepts are correlated.
- Variable-importance baselines: Standardized regression coefficients account for other concepts, but can be numerically unstable under multicollinearity.
- Variable-importance baselines: Partial R2 measures each concept’s unique variance after accounting for all others, but can shrink near zero under strong multicollinearity.
- Variable-importance baselines: Shapley R2 distributes shared variance symmetrically by averaging each concept’s marginal contribution across entry orderings.
- Score normalization: All methods are normalized over the supplied variable set, placing their scores on a common simplex and reducing layer-dimensionality effects.
- Experimental setup: The experiments use causal ToyBrains scenarios, trained neural networks, held-out test data, and ground truth obtained through interventions on the causal graph.
- Evaluation caveat: CAV and probe comparisons are limited because those methods provide rankings rather than variance decompositions, making their absolute TVD an upper bound on agreement.
4 Extended Data (Supplementary)
The supplementary material documents concept-based explainability methods, controlled shortcut-learning experiments, implementation resources, and sensitivity analyses. It reports that ICON remains most accurate across layer widths and outperforms baselines as concept sets grow or samples shrink.
- Linear probes: Linear probes predict concepts from representations, but decodability alone does not establish that the model uses those concepts for its task.
- CAV methods: CAV methods use activation-space directions and output gradients, yet their scores can vary with examples, probing datasets, and adversarial input perturbations.
- Concept entanglement: Concept entanglement lets estimates absorb correlated concepts’ contributions, as illustrated by nearly coincident beard and necktie CAV directions in CelebA.
- ICON: ICON jointly evaluates concepts through multivariate variance decomposition and explicitly allocates shared variance instead of counting it twice.
- Resources: The paper provides dataset-access information, archived ToyBrains configurations, and public implementation and analysis code.
- Sensitivity analysis: Across widths, ICON has the lowest error at every tested width and remains most accurate at both interpreted layers.