PaperlayerView exact revision on arXiv

Source-linked AI summary

Concept Whitening for Interpretable Image Recognition

Zhi Chen, Yijie Bei, Cynthia Rudin

arXiv:2002.01650v5cs.LGcs.AIcs.CVstat.ML

TL;DR

Neural networks’ hidden units are often not semantically understandable, and posthoc analyses may not satisfactorily reveal the concepts they represent. Concept whitening aligns latent-space axes with predefined concepts, providing clearer layerwise interpretations while maintaining accuracy on par with corresponding black-box models.

  • Problem

    Hidden-layer units are often not semantically understandable, while posthoc analyses may not provide satisfactory answers about the concepts they represent.

  • Method

    Concept whitening inserts a module that whitens latent representations and rotates their axes to align with predefined concepts.

  • Results

    Concept whitening makes layerwise concept learning clearer and achieves accuracy on par with corresponding original black-box networks on large datasets.

  • Takeaways & Limitations

    Concept whitening enables inherently interpretable CNNs that reveal how target concepts are learned across layers without harming the main objective’s performance.

  • Takeaways & Limitations

    Standard neural-network latent spaces may fail to separate concepts, even when mean-centered and standardized.

Abstract

from arXiv · show

What does a neural network encode about a concept as we traverse through the layers? Interpretability in machine learning is undoubtedly important, but the calculations of neural networks are very challenging to understand. Attempts to see inside their hidden layers can either be misleading, unusable, or rely on the latent space to possess properties that it may not have. In this work, rather than attempting to analyze a neural network posthoc, we introduce a mechanism, called concept whitening (CW), to alter a given layer of the network to allow us to better understand the computation leading up to that layer. When a concept whitening module is added to a CNN, the axes of the latent space are aligned with known concepts of interest. By experiment, we show that CW can provide us a much clearer understanding for how the network gradually learns concepts over layers. CW is an alternative to a batch normalization layer in that it normalizes, and also decorrelates (whitens) the latent space. CW can be used in any layer of the network without hurting predictive performance.

1. Introduction

Posthoc analyses may fail to provide semantically complete or assumption-free explanations of neural-network representations. Concept whitening addresses these limitations by aligning latent-space axes with target concepts, enabling visualization of how concepts are represented and refined across layers.

  • Hidden-layer units are generally not semantically understandable, making neural-network computations difficult to explain.
  • Posthoc node analyses may identify semantically aligned units without capturing the network’s full information about a concept, which may be scattered across nodes.
  • Concept-vector methods remain dependent on assumptions about the latent space’s geometry and concept organization.
  • Concept whitening inserts a module that constrains latent representations to align with target concepts and provides a straightforward way to extract them.CW aligns concepts along latent-space axes rather than forcing them to be learned as an intermediate step.
  • By examining concept-aligned images at successive layers, CW reveals increasingly sophisticated and complex representations of concepts.Experiments show that lower layers can represent complex concepts through lower-level abstractions, such as airplane features involving white or gray objects on blue backgrounds.
  • CW combines whitening with a rotation matrix that aligns concepts to axes, replacing a plain batch-normalization step while normalizing and decorrelating the latent space.The rotation is the key mechanism connecting whitening to interpretability through disentangled concepts.

2. Related work

Concept Whitening (CW) belongs to inherently interpretable approaches that shape latent representations during training, addressing limitations of post-hoc explanations and diffuse concept encoding. It also connects interpretability with whitening, distinguishing its decorrelation from batch normalization’s standardization-only operation.

  • Interpretability and explainability: CW is an inherently interpretable approach that enlightens network behavior without providing a full explanation of every computation.The paper contrasts inherently interpretable models with post-hoc explanations and places CW in the first category.
  • Interpretability and explainability: Saliency methods assign pixel-importance weights for predicted classes but often highlight image edges regardless of class.These methods are described as the most common form of post-hoc neural-network explanation.
  • Interpretability and explainability: Individual filters can align with predefined concepts, but concepts may instead be represented diffusely across many impure units.The network was not trained to express concepts purely through individual nodes.
  • Interpretability and explainability: CW shapes the latent space through training, avoiding diffuse concept representations while providing useful concept-aligned directions.This places CW closer to inherently interpretable neural networks while retaining the spirit of concept-vector methods.
  • Whitening and orthogonality: Whitening transforms a covariance matrix to the identity, whereas batch normalization retains standardization but not decorrelation.The paper presents whitening as a classical preprocessing step and batch normalization as a widely used deep-learning operation.

3. Methodology

Concept Whitening (CW) whitens latent representations and applies an orthogonal transformation to align predefined concepts with latent-space axes. It is trained within neural-network layers using alternating objectives, with adaptations for convolutional outputs and limited computational overhead.

  • Concept Whitening Module: CW first standardizes and decorrelates latent representations, then learns an orthogonal rotation aligning concept c_j with the jth latent axis.The whitening transformation removes correlations, while the rotation is optimized under an orthogonality constraint.
  • Concept Whitening Module: The rotation matrix is optimized using gradient-based methods on the Stiefel manifold while preserving orthogonality.The whitening matrix is non-unique and rotation-free, allowing a separate orthogonal transformation for concept alignment.
  • Training Procedure: CW can be inserted at any neural-network layer and trained as part of the network’s overall optimization procedure.The module’s forward pass produces predictions after whitening and concept alignment.
  • Training Procedure: Training alternates between mini-batches from the main dataset and auxiliary concept data, optimizing the main objective and concept-alignment objective in turn.The main objective is usually related to classification accuracy.
  • Convolutional Outputs: For CNN feature maps, concept activation is reduced from each h×w map to a scalar using statistics such as its mean or maximum.After training, a feature map represents whether a meaningful concept is detected at each image location.
  • Computational Efficiency: Concept-alignment optimization runs only every 20 batches, usually costing fewer than 20 matrix multiplications and 10 matrix inversions without notably slowing training.The method includes separate iterative steps for whitening normalization and concept alignment.

4. Experiments

Experiments show that concept whitening preserves predictive performance while aligning latent axes with increasingly semantic concepts across layers. CW also improves concept separability and purity, enabling concept-importance analysis in Places365 and ISIC.

  • Performance: CW models remain within 1% of the original pretrained model on both top-1 and top-5 accuracy after one additional training epoch.Accuracy also changes little across CW layer placements and numbers of trained concepts.
  • Layerwise representations: Lower-layer CW axes capture colors and textures, whereas higher-layer axes capture semantically coherent object concepts.For example, lower-layer “airplane” activations often show blue backgrounds with white or gray central objects.
  • Layerwise representations: Traversing layers changes an image’s concept rankings from low-level characteristics toward higher-level meanings, as illustrated by a sunset becoming more associated with “airplane” at higher layers.The sunset’s warm colors initially increase its “bedroom” rank, while higher layers decrease its “bed” rank.
  • Concept geometry: CW achieves greater concept separability, with an average inter-concept/intra-concept similarity ratio of 0.35 versus 0.94 for standard CNNs.Without normalization, CW’s average inter-concept similarity is 0.05 versus around 0.74 for standard neural networks; whitening is therefore crucial for pure concepts.
  • Concept geometry: Concept-distinction auxiliary loss does not naturally ensure concept separation: its average similarity ratio is 0.85, while whitening concentrates each concept’s signal on its own axis.In standard CNNs, concepts can instead be distributed throughout the latent space.
  • Concept importance: Concept-importance scores are larger for concepts meaningfully related to target classes, and analogous CW experiments on ISIC provide practical insights for skin-lesion diagnosis.The ISIC conclusions are similar to Places365 for accuracy, latent separability, concept purity, and axis correlation.

5. Conclusion and Future Work

Concept whitening disentangles CNN bottleneck representations and aligns their axes with predefined concepts, enabling inherently interpretable analysis of concept learning without harming performance. Future work includes automatically detecting new primitive concepts and relaxing complete filter-output decorrelation when concepts are correlated.

  • Conclusion: Concept whitening disentangles the CNN bottleneck latent space and aligns its axes with predefined concepts.It is placed at the bottleneck of a CNN.
  • Conclusion: Concept whitening enables an inherently interpretable CNN that reveals how the network gradually learns target concepts across layers without harming main-objective performance.It can also indicate whether the network needs those concepts.
  • Future Work: Future work could automatically detect and quantify new primitive abstract concepts that CW helps humans define.The passage relates this direction to Ghorbani et al. (2019).
  • Future Work: Requiring CW to completely decorrelate all filter outputs may be too strong when task-relevant concepts are highly correlated.The passage identifies this as a limitation for some tasks.

Supplementary Information … A.2. Top-10 Activated Images Based on Different Calculations

The supplement defines several ways to reduce a feature map to a scalar concept-activation value and compares their top-10 activated images. The comparison shows that calculation choice can change which images and lower-level characteristics the network appears to discover.

  • A.1. Calculations of Concept Activation Based on Feature Maps: A feature map from one filter has spatial dimensions h × w, but concept activation requires reducing it to a scalar for optimization and evaluation.Multiple reasonable reductions can be applied to the same feature map.
  • A.1. Calculations of Concept Activation Based on Feature Maps: The study evaluates four activation calculations: mean values, maximum value, mean positive values, and mean values after max-pooling downsampling.These are the four methods demonstrated in the supplementary comparison.
  • A.1. Calculations of Concept Activation Based on Feature Maps: Mean activation is more suitable for low-level concepts because they are distributed throughout the feature map.The method aggregates responses across the spatial extent of the map.
  • A.1. Calculations of Concept Activation Based on Feature Maps: Maximum and mean-positive activation are more powerful for high-level concepts, including objects concentrated in a single image location.Objects typically do not recur throughout the image, making localized strong responses informative.
  • A.2. Top-10 Activated Images Based on Different Calculations: Supplementary Figure 10 compares the top-10 activated images produced by the four activation calculations under the same CNN, dataset, and CW-module depth.Keeping these settings fixed isolates the effect of the activation calculation.
  • A.2. Top-10 Activated Images Based on Different Calculations: Different activation calculations can produce different most-activated images and lead the network to discover completely different lower-level characteristics.The comparison therefore affects the interpretation of what a concept represents.

A.3. Concept AUC Based on Different Activation Calculations

The study compares four concept-activation calculations using CW at the 2nd and 16th ResNet18 layers. Max-pool-mean performs well across both layers and all concepts, whereas max and mean each perform poorly in particular layers.

  • CW is evaluated with four feature-map-based concept-activation calculations at the 2nd and 16th ResNet18 layers.The dataset, CNN architecture, and concept AUC definition match those used in the main paper.
  • In the 2nd layer, the feature map’s max value has lower AUC than other methods for two of the three concepts.
  • In the 16th layer, the mean performs poorly compared with the other concept-activation methods.
  • Max-pool-mean performs well across both layers and all concepts, combining the advantages of mean and max for low- and high-level concepts.

B. Sensitivity Analysis of Main Objective Accuracy · B.1. Main Objective Accuracy when CW is Applied to Different Layers

The study evaluates how applying concept whitening at different layers affects main-objective accuracy across CNN architectures. Across Tables 3–6, layer choice does not substantially impact accuracy.

  • B. Sensitivity Analysis of Main Objective Accuracy: CW is evaluated at different network layers using main-objective test accuracy.The evaluation follows the main paper’s objective and measures accuracy layer by layer.
  • B.1. Main Objective Accuracy when CW is Applied to Different Layers: Tables 3 through 6 report layer-wise test accuracy for different CNN architectures.These tables provide the results used to compare CW placement across layers.
  • B.1. Main Objective Accuracy when CW is Applied to Different Layers: The experiments use the same dataset as in the main paper.Dataset consistency keeps the sensitivity analysis aligned with the original evaluation setting.
  • B.1. Main Objective Accuracy when CW is Applied to Different Layers: The experiments use the same CNN architectures as in the main paper.The analysis therefore examines CW placement without changing the evaluated architecture set.
  • B.1. Main Objective Accuracy when CW is Applied to Different Layers: Accuracy remains stable across the different layers where CW is applied.The reported layer-wise results do not show a substantial accuracy change attributable to CW placement.
  • B.1. Main Objective Accuracy when CW is Applied to Different Layers: Applying CW at any tested layer does not substantially impact main-objective accuracy.This is the overall conclusion drawn from Tables 3 through 6 across the evaluated CNN architectures.

B.2. Main Objective Accuracy versus Number of Concepts Trained in CW · C. Correlation Matrix when CW is Applied to Different Layers

On Places365 with ResNet18, concept whitening maintains top-1 and top-5 accuracy as the number of trained concepts increases from 3 to 9. Correlation heatmaps assess latent-axis decorrelation for CW and other concept-based methods across multiple ResNet-18 layers.

  • B.2. Main Objective Accuracy versus Number of Concepts Trained in CW: On Places365, the study measures main objective accuracy while varying the number of concepts trained within the CW module.The evaluated CNN architecture is ResNet18.
  • B.2. Main Objective Accuracy versus Number of Concepts Trained in CW: The results average each concept count over three groups of randomly selected concepts.
  • B.2. Main Objective Accuracy versus Number of Concepts Trained in CW: Results are additionally averaged over different layers to which CW is applied for each group of simultaneous concepts.
  • B.2. Main Objective Accuracy versus Number of Concepts Trained in CW: Both top-1 and top-5 accuracy are not significantly affected by the number of concepts.
  • B.2. Main Objective Accuracy versus Number of Concepts Trained in CW: The drop of accuracy is less than 0.5% when the number of concepts increases from 3 to 9.
  • C. Correlation Matrix when CW is Applied to Different Layers: The study calculates latent-space axis correlations to quantitatively compare CW with other concept-based methods.
  • C. Correlation Matrix when CW is Applied to Different Layers: Supplementary Figure 12 presents absolute correlation coefficient matrices as heatmaps for methods applied at the 2nd, 4th, 6th, 8th, 12th, 14th and 16th ResNet-18 layers.
  • C. Correlation Matrix when CW is Applied to Different Layers: The correlation coefficient is calculated on the test set, with darker off-diagonal elements indicating more decorrelated latent axes.

D. Results on More Concepts … F. Case Study: Skin Lesion Diagnosis

Across object and general-characteristic concepts, concept whitening captures both low-level and high-level information, with generally purer concepts than comparison methods. Additional visualizations extend this analysis to local receptive fields and a skin-lesion diagnosis case study.

  • D. Results on More Concepts: Experiments on more concepts cover both object concepts and general characteristics of objects and scenes.The general-characteristics concepts are defined using attributes from the SUN Attribute Database.
  • D.1. Object Concepts: The object-concept experiments train CW simultaneously on seven randomly selected concepts from an 80-concept MS COCO bank.The bank is constructed by cropping objects from bounding boxes.
  • D.1.1. TOP-10 ACTIVATED IMAGES: CW captures concept-related color and texture at the 2nd layer and high-level meaning at the 16th layer for object concepts.Supplementary Figure 13 shows top-10 activated images along seven concept axes at lower and higher layers.
  • D.1.2. AUC CONCEPT PURITY: CW concepts are generally purer than concepts learned by TCAV, IBD, and standard-CNN filters across the evaluated layers.Supplementary Figure 14 compares AUC concept purity for 14 concepts across eight layers and reports frequent dominance by CW.
  • D.2. General Characteristics Concepts: CW also decently captures high-level meaning and low-level aspects for weather and object-material concepts from the SUN Attribute Database.The evaluated weather concepts are “cold,” “moist/damp,” and “warm,” while the material concepts include “metal” and “rubber/plast…”.
  • E. Top Activated Images Visualized with Empirical Receptive Fields: Empirical receptive-field visualizations identify image locations whose blacking out most reduces activation along each CW output concept axis.The visualization uses 32 × 32 random covering patches with a stride of 5.

F.1. Test Accuracy … F.4. Correlation of Axes

Across four evaluations, concept whitening slightly improves balanced test accuracy, enhances concept separability, produces purer learned concepts for “age < 20,” and successfully decorrelates latent axes.

  • F.1. Test Accuracy: 72.26% test balanced accuracy is achieved by ResNet18 with CW, compared with 71.65% for standard ResNet18.The CW result averages performance across different layers where CW was applied.
  • F.2. Separability of Latent Representation: Adding CW significantly improves concept separability in the latent representation compared with a standard ResNet.The comparison is reported for latent spaces with and without the CW module.
  • F.3. AUC Concept Purity: The concept “age < 20” is purer with CW than with concept-based posthoc methods.Purity methods are approximately tied for the concept “size ≥10 mm.”
  • F.4. Correlation of Axes: Standard neural networks show very strong correlations among latent axes, near 1 in many cases, at ResNet18’s 16th layer.The passage suggests these correlations may negatively influence concept separation and stochastic gradient descent.
  • F.4. Correlation of Axes: CW successfully decorrelates the latent space at the 16th layer of ResNet18.This contrasts with the strong axis correlations observed without CW.

F.5. Concept Importance

On ISIC, concept importance is measured against the binary classification loss, focusing on CW applied at layer 16 because concepts are purer there. The analysis identifies axis 76 as the most important untrained axis and suggests it may capture lesion-border irregularity.

  • Concept importance on ISIC is measured by each concept’s contribution to the binary classification problem using balanced binary cross entropy loss.
  • For CW at layer 16, the analysis compares the selected concept axes with the maximum and mean importance across all 512 latent axes.Layer 16 was chosen because concepts are purer there.
  • Axis 76 is the most important axis despite not being trained to represent a concept.The authors examine its top-10 activated images and empirical receptive fields for possible future concepts.
  • Axis 76 appears to focus more consistently on lesion borders, suggesting irregular border as a possible future concept axis.
Loading 2002.01650v5…