PaperlayerView exact revision on arXiv

Source-linked AI summary

Unmasking Clever Hans Predictors and Assessing What Machines Really Learn

Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, Klaus-Robert Müller

arXiv:1902.10178v1cs.AIcs.CVcs.LGcs.NEstat.ML

TL;DR

The paper addresses whether high-performing learning machines actually use valid, generalizable strategies rather than artifacts or loopholes. It applies LRP explanations across computer-vision and Atari tasks and proposes SpRAy for semi-automated analysis. The results reveal a broad spectrum of behaviors, including strategic play and Clever Hans strategies that standard metrics can miss.

  • Problem

    High predictive performance does not show whether nonlinear machines use valid, generalizable decision strategies rather than spurious correlations.

  • Method

    The paper uses LRP to explain individual predictions and SpRAy to cluster relevance maps and identify prediction strategies across large datasets.

  • Results

    Across vision and Atari tasks, the models displayed behaviors ranging from strategic decision making to Clever Hans strategies exploiting dataset, game, and training artifacts, often unnoticed by common metrics.

  • Takeaways & Limitations

    Explaining decisions provides a practical basis for judging whether learned behavior is valid and useful beyond predictive performance alone.

  • Takeaways & Limitations

    Analyzing classifier behavior through individual heatmaps is laborious and costly without scalable automation.

Abstract

from arXiv · show

Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly "intelligent" behavior. Here we apply recent techniques for explaining decisions of state-of-the-art learning machines and analyze various tasks from computer vision and arcade games. This showcases a spectrum of problem-solving behaviors ranging from naive and short-sighted, to well-informed and strategic. We observe that standard performance evaluation metrics can be oblivious to distinguishing these diverse problem solving behaviors. Furthermore, we propose our semi-automated Spectral Relevance Analysis that provides a practically effective way of characterizing and validating the behavior of nonlinear learning machines. This helps to assess whether a learned model indeed delivers reliably for the problem that it was conceived for. Furthermore, our work intends to add a voice of caution to the ongoing excitement about machine intelligence and pledges to evaluate and judge some of these recent successes in a more nuanced manner.

1 Introduction

The paper argues that high predictive performance does not establish that nonlinear learning machines use valid, generalizable strategies. It motivates explaining individual decisions and introduces SpRAy to identify unexpected behaviors across large datasets.

  • Context: Machine-learning systems support products and services by converting large streams of data into potentially useful information.The paper situates its concerns within the growing everyday use of machine learning.
  • Motivation: Nonlinear machine-learning predictors are often black boxes, limiting scientific insight into what they have learned.This opacity has encouraged some fields to prefer linear models for interpretability.
  • Motivation: Predictive performance alone cannot distinguish valid problem solving from strategies based on spurious correlations.Such correlations may disappear after deployment, causing incorrect classifications and reduced usefulness.
  • Explanation: Individual-example explanations are especially important for nonlinear models because relevant features can vary across predictions.Traditional feature selection often summarizes relevance across the training ensemble or a class prototype.
  • Approach: The paper uses LRP explanations and extends them with semi-automated SpRAy to identify diverse, unexpected, or undesirable decision behaviors at dataset scale.The framework complements performance evaluation and can make large-scale behavior analysis practical.

2 Results

Explanation methods reveal that models with strong task performance can rely on spurious artifacts or loopholes, while other models develop strategic behavior. SpRAy extends this analysis from individual heatmaps to scalable whole-dataset investigation.

  • Valid and invalid problem-solving behaviors: LRP revealed that a Fisher-vector classifier used source tags rather than horses to classify some PASCAL VOC images, despite excellent test-set accuracy.The DNN instead focused on the horse and rider, while the Fisher-vector heatmap focused on a source tag in the image corner.
  • Valid and invalid problem-solving behaviors: Atari Pinball performance exploited a game loophole: the neural network nudged the table to activate scoring switches and ignored the flippers.This strategy is rational under the software’s scoring rules but would likely fail in a physical pinball machine that tilts after strong movements.
  • Valid and invalid problem-solving behaviors: Atari Breakout heatmaps showed a progression from ball control, to paddle targeting, to focus on the playing-field corners during learning.The changing relevance structure provides evidence of learned strategic behavior rather than a fixed feature reliance.
  • Whole-dataset analysis of classification behavior: Manual inspection of individual heatmaps is laborious and costly, motivating automated analysis for large datasets.SpRAy addresses this scaling problem through semi-automated clustering of relevance maps.
  • Whole-dataset analysis of classification behavior: SpRAy identified four horse-classification strategies: detecting the horse and rider, source tags in portrait images, horseback-riding context, and source tags in landscape images.It combines relevance maps, dimensionality reduction, spectral clustering, and visual inspection to separate prediction strategies.
  • Whole-dataset analysis of classification behavior: SpRAy uncovered an aeroplane strategy based on artificial image-border padding that manual inspection had missed, and changing padding strongly affected classifier output.The result shows that SpRAy can expose divergent input-feature reliance even when predictions appear similar.

3 Discussion

The discussion frames explanation as necessary for judging whether learned strategies are valid, highlighting a spectrum from strategic behavior to Clever Hans shortcuts. SpRAy provides a scalable way to systematize that assessment.

  • Discussion: The studied systems exhibited strategic, artifact-driven, and loophole-based behaviors that common evaluation metrics did not distinguish.Examples included dataset tags, game mechanics, image padding, and strategic play in Atari Breakout.
  • Discussion: SpRAy systematizes classifier-behavior analysis and scales semi-automatically to large datasets, revealing artifacts missed by manual heatmap inspection.The authors connect this capability to future assessment of trustworthiness, fairness, and accountability.
  • Discussion: Explaining decisions enables deeper judgment of learned representations than monitoring behavior and performance measures alone.The paper presents explanation as a way to assess whether machines embody particular strategic decision-making.

4 Methods

The paper uses LRP to decompose nonlinear model predictions into input-feature relevance scores, then applies SpRAy to cluster explanations and identify typical or atypical decision behaviors. These tools support inspection of individual decisions and broader validation of whether learned strategies are meaningful, generalizable, and reliable.

  • Layer-wise Relevance Propagation: LRP decomposes a model output into scores assigned to input variables, producing an explanation that can be visualized as a heatmap.The method applies a backward pass from the network output to input variables such as pixels.
  • Layer-wise Relevance Propagation: LRP has been designed for neural networks and other models, including kernel machines, Fisher vector models, Bag of Words classifiers, and LSTMs.The paper notes that rules differ in characteristics and domain of applicability.
  • Spectral Relevance Analysis: SpRAy applies spectral clustering to datasets of LRP explanations to identify typical and atypical decision behaviors efficiently.It addresses the time cost of screening many individual explanations when investigating classifier behavior on large datasets.
  • Motivation and Validation: Decision explanations complement task-level performance measures by revealing whether a learned strategy is valid and generalizable or relies on spurious correlations.The paper motivates interpretability as a way to assess behavior beyond predictive performance across scientific and game applications.
  • Layer-wise Relevance Propagation: LRP propagates relevance backward by redistributing each neuron’s relevance to lower-layer neurons according to their contributions.The relevance messages obey a local conservation property, and the procedure continues iteratively to the pixel layer.
  • Layer-wise Relevance Propagation: The propagation intuition is that neurons are relevant when they contribute to neurons that are themselves relevant.Different LRP rules can be selected according to model architecture and desired properties such as sparsity or negative evidence.

C.4 Experiments

The experiments analyze convolutional networks on Atari games and image classification, using decision explanations to compare learned behavior with apparent task performance.

  • The study examines convolutional networks in reinforcement learning for arcade games and supervised classification on Pascal VOC images.
  • The classification analysis contrasts a Fisher vector model with a pretrained deep neural network on Pascal VOC categories.
  • The Atari network uses three convolutional and two fully connected layers, producing action-wise Q-function predictions.

D Task I: Playing Atari Games

The Atari agents are trained with a deep Q-learning setup and explained with layer-wise relevance propagation to connect selected actions to game inputs.

  • The parameter update clips the Bellman-error difference to [−1, 1] to curb oscillations when the network is far from satisfying the Bellman equation.
  • The agent receives four grayscale 84 × 84 frames as a 4 × 84 × 84 tensor and selects the action with the highest predicted long-term reward.
  • The network has three convolutional layers and two inner product layers, with actions repeated for four time steps.
  • Training uses stochastic gradient descent, replay memory, a batch size of 32, and a learning rate of α = 2.5 · 10−4.
  • Layer-wise relevance propagation back-propagates the dominant action onto the input, producing a relevance tensor for visual interpretation.

D.1 Visualization and Comparison to Game Play

LRP visualizations reveal both strategic game behavior and brittle failure modes: agents exploit reward structures effectively but can stall in unfamiliar states.

  • Breakout: In Breakout, relevance shifts from tracking the ball to targeting a tunnel and then maintaining the ball above the brick wall for rapid score accumulation.
  • Breakout: The Breakout deadlock begins when the paddle becomes partly hidden; the model keeps predicting movement right despite minimal relevance to the paddle.
  • Video Pinball: In Video Pinball, the agent uses nudging and collisions to pass scoring triggers, secure an extra life, and then pursue additional rewards.
  • The agents maximize long-term reward within limited game states, producing strategies that follow game rules while exploiting weak points unintended by human designers.
  • LRP enables automated analysis of changing strategy by quantifying relevance assigned to game regions without manually watching long gameplay sequences.

D.3.1 Quantifying the Focus of the Deep Networks over Training

Relevance measurements track how Breakout agents refine their focus during training, moving from immediate object recognition toward tunnel-based reward strategies.

  • The analysis evaluates 200 model snapshots using 500 early- and 500 late-phase states sampled from a 2000-frame Breakout sequence.
  • Six training runs measure relevance for the ball, paddle, and tunnel using defined pixel regions and normalized relative-relevance scores.
  • Relative relevance is computed as object relevance divided by total frame relevance and scaled by frame area.
  • The ball becomes salient first, followed by the paddle, while tunnel regions come into focus after roughly 40–50 training epochs.
  • After tunnel construction, the ball ricochets between borders and high-valued blocks, reducing interaction while maximizing received reward.
  • Despite strategic tunnel building, agents trained for 200 epochs fail to restart a new ball because knowledge of pressing fire does not transfer to later game phases.

D.3.2 Varying Depth of Architecture

Comparing three network architectures shows that depth changes both the timing of attention to important game elements and the emergence of strategic tunnel building. The Small architecture recognizes the ball and paddle early but remains limited to sub-human gameplay.

  • The compared architectures differ in convolutional depth: the NIPS architecture has one fewer convolutional layer than the Nature architecture, while the Small architecture has the same number as Nature.
  • The NIPS architecture delays focus on the ball and paddle by approximately five training epochs relative to the Nature network.Its tunnel relevance shifts in two stages and ultimately exceeds the Nature architecture's level.
  • The Small architecture switches early to the ball and paddle but does not develop advanced tunnel-building strategies.Its agent follows the ball and successfully reflects it in three out of four interactions, while gameplay remains sub-human.
  • Replay memory capacity also affects the attention shift: capacities of 5 · 10^4, 10^6, and 5 · 10^6 all produce tunnel-directed attention, while the smallest memory yields a more gradual, earlier shift.
  • The Nature, NIPS, and Small architectures all shift attention toward tunnel building during training.

D.3.4 Changes of Model Attention during Game Play

The study tracks changing game-time attention and action decisiveness by aggregating relevance maps and comparing LRP with gradient-based Sensitivity Analysis. LRP reveals strategic and temporally localized behavior more clearly, whereas gradient maps blur or miss important strategy changes.

  • Changes of Model Attention during Game Play: Relevance maps are horizontally pooled and concatenated over time to represent where attention changes across a game sequence.The resulting representation retains vertical pixel location while using time as the horizontal axis.
  • Changes of Model Attention during Game Play: As training progresses, Q-value divergence increases at moments when actions strongly affect gameplay, and the trained network concentrates on a single action during ball interactions.The comparison uses 500-frame sequences at two, twenty, and two hundred epochs, with 100.000 training steps per epoch.
  • Changes of Model Attention during Game Play: LRP heatmaps track the ball without a dedicated tracker and identify the ball height at which action decisions are made.
  • Comparing Relevance Maps to Gradient Map: Sensitivity Analysis follows the ball's general area rather than the ball itself, assigns no tunnel-pixel weight, and does not reveal strategy changes.Its gradient peaks indicate where input changes would most steeply alter predicted Q-values, not necessarily where the ball is located.
  • Comparing Relevance Maps to Gradient Map: Gradient-map visualizations preserve the ball's vertical movement but are substantially more blurred and less local than LRP heatmaps.
  • The broader evaluation motivates analyzing classifier behavior beyond test-set error because finite datasets can leave models reliant on unusual backgrounds or artifacts.

E.3.1 Detecting Boats & Horses

LRP comparisons show that similar prediction accuracy can arise from different decision strategies: DNNs generally use object information, while FV models more often rely on contextual cues and dataset artifacts. These explanations expose failures that benchmark accuracy can miss.

  • Model comparison: DNN predictions are dominated by objects, whereas FV predictions often use background information, co-occurring objects, or other contextual elements.
  • Boats: For boats, the FV model relies on water while the DNN uses the boat silhouette and still finds positive evidence for a shipwreck on land.The FV model instead finds strongly contradictory evidence and predicts “aeroplane” for the shipwreck.
  • Horses: For horses, the FV model uses a copyright watermark as a strong cue, and removing it produces a false negative while leaving the remainder of its relevance pattern unchanged.Similar watermarks occurred in approximately 20% of horse images in the benchmark.
  • Context importance: The context-importance analysis uses positive relevance inside and outside object bounding boxes, with larger outside-to-inside ratios indicating stronger background reliance.
  • Model comparison: The DNN and FV models can both predict classes accurately while differing in whether they have learned object concepts or recurring contextual biases.The reported comparison specifically identifies “aeroplane” and especially “boat” as well-predicted classes affected by contextual bias.
  • Context importance: Interior-object classes such as “chair”, “diningtable”, and “sofa” remain outliers for the DNN because frequent co-occurrence makes the categories difficult to distinguish.

F Semi-automated Analysis of Classifier

The paper introduces SpRAy, a semi-automated analysis that uses relevance maps and spectral analysis to examine prediction strategies across whole datasets. It bridges dataset-level performance evaluation and costly manual inspection of individual explanations.

  • SpRAy combines relevance-map computation with spectral analysis to analyze classifier behavior semi-automatically.
  • Manual assessment of individual LRP heatmaps provides decision-process information but requires human attention for every data point and does not scale to whole datasets.
  • SpRAy is presented as the first method, to the authors' knowledge, to investigate classifier prediction strategies over a whole dataset in a semi-automated manner.
  • The method detects irregularly frequent reoccurrences of non-obvious, highly similar image features that may reflect meaningful class cues or co-occurring learned features.

F.1 Spectral Relevance Analysis

Spectral Relevance Analysis (SpRAy) combines relevance maps, spectral clustering, eigengap analysis, and optional t-SNE visualization to expose structured prediction behavior. Its graph-based analysis identifies clusters and characteristic features that may remain hidden in model-output comparisons.

  • Pipeline: SpRAy analyzes relevance maps through preprocessing, spectral clustering, eigengap-based cluster selection, and optional t-SNE visualization.The pipeline computes relevance maps, standardizes their shapes, clusters them, identifies interesting clusters, and visualizes them in lower dimensions.
  • Purpose: SpRAy can reveal features associated with target-class predictions even when those features are not obvious in pixel space.The analysis is designed to identify structured correlations in relevance maps that standard inspection may miss.
  • Spectral clustering: Spectral clustering builds a weighted affinity matrix W over samples, symmetrizes neighborhood relations, and derives clusters from the graph Laplacian eigenvectors.The affinity matrix can use k-nearest-neighbor relationships; symmetrization produces an undirected graph.
  • Eigenvalue analysis: The number of eigenvalues near zero and the eigengap indicate how many weakly or strongly separated sample groups are present.Exactly zero eigenvalues correspond to disjoint clusters, while eigenvalues close to zero and subsequent eigengaps reveal connected groups in non-synthetic data.
  • Visualization: t-SNE embeds samples using pairwise distances so groups of input images or relevance maps can be inspected for model-specific patterns.The embedding supports human interpretation of cluster structure identified by spectral analysis.

F.2 Uncovering Prediction Strategies of the Pascal VOC Classifiers

SpRAy exposes distinct prediction strategies in Pascal VOC classifiers, including reliance on water, hurdles, source tags, pose, and other contextual or artifact features. These strategies can differ substantially even when aggregate prediction performance appears comparable.

  • Horse strategies: SpRAy found that the FV horse classifier relied on a copyright watermark appearing in approximately one fifth of horse images.The watermark became a stronger indicator of horseness for the FV model than the pictured horses themselves.
  • Horse strategies: The FV model also formed clusters associated with hurdles, while DNN relevance maps attributed horse predictions to horses, riders, and pose rather than the hurdles.The hurdle correlation was present in the images, but the DNN relevance maps did not assign relevance to the hurdles themselves.
  • Evaluation: SpRAy identified model-relevant features that standard AP comparisons could not discover when the models had comparable performance.Feature-based recall analysis further showed that the copyright tag strongly affected the FV model but largely did not affect the DNN model.
  • Other classes: For boat images, the FV model relied on water around and below boats, whereas the DNN used boat shapes and related structures such as sails.The FV behavior was related to spatial pyramid mapping, which can facilitate spatial biases.
  • Aeroplane strategies: Aeroplane analyses revealed separated image and relevance-map clusters, including flight images against blue sky and DNN reactions to preprocessing-related border information.The DNN’s weak context dependence suggested that the border artifact occurred consistently without dominating correct airplane recognition.

F.2.1 Verifying the Detected Bias in Prediction behavior

Controlled preprocessing tests verified that the aeroplane DNN responds to uniform, sky-like image borders introduced during image preparation. Adding structured or high-frequency border content reduces this artifact, while constant-color padding can increase aeroplane scores even for non-aeroplane images.

  • Verification: Padding with non-border-copying strategies decreased aeroplane prediction scores, with random noise producing the largest decrease.High-frequency random pixels contradicted the model’s learned expectation of uniformly colored borders and removed the positive border contribution.
  • Verification: Uniform sky-blue or random-color padding reduced the score decrease relative to random noise, confirming sensitivity to low-frequency border structure.The model reacts more favorably to constant-color extensions than to structure-rich or noisy extensions.
  • Non-aeroplane controls: For non-aeroplane images, mirror padding, cropping, and random noise reduced airplaneness, whereas fitting constant-color padding comparatively increased it.Introducing structure into the border regions reduced the airplane prediction further.
  • Relevance localization: The DNN assigned strongest positive relevance to the top and bottom image borders, consistent with uniform sky-like areas supporting airplane predictions.The left and right borders received weaker relevance attribution.
  • Orientation: Constant border padding increased airplaneness more for vertically padded landscape images than for horizontally padded portrait images.The comparison covered approximately 3800 landscape-format and 900 portrait-format non-aeroplane images.
  • Implication: The authors warn that sky-like border reliance could produce false-positive aeroplane predictions outside laboratory settings.They identify preprocessing and training-data choices as possible targets for future model adaptation.

F.3 Spectral Relevance Analysis on Atari Gameplay Sequences

SpRAy separates temporal gameplay phases from relevance maps of Atari agents playing Breakout and Video Pinball. Relevance-map clusters are fewer and more distinctive than clusters in the raw input frames, making strategic phases easier to identify.

  • Spectral structure: SpRAy identified two to three main relevance-map clusters corresponding to distinct gameplay phases in Breakout and Video Pinball.Breakout showed two near-zero eigenvalues and two large eigengaps, while Video Pinball showed three near-zero eigenvalues followed by a distinct eigengap.
  • Inputs versus relevance: Input-frame clusters were more numerous and mainly reflected temporal changes in game scenes, whereas relevance-map clusters isolated the agent’s strategic phases.In Breakout, changing brick layouts, ball motion, and paddle position produced many dense input-frame groups.
  • Video Pinball: Video Pinball relevance clusters corresponded to early positioning near the top-right element and later repeated movement near the top-left element.The early phase occupied approximately 30% of recorded time, and later clusters reflected high versus low ball positions.
  • Breakout: After a mistake in Breakout, the agent returned to an earlier strategy to regain control of the ball.The re-entry occurred at t ≥880 after the ball ricocheted away from the top-left game element.
  • Conclusion: The Atari experiments demonstrate that relevance maps can isolate important temporal phases that are less clearly recognizable from the model’s input data alone.This supports using SpRAy for temporal behavior analysis in gameplay sequences.

F.4 Concluding Remarks on Spectral Relevance Analysis

SpRAy combines relevance-map analysis with spectral clustering to characterize nonlinear models beyond one-dimensional test-set metrics. Across image-classification and Atari gameplay analyses, it exposed model strategies, biases, preprocessing issues, and temporally distinct gameplay phases.

  • SpRAy uses LRP relevance maps as features for spectral analysis of a trained model’s prediction strategies.The approach focuses on meaningful sets of explanation heatmaps rather than isolated predictions.
  • In image classification, the pipeline identified systematic FV biases for “boat”, “horse” and “aeroplane” and an input-preprocessing issue in the DNN model.The preprocessing issue could be rectified in the model’s next development iteration.
  • For Video Pinball, relevance-map clustering revealed three main clusters whose assignments correlated with temporal game progression.The relevance maps made larger time-interval groups clearer than analysis of the input images.
  • SpRAy isolated important gameplay phases in Breakout and Video Pinball from relevance data, including phases that were less transparent in the recorded game frames.These games provide data with a strong temporal aspect for analyzing changing prediction strategies.
  • SpRAy’s future applicability depends partly on explanation and preprocessing choices, including alternatives to LRP and simple sum pooling.Different preprocessing choices can direct analysis toward specific heatmap aspects.
Loading 1902.10178v1…