Unmasking Clever Hans Predictors and Assessing What Machines Really Learn

Problem

• Nonlinear predictors are often treated as black boxes, leading scientific fields to prefer interpretable linear models even when this sacrifices predictive performance.Their opacity prevents researchers from extracting deeper insights into what the systems have learned.Source: §1 · p. 2

Motivation

• Recent successes in Atari [6], Go, and Texas hold’em poker motivate evaluating machine behavior beyond predictive performance measures such as test-set error.The authors argue that this need applies across application domains, including science and games.Source: §1 · p. 2
• Predictive metrics must be complemented by analysis of decision-making processes to reveal whether models rely on valid, generalizable strategies or spurious correlations—the Clever Hans phenomenon.Such artifact-based strategies may fail when deployed in settings where the correlations are absent.Source: §1 · pp. 2–3

Related work

• Individual-example explanations are especially important for nonlinear models because relevant features can vary across predictions, whereas traditional feature selection summarizes an entire dataset or class prototype [13] [14] [15] [16].Most existing explanations assign importance scores to input features or pixels and visualize them as heatmaps [13] [17] [19] [21].Source: Figure 1 · p. 2, §1 · p. 3

Contribution

• SpRAy extends decision explanations into a semi-automated analysis that identifies diverse learned behaviors and detects unexpected or undesirable strategies across large datasets.Built on heatmaps, it provides an end-to-end way to evaluate models beyond test-set accuracy or reinforcement-learning reward.Source: §1 · p. 3
• The paper uses LRP to assess machine behavior qualitatively and quantitatively, presenting decision explanations as an efficient complement or practical alternative to broader tests of machine intelligence [29].The approach is intended to assess classification behavior from relatively few examples.Source: §1 · p. 3

2.1 Identifying valid and invalid problem-solving behaviors

• The Fisher-vector classifier achieves strong PASCAL VOC accuracy but classifies horses using source tags rather than horses, a Clever Hans strategy confirmed by tag removal and insertion tests.The DNN focuses on the horse and rider, whereas the FV prediction weakens when tags are removed and changes from car to horse when tags are added.Source: §2.1 · p. 3, Figure 2 · p. 4
• The Atari Pinball agent attains high scores by moving the ball near high-scoring switches and repeatedly nudging the table, while ignoring the flippers.This exploits the software’s nudging threshold without triggering tilt, although comparable movements would likely cause a real pinball machine to tilt.Source: §2.1 · pp. 4–5, Figure 2 · p. 4
• High test accuracy or game scores can conceal cheating, invalid real-world performance, or socially unacceptable behavior; LRP explanations reveal this distinction.The FV classifier may fail on real-world horse images, while the Pinball agent may fail after the game prevents excessive nudging.Source: §2.1 · pp. 5–6
• In Atari Breakout, relevance patterns progress from ball control to paddle targeting and finally corner-focused tunnel digging, indicating the emergence of efficient strategic behavior.The final corner strategy is also used by human players.Source: Figure 2 · p. 4, §2.1 · p. 6

2.2 Whole-dataset analysis of classification behavior

• SpRAy semi-automatically identifies four horse-classification strategies—horse and rider, source tags in portrait or landscape images, and horseback-riding context—without human inspection.This addresses the laborious, poorly scalable requirement for experts to analyze individual heatmaps.Source: Figure 3 · p. 5, §2.2 · p. 6, §2.1 · p. 6
• SpRAy reveals that the fine-tuned DNN classifies aeroplanes partly through artificial blue-border padding, and changing the padding procedure strongly affects its outputs.Unlike decision-similarity methods, SpRAy grounds divergent prediction strategies in input features even when predictions appear identical.Source: §2.2 · pp. 6–7

Discussion

• Explaining decisions revealed behaviors ranging from strategic Atari Breakout play to dataset, game, and training artifacts that common evaluation metrics failed to detect.The examples include exploiting horse-image tags, nudging in Atari Pinball, and image padding.Source: §3 · p. 7
• SpRAy systematizes classifier-behavior analysis at scale and exposed a padding artifact that manual inspection of heatmaps for all 20 PASCAL VOC classes missed.It also identified misuse of source tags in horse images, demonstrating the value of automated large-scale analysis.Source: §3 · p. 7
• The proposed analysis is framed as an initial step toward assessing AI trustworthiness, fairness, and accountability, including in regulatory contexts such as the GDPR [46].The work also contributes to broader discussion about whether machines are truly “intelligent”.Source: §3 · p. 7
• The authors argue that decision explanations enable judgments about whether machines embody particular strategies, beyond merely monitoring behavior or applying performance measures.These insights may support better learning machines and applications of machine learning in the sciences.Source: §3 · p. 7

4.1 Layer-wise relevance propagation

• LRP decomposes a nonlinear model’s prediction into input-feature relevance scores, providing explanations for neural networks and other machine-learning models.The method has been applied to neural networks, kernel machines, Bag of Words classifiers, Fisher vector models, and LSTMs.Source: §4.1 · pp. 7–8
• LRP propagates relevance backward from the output to input variables, where the resulting scores can be visualized as heatmaps matching the input dimensions.Its propagation rule uses neuron contributions and is applied uniformly through successive network layers.Source: §4.1 · p. 8

4.2 Spectral relevance analysis

• SpRAy clusters large datasets of LRP heatmaps to identify typical, atypical, and potentially Clever Hans decision strategies efficiently.It analyzes relevance patterns rather than raw images, helping distinguish meaningful object features from unintended co-occurring features.Source: §4.2 · p. 8
• SpRAy computes relevance maps, standardizes their size, clusters them spectrally, and uses eigengap analysis to identify separable behavioral groups for inspection.t-SNE is optionally used to visualize the resulting structure.Source: §4.2 · p. 8

C Understanding AI Systems

• Interpretability exposes decision bases that test-set performance may miss, including spurious strategies such as a horse classifier relying on copyright watermarks.The example is characterized as the Clever Hans phenomenon, because the system solves the dataset task without understanding the horse concept.Source: C Understanding AI Systems · pp. 12–13

C.2 Pixel-Wise Decompositions

• Pixel-wise relevance decompositions conserve the model output and can be aggregated over image regions, color channels, or height to monitor attention over time.This aggregation enables height-aggregated Atari heatmaps and single relevance scores per pixel.Source: §C.2 · p. 13
• Pixel-wise decompositions comprise structure, analytic, and propagation approaches, with propagation methods tracing output scores through directed computational graphs under local conservation constraints.Structure methods identify summands, analytic methods use local linear expansions, and propagation methods propagate relevance toward the pixels.Source: §C.2 · p. 14

C.3 Layer-Wise Relevance Propagation

• LRP redistributes each neuron’s received relevance to incoming neurons using contribution-dependent messages, then iteratively pools relevance downward until pixel-level scores are obtained.The method applies broadly to image-classification and neural-reinforcement-learning architectures, as well as Fisher-vector SVMs and LSTMs; its rules include αβ-, ϵ-, w2-, and flat-rules, with specialized handling for max-pooling and local renormalization layers.Source: §C.3 · pp. 14–15, Table 1 · p. 15

• LRP visualizations for Breakout and Video Pinball exposed which game objects Atari CNNs used, linking their focus to game situations and learned strategies.Tracking object relevance throughout training quantified behavioral changes that game scores did not reveal.Source: §C.4 · p. 15
• A Fisher vector classifier for Pascal VOC horse detection relied on a persistent copyright watermark, while removing the tag or background reduced predictive performance; an ImageNet-pretrained deep network avoided these shortcomings [93].For ship classification, the classifier primarily focused on water in the image’s bottom half.Source: §C.4 · pp. 15–16
• The Atari network used three convolutional and two fully connected layers to predict the Q-function for every possible action.All layers used ReLU nonlinearities except the final fully connected layer.Source: Figure 5 · p. 16

D Task I: Playing Atari Games

• The network uses three convolutional and two inner-product layers, with target parameters θ∗ periodically copied from current parameters θ to stabilize training.The model is trained with replay memory, stochastic gradient descent, batch size 32, learning rate α = 2.5 · 10−4, and updates every fourth step.Source: D Task I: Playing Atari Games · pp. 16–17
• Training clips the Bellman-error difference to [−1, 1], yielding quadratic error up to 1 and constant error beyond that range.This clipping curbs oscillations when the network is far from satisfying the Bellman equation.Source: D Task I: Playing Atari Games · p. 16
• The agent receives four grayscale 84 × 84 frames, predicts expected long-term rewards for available actions, repeats each selected action for four time steps, and retains 10% random exploration.Training begins with 100% random actions and linearly reduces randomness to 10%.Source: D Task I: Playing Atari Games · pp. 16–17
• Layer-wise Relevance Propagation uses the αβ-rule with α = 1 and β = 0, while the lowest layer uses the w2-rule to assign relevance to all pixels.The αβ-rule distributes relevance according to positive forward contributions, and black background pixels otherwise have zero grayscale value.Source: D Task I: Playing Atari Games · p. 17

D.1 Visualization and Comparison to Game Play

• The Breakout agent tracks the ball, builds a vertical tunnel after the ball accelerates, then keeps the ball above the bricks to accumulate rewards.Relevance shifts from the ball and paddle to the tunnel and the region above the brick wall, revealing the agent’s staged strategy.Source: §D.1 · p. 17, Figure 6 · p. 18
• Despite outperforming humans on measured performance and reflexes, the Breakout agent deadlocks after losing the ball because it keeps issuing rightward commands when the paddle no longer changes.The failure reflects insufficient training coverage of nearly cleared boards and forces reliance on random action selection to restart.Source: §D.1 · pp. 17–19, Figure 7 · p. 19
• The Video Pinball agent ignores the paddles, uses controlled nudging to secure an extra life, then exploits the left pipe to increase long-term rewards.It passes through the right pipe four times, then repeatedly routes the ball through the left pipe while avoiding tilt penalties.Source: Figure 6 · p. 18, §D.1 · pp. 19–20

D.2 Is the AI Agent Cheating ?

• The tunnel and nudging behaviors maximize long-term reward within the game rules, but appear to humans as cheating because they exploit weaknesses rather than conforming to intended play.From the agent’s objective, these are clever reward-maximizing strategies rather than violations, unless fairness constraints are explicitly included in the objective.Source: §D.2 · p. 20
• LRP makes unintended strategies easier to detect than raw gameplay observation by exposing what the network tracks, targets, or ignores in a single game.The heatmaps reveal ball tracking, tunnel construction, and paddle neglect, while automated erratic-behavior detection depends on this richer information.Source: §D.2 · p. 20

D.3 Quantifying Strategy Shifts

• During training, LRP quantifies strategy shifts from recognizing the ball, to using the paddle, to acquiring long-term tunnel-building behavior.Objects can become reliably represented before rudimentary gameplay emerges, indicating lower layers track game elements while top layers have not yet formed an appropriate strategic response.Source: §D.3 · p. 21
• Monitoring relevance can distinguish tracking failures in lower network layers from decision-making shortcomings in top layers.Early training shows convolutional filters responding to moving elements while higher layers mainly correlate ball movement with score changes.Source: §D.3 · p. 21

D.3.1 Quantifying the Focus of the Deep Networks over Training

• The analysis compares 200 training snapshots from six Breakout runs using identical early- and late-game inputs, enabling relevance changes to be tracked across learning stages.Each network was trained for 200 epochs, with each epoch spanning 100,000 parameter updates; 500 early-phase and 500 late-phase frames were reused across snapshots.Source: §D.3.1 · p. 21
• Relative relevance is measured for the ball, paddle, and tunnel by normalized pixel-region relevance, with object-specific overlap and visibility conditions preventing confounded measurements.Object relevance is divided by object area and then normalized by average full-frame relevance to reduce effects from differing predicted Q-values.Source: §D.3.1 · pp. 21–22
• After 50 training epochs, tunnel regions come into focus after attention first grows on the ball and then the paddle, indicating a shift toward tunnel-building strategy.Tunnel relevance increases until about the 80th training epoch, while agents begin attempting to build tunnels after roughly 40 epochs.Source: Figure 8 · p. 22, §D.3.1 · p. 22
• Ball relevance emerges earliest because its position and movement directly predict immediate score changes or penalties, before networks learn effective paddle control.Paddle relevance appears only a few epochs later because controlling it affects rewards through delayed future outcomes.Source: §D.3.1 · p. 22
• Despite 200 training epochs, agents fail to restart a lost ball because they do not transfer the learned “fire” action from early to advanced game phases.This apparent lack of transfer knowledge qualifies tunnel building as strategic behavior while supporting the view that reinforcement learning may be driving delayed reward optimization.Source: §D.3.1 · p. 23

D.3.2 Varying Depth of Architecture

• The study compares Nature, its shallower NIPS predecessor, and a Small network with the same convolutional depth but only one fully connected layer.The investigated architecture specifications are summarized in Table 2.Source: §D.3.2 · p. 23, Table 2 · p. 23
• The NIPS architecture focuses on ball and paddle about five training epochs later than Nature, then develops a stronger tunnel relevance after a two-stage shift.The shallower convolutional architecture may slow recognition of important game elements while still permitting higher-level strategies once they are recognized.Source: §D.3.2 · p. 23, Figure 9 · p. 24
• The Small architecture switches early to ball and paddle, yet remains sub-human and never develops tunnel-building attention during training.Its three convolutional layers enable rapid object recognition, but one fully connected layer limits decision-making capacity; it reflects the ball in only three out of four interactions.Source: §D.3.2 · p. 23, Figure 9 · p. 24

D.3.3 Varying Size of Replay Memory

• Replay memory stores state, action, reward, and next state, replaces the earliest entries at capacity, and supplies random training batches.Each Atari interaction adds one game situation to the memory before sampling for training.Source: §D.3.3 · p. 23

D.3.2 Varying Depth of Architecture

• All three architectures shift strategy toward tunnel building, but NIPS does so in two stages with a larger total shift, whereas Small never makes the tunnel relevant.The comparison tracks relevance responses to the ball, paddle, and tunnel over training time.Source: Figure 9 · p. 24

D.3.3 Varying Size of Replay Memory

• Replay memories of 5 · 10^4, 10^6, and 5 · 10^6 states all shift attention toward the tunnel, but larger memories delay and sharpen this transition.The largest memory concentrates relevance more strongly on the tunnel, whereas the smallest memory shifts earlier and more gradually with less concentrated attention.Source: §D.3.3 · p. 24
• The largest replay memory delays tunnel recognition because early-game states remain longer, while smaller memories replace them faster but lack enough later-game observations for successful tunnel learning.This explains the trade-off between removing uncontrolled early-play records and retaining observations from later, more expert phases.Source: §D.3.3 · p. 24

D.3.4 Changes of Model Attention during Game Play

• The Atari analysis combines horizontally pooled relevance heatmaps with action-specific Q-value divergence to track changing attention and decision-critical events over game time.This representation preserves the vertical locations of the ball, paddle, and tunnel while aligning relevance changes with predicted action differences.Source: §D.3.4 · pp. 24–25
• LRP shows that fully trained Breakout networks focus predictions on frames requiring ball interaction, whereas earlier networks remain less decisive.Across training, Q-value divergence becomes concentrated at action-critical moments, while spatial relevance increasingly focuses on the ball and other game elements.Source: §D.3.4 · p. 25, Figure 11 · p. 26

D.3.5 Comparing Relevance Maps to Gradient Map

• Sensitivity Analysis produces blurred saliency around the ball, omits the tunnel, and fails to reveal strategy changes that LRP identifies.Gradient responses indicate where changing the input would most affect prediction rather than which present game elements are important, making the resulting maps less interpretable.Source: §D.3.5 · pp. 27–28, Figure 13 · p. 28

E Task II: Image Categorization

• Image-classification evaluation should explain classifier behavior because accuracy and test error alone may not establish reliable or conceptually appropriate problem-solving strategies.The section contrasts Fisher Vector and deep neural network predictors on Pascal VOC using LRP to expose their prediction strategies.Source: E Task II: Image Categorization · p. 29

E.1 The Fisher Vector Classifier

• The Fisher Vector classifier uses dense SIFT descriptors, a GMM-based Fisher representation, spatial pooling, and a linear SVM for 20 one-vs-all Pascal VOC object classes.Its reported Pascal VOC 2007 test performance is 61.69% mAP in and 59.99% mAP in.Source: §E.1 · p. 29

E.2 The Multilabel Deep Neural Network

• The fine-tuned multilabel deep network replaces softmax with hinge loss for 20 Pascal VOC classes and achieves 72.12% mAP on Pascal VOC 2007.Training uses augmented 227×227 crops from Pascal VOC 2012 and 15,000 fine-tuning iterations.Source: §E.2 · p. 30

E.3 Comparing Fisher Vector and Deep Network Prediction Strategies

• LRP compares the classifiers using model-specific decompositions: ε- and flat rules for the Fisher Vector pipeline, and the αβ-rule with α = 2, β = −1 for the deep network.These rules provide numerically stable pixel-level relevance decompositions while accounting for the different structures of feature extraction and neural layers.Source: §E.3 · p. 30
• Fisher Vector relevance maps have lower spatial granularity because the flat rule uniformly distributes descriptor relevance across receptive fields, unlike explicit neural connections.Local feature extraction can obscure direct pixel-to-neuron relationships, so the projection to pixels sacrifices heatmap resolution.Source: §E.3 · p. 30

E.3.1 Detecting Boats & Horses

• DNN predictions primarily use objects, while FV predictions often depend on background information, co-occurring objects, or other contextual elements.This difference is reflected qualitatively in relevance maps and in the models’ reported prediction performances.Source: §E.3.1 · p. 30, §E.3.2 · p. 33
• For boats, FV relies on water while DNNs use the boat silhouette; FV can therefore misclassify a shipwreck on land as “aeroplane”.The FV model’s water-based rule holds across almost all boat samples and also appears in contextualized predictions for other classes such as “aeroplane”.Source: §E.3.1 · p. 31, Figure 15 · p. 32
• FV horse detection exploits a copyright watermark: removing it causes false negatives, while adding it to a sports-car image causes false positives.The watermark occurred in approximately 20% of horse images, while horse-show obstacles also supplied easy contextual features; DNN predictions and heatmaps were invariant to watermark removal.Source: §E.3.1 · pp. 31–32, Figure 16 · p. 33

E.3.2 Importance of Context

• The outside-inside relevance ratio quantifies context importance: higher values indicate reliance on positively relevant background pixels, whereas lower values indicate object-centric predictions.Applied per class and model, the measure uses object bounding boxes and compares positive relevance inside versus outside those boxes.Source: §E.3.2 · p. 33
• FV achieves high performance for some classes through contextual bias rather than generalizable object concepts, making failures likely outside the controlled benchmark setting.For “aeroplane” and especially “boat”, contextual cues can support correct majority predictions despite poor concept generalization.Source: §E.3.2 · p. 33, Figure 17 · p. 34

E.3.3 Why do FV and DNNs exhibit different strategies ?

• FV’s spatial subdivision encourages concentration on predictive image regions, while Pascal VOC’s small, co-occurring, and insufficiently diverse data encourages contextual and cross-class biases.The FV mapping computes separate representations for image sub-areas, and frequent label co-occurrence allows features from other classes to support predictions.Source: §E.3.3 · pp. 33–34
• DNNs use lower-context, object-centered representations because ImageNet pretraining supplies robust, semantically overlapping prior knowledge that is less sensitive to Pascal VOC biases.Interior-object classes such as “chair”, “diningtable”, and “sofa” remain outliers because their features frequently co-occur in cluttered scenes, complicating category separation.Source: §E.3.3 · pp. 34–35

Motivation

• SpRAy bridges dataset-wide performance metrics, which reveal little about classifier internals, and single-prediction relevance analysis, which requires human inspection and does not scale.The method is intended to provide automated behavioral analysis while retaining information about decision processes.Source: F Semi-automated Analysis of Classifier · p. 35

Method contribution

• SpRAy is presented as the first method to investigate classifier prediction strategies across an entire dataset in a semi-automated manner.It combines relevance-map computation with spectral analysis.Source: F Semi-automated Analysis of Classifier · p. 35

Detection capabilities

• SpRAy detects irregularly frequent recurrences of non-obvious, highly similar image features that may expose suspicious patterns in training data, test data, or learned prediction strategies.The detected features can be benign class representatives or unintended co-occurring features, helping identify model and dataset weaknesses.Source: F Semi-automated Analysis of Classifier · p. 36

F.1 Spectral Relevance Analysis

• The FV classifier used a copyright watermark in roughly one fifth of horse images as a stronger horse indicator than the pictured animals.The watermark was present in both training and test images and did not visually describe horses.Source: §F.1 · p. 36
• SpRAy finds features correlated with a target class and detects them through concentrated relevance feedback in models’ relevance maps.Such correlated features may be non-obvious to human observers even when they strongly describe the prediction target.Source: §F.1 · p. 37
• The pipeline computes relevance maps, standardizes them through preprocessing, applies spectral clustering, identifies clusters by eigengap analysis, and optionally visualizes them with t-SNE.Preprocessing makes maps uniform in shape and can reduce dimensionality, accelerate analysis, improve robustness, and focus analysis on selected characteristics.Source: §F.1 · p. 37
• Spectral clustering builds an affinity graph, symmetrizes asymmetric neighborhood relations, and uses Laplacian eigenvalues and eigengaps to identify dense sample groups.Exactly zero eigenvalues indicate disjoint clusters, whereas eigenvalues near zero and eigengaps identify weakly connected groups; substantial distribution overlap can obscure the eigengap.Source: §F.1 · p. 38
• t-SNE embeds images or relevance maps in lower dimensions using pairwise distances, reusing spectral-clustering affinities to visualize prediction patterns.The affinity matrix can be converted into distances by assigning high distances to unconnected points with a small denominator offset.Source: §F.1 · pp. 38–39

Implications

• These localized strategies can fit small subsets of training and test samples yet cause overfitting when spurious correlations occur less often in real-world data.The analysis uses eigenvalue spectra, spectral clustering, t-SNE embeddings, and relevance maps to expose strategies that may be difficult to detect in raw images.Source: §F.2 · pp. 39, 41, 43

Horse classifiers

• SpRAy identifies four well-separated Fisher Vector prediction strategies for “horse,” including copyright tags and tourney hurdles, while DNN relevance maps form one connected cluster.The Fisher Vector model uses cues correlated with “horse” but not intended to represent the object, whereas the DNN focuses on horses and riders.Source: §F.2 · pp. 39–41, Figure 22 · p. 40

Comparison and limitation

• Model-output differences alone miss suspicious behavior: classes such as “horse” can have comparable model performance despite SpRAy exposing distinct prediction strategies and unintended cues.The replicated cue analysis is useful for verifying features already identified by SpRAy, but correlated features prevent it from reliably identifying which feature itself drives predictions.Source: §F.2 · pp. 41–43, Figure 24 · p. 42

Horse classifiers

• The Fisher Vector model’s copyright-tag reliance substantially increases Recall for “horse,” while the DNN is largely unaffected; both models favor reasonably sized, upright horses.Hurdles appear correlated with tall, slim bounding boxes, but relevance maps show that only the Fisher Vector model uses hurdles, whereas the DNN uses horses and rider knees.Source: §F.2 · pp. 42–43, Figure 24 · p. 42, Figure 25 · p. 43

Boat classifier

• For “boat,” Fisher Vector relevance maps form weak subclusters centered on water rather than boats, whereas the DNN uses boat shapes and related structures such as sails.The water bias is plausibly enabled by spatial pyramid mapping, which incorporates global scene geometry but can facilitate spatial biases.Source: §F.2 · pp. 43–44, Figure 27 · p. 45

Aeroplane classifier

• For “aeroplane,” DNN clusters reveal reliance on top and bottom image borders introduced by padding, although the artifact is consistent without dominating correct recognition.The Fisher Vector model instead reacts to uniform, structureless background—especially in the upper-left region—and generalizes that background rather than the intended class.Source: §F.2 · pp. 44–45

Verifying the Detected Bias in Prediction behavior

• Across aeroplane samples, mirror padding, cropping, and random-noise padding reduce airplaneness, whereas suitable constant-color padding yields comparatively higher scores; the same pattern appears for non-aeroplane images.Adding structured border content reduces the prediction, while uniformly colored extensions can provoke a positive aeroplane response even when the image is not labelled aeroplane.Source: §F.2.1 · pp. 46–48, Figure 31 · p. 48
• Random-noise padding produces the largest decrease in aeroplane prediction and removes the border artifact, while constant-color padding can preserve or increase the prediction.The relevance assigned to the image center remains nearly unchanged under random-noise padding, indicating that the output change is driven by the border artifact.Source: §F.2.1 · p. 46, Figure 29 · p. 47, Figure 30 · p. 47
• The relevance maps attribute strongest positive relevance to the top and bottom borders, and constant padding increases airplaneness more for vertically padded landscape images than horizontally padded portrait images.The observed spatial bias is consistent with sky-like regions appearing above and below photographed objects.Source: §F.2.1 · p. 48, Figure 32 · p. 49
• The classifier therefore detects airplanes partly through uniform image areas of constant sky-like color, creating a risk of false positives when camera images contain such borders.The authors identify preprocessing, training-data, and future model revisions as possible ways to avoid predictions based on low-frequency image borders.Source: §F.2.1 · pp. 48–49

F.3 Spectral Relevance Analysis on Atari Gameplay Sequences

• Applying SpRAy to roughly 95-second recordings shows that relevance maps isolate important temporal phases that are less clearly recognizable directly from model inputs.The analysis uses emulator frames and action-specific relevance maps from trained DNNs playing Breakout and Video Pinball.Source: §F.3 · pp. 49, 51
• Figure 33 shows two near-zero eigenvalues and two large eigengaps for Breakout, versus three near-zero eigenvalues and a smaller distinct eigengap for Video Pinball.These spectra indicate many smaller clusters among game frames but fewer, more distinctive clusters among relevance maps.Source: Figure 33 · p. 49, §F.3 · p. 50
• SpRAy relevance maps identify two to three distinct gameplay-phase clusters, whereas raw frames fragment into many temporally organized groups.In Breakout, relevance maps separate early play with an intact brick wall from late play after the DNN creates a tunnel; Video Pinball likewise forms three main relevance-map clusters.Source: Figure 34 · p. 50, §F.3 · pp. 50–51
• In Video Pinball, the first relevance cluster covers ≈30% of gameplay as the model positions the ball near the top-right element and targets four passes.Relevance concentrates on the ball and targeted top-right area during this opening phase.Source: §F.3 · p. 51
• The second and third Video Pinball clusters cover about 50% of gameplay, representing repeated high- and low-ball-position attempts near the top-left element.After a mistake sends the ball away, the DNN re-enters the initial control-and-positioning phase.Source: §F.3 · p. 51

Concluding Remarks

• SpRAy bridges dataset-level performance metrics and manual single-prediction heatmap assessment by rapidly revealing prediction-strategy composition through LRP relevance inputs and t-SNE embeddings.It identifies whether predictions are heterogeneous or rely on unintended information, while presenting spectral structure in a human-interpretable form.Source: §F.4 · p. 51
• The analysis found systematic FV-model biases for “boat”, “horse” and “aeroplane” linked to spatial pyramid mapping, and an input-preprocessing issue in the DNN model that could be rectified.These findings arose from comparative eigenvalue-spectrum analysis across Pascal VOC 2007 object categories, model architectures, and preprocessing choices.Source: §F.4 · p. 51
• SpRAy isolated important gameplay phases in Breakout and Video Pinball from temporally structured relevance data that were not as transparent in the recorded game frames.The method analyzed relevance maps from recorded Atari 2600 gameplay sequences and recovered phases previously identified manually.Source: §F.4 · p. 52
• Future extensions include alternative explanation methods, relevance preprocessing choices such as interpolation, and Fourier-domain analysis to emphasize different structures in explanation heatmaps.The initial preprocessing used sum pooling to preserve locality, while interpolation could act as a low-pass filter and Fourier transforms could shift analysis into the frequency domain.Source: §F.4 · p. 52