Interpretation of Deep Temporal Representations by Selective Visualization of Internally Activated Nodes

1 Introduction

• Industrial adoption of deep learning is hindered by demands for transparent decisions, while time-series data often lack temporal annotations that help locate semi-global patterns.The paper notes that image interpretation benefits from annotated segmentation information, which is generally unavailable for time-series datasets.Source: §1 · pp. 1–2
• The framework identifies representative temporal patterns activating each convolutional-network channel without hand-crafted segmentation labels.It does this by clustering temporal patterns of highly activated nodes.Source: §1 · p. 2
• It matches perceived subsequences to the closest representative patterns so users can check whether patterns are common or deviate from trained data.Source: §1 · p. 2
• Uncertainty estimates for representative patterns indicate the potential variance of input shapes that activate a channel.Source: §1 · p. 2
• The framework visualizes temporal representations across channels and hidden layers, with higher layers associated with long-term trends and lower layers with detailed features.Source: Figure 1 · p. 2

2 Related Work

• Network Dissection matches CNN units to labeled concepts, while LRP decomposes output relevance and CAM highlights class-specific regions; these approaches leave limited support for time-series interpretation.Network Dissection relies on segmentation labels, which are unavailable for most time-series datasets.Source: §2 · p. 3

3 Clustered Pattern of Highly Activated Period (CPHAP)

• The CPHAP process is presented as a workflow for extracting and patternizing highly activated periods.Figure 2 is identified as the process for applying CPHAP.Source: Figure 2 · p. 3

3.1 Convolutional Neural Networks

• CNN channels encode features in activation maps, and deeper-layer receptive fields cover longer input regions that can be examined to interpret channel responses.The paper chooses CNN-based models because interpreting hidden units in recurrent networks is complicated by their recursive structure.Source: §3.1 · p. 4

3.2 Extracting important input sub-sequences

• Highly Activated Nodes are channel nodes above a threshold set so that only 5% of activations exceed it; their input receptive fields form the Highly Activated Period.The periods are treated as important input subsequences contributing to highly activated nodes.Source: §3.2 · p. 4

3.3 Patternizing representative sub-sequences by clustering

• A Self Organizing Map clusters HAPs into representative patterns, with each CPHAP defined as the cluster’s average sequence.The method reports cluster variance as uncertainty about how confidently input subsequences belong to a cluster; the map uses 64 groups and preserves neighborhood relationships among patterns.Source: §3.3 · pp. 4–5
• The paper reports that training-derived CPHAPs match new test sequences, while some less-matched examples deviate at points associated with greater cluster uncertainty.Figure 3 describes CPHAPs from training data alongside sequences from an unused test dataset.Source: Figure 3 · p. 5, §3.3 · p. 5

4 Experimental Results

• Experiments use temporal CNNs and include UWaveGestureLibraryAll and HAR; ResNet CPHAP pattern lengths vary by hidden layer with model structure.The reported ResNet lengths are 15, 25, 31, 45, 55, 61, 75, 85, and 91.Source: §4 · pp. 5–6, Figure 4 · p. 6

4.1 Identifying role of the channel in CPHAP

• The frequency matrix links class labels to channels: examples include Channel 27 with Class 1 and an inverted-concave pattern, and Channel 10 with Class 6 and a W shape.The reported examples also associate Channel 52 with Class 2 and concave/convex patterns, and Channel 19 with Class 4 and a convex pattern.Source: Figure 5 · p. 6, §4.1 · p. 7

4.2 Comparisons with other interpretable methods

• Channel-LRP shows pointwise importance, whereas CPHAP visualizes continuous subsequences as patterns and also indicates uncertainty for each pattern.The uncertainty can differ across time points within a pattern.Source: §4.2 · p. 7, Figure 6 · p. 7

4.3 Perturbation analysis

• On Uwave, CPHAP is more robust than LRP and Random under Gaussian perturbations, and is better than LRP in most other tested cases.With zero perturbation at ratios 0.8 and 1.0, CPHAP is slightly worse than LRP, though the activation differences are close to zero.Source: Table 2 · p. 8, §4.3 · p. 8
• The perturbation test preserves method-selected regions, perturbs other input points with Gaussian, inverse, or zero changes, and measures absolute channel-activation differences.For LRP, preserved points are selected by relevance score, with their count matched to CPHAP; Random preserves none.Source: Figure 7 · p. 7, §4.3 · p. 8

5 Conclusion

• The method clusters highly activated temporal regions to visualize representative patterns and analyzes channel receptive fields in input space.It computes uncertainty for each cluster to inform interpretation of new inputs.Source: §5 · p. 8
• The framework breaks internal-activation interpretation into channel-by-channel roles for visualization.The paper says this helps users interpret temporal neural-network decisions intuitively.Source: §5 · p. 8
• Perturbation analysis finds temporal regions useful for analyzing node-activation contributions compared with points selected by relevance scores.Source: §5 · p. 8

6 Broader Impact

• The framework automatically extracts representative temporal patterns without human-annotated segmentation information.Source: §6 · p. 11
• The paper proposes CPHAP as a possible source technology for tasks requiring temporal segmented information and AI transparency.Source: §6 · p. 11
• The authors suggest the work may inspire understanding AI systems and applying AI technologies to real-world problems.Source: §6 · p. 11

7 Experimental Settings

• Table 3 is titled “Datasets Information.”Source: Table 3 · p. 11
• Table 4 is titled “CNN Structures used in our framework.”Source: Table 4 · p. 11
• Table 5 is titled “ResNet Structures used in our framework.”Source: Table 5 · p. 11

8.1 Visualization of CPHAP

• CPHAP visualizations distinguish the input sequence, receptive fields, patterns, and pattern uncertainty through separate line styles and colored shading.The input is black, receptive fields are colored dotted lines, CPHAPs are colored lines, and shadows indicate uncertainty.Source: §8.1 · p. 11
• Overlapping receptive fields can produce overlapping detected patterns, so some overlaps are removed to make CPHAPs clearer.Source: §8.1 · p. 12, Figure 9 · p. 12

8.2 Various Methods to Visualize Important Regions

• The method upsamples channel activations before highlighting important regions, adapting CAM-style heatmaps or Network Dissection-style thresholding to internal time-series activations.The section notes that CAM is limited to the last CNN layer and Network Dissection requires predefined concepts.Source: §8.2 · p. 12, Figure 10 · p. 12

9.1 CPHAP Results of ResNet

• Lower ResNet layers capture local changes, while higher layers capture global changes such as slow upward trends.Source: §9.1 · p. 13, Figure 11 · p. 13
• Channels detect patterns of increasing complexity across layers: a W-shaped pattern detected in layers 4 or 5 can appear as a U shape in layer 7.The authors also report that channels 55 and 51 in layer 1 detect sharp decreases and increases, respectively.Source: §9.1 · p. 14, Figure 12 · p. 14, Figure 13 · p. 15
• For a simple sample, lower layers focus on rapid changes or inflection points, while higher layers recognize extreme changes in softer patterns.Source: §9.1 · p. 15

9.2 Comparison of Various Filter Sizes

• Longer convolution filters capture longer patterns and suit global trends, whereas shorter filters are preferable for detecting local features in complex oscillations.The section includes examples labeled as longer- and shorter-pattern results.Source: §9.2 · p. 16, Figure 14 · p. 16, Table 6 · p. 16

• CPHAP is evaluated on test data not used to train the pattern clusters, including well-matched examples.Source: §10 · p. 18, Figure 16 · p. 18
• In less-matched examples, actual data deviate from the assigned pattern at some points.Source: §10 · p. 18, Figure 16 · p. 18
• The less-matched examples show large uncertainty where the data deviate, which the authors take as support for CPHAP's uncertainty capture.Source: §10 · p. 18

• The authors compare K-means, GMM, K-shape, and SOM to characterize subsequences by shape and distribution.K-means uses Euclidean distance, while K-shape accounts for scaling and shift invariances.Source: §11 · p. 19
• SOM is selected because cluster means represent their subsequences and nearby map groups indicate similar patterns.Source: §11 · p. 19
• For UWave, clusters detected by channel 1 in layer 2 are near one another in SOM map space, consistent with their similar patterns.The reported clusters are 3, 9, 10, 11, 12, 16, 17, and 18.Source: §11 · p. 19

12 Identifying role of the channel in CPHAP

• In HAR, channels 4, 25, 29, 45, and 54 are frequently activated for classes 0 or 2, while channels 2, 15, 42, and 57 activate often for class 5.Classes 3 and 5 appear to have similar class-channel relationships.Source: §12 · p. 21, Figure 21 · p. 21

13 Perturbation analysis

• The perturbation-analysis Y-axis represents activation change; a large change indicates that an algorithm failed to select points appropriate to the channel.Source: §13 · p. 22
• Under Gaussian perturbation, preserving CPHAP-selected regions makes activation more robust for most channels; under inverse or zero perturbation, channel 15 performs similarly to LRP.CPHAP generally selects more important points than LRP to maintain activation.Source: §13 · p. 22, Figure 22 · p. 22