Source-linked AI summary
Do Convolutional Neural Networks Learn Class Hierarchy?
Bilal Alsallakh, Amin Jourabloo, Mao Ye, Xiaoming Liu, Liu Ren
TL;DR
As class counts grow, CNN classification errors exhibit hierarchical structure, but existing analysis gives limited attention to collective errors and their relation to internal features. The paper presents Blocks, a visual-analytics system integrating samples, CNN representations, and results to analyze and improve CNNs. It reports that hierarchy-aware design improves accuracy and convergence, while Blocks remains limited in connectivity visualization and training-process monitoring.
Problem
Existing CNN visualization methods give little attention to classification errors and lack an integrated, scalable analysis of input images, internal data, and classification results.
Method
Blocks uses visual analytics to inspect class hierarchies, confusion patterns, CNN features, input samples, and classification results together.
Results
Hierarchy-aware CNN design yields significant gains in classification accuracy and convergence speed, including a top-5 error reduction of more than one third for a reference CNN.
Takeaways & Limitations
Class similarity structures provide actionable information for CNN design, training, robustness analysis, and identifying training-data quality issues.
Takeaways & Limitations
Blocks does not show layer connectivity, visualizes layer responses independently, and offers limited monitoring of training parameters.
Abstract
from arXiv · showhide
Convolutional Neural Networks (CNNs) currently achieve state-of-the-art accuracy in image classification. With a growing number of classes, the accuracy usually drops as the possibilities of confusion increase. Interestingly, the class confusion patterns follow a hierarchical structure over the classes. We present visual-analytics methods to reveal and analyze this hierarchy of similar classes in relation with CNN-internal data. We found that this hierarchy not only dictates the confusion patterns between the classes, it furthermore dictates the learning behavior of CNNs. In particular, the early layers in these networks develop feature detectors that can separate high-level groups of classes quite well, even after a few training epochs. In contrast, the latter layers require substantially more epochs to develop specialized feature detectors that can separate individual classes. We demonstrate how these insights are key to significant improvement in accuracy by designing hierarchy-aware CNNs that accelerate model convergence and alleviate overfitting. We further demonstrate how our methods help in identifying various quality issues in the training data.
1 INTRODUCTION
CNNs achieve strong large-scale image-classification results, but their errors form hierarchical patterns that prior visualization methods do not adequately expose. Blocks integrates input samples, internal representations, and classification results to diagnose errors and improve CNNs; hierarchy-aware training reduces top-5 error by more than one third.
- CNNs automatically extract discriminative features and outperform heuristic-feature approaches on large-scale datasets such as ImageNet.
- Visualizing learned features supports CNN analysis, but existing methods have given little attention to visualizing classification errors and refining CNNs accordingly.
- Blocks integrates input samples, CNN-internal representations, and classification results for scalable inspection at ImageNet scale.
- Blocks supports identifying error sources, exploiting class hierarchy in CNN design and training, and curating balanced data to improve robustness.
- More than one third: involving class hierarchy reduces the reference CNN’s top-5 error by more than one third.
2 MOTIVATION AND BACKGROUND
The paper motivates hierarchy-aware CNN analysis from structured ImageNet confusions and limitations in existing visualization tools. It proposes an integrated environment spanning images, internal data, and classification results to support scalable model refinement.
- 2.2 Class Hierarchy: More than 98% of misclassifications fall within two reordered confusion-matrix blocks, separating natural objects from artifacts.
- 2.1 ImageNet and the ILSVRC 2012 Dataset: ImageNet is a large labeled resource with more than 14 million images spanning more than 20,000 object categories.
- 2.1 ImageNet and the ILSVRC 2012 Dataset: ILSVRC 2012 contains 1,000 object categories, with about 1.2 million training images and 50,000 validation images organized as WordNet leaf nodes.
- 2.3.1 Feature Visualization: Existing feature-visualization techniques help diagnose CNNs but do not expose high-level patterns in collective responses across all classes.
- 2.3.3 Network-based Visualization: Network-based visualizations expose architecture, activations, redundancies, and learning-parameter issues, while architecture views support comparing large networks.
- 2.3.4 Training Data and Performance Visualization: Blocks fills a tooling gap by integrating input images, CNN-internal data, and classification results in a scalable exploration environment.
3 Blocks
Blocks analyzes class similarity hierarchies and their effects on classification results and learned CNN features. Its coordinated views connect hierarchy, confusions, responses, and selected image samples, using either predefined or interactively constructed hierarchies.
- Class similarity becomes increasingly strong within hierarchy branches as the number of classes grows, motivating Blocks’ hierarchy-centered design.
- Blocks analyzes how class hierarchy affects both classification results and the image features learned by CNNs.
- The interface combines a hierarchy viewer, confusion matrix, response map, and sample viewer with a unified hierarchy-based class order.
- The hierarchy may be predefined or constructed interactively from the confusion matrix, with examples using ILSVRC 2012 and GoogLeNet.
3.1 Class Hierarchy Viewer
The class hierarchy viewer represents class groups as nested rectangles and supports group-level performance analysis. Interaction enables sorting, compression, selection, and coordinated inspection across views.
- Each icicle-plot rectangle represents a class group, whose color can encode a group-level performance metric.
- Group-level precision measures the fraction of samples classified into a group that actually belong to it, while recall measures the fraction of group samples classified into it.
- The group-level F-measure is defined from group-level precision and recall.
- Inspecting group-level performance under varying conditions reveals hierarchy effects on CNN performance and sensitivity to data variation.
- Users can sort child nodes, contract single-child nodes, inspect group metrics, and select rectangles to update other views.
3.2 Confusion Matrix
Blocks reorders and interactively analyzes confusion matrices to expose hierarchical class groups, distinguish expected from unexpected errors, and investigate their causes.
- Reordering classes reveals dense diagonal blocks whose within-block confusions indicate highly similar classes.
- When no suitable hierarchy exists, seriation algorithms can discover block patterns and recursively refine plausible sub-blocks.
- Block outliers are non-diagonal confusions outside dense blocks, representing unexpected errors between apparently less-related classes.
- Sequential color mapping, logarithmic scaling, diagonal exclusion, filtering, and halo effects make frequent and subtle confusions more visible.
- Interactive selection and filtering connect matrix cells or hierarchy groups to samples, supporting investigation of classification and data-quality errors.
3.3 Response Map
The response map aggregates neuron activity by class and orders classes hierarchically, enabling analysis of group-level and class-specific features across CNN layers.
- 3.3.1 Visual encoding: The heatmap encodes each neuron's per-class average response, with hierarchy-ordered rows and interactively reordered neuron columns.
- 3.3.1 Visual encoding: Thresholded color encoding emphasizes unusually high responses and helps identify neurons responding specifically to selected classes.
- 3.3.1 Visual encoding: Downsampling large convolutional outputs consolidates channel variations so multiple neuron response profiles can be compared side by side.
- 3.3.2 Exploring group-level features: Relevance ranking compares group and non-group responses; advanced inception-6 neurons separate wheeled vehicles specifically, whereas inception-1 separates only higher-level groups.
- 3.3.2 Exploring group-level features: Group-level detectors often encode shape, pose, or background, while color and texture can be shared across classes without a specific group.
- 3.3.2 Exploring group-level features: Some groups lack shared detectors because their classes vary substantially and instead develop class-specific features.
- 3.3.3 Exploring neuron properties: Neuron profiles and saliency-linked sample inspection reveal the visual features supporting class detection and related misclassifications.
3.4 Sample Viewer
The sample viewer presents selected images with prediction and neuron-response information, enabling detailed inspection of errors, transformations, and classifier-specific behavior.
- The viewer displays selected thumbnails, groups samples by actual class, and labels what each current selection represents.
- Thumbnail borders distinguish top-1 correct, top-5 correct, and other predictions, while saliency maps show features associated with selected neurons.
- Users can filter samples by class or group membership, neuron activation, and classification results from different classifiers or transformations.
- Comparing filtered results helps identify properties such as rotation invariance, ease of discrimination, and classifier-specific strengths.
4 APPLICATIONS
Blocks reveals that CNNs learn class hierarchies progressively: early layers separate broad groups quickly, while deeper layers develop fine-grained separation later. Hierarchy-aware training uses these layer-specific capabilities to improve accuracy and convergence.
- Applications: Blocks examines training behavior, feature-detector separability, architecture improvement, and data curation through coordinated visual-analytics views.The system supports understanding, diagnosing, and improving CNNs.
- Understand: model convergence: Major confusion blocks emerge after one epoch, with broad group separation preceding subgroup and narrow-group separation.The final epoch contains fewer overall confusions, generally limited to narrow groups.
- Understand: model convergence: Early-layer responses converge within the first two epochs, whereas deeper-layer responses continue changing at later epochs.This pattern indicates different training timescales for broad and fine-grained feature development.
- Understand: model convergence: Early-layer features separate high-level groups nearly as well as output features, while deeper features are needed for fine-grained groups.Blocks assesses layer classification power using linear classifiers trained on layer responses.
- Improve: exploiting the class hierarchy: The adapted AlexNet adds layer-specific group-classification branches that back-propagate group error during training and are removed afterward.The first layer classifies three broad groups, with more groups introduced in subsequent layers.
- Improve: exploiting the class hierarchy: More than one third reduction in top-5 error is reported for the hierarchy-aware architecture, whose main branch matches baseline AlexNet complexity.The additional branches are used only during training; top-5 error reaches 24.6% after four epochs, and results improve more than HD-CNN on ILSVRC.
4.2 Sensitivity to Image Transformations
Blocks evaluates how grayscale conversion and rotation affect class-group accuracy, revealing which visual features CNNs rely on. The results connect transformation sensitivity to dataset variation and feature development.
- Color sensitivity: Grayscale conversion exposes whether classes depend on color features by comparing group-level precision with results from original color images.Red marks accuracy decreases caused by removing color information.
- Color sensitivity: Food-group accuracy drops by 60% for fruits, 43% for vegetables, and 67% for dishes after grayscale conversion.Artifact groups such as electronic equipment and seats show 0% change, while measuring instruments drop 1% and wheeled vehicles 3%.
- Color sensitivity: The CNN relies on color-dependent discriminative features for affected classes, even when those classes have distinctive shapes.Training-data curators can add grayscale or color-varied samples, although intrinsic color can remain necessary for some classes.
- Rotation sensitivity: Rotation-invariant classes commonly have circular shapes or rotation-invariant texture and color features, whereas rotation-sensitive classes have non-circular shapes and fixed poses.The comparison is based on reclassification after 90° image rotation.
- Rotation sensitivity: Aquatic birds lose 39% and hoofed mammals 45% after rotation, while invertebrates lose 4% when training samples contain varied rotations.The authors associate varied training rotations with rotation-invariant features and consistent poses with rotation-sensitive features.
4.3 Inspecting Data Quality
Blocks uses confusion patterns and sample inspection to identify dataset-quality problems that can distort evaluation and class learning. The analysis highlights semantic overlap, mislabeled samples, and hierarchy mismatches.
- Class-quality issues: Filtering frequent confusions and inspecting block outliers reveals redundancy, subclass, part-of, semantic-ambiguity, and abstract-class issues.These issues arise among ImageNet classes selected from the WordNet ontology.
- Class-quality issues: About 5% of classes exhibit these semantic issues, causing significant top-1 accuracy loss that is not caused by the classifier.The passage notes that reliance on top-5 error may leave such problems largely unnoticed.
- Sample and hierarchy inspection: Inspecting high-confidence misclassifications with low ground-truth probability identifies mislabeled samples such as a lion labeled as monkey.Blocks also supports restructuring ImageNet’s predefined hierarchy to better reflect visual similarity.
5 DISCUSSION
Blocks provides scalable, hierarchy-ordered views for inspecting large CNN classifiers, while revealing limitations in connectivity inspection and training-process monitoring.
- Relation to prior work: Blocks was designed to expose nested confusion patterns and group-level response patterns that prior confusion matrices and response maps did not comprehensively reveal.Earlier systems focused on alternative data facets, selected responses, or limited sample-level feature visualizations.
- Scalability: The hierarchy viewer, sample viewer, and similarity ordering help users identify class groups and inspect 1000×1000 confusion matrices without scrolling.Multiscale aggregation supports larger matrices but sacrifices individual-class confusion detail.
- Limitations: Blocks does not show layer connectivity or connection weights, and independent layer views hinder inspection of how feature detectors build on preceding layers.The authors suggest combining Blocks with CNNVis to address this limitation.
- Limitations: Training monitoring is limited to changes in the confusion matrix and response map, leaving initialization, learning-rate, and regularization effects for further study.The authors specifically identify pre-training, learning rate, and DropOut as training parameters requiring closer examination.
- Scope: Except for its response map, Blocks may generalize beyond CNNs to large-class classifiers with explicit or latent class hierarchies.The proposed broader use focuses on how hierarchy affects model behavior and performance.
6 CONCLUSION AND FUTURE WORK
The paper presents visual-analytics methods centered on hierarchical class similarity to inspect and improve large-scale CNN classification. It reports gains in accuracy and convergence speed while using class-similarity analysis to identify training-data quality issues and guide robustness improvements.
- Conclusion: The methods use hierarchical similarity structures to inspect CNNs and improve their design and accuracy on large-scale image classification.They examine how class structures affect CNN feature detectors across layers and training epochs.
- Conclusion: Hierarchy-aware CNN architectures yield significant gains in classification accuracy and convergence speed.The paper connects these gains to understanding how class similarity influences learned feature detectors.
- Conclusion: Class-similarity analysis reveals overlapping class semantics, labeling issues, and imbalanced training-data distributions.The authors use these findings to curate representative data and improve robustness to data variation.
- Future work: Future work will study class-similarity effects in other large-scale classifiers and domains beyond image classification.