Source-linked AI summary
Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks
Luca Bertinetto, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, Nicholas A. Lord
TL;DR
Standard classification metrics treat all incorrect classes alike, leaving mistake severity under-addressed despite fewer errors. The paper reviews hierarchy-aware approaches and introduces two simple cross-entropy modifications; experiments show they outperform prior methods across several metrics, while revealing a tradeoff between top-1 accuracy and hierarchical mistake severity. The usefulness of “better” mistakes also depends on the class relationships encoded by the hierarchy.
Problem
Modern image classifiers rarely exploit class relationships when measuring or learning mistakes, leaving the severity of incorrect predictions insufficiently addressed.
Method
The paper reviews hierarchy-aware methods and proposes two one-parameter drop-in generalisations of cross-entropy, including hierarchical cross-entropy that reweights lineage-based conditional terms.
Results
The proposed loss modifications outperform the few modern methods addressing this problem under several metrics, with their tradeoff curves subsuming prior art across tested settings.
Takeaways & Limitations
Better mistakes appear achievable, but prioritising top-1 accuracy versus hierarchical severity involves a delicate balance and depends crucially on the chosen class relationships.
Takeaways & Limitations
The evaluated WordNet and biological taxonomies reflect visual relationships but are not visual hierarchies themselves, leaving the importance of hierarchy structure open.
Abstract
from arXiv · showhide
Deep neural networks have improved image classification dramatically over the past decade, but have done so by focusing on performance measures that treat all classes other than the ground truth as equally wrong. This has led to a situation in which mistakes are less likely to be made than before, but are equally likely to be absurd or catastrophic when they do occur. Past works have recognised and tried to address this issue of mistake severity, often by using graph distances in class hierarchies, but this has largely been neglected since the advent of the current deep learning era in computer vision. In this paper, we aim to renew interest in this problem by reviewing past approaches and proposing two simple modifications of the cross-entropy loss which outperform the prior art under several metrics on two large datasets with complex class hierarchies: tieredImageNet and iNaturalist'19.
1. Introduction
Modern classifiers make fewer errors, but the severity distribution of their mistakes has changed little because standard metrics treat every non-ground-truth class as equally wrong. The paper revisits hierarchy-aware evaluation and proposes simple cross-entropy modifications to make better mistakes.
- Motivation: Standard top-k evaluation treats all classes other than the ground truth as equally wrong, which may conflict with desired classifier behaviour in applications such as autonomous driving.The paper motivates incorporating relationships between classes into the definition of a mistake.
- Hierarchy-aware evaluation: A taxonomic hierarchy tree provides a natural representation of relationships between classes for evaluating mistake severity.Prior work used WordNet and related hierarchies to quantify the semantic severity of classification errors.
- Related work: Earlier hierarchy-aware classifiers were reported to empirically outperform flat classifiers, while later ImageNet work explored hierarchy-derived costs and similarity functions.These approaches include hierarchical cost minimisation and compatibility functions based on semantic relationships.
- Motivation: Despite spectacular improvements in top-1 error, the severity distribution of mistakes on ImageNet has remained fairly unchanged.Figure 1 compares top-1 error with mistake severity relative to the WordNet hierarchy; dashed lines mark the best achievable metric values.
- Contributions: The paper reviews fragmented prior work, proposes two one-parameter drop-in generalisations of cross-entropy, and evaluates them against prior art.The loss variants can be tuned to trade off top-k and hierarchical performance and reduce to the standard setup in appropriate limits.
2. Framework and related work
The framework separates hierarchy-aware learning into label embeddings, hierarchical losses, and architecture changes. Prior methods modify class representations, penalties, or network structure to exploit relationships encoded by a hierarchy.
- General framework: The standard supervised framework minimises a loss comparing network outputs with embedded class labels, plus a regulariser.Training examples pair images with class labels, and the network predictor is parameterised by θ.
- General framework: With cross-entropy and one-hot labels, the standard framework is agnostic to relationships between classes.The framework therefore asks how a class hierarchy H can be incorporated into learning.
- Label-embedding methods: Label-embedding methods replace class representations with hierarchy- or semantics-informed vectors whose relative locations encode class relationships.Examples include DeViSE, which uses embeddings derived from unannotated Wikipedia text and optimises a ranking loss based on cosine similarity.
- Hierarchical losses: Hierarchical losses parameterise the loss by the class hierarchy so predictions of more distant relatives receive higher penalties.Prior examples modify logistic regression probabilities or directly minimise hierarchy-based costs, sometimes with hierarchy-aware regularisation.
- Hierarchical architectures: Hierarchy-aware architectures divide classification structurally, assigning inputs to superclasses earlier and distinguishing fine-grained classes later.Generalist/expert systems occupy an intermediate position between fully hierarchical and flat classification.
3. Method
The paper introduces two hierarchy-aware modifications of cross-entropy: hierarchical cross-entropy (HXE), which reweights lineage conditionals, and soft labels, which distribute target probability according to hierarchy distance.
- Soft labels: Soft labels replace one-hot targets with class distributions whose probabilities decay exponentially with LCA-based distance from the ground truth.The approach uses the standard cross-entropy with these hierarchy-informed target distributions.
- Hierarchical cross-entropy: HXE factorizes each class probability into conditional probabilities along its path from the leaf to the root.The tree hierarchy provides the conditional decomposition used by the loss.
- Hierarchical cross-entropy: HXE reweights the conditional cross-entropies associated with edges in the ground-truth class lineage.The loss can be applied to models outputting ordinary class probabilities, and becomes standard cross-entropy when all weights equal 1.
- Hierarchical cross-entropy: The weighting λ(C) = exp(−αh(C)) discounts information farther down the hierarchy, with α controlling preference for generic versus fine-grained information.Larger α assigns lower loss to errors involving nodes farther from the root.
- Visualization: Figure 2 contrasts HXE and soft-label loss representations, marking the ground-truth class and highlighting the edges contributing to the total HXE value.The two subfigures visualize the distinct hierarchy-aware loss constructions.
- Soft labels: As β increases, soft labels approach one-hot targets; as β decreases, they approach uniform distributions, with intermediate values favoring closely related classes.This representation can model visual confusion between related classes or impose correlations between classifier outputs, and connects to label smoothing.
4. Evaluation
The evaluation compares hierarchy-aware methods, metrics, and baselines on tieredImageNet-H and iNaturalist-19, then tests how hierarchy structure affects results. The proposed methods expose tradeoffs between conventional top-1 error and hierarchical mistake severity.
- Datasets: Experiments use tieredImageNet-H and iNaturalist-19, whose WordNet and biological-taxonomy hierarchies differ and span many visual concepts.tieredImageNet-H was resampled for standard classification and its hierarchy modified to satisfy the tree assumption.
- Metrics: The evaluation measures top-k error, hierarchical distance of mistakes, and average hierarchical distance of top-k predictions.Hierarchical measures use the lowest common ancestor height; average top-k distance evaluates all k most likely classes.
- Baselines and setup: HXE and soft labels are compared with flat cross-entropy, YOLO-v2, DeViSE, and Barz & Denzler under a shared experimental configuration.DeViSE is omitted on iNaturalist-19 because its class IDs are absent from the word2vec corpus.
- Results: For hierarchical distance, HXE dominates tieredImageNet-H tradeoffs, while soft labels generally perform better for average hierarchical distance at k=5 and k=20.Barz & Denzler is the exception on tieredImageNet-H, attaining slightly lower average distances at a substantial top-1 cost.
- Results: The methods’ hyperparameters tune a tradeoff between top-1 error and hierarchical performance, with validation trends largely replicated on test data.Soft labels with high β or HXE with low α suit top-1 prioritization, while lower β or higher α suit hierarchical prioritization.
- Hierarchy structure: Randomizing hierarchy nodes substantially worsens hierarchical metrics, suggesting effectiveness depends on structural relationships that reflect properties of the data.The authors report that application-specific hierarchy constraints may therefore be limited by underlying data properties.
5. Conclusion
The paper argues that reducing mistake severity remains an open problem: two simple cross-entropy modifications can improve hierarchical mistake quality, but the definition of “better” depends crucially on class relationships.
- Interest in making semantically better classification mistakes has nearly vanished since deep learning’s advent.
- Figure 5 evaluates the tradeoff between top-1 error and hierarchical distances for mistakes and top-20 predictions on iNaturalist-19.
- Two simple cross-entropy baselines outperform the few modern methods addressing mistake severity.
- Better mistakes can be achieved, but balancing standard top-1 accuracy against mistake severity remains delicate.
A. Outputting conditional probabilities with HXE
The HXE approach can represent hierarchy through conditional probabilities or through ordinary class probabilities with loss-based conversion. The latter implements hierarchy only in the loss and consistently performs better across the evaluated metrics.
- Conditional-probability outputs require one softmax at each hierarchy level and logits for every hierarchy node.
- Class-probability outputs use a single softmax over leaf nodes, then derive conditional probabilities for HXE.
- The class-probability implementation incorporates hierarchical information in the loss without requiring hierarchy knowledge during inference.
- Across α values and otherwise identical training settings, class-probability outputs consistently improve performance on all metrics.
B. Note on methods based on hierarchical architectures
The paper excludes generalist/expert hierarchical architectures because they lack hierarchical-measure evaluations, use discovered rather than supplied hierarchies, and alter model capacity.
- The authors therefore do not compare against these architectures.
- None of the surveyed generalist/expert methods report experiments using hierarchical measures.
- Their discovered hierarchies prevent direct comparison with methods using the supplied hierarchies considered here.
- Increased base-model capacity confounds whether hierarchical design itself explains their top-k accuracy gains.
C. More implementation details
The experiments use a common configuration while comparing modern hierarchy-aware methods, including DeViSE, Barz&Denzler, and HXE-related implementations. Implementation choices adapt these methods for consistent evaluation.
- All presented methods use a common network, optimiser, and augmentation configuration for meaningful comparisons.
- The baseline uses ImageNet-pretrained ResNet-18, Adam, 200,000 training steps, batch size 256, and 224×224 images.
- Compared methods: Related hierarchical architectures aim to degrade gracefully by retaining confidence in parent classes when child-level predictions are uncertain.
- The conditional-probability HXE variant outputs logits for every hierarchy node and sums conditional cross-entropies along the ground-truth path.
- Compared methods: DeViSE replaces the softmax with a learned mapping to 300-dimensional semantic word embeddings.
- Compared methods: Barz&Denzler embeds labels using normalized LCA-height similarities and combines feature-alignment with weighted cross-entropy.
- Compared methods: The Barz&Denzler implementation uses ResNet-18 and Adam rather than the original architectures and SGD with warm restarts.
D. Pruning the WordNet hierarchy
The WordNet graph was converted into a pruned tree to satisfy the hierarchical cross-entropy loss assumption, while other methods were evaluated on the same hierarchy for consistency.
- HXE assumes that the data hierarchy is a tree, so the WordNet graph was modified to obtain one.The resulting tree was used for the proposed hierarchical cross-entropy loss.
- For each class, the method selected a longest path to the root and resolved ties by adding the path with the fewest new nodes.This criterion was intended to preserve paths with the highest discriminative power while limiting added structure.
- The pruned hierarchy initially had ENTITY rather than PHYSICAL ENTITY as its root because several physical classes linked through abstract WordNet concepts.The passage identifies BUBBLE, TRAFFIC SIGN, and TRAFFIC LIGHTS as contributing connections through SPHERE and SIGN.
- Soft labels, the cross-entropy baseline, DeViSE, and Barz & Denzler were run with the pruned hierarchy for experimental consistency, despite not requiring a tree.
E. Supplementary figures
The supplementary figures examine mistake-severity distributions and compare architectural versus loss changes for hierarchical cross-entropy across two datasets.
- Random ImageNet/ILSVRC-12 image pairs produce a mistake-severity distribution substantially different from those of the studied DNN architectures.The caption states that dataset properties alone cannot explain the architectures’ distribution shapes.
- Figure 7 compares conditional-probability outputs with direct class-probability outputs on tieredImageNet-H and iNaturalist19-H.Points nearer the bottom-left represent better tradeoffs.