Source-linked AI summary
Label Refinery: Improving ImageNet Classification through Label Progression
Hessam Bagherinezhad, Maxwell Horton, Mohammad Rastegari, Ali Farhadi
TL;DR
Label quality is understudied even though supervised-learning labels can be incomplete, ambiguous, and redundant. The paper introduces Label Refinery, which iteratively updates crop labels during training using neural networks and data, and reports significant ImageNet gains across architectures.
Problem
Label properties have received little attention, while supervised-learning labels can be incomplete, ambiguous, and redundant.
Method
Label Refinery iteratively modifies augmented crop labels during training using networks trained on labels generated by preceding refiners.
Results
Label Refinery consistently improves ImageNet top-1 accuracy across architectures, including AlexNet from 59.3% to 67.2%.
Takeaways & Limitations
The method produces dynamically labeled augmented crops with soft targets and improves classification accuracy across popular network architectures.
Takeaways & Limitations
Validation accuracy does not always predict how well a model produces crop labels, and taxonomy-based refinement did not improve training.
Abstract
from arXiv · showhide
Among the three main components (data, labels, and models) of any supervised learning system, data and models have been the main subjects of active research. However, studying labels and their properties has received very little attention. Current principles and paradigms of labeling impose several challenges to machine learning algorithms. Labels are often incomplete, ambiguous, and redundant. In this paper we study the effects of various properties of labels and introduce the Label Refinery: an iterative procedure that updates the ground truth labels after examining the entire dataset. We show significant gain using refined labels across a wide range of models. Using a Label Refinery improves the state-of-the-art top-1 accuracy of (1) AlexNet from 59.3 to 67.2, (2) MobileNet from 70.6 to 73.39, (3) MobileNet-0.25 from 50.6 to 55.59, (4) VGG19 from 72.7 to 75.46, and (5) Darknet19 from 72.9 to 74.47.
1 Introduction
Supervised learning has advanced mainly through data and model improvements, while label quality remains comparatively understudied despite incompleteness, taxonomy dependence, and inconsistency. Label Refinery updates crop labels during training and improves classification accuracy across multiple architectures.
- Label research has received little effort compared with extensive work on datasets, augmentation, architectures, optimization, and loss functions.
- ImageNet labels can be incomplete, taxonomy-dependent, and inconsistent with cropped image content, creating challenges for learning algorithms.
- Label Refinery modifies crop labels during training using a neural network and the data, producing soft, informative, collective, and dynamic labels.
- 67.2% top-1 accuracy replaces 59.3% for AlexNet after label refinement on ImageNet.
- Label refinement improves top-1 accuracy from 70.6% to 73.39% for MobileNet and from 72.9% to 74.47% for DarkNet19.
- A trained model can refine labels for another model of the same architecture, improving AlexNet from 59.3% to 61.2% in the reported experiment.
2 Related Work
Related work improves learning through label smoothing, taxonomy information, data augmentation, and teacher-student training. Label Refinery addresses crop-label inconsistency while combining model-generated labels with augmented data.
- Label smoothing and random-label replacement regularize training, while other methods use models to improve label consistency.
- Taxonomy-based methods use cross-category relationships, hierarchical losses, or semantic hierarchies to address noisy or missing labels.
- Cropping, rotation, flipping, warping, synthetic oversampling, and GANs expand training data but can introduce inconsistent labels.
- Teacher-student methods use another network or an ensemble for distillation, compression, or training from a model ensemble.
3 Label Refinery
Label Refinery iteratively replaces inaccurate crop labels with soft labels generated by successive neural networks, improving label quality and downstream classification accuracy. The procedure can reuse architectures, incorporate batch-normalization handling, and generate adversarial examples to refine training.
- Sequential label refinement: The first refinery trains on image-level labels, while each subsequent network learns soft crop labels generated by its predecessor.The original ground-truth labels are not directly shown to subsequent models; their information is propagated through the refinery sequence.
- Sequential label refinement: Batch-normalization networks remain in training mode during label generation, producing more accurate labels and models according to the authors.Batch statistics are computed from the current training batch rather than saved running statistics.
- Sequential label refinement: The same architecture can improve labels across repeated passes because each model is trained on a more accurate refined dataset.The paper reports iterative improvement even when some or all refinery networks share one architecture.
- Empirical behavior: Validation accuracy does not always identify the best crop-label generator because severe training overfitting can reproduce image-level labels without producing better crop labels.The paper notes that direct quantitative assessment of crop-label quality is unavailable because crop-level labels are not provided.
- Sequential label refinement: 8% image crops often receive inaccurate original labels, motivating Label Refinery-generated labels for aggressive augmentation.The method augments both image inputs and crop labels, addressing a stated source of accuracy degradation for small crops.
- Additional data and adversarial examples: Refinery labels can be generated on-the-fly for unlabeled or synthetic images, because the refined training dataset depends on input images and refinery outputs.The paper also explores pairing the refinery with the trained network to generate adversarial examples where the two networks disagree.
- Additional data and adversarial examples: Adversarial jittering modifies a crop in the direction that increases KL-divergence disagreement between successive refinery networks, then batches it with the natural crop.This pairing is intended to prevent the trained model from becoming confused by unnatural adversarial inputs.
4 Experiments
Experiments evaluate Label Refinery across architectures, refinement settings, and analyses of label dynamics, generalization, and refinery selection. Refined labels improve accuracy and reduce overfitting, with benefits from cross-architecture, adversarial, dynamic, and soft-label refinement.
- Experimental design: ImageNet experiments compare self-refinement, cross-architecture refinement, adversarial inputs, label variants, category-versus-image refinement, and model generalization.The evaluation includes multiple network architectures and studies both performance gains and the behavior of models trained with refined labels.
- Self-Refinement: All six evaluated architectures improve through self-refinement, although AlexNet requires four successive refinements before convergence while MobileNet and ResNet-50 converge faster.Each model is trained from random initialization using labels refined by the preceding model in the sequence.
- Cross-Architecture Refinement: More than 7 percentage points of top-1 accuracy are gained by AlexNet and ResNetXnor-50, while Darknet19 improves by almost 4 points with cross-architecture refinement.A ResNet-50 refinery benefits architectures including compact MobileNet models and high-capacity VGG networks.
- Adversarial Inputs: Smaller models improve further when trained with adversarial inputs generated from patches where the refinery and trained model disagree.The adversarial label refinery is ResNet-50, and original and modified crops are batched together during training.
- Dynamic Labels vs. Soft Labels: Dynamic labeling and soft labeling each improve AlexNet accuracy, and combining them yields an additional improvement because they address different label issues.Dynamic labels re-label each crop, while soft labels represent distributions of visually similar objects.
- Model Generalization and Refinery Choice: Refined-label models show less overfitting: successive AlexNet models bring training accuracy closer to validation accuracy, while an overfit VGG16 is a poorer refinery than MobileNet.A heavily overfit refinery produces labels too similar to ground truth, limiting its usefulness.
5 Conclusion
The paper introduces Label Refinery to refine labels during training, producing dynamic soft targets that improve classification accuracy and generalization across diverse network architectures.
- Label Refinery dynamically labels augmented training crops with soft targets during training.
- The method achieves significant classification-accuracy gains across a wide range of network architectures.