PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Label Distribution Learning with Label Ambiguity

Bin-Bin Gao, Chao Xing, Chen-Wei Xie, Jianxin Wu, Xin Geng

arXiv:1611.01731v2cs.CV

TL;DR

Small or ambiguously labeled datasets make several visual recognition tasks difficult for conventional deep learning. DLDL converts labels into discrete distributions and trains deep ConvNets with KL divergence, exploiting ambiguity in feature and classifier learning. It reports better age and head pose estimation than state-of-the-art methods while also improving multi-label classification and semantic segmentation.

  • Problem

    Age estimation, head pose estimation, multi-label classification, and semantic segmentation often lack sufficient training images with precise, unambiguous labels.

  • Method

    DLDL converts each image’s label into a discrete distribution and minimizes KL divergence between predicted and ground-truth distributions with deep ConvNets.

  • Results

    DLDL outperforms state-of-the-art methods on age and head pose estimation and improves multi-label classification and semantic segmentation.

  • Takeaways & Limitations

    DLDL uses label ambiguity in both feature and classifier learning and helps prevent over-fitting when training sets are small.

Abstract

from arXiv · show

Convolutional Neural Networks (ConvNets) have achieved excellent recognition performance in various visual recognition tasks. A large labeled training set is one of the most important factors for its success. However, it is difficult to collect sufficient training images with precise labels in some domains such as apparent age estimation, head pose estimation, multi-label classification and semantic segmentation. Fortunately, there is ambiguous information among labels, which makes these tasks different from traditional classification. Based on this observation, we convert the label of each image into a discrete label distribution, and learn the label distribution by minimizing a Kullback-Leibler divergence between the predicted and ground-truth label distributions using deep ConvNets. The proposed DLDL (Deep Label Distribution Learning) method effectively utilizes the label ambiguity in both feature learning and classifier learning, which help prevent the network from over-fitting even when the training set is small. Experimental results show that the proposed approach produces significantly better results than state-of-the-art methods for age estimation and head pose estimation. At the same time, it also improves recognition performance for multi-label classification and semantic segmentation tasks.

I. INTRODUCTION

Many visual recognition tasks have small datasets and ambiguous labels, limiting conventional deep learning. DLDL addresses this by learning label distributions that exploit ambiguity during feature and classifier learning.

  • Motivation: Age, head pose, multi-label classification, and semantic segmentation are difficult because precise labels and sufficient training data are hard to obtain.Boundary pixels, apparent age, head pose, and sparse age or pose coverage create annotation and data-collection challenges.
  • Label ambiguity: Label ambiguity represents uncertainty among ground-truth labels and occurs naturally in age, head pose, segmentation boundaries, and difficult-to-recognize objects.For apparent age, neighboring ages and different annotators’ guesses may all describe the same face.
  • Label ambiguity: Traditional single-label and multi-label recognition identify applicable labels but do not describe the ambiguity associated with those labels.The paper argues that exploiting label ambiguity can improve recognition performance.
  • DLDL framework: DLDL converts each instance into a discrete label distribution, increasing the training instances associated with each class without increasing the total number of images.A deep ConvNet learns the distribution through both feature learning and classifier learning.
  • DLDL framework: DLDL uses label ambiguity to relax the requirement for large training sets and make neighboring labels useful, such as ages 24 and 26 for an image labeled 25.The framework is intended to provide more robust performance than existing classification and regression methods.
  • Results: Without an ensemble, DLDL outperforms state-of-the-art methods for age and head pose estimation and improves multi-label classification and semantic segmentation.These results span the four recognition settings emphasized by the paper.

II. RELATED WORK

Prior work modeled label correlations or learned deep representations, but these approaches did not fully exploit ambiguity in visual representation learning. DLDL integrates label-distribution learning with deep ConvNets and extends it across several recognition tasks.

  • Earlier approaches: Earlier visual recognition methods commonly extracted hand-crafted features before applying SVMs, random forests, or neural networks.Label distribution learning improved age and head pose estimation by exploiting correlations among adjacent labels.
  • Deep learning: Deep learning improved visual recognition, with ConvNets commonly using softmax loss for classification and squared ℓ2 loss for regression.These objectives treat classification and regression differently from label-distribution prediction.
  • Limitations of prior work: Label smoothing regularizes classifiers with a uniform distribution, but it does not mine task-specific ambiguous information among labels.The paper therefore treats label ambiguity as information that should be explicitly modeled.
  • DLDL scope: DLDL focuses on exploiting label ambiguity in deep ConvNets for age and head pose estimation, with extensions to multi-label classification and semantic segmentation.The framework is presented as applicable beyond the two primary facial estimation tasks.
  • DLDL approach: DLDL uses an end-to-end deep learning approach that incorporates label correlations in both feature learning and classifier learning.Earlier label distribution methods used features extracted without these correlations.
  • DLDL formulation: DLDL quantizes possible outputs into ordered labels and predicts a distribution over them, forcing the model to account for ambiguity among labels.For age estimation, the example label set contains integer ages from 1 through 85.
  • DLDL formulation: The framework defines an input image space, a task-specific complete label set, and a training dataset of image-distribution pairs.This formulation supports task-specific label distributions rather than a single universal label encoding.

B. Deep label distribution learning

DLDL maps ConvNet activations to predicted label distributions and trains them to match ground-truth distributions. Its outputs support single-label, multi-label, and real-valued predictions.

  • Prediction: A deep ConvNet transforms an input image into final-layer activations and applies softmax to produce a predicted label distribution.The distribution is the network output used for subsequent prediction.
  • Optimization: DLDL trains parameters so the predicted distribution resembles the ground-truth distribution, using Kullback-Leibler divergence as the similarity objective.The loss is minimized with stochastic gradient descent.
  • Prediction: For a single expected label, DLDL selects a class from the predicted distribution.The supplied passages state the forward-run generation of the distribution and the single-label output rule.
  • Prediction: For multiple labels, DLDL returns labels whose predicted probabilities exceed a predefined threshold.The threshold is denoted ξ and lies between 0 and 1.
  • Prediction: For real-valued outputs, DLDL predicts the expectation of the label distribution, making the framework suitable for both classification and regression.The same distributional output supports different task-specific decoding rules.

C. Label distribution construction

DLDL constructs task-specific discrete label distributions to represent ambiguity in age, pose, multi-label, and segmentation annotations. These distributions encode uncertainty while preserving differences among possible labels.

  • A valid label distribution is normalized, with probabilities in [0, 1] that differ across possible labels.
  • Age estimation: Age distributions concentrate around the ground-truth age using a normal distribution over an ordered label set.Apparent-age labels use the mean and standard deviation derived from multiple annotator votes.
  • Head pose estimation: Head pose is represented as a joint distribution over discrete pitch and yaw labels.The distribution is stored as an n1×n2 matrix whose entries correspond to pitch–yaw pairs.
  • Multi-label classification: Multi-label distributions assign ordered probabilities to Positive, Difficult, and Negative labels, preserving information that binary treatment loses.The construction uses pP > pD > pN and applies ℓ1 normalization.
  • Semantic segmentation: For semantic segmentation, a Gaussian kernel replaces one-hot pixel labels with distributions that encode ambiguity near object boundaries.The experiment uses K = 5, P = 2, and S = 1; the resulting distributions encode boundary ambiguity.

D. The DLDL architecture and training details

DLDL uses deep ConvNets whose task-specific output layer predicts label distributions and whose original softmax loss is replaced by KL loss. Training supports both initialization from scratch and fine-tuning of pretrained networks.

  • DLDL modifies ZF-Net or VGG-Net by changing the final output for the task and replacing softmax loss with KL loss.The architecture retains deep convolutional and fully connected layers while learning the target label distribution.
  • Training from scratch: The models can be trained from scratch with random Gaussian initialization, dropout, weight decay, and mini-batch SGD.For ZF-Net, dropout is applied to the final two fully connected layers at rate 0.5.
  • Fine-tuning: Fine-tuning removes pretrained classification and loss layers, then inserts a Gaussian-initialized label distribution layer and KL loss layer.The pretrained models include 16- and 19-layer VGG-Nets and VGG-Face.

IV. EXPERIMENTS

The experiments evaluate DLDL across age estimation, head pose estimation, multi-label classification, and semantic segmentation using a MatConvNet implementation.

  • DLDL is evaluated on four tasks: age estimation, head pose estimation, multi-label classification, and semantic segmentation.
  • The implementation is based on MatConvNet and experiments run on an NVIDIA K40 GPU with 12GB of memory.

A. Age estimation

The age-estimation experiments compare DLDL with published methods and ConvNet baselines on Morph and ChaLearn. DLDL consistently performs strongly, with fine-tuning substantially improving results on the small ChaLearn dataset.

  • Datasets: Morph contains 55,134 images from more than 13,000 subjects, with ages from 16 to 77, and uses 10-fold cross-validation.
  • Datasets: ChaLearn contains 4,699 in-the-wild images split into 2,476 training, 1,136 validation, and 1,087 testing images.Apparent ages are labeled by multiple individuals.
  • Evaluation: Age estimation is evaluated using MAE, CS, and ChaLearn’s ϵ-error, with CS measuring accuracy within an error threshold g.
  • Results: DLDL consistently outperforms baselines and published methods; its MAE gap over C-ConvNet on Morph is 0.51.The reported gap exceeds six times the sum of the methods’ standard deviations.
  • Results: Eq. 9 outperforms Eq. 8 in many cases, suggesting that Eq. 9 is more suitable for age estimation.
  • Results: Fine-tuning reduces ChaLearn MAE from 5.34 to 3.51 and ϵ-error from 0.44 to 0.31.The fine-tuned model achieves the best CS accuracy at every error level on ChaLearn and Morph.
  • Qualitative results: DLDL predictions are often accurate, while failures arise from missed or misaligned faces and extreme conditions such as occlusion or low resolution.

B. Head pose estimation

DLDL is evaluated for head pose estimation across Pointing’04, BJUT-3D, and AFLW using pitch, yaw, and joint pitch+yaw measures. It performs well with few training images and achieves the best performance on datasets with sufficient training data, while remaining vulnerable to extreme angles.

  • Evaluation setup: Head pose is represented by pitch and yaw angles, with evaluation using MAE and classification accuracy for pitch, yaw, and joint pitch+yaw predictions.For joint pitch+yaw, MAE is the Euclidean distance between predicted and ground-truth angle pairs.
  • Pointing’04: DLDL reaches 73.15% accuracy for pitch+yaw on Pointing’04, compared with 42.97% for C-ConvNet.Pointing’04 contains only 2,790 images, and other deep learning baselines were reported as poorly learned in this small-data setting.
  • BJUT-3D and AFLW: On BJUT-3D and AFLW, DLDL achieves the best performance, with lower MAE and higher accuracy than the compared methods.These datasets have enough training data, whereas competing convolutional methods show less robust performance across dense and sparse label settings.
  • Failure cases: DLDL estimates head pose with low errors on AFLW but produces more incorrect predictions when both yaw and pitch are large, such as ±90°.The paper attributes this vulnerability to fewer training examples for large angles.

C. Multi-label classification

DLDL is applied to multi-label image classification by learning label distributions with image- and proposal-based fine-tuning variants. The proposal-based approach gives the strongest VOC results, while image-based training is cheaper and avoids region proposals.

  • Dataset and evaluation: The experiments evaluate multi-label classification on PASCAL VOC2007 and VOC2012 using average precision and mean average precision.Images may contain one or several labels from 20 object categories, and training uses the TRAINVAL split before TEST evaluation.
  • DLDL variants: IF-DLDL fine-tunes ConvNets on images, whereas PF-DLDL further fine-tunes them using clustered region proposals.The two variants are defined by whether fine-tuning uses images or image proposals.
  • VOC2007 results: PF-DLDL achieves 92.3% mAP on VOC2007, exceeding HCP-VGG’s 90.9% without using bounding-box annotations.The reported comparison is for the single-model VOC2007 results.
  • Ablation comparisons: IF-DLDL outperforms IF-VGG-ℓ2 by 1.1% on VOC2007 and 1.3% on VOC2012, while improving IF-VGG-KL by about 0.2–0.3 mAP points.These comparisons support the reported usefulness of KL loss and label-distribution learning in the image-based setting.

D. Semantic segmentation

DLDL adapts label-distribution learning to semantic segmentation while retaining the FCN training images and model structure. It improves mean IU over FCN-8s, and an optional CRF further refines segmentation boundaries.

  • Dataset and metric: Semantic segmentation is evaluated on VOC2011 and VOC2012 using mean IU, the reported intersection-over-union metric.Training uses the union of SBD and VOC2011 training images.
  • Method: DLDL keeps FCN’s training images and model structure but replaces the training loss with KL divergence over label distributions.Evaluation still relies on ground-truth labels rather than label distributions.
  • Quantitative results: DLDL-8s raises VOC2011 mean IU from 62.7% for FCN-8s to 64.9%.On VOC2012, DLDL-8s improves mean IU by 2.3 points over FCN-8s.
  • Qualitative results: DLDL-8s improves small-object and object-boundary segmentation in examples, including cars, bicycles, horse legs, and plant leaves.The method can still misclassify objects, such as identifying a flowerpot as a potted plant.
  • CRF refinement: DLDL-8s-CRF refines coarse pixel predictions into sharper boundaries and finer-grained segmentations than DLDL-8s.The refinement is illustrated for structures such as plant leaves.

V. DISCUSSIONS

The discussion examines how DLDL learns features and label distributions, reducing over-fitting and accelerating training on small, ambiguous datasets. It also studies robustness and the effect of the label-distribution parameter σ.

  • Reduce over-fitting: DLDL uses label ambiguity to reduce over-fitting when training data are limited.Its label distributions contain multiple nonzero elements, increasing label diversity during learning.
  • Reduce over-fitting: DLDL augments data on the label side by maximizing likelihood for ambiguous labels alongside the traditional ConvNet loss.The objective combines the traditional loss with a term for ambiguous labels.
  • Accelerate convergence: DLDL’s MAE decreases quickly, whereas C-ConvNet and R-ConvNet often reduce training MAE slowly at the beginning.
  • Robust performance: DLDL is more robust than C-ConvNet and R-ConvNet, especially for sparse joint pitch+yaw estimation.R-ConvNet’s errors are roughly 20 times higher than those of DLDL and C-ConvNet in the cited case, while C-ConvNet can over-fit with insufficient data.
  • Analyze the hyper-parameter: The optimal label-distribution parameter σ depends on the task, with a value near the interval between neighboring labels generally performing well.The MAE curves are V-shaped, making cross-validation convenient; the experiments use σ = 2 for Morph and σ = 15° for Pointing’04.

VI. CONCLUSION

DLDL addresses learning with insufficient data and ambiguous labels by representing each image with a label distribution. The paper reports robust performance across visual recognition tasks, while noting that constructing suitable distributions for diverse label spaces remains challenging.

  • DLDL represents each image with a label distribution to exploit label ambiguity in feature and classifier learning.
  • DLDL prevents over-fitting with small training sets and produces robust, competitive performance across several visual recognition tasks.
  • Constructing a reasonable label distribution remains challenging because recognition tasks have diverse label spaces.The paper identifies extending DLDL to more recognition problems through task-specific distributions as an open direction.
Loading 1611.01731v2…