PaperlayerView exact revision on arXiv

Source-linked AI summary

Heterogeneous Face Attribute Estimation: A Deep Multi-Task Learning Approach

Hu Han, Anil K. Jain, Fang Wang, Shiguang Shan, Xilin Chen

arXiv:1706.00906v3cs.CV

TL;DR

Face-attribute methods often overlooked the joint effects of attribute correlation and heterogeneity. The paper introduces an end-to-end DMTL CNN with shared and category-specific feature learning, evaluates it across multiple benchmarks, and reports superior performance with cross-database generalization analysis.

  • Problem

    Most prior methods did not explicitly model attribute correlation and heterogeneity, including ordinal versus nominal and holistic versus local attributes, during feature representation learning.

  • Method

    DMTL jointly estimates multiple heterogeneous attributes using shared CNN feature learning followed by category-specific subnetworks.

  • Results

    The proposed approach outperforms state-of-the-art methods across multiple face-attribute benchmarks, including MORPH II, LFW+, CelebA, LFWA, and FotW.

  • Takeaways & Limitations

    The approach supports joint estimation across heterogeneous and single-attribute scenarios while retaining reported generalization ability under cross-database testing.

  • Takeaways & Limitations

    Cross-database performance drops because image distributions, including demographics and capture conditions, differ between databases.

Abstract

from arXiv · show

Face attribute estimation has many potential applications in video surveillance, face retrieval, and social media. While a number of methods have been proposed for face attribute estimation, most of them did not explicitly consider the attribute correlation and heterogeneity (e.g., ordinal vs. nominal and holistic vs. local) during feature representation learning. In this paper, we present a Deep Multi-Task Learning (DMTL) approach to jointly estimate multiple heterogeneous attributes from a single face image. In DMTL, we tackle attribute correlation and heterogeneity with convolutional neural networks (CNNs) consisting of shared feature learning for all the attributes, and category-specific feature learning for heterogeneous attributes. We also introduce an unconstrained face database (LFW+), an extension of public-domain LFW, with heterogeneous demographic attributes (age, gender, and race) obtained via crowdsourcing. Experimental results on benchmarks with multiple face attributes (MORPH II, LFW+, CelebA, LFWA, and FotW) show that the proposed approach has superior performance compared to state of the art. Finally, evaluations on a public-domain face database (LAP) with a single attribute show that the proposed approach has excellent generalization ability.

1 INTRODUCTION

Face attributes are correlated yet heterogeneous, motivating an end-to-end DMTL CNN that combines shared and category-specific feature learning and is evaluated across diverse databases.

  • Face attribute estimation supports applications including video surveillance, face retrieval, and social media.
  • Prior work often estimated a single attribute or learned separate models, while joint methods frequently reused identical features despite attribute heterogeneity.
  • Attributes differ in data type, scale, and semantic meaning: age and hair length are ordinal, whereas gender and race are nominal.
  • DMTL jointly estimates multiple heterogeneous attributes using shared feature learning followed by category-specific feature learning in one CNN.
  • The paper contributes an efficient joint-learning method, models correlation and heterogeneity, studies cross-database generalization, and compiles the crowdsourced LFW+ database.
  • The work extends preliminary versions with broader category-specific modeling, implementation details, six-database evaluations, and additional state-of-the-art comparisons.

2 RELATED WORK

Prior face-attribute research spans single-task and multi-task models, but the proposed approach distinguishes itself through end-to-end joint estimation that explicitly handles attribute heterogeneity.

  • Earlier face-attribute methods commonly used handcrafted features, while recent deep models learn compact and discriminative representations.
  • Published methods divide into single-task per-attribute classifiers and multi-task joint classifiers using shared representations.
  • The proposed approach targets joint estimation of multiple attributes from a face image rather than alignment, pose, or general human-attribute tasks.
  • Unlike auxiliary-task MTL, it aims to improve all face-attribute estimates by exploiting correlations while handling heterogeneities.
  • Unlike two-step CNN-feature pipelines followed by attribute classifiers, the proposed DMTL performs end-to-end learning.
  • The evaluation covers heterogeneous attributes, single-attribute estimation, and cross-database testing as practical scenarios.

3 PROPOSED APPROACH

The proposed DMTL jointly estimates correlated, heterogeneous face attributes by combining shared feature learning with category-specific subnetworks and losses. It models differences such as ordinal versus nominal and holistic versus local attributes within one CNN.

  • 3.1 Deep Multi-task Learning: Strong attribute correlations motivate joint estimation, while nonlinear mappings and heterogeneous attributes require CNN-based multi-task feature learning.CelebA examples include positive correlations between 5 O’Clock Shadow and Male, and Heavy Makeup and Wear Lipstick.
  • 3.1 Deep Multi-task Learning: The DMTL objective shares a deep representation across attributes and uses attribute-specific subnetworks to refine predictions.The shared network is controlled by Wc, while Wj controls updates relative to each attribute.
  • 3.2 Heterogeneous Face Attribute Estimation: Category-specific modeling separates heterogeneous attributes into groups whose members share feature learning and classification models.The framework considers data type and scale, including ordinal versus nominal attributes, and semantic meaning, including holistic versus local attributes.
  • 3.2 Heterogeneous Face Attribute Estimation: Ordinal attributes use regression with Euclidean loss, whereas nominal attributes use classification with cross-entropy loss.Age is treated as ordinal, while race is described as nominal without intrinsic ordering.
  • 3.2 Heterogeneous Face Attribute Estimation: Holistic attributes such as age, gender, and race can use different features from local attributes such as pointy nose and big lips.The proposed DMTL can model this distinction with multiple holistic and per-component attribute subnetworks.
  • 3.3 Implementation Details: The implementation uses a shared modified AlexNet-style network, category-specific fully connected subnetworks, and two-field labels containing attribute value and category.The category field allows input-attribute order to become irrelevant and determines whether cross-entropy or Euclidean loss is used.

4 EXPERIMENTAL RESULTS

Across heterogeneous and binary attribute benchmarks, the proposed DMTL approach generally outperforms or matches state-of-the-art methods, while also showing strong single-attribute generalization and efficient inference. Results indicate benefits from modeling attribute correlations and heterogeneity, although dataset imbalance, annotation inconsistency, and distribution shifts constrain performance.

  • Nominal and Ordinal Face Attributes: 3.0 years MAE on MORPH II age estimation outperformed multi-label regression baselines reporting 3.92 and 3.63 years MAE.Gender and race performance was comparable with those methods; category-specific subnetworks fine-tune shared features for individual attributes.
  • Nominal and Ordinal Face Attributes: 75.0% @ 5-year AE, 96.7% gender accuracy, and 94.9% race accuracy were achieved on the larger unconstrained LFW+ database.DIF reported 42.5%, 94%, and 90% on a smaller frontal LFW subset for the corresponding tasks.
  • Limitations and Error Analysis: Small numbers of young and old subjects in MORPH II and LFW+ made age and race estimation difficult despite robustness to pose and illumination variations.Some LAPAge2015 errors were attributed to overexposure-related loss of facial detail, and inconsistent CelebA labels also caused poor estimates.
  • Binary Face Attributes: The proposed approach outperformed several methods for most of CelebA and LFWA’s 40 binary attributes and generally exceeded separate-task learning.On LFWA, performance was similar to MCNN-AUX, while the larger CelebA training set improved both methods and favored the proposed method.
  • Single Face Attribute: 0.289 ϵ-error on LAPAge2015 after IMDB-WIKI pretraining was comparable to DEX’s best reported 0.265 ϵ-error.Training from scratch achieved 0.449 ϵ-error and 5.2 years MAE, comparable to the eighth-best challenge method.
  • Computational Cost: 8ms on a Titan X GPU and 35ms on an Intel Core I7 CPU enabled a prototype to run at about 16 fps on a commodity desktop.The approach was faster than most compared methods with reported GPU costs and about ten times faster than a single VGG-16 model used by DEX.

5 CONCLUSIONS

The paper presents a single-network deep multi-task approach for heterogeneous face attribute estimation and introduces LFW+ for broader evaluation. Its computational cost is profiled across multiple hardware platforms.

  • Conclusions: DMTL jointly estimates multiple face attributes while modeling attribute correlation and heterogeneity in a single network.The approach combines shared feature learning for all attributes with category-specific learning for heterogeneous attribute categories.
  • Conclusions: The computational costs of face attribute estimation methods are profiled on Tesla K20, Titan X, and Intel CPUs.The profiling includes Tesla K20 GPU, Titan X GPU, Intel Core2, Intel Core i3, and Intel Core i7 CPUs.
  • Conclusions: LFW+ extends LFW with 2,466 images of subjects aged 0–20 years to evaluate the approach over a wider age range.The database contains faces in the wild and heterogeneous demographic attributes obtained via crowdsourcing.
Loading 1706.00906v3…