Source-linked AI summary
Learning Deep Representation for Face Alignment with Auxiliary Attributes
Zhanpeng Zhang, Ping Luo, Chen Change Loy, Xiaoou Tang
TL;DR
Face alignment is difficult under occlusion and pose variation, and auxiliary tasks can help but have different convergence behavior. The paper introduces TCDCN to jointly learn correlated facial attributes with adaptive task coefficients and inter-task correlations. It reports improved robustness and accuracy over existing methods while avoiding complex cascaded architectures.
Problem
Robust landmark detection is challenging under partial occlusion and large pose variation, while heterogeneous auxiliary tasks can converge differently during joint learning.
Method
TCDCN jointly learns landmark detection and correlated facial attributes using dynamic task coefficients, inter-task correlation modeling, and alternating optimization.
Results
TCDCN outperforms existing face-alignment methods, including cascaded CNNs, and is more robust to severe occlusion and large pose variation.
Takeaways & Limitations
Shared representations learned from auxiliary attributes can support robust face alignment without requiring a cascaded deep model, while the lighter CNN supports real-time performance without GPU or parallel computing.
Takeaways & Limitations
The formulation assumes variance parameters such as σm, σt, and ε equal one, so the corresponding regularization parameters are ignored.
Abstract
from arXiv · showhide
In this study, we show that landmark detection or face alignment task is not a single and independent problem. Instead, its robustness can be greatly improved with auxiliary information. Specifically, we jointly optimize landmark detection together with the recognition of heterogeneous but subtly correlated facial attributes, such as gender, expression, and appearance attributes. This is non-trivial since different attribute inference tasks have different learning difficulties and convergence rates. To address this problem, we formulate a novel tasks-constrained deep model, which not only learns the inter-task correlation but also employs dynamic task coefficients to facilitate the optimization convergence when learning multiple complex tasks. Extensive evaluations show that the proposed task-constrained learning (i) outperforms existing face alignment methods, especially in dealing with faces with severe occlusion and pose variation, and (ii) reduces model complexity drastically compared to the state-of-the-art methods based on cascaded deep model.
1 INTRODUCTION
Face alignment is treated as a multi-task problem because auxiliary facial attributes can constrain landmark estimation, while heterogeneous task convergence complicates joint learning. TCDCN addresses this with adaptive task weighting and inter-task correlation modeling, achieving stronger alignment results than existing approaches.
- Motivation: Face alignment supports face analysis tasks but remains difficult under partial occlusion and large head-pose variations.It is used for facial attribute inference, face verification, and face recognition.
- Motivation: Facial attributes such as expression and pose provide correlated information that can divide the input and solution spaces into differently distributed subsets.Examples include smiling faces revealing mouth corners and yaw rotation affecting inter-ocular distance.
- Optimization challenge: Joint learning can initially improve landmark convergence but later become harmful when an auxiliary task reaches local minima or overfits.Continuing all tasks can jeopardize network convergence and degrade landmark detection.
- Method: TCDCN adaptively weights auxiliary tasks according to training and validation errors, preventing tasks judged unhelpful from contributing to learning.The dynamic coefficient generalizes task-wise early stopping without requiring a tuned stopping threshold.
- Results: The dynamic task coefficient is essential for reaching peak face-alignment performance, and the revised objective improves performance while enabling analysis of auxiliary-task usefulness.These additions distinguish the work from its earlier version.
- Method: TCDCN models inter-task relatedness in a covariance matrix and learns network parameters, dynamic coefficients, and correlations through alternating optimization.Inter-task correlation modeling targets feature learning, while dynamic coefficients address learning convergence.
- Results: The approach outperforms deep face-alignment methods, including cascaded CNNs, and transfers sparse-landmark representations to 68-point alignment on 300-W.Further gains are reported on 300-W, Helen, and COFW.
2 RELATED WORK
Prior face-alignment work includes regression, template fitting, part-based, and cascaded deep models. TCDCN differs by learning whole-face features with auxiliary tasks without pre-partitioned faces or cascaded networks.
- Conventional methods: Traditional face-alignment methods primarily use regression-based landmark estimation or template fitting.Examples include support-vector regression, cascaded fern regression, and face-template fitting.
- Related models: Part-based models can jointly address face detection, landmark detection, and pose estimation, while other deep methods use cascaded coarse-to-fine alignment.The cited approaches include successive auto-encoder networks and cascaded processing.
- TCDCN distinction: TCDCN requires neither pre-partitioned faces nor cascaded networks, reducing model complexity while retaining comparable or better accuracy.The model extracts features automatically from the whole face image rather than handcrafted local regions.
- Multi-task learning: Auxiliary-task learning can reduce deep-model overfitting because different tasks may have local minima in different places.This provides a motivation for learning multiple tasks within one neural network.
3 LANDMARK DETECTION WITH AUXILIARY ATTRIBUTES
The framework jointly learns facial landmarks and auxiliary attributes from shared deep features, modeling task correlations and adapting auxiliary-task influence during training.
- TCDCN extracts shared high-level features from face images and jointly predicts landmark coordinates and binary facial attributes.The model uses convolutional filters for feature extraction, with landmark detection as the main task and attribute prediction as auxiliary tasks.
- The model represents task relationships through a learned covariance matrix that captures correlations among landmark and attribute weights.The matrix-normal formulation allows the covariance structure to be learned during training.
- The objective combines landmark least-squares loss, attribute cross-entropy loss, task-correlation regularization, and dynamic-coefficient terms.Parameters are optimized by alternating updates, using stochastic gradient descent for network parameters and closed-form updates for dynamic coefficients.
- Dynamic coefficients adapt each auxiliary task’s contribution according to training and validation errors as optimization proceeds.This reduces the influence of tasks that become ineffective because of local minima or over-fitting and can allow them to resume later.
- The learned features show similar patterns for faces sharing poses or expressions, and are reported as robust to large poses and expressions.The paper also reports that the dynamic task coefficient is essential for reaching peak face-alignment performance.
4 IMPLEMENTATION AND EXPERIMENTS
Experiments evaluate TCDCN’s shared representation, dynamic task weighting, inter-task correlation modeling, auxiliary-task configurations, and efficiency across several face-alignment datasets.
- Implementation: TCDCN uses a shared feature vector for multiple tasks, taking 60×60 grayscale face images through four convolutional layers, three pooling layers, and one fully connected layer.The model reports mean error and failure rate, with failures defined as mean error above 10%.
- Datasets: MAFL contains 20,000 faces annotated with 22 binary attributes and five facial landmarks; 1,000 faces are reserved for testing.The attributes are grouped by affected face region and head pose.
- Inter-task Correlation Learning: Dynamic task coefficients make training and validation landmark errors converge faster and more stably than without them, while outperforming task-wise early stopping.The comparison tracks the main-task error through 200,000 iterations.
- Inter-task Correlation Learning: Learned correlations are region-specific: mouth, nose, and eye attributes align with corresponding landmarks, whereas global attributes correlate broadly and pose correlates strongly across landmarks.Pose has higher correlation than the other groups because head rotation directly affects landmark distribution.
- Evaluating the Effectiveness of Auxiliary Task: “FLD+all” reduces failure rate by over 7% versus “FLD”, while pose and global attributes provide the strongest single-group improvements and random attributes barely help.The results indicate that auxiliary tasks need to be related to the main task for performance gains.
- Comparison with Deep Learning based Methods: TCDCN achieves approximately 100K parameters and 18ms per face, compared with cascaded CNN’s approximately 990K parameters and 120ms.The proposed model uses one CNN, whereas the cascaded implementation uses 23 CNNs; it also outperforms cascaded CNN and CFAN in overall accuracy on MAFL and AFLW.
5 CONCLUSIONS
The proposed Tasks-Constrained DCN improves face alignment by jointly learning heterogeneous, subtly correlated auxiliary tasks. It achieves greater robustness to severe occlusion and large pose variation without requiring a cascaded deep model.
- Joint learning with appearance, expression, demographic, and head-pose tasks produces more robust landmark detection.
- The method is evaluated on COFW using mean errors and example alignment results.
- Dynamic task coefficients use auxiliary tasks more efficiently during optimization.
- The model remains robust to faces with severe occlusions and large pose variations compared to existing methods.
- A lighter-weight CNN avoids cascading and supports real-time performance without GPU or parallel processing.