PaperlayerView exact revision on arXiv

Source-linked AI summary

Peak-Piloted Deep Network for Facial Expression Recognition

Xiangyun Zhao, Xiaodan Liang, Luoqi Liu, Teng Li, Yugang Han, Nuno Vasconcelos, Shuicheng Yan

arXiv:1607.06997v2cs.CV

TL;DR

FER methods often learn samples independently despite strong relationships between easy peak and difficult non-peak expressions. PPDN embeds non-peak-to-peak evolution through paired feature supervision and PGS, achieving superior FER results and state-of-the-art pose-invariant face recognition performance.

  • Problem

    FER methods commonly ignore correlations between peak and non-peak samples, although non-peak expressions are common and difficult to distinguish.

  • Method

    PPDN uses peak-expression features to supervise corresponding non-peak features, jointly optimizing feature transformation and expression-recognition losses with directional PGS updates.

  • Results

    PPDN outperforms previous methods on CK+ and Oulu-CASIA and achieves state-of-the-art robust face recognition performance on Multi-PIE.

  • Takeaways & Limitations

    PPDN provides intensity-invariant FER and generalizes to other tasks through appropriate definitions of peak and non-peak samples.

Abstract

from arXiv · show

Objective functions for training of deep networks for face-related recognition tasks, such as facial expression recognition (FER), usually consider each sample independently. In this work, we present a novel peak-piloted deep network (PPDN) that uses a sample with peak expression (easy sample) to supervise the intermediate feature responses for a sample of non-peak expression (hard sample) of the same type and from the same subject. The expression evolving process from non-peak expression to peak expression can thus be implicitly embedded in the network to achieve the invariance to expression intensities. A special purpose back-propagation procedure, peak gradient suppression (PGS), is proposed for network training. It drives the intermediate-layer feature responses of non-peak expression samples towards those of the corresponding peak expression samples, while avoiding the inverse. This avoids degrading the recognition capability for samples of peak expression due to interference from their non-peak expression counterparts. Extensive comparisons on two popular FER datasets, Oulu-CASIA and CK+, demonstrate the superiority of the PPDN over state-ofthe-art FER methods, as well as the advantages of both the network structure and the optimization strategy. Moreover, it is shown that PPDN is a general architecture, extensible to other tasks by proper definition of peak and non-peak samples. This is validated by experiments that show state-of-the-art performance on pose-invariant face recognition, using the Multi-PIE dataset.

1 Introduction

FER remains difficult because expression intensity and appearance vary, especially in common non-peak samples. PPDN uses corresponding peak expressions to supervise non-peak features and learn intensity-invariant recognition.

  • Motivation: FER must handle diverse expression intensities, subject-specific appearance changes, and subtle non-peak expressions that are difficult to distinguish.Non-peak fear and sadness can appear similar, while non-peak expressions are more common than peak expressions.
  • Motivation: Most existing methods learn from samples independently, overlooking correlations between easy peak and hard non-peak examples.They also focus on clearly separable peak expressions while neglecting challenging non-peak samples.
  • PPDN: PPDN uses peak-expression feature responses to supervise corresponding non-peak samples from the same subject and expression type.This implicitly embeds expression evolution from non-peak to peak during learning.
  • PPDN: The resulting non-peak-to-peak feature mapping magnifies critical subtle details and supports recognition invariant to expression intensity.The approach operates on high-level feature representations rather than explicitly predicting detailed facial changes.
  • Optimization: PPDN jointly learns peak-piloted feature transformation and expression recognition, with PGS preventing the reverse transformation during back-propagation.The paired training procedure uses L2-norm feature losses alongside recognition losses.

2 Related Work

Prior FER research includes sequence-based methods using appearance and motion, and still-image methods using appearance patterns. CNN-based approaches commonly process different-intensity expressions independently.

  • Sequence-based methods: Sequence-based FER methods exploit both appearance and motion information from video sequences.
  • Still-image methods: Still-image FER methods recognize expressions from image appearance patterns without using sequence motion.
  • CNN-based methods: CNN-based FER methods have shown strong performance but treat expression instances of different intensities from the same subject independently.

3 The Peak-Piloted Deep Network (PPDN)

PPDN trains on paired peak and non-peak expressions, combining feature transformation with recognition. Its PGS optimization moves non-peak features toward peak features without the inverse update.

  • Architecture: PPDN takes paired peak and non-peak images of the same expression type and subject as training input.The pair passes through convolutional and fully connected layers to produce feature maps for both images.
  • Network optimization: The network minimizes the L2-norm difference between non-peak and peak feature maps while jointly optimizing recognition losses.The loss includes one feature-transformation term and two cross-entropy terms, one for each expression image.
  • Peak Gradient Suppression: PPDN uses a special-purpose PGS back-propagation procedure to drive non-peak responses toward corresponding peak responses.This directional transformation is integrated with facial expression recognition in one architecture.
  • Peak Gradient Suppression: PGS suppresses gradients arising from peak-expression feature responses, avoiding the contrary movement toward non-peak features.The resulting update is a descent direction for the overall loss, though not necessarily the steepest descent direction.

4 Experiments

Experiments evaluate PPDN on CK+, Oulu-CASIA, and Multi-PIE, using GoogLeNet as its basic structure and subject-independent cross-validation. Comparisons include state-of-the-art methods and GoogLeNet-based baselines.

  • Experimental scope: PPDN is evaluated on CK+ and Oulu-CASIA for FER and on Multi-PIE for face recognition.The Multi-PIE experiment tests whether the architecture generalizes beyond FER.
  • Network configuration: GoogLeNet serves as the basic network structure, with peak-piloted transformation applied in the last two fully connected layers.Alternative placements on convolutional layers are also reported.
  • Testing and evaluation: Testing uses one image as input and predicts its facial expression label under 10-fold subject-independent cross-validation.This testing protocol is stated for all experiments.
  • FER comparisons: Table 1 compares average CK+ classification accuracy with four state-of-the-art methods and a GoogLeNet baseline.The comparison uses six facial expressions and the 10-fold cross-validation evaluation.
  • FER comparisons: Table 2 compares average classification accuracy on Oulu-CASIA with UDCS and a GoogLeNet baseline under the same setting as UDCS.

Datasets.

The experiments use CK+ and Oulu-CASIA, whose video sequences capture expressions evolving toward peak frames. Standard comparisons evaluate nearly peak frames, with PPDN pairing non-peak and peak frames during input.

  • Datasets: CK+ contains 593 sequences from 123 subjects, with six basic and one non-basic facial expression.Only 327 sequences are annotated with expression labels.
  • Datasets: Oulu-CASIA contains 480 sequences of six facial expressions under normal illumination.
  • Standard evaluation: Standard CK+ evaluation uses only the last one to three frames per sequence, representing nearly peak expressions.PPDN is evaluated on the last three frames in the comparison described.

Training and Testing with More Non-peak Expressions.

Additional experiments test PPDN with more non-peak frames, compare it with sequence-based approaches, and examine architectural variants and pose-invariant face recognition. The reported results cover weak, peak, combined, and no-peak-information settings.

  • More non-peak expressions: Weak-expression tests use frames 7 to 9, while peak-expression tests use the last frame and combined tests use frames 7 through the last.Training includes all frames beyond the seventh in addition to standard-setting images.
  • Still-image comparisons: On weak-expression tests, PPDN reaches 83.36% and 67.95% versus GoogLeNet baseline values of 78.10% and 64.64% on CK+ and Oulu-CASIA, respectively.PPDN also outperforms PPDN trained with standard SGD and GoogLeNet on combined test sets.
  • Sequence-based comparisons: For sequence recognition, PPDN pairs a non-peak frame with a peak frame and predicts the peak-frame label as the sequence prediction.Unlike prior sequence-based approaches, it relies only on appearance information rather than motion during inference.
  • Sequence-based comparisons: PPDN gains 5.1% and 2% over STM-ExpLet and DTAGN(Joint) on CK+, respectively, and achieves 84.59% on Oulu-CASIA versus 75.52% for Atlases and 81.46% for DTAGN(Joint).
  • No peak information: Without peak information, PPDN achieves 99.2% on CK+ and 83.67% on Oulu-CASIA.Testing selects the image with the highest classification scores for all categories as the peak frame.
  • Additional evaluations: Additional evaluations report peak-piloted feature transformation across different convolutional layers and face recognition rates under two Multi-PIE pose settings.

PGS vs. standard SGD.

PGS suppresses gradients from peak expressions so non-peak features move toward corresponding peak features without the reverse update. It outperforms standard SGD across all three test sets.

  • Optimization comparison: PGS outperforms standard SGD on the weak-expression, peak-expression, and combined test sets.Standard SGD uses gradients from both non-peak and peak samples, whereas PGS suppresses peak-expression gradients.

Ablative Studies on Peak-Piloted Feature Transformation.

Ablations show that peak-piloted supervision is most effective when applied to highly semantic feature representations, while the method also supports training when peak frames must be estimated.

  • Layer placement: Applying peak-piloted transformation only to both fully connected layers achieves the best performance.The comparison evaluates supervision at inception layers and fully connected layers in GoogLeNet.
  • Layer placement: Adding losses to all inception layers produces roughly the same performance as supervising both fully connected layers.
  • Layer placement: Removing either fully connected-layer loss decreases performance by more than 1%.
  • Layer placement: The results indicate stronger usefulness for supervising highly semantic fully connected features than early convolutional features.Early layers capture fine-grained facial details, whereas later layers capture semantic mouth and eye patterns.
  • Unknown peak frames: PPDN can be trained when the peak frame is unknown by selecting, within each sequence, the frame with the highest cross-validated prediction score.A basic GoogLeNet model is trained with 10-fold cross validation to generate the prediction scores.

4.2 Generalization Ability of the PPDN

The PPDN generalizes the easy-to-hard sample framework from expression intensity to pose variation in face recognition, improving performance especially for difficult poses and across-pose averages on Multi-PIE.

  • Task generalization: PPDN treats near-frontal faces as easy examples and profile faces as hard samples for learning pose-invariant features.The analogy extends peak and non-peak expressions to face-recognition pose variation.
  • Experimental setup: The experiments use session 1 images from 249 subjects under two training and testing settings.
  • Results: 97.98% vs 95.99% under “setting 1” and 83.22% vs 74.84% under “setting 2” for average performance over all poses.These comparisons are between PPDN and “GoogLeNet (baseline)” on Multi-PIE.
  • Results: PPDN improves substantially over “GoogLeNet (baseline)” on hard poses −45° and 45° in both evaluated settings.
  • Results: PPDN beats all baselines by 2.52% under “setting 2”.

5 Conclusions

The conclusion presents PPDN as a network that embeds expression evolution from non-peak to peak and jointly trains feature transformation with expression recognition. PGS directs non-peak features toward peak features without the inverse direction.

  • Conclusions: PPDN embeds the evolution from non-peak to peak expressions into network parameters.
  • Conclusions: PPDN jointly optimizes an L2-norm peak-piloted feature-transformation loss and cross-entropy expression-recognition losses.
  • Conclusions: PGS drives intermediate-layer non-peak features toward peak-expression features while avoiding the inverse.

Appendix

The appendix argues that PGS remains a valid descent direction for the total loss under a gradient-similarity condition, although it is not generally the steepest descent direction.

  • Descent condition: The relevant gradient dot product is positive when the gradient-size condition is satisfied.The appendix expresses the dot product using the gradient norms and the cosine of the angle between A and B.
  • Descent condition: When the dot product is positive, the modified gradient is a descent direction for J1, though not a steepest descent direction.
  • Total loss: PGS is a descent direction for the total loss, and gradients from J2 and J3 can preserve this even if the stated condition is violated.This can occur when the J2 and J3 gradients are dominant.
  • Convergence: The appendix states that PGS is therefore likely to converge to a minimum of the loss.
Loading 1607.06997v2…