FERA 2017 - Addressing Head Pose in the Third Facial Expression Recognition and Analysis Challenge

I. INTRODUCTION

• Most existing systems and databases emphasize frontal faces or limited posed non-frontal expressions, while real recordings can include widely varying head poses.The paper notes that some camera angles make frontal views unrealistic and that existing multiview databases have restricted expression sets.Source: §I · p. 1
• Many benchmarks assume fixed expression intensity, and little annotated data supports intensity estimation; FERA 2015 used predominantly near-frontal views.The paper notes that intensity can be an important cue to expression meaning.Source: §I · p. 1
• Benchmarking also needs to move beyond short, single-expression clips toward unsegmented videos with multiple naturally transitioning expressions.The paper identifies video duration as an issue despite earlier efforts to evaluate videos longer than a few seconds.Source: §I · pp. 1–2
• FERA 2017 uses BP4D+ videos rendered in nine views and asks participants to detect AU occurrence and estimate intensity without knowing the facial view in advance.The intensity task is fully automatic, with AU occurrence unknown beforehand.Source: §I · p. 2

A. AU occurrence detection

• The review frames AU occurrence detection as a binary-classification problem and identifies ANN, boosting, and SVM as common classifier families.It describes SVMs as the currently most popular choice and notes boosting's feature-selection and training advantages.Source: A. AU occurrence detection · p. 2

B. AU intensity estimation

• Intensity estimation assigns each AU a per-frame label from 0 to 5, and regression models the ordinal differences between labels that classification does not capture.The review also notes that intensity's class overlap suggests an underlying continuous quantity.Source: B. AU intensity estimation · p. 2
• AU intensity can be estimated from binary-classifier confidence, as a six-class classification problem, or through regression.The cited work reports that multiclass and regression approaches detected intensity more accurately than distance-from-hyperplane measures.Source: B. AU intensity estimation · p. 2

III. DATA

• Training uses BP4D-Spontaneous, while validation and test data come from a subset of BP4D+.The challenge divides data into train, validation, and test partitions.Source: §III · p. 2
• Training and development partitions are public for system development and uniform performance reporting, while organizers retain the test partition for system evaluation.Participants submit trained systems for evaluation on the held-back test data.Source: §III · p. 2
• Occurrence detection covers 10 frequently occurring AUs, while intensity estimation covers a subset of seven AUs.AUs were selected based on frequency and sufficiently high inter-rater reliability.Source: §III · p. 2

A. Multiview Face Synthesis

• Figure 1 shows the different views considered for the challenge, illustrating the multiview orientations.The orientations are synthesized from frontal pose using pitch and yaw rotations.Source: Figure 1 · p. 3, A. Multiview Face Synthesis · p. 3
• Table I provides an overview of the AUs included in the occurrence-detection and intensity-estimation sub-challenges.Source: Table I · p. 3
• Nine views are generated by rotating 3D sequences in BP4D and BP4D+ to different face orientations.The rotations use combinations of pitch and yaw relative to frontal pose.Source: A. Multiview Face Synthesis · pp. 2–3

B. BP4D Database

• Train, validation, and test participants completed similar emotion-elicitation and video-recording procedures, but the datasets differ in collection details and AU annotation thresholds.The training set comes from BP4D-Original; validation and test are from BP4D+.Source: B. BP4D Database · p. 3
• BP4D-Train uses BP4D-Original, while validation and test use BP4D-Expanded (BP4D+); training data was collected first and is publicly available.BP4D+ includes 2D, 3D, thermal, and peripheral-physiology data, but the challenge does not include thermal or physiological data.Source: B. BP4D Database · p. 3
• BP4D-Train contains 41 participants, BP4D-Validation 20, and BP4D-Test 30; the partitions comprise 328, 159, and 120 sessions, respectively.Across nine views per subject, they contain 2,952 training, 1,431 development, and 1,080 test videos.Source: B. BP4D Database · p. 3

C. Action Unit Annotation

• Experts annotated AU occurrence and intensity frame by frame using FACS, which decomposes expressions into anatomically based units corresponding to facial muscles or muscle groups.Action units individually or in combinations can describe nearly all possible facial expressions.Source: C. Action Unit Annotation · p. 3
• For BP4D-Train, coders marked occurrence onset at intensity level A and offset below A; coded segments covered the most expressive 20 seconds of each task.Occurrence agreement averaged 0.91 MCC, ranging from 0.81 for AU 23 to 0.96 for AU 2.Source: C. Action Unit Annotation · pp. 3–4
• 0.79 average MCC for occurrence in BP4D-Validation and BP4D-Test, versus 0.91 in BP4D-Train; agreement was lower for every AU except AU 15 in the expanded dataset.The authors suggest the B-level rather than A-level occurrence threshold and the addition of two coders may partly explain the difference.Source: C. Action Unit Annotation · p. 4
• BP4D-Original intensity coding covered seven AUs, with average inter-coder ICC of 0.85 and individual-AU values from 0.79 to 0.92.B- and C-level intensity were most common except for AU1, AU4, and AU17, which had more A- than C-level intensity.Source: C. Action Unit Annotation · p. 4

IV. EVALUATION PROCEDURE

• Participants submit working programs for organizer-run evaluation on the held-back test set to support fair evaluation.Occurrence is measured with F1, while intensity uses ICC.Source: §IV · p. 4
• Each test video is evaluated without prior view information, and the nine views for each user and task share the same annotations.Participants are asked to treat views as independent videos.Source: §IV · p. 4
• The sub-challenge score averages per-AU F1 for occurrence or ICC for intensity after concatenating test-sequence labels.The intensity measures also include RMSE and PCC; the single score uses 10 occurrence AUs or 7 intensity AUs.Source: §IV · p. 4
• Occurrence uses F1, accuracy, and 2AFC for baseline analysis; 2AFC approximates ROC AUC and, unlike F1, accounts for true negatives.The challenge performance measure for occurrence is F1, the harmonic mean of precision and recall.Source: §IV · p. 4

V. BASELINE SYSTEM

• Baseline recognition results are provided for both development and test sets to support comparison with participant systems.The passage does not report the baseline scores.Source: §V · p. 5
• Baseline points and features are not made publicly available to participants.Participants submit standalone applications for evaluation.Source: §V · p. 5
• Submitted standalone applications must address the challenge tasks associated with the different views.Source: §V · p. 5

A. Baseline Features

• Setting rigid parameters and the pose-related parameter to zero leaves 19 non-rigid expression parameters as a low-dimensional, pose-normalised shape representation.The paper illustrates that the main expression remains after pose removal.Source: A. Baseline Features · p. 5, Figure 2 · p. 5
• For the left eye, right eye, and mouth regions, the baseline computes within-group distances and angles, yielding 71 features.The three groups comprise specified eye and eyebrow landmarks and the mouth landmarks.Source: A. Baseline Features · p. 5
• The feature vector adds 19 frame-to-frame parameter differences and 49 distances from aligned landmarks to a median of stable points, for 158 features per frame.For the first frame of each session, the difference features repeat the first 19 parameters.Source: A. Baseline Features · p. 5

B. Baseline Results

• The baseline uses a CRF for AU occurrence and a CORF for intensity, with 90-frame windows evaluated at a stride of 30 frames.Overlapping-window predictions are combined per frame by selecting the prediction with the maximum likelihood.Source: B. Baseline Results · pp. 5–6
• View 4 occurrence results were close to those for frontal views 5 and 6, while extreme views were harder to track and affected performance.View 4 was not used to train the models; the paper also reports intensity results above chance across views.Source: Table III · p. 6, Table IV · p. 7, B. Baseline Results · p. 8
• Training uses 90-frame segments with a 30-frame stride and frontal-view features from views 5 and 6; per-AU training samples are balanced when possible.The feature vector has 158 dimensions before Correlation Feature Selection reduces it.Source: B. Baseline Results · p. 6
• Occurrence performance is reported with F1, Accuracy, and 2AFC, while intensity uses ICC, RMSE, and PCC; participants are judged only on F1 and ICC.The tables report development and test results, with additional per-view results for both sub-challenges.Source: B. Baseline Results · pp. 6, 8, Table II · p. 6, Table V · p. 7

• The geometric-feature baseline leaves substantial room for improvement, especially on challenging views.Source: §VI · p. 8
• FERA 2017 extends BP4D into videos comprising nine views for FACS AU detection and intensity estimation.Source: §VI · p. 8
• The dataset is the first FACS AU-annotated dataset focused on expression analysis across different camera views, including extreme poses.Source: §VI · p. 8
• The challenge targets intensity estimation and robust AU detection under non-frontal poses and partial self-occlusions.Source: §VI · p. 8