PaperlayerView exact revision on arXiv

Source-linked AI summary

SAMM Long Videos: A Spontaneous Facial Micro- and Macro-Expressions Dataset

Chuin Hong Yap, Connah Kendrick, Moi Hoon Yap

arXiv:1911.01519v2cs.CV

TL;DR

FACS-coded long videos containing micro- and macro-expressions are limited, although the expression types may coexist or occur separately, making spotting challenging. This paper releases SAMM Long Videos, evaluates OpenFace-based AU detection and expression spotting, and reports that its spotting method outperforms the MEGC III baseline across precision, recall, and F1-score.

  • Problem

    FACS-coded long videos with micro- and macro-expressions are limited, despite the challenge of spotting expressions that may coexist or occur separately.

    Source: §I · p. 1

  • Method

    The paper releases SAMM Long Videos with FACS-coded micro- and macro-expressions and evaluates OpenFace AU detection and facial-expression spotting.

    Source: §I · p. 1, §V · p. 5

  • Results

    The spotting method outperformed the MEGC III baseline in precision, recall, and F1-score, with macro-expression spotting yielding the highest F1-score.

    Source: §V · p. 5, §IV · p. 5

  • Takeaways & Limitations

    SAMM Long Videos is released for research use in facial-expression recognition and spotting.

    Source: §I · p. 1

    OpenFace was not designed to detect micro-expressions and had lower accuracy on micro-only than macro-only videos (0.2903 versus 0.4502).

    Source: §V · p. 5

Abstract

from arXiv · show

With the growth of popularity of facial micro-expressions in recent years, the demand for long videos with micro- and macro-expressions remains high. Extended from SAMM, a micro-expressions dataset released in 2016, this paper presents SAMM Long Videos dataset for spontaneous micro- and macro-expressions recognition and spotting. SAMM Long Videos dataset consists of 147 long videos with 343 macro-expressions and 159 micro-expressions. The dataset is FACS-coded with detailed Action Units (AUs). We compare our dataset with Chinese Academy of Sciences Macro-Expressions and Micro-Expressions (CAS(ME)2) dataset, which is the only available fully annotated dataset with micro- and macro-expressions. Furthermore, we preprocess the long videos using OpenFace, which includes face alignment and detection of facial AUs. We conduct facial expression spotting using this dataset and compare it with the baseline of MEGC III. Our spotting method outperformed the baseline result with F1-score of 0.3299.

I. INTRODUCTION

Micro- and macro-expressions may coexist or occur in isolation, yet FACS-coded long-video datasets are limited. SAMM Long Videos addresses this gap alongside prior long-video resources and micro-expression spotting work.

I. INTRODUCTION

  • Micro-expressions last less than 500 ms, and their involuntary nature makes them an important cue in non-verbal communication.Facial-expression research also has applications in emotional study, behavioural psychology, security, well-being, and communication.Source: §I · p. 1
  • FACS-coded long videos with micro- and macro-expressions remain limited, although both types may coexist or occur separately, making spotting challenging.The paper presents and releases SAMM Long Videos with FACS-coded micro- and macro-expressions for research use.Source: §I · p. 1

II. RELATED WORK

  • CAS(ME)2 is described as the only fully annotated dataset with both expression types, with onset, apex, and offset annotations and labelled eye blinks.SAMM long videos released for MEGC II contained only micro-expression annotations and were made available only for that challenge.Source: §II · p. 1
  • Li et al. evaluated micro-expression spotting on SAMM and CAS(ME)2, finding Local Temporal Pattern outperformed Local Binary Pattern approaches.CAS(ME)2 included macro- and micro-expression labels, but the challenge focused on micro-expression spotting.Source: §II · p. 1

III. DATASET PROFILE AND EVALUATION

The section outlines the dataset profile, preprocessing, AU detection, movement detection, and evaluation metrics. The experiment labels all micro- and macro-movements, including movements such as eye blinks.

III. DATASET PROFILE AND EVALUATION

  • The section covers SAMM Long Videos specifications, preprocessing, OpenFace AU detection, facial-movement detection, and evaluation metrics.Source: §III · p. 1

A. Experiment

  • MEGC II used 79 SAMM long videos containing 159 micro-expressions, while macro-expression labels were not provided for those challenge videos.The micro-movement intervals were annotated from onset to offset.Source: A. Experiment · p. 1
  • All micro- and macro-movements are labelled, so spotted frames can include eye blinks as well as facial expressions.Source: A. Experiment · pp. 1–2
  • The figure contrasts a brief, low-intensity AU12 micro-expression with a higher-intensity AU12 macro-expression.The caption identifies the apex frame as the middle frame in each example.Source: Figure 1 · p. 2

B. Comparison of Facial Expressions and Dataset

SAMM Long Videos is compared with CAS(ME)2 in video characteristics and expression counts. Its release adds FACS-coded macro-expression annotations to the SAMM long-video resource.

B. Comparison of Facial Expressions and Dataset

C. Dataset Preprocessing

OpenFace preprocessing focuses on face alignment and AU detection. Landmark-based alignment maps face texture to a common reference frame, while the figure shows the original image, landmarks, and cropped face region.

C. Dataset Preprocessing

  • For this dataset, preprocessing focuses on face alignment and AU detection using OpenFace.Source: C. Dataset Preprocessing · p. 2
  • The preprocessing figure shows the original image, facial landmark detection, and cropped face ROI from left to right.Source: Figure 2 · p. 2
  • OpenFace uses a similarity transform from detected landmarks to neutral-expression landmarks, mapping face texture to a common frame and removing scale and in-plane rotation changes.The output image is 112×112 pixels with an interpupillary distance of 45 pixels.Source: C. Dataset Preprocessing · p. 2
  • OpenFace uses CE-CLM deep networks to detect and track facial landmarks; its simplified model has about 90,000 parameters and reported speed improvements.The account reports a 1.5× speedup from simplification, a 4× performance improvement from 11 orientation hypotheses, and a further 1.5× speed improvement from sparse response maps.Source: C. Dataset Preprocessing · p. 2

2) Detection of AUs:

OpenFace detects AU presence and intensity in long-video clips, with AU traces illustrating expression intervals. Its outputs use separate intensity and presence scales, and eye-blink AU45 is excluded from spotting.

2) Detection of AUs:

  • OpenFace detects facial expressions through the presence and intensity of AUs.The AU traces provide an example of this output on a long-video clip.Source: 2) Detection of AUs · pp. 2–3, Figure 3 · p. 3
  • The illustrated clip contains one micro-expression and two macro-expressions, traced using AU4 and AU7.AU4 is blue and AU7 is orange.Source: Figure 3 · p. 3
  • OpenFace represents AU intensity on a 5-point scale and AU presence as 0 (absent) or 1 (present).AU45, which detects eye blinks, is removed from the output signal because it does not carry significant information for expression classification and spotting.Source: 2) Detection of AUs · p. 3

1) Signal smoothing:

The spotting pipeline combines detected AU intensities, normalizes their sum, and smooths the resulting signal before plotting it by frame.

1) Signal smoothing:

  • Detected AU intensities are combined and normalized to a 0-to-1 scale for spotting.The resulting normalized sum is plotted with respect to frame number.Source: 1) Signal smoothing · p. 3
  • A 21st-order Savitzky-Golay filter is applied to the normalized AU-intensity signal for noise reduction.The smoothed signal is plotted alongside the normalized sum.Source: 1) Signal smoothing · p. 3
  • The plotted signal shows the normalized sum of AU intensities and its smoothed version against frame number.Source: 1) Signal smoothing · p. 3

2) Onset and offset frame acquisition:

The spotting procedure smooths the AU signal, detects peaks, and uses onset-to-offset intervals to identify and classify expressions.

2) Onset and offset frame acquisition:

  • A modified analysis algorithm uses Daubechies wavelets with scaling function 2 and level-3 decomposition to smooth the signal and obtain onset and offset frames.Source: 2) Onset and offset frame acquisition · p. 3
  • The 75th-percentile threshold filters low local peaks attributed to noise, while peaks must exceed the seven immediately preceding and following points.Figure 4 shows annotated peaks on the filtered signal.Source: 2) Onset and offset frame acquisition · p. 3, Figure 4 · p. 4
  • Intervals are classified using a 100-frame (0.5-second) cutoff at 200 fps: macro-expressions exceed 0.5 seconds and micro-expressions last 0.5 seconds or less.The spotted interval is obtained by subtracting onset from offset.Source: 2) Onset and offset frame acquisition · p. 3

E. Performance Metrics

The paper evaluates spotting with interval-overlap metrics and evaluates AU detection by comparison with ground-truth labels.

E. Performance Metrics

  • Spotting performance is measured with precision, recall, and F1-score using frame-interval overlap rather than individual-frame accuracy.The metric setup follows the spotting challenge criteria.Source: E. Performance Metrics · p. 4
  • The spotting criteria use IoU with k set to 0.5 and distinguish false positives from false negatives based on detected and ground-truth intervals.A false negative occurs when a ground-truth interval exists without a predicted interval.Source: E. Performance Metrics · p. 4
  • AU-detection accuracy is the number of correctly detected AUs divided by the total number of AUs in the ground truth.The supplied table captions identify AU-classification accuracy and a comparison between videos containing only micro- or macro-expressions, but provide no table values.Source: E. Performance Metrics · p. 4, Table III · p. 4, Table IV · p. 4

IV. RESULT AND DISCUSSION

The section evaluates OpenFace AU detection and facial-movement spotting on SAMM Long Videos. The spotting method outperformed the MEGC III baseline, while micro-expression detection and spotting were more difficult than macro-expression performance.

IV. RESULT AND DISCUSSION

  • AU classification compared OpenFace AU presence in each video against ground truth.This evaluation measures OpenFace’s ability to detect and distinguish AUs.Source: §IV · p. 4
  • OpenFace does not classify AU presence by expression class, and 84 AUs overlap between macro- and micro-expressions.The authors report that overlap contributes false positives and false negatives in the detected classes.Source: §IV · p. 4
  • Micro-expression detection performed worse than macro-expression detection in videos containing only one expression class.The comparison used 18 micro-only and 68 macro-only videos; the authors describe this as a fairer evaluation of detection accuracy.Source: §IV · p. 5
  • The spotting evaluation labelled 43 micro-expressions misclassified as macro and 9 macro-expressions misclassified as micro as false positives and false negatives.A further 93 true positives did not meet the MEGC III interval-overlap criterion; the authors report that total ground-truth expressions matched true positives plus false negatives.Source: §IV · p. 5
  • The spotting method outperformed the MEGC III baseline; macro-expression spotting had the highest F1-score in both results.The authors attribute the easier macro-expression spotting to longer intervals (> 0.5s); they note that OpenFace training data lacked labelled microexpressions.Source: §IV · p. 5, Table V · p. 5

V. CONCLUSION

The paper presents SAMM Long Videos and evaluates OpenFace for AU detection and spotting. Spotting outperformed the MEGC III baseline across precision, recall, and F1-score, while future dataset expansion is intended to improve class balance.

V. CONCLUSION

  • SAMM Long Videos is a dataset of facial micro- and macro-expressions, and the paper evaluates OpenFace facial behaviour tools for AU detection.Source: §V · p. 5
  • The spotting results outperformed the MEGC III baseline in Precision, Recall, and F1-Score.OpenFace’s overall Accuracy was 0.4712 and F1-Score was 0.3553; it performed worse in micro-only than macro-only videos, with Accuracy 0.2903 versus 0.4502.Source: §V · p. 5
  • The dataset is described as benefiting researchers in affective computing, human behavioural research, computer vision, and machine learning.Source: §V · p. 5
  • The authors identify improving class balance by adding long videos containing more microexpressions as future work.They also propose investigating more effective micro-expression elicitation methods.Source: §V · p. 5
Loading 1911.01519v2…