PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Reading Hidden Emotions: A comparative Study of Spontaneous Micro-expression Spotting and Recognition Methods

Xiaobai Li, Xiaopeng Hong, Antti Moilanen, Xiaohua Huang, Tomas Pfister, Guoying Zhao, Matti Pietikäinen

arXiv:1511.00423v2cs.CV

TL;DR

Spontaneous micro-expressions are difficult to analyze because they are brief, subtle, and often mixed with other facial motion, while prior spotting work mainly used posed videos. The paper introduces training-free spontaneous spotting based on feature-difference contrast and an enhanced recognition framework, reporting state-of-the-art results on SMIC and CASMEII and a complete spotting-recognition system.

  • Problem

    Spontaneous ME analysis is limited by short duration, low intensity, scarce databases, and interference from ordinary facial motion.

  • Method

    The paper combines training-free feature-difference spotting for spontaneous long videos with recognition using motion magnification and LBP, HOG, and HIGO descriptors.

  • Results

    The method outperforms prior work on SMIC and CASMEII recognition, while LBP spotting reaches 84.53% AUC on SMIC-E-VIS and 92.98% AUC on CASMEII.

  • Takeaways & Limitations

    MESR provides an automatic system for spotting and recognizing spontaneous MEs, with combined performance comparable to humans.

  • Takeaways & Limitations

    Spotting produces many false positives from eye blinks and gaze changes, and recognition performance on NIR data is unsatisfactory.

Abstract

from arXiv · show

Micro-expressions (MEs) are rapid, involuntary facial expressions which reveal emotions that people do not intend to show. Studying MEs is valuable as recognizing them has many important applications, particularly in forensic science and psychotherapy. However, analyzing spontaneous MEs is very challenging due to their short duration and low intensity. Automatic ME analysis includes two tasks: ME spotting and ME recognition. For ME spotting, previous studies have focused on posed rather than spontaneous videos. For ME recognition, the performance of previous studies is low. To address these challenges, we make the following contributions: (i)We propose the first method for spotting spontaneous MEs in long videos (by exploiting feature difference contrast). This method is training free and works on arbitrary unseen videos. (ii)We present an advanced ME recognition framework, which outperforms previous work by a large margin on two challenging spontaneous ME databases (SMIC and CASMEII). (iii)We propose the first automatic ME analysis system (MESR), which can spot and recognize MEs from spontaneous video data. Finally, we show our method outperforms humans in the ME recognition task by a large margin, and achieves comparable performance to humans at the very challenging task of spotting and then recognizing spontaneous MEs.

1 INTRODUCTION

Spontaneous micro-expressions are brief, involuntary, low-intensity facial signals with potential applications including lie detection and psychotherapy. Automatic analysis is difficult because spontaneous data are scarce and MEs must be both spotted in video context and recognized despite subtle, rapid movements.

  • Motivation: Micro-expressions are brief, involuntary facial expressions that can reveal concealed emotions.Their duration is approximately 1/25 to 1/3 second, and their muscle movements are subtle.
  • Motivation: MEs are relevant to lie detection and may help understand patients’ genuine emotions in psychotherapy.The paper also mentions possible future use in border control, but presents these as potential applications.
  • Challenges: Untrained people perform only slightly above chance on ME recognition, while training specialists is time-consuming and expensive.Special training improves performance, but it remains below the reported efficient level.
  • Challenges: Automatic ME analysis is hindered by limited spontaneous databases, short durations, low intensity, and interference from head movements and eye blinks.The two core tasks are spotting when an ME occurs and recognizing which expression it represents.
  • Contributions: The paper proposes a training-free spontaneous-ME spotting method and an advanced recognition framework using motion magnification and multiple feature descriptors.The stated contributions include a complete system for spotting and recognizing spontaneous MEs.

2 RELATED WORK

Prior ME research was constrained by scarce spontaneous databases and heavy reliance on posed or synthetic data. This paper focuses on spontaneous datasets, especially SMIC and CASMEII, while distinguishing spotting from recognition.

  • 2.1 ME databases: ME research has fewer available databases than conventional facial-expression research, making adequate training data difficult to obtain.The paper focuses on spontaneous MEs and uses SMIC and CASMEII as comprehensive publicly available databases.
  • 2.1 ME databases: Spontaneous ME databases are elicited under high-stake paradigms designed to induce participants to conceal emotional reactions.The SMIC protocol used emotion-inducing videos, neutral instructions, monitoring, and a stated punishment for visible emotion leaks.
  • 2.2 State of the art of ME spotting research: Posed or synthetic ME data do not adequately represent spontaneous expressions because their spatial and temporal properties differ.Prior spotting studies using posed data often had cleaner head motion and more abrupt or controlled expression transitions.
  • 2.1 ME databases: SMIC contains HS, VIS, and NIR recordings, while CASME and CASMEII provide spontaneous samples with camera, action-unit, and emotion-label information.The full SMIC version contains 164 MEs from 16 subjects; CASMEII contains 247 samples from 26 subjects.
  • 2.2 State of the art of ME spotting research: ME spotting detects the temporal interval of an expression, whereas ME recognition classifies a spotted interval into expression classes.Earlier spontaneous recognition work combined temporal interpolation, LBP-TOP features, and multiple-kernel learning, achieving 71.4% accuracy on an early SMIC version.

3 METHOD FOR ME SPOTTING

The proposed spotting method aligns faces, extracts appearance or optical-flow features, and uses feature-difference contrast over temporal windows to identify rapid movements. Thresholding and peak detection then locate candidate ME frames.

  • Pipeline: The spotting pipeline combines facial alignment, block-based feature extraction, feature-difference analysis, contrasting, and peak detection.The method is presented as a four-step workflow combining appearance-based descriptors with FD analysis.
  • 3.1 Facial points tracking and block division: Three facial points are tracked to correct in-plane rotation and scale variation before dividing the face into equal-sized blocks.The tracked points are two inner eye corners and a nasal spine point.
  • 3.2 Feature extraction: The method evaluates LBP and HOOF descriptors, constructing frame-level feature vectors from block histograms or optical-flow orientations.LBP uses normalized histograms concatenated across blocks; HOOF summarizes flow-field orientations.
  • 3.3 Feature difference (FD) analysis: FD analysis compares each current frame with an average feature frame formed from temporal neighbors within a sliding window.The current, tail, head, and average feature frames define the comparison used to measure rapid change.
  • 3.3 Feature difference (FD) analysis: Large FD values indicate rapid facial movements, while slower movements produce smaller values over the same short interval.The FD is computed using Chi-Squared distance between feature histograms.
  • 3.3 Feature difference (FD) analysis: For each frame, the method averages the 12 largest block FD values, contrasts them against surrounding values, zeros negatives, and detects thresholded peaks.The 12 selected blocks equal one third of the 36 face blocks; the threshold depends on the mean, maximum, and parameter τ.

4 METHOD FOR ME RECOGNITION

The recognition framework normalizes faces, amplifies subtle motion, interpolates short sequences, and compares spatial-temporal descriptors before classification.

  • Face alignment: Face alignment registers each clip to a neutral model face using 68 landmarks and a Local Weighted Mean transformation.All frames are normalized with the resulting transform, then face regions are cropped.
  • Motion magnification: Eulerian video magnification amplifies subtle facial motions before feature extraction, while larger α values increase both motion amplification and artifact risk.The method varies α across ten levels to study its effect on recognition.
  • Temporal interpolation: Temporal interpolation maps short ME sequences onto continuous temporal curves and resamples them into a common length.This supports up-sampling clips with few frames and stabilizes feature-descriptor performance.
  • Feature descriptors: The framework compares LBP-TOP with HOG and HIGO descriptors for ME recognition.HOG uses weighted orientation votes, whereas HIGO uses simple votes for histogram bins.
  • LBP-TOP: LBP-TOP extracts and concatenates histograms from XY, XT, and YT planes, while the method evaluates five combinations of these histograms.The XT and YT planes capture motion-transition information in addition to spatial texture.

5 EXPERIMENTAL RESULTS AND DISCUSSIONS

The experiments evaluate ME spotting, recognition, a combined automatic analysis system, and comparisons with human performance.

  • ME spotting: ME spotting results are presented in Section V.A.
  • ME recognition: ME recognition is evaluated through four sub-experiments in Section V.B.
  • Integrated evaluation: Section V.C tests an automatic ME analysis system combining spotting and recognition, while Section V.D compares it with human subjects.

5.1 ME spotting

The proposed training-free method spots spontaneous MEs by contrasting feature differences and detecting peaks, with LBP outperforming HOOF across the evaluated datasets. Performance is strongest on CASMEII, while eye movements account for many false spots and remain difficult to suppress without reducing true detections.

  • Evaluation: The method evaluates spontaneous ME spotting on CASMEII and three extended SMIC-E datasets using ROC curves and AUC.SMIC-E was created with longer clips around ME occurrences, unlike the original SMIC recognition clips.
  • Threshold analysis: Lowering the threshold increases correctly spotted MEs but also increases FPR, reflecting the spotting trade-off.ROC points are generated by varying τ from 0 to 1 in steps of 0.05.
  • Feature comparison: LBP outperforms HOOF because it achieves higher AUC values and lower false positive rates.The comparison is reported for the proposed spotting method across CASMEII and the SMIC-E datasets.
  • Results: 70% of MEs are spotted with 13.5% FPR and 84.53% AUC on SMIC-E-VIS using LBP.SMIC-E-VIS provides the best spotting performance among the three SMIC-E datasets.
  • Results: LBP reaches 92.98% AUC on CASMEII, where shorter video clips may make spotting easier than on SMIC-E.The explanation for CASMEII’s higher AUC is presented as a hypothesis.
  • Failure cases: Eye blinks and gaze changes generate many false spots because the method detects transient movement across the whole face.Excluding eye regions or filtering eye-blink peaks lowers FPR but also lowers TPR, partly because many MEs involve eye muscles or overlap with blinks.

5.2 ME recognition

The recognition framework evaluates temporal interpolation, feature descriptors, camera modalities, and motion magnification on SMIC and CASMEII. Combining TIM10, motion magnification, and HIGO produces the strongest reported results and outperforms prior methods across four datasets.

  • 5.2.1 Effect of the interpolation length: TIM10 achieved the best performance on all three SMIC datasets, and interpolation beyond 10 frames did not improve accuracy.The authors attribute this to unified input lengths and possible dilution of temporal changes in longer sequences.
  • 5.2.1 Effect of the interpolation length: Higher interpolation frame rates increased computation time and storage requirements without consistently improving accuracy.
  • 5.2.2 Comparison of three features: HIGO-XOT achieved the best SMIC result at 76.06%, while gradient-based HOG and HIGO generally outperformed LBP on RGB datasets; LBP performed best on SMIC-NIR.The results indicate that temporal-dimension information is most important, while XY histograms may add redundant facial-appearance information.
  • 5.2.4 Motion magnification: Motion magnification generally improved recognition over α = 1, with the best performance generally occurring for magnification levels from 8 to 16.Higher magnification could degrade performance because it induces artifacts.
  • 5.2.5 Comparison to the state of the art: 81.7% accuracy on SMIC-VIS, 68.29% on SMIC-HS, 67.61% on SMIC-NIR, and 67.21% on CASMEII were achieved using motion magnification, TIM10, and HIGO.Using leave-one-subject-out validation, the framework outperformed all previous methods on all four ME datasets.

5.3 An automatic ME analysis system (MESR) combining Spotting and Recognition

MESR combines spontaneous-ME spotting and recognition by first locating candidate events in long videos and then classifying them into positive, negative, or surprise categories. On SMIC-E-VIS, its overall performance was 42.42% under the selected spotting threshold, while the authors report comparable performance to humans for the combined task.

  • 5.3 An automatic ME analysis system (MESR) combining Spotting and Recognition: MESR integrates spotting and recognition, addressing the prior separation of these tasks in automatic ME analysis.The system first spots MEs in long videos and then classifies them into positive, negative, and surprise categories.
  • 5.3 An automatic ME analysis system (MESR) combining Spotting and Recognition: The system treats each input video as an unknown test sample and trains its classifier on labeled MEs from other subjects.
  • 5.3 An automatic ME analysis system (MESR) combining Spotting and Recognition: At TPR = 74.86% and FPR = 22.98%, correctly spotted sequences reached 56.67% emotion-recognition accuracy.The recognition accuracy was lower than in experiments using hand-labeled ME boundaries because automatically spotted sequences included imprecise or non-ME frames.
  • 5.3 An automatic ME analysis system (MESR) combining Spotting and Recognition: 42.42% overall MESR accuracy resulted from multiplying the 74.86% spotting rate by the 56.67% recognition accuracy.
  • 5.3 An automatic ME analysis system (MESR) combining Spotting and Recognition: The authors report that MESR performs comparatively to humans on the difficult task of spotting and recognizing spontaneous MEs.

5.4 Human test results

Human performance was evaluated on ME recognition and on spotting followed by recognition using spontaneous-video experiments. The automatic MESR system was comparable to humans on the combined task, but its false spot rate remained a limitation.

  • ME recognition: 15 subjects participated in the ME recognition experiment using 71 SMIC-VIS ME samples.Subjects were shown the ME definition and three sample videos before testing.
  • ME spotting and recognition: The second experiment asked 15 subjects to spot MEs in long videos and then identify their emotion category.It used 71 SMIC-E-VIS clips and five neutral clips without MEs.
  • Evaluation: Human recognition performance was measured using accuracy and false-positive rate, based on correctly spotted and recognized MEs.Accuracy was divided by 71, while false-positive rate used 152 as the maximum possible false spots.
  • Human results: 49.74% ± 8.04% was the mean human accuracy, with a mean false-positive rate of 7.31% ± 5.45%.Recognition ability varied across individual subjects.
  • Automatic-system comparison: 42.42% was MESR’s accuracy, comparable to the mean human accuracy within one standard deviation.MESR’s main shortage was its 22.98% false spot rate, which could be reduced by excluding fast non-ME movements such as eye blinks.

6 CONCLUSIONS

The paper develops methods for spotting and recognizing spontaneous MEs, which are harder to analyze than posed expressions. Its system outperforms humans on recognition and is comparable to humans on combined spotting and recognition, while several scope limitations remain.

  • Scope: The study targets spontaneous MEs, which are much more difficult to analyze than posed expressions.
  • ME spotting: The proposed spotting method uses feature-difference comparison with LBP and HOOF features, and LBP outperforms HOOF on two databases.
  • ME recognition: Motion magnification is incorporated into the recognition framework to counter the low intensity of MEs, which outperforms the state of the art on SMIC and CASMEII.
  • MESR: The automatic ME analysis system first spots and then recognizes MEs, outperforming humans on recognition and performing comparably on the combined task.
  • Limitations: The spotting method uses fixed intervals for peak detection, may confuse eye blinks with MEs, and was evaluated on limited near-frontal datasets.Future work proposes more precise onset and offset detection, AU detection, and larger, more natural datasets with complex head views and interactions.
Loading 1511.00423v2…