PaperlayerView exact revision on arXiv

Source-linked AI summary

Short and Long Range Relation Based Spatio-Temporal Transformer for Micro-Expression Recognition

Liangfei Zhang, Xiaopeng Hong, Ognjen Arandjelovic, Guoying Zhao

arXiv:2112.05851v4cs.CV

TL;DR

Micro-expressions are subtle, involve less facial movement, and last 0.04s to 0.2s, making them harder to recognize than macro-expressions. SLSTT uses a transformer encoder to learn spatial relations from long-term optical-flow frames and a temporal aggregation block to model temporal relations. The method achieves state-of-the-art results across SMIC, CASME II, and SAMM in both sole- and composite-database evaluations.

  • Problem

    Micro-expressions are subtle, involve less facial movement, and last 0.04s to 0.2s, making them harder to recognize than macro-expressions.

    Source: §1 · p. 1

  • Method

    SLSTT uses a transformer encoder to learn spatial relations from long-term optical-flow frames and a temporal aggregation block to model temporal relations.

    Source: §1 · p. 2, §5 · p. 10

  • Results

    The method achieves state-of-the-art results across SMIC, CASME II, and SAMM in both sole- and composite-database evaluations.

    Source: §4.4 · p. 9

  • Takeaways & Limitations

    The proposed framework demonstrates that a deep-learning micro-expression recognizer can be built entirely around transformers, without CNN use.

    Source: §1 · p. 2, §5 · p. 10

    Because current MER corpora use visual data and emotion-to-action-unit mappings are not bijective, their theoretical highest automated-recognition accuracy is below 100%.

    Source: §4.4 · p. 10

Abstract

from arXiv · show

Being spontaneous, micro-expressions are useful in the inference of a person's true emotions even if an attempt is made to conceal them. Due to their short duration and low intensity, the recognition of micro-expressions is a difficult task in affective computing. The early work based on handcrafted spatio-temporal features which showed some promise, has recently been superseded by different deep learning approaches which now compete for the state of the art performance. Nevertheless, the problem of capturing both local and global spatio-temporal patterns remains challenging. To this end, herein we propose a novel spatio-temporal transformer architecture -- to the best of our knowledge, the first purely transformer based approach (i.e. void of any convolutional network use) for micro-expression recognition. The architecture comprises a spatial encoder which learns spatial patterns, a temporal aggregator for temporal dimension analysis, and a classification head. A comprehensive evaluation on three widely used spontaneous micro-expression data sets, namely SMIC-HS, CASME II and SAMM, shows that the proposed approach consistently outperforms the state of the art, and is the first framework in the published literature on micro-expression recognition to achieve the unweighted F1-score greater than 0.9 on any of the aforementioned data sets.

1 INTRODUCTION

Micro-expressions are difficult to recognize because they are brief and subtle, while existing approaches struggle to learn local and global spatial relations jointly. The paper introduces SLSTT, a transformer-only video-recognition framework using long-term optical flow and temporal aggregation, and reports state-of-the-art results across three databases.

1 INTRODUCTION

  • Micro-expressions last 0.04s to 0.2s and involve less facial movement than macro-expressions, making them harder to recognize.Their involuntary expression under psychological inhibition can reveal felt emotion even when a person attempts to conceal it.Source: §1 · p. 1
  • Existing approaches that learn global spatial relations after extracting local features cannot learn both relation types jointly.The paper motivates using self-attention to address this gap by considering sequence elements in their entirety.Source: §1 · p. 2
  • SLSTT is a video-based, CNN-free transformer framework that uses long-term optical-flow matrices, learns short- and long-range spatial relations, and aggregates temporal features.Its temporal aggregator connects spatial-relation features extracted from each frame.Source: §1 · p. 2, Figure 2 · p. 2
  • The method achieves state-of-the-art results across SMIC, CASME II, and SAMM in both sole- and composite-database evaluations.The authors report that it performs best in most tested scenarios.Source: §1 · p. 2

2 RELATED WORK

Prior micro-expression recognition work spans handcrafted and deep-learning approaches to spatial and temporal features. The review describes transformer vision methods and notes that existing micro-expression approaches do not adequately use both spatial and temporal information.

2.1 Micro-expression Recognition

  • Micro-expression recognition has progressed from handcrafted computer-vision features to deep-learning approaches, with research organized around spatial or temporal strategies.Most published work is video-based, while a smaller body analyzes single frames.Source: §2.1 · p. 3, §2.1.2 · p. 3

2.1.1 Spatial Features

  • Spatial methods include local appearance descriptors, facial-region features, CNN extraction, attention blocks, and AU-based graphs.Region-based approaches partition faces into different numbers of regions, while AU graphs encode relationships between action-unit engagements.Source: §2.1.1 · p. 3

2.1.2 Temporal Features

  • Temporal methods use apex frames or onset-to-offset sequences, including interpolated frames, 3D spatiotemporal descriptors, optical flow, and recurrent models.Handcrafted approaches often apply spatial operators to temporal planes, while deep-learning work also uses RNNs or LSTMs for longer-range patterns.Source: §2.1.2 · p. 3

2.2 Transformers in Computer Vision

  • Transformers use self-attention to model long-range sequence dependencies, and image patches let them learn spatial dependencies.Vision Transformer encodes image patches directly, whereas DETR retains a CNN backbone; existing micro-expression approaches do not adequately use both spatial and temporal information.Source: §2.2 · p. 4

3 PROPOSED METHOD

SLSTT combines long-term optical flow with transformer-based spatial feature extraction and an LSTM temporal aggregator to capture short- and long-range spatio-temporal relations. Its pipeline embeds flow fields, encodes spatial features, aggregates frame outputs into a video-level feature, and classifies the result.

3.1 Long-term Optical Flow

  • Long-term optical flow is computed between each frame and the onset frame, producing more structured motion patterns than consecutive-frame flow.The fields increase in magnitude toward the apex and decline afterward.Source: Figure 3 · p. 4, §3.1 · p. 4

3.2 Spatial Feature Extraction

  • The method treats each frame as a sequence of patches with local spatial features, then uses a transformer encoder to extract long-range spatial relations.Long-term optical flow fields, rather than original appearance images, are the inputs.Source: §3.2 · p. 4, §3.2.1 · pp. 4–5

3.2.1 Input Embedding and Short-Range Spatial Relation Learning

  • A patch-wise fully connected layer extracts local features before the patches are flattened into vectors with class and position embeddings.The resulting sequence contains N + 1 vectors, including the class token.Source: §3.2.1 · p. 5

3.2.2 Long-Range Spatial Relation Learning by Transformer Encoder

  • The spatial encoder applies 12 transformer layers, each combining multi-head self-attention and a position-wise feed-forward block with layer normalization and residual connections.The feed-forward block expands the embedding from D to 4D and reduces it back to D; D is 768 in the experiments.Source: §3.2.2 · p. 5

3.3 Temporal Aggregation

  • An LSTM aggregator processes frame-level encoder outputs as a sequence and produces one feature set for the whole video sample.The inputs are not concatenated, and the aggregator uses three LSTM layers.Source: §3.3 · p. 6

3.4 Network Optimization

  • The final video feature is classified by two fully connected layers with SoftMax, and cross-entropy loss is used for training.Cosine annealing adjusts the learning rate during optimization.Source: §3.4 · pp. 6–7

4 EXPERIMENTS AND EVALUATION

The section outlines the experiments, from dataset and preprocessing descriptions through implementation and evaluation metrics, before reporting and discussing results.

4 EXPERIMENTS AND EVALUATION

  • The experiments are organized around descriptions of the data sets, preprocessing, implementation details, and evaluation metrics.The section then reports results and discusses the findings.Source: §4 · p. 7
  • Preprocessing and implementation details are included before the results are reported.The passage gives no further details about these procedures.Source: §4 · p. 7
  • The section concludes with results and discussion of the findings.No specific results are provided in this passage.Source: §4 · p. 7

4.1 Databases

The evaluation uses three spontaneous-expression databases, with SMIC-HS selected from SMIC for comparability to CASME II and SAMM, which provide high-speed-camera videos.

4.1 Databases

  • The evaluation selects three widely used spontaneous-expression databases to support diversity, evaluation scale, and comparison with prior work.All samples capture spontaneous rather than acted micro-expressions.Source: §4.1 · p. 7

4.1.1 SMIC

  • SMIC-HS contains 164 video samples from 16 subjects and uses three expression labels: negative, positive, and surprised.SMIC also has VIS and NIR subsets, but the study uses HS to match the high-speed-camera videos in CASME II and SAMM.Source: §4.1.1 · p. 7

4.1.2 CASME II

  • CASME II contains 247 samples from 26 Chinese participants, labeled into five emotion classes.The classes are Happiness (33), Disgust (60), Surprise (25), Repression (27), and Others (102).Source: §4.1.2 · p. 7

4.1.3 SAMM

  • SAMM contains 159 short videos from 32 participants across 13 ethnicities, with samples assigned to eight emotion classes.The database was recorded at 200 fps; its labels include Anger (57), Happiness (26), Other (26), Surprise (15), Contempt (12), Disgust (9), Fear (8), and Sadness (6).Source: §4.1.3 · p. 7

4.2 Data Pre-Processing

Preprocessing standardizes face extraction and cropping across databases, then increases temporal sampling with optical-flow-oriented frame interpolation.

4.2.1 Face Cropping

  • The study uses DLib's Ensemble of Regression Trees to locate 68 facial landmarks uniformly across databases, including SAMM, which lacks supplied cropped faces.SMIC-HS and CASME II supply cropped faces, but their original face-extraction methods differ from the study's uniform approach.Source: §4.2.1 · pp. 7–8
  • A nonreflective 2D Euclidean transformation replaces the original registration strategy to reduce resemblance between samples while aligning salient points and retaining facial-contour variability.Landmarks are detected at the apex rather than the first frame; point 30 is the crop center, and a square size is calculated from eyebrow-to-chin distances.Source: §4.2.1 · p. 8
  • The landmark illustration marks onset-frame points in green and apex-frame points in red.Points 27–30 identify the nose center line, with point 30 used as the crop center.Source: Figure 8 · p. 8, §4.2.1 · p. 8

4.2.2 Temporal Interpolation

  • RIFE is used to interpolate intermediate frames directly from optical flow, targeting smoother flow and fewer motion-boundary artifacts than bidirectional-flow approximation.The method recursively interpolates between consecutive frames and prioritizes frames near the apex; for SMIC-HS, the middle frame is treated as the apex.Source: §4.2.2 · p. 8

4.3 Experimental Settings

The experiments use a ViT spatial encoder and compare mean with LSTM temporal aggregation, evaluating both individual databases and a three-database composite under LOSO cross-validation.

4.3.1 Implementation Details

  • The spatial encoder uses a 12-layer ViT-B/16 initialized from ImageNet, with 384 × 384 inputs divided into 16 × 16 patches.For temporal aggregation, 11 frames centered on the apex feed the mean or LSTM aggregator; the paper reports that trying other frame counts did not work better.Source: §4.3.1 · p. 8

4.3.2 Mean Versus LSTM Aggregator

  • The experiments compare LSTM aggregation with mean aggregation, with LSTM expected to distinguish emotions better through its greater expressive capability.Both aggregators use outputs from the spatial encoder for each frame.Source: §4.3.2 · pp. 8–9

4.3.3 Evaluation Metrics

  • Evaluation uses LOSO cross-validation on individual databases and their composite; individual-database tests use accuracy and macro F1, while composite tests use UF1 and UAR.The composite database combines SMIC, CASME II, and SAMM into negative, positive, and surprise classes and contains 68 subjects.Source: §4.3.3 · p. 9

4.4 Results and Discussion

SLSTT performs near the top across sole- and composite-database evaluations, though its dominance is less pronounced in the composite setting. The discussion links performance to long-range relation learning and notes that labeling and visual-only data constrain recognition.

4.4 Results and Discussion

  • In SDE, SLSTT ranks first in all but one testing scenario, where it is second behind Sun et al.; in CDE, it is first or second in most cases.For CDE, it achieves the best UF1 and UAR on CASME II and best UF1 on the full composite database; it is second on composite UAR and SMIC-HS UF1.Source: §4.4 · p. 9, Table 1 · p. 10, Table 2 · p. 10
  • SLSTT is less dominant in CDE than SDE, which the authors attribute to its stronger performance on finer-grained classes than on a simpler three-class setting.The authors also report that CASME II performance is consistently higher than on other datasets and suggest ethnic diversity may increase the MER challenge.Source: §4.4 · p. 9
  • The authors identify long-range spatial learning as key: short-range-only CNN approaches do not outperform handcrafted methods, while SLSTT exceeds the corresponding methods using multi-head self-attention.They also compare the proposed LSTM temporal aggregator with a simpler mean operator aggregator in Tables 1 and 2.Source: §4.4 · p. 9, Table 1 · p. 10, Table 2 · p. 10
  • CASME II contains inconsistently labeled AU4-only samples assigned to either Disgust or Others, complicating that distinction.The authors also report that some SAMM Contempt AU labels were not manually verified, causing confusion with Happiness; they note that AU-to-emotion mapping is not bijective and visual-only data provides imperfect information, so the theoretical accuracy ceiling is below 100%.Source: §4.4 · pp. 9–10

5 CONCLUSION

The conclusion presents SLSTT as a convolution-free transformer framework that learns short- and long-range spatial and temporal relations for micro-expression recognition. Across three databases and sole and composite evaluation settings, the authors report that it consistently outperforms the state of the art.

5 CONCLUSION

  • SLSTT uses transformer encoders with multi-head self-attention on visualized long-term optical-flow frames for spatial relations, plus a temporal aggregation block for temporal relations.The framework is described as entirely convolution-free.Source: §5 · p. 10
  • The framework learns short- and long-term relations between pixels in both spatial and temporal directions of sample videos.Spatial relations are learned from visualized long-term optical-flow frames, while temporal relations are handled by the temporal aggregation block.Source: §5 · p. 10
  • Across three MER databases, sole and composite evaluation settings, and LOSO cross-validation, the authors report that SLSTT consistently outperforms the state of the art.The authors say these findings motivate further research on transformer architectures rather than CNNs for micro-expression analysis.Source: §5 · p. 11

Related papers

Loading 2112.05851v4…