PaperlayerView exact revision on arXiv

Source-linked AI summary

MoTop: Motion-Topological Model For Micro AU Detection

Huai-Qian Khor, Mengting Wei, Yante Li, Chu Kiong Loo, Guoying Zhao

arXiv:2609.31285v1cs.CV

TL;DR

Micro-AU detection is challenging because overlapping AU activity and subtle facial motion complicate distinguishing muscle movements. MoTop combines optical-flow context with landmark transitions, amplifies those transitions, and builds anatomical clusters for multi-scale geometry. On CD6ME, it performs best in 6 of 12 AU classes and comparably in the others.

  • Problem

    Overlapping AU activity and the subtle motion of micro-expressions make micro-AU detection challenging, despite its role as a fine-grained step toward emotion analysis.

    Source: §I · p. 1

  • Method

    MoTop injects optical-flow motion context as regional guidance for facial landmarks, amplifies landmark transitions through linear extrapolation, and uses anatomical clusters for multi-scale geometry.

    Source: §I · p. 2

  • Results

    MoTop achieves the best performance in 6 of 12 AU classes and comparable performance in the others; ablations report 0.4151 to 0.4284 with facial clusters and 0.4284 to 0.4610 with ADN.

    Source: C. Comparison with State-of-the-art · p. 4, D. Ablation Study · p. 4

  • Takeaways & Limitations

    The authors report that MoTop models localized, fine-scale motion-topological transitions to alleviate AU ambiguity.

    Source: E. Qualitative Analysis · p. 5, §IV · p. 5

Abstract

from arXiv · show

Facial micro-expressions are spontaneous, brief, and subtle facial movements that reveal suppressed emotions in high-stakes environments. In contrast to classic expression analysis, detecting action unit (AU) yields a finer representation of facial movements, serving as a preliminary step before defining expression classes and other downstream tasks. Therefore, it represents a crucial upstream task in facial analysis, and improving an AU detection module increases the precision of facial analysis. Despite that, detecting AU is challenging because of the constrictive nature of the AU activation regions, leading to confusion among different AUs known as AU ambiguity. To model the fine-scale changes, we propose \textbf{MoTop}, a motion-topological model that is augmented with a learnable motion context, yielding regional soft guidance for facial activity, followed by facial landmarks that capture the fine-scale topological changes of micro AUs. To increase the micro facial landmark representations, we amplify the encoded facial landmark transitions via linear extrapolation, thereby increasing the spatial proximity of landmarks and enhancing the low-intensity landmark dynamics. In addition, we design anatomical facial clusters that enhance the hierarchical representation, facilitating multi-scale modelling of facial geometry and improving micro-topological representations. With these contributions, we have achieved state-of-the-art performance on the CD6ME protocol for the micro AU detection task.

I. INTRODUCTION

Micro-AU detection is a fine-grained step toward emotion analysis, but overlapping AU activity and subtle landmark motion make it difficult. MoTop combines motion context, landmark transitions, linear extrapolation, and anatomical clusters to model these changes at multiple scales.

I. INTRODUCTION

  • AU detection discretizes facial muscle movements as a fine-grained first-level analysis used in downstream categorization and insight mining.The paper describes detecting 12 multilabel micro AUs as approximating labor-intensive annotation; selected AUs reflect sample size and downstream significance.Source: §I · p. 1
  • Overlapping AUs and subtle activity regions make micro-AU detection difficult, with ambiguity and weak landmark dynamics mutually reinforcing the challenge.The paper gives AU 1 and AU 4 around the eyebrows as an example of concurrent, overlapping appearances.Source: §I · p. 1
  • MoTop jointly learns optical flow and landmark transitions, injecting motion context as regional guidance for fine-scale micro-AU detection.The approach uses motion context to guide landmarks as they encode fine-scale topological changes.Source: §I · p. 2
  • Linear extrapolation amplifies facial landmark transitions to increase node proximity and enhance representations of low-intensity micro-AU dynamics.The model also designs anatomical facial clusters to build a node-to-organ hierarchy for multi-scale facial geometry.Source: §I · p. 2

II. METHODOLOGY

MoTop combines optical-flow context and facial landmarks to model motion-topological changes. It amplifies landmark transitions, pools them into anatomical clusters, and fuses the resulting features with a separate motion-only AU detector.

II. METHODOLOGY

  • MoTop injects encoded optical-flow motion context to provide regional guidance for facial landmarks.A separate motion context network is fine-tuned to supply this context.Source: §II · p. 2
  • Linear extrapolation amplifies landmark changes, while anatomical-cluster graph pooling produces multi-scale facial geometry for fine-scale AU detection.The model uses 2D facial landmarks to learn topological changes.Source: §II · p. 2
  • A separate motion-only AU Detection Network is fused with MoTop's multi-scale motion-topological features.Source: §II · p. 2

A. Preliminaries

The preliminaries describe optical-flow inputs and a graph-temporal representation of facial landmarks. The model computes flow and strain between onset and apex frames, then uses landmark graphs and temporal convolution to represent changes over time.

A. Preliminaries

  • Optical-flow changes and their derivative, optical strain, are computed between onset and apex frames for both motion networks.Source: A. Preliminaries · p. 2
  • The method extracts 68 facial landmarks, filters out unlikely motion sites to retain 47, and models intra-part node changes with a learnable adjacency matrix.Source: A. Preliminaries · p. 2
  • Four successive GCN-TCN blocks model landmark connections and temporal changes, with a skip connection accompanying the temporal learner to address vanishing gradients.The block dimensions are 32, 64, 64, and 128; the paper notes that micro-AU node changes can be extremely subtle.Source: A. Preliminaries · p. 2

B. MoTop

MoTop amplifies subtle landmark transitions, combines landmark features with learned motion context, and pools the resulting topology into anatomical clusters for AU prediction.

B. MoTop

  • Linear extrapolation amplifies differences between successive landmark frames to represent low-intensity micro-AU dynamics.The extrapolated frames represent transitions between onset–apex and apex–offset frames.Source: B. MoTop · p. 2
  • Motion tokens provide soft regional context, and MCN produces AU-responsive motion context aligned with facial landmark nodes.MCN uses adaptive average pooling to create M47 and optimizes it toward AU ground truth.Source: B. MoTop · pp. 2–3
  • MoTop attaches motion context to normalized landmark coordinates so its input represents localized, fine-scale landmark changes.The combined input has 130 dimensions: two landmark coordinates plus a 128-dimensional motion-context embedding.Source: B. MoTop · p. 3
  • Anatomical graph pooling summarizes 47 landmarks into six AU-relevant facial clusters, whose temporal changes feed AU prediction.The clusters cover the right and left brows, right and left eyes, nose tip, and mouth; temporal averaging and a fully connected layer produce predictions.Source: B. MoTop · p. 3

C. Micro AU Detection

The detection design pairs MoTop’s motion-topological features with a separate optical-flow detector, then fuses their outputs for AU detection.

C. Micro AU Detection

  • A separate motion-only AU Detection Network (ADN) complements MoTop where landmarks may miss AU-active regions such as the cheek.ADN uses TinyViT and is trained directly on optical flow; its output joins MoTop through decision-level fusion.Source: C. Micro AU Detection · p. 3
  • Decision-level fusion combines ADN’s patch-based learning with MoTop’s fine, multi-scale features for final AU detection.The two networks are trained with separate focal losses.Source: C. Micro AU Detection · p. 3
  • ADN optimizes optical-flow features, MCN optimizes AU-specific motion context, and MoTop optimizes regional fine-scale variations.Each model is fine-tuned with its own binary cross-entropy focal loss.Source: C. Micro AU Detection · p. 3

A. Dataset & Protocol

The evaluation combines six micro-expression databases under the CD6ME protocol, testing each database in turn as the held-out set.

A. Dataset & Protocol

  • The experiments combine CASME, CASME II, CAS(ME)3, 4DME, MMEW, and SAMM with 12 overlapping AUs.The combination follows the cited protocol because emotional-label definitions can vary across databases.Source: A. Dataset & Protocol · pp. 3–4
  • CD6ME uses leave-one-database-out cross-validation, training on the remaining databases and testing on the held-out database.The process repeats until each database has served once as the test set.Source: A. Dataset & Protocol · pp. 3–4
  • Performance under the protocol is averaged across databases using Macro-F1 scores.Source: A. Dataset & Protocol · p. 4

B. Experimental Setup

The experimental setup optimizes three separate models with Adam and uses different learning-rate decay schedules for MoTop versus ADN and MCN.

B. Experimental Setup

C. Comparison with State-of-the-art

MoTop performs best in 6 of 12 AU classes under CD6ME and is comparable on the others, while its ablations and extrapolation analysis examine the contributions of its components.

C. Comparison with State-of-the-art

  • MoTop achieves the best performance in 6 of 12 AU classes and comparable performance in the others under CD6ME, with persistent difficulty on AU 9.The authors also report improvements over InfuseNet and RFT for AU 2, AU 12, and AU 17, and note that cheek-related AUs remain challenging.Source: C. Comparison with State-of-the-art · p. 4

D. Ablation Study

  • 0.4151 to 0.4284 (+1%) with facial clusters, 0.4284 to 0.4610 (+3%) with ADN, and 0.4610 to 0.4075 (-5%) when motion-topological features are excluded.The ablation indicates that landmarks without motion context are insufficient for topological learning, while ADN complements MoTop with potentially missing representations.Source: D. Ablation Study · p. 4
  • MF1 growth with increasing extrapolation factor decreases from 0.56% to 0.11%, indicating that the optimum factor lies below 10.The authors report that larger factors increase node-change magnitude, making the topological learner more sensitive to larger displacements.Source: Figure 3 · p. 4, D. Ablation Study · p. 4

E. Qualitative Analysis

The qualitative analysis illustrates motion and topological activations for AU detection and examines how learned adjacency weights change across GCN layers.

E. Qualitative Analysis

  • Grad-CAM visualizations show localized motion patches, while topological features help resolve the illustrated AU 4 ambiguity and focus on active eyebrow, eye, and mouth regions.The figure includes correct detections of AU14, AU15, AU4, and AU7, alongside the last GCN-TCN block’s adjacency matrix.Source: E. Qualitative Analysis · p. 4, Figure 4 · p. 4
  • Earlier GCN layers emphasize regional and cross-region eye motion, whereas GCN 3 and GCN 4 focus on more localized, balanced regions.The authors interpret the layer transition as adaptation toward localized, fine-scale motion-topological features that alleviate AU ambiguity.Source: E. Qualitative Analysis · p. 4, Figure 5 · p. 5
  • Table I compares the proposed algorithm with baselines across AUs, marking best results in bold and second-best results with underlining; Table II studies MoTop component efficacy.The supplied captions identify the tables’ scope but provide no cell values.Source: Table I · p. 5, Table II · p. 5

F. Detect-Then-Recognise (DTR) Task

The DTR task uses MoTop-detected micro AUs to infer coarse- and fine-grained emotions on SMIC. The conclusion reports CD6ME state-of-the-art performance while noting that micro-AU detection remains challenging.

F. Detect-Then-Recognise (DTR) Task

  • On SMIC, DTR maps detected micro AUs to three coarse-grained emotion classes with labels and five fine-grained classes without ground-truth annotations.The mappings follow SMIC protocols for coarse categories and CASME II protocols for fine-grained categories; predicted emotions are reported as semantically coherent at both granularities.Source: F. Detect-Then-Recognise (DTR) Task · p. 5, Figure 6 · p. 5
  • Meaningful fine-grained emotion categories emerge on the unseen SMIC dataset despite the absence of fine-level annotations.The authors state that this demonstrates MoTop’s ability to infer micro-expressions from detected micro AUs in this setting.Source: F. Detect-Then-Recognise (DTR) Task · p. 5

IV. CONCLUSION

  • The conclusion reports state-of-the-art performance on CD6ME, but domain differences, class imbalance, AU ambiguity, and minuscule AU activity remain challenging.MoTop uses late fusion with the AU Detection Network, motion context for regional guidance, linear extrapolation for landmark displacements, and anatomical clusters for multi-scale geometry.Source: §IV · p. 5

Related papers

Loading 2609.31285v1…