PaperlayerView exact revision on arXiv

Source-linked AI summary

GM-GReFEL: Geometry-Guided Spatiotemporal State-Space Learning for Robust Dynamic Facial Expression Recognition

Yudhistira Arditya Pratama, Yi-Zeng Hsieh

arXiv:2610.31140v1cs.CVcs.LG

TL;DR

Unconstrained-video facial expression recognition must contend with quadratic self-attention cost and noisy crowdsourced labels. GM-GReFEL combines grouped state-space modeling with entropy-guided geometric reliability, achieving competitive results across three benchmarks, though its five-fold MAFW mean remains below leading models.

  • Problem

    Unconstrained-video expression recognition must handle noisy labels while modeling temporal information efficiently.

    Source: §I · p. 1

  • Method

    GM-GReFEL combines a linear-complexity Modulated GroupMamba backbone with entropy-guided geometric reliability and static-to-dynamic initialization.

    Source: §I · p. 1, §VI · p. 12

  • Results

    Five-fold means are comparable to recent pre-trained models on DFEW, lower on MAFW, and competitive on FERV39k; DFEW mean scores are 64.1% UAR and 74.5% WAR.

    Source: 4) Cross-Dataset Summary · p. 9

  • Takeaways & Limitations

    GM-GReFEL is competitive with large pre-trained models but is not uniformly ahead of them.

    Source: 4) Cross-Dataset Summary · p. 9

    The study uses visual input only, performs categorical rather than valence–arousal recognition, and requires a laboratory-grade GPU for its reported latency.

    Source: §V · p. 12

Abstract

from arXiv · show

Recognizing facial expressions in unconstrained video suffers from two coupled problems: self-attention grows quadratically in cost with clip length, and crowdsourced labels are noisy. We present GM-GReFEL, a spatiotemporal state-space network with a geometry-aware decision stage. Its Modulated GroupMamba backbone replaces a spatiotemporal Vision Transformer. Six channel groups are scanned separately, four over the frame plane and two over time, so the cost grows linearly with sequence length ($\mathcal{O}(N)$); after each layer, a Channel Affinity Modulation gate mixes the groups again. On top of the backbone, a GReFEL reliability module turns the normalized entropy of each prediction into a mixing weight between the classifier output and a vote from trainable geometric anchors, so that facial geometry counts for more when the network is unsure. The model is initialized by static-to-dynamic weight transfer and trained with a Triple Loss. In five-fold cross-validation, the best fold scores 66.57\%/76.46\% UAR/WAR on DFEW, 0.3 points or less below the strongest compared method, and 45.66\%/62.27\% on MAFW, ahead of all compared methods; on FERV39k the mean over five splits, 42.30\%/52.66\%, is competitive. The five-fold means are 64.07\%/74.54\% on DFEW, on par with recent large pre-trained models, and 38.90\%/54.37\% on MAFW, below the best of them. A paired $t$-test on DFEW indicates a modest benefit of temporal modeling (+1.61 UAR points, one-sided $p=0.031$). Attention stays on plausible facial regions, and latency is close to a ViT-B/16 baseline (47.4 vs.\ 46.8 ms per clip). Code: https://github.com/ardiytama/GroupMamba-DFER.

I. INTRODUCTION

GM-GReFEL couples linear-complexity spatiotemporal modeling with geometry-based reliability to address computational cost and noisy labels in unconstrained-video expression recognition. Its best-fold results are strongest among compared methods on MAFW and within 0.3 points of S4D on DFEW.

I. INTRODUCTION

  • Crowdsourced labels disagree on ambiguous expressions, while self-attention’s O(N^2) cost limits clip length, resolution, and memory.Plain cross-entropy can absorb label noise, with a documented bias toward majority classes such as Happy and Neutral.Source: §I · p. 1

C. Transformer-Based Spatiotemporal Modeling

  • GM-GReFEL replaces the quadratic attention core with six channel groups scanned spatially or temporally, then re-coupled by Channel Affinity Modulation.The four spatial groups use separate frame-plane scan directions, while two groups scan forward and backward over time.Source: §I · p. 1, D. Selective State-Space Backbones in… · p. 2

I. INTRODUCTION

  • On its best fold, GM-GReFEL has the highest UAR and WAR among compared methods on MAFW and comes within 0.3 points of S4D on DFEW.Its five-fold mean is comparable to recent pre-trained models on DFEW and lower on MAFW; FERV39k results are competitive.Source: §I · pp. 1–2

A. From Image-Based to Video-Based Expression Analysis

  • Earlier DFER approaches use recurrent models or 3D convolutions, but long clips challenge recurrent models and fixed receptive fields favor rapid local motion.Larger convolution kernels raise parameter count and overfitting risk on imbalanced in-the-wild collections.Source: A. From Image-Based to Video-Based… · p. 2

B. Face-Detection-Free DFER Paradigms

  • FDA-free recognition is complementary to GM-GReFEL because its geometric anchors rely on a stable, aligned facial region.Extending the reliability mechanism to unaligned frames is identified as a separate research direction.Source: B. Face-Detection-Free DFER Paradigms · p. 2

D. Selective State-Space Backbones in Vision

  • Prior video state-space work adapts selective scanning to images and video, while GM-GReFEL combines grouped linear-complexity scanning with geometric reliability for facial expression recognition.The paper states that, to its knowledge, prior work had not combined these elements.Source: D. Selective State-Space Backbones in… · p. 2

E. Label Noise and Geometric Reliability

GM-GReFEL extends geometry-grounded reliability from still images to video, using predictive entropy to determine the balance between classifier output and geometric evidence. Its pipeline also uses temporally jittered sampling and consistent spatial alignment and normalization.

E. Label Noise and Geometric Reliability

A. Overview

  • The pipeline tokenizes sampled video, processes it with Modulated GroupMamba and Channel Affinity Modulation, then blends classification with a geometry-grounded vote under Triple Loss.Initialization uses static-to-dynamic weight transfer.Source: A. Overview · p. 2, Figure 1 · p. 3

1) Jittered Temporal Sampling:

  • At test time, each clip is divided into T equal segments and the center frame of each segment is selected.The paper describes this sampling procedure as fully reproducible.Source: 1) Jittered Temporal Sampling · p. 3
  • Random temporal jitter shifts the selected frame offset while preserving temporal order, aiming to make learned motion patterns robust to small sampling shifts.The offset is drawn per clip and epoch from a discrete uniform distribution over [0, floor(Ltotal/T)-1].Source: 1) Jittered Temporal Sampling · p. 3

2) Spatial Alignment and Normalization:

  • Frames are resized after detecting the face in the first frame, and the same crop is applied across all T frames to avoid crop jitter that could resemble facial motion.Pixel intensities are standardized per channel using mean and standard deviation computed over the training split.Source: 2) Spatial Alignment and Normalization · p. 3

C. Spatiotemporal Tubelet Tokenization

GM-GReFEL converts video into a sequence of 3D tubelet tokens before applying grouped spatial and temporal state-space scans. The tubelet size halves the temporal sequence length at the input stage.

C. Spatiotemporal Tubelet Tokenization

  • Setting Pt = 2 halves the temporal sequence length at the first layer, helping keep token count and downstream memory footprint tractable for a 16-frame clip.The paper contrasts this with frame-by-frame tokenization, which would increase sequence length and restore memory pressure.Source: C. Spatiotemporal Tubelet Tokenization · p. 3
  • With 16 frames resized to 224 × 224, 3D tubelets of size 2 × 4 × 4 yield 25,088 tokens.Each flattened tubelet has 96 dimensions and is projected to D = 96 with a learned matrix and trainable positional encoding.Source: C. Spatiotemporal Tubelet Tokenization · pp. 3–4

1) Orthogonal Channel Partitioning:

  • The 96 channels are divided into six groups: four scan the frame plane for spatial texture, and two scan forward and backward along time.The two temporal pathways are described as tracking muscle-contraction velocity from onset to apex.Source: 1) Orthogonal Channel Partitioning · p. 4

2) Discretized Selective State-Space Recurrence:

The VSSS blocks use input-dependent state-space parameters for selective sequence processing, while Channel Affinity Modulation re-couples the independently scanned groups. After three layers, temporal pooling produces a clip-level feature vector.

2) Discretized Selective State-Space Recurrence:

  • The VSSS recurrence makes its step size and projection matrices input-dependent, following Mamba’s selectivity mechanism.The step size acts as a forgetting gate; for irrelevant tokens such as lighting changes or blinks, it can preserve the accumulated expression representation.Source: 2) Discretized Selective State-Space… · p. 4

3) Channel Affinity Modulation:

  • Because independent group scans leave spatial and temporal pathways isolated, Channel Affinity Modulation pools their concatenated outputs and maps them to per-channel weights.The output is gated by those weights alongside a residual SiLU-activated path.Source: 3) Channel Affinity Modulation · p. 4
  • After three stacked GroupMamba layers by default, temporal pooling collapses the sequence into a 768-dimensional feature vector summarizing the clip’s spatiotemporal signature.The default depth is L = 3.Source: 3) Channel Affinity Modulation · p. 4

E. Geometry-Aware Reliability Balancing

GReFEL adds a geometry-based check to a classifier trained on class-imbalanced, noisy labels. It uses prediction uncertainty and similarities to trainable geometric anchors to balance the two signals.

E. Geometry-Aware Reliability Balancing

  • GReFEL pairs a primary MLP classifier with a geometric track that checks predictions against trainable anchors.The anchors provide an independent, anatomically grounded check on the classifier.Source: E. Geometry-Aware Reliability Balancing · p. 5

1) Normalized Entropy as a Confidence Signal:

2) Geometric Anchor Matching:

  • Cosine similarities between video features and trainable, class-specific geometric anchors produce a pseudo-target distribution.Each anchor is an idealized structural prototype for an emotion class.Source: 2) Geometric Anchor Matching · p. 5

3) Entropy-Weighted Calibration:

The final prediction shifts between classifier probabilities and geometric-anchor votes according to classifier confidence. Training combines classification with anchor-separation and feature-to-anchor objectives.

3) Entropy-Weighted Calibration:

  • The final prediction blends classifier probabilities and the geometric-anchor target according to the network’s confidence.Low entropy favors the classifier; high entropy shifts the decision toward the anchors.Source: 3) Entropy-Weighted Calibration · p. 5

4) Triple Loss Formulation:

  • The Triple Loss combines classification, anchor orthogonality, and center loss to regularize predictions and anchor-feature geometry.Orthogonality discourages anchor collapse, while center loss pulls each feature toward its ground-truth anchor.Source: 4) Triple Loss Formulation · p. 5

F. Static-to-Dynamic (S2D) Knowledge Transfer

Static-to-dynamic transfer initializes the video model with spatial weights and anchors learned from still images. The section also describes linear token-count complexity and the class-sensitive evaluation metrics.

F. Static-to-Dynamic (S2D) Knowledge Transfer

  • Static-to-dynamic transfer inflates 2D patch-embedding weights for video and initializes spatial scanning groups from a converged 2D model.The temporal groups start near zero, while imported anchors begin from their static-model values.Source: F. Static-to-Dynamic (S2D) Knowledge… · p. 5

G. Computational Complexity and Inference Behavior

  • Modulated GroupMamba has linear complexity in token count, so doubling clip length from 16 to 32 frames doubles compute rather than quadrupling it.At inference, calibrated predictions use the network’s entropy and frozen anchors without external supervision.Source: G. Computational Complexity and… · pp. 5–6

H. Evaluation Metrics

  • WAR favors majority classes, whereas UAR weights each class’s recall equally regardless of sample count.The section identifies UAR as more informative for assessing protection of minority classes.Source: H. Evaluation Metrics · p. 6

IV. EXPERIMENTS

The DFEW component ablation identifies temporal modeling as the only component with a measurable average-accuracy effect in this setting. The experiments use five-fold evaluation and report implementation and dataset protocols.

1) Datasets:

  • Evaluation uses strict five-fold cross-validation; DFEW and MAFW report best-fold and five-fold mean results, while FERV39k reports the five-split mean.The authors treat five-fold means as like-for-like comparisons with baseline figures and best-fold results as upper bounds.Source: 1) Datasets · p. 6

2) Implementation Details:

  • Training uses the two-stage S2D protocol and AdamW, with AMP needed to fit 25,088-token sequences into 24 GB of VRAM.Experiments used one NVIDIA RTX 3090 GPU.Source: 2) Implementation Details · p. 6

B. Ablation Analysis

  • The ablation examines component contributions, statistical significance of the DFEW gain, and performance consistency across splits.Its rows are five-fold means and standard deviations, not directly comparable to the best-fold headline.Source: B. Ablation Analysis · p. 6, 1) Component Ablation · p. 6

1) Component Ablation:

  • Removing temporal modeling lowers mean DFEW UAR from 64.69% to 63.08%, while the other four variants remain within 0.3 points of the full model.The other variants’ small mean-UAR differences are comparable to fold-to-fold standard deviations of 1.5 to 2.2 points.Source: 1) Component Ablation · p. 6, Table I · p. 7
  • The authors conclude that average-accuracy effects of the anchors, model depth, training schedule, and jittered sampling are not measurable in this setting.The ablation does not test whether those components improve training stability or best-fold results.Source: 1) Component Ablation · p. 7

2) Hyperparameter Sensitivity:

The section compares GM-GReFEL with DFEW baselines, examines hyperparameter sensitivity and temporal modeling, and reports cross-fold performance and variability.

2) Hyperparameter Sensitivity:

  • Repeated default runs differ by about two UAR points, while τ = 0.10 reaches the sweep’s highest UAR at 65.20%, suggestive but not conclusive.The accelerated sweep used DFEW Fold 1; differences below about two points are within run-to-run variation.Source: 2) Hyperparameter Sensitivity · p. 7, Table II · p. 7

3) Statistical Significance:

  • On DFEW, the full model exceeded the baseline by a mean paired 1.61 UAR points across five folds (SD 1.41; t(4) = 2.56; one-sided p = 0.031), with moderate, not strong, evidence.The two-sided p-value was 0.063, and the 95% confidence interval was [−0.14, 3.36] points. The test was run on DFEW only.Source: 3) Statistical Significance · p. 7

1) DFEW:

  • 66.57% UAR and 76.46% WAR put the best DFEW fold 0.23 UAR points and 0.14 WAR points below S4D.The five-fold mean is 64.07% UAR and 74.54% WAR, 2.7 UAR points below S4D.Source: 1) DFEW · pp. 7–8

4) Cross-Fold Stability:

  • 64.07% UAR is the DFEW five-fold mean; MAFW has the widest spread (±5.65 / ±6.35), while FERV39k has the tightest (±0.84 / ±0.69).MAFW Fold 4 reaches 45.66% / 62.27%, but the other folds range from 31.51% to 43.31% UAR.Source: 4) Cross-Fold Stability · p. 7, Table III · p. 8

1) DFEW:

  • Happy recall stays between 91.2% and 94.9% across folds, while minority-class Disgust shows substantially more interfold variance.The authors attribute the minority-class variability to scarce samples in those splits.Source: 1) DFEW · p. 8

2) FERV39k:

GM-GReFEL improves on the strongest published FERV39k baseline and leads the compared methods on MAFW’s best fold, while its cross-dataset five-fold results are competitive but not uniformly ahead of large pre-trained models.

2) FERV39k:

  • 42.30% UAR raises the strongest published FERV39k baseline by 6.36 points, with 52.66% WAR across five splits.The mean is reported because FERV39k split variation is low; several large pre-trained ViT-B/16 models remain ahead.Source: 2) FERV39k · p. 8

3) MAFW:

  • 45.66% UAR and 62.27% WAR on MAFW’s best fold exceed S4D by 1.94 UAR points and 3.83 WAR points.The five-fold mean is 38.90% UAR and 54.37% WAR, below several stronger models on UAR.Source: 3) MAFW · p. 8
  • MAFW errors concentrate in ambiguous compound classes such as Surprise and Anxiety, which scatter toward anatomically adjacent basic emotions.Disgust, Fear, Disappointment, and Anger each exceed 78% true-positive rate.Source: 3) MAFW · p. 8

4) Cross-Dataset Summary:

  • GM-GReFEL’s best fold leads all listed methods on MAFW, but its five-fold mean UAR trails stronger models and its mean WAR is similar to SVFAP and MAE-DFER.The best fold reaches 45.66% UAR and 62.27% WAR; the five-fold mean is 38.90% UAR and 54.37% WAR.Source: 3) MAFW · p. 8, 4) Cross-Dataset Summary · p. 9
  • GM-GReFEL has the highest UAR and WAR among listed MAFW methods and comes within 0.3 points of S4D on DFEW (66.6% vs 66.8% UAR; 76.5% vs 76.6% WAR).Five-fold means are the like-for-like basis for most baselines, and on that basis the comparison is more balanced.Source: 4) Cross-Dataset Summary · p. 9

D. Visualization

Across datasets, training converges consistently, while illustrative attention maps show the model tracking facial regions and remaining more localized than mean pooling.

1) Training Convergence:

  • Training and validation losses stabilize across folds, with DFEW illustrating the same qualitative convergence pattern seen on all three datasets.Mean training loss plateaus near 1.5 and validation loss near 0.9, attributed to 50% dropout after the CAM layer during training.Source: 1) Training Convergence · pp. 9–10, Figure 4 · p. 10
  • UAR and WAR rise sharply during the first 20 epochs and then plateau, with narrow final-epoch variance bands across folds.The progression is shown averaged across five folds.Source: 1) Training Convergence · p. 10, Figure 5 · p. 10

2) Interpretability and Attention Analysis:

  • In an illustrative Angry clip, Grad-CAM attention shifts from the eyes toward the mouth and jawline as the expression reaches its apex.The authors describe these case studies as illustrative, not statistical evidence.Source: 2) Interpretability and Attention… · p. 11, Figure 6 · p. 11
  • On the same Surprise sample, full GroupMamba attention stays localized on the mouth and eyes, while mean-pooling attention diffuses into the background.The full-model attention remains stable across the 16-frame sequence; the authors note the anchors’ role is not isolated by the accuracy ablation.Source: 2) Interpretability and Attention… · p. 11, Figure 7 · p. 11

3) t-SNE Feature Space Analysis:

The learned features form separated clusters across datasets, while misclassification patterns vary by benchmark. The model’s inference speed is close to ViT-B/16, but its scope and parameter count constrain deployment and interpretation.

3) t-SNE Feature Space Analysis:

  • t-SNE projections show compact, well-separated clusters with little overlap across DFEW, FERV39k, and MAFW, including MAFW’s 11 compound classes.The paper relates this pattern to Triple Loss reducing intra-class variance and enlarging inter-class margins.Source: 3) t-SNE Feature Space Analysis · p. 11, Figure 8 · p. 11

4) Confusion and Misclassification Patterns:

  • On FERV39k, seven of the ten most frequent errors involve Neutral; MAFW errors instead spread across related compound-emotion pairs, with only three involving Neutral.The authors suggest expression onset and offset are a remaining difficulty on FERV39k.Source: 4) Confusion and Misclassification… · p. 11

E. Computational Efficiency

V. LIMITATIONS

  • The study is visual-only and categorical, and its reported latency still requires a laboratory-grade GPU; audio fusion, dimensional affect, and edge compression remain out of scope.The geometric anchors are tied to discrete class prototypes and cannot directly track blended or graded affect.Source: §V · p. 12

VI. CONCLUSION

  • The conclusion reports best-fold leadership on MAFW, near-parity with the strongest DFEW method, and competitive FERV39k results, with temporal modeling measurably helping DFEW accuracy.It also reports face-localized attention, separated feature clusters, and inference speed close to ViT-B/16.Source: §VI · p. 12
Loading 2610.31140v1…