Source-linked AI summary
iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis
Xin Liu, Henglin Shi, Haoyu Chen, Zitong Yu, Xiaobai Li, Guoying Zhaoz?
TL;DR
Existing emotional gesture research often uses sensitive biometric data and commonly evaluates intentionally performed gestures or isolated behavior recognition, leaving identity-free micro-gestures and holistic emotion analysis less addressed. iMiGUE introduces a dataset with two-level annotations and an unsupervised model for class imbalance. Experiments report competitive micro-gesture recognition and better emotion understanding than action/gesture-recognition models.
Problem
Emotional gesture research commonly uses sensitive biometric data and existing gesture datasets often focus on intentional behaviors or isolated behavior recognition rather than underlying emotions.
Source: §1 · pp. 1–2Method
iMiGUE is an identity-free micro-gesture dataset with clip-level gesture and video-level emotion labels, accompanied by an unsupervised model to address class imbalance.
Source: §1 · p. 2Results
Micro-gesture-based emotion analysis with a simple RNN beat action/gesture-recognition models, while fully supervised models did not achieve satisfactory micro-gesture recognition accuracy.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
We introduce a new dataset for the emotional artificial intelligence research: identity-free video dataset for Micro-Gesture Understanding and Emotion analysis (iMiGUE). Different from existing public datasets, iMiGUE focuses on nonverbal body gestures without using any identity information, while the predominant researches of emotion analysis concern sensitive biometric data, like face and speech. Most importantly, iMiGUE focuses on micro-gestures, i.e., unintentional behaviors driven by inner feelings, which are different from ordinary scope of gestures from other gesture datasets which are mostly intentionally performed for illustrative purposes. Furthermore, iMiGUE is designed to evaluate the ability of models to analyze the emotional states by integrating information of recognized micro-gesture, rather than just recognizing prototypes in the sequences separately (or isolatedly). This is because the real need for emotion AI is to understand the emotional states behind gestures in a holistic way. Moreover, to counter for the challenge of imbalanced sample distribution of this dataset, an unsupervised learning method is proposed to capture latent representations from the micro-gesture sequences themselves. We systematically investigate representative methods on this dataset, and comprehensive experimental results reveal several interesting insights from the iMiGUE, e.g., micro-gesture-based analysis can promote emotion understanding. We confirm that the new iMiGUE dataset could advance studies of micro-gesture and emotion AI.
1. Introduction
iMiGUE addresses gaps in emotional gesture research by focusing on identity-free, unintentional micro-gestures and holistic emotion understanding. It introduces a two-level annotated dataset and reports that micro-gesture-based analysis can improve emotion understanding.
1. Introduction
- Most gesture datasets assess recognition of intentionally performed illustrative behaviors, while micro-gestures are unintentional behaviors linked to inner feelings and may reveal suppressed emotions.The paper notes that no public computer-vision dataset for emotional micro-gesture research was available to the authors’ knowledge.Source: §1 · pp. 1–2
- Existing emotion datasets often involve sensitive biometric data, whereas iMiGUE explores identity-free micro-gestures for hidden-emotion understanding while preserving individuals’ privacy.Source: §1 · p. 2
- The paper proposes an unsupervised model to address the dataset’s imbalanced class distribution.Source: §1 · p. 2
- iMiGUE links clip-level micro-gesture categories with video-level emotion categories, enabling holistic exploration of their relationship rather than isolated gesture recognition.Source: §1 · p. 2
- A simple RNN’s holistic micro-gesture analysis beat action/gesture-recognition models for emotion understanding, while fully supervised models did not achieve satisfactory micro-gesture recognition accuracy.The proposed unsupervised method achieved competitive performance with many supervised models.Source: §1 · p. 2
2. Related Work
Prior work spans posed and spontaneous emotional gestures, alongside action-recognition methods that learn from labeled data or video sequences. The review positions fine-grained micro-gestures as comparatively underexplored.
2. Related Work
- Body-activity datasets commonly target action recognition rather than emotional states, motivating the review’s focus on emotional gesture benchmarks.Source: §2 · p. 2
2.1. Emotional Gesture-based Datasets
- Emotional gesture datasets progressed from posed, controlled recordings to spontaneous or in-the-wild material, but few studies address fine-grained micro-gestures linked to concealed emotions.The reviewed examples include acted datasets, movie and game-video gestures, and in-the-wild perceived emotions.Source: §2.1 · pp. 2–3
2.2. Methods for Gesture/Action Recognition
- Gesture/action recognition includes supervised 2D and 3D CNNs, temporal RNNs, and skeleton-based RNN or GCN methods; unsupervised approaches use temporal learning or sequence reconstruction.The review notes that P&C uses element-wise reconstruction loss without an informative constraint or prior and fixed-length inputs that may lose long-term motion information.Source: §2.2 · p. 3
3. The iMiGUE dataset
iMiGUE is an identity-free dataset of spontaneous micro-gestures, built to link clip-level gesture categories with video-level emotions. It uses post-match press conferences and a two-level annotation scheme, with match outcomes as emotion labels.
3.1. Key Challenges
- Category design draws on psychological studies to distinguish micro-gestures from illustrative behaviors, and Figure 2 shows categories, sample percentages, and masked-face examples.The dataset also treats collecting spontaneous instances and identifying their emotional context as construction challenges.Source: §3.1 · pp. 3–4, Figure 2 · p. 4
3.2. Dataset Construction
- Grand Slam press conferences provide spontaneous gestures; clips receive micro-gesture labels and videos receive positive or negative emotion labels based on match outcomes.Five annotators were trained, each clip was labeled by two people, and average interlabeler reliability was 0.81.Source: §3.2 · pp. 4–5
3.3. Dataset Statistics and Properties
- iMiGUE contains 359 post-match press-conference videos and 18,499 labeled micro-gesture samples across 32 categories, with a long-tailed category distribution.The videos cover 258 wins and 101 losses; the emotion labels use winning and losing as objective references.Source: §3.3 · p. 5
4. Unsupervised Learning for MG Recognition
To address long-tailed class imbalance, the paper proposes an unsupervised sequence encoder-decoder that learns latent representations from complete micro-gesture pose sequences. Its joint objective combines reconstruction with a KL-divergence term.
4. Unsupervised Learning for MG Recognition
- The method targets iMiGUE's long-tailed class imbalance and learns representations without human-labeled data, while reading complete sequences rather than fixed-length portions.The paper identifies imbalance as a challenge for fully supervised models.Source: §4 · p. 6
- The U-S-VAE encodes a complete body-keypoint sequence with a multi-layer BLSTM into a final state, then uses an LSTM decoder to reconstruct the sequence.The sequential VAE is described as capturing latent patterns from the whole input sequence.Source: §4 · p. 6
- Training combines element-wise regeneration loss with KL divergence, which the authors use to encourage mutual information between input and regenerated sequences.The decoder receives zeros after initialization, making it rely on the encoder's final state for prediction.Source: §4 · p. 6
5. Experiments
Experiments evaluate iMiGUE at clip-level micro-gesture recognition and video-level emotion understanding, with additional tests of U-S-VAE on another dataset and across model variants.
5.1. Benchmark Evaluations
- Evaluations use a 37-subject training and 35-subject test split for clip recognition, and separate video-level training and test sets for emotion classification.The video sets contain 102 training videos and 100 test videos, balanced between wins and losses.Source: §5.1 · pp. 6–7
5.2. Implementation details
- U-S-VAE uses a one-layer bidirectional LSTM encoder and a one-layer LSTM decoder, trained until loss stabilizes.Each layer has 256 units; experiments ran on a Titan RTX GPU, with configurations following original papers unless stated otherwise.Source: §5.2 · p. 7
5.3. Clip-level Micro-gesture Recognition
- Almost all clip-recognition methods remain below 60% Top-1 accuracy; TSM and TRN lead by large margins, while U-S-VAE is competitive with supervised 3DCNN and RNN models.The authors also report that 3DCNN and RNN-based models remain below 35% Top-1 accuracy, and U-S-VAE outperforms P&C.Source: §5.3 · p. 7, Table 2 · p. 7
5.4. Video-level Emotion Understanding
- Micro-gesture-based analysis helps emotion understanding: TSM + LSTM achieves the best score, while U-S-VAE + LSTM beats most compared methods.Direct RGB and pose methods reach about 53% emotion-classification accuracy for TSM, TRN, ST-GCN, and MS-G3D; I3D scores best among those direct methods.Source: §5.4 · p. 8, Table 3 · p. 8
5.5. Analysis and Discussion
- U-S-VAE 1L outperforms P&C on cross-subject evaluations of iMiGUE and NTU RGB+D, and performs best among the tested U-S-VAE variants on both datasets.P&C does better in cross-view evaluation, which the authors attribute to its view-invariant preprocessing and additional feature-level autoencoder; U-S-VAE hidden states also outperform its cell states.Source: §5.5 · p. 8, Table 4 · p. 8
6. Conclusions
The paper introduces iMiGUE, a dataset for micro-gesture study that also supports using micro-gestures to understand emotional states. It identifies future work on relationships between micro-gesture groups and emotions.
- iMiGUE investigates emotional-state understanding from micro-gestures alongside micro-gesture recognition.The authors hope the dataset and investigations will facilitate advances in emotion AI.Source: §6 · p. 8
- Future work will study relationships between micro-gesture groups and emotional states and explore more sophisticated models for their latent mapping.The paper aims to address these relationships more holistically.Source: §6 · pp. 8–9