PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Micro-action Recognition: Dataset, Methods, and Applications

Dan Guo, Kun Li, Bin Hu, Yan Zhang, Meng Wang

arXiv:2403.05234v2cs.CV

TL;DR

Micro-actions offer cues about mental states, but their recognition and use in emotion analysis remain limited. The paper introduces a whole-body interview dataset and a recognition benchmark, then reports improved emotion recognition when micro-actions are captured.

  • Problem

    Micro-actions can provide nonverbal cues to mental state, while their use in emotion analysis remains limited relative to psychological theory.

    Source: §I · p. 1, §VI · p. 10

  • Method

    The paper builds the whole-body MA-52 dataset from psychological interviews and benchmarks MANet, which combines SE and TSM with a joint-embedding loss.

    Source: §I · p. 2, C. Our Method · pp. 6–7

  • Results

    MANet outperforms existing methods on micro-action recognition, and experiments show that capturing micro-actions significantly enhances emotion recognition.

    Source: §I · p. 2

  • Takeaways & Limitations

    The experiments support the utility of the proposed data and method for emotion analysis.

    Source: §I · p. 2

Abstract

from arXiv · show

Micro-action is an imperceptible non-verbal behaviour characterised by low-intensity movement. It offers insights into the feelings and intentions of individuals and is important for human-oriented applications such as emotion recognition and psychological assessment. However, the identification, differentiation, and understanding of micro-actions pose challenges due to the imperceptible and inaccessible nature of these subtle human behaviors in everyday life. In this study, we innovatively collect a new micro-action dataset designated as Micro-action-52 (MA-52), and propose a benchmark named micro-action network (MANet) for micro-action recognition (MAR) task. Uniquely, MA-52 provides the whole-body perspective including gestures, upper- and lower-limb movements, attempting to reveal comprehensive micro-action cues. In detail, MA-52 contains 52 micro-action categories along with seven body part labels, and encompasses a full array of realistic and natural micro-actions, accounting for 205 participants and 22,422 video instances collated from the psychological interviews. Based on the proposed dataset, we assess MANet and other nine prevalent action recognition methods. MANet incorporates squeeze-and excitation (SE) and temporal shift module (TSM) into the ResNet architecture for modeling the spatiotemporal characteristics of micro-actions. Then a joint-embedding loss is designed for semantic matching between video and action labels; the loss is used to better distinguish between visually similar yet distinct micro-action categories. The extended application in emotion recognition has demonstrated one of the important values of our proposed dataset and method. In the future, further exploration of human behaviour, emotion, and psychological assessment will be conducted in depth. The dataset and source code are released at https://github.com/VUT-HFUT/Micro-Action.

I. INTRODUCTION

The introduction frames micro-actions as subtle, informative nonverbal cues and describes challenges in recognizing them. It presents MA-52 and MANet as contributions, alongside an emotion-analysis application.

I. INTRODUCTION

  • Micro-actions are brief, low-intensity movements that can convey feelings and intentions, but subtle visual changes, similar categories, long-tailed distributions, and limited data challenge recognition.The described temporal span is 1/25s to 1/3s.Source: §I · pp. 1–2
  • Prior generic action datasets emphasize coarse activities, while fine-grained datasets often focus on specialized scenarios rather than broad micro-action coverage.The introduction contrasts examples such as running and jumping with finer variations such as raising the head and nodding.Source: §I · p. 2
  • MA-52 provides 52 whole-body micro-action categories, seven body-part labels, and 22,422 videos from 205 participants recruited through psychological interviews.The interview approach encourages natural, spontaneous behavior, including lower-body movements and complex bodily interactions.Source: §I · p. 2
  • The paper introduces MANet and evaluates it alongside nine prevalent action-recognition methods; MANet combines SE and TSM with a ResNet backbone and a joint-embedding loss.The loss constrains semantic distance between video data and action labels.Source: §I · p. 2
  • The authors also investigate micro-action recognition for emotion analysis using an augmented MA-52-Pro dataset and a dual-path MANet-based network.They report that capturing micro-actions significantly enhances emotion recognition.Source: §I · p. 2

II. RELATED WORK

Related work spans coarse and fine-grained action datasets and methods, while micro-action recognition remains constrained by data acquisition, labeling, and subtle behavioral distinctions.

A. Tasks and Datasets

  • Generic action datasets use coarse labels, while fine-grained datasets target narrower action subclasses and specialized behavioral patterns.Micro-actions additionally require sensitivity to subtle visual changes and distinctions between similar categories.Source: A. Tasks and Datasets · p. 3
  • Existing micro-action datasets include iMiGUE, SMG, and BBSI, but differ in participant numbers, settings, behavior coverage, and potential authenticity constraints.The paper contrasts sports press conferences, prompted storytelling, and group conversations with its psychological-interview collection of 205 participants and 52 categories.Source: A. Tasks and Datasets · p. 3

B. Approaches

  • Generic and fine-grained recognition methods include temporal networks, transformers, attention mechanisms, and frameworks for aggregating body-region or spatiotemporal features.The cited approaches target generic actions or fine-grained distinctions across specialized tasks.Source: B. Approaches · p. 3
  • Micro-action approaches address dataset-specific challenges: an unsupervised encoder-decoder for iMiGUE imbalance, skeleton and RGB comparisons for SMG, and temporal modeling for BBSI.For SMG, the cited study reports that compact GCN-based skeleton models outperform RGB-based counterparts.Source: B. Approaches · pp. 3–4

C. Applications

The paper situates micro-actions in human interaction and emotion applications, then describes an interview-based collection process and MA-52’s scale, labels, and participant profile.

C. Applications

  • Prior work applies micro-actions to driver gestures, authentication, extended reality, and emotion analysis using facial cues, upper-limb actions, or holistic body information.The cited applications connect body movement and posture variation with understanding human emotions.Source: C. Applications · p. 4

A. Data Collection and Annotation

  • The collection approach uses professional psychological interviews to encourage spontaneous behavior, addressing authenticity concerns associated with prompted or uncontrolled recordings.Counselors use open-ended questions and follow-ups; seated whole-body recording includes lower-limb and inter-body-part movements.Source: A. Data Collection and Annotation · p. 4

B. Dataset Statistics and Properties

  • MA-52 contains 205 participants and 22,422 videos recorded at 1920×1080 and 30 fps, with video durations from 1 to 7 seconds and a 2:1:1 train-validation-test split.Its labels comprise 52 fine-grained actions grouped under seven coarse body-part categories.Source: B. Dataset Statistics and Properties · p. 5
  • The dataset emphasizes seated whole-body coverage, including lower-limb actions and bodily interactions, while its participant pool spans ages and genders.The authors note that long-tailed action distributions persist despite efforts to make the participant sample representative.Source: B. Dataset Statistics and Properties · pp. 5–6

IV. METHODOLOGIES

The methodology targets classification of micro-actions from video and groups fine-grained action labels under coarse body-part labels. It motivates a ResNet-based approach with SE and TSM for subtle movements.

IV. METHODOLOGIES

  • Because similar micro-actions are difficult to distinguish, the methodology combines a ResNet backbone with SE and TSM for recognition.The passage introduces these components as the basis for the proposed methodology.Source: §IV · p. 6

A. Task Definition

  • The task is to classify the micro-actions in a video sequence V from a set of micro-action labels.The input is represented as a sequence of frames, with T denoting video length.Source: A. Task Definition · p. 6
  • Each fine-grained micro-action is affiliated with a coarse-grained body-part category, which can be identified by the recognized action label.The task definition distinguishes the number of action categories from the number of body-part categories.Source: A. Task Definition · p. 6

B. General Action Recognition Benchmarks

The section compares nine generic action-recognition algorithms and presents MANet, which combines ResNet with temporal SE and TSM and uses an embedding loss to align video features with action labels.

B. General Action Recognition Benchmarks

  • The benchmark evaluates nine algorithms: three 2D CNNs, three 3D CNNs, and three transformer-based methods.The methods are evaluated for generic action recognition on the micro-action task.Source: B. General Action Recognition Benchmarks · p. 6
  • Each benchmark algorithm receives a fully connected action-classifier layer that outputs probabilities over the micro-action categories.The classifier uses the algorithm’s video representation vector as input.Source: B. General Action Recognition Benchmarks · p. 6

C. Our Method

  • MANet combines a ResNet backbone with temporal SE for channel-wise excitation and TSM for temporal information exchange.The network models micro-actions using both spatial and temporal learning modules.Source: C. Our Method · p. 6
  • MANet uses four ResNet blocks and outputs a video feature representation after spatiotemporal modeling.The described blocks include SE and TSM operations; the TSM stage splits features into eight chunks for temporal shifting.Source: C. Our Method · pp. 6–7
  • An embedding loss aligns video features with GloVe action-label vectors in a joint space using Euclidean distance.The loss is combined with cross-entropy; α balances the loss terms.Source: C. Our Method · p. 7

V. EXPERIMENTS

The evaluation uses accuracy and F1 measures for coarse- and fine-grained labels. Its F1 variants provide different views of performance across the long-tailed action categories.

Evaluation Metrics

  • Acc-Top1 and Acc-Top5 are computed for both coarse- and fine-grained action labels.The evaluation also reports F1 scores.Source: A. Evaluation Metrics · p. 7
  • F1micro summarizes true positives, false negatives, and false positives across samples, while F1macro weights action categories equally.The paper uses multiple F1 variants because the dataset has a long-tailed category distribution.Source: A. Evaluation Metrics · p. 7
  • The metric notation distinguishes F1 measures for the dataset’s coarse- and fine-grained action labels.The passage introduces a mean F1 calculation across the label levels.Source: A. Evaluation Metrics · p. 7

B. Implementation Details

The implementation samples eight resized frames per video and trains MANet with specified architecture and optimization settings.

B. Implementation Details

  • Each video is represented by eight frames resized to 224×224, based on an average duration of 1.9 seconds.The frame count was selected empirically.Source: B. Implementation Details · p. 7
  • MANet uses ResNet with SE ratio 4 and TSM shift length 1/8, and sets the loss-balancing hyperparameter α to 5.Source: B. Implementation Details · p. 7
  • Training uses SGD with learning rate 0.001, momentum 0.9, weight decay 1e-4, batch size 10, and 80 epochs.The learning rate is reduced tenfold at epochs 30 and 60.Source: B. Implementation Details · p. 7

C. Experimental Analysis

MANet leads the MA-52 benchmark comparison, while fine-grained recognition remains harder than coarse-grained recognition. Ablations support the contributions of label embedding, loss balancing, and sampling eight frames per video.

C. Experimental Analysis

  • MANet outperforms UniFormer by 1.16% on F1mean and TSM by 4.20% on F1mean, setting new benchmarks across major metrics.The comparison covers coarse- and fine-grained labels.Source: C. Experimental Analysis · p. 7, Table II · p. 8
  • Fine-grained recognition scores are consistently lower; MANet’s F1macro is 72.87% coarse-grained versus 49.22% fine-grained.Fine-grained body-part Acc-Top1 is below 61.33%, compared with above 78.95% coarse-grained.Source: C. Experimental Analysis · p. 7
  • Removing label embedding or SE lowers MANet’s F1mean by 2.04% or 0.82%, respectively, across fine- and coarse-label settings.The authors attribute the embedding component’s value to distinguishing similar but distinct micro-actions.Source: C. Experimental Analysis · p. 8, Table V · p. 8
  • At α=5, F1mean with fine and coarse labels is optimal at 65.59%.α balances the classification and embedding losses; α=0, which removes embedding, gives the lowest performance across metrics.Source: C. Experimental Analysis · p. 8, Table IV · p. 8
  • With T=4, fine-label Acc-Top1 drops 6.76% versus T=8; T=16 lowers F1mean from 65.59 to 64.66.The authors select T=8 as a balance that retains important information while reducing computing costs.Source: C. Experimental Analysis · p. 8, Table III · p. 8

D. Visualization Examples

Visualizations show MANet’s category features cluster more clearly than TSM’s, and its predictions generally align with ground truth. Examples also illustrate that similar micro-actions remain difficult to distinguish.

D. Visualization Examples

  • MANet’s t-SNE features cluster coarse- and fine-grained categories more clearly than TSM’s, including separate clusters for nodding and head up.The plotted points represent video instances, with colors indicating micro-action categories.Source: D. Visualization Examples · pp. 8–9, Figure 5 · p. 9
  • MANet’s coarse- and fine-grained confusion matrices are predominantly diagonal, but turning around and shrugging are among the misclassified actions.The authors suggest approximate inter-class differences and insufficient training data may contribute to these errors.Source: D. Visualization Examples · p. 9, Figure 6 · p. 9
  • Qualitative examples show MANet distinguishing actions such as touching nose and patting legs where TSM predicts alternatives, though subtle action differences remain challenging.The examples also contrast hands touching fingers with TSM’s incorrect coarse- and fine-grained predictions.Source: D. Visualization Examples · pp. 9–10, Figure 4 · p. 9

VI. APPLICATION IN EMOTION ANALYSIS

The paper applies micro-action recognition to emotion analysis, where facial-expression approaches face practical constraints and micro-action methods remain limited. It extends the data and uses a dual-branch MANet framework for joint emotion and multi-label micro-action prediction.

VI. APPLICATION IN EMOTION ANALYSIS

  • The application addresses emotion analysis, where facial-expression methods face resolution, illumination, viewing-angle, and privacy constraints, while micro-action recognition remains limited.Prior work suggests micro-action cues can offer insights for emotion analysis.Source: §VI · p. 10

A. Data Preparation and Methodology

  • MA-52-Pro extends MA-52 for multi-label micro-action and emotion recognition, with emotion shifts frequently accompanied by multiple simultaneous micro-actions.Instances are sequentially annotated with an emotion label followed by a micro-action label.Source: A. Data Preparation and Methodology · p. 10
  • MA-52-Pro contains 7,818 instances and 29,503 micro-action labels, with five emotion categories and instances bearing 1–15 micro-actions.Emotion annotations use holistic participant data; the dataset describes spontaneous emotional responses and comprehensive annotations.Source: A. Data Preparation and Methodology · p. 10
  • The unified framework combines facial and micro-action MANet branches for emotion prediction and uses a two-layer transformer to model long-term micro-action associations.Training adds emotion cross-entropy and multilabel-action binary cross-entropy losses, weighted by β=0.03.Source: A. Data Preparation and Methodology · p. 10

B. Experiment Analysis of Application

The application experiments show strong emotion-recognition results, while multi-label micro-action recognition remains difficult for videos with many simultaneous actions. Lower-limb cues contribute to emotion recognition, and the visual examples show both the value of micro-action cues and limits in recognizing co-occurrences.

B. Experiment Analysis of Application

  • In two examples, the full model uses micro-action cues to correctly distinguish anger from surprise and joy from sadness when facial expressions alone are insufficient.Figure 8 compares the Face-only, Micro-action-only, and full-model emotion predictions.Source: Figure 8 · p. 11, B. Experiment Analysis of Application · p. 12
  • The examples show successful emotion prediction and key-action detection in some videos, but many simultaneous actions in extended videos remain challenging.Figure 9 presents four samples with predicted micro-actions, ground-truth labels, and emotion probabilities.Source: Figure 9 · p. 11, B. Experiment Analysis of Application · p. 12
  • The model learns co-occurrences from head and upper-limb actions, but struggles with full-body, lower-limb, and head-hand co-occurrences.The confusion matrices concern fine- and coarse-grained categories on the MA-52-Pro test set.Source: Figure 10 · p. 11, B. Experiment Analysis of Application · p. 11
  • 62.38% Acc, 58.49% F1weight, 41.75% UF1, and 42.88% UAR: the proposed method achieves the best emotion-recognition results.The comparison includes emotion-recognition methods; implementation samples 16 frames per video with β=0.03.Source: B. Experiment Analysis of Application · pp. 10–11
  • 15.40% micro-action recognition mAP, with incomplete prediction especially when many actions occur simultaneously in long videos.The authors characterize this result as acceptable, while noting that predicting all action categories remains difficult.Source: B. Experiment Analysis of Application · p. 11
  • Removing lower-limb actions drops Acc Avg 1.73%, F1weight 1.59%, and UAR 1.23%, while mAP increases.Lower-limb actions account for 5,379 labels (18.23% of the total) and a non-negligible share across all five emotion categories.Source: Table VII · p. 12, B. Experiment Analysis of Application · p. 12

VII. FURTHER RESEARCH DIRECTION

The paper outlines future directions spanning composite and multimodal recognition, combining micro-expressions with micro-actions, data generation, and more realistic settings. Its conclusion presents MA-52 and MANet as a whole-body dataset and recognition approach, and reports their application to emotion recognition.

VII. FURTHER RESEARCH DIRECTION

  • Future work should address composite actions, combine modalities such as audio and depth, and jointly study micro-expressions and micro-actions.The paper frames these as directions for recognizing complex behaviors and understanding the linkage between micro-expressions and micro-actions.Source: §VII · p. 12
  • Lower-limb micro-actions account for a non-negligible proportion of micro-actions used for emotion recognition, as illustrated by comparing full labels with labels excluding lower-limb actions across emotion categories.Source: Figure 11 · p. 12
  • The authors propose exploring generated micro-action data and improving robustness and generalization under real-world noise, including occlusion and camera shake.They note that generating data could increase its variety and volume, addressing the expense of traditional collection and annotation.Source: §VII · p. 12

VIII. CONCLUSION

  • The conclusion presents MA-52 as a whole-body dataset and MANet as a CNN combining SE and TSM within ResNet, applied to emotion recognition alongside multi-label action recognition.The paper reports that experiments demonstrate dataset reliability and method effectiveness, including proficiency in emotion-analysis applications.Source: §VIII · pp. 12–13
Loading 2403.05234v2…