PaperlayerView exact revision on arXiv

Source-linked AI summary

Survey on Emotional Body Gesture Recognition

Fatemeh Noroozi, Ciprian Adrian Corneanu, Dorota Kamińska, Tomasz Sapiński, Sergio Escalera, Gholamreza Anbarjafari

arXiv:1801.07481v1cs.CV

TL;DR

Automatic emotion recognition has concentrated on faces and speech, leaving body gestures comparatively underexplored. This survey synthesizes emotional body-gesture research, defines an end-to-end recognition framework, and reviews databases, representations, recognition methods, and multimodal approaches. It concludes that preprocessing is mature, but labelled data, output taxonomies, and learned representations remain limited.

  • Problem

    Automatic emotion recognition has focused mainly on facial expressions and speech, while recognition from body gestures remains less explored.

  • Method

    The paper presents a comprehensive survey that defines an Emotion Body Gesture Recognition pipeline and reviews pose estimation, representations, emotion recognition, databases, and multimodal methods.

  • Results

    93% recognition rate is reported for five emotion classes using neural networks, while multimodal methods combining body gestures with speech or face report improved results.

  • Takeaways & Limitations

    The survey identifies body gestures as a distinct emotion-recognition domain and organizes its methods and resources within a common framework.

  • Takeaways & Limitations

    Complex learned representations remain scarce because labelled data are limited and emotion taxonomies are inconsistent across databases.

Abstract

from arXiv · show

Automatic emotion recognition has become a trending research topic in the past decade. While works based on facial expressions or speech abound, recognizing affect from body gestures remains a less explored topic. We present a new comprehensive survey hoping to boost research in the field. We first introduce emotional body gestures as a component of what is commonly known as "body language" and comment general aspects as gender differences and culture dependence. We then define a complete framework for automatic emotional body gesture recognition. We introduce person detection and comment static and dynamic body pose estimation methods both in RGB and 3D. We then comment the recent literature related to representation learning and emotion recognition from images of emotionally expressive gestures. We also discuss multi-modal approaches that combine speech or face with body gestures for improved emotion recognition. While pre-processing methodologies (e.g. human detection and pose estimation) are nowadays mature technologies fully developed for robust large scale analysis, we show that for emotion recognition the quantity of labelled data is scarce, there is no agreement on clearly defined output spaces and the representations are shallow and largely based on naive geometrical representations.

1 INTRODUCTION

Body language conveys emotional and cognitive states, yet automatic emotion recognition has focused overwhelmingly on faces and speech rather than body gestures. The survey addresses this gap by reviewing emotional body gestures and organizing recent recognition methods into a broader framework.

  • Body language comprises nonverbal signals such as facial expressions, posture, gestures, and eye movements that communicate feelings and thoughts.
  • Understanding emotional state can improve human–machine interaction by helping machines adapt to users and cooperate more naturally.
  • About 95% of emotion-analysis literature has focused on faces, while considerably fewer studies have addressed body gestures and posture.
  • The survey reviews recent automatic emotion-recognition advances from body gestures, while treating facial and speech recognition mainly as complementary topics.
  • The paper organizes the field around emotional body gestures, a recognition pipeline, publicly available databases, and future research directions.

2 EXPRESSING EMOTION THROUGH BODY LAN-

Emotional body language draws on coordinated movements across body parts, but its interpretation depends on culture and gender-related factors. The section surveys these expressive cues and highlights the need to consider body regions together.

  • Body language expresses inner states through gaze, limb position, posture, and movement, with hands described as a particularly rich information source.
  • Head position and torso angle provide emotion-related cues, including signals associated with patience, superiority, submission, aggression, confidence, and curiosity.
  • Correct interpretation requires considering multiple body parts simultaneously, supported by behavioral protocols for six basic emotions.
  • 2.1 Culture differences: Gestures are strongly culture-dependent, and the same posture or symbol can acquire different meanings across cultures and generations.
  • 2.2 Gender differences: Body-language communication differs by gender, although reported tendencies are increasingly treated as gender stereotypes rather than fixed generalizations.

3 MODELS OF THE HUMAN BODY AND EMOTION

The paper frames body and emotion modelling as key design choices for mapping human-body dynamics to predefined affect representations. Body models range from independently detected parts to kinematic joint structures, while emotion models are categorical, dimensional, or hybrid.

  • Human-body input is abstracted as body structure and dynamics before machine-learning methods map it to a predefined emotion abstraction.
  • Part Based Models: Part-based models detect flexible body components independently and refine them with structural priors or compositional rules.
  • Kinematic Models: Kinematic models represent the body as interconnected joints with degrees of freedom, ranging from planar projections to 3D meshes.
  • Categorical models: Categorical emotion models assign affect to distinct classes such as happiness, sadness, fear, anger, disgust, and surprise.
  • Dimensional models: Dimensional models place emotions along continuous dimensions such as valence, activation, and control, but systems often simplify them into limited categories.
  • Figure 5 illustrates neural-network-based pose-estimation approaches, including LSSVM, associative embedding, CNN–Markov random field hybrids, and iterative error-feedback CNNs.

4 BODY GESTURE BASED EMOTION RECOGNI-

An Emotion Body Gesture Recognition system is organized as a pipeline whose modelling choices determine compatible datasets and components. It detects people, estimates pose, and then extracts or learns representations for emotion recognition.

  • An EBGR system first detects the body to subtract background, then detects and tracks pose to reduce posture-related variation.
  • The final pipeline stage extracts or learns a representation and maps it to a predefined emotion model using pattern-recognition methods.
  • Choosing human-body and emotion models is an early design decision that affects subsequent system components and database selection.

4.1 Human Detection

Human detection isolates people from images before pose and emotion analysis, but variation in body appearance and the speed–accuracy trade-off complicate robust detection. Methods progress from cascades and hand-crafted features to part-based and deep models.

  • Human detection estimates bounding boxes despite variation from non-rigid bodies, pose, clothing, illumination, and occlusion.
  • A conventional detection pipeline generates candidate regions, represents and classifies them, then merges positive regions into final human detections.
  • Depth information can limit the search space and simplify background subtraction when it is available.
  • Classical approaches include Viola–Jones cascades with AdaBoost, HOG shape features, and deformable part models with geometry priors.
  • Deep neural networks can achieve performance comparable to classical DPM approaches, but their speed makes pedestrian detection challenging.
  • Cascaded detection passes only high-confidence regions to deeper networks to trade detection speed against accuracy.

4.2 Body Pose Detection

Body pose detection estimates changing body-model parameters from images or frame sequences, using model fitting, learning, and temporal tracking. The field has shifted toward deep networks, while initialization, data, computation, and 3D preprocessing remain constraints.

  • Pose estimation is challenging because high-dimensional search, many degrees of freedom, clutter, illumination, and impossible body configurations must be handled.
  • Model-based methods fit expected skeletal models as inverse-kinematic problems, but they can be vulnerable to local extrema and require initialization.
  • Learning-based pose estimation is computationally expensive and requires large labeled skeletal databases.
  • DeepPose and later CNN methods shifted HPE toward direct 2D regression, multiscale heatmaps, and CNN-based 3D pose estimation.
  • Dynamic tracking estimates body position and shape across consecutive frames using iterative prediction, model updates, EM, HMMs, or dynamic Bayesian networks.
  • Voxelized 3D tracking can provide efficient and reliable data, but converting images into 3D reconstructions requires preprocessing and dedicated hardware for real-time performance.

4.3 Representation Learning and Emotion Recognition

The surveyed systems represent emotional gestures using static or dynamic geometric, motion, skeletal, appearance, and contextual features, then classify them into emotion categories or affective dimensions. Complex learned representations remain uncommon, while reported studies distinguish emotion groups using body trajectories and standard classifiers.

  • Representation Learning: Representations combine static or dynamic geometry, appearance, motion cues, skeletal features, and body-part-specific information.Examples include joint coordinates, velocities, accelerations, jerk, silhouettes, hand trajectories, and motion indices.
  • Representation Learning: Hand and head motion can be characterized through attack, release, local maxima, and peak-duration ratios to estimate movement impulsiveness.These features describe the slopes around motion extrema and the temporal structure of the largest peak.
  • Representation Learning: Complex representations are scarce, with only a few studies using HOG, bag-of-words appearance features, multichannel CNNs, or spiking neural networks.The survey contrasts these examples with the broader reliance on simpler representations.
  • Emotion Recognition: Trajectory representations of the head and hands separated emotion groups associated with four valence/arousal quadrants from frontal and lateral views.The groups were classified by their positions in the dimensional affect space.
  • Emotion Recognition: With 156 samples from three subjects, a Bayesian Net performed best among standard classifiers for six upper-body emotional categories.The categories grouped the original output space into combined emotion labels.
  • Emotion Recognition: DTW-1NN performed best for classifying dynamic gestures as anger, joy, pleasure, or sadness, while ensemble trees performed best for five skeletal-feature emotions.The two studies compared multiple conventional classifiers on different gesture representations.
  • Emotion Recognition: Fusing facial action units with body-gesture representations produced better recognition than either unimodal result.This finding supports combining body gestures with other affective modalities.

4.4 Applications

Gesture-based affect recognition is applied to detecting users’ emotions, animating conversational agents, and building systems intended to feel emotions. Multimodal face, voice, and body systems are positioned for perceptual user interfaces and affective human–machine interaction.

  • Applications: Applications include systems that detect users’ emotions, animated conversational agents, and systems intended to feel emotions.The agent category includes robots and avatars expected to act like humans when expressing a feeling.
  • Applications: Face, voice, and body gesture can be combined in perceptual user interfaces for online shops and affective human–machine systems.The surveyed examples include pervasive interfaces and machines that understand and react to human emotions.

5 DATA

The survey reviews public databases for training emotional body-gesture recognition systems, spanning RGB, depth, multimodal, upper-body, full-body, and skeletal recordings. These resources vary substantially in participants, emotions, modalities, and recording conditions.

  • Database Overview: The survey presents public affective-gesture databases and organizes them into RGB, depth, and RGB-plus-depth groups.Table 2 summarizes database characteristics, while Figure 9 shows selected samples.
  • RGB Databases: Gunes and Piccardi’s database contains 206 samples covering six basic emotions plus neutral, anxiety, boredom, and uncertainty.The split includes 156 training samples and 50 test samples.
  • RGB Databases: The HUMAINE subset contains 240 gestures from 10 participants acting eight emotions, with analyses focused on anger, joy, pleasure, and sadness.Participants included six men and four women, and the emotions were distributed across valence-arousal space.
  • RGB Databases: GEMEP contains more than 7000 audiovisual portrayals from 10 actors, while GEMEP-FERA uses short upper-body videos from training and test actors with partial subject overlap.GEMEP-FERA’s videos average 2.67 seconds and do not begin from a neutral state.
  • Depth and Skeletal Databases: Other resources include small culturally diverse gesture recordings, balanced six-emotion upper-body videos, controlled skeletal datasets, and action databases with 15-joint or 20-action annotations.Examples span participants from Japanese, Chinese, and Vietnamese nationalities; LIRIS-ACCEDE includes 64 subjects; UCFKinect records 1280 actions; MSR Action 3D contains 20 actions.
  • Multimodal Databases: emoFBVP combines face, body gesture, voice, physiological signals, facial tracking, and skeletal tracking across 23 emotions and three expression intensities.Ten professional actors contributed recordings in standing and seated positions.

6 DISCUSSION

The survey finds that emotional body gesture recognition remains constrained by limited and inconsistent data, shallow representations, and fragmented emotion taxonomies. Dynamic features and multimodal processing improve recognition, but reliable comparison across studies remains difficult.

  • 6.1 Data: Publicly available databases are few, heterogeneous, and often use acted expressions that may not reflect spontaneous real-world emotions.Acted recordings are easier to collect and label, but can be redundant, actor-dependent, and limited to basic emotions; spontaneous data are harder to evaluate and rarely public.
  • 6.1 Data: Emotion databases vary widely in their affective labels, with many emphasizing six basic emotions while omitting or inconsistently representing other states.Examples of less consistently represented states include uncertainty, unconcern, aghastness, shame, and tenderness.
  • 6.1 Data: There is no consistent taxonomy for naming affective states, making it difficult to determine whether labels such as joy and happiness denote the same or different states.Database heterogeneity also makes comparison of database quality problematic.
  • 6.2 Representation Learning and Emotion Recognition: Most recognition methods use simple geometrical features based on body-part or joint coordinates, while complex learned representations remain scarce.Only a few examples use multichannel CNNs, spiking neural networks, or richer HOG-based representations.
  • 6.2 Representation Learning and Emotion Recognition: Combining dynamic features such as acceleration, movement gain, and velocity with static features usually produces higher recognition rates than using static features alone.The survey attributes this pattern to richer representations and the fact that some emotional traits are expressed mainly through body dynamics.
  • 6.2 Representation Learning and Emotion Recognition: Multimodal systems combining body gestures with speech or facial information consistently report improvements over monomodal equivalents, although the research remains scarce and simplistic.Existing studies mainly use shallow representations and simple fusion techniques, leaving the potential of multimodal recognition largely unexplored.
  • 6.2 Representation Learning and Emotion Recognition: Reducing the number of emotion classes generally improves classifier performance, with a reported best recognition rate of 93% for five classes using neural networks.The expected trend can be violated by low-quality samples or features.
  • 6.2 Representation Learning and Emotion Recognition: Reported accuracies above 90% are often specific to particular databases, so reliability requires testing across varied backgrounds and training-testing strategies.Different evaluation protocols can produce different performance rates.

7 CONCLUSION

The survey organizes emotional body gesture recognition into a general pipeline while identifying shallow representations, scarce data, inconsistent labels, and the need for richer shared output spaces.

  • The paper defines and details a general pipeline for automatic emotional body gesture recognition.It covers person detection, body pose estimation, representation learning, and emotion recognition methods.
  • Current representations remain shallow and largely rely on naive geometrical descriptions of skeletal structures or independently detected body parts.Common features include motion cues, distances, orientations, and shape descriptors.
  • Scarce body-gesture and multimedia affective data constrain the development of deep meaningful representations for body analysis.The paper notes that this scarcity is only recently beginning to be overcome for facial analysis.
  • Public datasets use varied, redundant, or confusing label taxonomies, indicating a lack of agreement on output spaces.The paper recommends sufficiently simple and well-defined output spaces for publishing larger quantities of labelled and unlabelled data.
  • The paper argues for large, high-quality labelled and unlabelled datasets to support learning rich deep statistical representations of affective body language.This recommendation is paired with the need for clearly defined output spaces.
Loading 1801.07481v1…