Source-linked AI summary
Emotion Intensity Matters: Generating Realistic Expressions in Virtual Humans with CVAEs
Vitor Miguel Xavier Peres, Lara Volpato, Gabriel Ferri Scnheider, Soraia Raupp Musse
TL;DR
Generating naturalistic, intensity-varying facial expressions for virtual humans is difficult, especially from limited real-expression data. The paper trains a CVAE conditioned on emotion, intensity, and gender, then applies residual enhancement to generated facial dynamics. The resulting system preserves key expressive characteristics across intensity levels and produces coherent emotional variations while supporting controllable synthesis.
Problem
Generating naturalistic facial expressions that vary in emotional intensity remains a central challenge for virtual humans.
Method
A CVAE conditioned on emotion, intensity, and gender generates MetaHuman facial expressions, followed by LSTM-based residual enhancement of facial dynamics.
Results
The model preserves key expressive characteristics across intensity levels and generates coherent emotional variations, while residual enhancement generally increases correspondence with real data.
Takeaways & Limitations
The framework supports controllable emotional expression synthesis for virtual humans from relatively small real-expression datasets without actor-specific supervision or manual artistic authoring.
Abstract
from arXiv · showhide
Generating expressive facial behavior in virtual humans (VHs) remains a central challenge in affective computing and character animation. This paper presents a novel approach based on Conditional Variational Autoencoders (CVAEs), trained on real human facial expression data, to synthesize controllable emotional expressions at varying intensities. Using a dataset comprising six basic emotions represented at two intensity levels (low and high), we train a CVAE model to generate synthetic facial expression data while preserving semantic consistency with real human expressions. Despite the limited amount of training data (only 7,680 facial expression samples), the proposed approach learns meaningful latent representations and generates coherent emotional variations. Our method enables control over emotional intensity, making it suitable for animating virtual characters without requiring actor performances or manual artistic intervention. Our research aimed to evaluate whether the method (CVAE) preserves the characteristics associated with the different intensity levels present in the dataset. Results show that the proposed model preserves key expressive characteristics across intensity levels while supporting generalization across emotional intensity levels, contributing to the creation of emotionally expressive virtual characters from relatively small datasets.
I. INTRODUCTION
The paper addresses the challenge of generating naturalistic facial expressions for virtual humans, particularly expressions that vary dynamically in emotional intensity. It proposes a CVAE-based framework conditioned on emotion, intensity, and gender to generate controllable expressions from limited real facial-expression data.
- Generating expressive facial behavior remains difficult because emotional expressions are dynamic and vary in intensity.
- The CVAE is trained on six basic emotions represented at low and high intensity levels.
- Conditioning on emotion, intensity, and gender enables controllable synthesis while preserving semantic consistency across emotional states.
- The method targets learning expressive facial representations from limited real-world training data.
- It generates expressions across intensity levels without artistic authoring or actor-specific supervision.
- A residual enhancement framework recovers facial dynamics and emotional intensity through temporal residual learning.
II. RELATED WORK
Related work establishes latent-space generative modeling and residual learning as foundations for facial synthesis and refinement. Prior conditional generative approaches support multimodal outputs, while residual networks provide a framework for learning corrections relative to inputs.
- VAEs use structured, continuous latent spaces that support smooth exploration and diverse data generation.
- These latent-space properties are relevant to modeling subtle facial-expression intensity variations and generalizing to unseen data.
- CVAEs extend VAEs for structured prediction by modeling complex conditional distributions and generating multiple plausible outputs from the same input.
- Residual learning models functions relative to the input, helping address degradation in deep networks and facilitating optimization and convergence.
III. METHODOLOGY
The methodology extracts MetaHuman facial parameters from acted-expression videos, trains a conditional VAE, and applies its outputs to synthesize new expressions. An LSTM residual stage then predicts corrective facial dynamics for the synthetic data.
- The study collects 864 videos from MEAD and RAVDESS featuring six emotions at low and high intensity levels.
- Live Link Hub processes videos into frame-by-frame facial capture data containing head pose, landmarks, and animation curves.
- Extracted MetaHuman facial parameters form the CVAE training dataset and numerically represent facial controls over time.
- The trained CVAE generates novel MetaHuman parameters conditioned on emotion, gender, and intensity labels, which are applied directly to a MetaHuman model.
- An LSTM models differences between real and CVAE-generated expressions and applies predicted corrective dynamics to the original synthetic data.
A. Dataset Preprocessing (1)
The preprocessing pipeline converts controlled audiovisual recordings into compact MetaHuman facial-control sequences and selects highly expressive frames for CVAE analysis. Figure 1 places this capture and preparation stage before model training and expression generation.
- The source datasets contain actors expressing six emotions at low, medium, and high intensity, but this study uses front-view captures and two intensity levels.
- Live Link Hub converts videos into frame-by-frame facial capture data, from which MetaHuman facial controls are exported as numerical sequences.
- The method overview places dataset acquisition, capture preparation, and MetaHuman dataset construction before CVAE training and generation.
- Each dataset row represents an animation frame, while the 220 facial controls represent the facial motion attributes used for training.
- 7,680 frames were retained by selecting the 20 highest-scoring frames from each of 384 videos.The score sums the absolute values of the 220 facial-control parameters; higher scores indicate stronger facial activations.
C. CVAE Training (4)
The CVAE conditions facial-expression generation on semantic labels and reconstructs 220 facial controls from latent representations. Its training uses a fully connected encoder-decoder with normalized outputs and specified optimization settings.
- The CVAE conditions generation on emotion, intensity, gender, and optionally video identity to support controllable facial-expression synthesis.
- The decoder reconstructs 220 facial control values from a latent-space vector using two fully connected hidden layers and a 220-neuron output layer.
- The model was trained with Adam at a 0.0001 learning rate, batch size 32, and 400 epochs.
- The training data represent facial behavior through 220 control parameters, with Figure 2 illustrating a subject used for CVAE training.
E. Residual Check (6) and Enhancement (7)
The enhancement stage models residual differences between real and CVAE-generated facial expressions with an LSTM. The enhanced outputs are presented as more dynamically expressive than standard CVAE generations.
- The residual stage computes differences between real and synthetic facial-control sequences and trains an LSTM to predict corrective facial dynamics.
- The residual-learning design targets the smoothing of CVAE outputs by learning corrections rather than reconstructing the complete facial-expression signal.
- Table I reports average p-values and standard deviations for comparisons between real and synthetic data across emotion, gender, and intensity conditions.
- Table II reports the corresponding comparisons between real and enhanced synthetic data across emotion, gender, and intensity conditions.
- Figure 2 shows the enhanced model producing more pronounced mouth deformations than the standard CVAE while generating new expressions from conditioning labels.
IV. EXPERIMENTAL RESULTS
The experiments compare real, synthetic, and enhanced facial-control data using statistical tests and Py-Feat emotion classification. Qualitatively, enhancement generally amplifies facial dynamics, particularly around the mouth.
- The evaluation compares Unreal facial-control values from real and synthetic datasets using independent-samples t-tests and also applies Py-Feat emotion classification.
- Table III organizes average p-values and standard deviations for synthetic-versus-enhanced data by emotion, gender, and intensity level.
- The enhancement stage generally amplifies facial dynamics, with the clearest qualitative effect appearing in mouth-related expressions.
A. Comparison Between Real and Generated and Enhanced Generated Data
Statistical comparisons aggregate 220 facial-control tests through average p-values. The CVAE shows moderate correspondence with real data before enhancement, while residual enhancement increases correspondence across evaluated conditions.
- The analysis averages 220 independent Student’s t-test p-values for each emotion, intensity, and gender combination to measure aggregate correspondence.
- All average p-values exceed 0.05, indicating no statistically significant differences between the compared real and synthetic datasets.
- Before enhancement, equivalence values range from approximately 0.27 to 0.40, indicating moderate correspondence between synthetic and real datasets.
- After enhancement, equivalence values increase consistently across evaluated conditions, effectively doubling relative to the first generation.
- Figure 2 compares real, CVAE-rendered, and enhanced-CVAE faces across six emotions and two intensity levels for two individuals.
B. Comparison Generated vs Enhanced Generated
Residual enhancement generally improves correspondence and emotion recognition relative to the original CVAE outputs, while preserving emotional identity across low- and high-intensity expressions.
- High equivalence values between CVAE and enhanced synthetic data ranged from approximately 0.58 to 0.70 across emotions, genders, and intensity levels.
- Recognition performance generally decreased after CVAE generation but improved substantially after residual enhancement.
- Anger: 75.00% recognition for men and 56.25% for women followed enhancement of anger expressions.
- Disgust: 31.25% recognition for men and 62.50% for women followed enhancement of disgust expressions, decreasing for men but improving for women.
- Fear: 50.00% recognition for men and 37.50% for women followed enhancement of fear expressions, both below the CVAE rate of 56.25%.
- Happy: 100.00% recognition for men and 81.25% for women followed enhancement of happy expressions, compared with 93.75% and 87.50% for CVAE outputs.
- Surprise: 68.75% recognition for men and 50.00% for women followed enhancement of surprise expressions, improving especially for men from 43.75%.
C. Emotions Distances by Intensities Levels
Real expressions had the highest estimated intensity, whereas baseline CVAE expressions were attenuated across emotions and intensity levels.
- Real facial expressions consistently exhibited the highest average intensity across emotions and target intensity levels.
- Baseline CVAE-generated expressions showed substantially lower intensity across all emotions and levels, indicating generally attenuated outputs.
V. FINAL CONSIDERATIONS
The proposed CVAE framework generates controllable emotional facial behavior from limited real-performance data, while residual enhancement improves correspondence and recognizability. Objective evaluation supports the approach, but human perceptual validation remains future work.
- The method models six basic emotions at two intensity levels with CVAEs trained on real facial expression data.
- The framework synthesizes expressive facial behaviors without actor-specific supervision, motion-capture transfer, or manual artistic authoring.
- 7,680 facial expression samples supported learning meaningful emotional representations while maintaining controllability across emotions and intensity levels.
- Residual temporal learning compensates for CVAE smoothing by recovering expressive dynamics and increasing correspondence without introducing entirely new expression patterns.
- Emotion recognition supported the findings because enhanced expressions generally recovered or improved recognizability across most emotion categories.
- Future human-participant perceptual studies will assess realism, naturalness, emotional intensity, and recognizability alongside objective measures.