Source-linked AI summary
Capturing Dynamics: The 4D Facial Expression Intensity Dataset
Zesheng Wang, Alexandre Bruckert, Pierre Lebreton, Patrick Le Callet, Yante Li, Guoying Zhao
TL;DR
Frame-level 2D research and existing 4D datasets leave continuous expression intensity underrepresented, although expressions unfold through dynamic 3D cues. The paper introduces 4D-FEID, a dataset with human intensity ratings, and reports that difference-aware dynamic 3D models outperform 2D and parametric baselines.
Problem
Prior work primarily estimates intensity from frame-level 2D representations, while existing 4D datasets often emphasize categories or identity rather than continuous intensity.
Source: §I · p. 1Method
The paper constructs 4D-FEID with mesh sequences, corresponding FLAME parameters, rendered videos, and human-annotated intensity ratings.
Source: §III · p. 2, §V · p. 5Results
Dynamic 3D representations with difference-aware learning outperform traditional 2D and parametric baselines; 4D-Vertex MAE is 0.139 versus 0.192 for the FLAME-based baseline.
Source: §V · p. 5, D. Results and Discussion · p. 5Takeaways & Limitations
4D-FEID provides a dataset and benchmark for research on facial expression intensity perception using dynamic 3D data.
Source: §V · p. 5
Abstract
from arXiv · showhide
The estimation and analysis of facial expression intensity play a crucial role in affective communication and human-computer interaction. Previous research has primarily focused on detecting and estimating facial expression intensity from frame-level 2D representations. However, this limitation restricts a comprehensive understanding of real-world facial expressions, as they are inherently 3D and temporally continuous. This paper investigates the perception of facial expression intensity by introducing the 4D Facial Expression Intensity Dataset (4DFEID). We employ a parametric face model and compile a total of 2,869 mesh sequences with controlled geometric variations, generating 4D data instances with diverse peak intensities and identity attributes. Using a Likert scale, we collect more than 90,000 subjective intensity perception ratings via a crowdsourcing platform. We explore various architectures and aggregation methods to establish baselines for episode intensity estimation on the new dataset, revealing that spatial-temporal graph models consistently outperform traditional frame-aggregation methods. In contrast to existing datasets that rely on 2D static imagery, the proposed 4D-FEID dataset provides the community with a unique and vital resource for investigating the perception of facial expression intensity through the use of dynamic 3D stimuli. By offering high-fidelity, spatio-temporally coherent facial data, 4D-FEID establishes a new foundation for research into more nuanced and naturalistic expression analysis, thereby addressing a gap in the current landscape of affective computing and human-computer interaction studies. The dataset is available at link.
I. INTRODUCTION
The introduction argues that frame-level 2D AU intensity estimates miss dynamic 3D expression perception and face imaging and annotation constraints. It presents 4D-FEID as a dataset and benchmark for episode-level intensity estimation.
I. INTRODUCTION
- Prior intensity datasets emphasize frame-level AU recognition from static 2D images, which can be affected by illumination and viewpoint and require certified FACS experts to annotate.The paper notes that expert annotation limits dataset scalability.Source: §I · p. 1
- Existing 4D facial datasets often focus on categorical labels or identity reconstruction, leaving continuous expression intensity less addressed despite humans' use of dynamic 3D cues.The paper proposes a facial mesh dataset for intensity estimation under more natural conditions.Source: §I · p. 1
- The dataset uses FLAME to generate personalized 4D sequences with controlled expression intensities, decoupling intensity shifts from identity variation.The approach samples an expression manifold derived from high-fidelity 4D captures or scans.Source: §I · p. 1
- The paper defines facial expression intensity as an episode attribute: the holistic perceived score observers assign to an entire sequence.This framing reflects perception across the sequence's global spatio-temporal evolution rather than isolated expression phases.Source: §I · p. 2
- 4D-FEID contains 2,869 mesh sequences paired with FLAME parameters and uses a 5-point Likert scale to collect over 90,000 human intensity ratings.The dataset is intended to support geometric and parametric analysis.Source: §I · p. 2
- The authors evaluate three method types on 4D-FEID to establish its first benchmark for 4D facial expression intensity estimation.They position the benchmark as a foundation for research on dynamic 3D facial behavior.Source: §I · p. 2
II. RELATED WORK
Related work describes established intensity rating scales and the limitations of static-image assessment, then surveys computational methods ranging from classification and ordinal regression to deep-learning approaches.
A. Perceived Intensity and Estimation
- Standardized ordinal scales, including FACS and DISFA annotations, encode AU intensity, while static-image assessment remains vulnerable to illumination artifacts and sensor noise.Ekman et al. used rating scales to examine cross-cultural intensity judgments and reported universal trends and cultural variations.Source: A. Perceived Intensity and Estimation · p. 2
- Early computational approaches treated AU intensity estimation as multi-class classification or ordinal regression.The cited approaches include and,, respectively.Source: A. Perceived Intensity and Estimation · p. 2
- Later methods include CNN-based joint AU detection and intensity prediction, structured learning for AU relationships and imbalance, heatmap regression, and self-supervised pre-training.The passage attributes these approaches to Gudi et al., Walecki et al.,, and.Source: A. Perceived Intensity and Estimation · p. 2
B. Relevant Datasets
Existing expression-intensity resources are limited by 2D projections or sparse, frame-level annotations, while 4D-FEID provides controlled dynamic mesh sequences with aligned parameters and rendered videos. Its construction filters and scales expression parameters, then applies quality control to produce 2,869 sequences.
B. Relevant Datasets
- DISFA relies on 2D video, while BP4D has 3D geometry but mostly 2D-derived, sparse-AU, frame-level intensity labels; 4D-FEID supplies continuous sequence-level ratings and controlled 4D dynamics.The paper describes existing corpora as constrained by 2D projections and the absence of holistic 4D expression dynamics.Source: B. Relevant Datasets · p. 2
III. THE 4D-FEID DATASET
- 4D-FEID aligns mesh sequences, FLAME parameters, and frontal-view videos, using D3DFACS mesh sequences captured at 60 fps.Each sequence stores a fixed face topology and 3D vertex coordinates across frames.Source: §III · p. 2
- The authors register each frame to FLAME, optimize identity shape on the neutral onset frame, hold it constant, and then estimate frame-wise pose and expression parameters.Keeping identity shape fixed is intended to prevent identity jitter across the sequence.Source: §III · pp. 2–3
- A fifth-order Butterworth low-pass filter with forward-backward filtering reduces spike-noise discontinuities while preserving the underlying expression dynamics.Forward-backward filtering is used to eliminate phase lag and ensure zero-phase distortion.Source: §III · p. 3
- Only expression channels whose deviation exceeds the threshold are scaled, leaving smaller physiological tremors unaltered.Figure 1 illustrates intensity-based pruning of expression-parameter dimensions.Source: §III · p. 3, Figure 1 · p. 3
- The selected expression parameters are scaled across frames using factors from 0.1 to 5.0, including attenuation factors that reduce intensity.The paper reports a 0.5 step size for scale factors.Source: §III · p. 3
- 2,869 distinct sequences spanning a wide range of expression intensities remain after manually filtering artifacts such as mesh self-intersections and implausible lip or eye distortions.The sequences were manually screened to eliminate unnatural artifacts.Source: §III · p. 3
- The dataset also renders frontal-view videos and shows peak-frame geometry with per-vertex displacement heatmaps to visualize intensity differences.The heatmaps represent displacement between peak-expression and onset frames.Source: §III · pp. 3–4
A. Subjective Study
The subjective study used a five-point Likert scale to collect ratings of perceived expression intensity from rendered videos. Each sequence received ratings from at least 32 unique raters.
A. Subjective Study
- The study used five Likert anchors—Trace, Slight, Marked & Pronounced, Severe & Extreme, and Maximum—aligned with the FACS five-level intensity metric.Participants viewed a brief tutorial, then rated each rendered video through the experiment interface.Source: A. Subjective Study · p. 4
- Each sequence was evaluated by at least 32 unique raters, yielding 91,584 ratings across 14 sessions of approximately 200 videos each.Videos generated from the same 4D sequence were grouped within a session; each video appeared once, and sessions lasted around 15 minutes.Source: A. Subjective Study · p. 4
B. Subjective Data Analysis
The authors mapped predictions to a five-point intensity scale, assessed inter-observer reliability by session, and averaged ratings to form each sequence’s ground-truth MOS.
B. Subjective Data Analysis
- Session ICC values ranged from 0.96 to 0.98, exceeding the stated 0.80 threshold for excellent rater agreement.Ratings were averaged across observers to derive the ground-truth mean opinion score for each sequence.Source: B. Subjective Data Analysis · p. 4
A. Experimental Setup
The evaluation benchmarks expression-intensity estimation on training, test, and cross-identity generalization subsets using rendered videos and 4D meshes.
A. Experimental Setup
- The cross-identity generalization set holds out 3D shape identities and AU-intensity combinations from training to test performance on unseen identities and expressions.The dataset is partitioned into training, same-distribution test, and cross-identity generalization subsets.Source: A. Experimental Setup · p. 4
- The benchmark compares three baseline families: geometry-based 4D-Vertex, rendered-video Video-CNN, and a FLAME-coefficient-based model.4D-Vertex uses vertex coordinates and a Bi-LSTM; Video-CNN uses ResNet-18 frame features with Bi-LSTM temporal attention; the FLAME-based model uses 1D convolutions, an LSTM, and an MLP regressor.Source: A. Experimental Setup · p. 4
B. Implementation Details
The implementation uses random training and test splits alongside a cross-identity set, with standardized sequence sampling and training conditions. Evaluation reports accuracy, correlation, and consistency metrics.
B. Implementation Details
- The dataset is randomly split into 80% training and 20% test data, with a separate cross-identity set holding out shape identities and AU-intensity combinations.The held-out set evaluates generalization to unseen morphological structures and expression patterns.Source: B. Implementation Details · p. 4
- Models train for 100 epochs with Adam, using 32 sampled frames per sequence, MSE loss, and a constant learning rate of 1 × 10−3.These shared conditions standardize comparisons across benchmark models.Source: B. Implementation Details · p. 4
C. Metrics
- Test-set metrics include MAE for absolute prediction precision and PCC for linear trend alignment.The evaluation also reports ICC(3,1) for consistency and reliability relative to human labels, and CCC as a combined measure of precision and accuracy.Source: C. Metrics · pp. 4–5
D. Results and Discussion
4D-Vertex outperformed the cited Video-CNN and FLAME-based baselines on key standard-test measures, while neutral-state residual training helped Video-CNN and 4D-Vertex overcome raw-input optimization challenges. Performance degradation was small on the cross-identity set with unseen facial shapes and AU combinations.
D. Results and Discussion
- 4D-Vertex achieved ICC 0.975 and beat the FLAME-based baseline on MAE (0.139 vs. 0.192), while Video-CNN had the highest MAE (0.289).These are standard-test comparisons.Source: Table III · p. 5, D. Results and Discussion · p. 5
- On unseen facial shapes and AU combinations, 4D-Vertex MAE rose only from 0.139 to 0.146, indicating robust cross-identity generalization.The cross-identity set excludes these shapes and AU combinations from training.Source: Table IV · p. 5, D. Results and Discussion · p. 5
- Neutral-state residual training let Video-CNN and 4D-Vertex overcome raw-input optimization failure (Ltrain > 0.7 after 50 epochs), reaching PCC/CCC > 0.9.The residuals are measured against each identity's initial neutral state.Source: D. Results and Discussion · p. 5
V. CONCLUSION
The paper presents 4D-FEID, a dataset of facial sequences with human intensity annotations, and benchmarks representative deep networks across modalities. The evaluations report that dynamic 3D representations, especially with difference-aware learning, outperform traditional 2D and parametric baselines.
V. CONCLUSION
- 4D-FEID contains 2,869 4D facial sequences, rendered videos at 60 FPS, and more than 90,000 human-annotated intensity ratings.The dataset is designed to support research on facial expression intensity perception.Source: §V · p. 5
- Dynamic 3D representations, particularly with difference-aware learning, significantly outperformed traditional 2D and parametric baselines in the benchmark evaluations.The strategy filters out static morphological noise.Source: §V · p. 5