Source-linked AI summary
Emo-DVS: A Multimodal Benchmark for Privacy-Aware Emotion Recognition with Event Cameras
Jiaqi Chen, Qinfu Xu, Hao Zhuang, Liyuan Pan
TL;DR
RGB-based emotion analysis creates privacy concerns, while existing event-based methods lack sufficiently diverse datasets and multimodal modeling for complex settings. The paper introduces Emo-DVS and the tri-modal IGF framework, whose experiments establish state-of-the-art privacy-preserving performance across varying illumination conditions.
Problem
Existing event-based emotion analysis is limited by small datasets, simple acquisition conditions, and reliance on single-modality visual cues, while RGB sensing raises privacy concerns.
Method
The paper constructs the Emo-DVS dataset and tri-modal event-audio-text benchmark, then uses IGF with event pre-training, adaptive modality gating, and mutual-information alignment.
Results
Extensive experiments establish state-of-the-art privacy-preserving emotion-analysis performance across varying illumination conditions.
Takeaways & Limitations
Emo-DVS provides a large-scale benchmark for studying cross-modal emotion analysis with event cameras under challenging illumination and privacy constraints.
Takeaways & Limitations
Dataset access is restricted to verified academic researchers under a formal data use agreement that prohibits commercial use and reidentification attempts.
Abstract
from arXiv · showhide
Emotion analysis is a fundamental task in computer vision, but its practical deployment remains constrained by the privacy risks inherent to conventional RGB cameras. Bio-inspired event cameras present a promising hardware-level solution because they capture asynchronous brightness changes, thereby reducing exposure of facial identity details while leveraging high dynamic range for robust perception under challenging illumination conditions. Despite these advantages, existing event-based methods struggle in complex real-world settings due to limited dataset scales, simple acquisition conditions, and reliance on single-modality visual cues. To address these, we establish a challenging tri-modal benchmark with event, audio, and text modalities and propose the Information-Guided Gated Fusion (IGF) framework, which first pre-trains an event encoder on the FAU subset of Emo-DVS to capture fine-grained facial dynamics, then employs adaptive modality gating to suppress modality-specific noise, and finally leverages mutual information maximization to align robust cross-modal representations. To alleviate data scarcity, we introduce Emo-DVS, the first large-scale event-based emotion analysis dataset, which couples dynamic illumination with the Facial Action Unit (FAU) subset and emotion subset. Extensive experiments demonstrate that IGF achieves state-of-the-art performance.
1 Introduction
RGB-based emotion analysis raises privacy concerns because it captures identifiable facial details, while event cameras reduce identity exposure and tolerate difficult illumination. Emo-DVS addresses remaining dataset and modality gaps with a tri-modal benchmark and the IGF framework.
- RGB sensors capture identifiable facial details, creating privacy leakage concerns that restrict real-world emotion-analysis deployment.
- Event cameras record asynchronous brightness changes, suppress facial identity details, and remain robust under low-light and overexposed conditions.
- Existing event-based methods are limited by small datasets, simple environments, and reliance on single-modality visual cues.
- Emo-DVS combines dynamic-illumination recordings, fine-grained FAU annotations, simulated emotional data, and a tri-modal event-audio-text benchmark.
- Extensive experiments establish state-of-the-art privacy-preserving emotion-analysis performance across varying illumination conditions.
- IGF pre-trains an event encoder, adaptively gates modality features, and maximizes mutual information to align cross-modal representations.
2 Related Work
Prior event-based emotion datasets remain limited in scale, annotation consistency, and environmental diversity, while neuromorphic tri-modal emotion analysis is underexplored. Emo-DVS responds with a large-scale benchmark combining event data with richer environmental and multimodal coverage.
- Existing event-based emotion datasets provide varied modalities and annotations, but synthetic event data may exhibit domain gaps from real recordings.
- Existing datasets are limited in scale, annotation consistency, and environmental diversity.
- Video Emotion Analysis includes categorical labels such as Ekman’s six basic emotions and dimensional representations using Valence, Arousal, and Dominance.
- Tri-modal emotion analysis combining neuromorphic event data, audio, and text remains underexplored.
3 The Emo-DVS Dataset
Emo-DVS is a large-scale multimodal event-based benchmark designed around hierarchical facial dynamics and emotional semantics. Its construction combines real and simulated event data, diverse illumination and expression settings, aligned annotations, and privacy-controlled access.
- Dataset scale: Emo-DVS contains 13,066 clips, comprising 4,042 FAU clips and 9,024 Emotion clips.
- Dataset organization: The hierarchical design links a fine-grained FAU physical layer with high-level emotional semantics from spontaneous and posed expressions.
- Acquisition conditions: The capture system synchronizes event and RGB sensors and samples illumination from 100–150 lux down to 2–5 lux, with recordings both with and without glasses.
- Data collection: The FAU subset records participants imitating demonstrated actions with event cameras, while the Emotion subset converts selected spontaneous and posed videos into event streams.
- Annotation strategy: Audio tracks are standardized and transcribed into text, while FAU labels are converted into structured descriptions to support cross-modal alignment.
- Dataset statistics: The FAU subset contains 4,042 clips covering 29 action units, while the Emotion subset contains 9,024 sequences including 3,582 spontaneous and 5,442 posed clips.
- Privacy and access: Event representations suppress static facial textures and identity details, and dataset access is restricted to verified academic researchers under a data-use agreement.
4 Method
The method converts sparse event streams into temporal representations, pre-trains the event encoder with text-anchored FAU alignment, and fuses event, audio, and text through adaptive weighting and information-guided objectives.
- Event representation: Events are aggregated into temporal bins or a time-windowed dense 3D frame for processing by spatial-temporal architectures.Accumulated event polarities form the dense representation used downstream.
- Event representation pre-training: The event encoder is pre-trained on FAU recognition to capture facial dynamic priors despite sparse event data and missing RGB texture.A frozen CLIP spatial backbone extracts frame representations, followed by a Temporal Transformer for macro-level temporal deformations.
- Event representation pre-training: Text-anchored contrastive pre-training aligns event representations with textual FAU categories using cosine similarity and cross-entropy.Learnable context tokens are included in the textual anchors, and the alignment is intended to resist modality-irrelevant noise.
- Multimodal fusion: Temporal attention pooling summarizes variable-length event, acoustic, and linguistic sequences into compact unimodal descriptors.The mechanism uses positional features and a learnable query to attend to informative temporal segments.
- Multimodal fusion: Dynamic modality gating computes input-dependent weights, scales each unimodal feature, and suppresses interference from redundant noise.The contribution vector is produced by an MLP and Softmax normalization before feature modulation.
- Multimodal fusion: IGF combines modality-specific, shared, and gate features, then uses mutual-information, alignment, and redundancy objectives during end-to-end optimization.The resulting fused representation feeds the emotion classification head.
5 Experiment
Experiments evaluate IGF across event-based emotion datasets, ablations, modality settings, illumination conditions, and privacy-related face recognition. Results show strong performance, benefits from pre-training and adaptive fusion, and improved robustness of event representations under challenging illumination.
- Main Results: IGF achieves the best performance among compared methods on VideoEmotion8*, Ekman6*, and FAU, reaching 41.05%, 42.54%, and 56.40% Acc, respectively.The evaluation compares the framework with representative RGB-based and event-based baselines under a unified protocol.
- Main Results: 59.43% Acc and 57.12% F1 are achieved after transferring the event-pretrained framework to RGB emotion classification on Ekman6, outperforming compared RGB-based methods.The result indicates cross-modal transferability beyond the event-domain evaluation.
- Ablation Studies: Removing FAU pre-training causes an 8.03% performance drop, while replacing temporal attention pooling with average pooling causes a 1.76% drop.These ablations support the contributions of low-level facial-dynamics priors and learnable temporal aggregation.
- Parameter Analysis: Moderate λ_mi and τ perform best, whereas excessively large values interfere with classification; small non-zero λ_alg and λ_red values improve consistency and complementarity.The auxiliary objectives are beneficial when properly balanced.
- Illumination and Privacy: Event data reduces face recognition accuracy by 66.22%, contrasting with RGB’s 98.34% identification rate and indicating suppression of identity-related appearance cues.This comparison directly evaluates the privacy-related behavior of the two modalities.
- Illumination and Privacy: Event representations outperform RGB across illumination settings, reaching 37.35% under low light versus 8.93% for RGB, and remaining superior under medium and high light.Event achieves 39.27% and 38.49% under medium and high light, compared with 36.28% and 35.27% for RGB.
- Modality Analysis: Bimodal settings outperform single modalities, while the full event, acoustic, and textual model performs best on both Ekman* and VideoEmotion8*.On Ekman*, E+T reaches 54.17% Acc and 53.33% F1; on VideoEmotion8*, A+T reaches 39.39% Acc and 39.46% F1.
6 Conclusion
The paper presents Emo-DVS as a large-scale event-based emotion dataset and establishes a tri-modal benchmark for challenging illumination and privacy-constrained settings. It proposes IGF to align multimodal cues and reports superiority over state-of-the-art methods.
- Conclusion: Emo-DVS is a large-scale event-based emotion analysis dataset paired with a tri-modal benchmark for challenging illumination and privacy-constrained environments.The benchmark combines event, audio, and text modalities.
- Conclusion: IGF extracts spatio-temporal priors with a pre-trained event encoder and uses mutual information maximization to align multimodal cues for emotion classification.The framework is designed for robust multimodal emotion analysis.
- Conclusion: Extensive experiments demonstrate that IGF outperforms state-of-the-art methods.The conclusion reports superiority across the paper’s evaluations.