Source-linked AI summary
RAFM-SER++: A Lightweight Multimodal Emotion Recognition Framework for Real-Time Behavioral Monitoring in Surveillance Systems
Ngo Truong Dinh, Tung-Lam Bui, Chi-Trung Duong, Vien Nguyen Thi, Viet-Anh Nguyen, Phuc-Lu Le
TL;DR
Interaction-heavy multimodal SER can be too costly for latency-sensitive, resource-constrained surveillance systems. RAFM SER++ uses asymmetric residual attention with cross-modal alignment and attention-guided pooling, achieving competitive or superior recognition with over 60% fewer trainable parameters and faster inference than transformer-based baselines.
Problem
Interaction-heavy cross-modal transformers improve multimodal SER but impose computational and latency costs that limit deployment on resource-constrained surveillance platforms.
Method
RAFM SER++ combines one-directional residual attention, BYOL-inspired cross-modal alignment, and attention-guided pooling for lightweight multimodal representation learning.
Results
More than 60% fewer trainable parameters and faster inference accompany competitive or superior recognition performance on IEMOCAP and ESD compared with transformer-based baselines.
Takeaways & Limitations
Lightweight asymmetric multimodal interaction can balance recognition accuracy and deployment efficiency without requiring computationally intensive bidirectional cross-modal interaction.
Abstract
from arXiv · showhide
Recent multimodal Speech Emotion Recognition (SER) systems achieve high accuracy through interaction-heavy cross-modal transformers, but their computational cost limits deployment in latency-sensitive and resource-constrained surveillance systems. To address this challenge, we propose RAFM_SER++, a lightweight multimodal SER framework featuring an asymmetric Residual Attention Fusion Mechanism (RAFM). Rather than relying on computationally expensive bidirectional interactions, RAFM injects affective speech cues into semantic text representations through a one-directional residual attention pathway. Combined with a BYOL-inspired cross-modal alignment objective and attention-guided pooling, the proposed framework improves multimodal representation learning while maintaining low computational overhead. Experiments on the IEMOCAP and ESD benchmarks demonstrate that RAFM_SER++ consistently outperforms the HuBERT-Base baseline and achieves a superior accuracy-efficiency trade-off compared with the state-of-the-art MemoCMT. Specifically, RAFM_SER++ reduces trainable parameters by more than 60%, achieves faster inference (79.60 it/s), and attains BACC scores of 81.10% on IEMOCAP and 95.39% on ESD. These results indicate that lightweight asymmetric multimodal fusion is an effective alternative to interaction-heavy cross-modal transformers for real-time surveillance applications.
I. INTRODUCTION
RAFM SER++ addresses the deployment cost of multimodal SER by using lightweight asymmetric fusion and complementary representation-learning components. Across IEMOCAP and ESD, it reports competitive or superior recognition with substantially lower parameter count and faster inference.
- Multimodal SER improves robustness for surveillance, but dense bidirectional cross-modal transformers impose computation and latency costs on resource-constrained platforms.
- RAFM injects affective speech cues into semantic text representations through a one-directional residual attention pathway.
- RAFM SER++ is formulated as a lightweight audio intelligence module for behavioral monitoring, emphasizing computational efficiency and real-time deployment.
- BYOL-inspired cross-modal alignment and attention-guided pooling support representation consistency and adaptive feature aggregation.
- More than 60% fewer trainable parameters and faster inference are reported across evaluations on IEMOCAP and ESD than stronger transformer-based baselines.
II. RELATED WORK
Related work shows that multimodal SER combines complementary speech and text information but often concentrates computational cost in fusion. RAFM SER++ instead constrains interaction through asymmetric residual attention while retaining cross-modal complementarity.
- Self-supervised speech encoders and multimodal learning have improved SER by strengthening acoustic representations and combining complementary speech, text, and vision cues.
- Early fusion can suffer from dimensionality, modality imbalance, and overfitting, whereas late fusion is efficient but weaker at modeling fine-grained dependencies.
- Even after encoder simplification, cross-modal transformers and repeated attention blocks remain a major bottleneck for real-time deployment.
- CM-RoBERTa and MemoCMT increase interaction capacity through self-attention, cross-attention, bidirectional transformers, or memory tokens, at higher computation, memory, and latency costs.
- RAFM SER++ uses constrained one-directional residual attention to preserve speech–text complementarity while reducing interaction complexity.
III. METHOD
RAFM SER++ is organized as a lightweight multimodal SER pipeline for real-time behavioral monitoring. It encodes modalities separately, projects them into a shared space, fuses them asymmetrically, and predicts from attention-guided representations.
- RAFM SER++ operates as an auxiliary audio intelligence module that complements video analytics with affective speech cues for behavioral monitoring.
- The framework has four stages: modality-specific encoding, shared-latent projection, RAFM fusion, and attention-guided prediction with joint optimization.
- Speech uses HuBERT and transcripts use BERT-base, with both representations projected into a shared embedding space before fusion.
- Text provides queries while speech provides keys and values, allowing asymmetric attention to refine textual representations through a residual pathway.
C. Residual Attention Fusion Mechanism
RAFM performs one-way speech-to-text residual attention, then attention-guided pooling emphasizes emotionally informative fused representations. This design targets lower interaction cost while preserving localized affective evidence.
- Residual Attention Fusion: Textual embeddings act as queries and acoustic embeddings as keys and values in a single asymmetric cross-attention operation.
- Residual Attention Fusion: F = LayerNorm( ˆT + Attn(Q, K, V )) adds the attention output to the projected text representation before normalization.
- Residual Attention Fusion: The residual formulation lets affective speech cues refine semantic representations while avoiding redundant bidirectional interaction, reducing computation and inference latency.
- Attention-Guided Pooling: Because emotional information is concentrated in limited words or acoustic segments, uniform mean or max pooling can dilute discriminative cues.
- Attention-Guided Pooling: Attention-guided pooling adaptively estimates each fused representation's contribution and aggregates the sequence into an utterance-level representation.
- Attention-Guided Pooling: The pooling mechanism emphasizes emotionally informative speech–text representations and suppresses redundant context, focusing classification on localized affective evidence.
E. Cross-Modal Alignment
RAFM SER++ uses a BYOL-inspired alignment loss to improve consistency between paired speech and text embeddings without negative samples. The alignment term functions as a lightweight regularizer with negligible computational overhead.
- The BYOL-inspired alignment loss maximizes cosine similarity between paired speech and text embeddings without requiring negative samples.
- The alignment loss acts as a lightweight regularizer with negligible computational overhead.
F. Deployment
RAFM SER++ is designed as an auxiliary audio perception module for multimodal surveillance systems, supporting higher-level behavioral analysis without directly inferring security threats. It is evaluated on complementary SER benchmarks under controlled recording conditions.
- RAFM SER++ supplies contextual affective cues that complement visual perception modules for higher-level behavioral analysis.
- The framework serves as an additional information source for multimodal decision making rather than directly inferring security threats.
- IEMOCAP and ESD provide complementary evaluation scenarios covering spontaneous conversational speech, multilingual recordings, and balanced emotional distributions.
- IEMOCAP contains scripted and improvised conversations with synchronized speech and transcripts and evaluates four emotion categories under a challenging class-imbalanced protocol.
- ESD contains over 29 hours of bilingual English and Mandarin emotional speech from 20 speakers, with cleaner recordings and a more balanced class distribution than IEMOCAP.
- Both datasets are collected under controlled recording conditions while serving as standard public benchmarks for evaluating surveillance-oriented affective perception models.
B. Implementation Details
The implementation uses pretrained HuBERT and BERT-base encoders, shared-space projection, a compact utterance representation, and lightweight classification. Training uses Adam on a single RTX 4090, while evaluation reports recognition metrics and inference throughput.
- RAFM SER++ is trained with Adam using β1 = 0.9, β2 = 0.999, ϵ = 10−8, an initial learning rate of 10−4, and weight decay of 10−6.
- Models are trained for 100 epochs with batch size 32, while the learning rate is decayed by 0.1 every 100 epochs.
- Speech and text representations are extracted with pretrained HuBERT and BERT-base encoders, then projected into a shared latent space.
- The final utterance-level representation is compressed into a 128-dimensional embedding and classified by a lightweight MLP.
- All experiments run on a single NVIDIA RTX 4090 GPU, and inference throughput is reported in iterations per second for deployment-efficiency assessment.
- Evaluation uses Accuracy, Balanced Accuracy, Macro-F1, Weighted-F1, and inference throughput, with BACC as the primary metric for class-imbalanced SER datasets.
- BACC measures average recall across emotion classes, while Macro-F1 is class-balanced, Weighted-F1 accounts for class frequency, and throughput measures deployment efficiency.
D. Results on ESD
On ESD, RAFM SER++ delivers strong recognition with substantially lower model size and faster inference than MemoCMT. Its attention-guided fusion is also associated with fewer confusion between Happiness and Neutral.
- 95.39% BACC, ACC, Macro-F1, and Weighted-F1 are achieved by RAFM SER++ on ESD.
- 3.6M trainable parameters and 79.60 it/s inference outperform MemoCMT's 8.9M parameters and 45.19 it/s.
- More than 60% fewer trainable parameters are achieved while preserving improved recognition performance.The authors attribute this balance to asymmetric residual fusion without computationally expensive bidirectional transformer interactions.
- RAFM SER++ produces fewer misclassifications than MemoCMT, particularly between Happiness and Neutral.The comparison uses confusion matrices with MemoCMT using Min pooling and RAFM SER++ using attention-guided pooling.
F. Ablation and Efficiency Analysis
Ablation and efficiency results show that both cross-modal alignment and attention-guided pooling improve performance while the full framework substantially reduces computational cost.
- BYOL-inspired alignment increases IEMOCAP BACC from 74.73% to 77.69%.
- Attention-guided pooling further increases IEMOCAP BACC to 81.10%.The ablation identifies adaptive feature aggregation as an additional contributor beyond alignment.
- More than 60% fewer trainable parameters are achieved than with MemoCMT while maintaining higher BACC and faster inference.The reported efficiency profile supports real-time deployment.
G. Discussion
Across IEMOCAP and ESD, RAFM SER++ balances recognition accuracy with deployment efficiency through constrained asymmetric multimodal interaction. The authors position it as an auxiliary audio module whose benchmark results still require validation in realistic surveillance conditions.
- RAFM SER++ provides competitive or superior performance with more than 60% fewer trainable parameters and faster inference than MemoCMT.
- Speech is modeled as an affective residual correction over semantic text representations through asymmetric attention.BYOL-inspired alignment and attention-guided pooling improve representation consistency and discriminative aggregation.
- RAFM SER++ is intended to complement visual perception as an auxiliary audio intelligence module rather than replace it.
- The experiments use benchmark SER datasets, while realistic surveillance introduces noise, reverberation, multilingual speech, speaker overlap, domain shift, and ASR errors.Future work will evaluate the framework under these conditions and investigate adaptive modality reliability.
- The paper concludes that lightweight asymmetric multimodal interaction can balance recognition accuracy and deployment efficiency for real-time surveillance.