From Affect to Complex Behavior: Advancing Multimodal Human-Centered AI at the 10th ABAW Workshop & Competition

Introduction

• Affect and behavior are expressed through dynamic combinations of facial, bodily, vocal, linguistic, and sometimes physiological cues, motivating multimodal learning for richer real-world understanding.Affect includes emotional and expressive states, while behavior also encompasses observable actions and social signals such as gesture, attention, mimicry, and hesitation.Source: §1 · pp. 1–2
• Affect and behavior analysis has scientific and practical value, but real-world systems must address temporal dynamics, modality fusion and noise, demographic bias, privacy, trustworthiness, fairness, and generalizability.These challenges have pushed the field beyond narrow unimodal recognition toward richer multimodal formulations.Source: §1 · p. 2
• ABAW has evolved from core affective computing toward socially grounded, responsible multimodal AI, while accepted papers span pose, motion and behavior, affect and multimodal learning, benchmarks, datasets, evaluation, fairness, robustness, and deployment.The workshop brings together researchers across fields and serves as both a benchmark-driven venue and a forum for discussing robust, equitable, human-centered technologies.Source: §1 · p. 2
• The 10th edition’s six challenges cover valence-arousal estimation, expression recognition, action unit detection, fine-grained violence, emotional mimicry intensity, and ambivalence/hesitancy.Across editions, ABAW competitions have expanded from core affect tasks toward more nuanced, socially significant, and temporally structured behaviors.Source: §1 · p. 2
• The paper presents the 10th ABAW Workshop, summarizes its competition, and positions the edition within the development of affective and behavior analysis in-the-wild.The edition reflects a field that is increasingly multimodal, behavior-centric, and human-centered.Source: §1 · p. 2

2.1. Pose, Motion and Behavior Estimation

• MuPPet and VSDPose address multi-person and multi-view 3D pose estimation, while Pose2Lang3D connects 2D skeletons with language-aligned representations and 3D supervision.MuPPet models inter-person correlations; VSDPose uses voxel-based self-distillation. Pose2Lang3D supports semantic reasoning from 2D inputs.Source: §2.1 · pp. 2–3
• SBF enriches skeleton-based action recognition with depth, contour and motion cues, while VGGT-HPE frames head pose estimation as relative prediction.VGGT-HPE uses a geometry-based foundation model trained on synthetic data.Source: §2.1 · p. 3

2.2. Affect Modelling and Multimodal Learning

• Affect and multimodal learning contributions model affect dynamics, annotation uncertainty, physiological stress and emotion representations.LaScA combines language-derived semantics with structured features; another framework captures annotator variability in continuous affect prediction. Stress assessment uses reliability-aware dynamic modality weighting under noisy conditions, while Dimensional Distribution Emotion State uses valence and arousal as a common embedding space across datasets.Source: §2.2 · p. 3

2.3. Benchmarks, Datasets & Evaluation Protocols

• Contributions introduce resources and methods for explainable content moderation, temporal laughter localization, event-centric anomaly evaluation and more stable AU evaluation.SenBen supplies grounded scene-graph annotations; MTLLFM provides temporally annotated datasets and a multimodal model. The anomaly-detection work critiques frame-level metrics, while the AU evaluation work analyzes cross-validation split variability and advocates leave-one-dataset-out settings.Source: §2.3 · p. 3

2.4. Fairness, Robustness and Deployment

• Fairness and robustness contributions address confounding in facial-expression fairness assessment and determine when test-time adaptation should be applied under domain shift.The fairness work identifies head pose and illumination as visual confounders and proposes a standardized protocol. The topology-guided framework uses persistent homology and aims to improve robustness without degrading performance.Source: §2.4 · p. 3

3.1. Valence-Arousal (VA) Estimation Challenge

• The VA challenge estimates per-frame valence and arousal on Aff-Wild2, with RAS [16] winning through a multimodal facial, behavioral, and audio framework.The dataset contains 2,993,081 annotated frames from 584 subjects across 594 videos; the validation baseline achieved average CCC 0.22. EmoDX [12] and IMLAB [6] placed second and third with reliability-aware and semantically guided multimodal approaches, respectively. Only uni-task solutions were allowed, and refinement had to use VA annotations.Source: §3.1 · pp. 3–4

3.2. Expression (EXPR) Recognition Challenge

• The EXPR challenge classifies each frame into eight expression categories; EagleonPamir1 [19] won with a two-stage visual-audio framework.Its DINOv2-based visual stage used padding-aware augmentation and a training-only MoE head, followed by gated fusion of facial features and Wav2Vec 2.0 audio with temporal smoothing. HSEmotion [17] used confidence-gated calibration, while USTC-IAT-United [22] addressed missing modalities and class imbalance. The VGGFace VGG16 baseline scored 0.25 on validation.Source: §3.2 · pp. 4–5

3.3. Action Unit (AU) Detection Challenge

• The AU challenge detects 12 facial action units, and USTC-IAT-United [21] won with hierarchical visual-audio alignment and temporal modeling.Its framework combined pretrained DINOv2 vision and WavLM audio, aligning local facial regions with global context and using an audio-guided state-space model for visual updates. HSEmotion [17] placed second with per-AU weighting, threshold tuning, and temporal smoothing; the VGG16-based validation baseline averaged F1 0.39. Solutions were uni-task and restricted to AU annotations during refinement.Source: §3.3 · pp. 5–6

3.4. Fine-Grained Violence Detection (VD) Challenge

• The fine-grained VD challenge classifies violent versus non-violent frames; HSEmotion won with fused visual and skeleton streams.The challenge used 172 videos with frame-level labels covering 1,389,976 frames. HSEmotion combined ConvNeXt-T visual features and a dilated TCN with MediaPipe-derived skeleton, motion, and interaction features, fusing them through bidirectional cross-attention before a two-layer BiLSTM. The ResNet-50 plus bidirectional LSTM validation baseline had macro F1 0.64.Source: §3.4 · pp. 6–7

3.5. Emotional Mimicry Intensity (EMI) Estimation Challenge

• The EMI challenge predicts intensity across six emotions from naturalistic mimicry videos, with USTC-IAT-United [23] winning using text-anchored multimodal attention.HUME-Vidmimic2 contains 15,000+ videos from 557 participants, with mimicry rated on a 0–100 scale. The baseline validation correlations were 0.09 for vision, 0.24 for audio, and 0.255 for multimodal input. TAEMI uses text to align and filter visual and audio signals and includes missing-modality tokens and modality dropout.Source: §3.5 · p. 7

3.6. Ambivalence/Hesitancy (AH) Video Recognition Challenge

• The AH challenge predicts ambivalence/hesitancy (A/H) at video level as binary classification, using a new, expert-annotated BAH dataset with 1,427 videos from 300 participants and frame- and video-level annotations.The zero-shot Video-LLaVA vision-only baseline scored 0.2827 on the public test set.Source: §3.6 · pp. 7–8
• VisPBF [13] won with BROTHER, an ensemble of visual, audio, and contextual expert networks using diversity constraints, consistency regularization, and sequence modeling.The framework combined complementary behavioral cues across modality-specific models.Source: §3.6 · p. 8
• Fennec [1] placed second with ConflictAwareAH, which used pretrained video, audio, and text encoders and pairwise embedding differences to capture cross-modal disagreement.Attention pooling aggregated the modality embeddings.Source: §3.6 · p. 8
• LEYA [15] placed third with pretrained visual, audio, and text features refined by temporal modules and adaptively weighted by modality reliability.The framework also used auxiliary supervision and regularization.Source: §3.6 · p. 8

• The edition’s results underscore the importance of multimodal fusion, strong pretrained models, temporal reasoning and deployment-aware design for real-world affective and behavioral analysis.Source: §4 · p. 8
• Its six competition challenges and diverse paper track provide a platform for benchmarking, collaboration and innovation in multimodal, robust and human-centered AI.Source: §4 · p. 8