PaperlayerView exact revision on arXiv

Source-linked AI summary

AVERE: Improving Audiovisual Emotion Reasoning with Preference Optimization

Ashutosh Chaubey, Jiacheng Pang, Maksim Siniukov, Mohammad Soleymani

arXiv:2602.07054v1cs.LGcs.CVcs.HC

TL;DR

MLLMs for emotion understanding can associate emotions with irrelevant audiovisual cues and hallucinate cues driven by text priors, undermining reliable reasoning. The paper introduces EmoReAlM to evaluate these problems and AVEm-DPO to align responses with audiovisual inputs while reducing text-prior bias. AVEm-DPO outperforms open-source baselines on proposed and existing benchmarks in zero-shot settings, while limitations remain for disgust recognition and spurious audio associations.

  • Problem

    MLLMs can ground emotions in irrelevant audiovisual cues or hallucinate cues driven by text priors, while existing benchmarks insufficiently evaluate these emotion-specific failures.

  • Method

    The paper introduces the EmoReAlM benchmark and AVEm-DPO, which uses multimodal preferences and text-prior debiasing to align responses with audiovisual inputs.

  • Results

    AVEm-DPO outperforms open-source baselines on proposed and existing emotion understanding benchmarks under a zero-shot setting.

  • Takeaways & Limitations

    The benchmark and preference-optimization method support evaluation and mitigation of spurious audiovisual associations and cue hallucinations in MLLMs.

  • Takeaways & Limitations

    Performance remains poor for disgust recognition and leaves room to reduce spurious audio cue–emotion associations.

Abstract

from arXiv · show

Emotion understanding is essential for building socially intelligent agents. Although recent multimodal large language models have shown strong performance on this task, two key challenges remain - spurious associations between emotions and irrelevant audiovisual cues, and hallucinations of audiovisual cues driven by text priors in the language model backbone. To quantify and understand these issues, we introduce EmoReAlM, a benchmark designed to evaluate MLLMs for cue-emotion associations, hallucinations and modality agreement. We then propose AVEm-DPO, a preference optimization technique that aligns model responses with both audiovisual inputs and emotion-centric queries. Specifically, we construct preferences over responses exhibiting spurious associations or hallucinations, and audiovisual input pairs guided by textual prompts. We also include a regularization term that penalizes reliance on text priors, thereby mitigating modality-specific cue hallucinations. Experimental results on DFEW, RAVDESS and EMER demonstrate that our method significantly improves the performance of the reference baseline models with 6-19% of relative performance gains in zero-shot settings. By providing both a rigorous benchmark and a robust optimization framework, this work enables principled evaluation and improvement of MLLMs for emotion understanding and social AI. Code, models and benchmark will be released at https://avere-iclr.github.io.

1 INTRODUCTION

Emotion understanding supports socially intelligent agents, but audiovisual MLLMs can rely on irrelevant cues or hallucinate cues shaped by text priors. EmoReAlM and AVEm-DPO address these reliability problems through targeted evaluation and preference optimization.

  • Emotion understanding supports tailored responses and meaningful human–machine interactions, including applications in health and education.
  • MLLMs improve interpretability beyond classification by generating detailed emotion descriptions and broader emotion-related keywords.
  • MLLMs make reasoning errors by grounding emotions in irrelevant audiovisual cues and perception errors by hallucinating cues to justify emotions.Text priors can bias models toward cues that commonly co-occur with particular emotions.
  • AVEm-DPO uses explicit prompt-based audiovisual preferences and text-prior debiasing to improve reference MLLMs in zero-shot evaluation.The method penalizes policy reward for responses to text-only inputs.
  • EmoReAlM provides 4,000 human-verified MCQA samples for evaluating emotion reasoning, modality agreement, and emotion-related hallucinations.Its MCQA format supports transparent, reproducible, and scalable evaluation without an additional LLM during inference.

2 RELATED WORK

Prior work develops audiovisual emotion models, multimodal benchmarks, and preference-optimization methods, but this paper focuses their combination on emotion-specific audiovisual reasoning. Its benchmark emphasizes standardized evaluation of cue associations, modality agreement, and hallucinations.

  • MLLMs for Emotion: Emotion-focused MLLMs use domain-specific tuning, open-vocabulary recognition, audiovisual fusion, mixture-of-experts routing, modality-specific experts, or retrieval-augmented reasoning.
  • Preference Optimization: This work applies multimodal preference optimization and text-prior debiasing to audiovisual emotion reasoning.
  • Benchmarks: EmoReAlM differs from broader multimodal hallucination benchmarks by focusing on audiovisual emotion understanding with standardized evaluation.
  • Benchmarks: Existing emotion reasoning benchmarks provide detailed transcript, audio, and visual cues that can support GPT-based evaluation.
  • Preference Optimization: Direct preference optimization aligns language models to human preferences and has been used to mitigate hallucinations in vision-language models.

3 EMOREALM BENCHMARK

EmoReAlM evaluates audiovisual emotion reasoning through basic reasoning, modality agreement, and stress tests for spurious associations and hallucinations. Its construction separates modalities, verifies emotion relevance, generates MCQA items, and applies human verification.

  • The benchmark tests relevant-cue reasoning, audio–visual emotional agreement, spurious cue–emotion associations, and hallucinations driven by text-only biases.
  • Emotion Reasoning – Basic: Basic Emotion Reasoning links speech, tone, facial expression, and body language to an emotion without providing the ground-truth emotion in the question.
  • Modality Agreement: Modality Agreement assesses whether audio and visual modalities convey the same emotional state.
  • Emotion Reasoning – Stress Test: Stress Test includes No Hallucination, Spurious Cue-Emotion Association, and Emotion-Relevant X-Hallucination subtasks for each modality.It tests both irrelevant observed cues and cues hallucinated because they commonly co-occur with emotions.
  • Benchmark Creation: The creation pipeline separately captions audio and video, verifies emotion relevance with text-based prediction, generates MCQA samples, and manually verifies them.GPT-4o performs caption extraction, emotion classification, and question–answer generation.
  • Benchmark Statistics: EmoReAlM contains 4,000 questions over 2,649 unique videos and is used only as a test set.Preference optimization uses a different dataset.

4 AVEM-DPO

AVEm-DPO aligns audiovisual emotion reasoning with both multimodal inputs and prompt-specific modality requirements. It combines preference learning over inputs and responses with text-prior debiasing to discourage spurious associations and hallucinated cues.

  • Multimodal Preference Optimization: AVEm-DPO extends DPO to align responses with chosen audiovisual inputs rather than relying only on response preferences.The formulation contrasts chosen and rejected multimodal inputs while retaining a reference model.
  • Prompt-based Modality Preference: Prompt-based modality preferences modify only the modality relevant to the query, encouraging responses grounded in the requested audio or visual input.The rejected audiovisual input is constructed conditionally on the prompt modality; emotion-different samples produced the best results among tested negative samples.
  • Emotion-based Response Preference: Response preferences reject cues that are relevant to the video but spuriously associated with the emotion, as well as emotion-related cues absent from the audiovisual input.Each chosen response is paired with two rejected response types targeting spurious associations and hallucinated cues.
  • Text Prior Debiasing: Text-prior debiasing penalizes rewards for responses generated from text-only inputs, discounting explanations attributable to language-model priors.The penalty uses a text language model and hyperparameter γTPD; responses supported by audio or video receive relative credit.
  • Preference Data: The preference data are generated from audiovisual emotion datasets, with variations of correct answers created using altered spurious or hallucinated cues.The generated preference data are not manually verified, yet the authors report that training still produces a performance gain.

5 EXPERIMENTS

Experiments evaluate AVEm-DPO against audiovisual MLLM references and preference-optimization baselines across EmoReAlM and established emotion benchmarks. AVEm-DPO improves reasoning, reduces hallucinations and spurious associations, redistributes attention toward relevant modalities, and remains robust to irrelevant-input changes.

  • EmoReAlM Results: AVEm-DPO achieves substantial gains on EmoReAlM, where reference models struggle especially on Modality Agreement and Stress-Test evaluations.The method also surpasses Vista-DPO and Naive-DPO by significant margins.
  • Existing Benchmarks: Preference tuning boosts existing-benchmark performance, especially on EMER, while reducing spurious cue-emotion associations and hallucinations.The reference and preference-tuned models also perform significantly better than most baselines on out-of-domain RAVDESS.
  • Ablations: Removing any AVEm-DPO component causes a significant performance drop, with Text Prior Debiasing especially important for hallucination stress-test samples.The ablation evaluates Prompt-based Modality Preference, Emotion-based Response Preference, and Text Prior Debiasing.
  • Design Choices: AVEm-DPO is significantly better than training-free contrastive decoding, while emotion-specific audiovisual inputs outperform random or diffused alternatives.Emotion-relevant and video-relevant rejected responses also outperform using only one type or an irrelevant response.
  • Analysis: AVEm-DPO increases attention to the relevant modality and produces negligible correct-answer log-likelihood shifts when the irrelevant audio input changes.These analyses support modality-grounded responses and robustness to adversarial audiovisual inputs.
  • Data Verification: Human verification used 90 participants for 1000 generated preference samples, with each sample checked by at least three annotators.The reported majority judgments validate the automatically generated preference data.

6 LIMITATIONS AND FUTURE WORK

The benchmark inherits limitations from its source data and task duration, while AVEm-DPO still struggles with disgust recognition and some spurious audio cue-emotion associations.

  • Benchmark Scope: EmoReAlM may inherit cultural biases because it is derived from DFEW and uses DFEW emotion labels.This constrains how broadly the benchmark's results should be interpreted across cultures.
  • Benchmark Scope: Long-video emotion understanding remains open because the benchmark and training data use short videos lasting approximately 2–10 seconds.The authors identify longer temporal contexts as future work.
  • Method Limitations: AVEm-DPO performs poorly on disgust recognition, which the authors attribute to the limited number of training samples for this emotion class.The paper also reports remaining room to mitigate spurious audio cue-emotion associations in stress-test subtasks.

7 CONCLUSION

The paper introduces EmoReAlM and AVEm-DPO to evaluate and mitigate spurious audiovisual-emotion associations and cue hallucinations in MLLMs. It reports improved zero-shot benchmark performance, supported by ablations and analyses of attention redistribution and log-likelihood shifts.

  • Conclusion: The work contributes EmoReAlM for complex emotion-reasoning evaluation and AVEm-DPO for reducing spurious associations and audiovisual cue hallucinations.The method uses prompt-based modality preference and text-prior debiasing.
  • Conclusion: AVEm-DPO outperforms open-source baselines on proposed and existing emotion-understanding benchmarks in zero-shot settings.Ablations and attention and likelihood analyses support the proposed optimization components.
  • Data and Reproducibility: The work uses publicly available DFEW, MAFW, and MER2025 data without collecting new audiovisual data.The benchmark release contains generated and human-verified question–answer pairs, while users obtain source videos independently.
  • Responsible Use: The paper cautions against deploying the benchmark and models in sensitive real-world contexts without domain-specific validation and safeguards.Risks include misinterpretation, bias reinforcement, and misuse in surveillance or high-stakes applications.
  • Benchmark Construction: Benchmark question–answer construction combines audiovisual captioning, emotion prediction, task-specific question generation, text-only filtering, and human verification.The pipeline uses audio and visual captions, source emotion labels, and participant review to finalize samples.

B.3 BENCHMARK STATISTICS

Benchmark statistics describe emotion and task distributions, while construction analyses document balanced stress-test categories and the use of sampled video frames for visual captioning.

  • Benchmark Statistics: EmoReAlM's emotion-label distribution is similar to that of its DFEW source dataset.The benchmark therefore retains the source dataset's broad label distribution.
  • Benchmark Statistics: The benchmark reports distributions of audiovisual languages, tasks, and stress-test subtasks, alongside examples from its benchmark and preference datasets.These statistics characterize the composition of EmoReAlM and its preference data.
  • Stress-Test Distribution: Stress-test samples are balanced between Yes and No answers, and No-answer samples are almost equally distributed across the tested subtasks.The subtasks include No hallucination, Spurious Association, and Audio/Visual Hallucination.
  • Captioning Analysis: Visual captioning uses sampled video frames, with the frame-count choice evaluated on the short-video EMER dataset.The analysis targets caption quality for subtle visual emotion cues.
  • Evaluation Inputs: Text subtitles shown in benchmark examples are reference material and are not passed to models during evaluation.This separates example presentation from the evaluation input configuration.

C.2 PREFERENCE DATA

The preference-data setup uses MAFW and MER2025 training data, automatic annotations, and multimodal or text-prior-aware evaluation procedures. It also specifies the reference models and baseline DPO implementations used for comparison.

  • Preference data construction: Preference data are constructed from the MAFW and MER2025 Track 1 training sets using a pipeline similar to Fig. 3.Gemini 2.5 Flash provides the automatic annotations used to create the training dataset.
  • Training setup: AVEm-DPO training uses one epoch with learning rate 5e−7, batch size 2 per GPU, and LoRA modules attached to the language-model backbone.Training uses eight NVIDIA H100 GPUs, LoRA rank 8, scale 4, and gradient accumulation over four iterations.
  • Evaluation: The evaluation reports average task accuracy and, for Modality Agreement and Stress Test, precision, recall, and F1.F1 is defined as the harmonic mean of precision and recall.
  • DPO baselines: The study compares AVEm-DPO with Naive-DPO and Vista-DPO adaptations using preference data and distinct preference objectives.Vista-DPO replaces prompt-based modality preferences with audiovisual inputs depicting different emotions.

D.5 EXPERIMENTAL SETUP FOR ABLATION STUDY

The ablation study defines aggregate metrics for benchmark subtasks and compares variants that remove prompt-based modality preference, emotion-based response preference, or text-prior debiasing.

  • Metrics: Basic is the unweighted average of visual and audio reasoning accuracy, while Stress is the unweighted average of visual and audio F1 scores.Agree is reported using F1 over Modality Agreement samples.
  • Ablation variants: The ablations remove prompt-based modality preference, emotion-based response preference, or text-prior debiasing from AVEm-DPO.These variants isolate the contributions of the different preference and regularization components.
  • Comparison: Table 13 compares methods on EmoReAlM and reports accuracy, precision, recall, and relative improvement over reference models.It also includes video-only and audio-only baselines.

E.2 EMOREALM RESULTS ON DIFFERENT STRESS TEST SUBTASKS

Stress-test analysis shows that hallucinated emotion-relevant cues are a larger bottleneck than spurious cue-emotion associations, while AVEm-DPO improves performance across subtasks.

  • Stress-test subtasks: AVEm-DPO significantly improves performance over the reference model on both spurious cue-emotion association and emotion-relevant cue hallucination subtasks.These subtasks use Stress Test samples whose correct answer is “No.”
  • Bottlenecks: Hallucination is a bigger bottleneck than spurious cue-emotion association within both audio and visual reasoning.The analysis compares the two “No”-answer subtasks directly.
  • Modality comparisons: Audio-only models perform worse than audiovisual models, whereas video-only performance is better than audiovisual performance on these reasoning evaluations.The same modality pattern is reported in the expanded benchmark results.
  • DFEW: AVEm-DPO and the base model achieve the best and second-best unweighted and weighted average recalls on DFEW across emotion classes.The proposed method also yields fairer performance across emotion categories than baselines.

E.4 USER EVALUATION

The user evaluation presents anonymized model captions alongside videos and asks participants to select the best caption across emotion description, cue association, and hallucination criteria.

  • Study design: 40 Prolific participants evaluated randomly sampled EMER videos with anonymized model generations displayed as captions.The survey was created in Qualtrics.
  • Evaluation criteria: Participants selected captions describing the person’s emotional state and associating that emotion with audiovisual cues.The evaluation also checks for inconsistencies or hallucinations.
  • Preference ablation: The multimodal-preference ablation tests random tensors, random videos, diffusion-noised inputs, and inputs depicting different emotions.Using a different-emotion video produces the best results among the tested alternatives.

E.6 RESPONSE PREFERENCE ABLATION

The ablation shows that response-preference choices, text-prior debiasing, and prompt-based modality preference each affect emotion-reasoning performance. AVEm-DPO also improves audiovisual grounding, adversarial robustness, and multimodal emotion prediction.

  • Response preference ablation: Using y_vr as the first rejected response gives AVEm-DPO its best performance across all EmoReAlM tasks.The tested alternatives include video-relevant, emotion-relevant, and completely irrelevant rejected responses.
  • Response preference ablation: Single rejected responses improve different subtasks: y_vr benefits Spurious Cue-Emotion Association, while y_er benefits Emotion-relevant Cue Hallucination.Using y_irr as the second rejected response produces the same or worse performance than y_vr.
  • Sensitivity to hyperparameters: Increasing β_vr mitigates spurious cue-emotion associations, whereas increasing β_er improves performance on hallucinated-cue samples.Text-prior debiasing improves hallucinated-cue performance at γ_TPD = 0.1 and saturates above 0.2; prompt-based modality preference saturates above λ_av = 1.0.
  • Attention analysis: AVEm-DPO increases attention to relevant modalities during reasoning and distributes attention more fairly across modalities for Modality Agreement.It also increases total attention to multimodal tokens rather than relying only on the input text prompt.
  • Adversarial robustness: AVEm-DPO remains unchanged when the prompt-irrelevant modality is replaced adversarially, while removing prompt-based modality preference produces wrong predictions.The adversarial tests evaluate visual and audio reasoning under irrelevant-modality perturbations.
  • Ablation across modalities: Audiovisual inputs outperform single-modality inputs for emotion prediction on RAVDESS, while video-only input outperforms audio-only input.The comparison covers audiovisual, video-only, and audio-only settings.

F QUALITATIVE SAMPLES

Qualitative examples show that AVEm-DPO produces emotion descriptions grounded in correct audiovisual cues and improves responses under spurious-association and cue-hallucination stress tests. The benchmark prompts separately construct audio, visual, agreement, and stress-test reasoning questions.

  • EMER emotion descriptions: AVEm-DPO produces correct emotion descriptions with consistent audiovisual cues and avoids associating irrelevant or background information with emotions.The comparison uses EMER samples and the prompt “Describe the audiovisual content relevant to emotion in detail.”
  • EmoReAlM sample outputs: AVEm-DPO improves model responses for EmoReAlM stress-test cases involving spurious emotion-cue associations and emotion-cue hallucinations.The examples are drawn from the Emotion Reasoning - Stress Test.
  • Captioning and prediction prompts: The benchmark generation prompts separately caption audio and video, predict emotions from each caption, and ask modality-specific cue-reasoning questions.Visual captioning explicitly ignores subtitles, while audio and video prediction use seven basic emotion categories.
  • Basic reasoning prompts: Audio and visual reasoning prompts require four-choice questions focused on cues supporting emotion without explicitly naming the emotion label.Incorrect choices may include plausible cues that are absent from the input.
  • Agreement and stress-test prompts: The Modality Agreement prompt asks whether audio and video align in conveying the emotion, while the stress-test audio prompt requires a mentioned cue to be present and emotion-supporting.Question-answer pairs are not generated when neither modality conveys the emotion label.
Loading 2602.07054v1…