An Agentic Approach with Evidence-Grounded Verification for Multimodal Emotion Reasoning in Conversation

1 Introduction

• Single-channel emotion systems proved unreliable, prompting multimodal fusion, but the task was still treated as mapping features to labels [Li and Deng, 2020; Schuller, 2018; Devlin et al., 2019; He et al., 2022, Tsai et al., 2020].The introduction argues that emotion perception is inference, not a lookup, and becomes harder as signals grow richer.Source: §1 · pp. 1–2
• MLLMs enable generative affect reasoning, yet incorrect claims are often judged by another model rather than checked against their visual, acoustic, or lexical evidence [Cheng et al., 2024; Lian et al., 2025a; Yang et al., 2025, Wang et al., 2026].A plausible explanation may leave both the reality of its evidence and the support for its conclusion unestablished.Source: §1 · p. 2
• The framework combines modality-specific cited observations, retrieved precedent, an integration agent and one possible critic revision with a separate evidence-grounded verification and targeted prompt-repair loop.Patches are validated on held-out data; on MER2025 Track 3, targeted repair beats a static pipeline on all four MLLMs and matches or beats two reflection baselines on each.Source: §1 · p. 2

2.1 LLMs and MLLMs for Emotion Understanding

• Frozen-prompting studies apply general-purpose language and vision-language models to affect tasks, while tuning approaches adapt model parameters with task-specific or emotion-oriented supervision [Feng et al., 2024; Etesam et al., 2024; Cheng et al., 2024; Lian et al., 2025a].These approaches do not independently re-derive each affective claim from cited raw signals and attribute failures to specific components.Source: §2.1 · pp. 2–3
• MERMAID and AffectAgent decompose emotion reasoning across agents using cross-modal reflection or retrieval-augmented collaboration, but are not designed for mechanical evidence validation and targeted prompt repair [Wang et al., 2026].Their mechanisms primarily aim to improve predictions rather than localize wrong final predictions to responsible agents.Source: §2.1 · p. 3
• EmoBench-M and EmotionHallucer evaluate emotional intelligence or emotion hallucinations, but assess model outputs rather than internal multi-agent reasoning traces [Hu et al., 2025; Xing et al., 2026].The related-work section frames process-level evidence verification, component attribution, and targeted repair as a distinct problem.Source: §2.1 · p. 3

3 Methodology

• The pipeline separates agentic draft generation from verification and prompt-space reinforcement, and leaves model weights unchanged.Verification audits incorrect predictions; attributed failures can drive later prompt updates.Source: §3 · p. 3, §3.2 · p. 5, §3.2.3 · pp. 6–7

3.1 Agentic Multimodal Emotion Reasoning

• Three frozen visual, acoustic, and lexical agents post observations with frame, time-span, or transcript-span evidence pointers to a shared board.The acoustic agent uses precomputed pitch, RMS energy, speech rate, and pause-length descriptors; the visual agent samples up to eight frames.Source: §3.1 · p. 4
• Retrieval independently returns five text-formatted neighbors using equally weighted transcript-text and keyframe-vision similarities, which integration reconciles with the evidence board.Integration can test cross-modal conflicts and issue one clarifying query; its draft contains an open-vocabulary description, label set, confidence, and citations.Source: §3.1 · p. 4
• The critic reviews the draft against the board and can request at most one redraft; its verdict is logged but excluded from later blame assignment.The final draft proceeds to verification after a pass or when the revision cap is reached.Source: §3.1 · p. 4

3.2 Verification Process

• Verification combines rule-based checks of cited evidence and references with an entailment check of whether the draft's claims are supported by that evidence.Deterministic checks test structural validity, while entailment evaluates claim content; only incorrect predictions are audited for failure attribution.Source: §3.2 · p. 5, §3.2.1 · p. 5

3.2.1 Deterministic Checks

• Deterministic checks re-derive cited visual and acoustic clues, check retrieval sanity, and validate citation provenance, but do not judge whether an emotional interpretation is correct.Acoustic values are checked within a 15% relative tolerance; citation checks cover source-reference validity, hash consistency, and resolution to a board entry or retrieved neighbor.Source: §3.2.1 · pp. 5–6

3.2.2 Entailment Check

• The entailment check decomposes the draft into claims and tests each against its cited evidence; wrong predictions with no traceable verification failure are excluded from patching.The entailment threshold is 0.5, and the selected NLI scorer achieved 84% agreement with human labels on 56 claim–evidence pairs.Source: §3.2.2 · p. 6

3.2.3 Prompt-Space Reinforcement

• Failures are clustered by responsible component and defect type, and patches are synthesized only after at least five same-type failures accumulate in a training round.A candidate is tested by rerunning the complete pipeline on held-out validation data and admitted only if it does not reduce open-vocabulary F-score; rejected patches leave the prompt unchanged.Source: §3.2.3 · p. 7

4.2 Main Results

• Targeted repair beats the static pipeline on all four models, leads on three, and ties on VideoLLaMA3-2B; global reflection trails static throughout.The targeted-versus-static gains are 2.2 F for Qwen2.5-VL-7B and 0.6 F for VideoLLaMA3-2B.Source: Table 1 · p. 8, §4.2 · pp. 8–9

4.3 Ablation Study

• Every component removal lowers F-score on InternVL2.5-38B; removing re-evaluation causes the largest drop (−3.0 F), while removing the critic causes the smallest (−0.5 F).Re-evaluation also causes the largest precision drop (−4.5); it is the pipeline’s safeguard against unconfirmed patches reaching later rounds.Source: Table 2 · p. 9, §4.3 · p. 9
• Removing retrieval (−2.1 F) and clustering (−1.8 F) causes larger drops than removing deterministic checks (−0.9 F) or the entailment check (−1.2 F).Retrieval supplies precedent, clustering aggregates similar defects, and entailment directly detects semantic corruption.Source: §4.3 · p. 9

4.4 Training Curve

• Across three models, F-score rises steadily as training data grows, with no meaningful decline at any tested fraction.The tested fractions range from 5% to 100% in nested subsets, evaluated on the held-out test split.Source: Figure 3 · p. 10, §4.4 · p. 10
• From 5% to 100% of training data, F-score gains are +2.0 F for Qwen2.5-VL-7B, +1.9 F for InternVL2.5-38B, and +1.2 F for Phi-3.5-Vision.The reported gains are modest in absolute terms, so a small labeled set captures much of the benefit while performance continues to climb.Source: §4.4 · p. 10
• Precision climbs later in the training-data range, while recall peaks around 70% to 75% and then declines slightly; F-score keeps improving.The recall rollover is sharpest for InternVL2.5-38B; the authors attribute continued F-score gains to precision increases outweighing the small recall loss.Source: §4.4 · p. 10

5 Limitations

• Errors whose citations pass every verification check remain irreducible, while sequential model calls add unreported latency, throughput, and per-clip cost.The repair loop’s reach is capped by the portion of errors attributable through the cited evidence.Source: §5 · p. 10

6 Conclusion and Future Work

• The framework traces errors to responsible agents through cited specialist observations, an integration draft refined by an inline critic, and a separate verification layer.The verification layer uses deterministic and entailment checks rather than relying on another model’s plausibility judgment.Source: §6 · pp. 10–11
• Across four MLLMs from 2B to 38B parameters on MER2025, targeted repair outperformed the frozen static pipeline on every backbone and matched or outperformed two reflection baselines.The ablation study found that each pipeline component contributes a measurable share of the gain.Source: §6 · p. 11
• Future work could use per-agent attributions to supervise fine-tuning and per-claim verification scores as process-level reinforcement-learning rewards.These directions would extend the current training-free loop to weight updates.Source: §6 · p. 11

Perceiver prompts

• Visual and lexical perceivers extract emotion-relevant clues grounded in specific frame IDs or verbatim transcript spans.The visual prompt requests 1–8 clues; the lexical prompt requests 3–8.Source: A Prompts · p. 13
• The acoustic perceiver produces one clue per segment, copying f0, RMS, speech-rate values and time spans exactly before interpreting them.It requires all three acoustic values in each clue and exactly one clue per listed segment.Source: A Prompts · p. 13

Integration and critic prompts

• The integrator combines board clues with retrieved examples as vocabulary context, reconciles cross-modal conflicts, and must account for or explain setting aside acoustic clues.It may request one follow-up from a single perceiver only when a specific ambiguity cannot be resolved from existing clues.Source: A Prompts · p. 14
• The critic checks whether claims are supported by cited clues, whether citations are valid, and whether deterministic checks pass; on failure it names one responsible agent and an actionable defect.A failing deterministic check is a strong signal that the draft should fail.Source: A Prompts · p. 14

Baseline prompts

• The Direct and Chain-of-Thought single-call baselines use the same transcript and thumbnail inputs; only the latter elicits a reasoning step.These are prompts for conditions 1 and 2, respectively.Source: A Prompts · p. 15

Round-3 repair

• The round-3 repair example appends a visual constraint against unsupported face claims and integrator guidance to align descriptions and labels with multimodal evidence and retrieved-neighbor labels.The visual patch comes from verifier-attributed visual-claim failures; the integrator guidance consolidates verifier-attributed failures.Source: A Prompts · p. 15

A Prompts

• By round 3, verifier-guided patches were appended verbatim to the corresponding round-0 prompts after their defect clusters passed the admission floor.Listing 8 gives examples of these patches for the targeted-repair condition.Source: A Prompts · p. 15

NeurIPS Paper Checklist

• The checklist answer is Yes: the abstract and introduction accurately reflect the paper’s contributions and scope.Source: NeurIPS Paper Checklist · p. 16
• The justification says the framework, MER2025 evaluation across four MLLMs, and F-score gains are substantiated in Sections 3–4 and the ablation study.Source: NeurIPS Paper Checklist · p. 16
• The checklist guidance says claims should match results and reflect expected generalization, while distinguishing attained claims from aspirational goals.Source: NeurIPS Paper Checklist · p. 16

• The verification stage cannot attribute or repair some errors, and the multi-call pipeline’s inference-time cost relative to single-call baselines is unquantified.Source: §2 · p. 16
• Tables 2–3 and Figure 2 report single-run point estimates on a fixed 480-sample test split without error bars, confidence intervals, or multiseed variance.Source: §2 · p. 19
• The paper does not report GPU or hardware type, memory, or wall-clock time for the perception, verification, or repair stages.Source: §2 · p. 19
• The paper includes no societal-impact discussion, leaving emotion recognition’s dual-use risk, including non-consensual affect inference or surveillance, unaddressed.Source: §2 · p. 20
• For the entailment scorer’s check against 56 human-labeled claims, the paper does not describe annotator instructions or compensation.Source: §2 · p. 21