Source-linked AI summary
Conflict-Aware Multimodal Fusion for Ambivalence and Hesitancy Recognition
Salah Eddine Bekhouche, Hichem Telli, Azeddine Benlamoudi, Salah Eddine Herrouz, Abdelmalik Taleb-Ahmed, Abdenour Hadid
TL;DR
A/H recognition is difficult because clinically meaningful evidence often lies in disagreements among language, voice, and facial behaviour, while text-only systems can over-detect A/H. ConflictAwareAH combines pretrained video, audio, and text encoders with bidirectional pairwise conflict features and text-guided late fusion, achieving 0.694 Macro F1 on the labelled test split and 0.715 on the private leaderboard.
Problem
A/H is difficult to detect automatically because clinically relevant evidence appears in conflicting signals across modalities, while text-dominant approaches struggle to confirm its absence.
Method
ConflictAwareAH extracts video, audio, and text representations, computes pairwise absolute-difference conflict features, and blends the full model with a text-only auxiliary head.
Results
0.694 Macro F1 on the labelled test split and 0.715 on the ABAW10 private leaderboard were achieved, outperforming published BAH multimodal baselines.
Takeaways & Limitations
+4.6 points F1-NoAH over text alone and a halved class-performance gap show that conflict-aware fusion improves confirmation of the negative class, while late fusion adds +4.1 Macro F1.
Takeaways & Limitations
Threshold tuning may not transfer under significant test-distribution shifts, and 16-frame windows do not fully solve temporal modelling; the conflict features capture geometric rather than semantic distance.
Abstract
from arXiv · showhide
Ambivalence and hesitancy (A/H) are subtle affective states where a person shows conflicting signals through different channels -- saying one thing while their face or voice tells another story. Recognising these states automatically is valuable in clinical settings, but it is hard for machines because the key evidence lives in the \emph{disagreements} between what is said, how it sounds, and what the face shows. We present \textbf{ConflictAwareAH}, a multimodal framework built for this problem. Three pre-trained encoders extract video, audio, and text representations. Pairwise conflict features -- element-wise absolute differences between modality embeddings -- serve as \emph{bidirectional} cues: large cross-modal differences flag A/H, while small differences confirm behavioural consistency and anchor the negative class. This conflict-aware design addresses a key limitation of text-dominant approaches, which tend to over-detect A/H (high F1-AH) while struggling to confirm its absence: our multimodal model improves F1-NoAH by +4.6 points over text alone and halves the class-performance gap. A complementary \emph{text-guided late fusion} strategy blends a text-only auxiliary head with the full model at inference, adding +4.1 Macro F1. On the BAH dataset from the ABAW10 Ambivalence/Hesitancy Challenge, our method reaches \textbf{0.694 Macro F1} on the labelled test split and \textbf{0.715} on the private leaderboard, outperforming published multimodal baselines by over 10 points -- all on a single GPU in under 25 minutes of training.
1. Introduction
Ambivalence and hesitancy are clinically relevant states marked by conflicting signals across channels, making them difficult for standard systems to detect. ConflictAwareAH addresses this challenge by modelling cross-modal disagreement and consistency rather than relying on agreement alone.
- Ambivalence involves conflicting feelings, while hesitancy involves reluctance or delay before committing; both occur frequently in clinical psychology, behavioural economics, and human–computer interaction.
- In medical interviews, verbal agreement can coexist with facial signs of doubt or stress, creating clinically relevant mismatches.Detecting such mismatches can help clinicians adjust motivational interviewing and identify attitudes linked to vaccine hesitancy or treatment refusal.
- A/H differs from basic emotions because its key evidence often appears as contradictory signals across facial, vocal, and linguistic channels.Standard multimodal fusion may suppress these disagreements because it is designed to identify cross-modal agreement.
- +4.6 points F1-NoAH over text alone and a halved class-performance gap demonstrate the benefit of conflict-aware multimodal fusion.The framework uses explicit pairwise conflict features as bidirectional cues: large differences flag A/H, while small differences anchor the negative class.
- +4.1 Macro F1 comes from text-guided late fusion of a text-only auxiliary head with the full multimodal model at inference.
2. Related Work
Related work establishes BAH as a small-data multimodal benchmark shaped by strong linguistic signals, clinically important cross-modal inconsistencies, and established video, speech, language, pooling, and fusion techniques. ConflictAwareAH builds on these foundations with explicit conflict modelling and efficient temporal coverage.
- Affective Behaviour Analysis and the ABAW Challenges: ABAW10 extends the A/H challenge with the full BAH dataset, participant-wise splits, and an unlabelled hold-out for evaluation.
- Prior A/H Recognition: Savchenko’s 0.772 Macro F1 established text as the primary carrier of A/H information, while later multimodal work confirmed speech semantics were more discriminative than acoustic prosody alone.
- BAH Dataset and A/H Recognition: Facial–language inconsistency accounts for 42.6% of annotated conflict segments, motivating explicit modelling of cross-modal disagreement.The best prior three-modality fusion reached 0.59 Macro F1, while transcript-based zero-shot multimodal-LLM inference reached 0.63.
- Video Representation Learning: VideoMAE-Base provides 16-frame visual representations, while VideoFocalNet offers a visual-only alternative with 0.566 Macro F1 on the BAH benchmark.
- Temporal Modelling and Small Affective Datasets: Five randomly sampled 16-frame windows provide implicit temporal coverage without an explicit temporal decoder, trading richer temporal modelling for efficiency on 778 training videos.
- Speech and Language Representations: HuBERT captures prosodic cues such as pauses, pitch variation, and stuttering, complementing emotion-aware language representations sensitive to confusion, doubt, and uncertainty.
- Multimodal Fusion and Conflict Modelling: The framework uses hybrid fusion: conflict-augmented early fusion is followed by late blending with a unimodal text branch.
- Attention Pooling: Attention pooling summarizes variable-length encoder outputs by upweighting emotionally salient moments and downweighting uninformative tokens.
3. Method
ConflictAwareAH combines pre-trained video, audio, and text encoders with attention pooling and explicit pairwise conflict features to model cross-modal disagreement. A text-only auxiliary head is jointly trained and blended with the full-fusion prediction at inference.
- Multimodal Encoders: The model predicts video-level A/H labels from aligned video, audio, and transcript inputs using three pre-trained modality encoders.All three encoder backbones produce 768-dimensional embeddings.
- Multimodal Encoders: Attention pooling reduces each encoder’s variable-length sequence to a fixed-size embedding while emphasizing discriminative hesitation markers.Padding positions are masked, and modality-specific projections map outputs to the shared dimension D=768.
- Conflict-Aware Fusion: Pairwise absolute differences between modality embeddings expose cross-modal disagreement as explicit conflict features for A/H prediction.The full fusion representation concatenates modality embeddings with conflict features; the classifier learns which disagreements matter.
- Conflict-Aware Fusion: The fusion design is not pre-aligned contrastively: modality projections learn alignment entirely from the classification signal.The full-fusion representation is processed by a two-layer feed-forward network and classification head.
- Text-Guided Late Fusion: A parallel text-only head is jointly trained and blended with the full-fusion branch at inference to retain text’s strong discriminative signal.The blend uses p = α · σ(ℓtext) + (1 − α) · σ(ℓfull), with α=0.6 tuned on validation data.
- Training and Inference: Training uses frozen encoders by default, while multi-window inference averages five random 16-frame windows for broader visual coverage.The implementation uses AdamW, mixed precision, and an effective batch size of 16; the method is presented as an efficiency trade-off for the small training set.
4. Experiments
Experiments evaluate ConflictAwareAH on the participant-wise BAH benchmark, showing gains from text-guided fusion, conflict features, and checkpoint ensembling while exposing text-only class bias and small-data limitations.
- 4.3. Main Results: 0.694 Macro F1 is achieved by the two-checkpoint ensemble, exceeding the strongest BAH baseline at 0.634 by +5.6 points.A single model already surpasses that baseline, while the ensemble produces the reported labelled-test score.
- 4.3. Main Results: 0.715 official Macro F1 is obtained on the ABAW10 private test split containing 161 videos with withheld labels.The submitted ensemble is evaluated by the challenge organizers on this private split.
- 4.4. Ablation Studies: +4.1 Macro F1 results from blending the full model with a text-only branch at α=0.6, increasing performance from 0.649 to 0.690.The text-only branch alone reaches 0.700 Macro F1, the best single-branch score in this ablation.
- 4.4. Ablation Studies: +0.9 points are contributed by explicit conflict features, raising Macro F1 from 0.681 without them to 0.690 in the full model.The comparison supports a measurable, though modest, contribution from cross-modal disagreement features.
- 4.5. Analysis and Discussion: Text-only achieves 0.804 F1-AH but 0.670 F1-NoAH, whereas the conflict-aware model reaches 0.777 and 0.716, narrowing the class-performance gap from 0.134 to 0.061.The multimodal model improves F1-NoAH by +4.6 points while sacrificing 2.7 points on F1-AH.
5. Conclusion
ConflictAwareAH combines conflict-aware multimodal fusion with text-guided late fusion to address text-only bias in A/H recognition. It achieves strong BAH benchmark performance while remaining efficient to train.
- +4.6 points in F1-NoAH and a halved class-performance gap compared to text alone result from explicit pairwise cross-modal difference features.Large differences flag A/H, while small differences anchor the negative class.
- +4.1 Macro F1 comes from blending a text-only auxiliary head with the full multimodal output.
- 0.694 Macro F1 on the labelled test split and 0.715 on the ABAW10 private leaderboard are the method’s best reported results.The best result ensembles two checkpoints with complementary regularisation and multiple inference windows.
- All published BAH multimodal baselines are outperformed, including the best baseline at 0.593 and the zero-shot multimodal LLM at 0.634.
- Training takes under 25 minutes on a single GPU, while future work targets alignment-aware conflict representations, temporal modelling, and richer regularisation.