Source-linked AI summary
Uncovering Ordinal-Matching Bias in Audio-Visual LLMs
Jihoo Jung, Youngjoon Jang, Hyebin Cho, Suho Yoo, Joon Son Chung
TL;DR
Current AVLLMs often misattribute speech to visible speakers, raising questions about whether they use genuine audio-visual correspondence. The paper diagnoses ordinal-matching bias and tests fine-tuning with randomized speaker positions and utterance order, finding reduced bias and improved performance on real-world benchmarks.
Problem
Despite strong performance on general audio-visual benchmarks, current AVLLMs often assign utterances to the wrong visible speaker, an important capability for conversation-rich videos.
Source: §1 · p. 1Method
The authors evaluate three open-source AVLLMs on a synthetic diagnostic corpus and fine-tune models on 400 synthetic videos with randomized speaker positions and utterance orders.
Source: §1 · p. 2Results
Across three real-world audio-visual benchmarks, OD-FT yields average gains of 8.27% for Qwen2.5-Omni and 2.57% for video-SALMONN2+, while suppressing ordinal-matching bias.
Source: §1 · p. 2Takeaways & Limitations
The findings highlight the need for AVLLMs built on genuine audio-visual alignment rather than fragile shortcuts.
Source: §4 · p. 4The diagnostic design avoids potential confounds from human demographic attributes by using animal speakers.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
This work aims to improve how audio-visual large language models (AVLLMs) associate speech with the correct visible speaker in multi-speaker scenes. We find that current AVLLMs frequently fail at this task, and analyze the nature of these failures. To this end, we construct a synthetic diagnostic dataset in which multiple visible speakers each utter a single word. Analysis on this corpus reveals a consistent error pattern across three recent open-source AVLLMs: models attribute utterances by simply matching the order of spoken sentences with the left-to-right, top-to-bottom arrangement of visible faces, rather than relying on audio-visual cues such as lip synchronization. We term this behavior \emph{ordinal-matching bias}. We further show that this bias can be substantially mitigated through a simple remedy, Ordinal-Decoupled Fine-Tuning (OD-FT), in which models are fine-tuned on synthetic videos where spatial positions of speakers and speaking order are independently randomized. Despite using only 400 synthetic training videos, OD-FT not only suppresses ordinal-matching bias but also improves audio-visual understanding on real-world videos, yielding average gains of 8.27\% for Qwen2.5-Omni and 2.57\% for video-SALMONN2+ across three audio-visual benchmarks.
1. INTRODUCTION
Current AVLLMs often misattribute speech to visible speakers, and a preliminary experiment points to reliance on visual position rather than genuine audio-visual correspondence. The paper identifies ordinal-matching bias and introduces a fine-tuning remedy that improves results on real-world benchmarks.
1. INTRODUCTION
- Speaker-utterance attribution is important for conversation-rich videos, yet current AVLLMs often assign utterances to the wrong visible person, motivating analysis of spatial and temporal shortcuts.The proposed shortcut is reliance on face arrangement and utterance order rather than genuine audio-visual correspondence.Source: §1 · p. 1
- In a two-speaker test, the model was more accurate when the first speaker was on the left, and the advantage persisted after horizontal flipping of the same videos.The flip changed speaker positions while preserving semantic content.Source: §1 · pp. 1–2
- Across a controlled corpus, three open-source AVLLMs systematically match the k-th utterance to the k-th speaker in visual order instead of relying on cues such as lip synchronization.The paper terms this pattern ordinal-matching bias.Source: §1 · p. 2
- OD-FT, trained on 400 synthetic videos, suppresses ordinal-matching bias and yields average real-world benchmark gains of 8.27% for Qwen2.5-Omni and 2.57% for video-SALMONN2+.The training videos randomize speaker positions and utterance order independently.Source: §1 · p. 2
2. IDENTIFYING ORDINAL-MATCHING BIAS
The section uses a synthetic diagnostic corpus to identify ordinal-matching bias in speaker attribution, then tests that bias through positional interventions. Open-source AVLLMs perform at or near chance, while their predictions track spatial speaker order; under intervention, they follow rewritten positions.
2. IDENTIFYING ORDINAL-MATCHING BIAS
- The section constructs a synthetic diagnostic corpus to identify ordinal-matching bias and provide causal evidence for it.Source: §2 · p. 2
2.1. Analysis Setup
- The diagnostic corpus uses 4–6 distinct animal speakers at fixed positions, each uttering one country name in a randomized sequence.It evaluates Says-What and Who-Says tasks; animal speakers avoid potential confounds from human demographic attributes such as gender.Source: §2.1 · p. 2
2.2. Analysis Results
- Open-source AVLLMs perform at or near chance, while their ordinal-matching rate is high; under positional interventions, predictions follow rewritten positions.The intervention lowers matching to the original row-major order while matching to the injected order remains high.Source: Table 1 · p. 3, §2.2 · p. 3, Table 2 · p. 3
2.3. Causal Validation via Positional Intervention
- Swap and Rotate interventions rewrite visual positional coordinates while preserving the on-screen order, testing whether predictions follow original or intervened positions.Across intervention types and models, ordinal matching to the original order drops, while matching to the injected order stays high.Source: §2.3 · p. 3, Table 2 · p. 3
3. MITIGATING ORDINAL-MATCHING BIAS
The section tests whether counterfactual fine-tuning can mitigate ordinal-matching bias and improve real-world benchmark performance. OD-FT reduces the bias and raises performance across three benchmarks, while increasing the share of decoupled examples also raises SocialOmni accuracy.
3.1. Ordinal-Decoupled Fine-tuning Experiments
- The fine-tuning corpus contains 400 videos across two- and four-speaker layouts, while OC-FT preserves strict ordinal correspondence.The section tests whether counterfactual fine-tuning can mitigate bias and improve real-world benchmark performance.Source: §3.1 · pp. 3–4, §3 · p. 3
3.2. Experimental Results
- OD-FT improves accuracy across three real-world benchmarks, with average gains of 8.27% for Qwen2.5-Omni and 2.57% for video-SALMONN2+, while reducing ordinal-matching to near-chance levels.OC-FT provides little synthetic-corpus accuracy improvement, exacerbates the bias, and yields smaller real-world gains.Source: Table 3 · p. 4, §3.2 · p. 4
- On SocialOmni, accuracy increases as decoupled examples rise from 0% (OC-FT) to 100% (OD-FT).The ablation also evaluates mixtures containing 25%, 50%, and 75% decoupled examples.Source: Figure 3 · p. 4, §3.2 · p. 4
4. CONCLUSION
Current open-source AVLLMs show ordinal-matching bias in speaker attribution, but fine-tuning on 400 synthetic videos suppresses it and improves performance across three real-world benchmarks.
4. CONCLUSION
- Current open-source AVLLMs remain unreliable at attributing utterances to the correct visible speaker and follow a systematic ordinal-matching bias.They pair the k-th utterance with the k-th speaker in visual position order.Source: §4 · p. 4
- Ordinal-Decoupled Fine-Tuning on only 400 synthetic videos suppresses the bias, and the resulting improvements transfer to three real-world benchmarks.Source: §4 · p. 4
- The evaluation centers on structured environments and clean speech; dynamic multi-party extensions and root-cause analyses remain future work.Source: §4 · p. 4