PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMs

Jihoo Jung, Youngjoon Jang, Joon Son Chung

arXiv:2609.31193v1cs.CVcs.SDeess.AS

TL;DR

AVLLMs struggle to resolve who says what in multi-speaker videos, where correct reasoning requires trimodal binding. This paper analyzes symbolic binding mechanisms and introduces ASD-based visual prompting. ASD fine-tuning improves conversation-centric accuracy and also yields gains on general benchmarks.

  • Problem

    How AVLLMs resolve trimodal binding in multi-speaker videos remains open, although correct speaker attribution requires linking text, audio, and visual entities.

    Source: §1 · pp. 1–2

  • Method

    The paper analyzes modality-specific symbolic IDs and uses an off-the-shelf Active Speaker Detection model to overlay bounding boxes on active speakers.

    Source: §1 · p. 2

  • Results

    7.9%, 9.7%, and 3.5% accuracy gains follow ASD fine-tuning for video-SALMONN2+, Qwen2.5-Omni, and MiniCPM-o-4.5, with gains also reported on general benchmarks.

    Source: §1 · p. 2, §5.2 · p. 9

  • Takeaways & Limitations

    Faulty audio-visual binding, rather than text-audio or text-visual grounding, is the primary bottleneck for overall binding failures.

    Source: §6 · p. 10

    The main analysis relies on synthetic toy datasets and the single-shot setting, so its conclusions are focused rather than exhaustive across practical conditions.

    Source: E Limitations · p. 29

Abstract

from arXiv · show

Current Audio-Visual LLMs (AVLLMs) struggle with reasoning over videos featuring multi-speaker dialogues. In such videos, resolving "who says what" is crucial, which necessitates trimodal (text-audio-visual) binding. Motivated by these challenges, we systematically investigate how this trimodal binding is achieved in AVLLMs. Specifically, we identify emergent symbolic trimodal binding mechanisms in AVLLMs that utilize modality-specific symbolic variables. By encoding auditory and visual components into symbolic variables-capturing temporal utterance sequences and spatial entity coordinates, respectively-the model establishes cross-modal linking within this abstract space. Crucially, we reveal that when trimodal binding fails, the breakdown predominantly stems from misaligned audio-visual connections. To overcome this bottleneck, we introduce an audio-visual prompting method utilizing an off-the-shelf Active Speaker Detection (ASD) model. By simply overlaying visual bounding boxes on active speakers, this training-free approach yields immediate performance gains across four conversation-centric benchmarks. Moreover, lightweight fine-tuning of fewer than 300 steps on these ASD-prompted-videos extends these gains to three general AV benchmarks, suggesting the generalizability of our method.

1 Introduction

The introduction frames multi-speaker “who says what” reasoning as an unresolved trimodal binding problem for AVLLMs. It outlines a symbolic mechanism analysis and an ASD-based prompting method that improves benchmark performance.

1 Introduction

  • Resolving what “the man in the gray shirt” is saying requires robust in-context associations that jointly bind the textual description, corresponding visual subject, and target acoustic utterance.Models use content-independent symbolic IDs to track and parse objects, converting raw content into abstract symbolic representations [17] [56] [1].Source: §1 · p. 1
  • How AVLLMs perform trimodal binding remains an open question, especially in multi-speaker videos where attribution errors can assign an utterance to the wrong person.The paper investigates whether failures stem from text-to-modality grounding or direct audio-visual alignment.Source: §1 · pp. 1–2
  • The study tests binding in single-shot videos with simultaneous active and inactive speakers, using a four-animal toy task with utterance and visual anchors.The task asks the model to identify the corresponding attribute in the opposite modality.Source: §1 · p. 2
  • Binding failures primarily arise during target ID selection, indicating faulty audio-visual alignment rather than a failure of text grounding.The proposed ASD prompting method overlays bounding boxes on active speakers to address this deficit.Source: §1 · p. 2
  • AVLLMs use modality-specific symbolic IDs: temporal IDs represent utterance timing, while spatial IDs represent speaker locations.The three stages are anchor ID retrieval, target ID selection, and feature retrieval.Source: §1 · p. 2
  • 7.9%, 9.7%, and 3.5% accuracy gains follow ASD fine-tuning for video-SALMONN2+, Qwen2.5-Omni, and MiniCPM-o-4.5, with improvements also reported on general benchmarks.The training-free ASD approach also improves results across three conversation-centric benchmarks for Qwen2.5-Omni and MiniCPM-o-4.5.Source: §1 · p. 2

2 Related Works

Prior work examined symbolic binding in LLMs [12] [21] [17] [40] and multimodal binding in VLMs [1] [3] [22]; this work extends inquiry to trimodal AVLLM binding.

Audio-visual large language models

  • AVLLMs extend text-centric reasoning to audio and vision, but still struggle with intricate multi-speaker videos requiring precise cross-modal binding [46] [7] [36].The cited models include video-SALMONN2+ [45], Qwen2.5-Omni [53], and MiniCPM-o-4.5 [11].Source: §2 · pp. 2–3

Binding mechanisms in LLMs/VLMs

  • Research on LLMs studies symbolic IDs linking entities to attributes, while VLM work connects binding failures to multi-object reasoning and proposes mitigations [5] [22].This paper extends binding analysis from unimodal and bimodal contexts to text, audio, and vision.Source: §2 · p. 3

Visual prompting in VLMs

3 Symbolic Trimodal Binding Mechanisms in AVLLMs

AVLLMs use modality-specific symbolic IDs and a three-stage process to bind text, audio, and visual attributes. Representational and causal analyses support this account, with target ID selection emerging as the main failure point and the symbolic pattern also appearing in real-world data.

3.1 Trimodal Binding Task

  • The trimodal binding task tests association of textual, acoustic, and visual information in single-shot, multi-speaker videos, using acoustic or visual anchors to retrieve the complementary attribute.These define the AAVR and VAAR tasks, respectively.Source: §3.1 · p. 3

3.2 Symbolic Trimodal Binding Mechanisms

  • The proposed mechanism abstracts attributes into content-independent IDs: temporal IDs track acoustic attributes by speaking order, while position IDs track visual attributes by location.It proceeds through anchor ID retrieval, target ID selection, and feature retrieval.Source: §3.2 · p. 4

3.3 Representational Analysis

  • RSA places temporal IDs in AAVR and position IDs in VAAR at the anchor token, then target IDs at the last token before semantic information peaks in deepest layers.These patterns correspond to anchor ID retrieval, target ID selection, and feature retrieval.Source: Figure 2 · p. 4, §3.3 · p. 5

3.4 Causal Mediation Analysis

  • Causal Mediation Analysis patches states from modified-context runs into original runs and measures whether the intervention shifts the prediction toward an expected answer.A higher CM Score indicates that the patched embedding can steer the prediction toward that answer.Source: §3.4 · p. 5
  • Target ID manipulation has the strongest late-layer causal effect, while anchor ID manipulation peaks in mid-to-late layers and semantic manipulation in the deepest layers.These patterns, averaged over 960 samples, align with anchor ID retrieval, target ID selection, and feature retrieval, respectively.Source: §3.4 · p. 6, Figure 4 · p. 6

3.5 Real-World Generalization

  • In 2,000 SocialOmni clips, VAAR shows position IDs at the anchor token and temporal IDs at the last token, supporting real-world generalization.Overall correlations are attenuated by real-world variability; similar trends appear across other models and tasks.Source: Figure 5 · p. 6, §3.5 · p. 7

4 Diagnosing and Mitigating Trimodal Binding Failures

The section locates unprimed trimodal-binding failures primarily in target ID selection, where audio-visual correspondence breaks down. It then uses Active Speaker Detection (ASD) prompting to address that bottleneck.

4.1 Locating the Failure Bottleneck in Trimodal Binding

  • In the unprimed setting, target ID selection diverges substantially from the primed setting, while anchor ID retrieval shows negligible divergence, implicating audio-visual correspondence.The pattern holds across models and tasks.Source: §4.1 · p. 7
  • Patching correct symbolic IDs improves accuracy at target ID selection but has little effect at anchor ID retrieval, identifying the former as the key bottleneck.The analysis selects head-layer pairs and patches representations averaged over 800 primed instances into the unprimed setting.Source: §4.1 · p. 7

4.2 Audio-Visual Prompting via Active Speaker Detection

  • ASD-FT yields consistent gains across all three AVLLMs and all seven benchmarks, while training-free ASD improves conversation-centric tasks for Qwen2.5-Omni and MiniCPM-o-4.5.The general benchmarks often lack speech or visible speakers, so training-free ASD is not applied there.Source: Table 1 · p. 8
  • The proposed prompt overlays a red box on the detected active speaker and explains the cue in text; the authors also fine-tune with ASD-prompted videos using LoRA.The prompting approach can work without training.Source: §4.2 · p. 8

5 Experiments

Experiments evaluate ASD prompting and fine-tuning on conversation-centric and general audio-visual benchmarks. ASD-FT improves results across both benchmark groups, while prompting-style and fine-tuning comparisons probe what contributes to the gains.

5.1 Experimental Setup

  • The fine-tuning corpus contains 400 synthetic human-speaker videos with red boxes, paired with captioning and multiple-choice tasks; training uses LoRA for fewer than 300 steps.The evaluation uses three AVLLMs; fine-tuning is conducted on a single NVIDIA RTX A6000 GPU.Source: §5.1 · p. 8
  • The evaluation covers four conversation-centric benchmarks and three general audio-visual benchmarks, using ASR and REF for conversation tasks.The setup includes multiple-choice QA, dialogue captioning, egocentric videos, image-audio pairs, and diverse audio categories.Source: §5.1 · p. 8

5.2 Experimental Results

  • On AVSpeaker with Qwen2.5-Omni (7B), training-free ASD outperforms the compared decoding- and visual-prompting methods.The comparison includes AVCD, FMD, OutRo, NumPro, and VISER.Source: Table 2 · p. 9, §5.2 · p. 9
  • After ASD-FT, target-ID correlation is higher in late layers on unprimed AAVR and VAAR, consistent with mitigating target ID selection errors.Figure 7 compares the vanilla model with ASD-FT.Source: Figure 7 · p. 9, §5.2 · p. 9
  • ASD-FT improves all three models across three conversation-centric datasets and brings consistent gains on general datasets; training-free ASD helps on three speech-centric datasets for two models.ASD-FT also shows consistent gains on the broader general datasets.Source: §5.2 · pp. 8–9
  • Bounding boxes on inactive speakers or random regions degrade performance, removing the explanatory prompt also hurts, and gains are robust to box color and thickness.These results attribute the gains to accurate active-speaker localization and explicit visual grounding instructions.Source: §5.2 · p. 9, Table 3 · p. 9
  • ASD-FT yields larger gains than unboxed fine-tuning on 4 of 5 datasets, suggesting the prompted setup directs learning toward audio-visual alignment.Unboxed fine-tuning also improves performance.Source: §5.2 · p. 9, Table 4 · p. 9

6 Conclusion

The analysis identifies modality-specific symbolic IDs and faulty audio-visual binding as the primary bottleneck in trimodal binding failures, then proposes ASD-based visual prompting to address it.

6 Conclusion

  • Faulty audio-visual binding, rather than text-audio or text-visual grounding, is identified as the primary bottleneck, motivating prompting with an off-the-shelf ASD model.The conclusion presents these findings as foundational insights for future AVLLM architecture and training strategies.Source: §6 · p. 10

A.1 Reasons to Introduce Priming

  • Priming raises accuracy from below 30% in unprimed settings to near-ceiling levels, enabling causal analysis on reliable binding trajectories.The authors argue that causal patching is difficult to interpret when performance is near chance.Source: §A.1 · p. 15
  • Removing audio and video while keeping the prime yields near-floor accuracy, supporting the conclusion that primed answers require audiovisual content.The reported prompt-only accuracies are 1.0% for video-SALMONN2+, 12.5% for Qwen2.5-Omni (7B), and 0.06% for MiniCPM-o-4.5.Source: §A.1 · p. 15
  • Patching anchor-position symbolic IDs shifts output logits toward the associated target, while primed-head activation injection improves unprimed accuracy across three models.Together, these results support the claim that priming makes the same binding circuit measurable rather than inducing a separate circuit.Source: §A.1 · p. 16

A.2.1 Qwen2.5-Omni(7B)

  • For Qwen2.5-Omni (7B), the three-stage flow manifests across prompt and last tokens collectively, not at fixed token positions.The reported VAAR analysis has target-ID selection beginning at prompt-token layers, a model-specific variation consistent with the framework.Source: §A.2.1 · p. 16

A.2.2 Qwen2.5-Omni(3B)

  • Qwen2.5-Omni (3B) shows anchor-ID retrieval, then target-ID selection, then feature retrieval in representational and causal analyses.The anchor ID dominates at the anchor token in mid-to-late layers; target-ID effects peak at the last token in late layers, followed by semantic effects in deepest layers.Source: §A.2.2 · p. 16

A.2.3 MiniCPM-o-4.5

  • MiniCPM-o-4.5 (9B) shows the same three-stage pattern, supporting generalization across model families and parameter scales.Its causal mediation results largely echo representational patterns, with minor stage-specific deviations.Source: §A.2.3 · p. 16

A.3 Generalization of Binding Mechanism Beyond the Controlled Four-Speaker Setting

  • The binding pattern persists in five- and six-speaker settings, including five-speaker videos with MUSAN noise at 20 and 10 dB SNR.The authors report the mechanism also generalizes beyond the specific 2 × 2 grid to broader synthetic and real-world settings.Source: §A.3 · p. 17

A.4 Representation Analysis on Different Models on Primed vs. Unprimed Settings

Across four AVLLMs, primed and unprimed settings show similar anchor-ID retrieval, but diverge at target-ID selection.

A.4 Representation Analysis on Different Models on Primed vs. Unprimed Settings

  • Across the four models and both AAVR and VAAR tasks, anchor-ID retrieval shows no observable primed–unprimed difference, while unprimed target-ID selection is less effective.The comparison covers video-SALMONN2+, Qwen2.5-Omni (7B and 3B), and MiniCPM-o-4.5.Source: §A.4 · pp. 17–18

A.5 Real World Generalization Analysis on Different Models

Real-world analyses across models show trends consistent with the proposed binding mechanism, though less pronounced than in toy data. The appendix also details the prompting procedure, synthetic training corpus, and training setup.

A.5 Real World Generalization Analysis on Different Models

  • In real-world data, AAVR shows temporal-ID correlation at the anchor token and stronger position-ID correlation at the last token; VAAR shows the converse pattern.These trends are consistent with the proposed mechanism across the tested models, though less pronounced than in the toy dataset.Source: §A.5 · pp. 19–20

B.1 Details of Training Free Prompting

  • Training-free prompting marks the active speaker with a red box and adds task-specific instructions directing QA or captioning models to attend to it.For QA, the instruction is prepended; for captioning, the note is appended to the original prompt.Source: §B.1 · p. 20
  • With weaker ASD models, ASD-FT reaches 48.4% and 48.6% on AVSpeaker, close to 49.1% with [37] and above the vanilla model’s 44.7%.The passage reports these results as evidence of robustness to ASD quality.Source: §B.1 · p. 20

B.2.1 Training Datasets

  • The synthetic corpus uses red-box active-speaker annotations and three configurations: two- or four-person single-shot videos and two-person multi-shot videos.The configurations vary the number of people on screen and whether speakers appear together or across consecutive shots.Source: §B.2.1 · p. 21

B.2.2 Implementation Details

  • Training used 140 steps for video-SALMONN2+ and 250 steps for the other models, with exact boxes generated during synthetic data creation.Evaluation used greedy decoding.Source: §B.2.2 · p. 24

B.3 Discussion on the Experimental Results

Training-free ASD prompting depends on zero-shot visual grounding, while fine-tuning can address that dependency. The VAAR illustration traces binding from a visual attribute to a spoken word.

B.3 Discussion on the Experimental Results

  • Training-free ASD prompting relies on zero-shot visual grounding of overlaid markers; video-SALMONN2+ does not benefit from it, while ASD-FT bridges that gap.The discussion attributes the difference to whether models can interpret bounding boxes as active-speaker indicators.Source: §B.3 · p. 25

C.1 Illustration of Symbolic Mechanism in VAAR task

  • In VAAR, a visual attribute retrieves a position ID, which maps to a temporal ID used to extract the target word’s feature.The illustration’s example maps the position ID to temporal ID 1 and retrieves “Japan.”Source: §C.1 · p. 25

C.2 Illustration of Causal Mediation Analysis for AAVR task

The AAVR causal mediation analysis tests a three-stage binding hypothesis by manipulating temporal IDs, position IDs, or semantic content at each stage. Each stage is predicted to affect the output only when its corresponding information type is manipulated.

C.2 Illustration of Causal Mediation Analysis for AAVR task

  • The configurations form a 3 × 3 grid crossing three binding stages with temporal ID, position ID, and semantic content manipulations.The reported predictions are consistent with the empirical results and support the view that each stage encodes a distinct, causally separable information type.Source: §C.2 · pp. 25–26
  • At anchor ID retrieval, temporal ID manipulation alone is predicted to flip the output to the counterfactual target.Position ID and semantic content manipulations are predicted to leave the output essentially unchanged.Source: §C.2 · p. 26, Figure 21 · p. 26, Figure 22 · p. 26
  • At target ID selection, position ID manipulation alone is predicted to flip the output to the counterfactual target.Temporal ID and semantic content manipulations are predicted to be ineffective.Source: §C.2 · p. 26
  • At feature retrieval, semantic content manipulation alone is predicted to flip the output to the counterfactual target.Temporal and position ID manipulations are predicted to have no comparable effect.Source: §C.2 · p. 26

D Qualitative Results

Qualitative examples show that audio-visual prompting can address speaker-attribution errors in both multi-speaker question answering and free-form captioning. The captioning example also shows consistently aligned speaker–utterance pairings with the ASD variant.

D Qualitative Results

  • Bounding-box prompting lets the model correctly align utterances with speakers in multi-speaker QA examples where the vanilla model misattributes them.The paper describes this as reflecting the audio-visual binding bottleneck identified in its analysis.Source: D Qualitative Results · pp. 26, 29
  • On a DiaDemBench multi-speaker captioning example, the ASD variant consistently aligns utterances with the correct speakers, unlike the vanilla model.The example illustrates the prompting effect in a generative setting as well as the discriminative QA examples.Source: D Qualitative Results · p. 29

E Limitations

The paper bounds its findings as a focused analysis rather than an exhaustive evaluation of practical audiovisual conditions, and reports the computational resources used. It also identifies possible downstream effects of biases in the off-the-shelf speaker detector.

E Limitations

  • The main analysis uses synthetic toy datasets and excludes multi-shot scenarios, limiting coverage of real-world scene complexity and settings.The authors focus on the more challenging single-shot setting and say their method does not modify AVLLMs’ intrinsic capabilities.Source: E Limitations · p. 29

F Computational Resource

  • Most experiments used an AMD EPYC 7513 CPU and one NVIDIA RTX A6000 GPU; WorldSense evaluation used an NVIDIA B6000 GPU.The WorldSense evaluation required larger memory.Source: F Computational Resource · p. 29

G Social Impact

  • Biases or demographic disparities in the off-the-shelf Active Speaker Detection model could propagate into downstream predictions and affect populations unequally.The authors recommend consent frameworks, fairness audits across demographic groups, and dataset documentation practices to mitigate these risks.Source: G Social Impact · pp. 29, 32
Loading 2609.31193v1…