Source-linked AI summary
CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios
Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, Xiaochun Cao
TL;DR
Dynamic audio-visual question answering remains challenging because MLLMs can give ambiguous descriptions of specific events. CAT combines question-aware clue aggregation, mixed audio-visual training, and ambiguity-aware preference optimization. The authors report advantages on AVQA and results across other multimodal tasks.
Problem
In dynamic audio-visual scenarios, MLLMs can respond ambiguously and fail to describe specific objects, while cross-domain alignment makes detailed AVQA answers challenging.
Source: §1 · p. 2Method
CAT aggregates question-related audio-visual clues, trains on mixed multimodal data including AVinstruct, and uses AI-assisted ambiguity-aware preference optimization to favor precise responses.
Source: §1 · p. 3Results
CAT outperforms state-of-the-art methods on a variety of AVQA tasks and achieves notable results in video-based text generation and zero-shot video question answering.
Source: §1 · p. 4Takeaways & Limitations
The authors conclude that CAT achieves comparable results across a variety of complex audio-visual scenarios.
Source: §5 · p. 15
Abstract
from arXiv · showhide
This paper focuses on the challenge of answering questions in scenarios that are composed of rich and complex dynamic audio-visual components. Although existing Multimodal Large Language Models (MLLMs) can respond to audio-visual content, these responses are sometimes ambiguous and fail to describe specific audio-visual events. To overcome this limitation, we introduce the CAT, which enhances MLLM in three ways: 1) besides straightforwardly bridging audio and video, we design a clue aggregator that aggregates question-related clues in dynamic audio-visual scenarios to enrich the detailed knowledge required for large language models. 2) CAT is trained on a mixed multimodal dataset, allowing direct application in audio-visual scenarios. Notably, we collect an audio-visual joint instruction dataset named AVinstruct, to further enhance the capacity of CAT to model cross-semantic correlations. 3) we propose AI-assisted ambiguity-aware direct preference optimization, a strategy specialized in retraining the model to favor the non-ambiguity response and improve the ability to localize specific audio-visual objects. Extensive experimental results demonstrate that CAT outperforms existing methods on multimodal tasks, especially in Audio-Visual Question Answering (AVQA) tasks. The codes and the collected instructions are released at https://github.com/rikeilong/Bay-CAT.
1 Introduction
CAT addresses ambiguous and insufficiently question-aware responses in dynamic audio-visual scenarios with question-related clue aggregation, mixed audio-visual training, and ambiguity-aware preference optimization. The authors report strong results across AVQA and other multimodal tasks.
1 Introduction
- Dynamic audio-visual questions remain challenging because modality alignment is difficult, and ambiguous descriptions of specific objects can impair detailed AVQA answers.Existing systems may miss question-relevant details or produce random guesses and overly long, unhelpful responses.Source: §1 · pp. 2–3
- CAT's clue aggregator captures question-aware audio-visual features, while mixed video, audio, and joint-instruction training supports use in scenarios containing both modalities.The authors introduce AVinstruct, an audio-visual joint instruction dataset intended to further support AVQA.Source: §1 · p. 3
- AI-assisted ambiguity-aware direct preference optimization trains CAT to favor precise rewritten responses over ambiguous descriptions of specific audio-visual objects.Ambiguous responses are treated as negative examples and rewritten with GPT into positive responses.Source: §1 · pp. 3–4
- CAT outperforms state-of-the-art methods on several AVQA tasks and achieves notable results in video-based text generation and zero-shot video question answering.The introduction reports these outcomes without specifying benchmark scores.Source: §1 · p. 4
2 Related Works
The related work describes AVQA's shift toward dynamic audio-visual reasoning and reviews bridging and preference-learning approaches. It notes that existing bridging methods can struggle to capture fine-grained information or depict specific events accurately.
Audio-visual question answering
- Dynamic AVQA datasets require temporal and spatial reasoning across modalities, extending earlier tasks that often sought a simple answer about the whole video.Examples include identifying instruments from audio or choosing among answers using dynamic audio-visual clues.Source: §2 · p. 4
MLLMs for audio-visual question answering
- Bridging methods range from linear feature-alignment layers to more complex networks, but both can struggle to capture fine-grained information or depict specific audio-visual events.The passage identifies multimodal-corpus alignment as a challenge for these approaches.Source: §2 · p. 4
Human-preference learning
- Human-preference learning uses preference signals to shape model responses; Direct Preference Optimization learns preferences directly without a reward model.The paper cites DPO as inspiration for addressing ambiguity.Source: §2 · pp. 4–5
3 Our Approach
CAT combines visual, audio, and question-related clues for an LLM, then trains on multimodal instructions and preference feedback to favor non-ambiguous descriptions. Its approach includes multimodal inputs, question-aware clue aggregation, staged training, and AI-assisted ADPO.
3.1 Multimodal Inputs
- CAT encodes video and audio with a frozen ImageBind encoder, temporally compresses video features, and projects both modalities into tokens for the LLM.The encoded features are aligned to the LLM’s hidden dimensions with linear projection layers.Source: §3.1 · pp. 5–6
3.2 Aggregating Key Clues
- The Clue Aggregator uses question-aware perceivers to localize visual and audio clues, then pools them into clue tokens for the LLM.Visual and audio perceivers share parameters; a learnable query vector of length 48 extracts information from their question-aware features.Source: §3.2 · pp. 6–7
3.3 Multimodal Training Strategy
- CAT uses separate video-text and audio-text feature-alignment stages, followed by joint instruction tuning on video instructions and AVinstruct.AVinstruct is assembled from videos with audio, AVQA question-answer pairs, generated video descriptions, and GPT-synthesized question-guided audio-visual descriptions.Source: §3.3 · p. 7
3.4 AI-assisted Ambiguity-aware Direct Preference Optimization
- ADPO uses GPT to identify ambiguous trained-CAT responses and rewrite them as positive responses, then trains CAT to favor positives over negatives.It combines DPO with an SFT-like objective to stabilize the model’s bias toward positive responses.Source: Figure 4 · p. 8, §3.4 · pp. 8–9
4 Experiments
CAT is evaluated on video understanding, video question answering, and closed- and open-ended audio-visual question answering, with component ablations and qualitative comparisons. The reported comparisons favor CAT across several tasks, while the analyses examine clue aggregation, input tokens, and ADPO.
4.1 Datasets
- The evaluation covers video-description metrics, zero-shot video QA, closed-ended Music-AVQA and AVQA, and zero-shot open-ended AVSD and VALOR.Closed-ended tasks use accuracy or candidate selection; open-ended AVQA is evaluated with CIDEr, while video-description and zero-shot video QA use GPT-based scores.Source: §4.1 · pp. 9–10, §4.2 · p. 10
4.2 Experimental Setup
- The experiments use frozen ImageBind and LLaMA2-7B as audio-visual encoders and the language model, with feature alignment, instruction tuning, and ADPO training.Training uses one NVIDIA A100 GPU; ADPO optimizes the instruction-tuning training set.Source: §4.2 · p. 10
4.3 Comparison to State-of-the-Art
- CAT outperforms state-of-the-art methods on video-description correctness, detail, and coherence, and on zero-shot MSRVTT-QA and ActivityNet-QA.For time-related video descriptions, CAT is reported as competitive despite not parsing longer visual tokens.Source: §4.3 · p. 11
- CAT-7B scores 92.0 accuracy on fine-tuned AVQA, versus 90.2 for PSTP-Net, 89.0 for HCRN + HAVF, and 87.7 for HGA + HAVF.The passage reports that CAT comprehensively outperforms non-LLM-based models under full supervision.Source: §4.3 · pp. 11–12, Table 4 · p. 12
- CAT surpasses OneLLM and ChatBridge on zero-shot AVSD and VALOR, and is close to fully supervised FA on these open-ended AVQA tasks.The comparison evaluates CAT zero-shot; the passage describes AVSD as complex text reasoning and reports the comparison with FA as close.Source: §4.3 · p. 12, Table 5 · p. 12
4.4 Ablations and Analyses
- Ablations find that question-related clues alone outperform joint audio-visual knowledge, clue fusion adds useful information, visual attention matters more than auditory attention, and 48 Q-Former queries work best.Increasing visual-token count slightly lowers video-description accuracy; ADPO improves descriptive power for both Video-LLaMA and CAT without much learning cost.Source: §4.4 · pp. 12–14, Table 6 · p. 13, Table 8 · p. 13
4.5 Qualitative Analysis
- Qualitative examples show CAT describing complex indoor scenes and background sounds, answering an instrument-count question, and correctly grounding an audio-visual answer where Video-LLaMA errs.The figure presents video-specific contextual inference, audio-visual contextual inference, audio-specific QA, and audio-visual QA examples.Source: §4.5 · p. 14, Figure 6 · p. 14
5 Conclusion
The conclusion presents CAT as a model for dynamic audio-visual understanding, combining question-related clue aggregation, mixed audio-video training, and ambiguity-aware preference optimization. It reports comparable results across complex audio-visual scenarios and identifies further modality extensions as future work.
5 Conclusion
- CAT combines question-related clue aggregation, mixed audio-video datasets, an audio-visual joint instruction dataset, and ambiguity-aware preference optimization for dynamic audio-visual understanding.The optimization strategy is specialized to retrain the model to reduce ambiguous descriptions and improve responses about specific audio-visual objects.Source: §5 · pp. 14–15
- The authors report comparable results across complex audio-visual scenarios and leave further modality extensions for future work.Source: §5 · p. 15