PaperlayerView exact revision on arXiv

Source-linked AI summary

video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models

Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zejun Ma, Chao Zhang

arXiv:2506.15220v3cs.CVcs.CLcs.SD

TL;DR

Video-SALMONN 2 addresses the difficulty of generating detailed, faithful captions for videos containing coupled visual and audio events. It introduces multi-round preference optimisation with refreshed LoRA-based references and uses improved captions to train models for broader video understanding. The resulting systems achieve strong captioning and video-QA performance across model scales, with 3B and 7B models leading comparable-size systems and the 72B model surpassing open-source competitors.

  • Problem

    Video captioning must capture detailed spatial and temporal audio-visual events, but existing metrics and training strategies provide limited support for completeness and factuality.

  • Method

    The method combines audio-visual modeling with MrDPO, which refreshes the reference through re-initialised LoRA adapters and uses caption-quality preferences plus ground-truth guidance.

  • Results

    The models achieve SOTA captioning and video-QA performance, with 3B and 7B models leading comparable-size systems and the 72B model surpassing open-source competitors.

  • Takeaways & Limitations

    Higher-quality captions can provide improved supervision for SFT, transferring captioning gains to general video understanding.

Abstract

from arXiv · show

We present video-SALMONN 2, a family of audio-visual large language models that set new state-of-the-art (SOTA) results in video description and question answering (QA). Our core contribution is multi-round direct preference optimisation (MrDPO), paired with a caption-quality objective that jointly rewards completeness and factual accuracy. Unlike standard DPO with a fixed reference policy, MrDPO periodically refreshes the reference by bootstrapping from a newly re-initialised lightweight adapter trained on the latest preferences, avoiding reference staleness and enabling continual improvement. This strategy produces captions that are consistently more detailed and accurate than those from proprietary systems such as GPT-4o and Gemini-1.5 Pro. We further distil these gains by using our model to generate a high-quality video-caption corpus for supervised fine-tuning of new models, transferring benefits beyond captioning to strong performance on complex video-QA tasks. Across widely used audio-visual and visual-only understanding benchmarks (including Video-MME, WorldSense, AVUT, Video-Holmes, DailyOmni, MLVU, and LVBench), our 3B and 7B models achieve SOTA results at comparable scales, while the 72B model surpasses all other open-source systems. Our source code, models, and data are released at \href{https://github.com/bytedance/video-SALMONN-2}{https://github.com/bytedance/video-SALMONN-2}.

1 INTRODUCTION

video-SALMONN 2 targets the difficulty of producing detailed, faithful audio-visual captions and transfers captioning gains to video understanding. It combines audio-visual modeling, completeness and factuality evaluation, MrDPO, and caption-based supervised fine-tuning.

  • Video captioning remains difficult because videos combine spatial content with evolving audio-visual events, while existing metrics and training strategies inadequately measure caption quality.
  • The proposed models process audio and visual inputs jointly to capture fine-grained temporal interactions and produce detailed, holistic captions.
  • MrDPO uses completeness and factuality signals to optimise captions while periodically updating the DPO reference policy with a lightweight adapter.
  • Caption improvements transfer to video QA because MrDPO-generated captions provide higher-quality supervision for supervised fine-tuning.
  • The resulting models surpass proprietary systems such as GPT-4o and Gemini-1.5 Pro in captioning and achieve new open-source SOTA results across multiple video-QA benchmarks.

2 RELATED WORK

Related work develops multimodal LLMs by connecting pretrained encoders to language models, extending from visual and silent-video understanding to audio-visual reasoning. Reinforcement-learning methods increasingly optimise language models directly from preference or model-generated feedback.

  • Adapter-based multimodal LLMs connect pretrained visual encoders to LLMs, supporting image and video understanding through instruction tuning, Q-Formers, pretraining, and encoder scaling.
  • Audio-focused systems use dual encoders or large audio datasets for zero-shot reasoning, audio event QA, broad audio tasks, and spatial-audio perception.
  • Non-silent video models combine visual frames with speech or audio, including transcription-enhanced captioning, multi-resolution audio-visual modeling, process DPO, and native multimodal architectures.
  • RLHF, DPO, iterative RPO, KTO, AI feedback, and GRPO progressively reduce reliance on separate reward or value models while targeting performance, efficiency, or stability.

3 METHODS

The method combines separate audio and visual branches with staged training and a caption-focused MrDPO procedure. MrDPO evaluates caption completeness and factuality, refreshes the reference through LoRA proxies, and stabilises optimisation with ground-truth guidance.

  • 3.1 MODEL ARCHITECTURE: Separate audio and visual encoders produce tokens that are synchronously interleaved with text tokens before entering the LLM backbone.
  • 3.2 MULTI-STAGE TRAINING FOR AUDIO–VISUAL CAPTIONING: Training proceeds through audio alignment, audio-visual SFT, and MrDPO, while the backbone LLM and encoders remain frozen to preserve visual capabilities.
  • 3.3 CAPTION QUALITY OPTIMISATION: Caption preferences are built by decomposing ground-truth descriptions into atomic events and selecting generated captions with lower missing-information and hallucination rates.
  • 3.3.2 MULTI-ROUND DPO WITH LORA PROXY: MrDPO merges the prior LoRA into the backbone, adds a newly initialised LoRA proxy, and adopts the resulting policy as the next reference to avoid reference staleness.
  • 3.3.2 MULTI-ROUND DPO WITH LORA PROXY: The guided DPO loss adds cross-entropy on ground-truth captions to standard preference optimisation, stabilising training while captions are judged by atomic-event metrics.
  • 3.4 TRANSFERRING CAPTION OPTIMISATION GAINS TO VIDEO QA: MrDPO substantially improves captioning, but subsequent caption-focused reinforcement learning does not further improve general video understanding beyond audio-visual SFT.

4 EXPERIMENTAL SETUP

The experiments cover multiple model scales and training variants, using audio-visual data, modality alignment, supervised fine-tuning, MrDPO, and human-annotated caption evaluation. The setup also includes large-scale re-annotation for downstream supervision.

  • The model family includes 3B, 7B, and 72B video-SALMONN 2+ variants, alongside 7B video-SALMONN 2 and video-SALMONN 2F-16 models with different frame rates and capacities.
  • All models receive modality alignment, while the principal 7B models additionally use audio-visual SFT and MrDPO; smaller and larger variants are trained from data annotated by 7B models.
  • Audio alignment uses LibriSpeech-960h for speech recognition and AudioCaps for audio captioning, followed by video SFT on FineVideo, CinePile, and audio-rich LLaVA-Video data.
  • During MrDPO, each video receives a generated caption pair evaluated for missing and hallucinated information, with suitable pairs selected for preference training.
  • A further 100k training videos are re-annotated with 7B models to provide additional supervision.
  • The caption benchmark contains 483 human-labelled videos with complete audio-visual captions and evaluates event missing and hallucination rates.

5 EXPERIMENTAL RESULTS

video-SALMONN 2 demonstrates strong captioning and video-QA performance, while ablations support MrDPO’s gDPO loss and proxy-based updates. Generated SFT data also improves downstream evaluation, and the model leads comparable-scale systems across audio-visual benchmarks.

  • 5.1 CAPTIONING EVALUATION RESULTS AND ANALYSIS: Video-SALMONN 2 provides more detailed and accurate captions than existing open-source systems, while Qwen2.5-VL is the strongest existing open-source captioner.
  • 5.2 ABLATION ON MRDPO: From the second round onward, gDPO consistently outperforms classical DPO, and Proxy performs better than Direct, especially early in training.The comparison uses total caption-benchmark error rates across six rounds.
  • 5.3 ABLATION ON SFT DATA: Models trained with video-SALMONN 2-generated captions consistently outperform models trained with the corresponding original SFT data on Video-MME.
  • 5.4 OVERALL RESULTS ON VIDEO QA BENCHMARKS: video-SALMONN 2+ (3B) outperforms all existing 7B models, while the 7B model is competitive with existing 72B models across audio-visual benchmarks.
  • 5.4 OVERALL RESULTS ON VIDEO QA BENCHMARKS: The 72B model consistently outperforms GPT-4o and Qwen3-Omni-Flash and surpasses Gemini-1.5 Pro on most audio-visual QA benchmarks.

6 CONCLUSIONS

video-SALMONN-2 combines MrDPO-based caption optimisation with generated-caption supervision to improve audio-visual captioning and video understanding. Its 3B and 7B models lead their size classes, while the 72B model surpasses open-source competitors.

  • MrDPO addresses DPO reference staleness and produces captions richer and more faithful than GPT-4o and Gemini-1.5 Pro.
  • Generated captions provide higher-quality supervision for SFT, transferring captioning gains to general video understanding.
  • Across standard video-QA benchmarks, the 3B and 7B models lead their size classes, while the 72B model surpasses all open-source competitors.

B TRAINING COST FOR VIDEO-SALMONN 2 (7B)

The training-cost section reports resource consumption for video-SALMONN 2 (7B) across training stages and notes that the 7B+ model uses similar settings.

  • MrDPO training uses six gDPO rounds in the reported experiments.
  • Resource consumption is reported for each training stage of video-SALMONN 2 (7B), with video-SALMONN 2+ (7B) trained using similar settings.

C TRAINING WITH MODELS WITH SMALLER SCALE AS EVALUATOR

Qwen3-4B can evaluate video-caption quality during training in place of GPT-3.5, with similar results under unchanged settings.

  • C TRAINING WITH MODELS WITH SMALLER SCALE AS EVALUATOR: Qwen3-4B and GPT-3.5 produce similar captioning results when used as evaluators in the first gDPO round.The prompts, generation settings, and training settings remain the same for both evaluators.

D SAMPLES SELECTING CRITERIA IN MRDPO

MrDPO selects preference pairs when one caption is sufficiently better than the other, balancing error reduction, repetition, and cross-entropy regularisation.

  • D SAMPLES SELECTING CRITERIA IN MRDPO: MrDPO selects pairs when one caption is significantly better, using total error-rate difference and repetition-rate difference to avoid weak or redundant comparisons.Thresholds vary across rounds, and negative thresholds permit the chosen sample to be somewhat worse on a metric.
  • E HYPERPARAMETER SEARCH IN GDPO: λ=0.1 is selected because λ>1 makes cross-entropy overwhelm DPO optimisation, whereas λ<0.01 does not sufficiently stabilise MrDPO training.The choice is based on first-round results and the observed training behaviour shown in Figure 3a.

F ABOUT THE TEST DATASET

The caption benchmark covers diverse 30–60-second videos and evaluates captions through missing, incorrect, and hallucinated atomic events across visual, audio, and audio–visual inputs.

  • F ABOUT THE TEST DATASET: The benchmark spans 14 fields and videos lasting 30–60 seconds, with an average duration of 51 seconds.It contains 6.1 audio-related atomic events per video, including 4.6 speech-related and 1.5 non-speech events.
  • F ABOUT THE TEST DATASET: The audio–visual model achieves the best result, while visual-only input reduces hallucinations but still misses events without audio and audio-only input misses more visual events.The comparison includes the base visual model, visual-only input, audio-only input, and audio–visual input.
  • G EVALUATION PROCESS ON OUR CAPTION BENCHMARK: Evaluation identifies missing events, incorrectly described ground-truth events, and hallucinated events absent from the atomic-event list.Incorrect and hallucinated events jointly contribute to the hallucination rate.
  • G EVALUATION PROCESS ON OUR CAPTION BENCHMARK: Missing rate divides missing events by all atomic events, hallucination rate divides incorrect plus hallucinated events by all atomic events, and total error rate sums both rates.The prompts used for testing are available in the released test-data scripts.

H ELO RANKING RESULTS

Human Elo ranking compares captions from video-SALMONN 2, its visual base model, GPT-4o, and Gemini-1.5 Pro using paired video-caption judgments focused on completeness and correctness.

  • H ELO RANKING RESULTS: The Elo ranking evaluates Gemini-1.5 Pro, GPT-4o, the visual base model, and final video-SALMONN 2 using human ratings.The ranking-system parameters are provided in Table 10.
  • H ELO RANKING RESULTS: 180 caption pairs are judged as simulated matches, with annotators selecting the better caption for completeness and correctness.The final Elo ratings and benchmark total error rates are reported for each model.
  • H ELO RANKING RESULTS: The benchmark uses atomic-event lists to represent the events that captions should cover for comparison.Figure 5 provides an example of an atomic-event list for a video.
Loading 2506.15220v3…