PaperlayerView exact revision on arXiv

Source-linked AI summary

video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models

Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, Chao Zhang

arXiv:2406.15704v1cs.CV

TL;DR

Speech remains under-explored in audio-visual LLMs despite carrying information important to comprehensive video understanding. video-SALMONN combines speech with visual frames, audio events, and music using a multi-resolution causal aligner and dedicated training strategies. On SAVE, it reports accuracy improvements on Video QA and audio-visual QA, alongside audio-visual co-reasoning abilities.

  • Problem

    Speech is under-explored in av-LLMs, although it carries linguistic, paralinguistic, and speaker information needed for comprehensive video understanding.

    Source: §1 · p. 1

  • Method

    video-SALMONN integrates visual frames, speech, audio events, and music through an MRC Q-Former, diversity loss, and unpaired audio-visual mixed training.

    Source: §1 · p. 2

  • Results

    25% accuracy improvements on Video QA and over 30% on audio-visual QA compared to Video-LLaMA are reported on SAVE.

    Source: §6 · p. 8

  • Takeaways & Limitations

    video-SALMONN demonstrates audio-visual, particularly speech-visual, co-reasoning abilities described as unprecedented for other av-LLMs.

    Source: §6 · p. 9

    The model may inherit demographic performance disparities from its pretrained audio and visual encoders and LLMs.

    Source: Impact Statement · p. 9

Abstract

from arXiv · show

Speech understanding as an element of the more generic video understanding using audio-visual large language models (av-LLMs) is a crucial yet understudied aspect. This paper proposes video-SALMONN, a single end-to-end av-LLM for video processing, which can understand not only visual frame sequences, audio events and music, but speech as well. To obtain fine-grained temporal information required by speech understanding, while keeping efficient for other video elements, this paper proposes a novel multi-resolution causal Q-Former (MRC Q-Former) structure to connect pre-trained audio-visual encoders and the backbone large language model. Moreover, dedicated training approaches including the diversity loss and the unpaired audio-visual mixed training scheme are proposed to avoid frames or modality dominance. On the introduced speech-audio-visual evaluation benchmark, video-SALMONN achieves more than 25\% absolute accuracy improvements on the video-QA task and over 30\% absolute accuracy improvements on audio-visual QA tasks with human speech. In addition, video-SALMONN demonstrates remarkable video comprehension and reasoning abilities on tasks that are unprecedented by other av-LLMs. Our training code and model checkpoints are available at \texttt{\url{https://github.com/bytedance/SALMONN/}}.

1. Introduction

Speech is an under-explored but important part of video understanding because it carries linguistic, paralinguistic, and speaker information. The paper introduces video-SALMONN, an end-to-end audio-visual model, and reports strong results on speech-audio-visual evaluation tasks.

1. Introduction

  • Speech carries linguistic, paralinguistic, and speaker information, but remains considerably under-explored in av-LLMs.Understanding it requires fine-grained temporal modelling that also interacts with other modalities at coarse and fine time scales.Source: §1 · p. 1
  • video-SALMONN combines visual frames, speech, audio events, and music in a single speech-enhanced av-LLM.Its MRC Q-Former aligns synchronized audio-visual features with text at three temporal scales, while diversity loss and unpaired audio-visual mixing address frame or modality dominance.Source: §1 · pp. 1–2
  • 25% accuracy improvement on Video QA and over 30% absolute accuracy improvement on audio-visual QA show video-SALMONN’s strong SAVE benchmark performance.The paper also reports that it is the only av-LLM able to perform speech-dependent tasks such as AVSR and speech-content-based QA.Source: §1 · p. 2
  • The introduced SAVE benchmark contains six single-modal tasks and four audio-visual tasks for evaluating general video understanding.The paper positions video-SALMONN as the first av-LLM tailored for general video understanding.Source: §1 · p. 2

2. Related Work

Prior av-LLMs and related approaches handle video, audio, and speech with varying coverage, but some omit speech or fine-grained cross-modal interaction. The paper positions video-SALMONN’s multi-resolution aligner and integrated speech processing against these approaches.

2. Related Work

  • Video-LLaMA (Zhang et al., 2023b) and Macaw-LLM (Lyu et al., 2023) handle non-speech audio, while X-LLM (Chen et al., 2023a) supports Chinese speech but not audio events or music.The paper contrasts these models with video-SALMONN’s speech understanding and fine-grained modality interactions.Source: §2 · p. 2
  • External ASR can supply subtitle text to an av-LLM, but does not capture speech’s paralinguistic and speaker information unless additional systems extract it.Video-SALMONN instead unifies hearing tasks such as ASR, speaker diarisation, and emotion recognition with visual perception in one model.Source: §2 · p. 2
  • The MRC Q-Former targets sequential, multi-resolution video features; Figure 1 depicts synchronized audio and visual streams processed at different time scales.The figure caption states that the individual encoders are not updated during training.Source: §2 · p. 2, Figure 1 · p. 3
  • The paper describes its work as a revision of an unpublished study that first explored video understanding with general audio, including events, speech, and music.Source: §2 · p. 3

3. video-SALMONN

video-SALMONN encodes synchronized visual, speech, and non-speech audio features, then uses a multi-resolution causal Q-Former and dedicated training losses and data mixing to prepare them for an LLM.

3. video-SALMONN

  • Visual frames, speech, and non-speech audio are encoded separately; video frames are sampled at 2 Hz and audio features at 50 Hz.Whisper encodes speech, BEATs encodes non-speech audio, and InstructBLIP encodes images or video frames.Source: §3 · p. 3

3.1. Temporal Fine-grained Synchronisation

  • Audio and visual sequences are synchronized at each video frame, every 0.5 seconds, using zero padding to equalize sequence lengths.Higher visual frame rates are supported at increased computation and storage costs; missing modalities can also be zero-padded.Source: §3.1 · p. 3

3.2. MRC Q-Former

  • The MRC Q-Former processes synchronized inputs in sliding windows at multiple temporal resolutions, balancing retained information against computation and storage costs.Resolution-specific outputs are synchronized to a fixed total number of query vectors, with fewer queries for finer windows; parameters are shared across resolutions except for query vectors.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.2.1. CAUSAL STRUCTURE

  • An added causal self-attention module lets each frame encoding include information from all previous frames, supporting questions about what happens next.The module captures causal temporal correlations among independently extracted frames.Source: §3.2.1 · p. 4, Figure 3 · p. 4

3.3. System Training

  • A diversity loss encourages the Q-Former to extract varied sequence information, using cosine similarity and applying only at low-resolution levels with enough frames.Training combines cross-entropy and diversity loss, with λ controlling the diversity-loss weight.Source: §3.3 · pp. 4–5
  • Unpaired audio-visual mixed training supplements scarce paired data, combining original audio and video tasks in prompts to reduce modality dominance.The authors report that this strategy improves modality balance and is a crucial factor in audio-visual understanding and co-reasoning.Source: §3.3 · p. 5

4. Experimental Setup

The SAVE benchmark evaluates single-modal and audio-visual tasks, including speech-focused QA and tasks requiring audio-visual interaction. The setup uses pretrained audio, speech, and visual encoders with a shared language model, trained on a mixture of single-modal and paired data.

4.1. Speech-Audio-Visual Evaluation Benchmark

  • SAVE covers six single-modal tasks and four audio-visual tasks, including speech-focused Ego4D-QA and Presentation-QA, audio-visual matching, and sound-source detection.Ego4D-QA and Presentation-QA test questions are generated from video descriptions and ASR transcriptions; AVM tests whether spoken descriptions or audio clips match visual content.Source: §4.1 · p. 5

4.2. Model Configurations

  • The model combines Vicuna-v1.5 with Whisper speech, BEATs audio, and InstructBLIP visual encoders; its visual encoder produces 32 vectors per frame and its audio encoder 50 per second.The MRC Q-Former uses 0.5-second and 5-second resolutions by default, with 3 and 30 output queries per window, respectively.Source: §4.2 · p. 5

4.3. Training Data and Specifications

  • Training uses multi-task instruction fine-tuning on single-modal and audio-visual data, with fewer than 300,000 video samples among 1 million samples overall.The mix includes public audio, image, and video datasets, plus paired audio-visual data; NExT-QA test instructions and videos are unseen during training.Source: §4.3 · pp. 5–6

5. Results and Discussions

video-SALMONN performs across SAVE’s single-modal and audio-visual tasks, with reported gains in video QA, AVQA, and AVSR. Ablations and analyses associate its balanced performance with multi-resolution modeling, temporal synchronization, diversity loss, and mixed training.

5.1. Main Results

  • Over 30% accuracy improvements on AVQA tasks over Video-LLaMA and a 7.2% relative WER reduction on AVSR over Whisper-large-v2 accompany strong performance across SAVE tasks.The paper also reports the lowest WER and highest SPIDEr on audio-based tasks, best results on IC, OCR, and Video QA, and on-par VQA with same-data fine-tuned InstructBLIP.Source: §5.1 · p. 6, Table 3 · p. 7

5.2. Ablation Studies

  • High and low resolutions complement one another: high resolution favors ASR and AVSR, low resolution favors OCR and Video QA, and their joint use gives the most balanced task performance.Removing MRC Q-Former degrades all tasks, especially ASR, AVSR, and Video QA; removing temporal synchronization further degrades AVSR and AVSSD.Source: §5.2 · p. 7

5.3. Analysis on Multi-resolution

  • Smaller windows improve speech WER but degrade video QA, while the two resolution levels divide speech-content and higher-level visual information between them.The LLM combines the complementary outputs; Figure 4 reports window-size experiments on systems trained with 10% of randomly sampled data.Source: §5.3 · p. 8, Figure 4 · p. 8, Table 5 · p. 8

5.4. Analysis of the Diversity Loss

  • A high diversity-loss factor can make diverse output queries confuse the LLM, causing severe hallucinations such as increased WER insertion rates and degraded performance.The analysis uses 10% of the training data.Source: §5.4 · p. 8, Figure 5 · p. 8

5.5. Emergent Speech-Audio-Visual Co-reasoning

  • Examples show speech-audio-visual co-reasoning across questions about speech and images, stories from unpaired audio and video, and interpretations combining dialogue, visuals, and background sound.The paper also describes using speech to identify a fish species and attributing utterances to characters based on speech-video event co-occurrence.Source: §5.5 · p. 8

6. Conclusions

video-SALMONN is presented as an end-to-end model for visual frames, speech, audio events, and music, with MRC Q-Former, fine-grained synchronisation, and mixed training among its designs. On SAVE, it outperforms single-modal baselines and improves accuracy over Video-LLaMA on Video QA and audio-visual QA; the paper also reports emergent audio-visual co-reasoning abilities.

6. Conclusions

  • 25% accuracy improvements on Video QA and over 30% on audio-visual QA compared to Video-LLaMA, alongside superior performance to single-modal baselines on SAVE.The paper also describes fine-grained synchronisation and mixed training as designs for strengthening speech and comprehensive video understanding.Source: §6 · p. 8
  • video-SALMONN demonstrates unprecedented audio-visual, particularly speech-visual, co-reasoning abilities in illustrated examples.Source: §6 · p. 9

Impact Statement

The impact statement frames speech-enabled video understanding as potentially broadening human interaction with technology, while noting inherited model bias and possible misuse. It describes potential applications in accessibility and education and says legal experts were consulted on usage guidelines.

Impact Statement

  • Speech-enabled video understanding may support more natural technology interactions and accessibility for people with motor impairments that make typing difficult.Source: Impact Statement · p. 9
  • The statement identifies video-QA use in academic presentations and education as a potential way to facilitate learning.Source: Impact Statement · p. 9
  • The system may inherit demographic performance disparities from its audio and visual encoders and biases from the LLMs used.The authors state their approach adds no potential biases beyond those inherited from pre-trained checkpoints.Source: Impact Statement · p. 9
  • The authors identify possible surveillance and eavesdropping abuses and report consulting legal experts to establish usage guidelines.Source: Impact Statement · p. 9

A. Training Set and Benchmark Details

The appendix points to Tables 6 and 7 for dataset and benchmark details, including individual descriptions and prompt designs.

A. Training Set and Benchmark Details

B. Examples of the AVQA Dataset

The appendix identifies example items from two English audio-visual QA datasets in Figures 6 and 7.

B. Examples of the AVQA Dataset

C. Evaluation Details

The evaluation uses task-specific metrics, including ChatGPT-assisted scoring for full-sentence answers. It also visualizes how diversity-loss factors relate to cosine similarities among causal Q-Former output queries.

C. Evaluation Details

  • Video QA and VQA use top-1 accuracy with exact-answer matching, while ASR, AAC, IC, OCR, AVM, and AVSSD use task-specific scoring procedures.The procedures include WER for ASR, SPIDEr for AAC, CIDEr for IC, and ChatGPT-assisted scoring for AVSSD.Source: C. Evaluation Details · p. 14

D. GPT Scoring Prompt Design

  • ChatGPT (GPT-3.5-turbo) assists evaluation of VGGSS open-ended answers because full-sentence responses are difficult to score by string matching.Table 8 provides prompt designs for ChatGPT-based evaluation.Source: D. GPT Scoring Prompt Design · p. 14, Table 8 · p. 14

E. Visualisation of Diversity Loss Effect

F. Additional Results on Lip Reading

Additional experiments report lip-reading, MUSIC-AVQA, backbone, and static-image spotlight results. Case studies illustrate video-SALMONN’s multimodal answers across diverse clips.

F. Additional Results on Lip Reading

G. Additional Results on MUSIC-AVQA

H. Comparison between Vicuna and Llama-2 as Backbone LLMs

I. Spotlight for Static Image

  • Static-image spotlight improves OCR but slightly degrades video-task performance, which the authors attribute possibly to its scan-like sequence confusing the model.The approach splits an image into sequential sub-images; the authors also note OCR performance is highly dependent on image resolution.Source: §I · pp. 15–16

J. Case Studies

  • Further cases show audio-video storytelling across different sources, speech helping identify a shark, and multimodal resolution of character names and who flew to Florence.The examples include the sloth character Flash and attribution of “Nick” to the sloth.Source: Figure 11 · p. 17, Figure 13 · p. 18, Figure 14 · p. 19
  • Case studies show answers combining speech, images, and audio, including Titanic dialogue and music, speaker attribution, and a meme’s sound, words, and facial expression.Other examples illustrate audio-visual matching with explanation.Source: Figure 10 · p. 17, Figure 12 · p. 18, Figure 15 · p. 19

Related papers

Loading 2406.15704v1…