PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Audio and Speech Understanding

Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, James Glass

arXiv:2309.14405v3cs.SDcs.AIeess.AS

TL;DR

Existing models often recognize speech or audio events separately and have limited reasoning, despite human understanding drawing on their relationship. LTU-AS combines Whisper-based perception with LLaMA reasoning and trains on Open-ASQA to jointly understand spoken text, paralinguistics, and non-speech events. It reports an instruction-following rate over 95% on free-form audio and speech questions, with emerging joint reasoning.

  • Problem

    Most existing models recognize either speech or audio events and have limited reasoning, although human understanding often integrates both and their relationships.

    Source: §1 · p. 1

  • Method

    LTU-AS combines Whisper and TLTR for audio perception with LLaMA reasoning, and trains on Open-ASQA, which brings together audio and speech question-answer data.

    Source: §2.1 · p. 2, §3 · p. 2

  • Results

    Over 95% instruction following rate, evaluated by GPT-4, on free-form open-ended audio and speech questions, with emerging joint audio-speech reasoning.

    Source: §1 · p. 1

  • Takeaways & Limitations

    In examples, LTU-AS uses spoken content and acoustic cues to infer speaker context and provide situation-specific suggestions.

    Source: §5.2 · p. 6

    The automatic instruction-following evaluation may overestimate actual performance because questions were GPT-4-generated and GPT-4 may favor answers resembling its outputs.

    Source: §5.2 · p. 6

Abstract

from arXiv · show

Humans are surrounded by audio signals that include both speech and non-speech sounds. The recognition and understanding of speech and non-speech audio events, along with a profound comprehension of the relationship between them, constitute fundamental cognitive capabilities. For the first time, we build a machine learning model, called LTU-AS, that has a conceptually similar universal audio perception and advanced reasoning ability. Specifically, by integrating Whisper as a perception module and LLaMA as a reasoning module, LTU-AS can simultaneously recognize and jointly understand spoken text, speech paralinguistics, and non-speech audio events - almost everything perceivable from audio signals.

1. INTRODUCTION

LTU-AS targets the gap between speech-only or audio-event-only systems with limited reasoning and human understanding that integrates speech with other sounds. It combines perception and reasoning components to jointly understand audio and speech, reporting strong task performance and emerging joint reasoning.

1. INTRODUCTION

  • Human audio understanding integrates speech and non-speech sounds and their relationships, such as combining a warning with a car horn to infer danger.The introduction also illustrates integration with bird sounds and lyrics with melody.Source: §1 · p. 1
  • Existing models generally recognize either speech or audio events, and even strong perceptual models have limited reasoning and understanding capabilities.The authors motivate a model that recognizes and jointly understands speech and audio events.Source: §1 · p. 1
  • Over 95% instruction following rate on free-form open-ended audio and speech questions, evaluated by GPT-4, alongside emerging joint audio-speech reasoning.The passage also reports strong performance across audio and speech tasks.Source: §1 · p. 1
  • To our knowledge, LTU-AS is the first joint audio and speech understanding model, improving on LTU [5] by recognizing both speech and audio events.It uses Whisper [1] and TLTR instead of LTU's AST encoder.Source: §1 · p. 1

2. LTU-AS MODEL ARCHITECTURE

LTU-AS preserves spoken text and non-linguistic audio information as separate inputs to LLaMA. Its design combines Whisper transcription, TLTR audio encoding, and parameter-efficient LLaMA adaptation.

2.1. Design Overview

  • Whisper transcribes speech while TLTR encodes audio events and speech paralinguistics from intermediate encoder layers; their outputs become audio tokens combined with spoken-text and question tokens for LLaMA.The audio-token sequence is capped at 25 tokens, corresponding to 10 seconds of audio.Source: §2.1 · p. 2
  • Whisper is frozen during training, while trainable TLTR attends across time and layers to use information distributed across Whisper representations.The authors freeze Whisper to retain its ASR ability and use TLTR to support universal perception.Source: §2.1 · p. 2

2.2. Audio Encoder

  • Whisper-large supplies a 32-layer encoder-decoder, and AudioSet-pretrained TLTR processes its features; a linear layer maps TLTR's 1280-dimensional output to LLaMA's 4096-dimensional embedding.Whisper and TLTR pool audio by a factor of 40, producing 25 TLTR output tokens per 10 seconds.Source: §2.2 · p. 2

2.3. LLaMA Large Language Model

  • With LLaMA frozen, LoRA adapters tune attention keys and queries for next-token prediction conditioned on audio tokens, spoken text, and the question.About 49 million of LTU-AS's 8.5 billion parameters are trainable.Source: §2.3 · p. 2

3. THE OPEN-ASQA DATASET

Open-ASQA expands OpenAQA-5M into a 9.6-million-example dataset by adding speech and joint audio-speech questions to existing data and standardizing samples as audio-question-answer tuples. It combines rule-based closed-ended pairs with GPT-assisted open-ended generation.

3. THE OPEN-ASQA DATASET

  • Open-ASQA adds 2.7 million speech AQAs and 1.2 million joint audio-speech AQAs to OpenAQA-5M, relabeling 13 public datasets and excluding test or evaluation samples.The resulting dataset contains 9.6 million AQAs.Source: §3 · p. 2
  • All samples use audio-question-answer tuples, with audio and question as inputs and the answer as ground truth, unifying labels in a semantic space for varied tasks.This common format also makes it possible to train LTU-AS across different tasks.Source: §3 · p. 3

3.1. Closed-Ended AQA Generation

  • Closed-ended AQAs combine paraphrased questions with rule-generated, fixed-format answers, drawing on audio-event data, speech labels, and a smaller joint music-genre task.The audio questions come from OpenAQA; speech AQAs cover four datasets, while the joint task uses FMA and requires lyrics and acoustic information.Source: §3.1 · p. 3

3.2. Open-Ended AQA Generation

  • Open-ended AQAs use GPT-3.5-Turbo-assisted Audio Instruction Generation, with text-form audio metadata as a surrogate input while LTU-AS trains and answers using raw audio.The approach addresses the impracticality of large-scale human generation and the limited output diversity of rule-based methods.Source: §3.2 · p. 3
  • Open-ended speech AQAs give GPT dataset labels, extracted speech-style features, and Whisper transcripts; age, pitch, speed, and volume are also quantized into five categories.Source: §3.2 · p. 4
  • For joint audio-speech AQAs, an AudioSet subset yielded 82,000 samples after transcription and speech-content filtering, while FMA supplied about 1.1 million lyric-and-melody questions.FMA generation uses genre lists, available titles, and Whisper-transcribed lyrics as GPT input.Source: §3.2 · p. 4

4. TRAINING LTU-AS

LTU-AS is trained with a three-stage curriculum that gradually unfreezes components and initially emphasizes classification AQAs to encourage attention to audio input.

  • Training follows three stages: only the randomly initialized projection layer is trainable first, then TLTR and LoRA adapters are unfrozen in stages two and three.Source: §4 · p. 4
  • The first two stages use classification AQAs, where wrong predictions receive a high penalty, to encourage the model to attend to audio rather than hallucinate from language ability.Source: §4 · p. 4
  • Unfreezing TLTR and LoRA after the projection-only first stage is intended to stabilize training.Source: §4 · p. 4

5. EXPERIMENTS

LTU-AS performs well across closed-ended audio and speech tasks, with particular strength on tasks requiring both modalities. On open-ended questions, it follows instructions at over 94% for both audio and speech, while examples show it combining spoken and acoustic information.

5.1. Closed-Ended Tasks Evaluation

  • LTU-AS performs well on both audio and speech benchmarks, especially tasks requiring joint understanding, including zero-shot music genre classification and speaker age prediction.It nearly doubles CLAP accuracy on GTZAN and achieves lower speaker-age MAE than the task-specific SOTA model; unlike CLIP-like models, it needs no predefined label set.Source: §5.1 · p. 5
  • Training on both non-speech audio and speech supports unified perception; audio- or speech-specialized LTUs nearly fail on tasks outside their training domain.Each specialized model performs slightly better on tasks within its own training domain.Source: §5.1 · p. 5
  • Removing either audio or spoken-text input drops performance on most tasks, while including spoken text causes only a slight drop on audio and gender classification.LTU-AS has 4.9% WER versus 3.5% for its internal Whisper, attributed to occasional instruction noncompliance and spelling changes.Source: §5.1 · p. 5

5.2. Open-Ended Audio Question Answering Evaluation

  • The open-ended evaluation used GPT-4 to generate 100 questions from AudioSet and VoxCeleb evaluation sets, then judge whether model answers directly addressed them.Questions and corresponding audio were given to LTU-AS and audio-only- or speech-only-trained LTU models.Source: §5.2 · p. 5
  • Over 94% instruction-following rate on both audio and speech questions; single-domain LTU models do not follow instructions well outside their training domain.The evaluation used GPT-4-generated questions and GPT-4 ratings, which may overestimate the rate.Source: §5.2 · pp. 5–6, Table 5 · p. 5
  • Evaluation examples show LTU-AS combining speech and acoustic cues to identify a basketball coach and using audio-and-speech context to suggest actions or interpret situations.Table 6 marks acoustic and speech information in different colors.Source: §5.2 · p. 6, Table 6 · p. 6

6. CONCLUSIONS

LTU-AS jointly recognizes and understands spoken text, speech paralinguistics, and non-speech audio events. Its tested benchmark performance and open-ended examples indicate that it combines audio and speech information and exhibits emerging reasoning abilities.

6. CONCLUSIONS

  • LTU-AS jointly recognizes and understands spoken text, speech paralinguistics, and non-speech audio events.The system uses Whisper for perception, LLaMA for reasoning, Open-ASQA for diverse audio and speech training, and a multi-stage curriculum to mitigate hallucination.Source: §6 · p. 6
  • LTU-AS performs well on tested closed-ended audio and speech benchmarks, particularly tasks requiring joint understanding, and its open-ended answers combine audio and speech information.The paper reports emerging reasoning abilities in open-ended responses.Source: §6 · p. 6
Loading 2309.14405v3…