PaperlayerView exact revision on arXiv

Source-linked AI summary

Pengi: An Audio Language Model for Audio Tasks

Soham Deshmukh, Benjamin Elizalde, Rita Singh, Huaming Wang

arXiv:2305.11834v2eess.AScs.SD

TL;DR

Existing zero-shot audio models perform well on close-ended tasks but lack the language generation needed for open-ended tasks. Pengi frames audio tasks as text generation, conditioning a frozen language model on audio and text prefixes, and achieves state-of-the-art performance in several of 21 downstream tasks. Its reported trade-off is weaker text-based retrieval than contrastive models, while language-model limitations such as hallucination remain applicable.

  • Problem

    Zero-shot audio models can perform close-ended classification and retrieval but lack the capacity to generate language for open-ended captioning and question answering.

  • Method

    Pengi frames all audio tasks as audio-and-text input to text output, using encoded audio and text prefixes to prompt a pretrained frozen language model without task-specific extensions.

  • Results

    Pengi was evaluated on 21 downstream tasks and achieved state-of-the-art performance in several, including relative gains of 6.6% on AudioCaps and 26% on Clotho captioning.

  • Takeaways & Limitations

    Connecting language models with audio supports a unified approach to general-purpose audio understanding across close-ended and open-ended tasks.

  • Takeaways & Limitations

    Pengi’s text-based retrieval performance lags behind contrastive models, and inherited language-model limitations include hallucinated or audio-ungrounded responses.

Abstract

from arXiv · show

In the domain of audio processing, Transfer Learning has facilitated the rise of Self-Supervised Learning and Zero-Shot Learning techniques. These approaches have led to the development of versatile models capable of tackling a wide array of tasks, while delivering state-of-the-art performance. However, current models inherently lack the capacity to produce the requisite language for open-ended tasks, such as Audio Captioning or Audio Question & Answering. We introduce Pengi, a novel Audio Language Model that leverages Transfer Learning by framing all audio tasks as text-generation tasks. It takes as input, an audio recording, and text, and generates free-form text as output. The input audio is represented as a sequence of continuous embeddings by an audio encoder. A text encoder does the same for the corresponding text input. Both sequences are combined as a prefix to prompt a pre-trained frozen language model. The unified architecture of Pengi enables open-ended tasks and close-ended tasks without any additional fine-tuning or task-specific extensions. When evaluated on 22 downstream tasks, our approach yields state-of-the-art performance in several of them. Our results show that connecting language models with audio models is a major step towards general-purpose audio understanding

1 Introduction

Existing audio transfer-learning models perform well on close-ended tasks but generally cannot generate language for open-ended tasks. Pengi addresses this gap with a unified Audio Language Model intended to support both task types without task-specific extensions.

  • Self-supervised and unsupervised audio models learn general-purpose representations but require additional fine-tuning before downstream application.
  • Zero-shot audio models achieve strong performance on classification and retrieval without target-task fine-tuning, but cannot produce language for captioning or audio question answering.
  • Current open-ended audio models do not support or have not been evaluated on closed-ended tasks, leaving a unified transfer-learning approach unexplored.
  • Pengi takes an audio recording and text prompt as input and generates free-form text as output.
  • The proposed Audio Language Model supports multiple close-ended and open-ended audio tasks without additional fine-tuning or task-specific architectural extensions.
  • Pengi frames audio tasks as audio-and-text input to text-output problems using a single training procedure and captioning objective.

2 Related Work

Related work established unified transfer-learning paradigms in language and vision, while audio models remained divided between close-ended and open-ended tasks. Pengi adapts text-to-text and language-model alignment ideas to audio.

  • Audio transfer-learning methods, including self-supervised and zero-shot approaches, support diverse tasks and strong performance but typically handle either close-ended or open-ended tasks.
  • Language Models: T5 unified text tasks by converting them into text-to-text problems, while FLAN used varied instructions to improve responses to similar instructions.
  • Language Models: Pengi adapts this unified text-generation framing to audio by representing tasks as audio-text input to text output.
  • Visual Language Models: Visual language models extend frozen or pretrained language models by aligning image embeddings or converting visual information into tokens.

3 Approach

Pengi converts audio and text prompts into a shared prefix for a frozen autoregressive language model. Training optimizes the audio encoder and mapping networks so the model generates text conditioned on both inputs.

  • Pengi’s unified architecture accepts audio and a text prompt, then generates free-form text for both open-ended and close-ended tasks.
  • Unified Architecture: The audio encoder and mapping network convert audio into a sequence of continuous embeddings, while the text encoder and mapping network do the same for the prompt.
  • Unified Architecture: The audio and text embedding sequences are concatenated into a fixed-length prefix for a pretrained frozen autoregressive language model.
  • Training and Inference: The text encoder and causal language model remain frozen, while the audio encoder and both mapping networks are trainable.
  • Training and Inference: Training uses audio-text-to-text examples and teaches the model to predict output text autoregressively from the prefix.
  • Training and Inference: At inference, the frozen language model generates tokens sequentially from the test-audio and prompt prefix, using beam search with beam size 5 in the experiments.

4 Experiments

Pengi is trained with instruction-inspired audio-task templates and evaluated across open-ended and close-ended downstream tasks. Its unified design supports generation while providing procedures for evaluating predefined outputs.

  • Training Datasets and Templates: Eight audio-task templates convert datasets into audio-text-to-text examples containing audio input, a text prompt, and text output.
  • Training Datasets and Templates: The templates include task-specific prompts and an auxiliary metadata-generation task for adding non-task-specific audio-text pairs.
  • Training Datasets and Templates: 3.4 million audio-text pairs from multiple datasets were mapped to the eight templates for training.
  • Downstream Tasks: Pengi’s unified architecture supports open-ended tasks such as captioning and question answering without additional fine-tuning or task-specific components.
  • Downstream Tasks: Close-ended tasks include classification, regression, and retrieval, with predefined outputs evaluated through log-likelihood or text matching.
  • Downstream Tasks: The benchmark covers 21 downstream tasks, with the first two domains open-ended and the remaining domains close-ended.

5 Results

Pengi is evaluated across diverse open-ended and close-ended audio tasks, achieving strong results against supervised, self-supervised, and zero-shot baselines. It sets new state-of-the-art performance on open-ended tasks, outperforms supervised captioning models, and remains competitive on classification and retrieval.

  • Benchmarking Pengi: Pengi was evaluated on 21 downstream tasks spanning multiple audio domains, with CLAP used as the principal broad-coverage baseline.A fair comparison with another model supporting both open-ended and close-ended tasks was unavailable.
  • Benchmarking Pengi: Pengi sets new state-of-the-art performance on the open-ended tasks of Audio Captioning and Audio Question Answering.CLAP supports only close-ended tasks, so open-ended comparisons use supervised models.
  • Benchmarking Pengi: Pengi performs better than CLAP on most audio classification tasks and can also outperform results reported in the literature.The comparison involves different learning methods and amounts of training data.
  • Audio Captioning and AQA: Pengi outperformed supervised models on AudioCaps and Clotho by relative margins of 6.6% and 26%, respectively, using SPIDEr for captioning evaluation.All compared models used both AudioCaps and Clotho datasets in training.
  • Audio Captioning and AQA: Shared audio-encoder training consistently improves AudioCaps and Clotho captioning performance compared with training only on audio-captioning data.The ablation contrasts a captioning-only setup with the proposed multi-task training setup.
  • Audio Captioning and AQA: Pengi achieved 64.5% on ClothoAQA, outperforming the existing supervised benchmark by a relative 1.5% without further fine-tuning on the task.The evaluation required generated answers to directly match human responses, while the prior work converted AQA into classification.
  • Zero-Shot Sound Event Classification: On zero-shot ESC50, Pengi beats AudioCLIP, CLAP, and LAION-CLAP by 32%, 11%, and 1%, respectively, while achieving 92% accuracy.On US8k, Pengi outperformed Wav2CLIP and AudioCLIP but trailed CLAP and LAION-CLAP.
  • Text-to-Audio Retrieval: Pengi outperforms generative retrieval models on R@1, although contrastive models are substantially better at directly matching text to audio.Pengi indexes audio by generating captions and matches user queries against those captions using text embeddings.

6 Limitations

Pengi’s performance reflects a trade-off between close-ended and open-ended tasks, while its frozen language-model component inherits hallucination, grounding, bias, and offensive-language risks.

  • Trade-off between close-ended and open-ended tasks performance: Pengi’s classification and text-generation performance is competitive with contrastive models, but its text-based retrieval performance lags behind them.Contrastive models excel at retrieval but are limited to close-ended tasks, creating a trade-off between the learning approaches.
  • Limitations inherent to Language Models: Pengi can produce responses that are not grounded or conditioned on audio, especially when audio knowledge is limited or absent.In those cases, it can fall back to the behavior of the pretrained language model.
  • Limitations inherent to Language Models: Pengi inherits language-model risks including hallucination, stereotypes, biases, and potentially offensive language.These limitations arise because a pretrained language model is a component of Pengi.

7 Conclusions

Pengi frames audio tasks as text-generation tasks, accepting audio and a text prompt to generate free-form text for both close-ended and open-ended tasks. Across 21 downstream tasks, it achieves state-of-the-art performance in several tasks.

  • 7 Conclusions: Pengi frames all audio tasks as text-generation tasks, taking an audio recording and text prompt as input and producing free-form text.This approach is presented as a Transfer Learning strategy for Audio Language Models.
  • 7 Conclusions: Pengi handles both close-ended and open-ended audio tasks within the same architecture.The paper benchmarks this capability across 21 downstream tasks.
  • 7 Conclusions: 21 downstream tasks were used to benchmark Pengi, which achieved state-of-the-art performance in several of them.The authors present these findings as progress toward general-purpose audio understanding.

A Additional text input

Pengi combines audio and text representations into a prefix for a frozen language model, using text inputs to induce tasks and guide generated responses. A second text input can produce meaningful outputs in limited settings but may disrupt audio grounding.

  • A Additional text input: Pengi maps audio and text inputs into embedding sequences, concatenates them as a prefix, and prompts a pretrained frozen language model to generate tokens.The audio and text representations are produced by separate encoders and mapping networks.
  • A Additional text input: The first text input acts as task induction, directing the language model toward outputs such as captions or sentiment labels.Different prompts can therefore request different response types for the same audio recording.
  • A Additional text input: A second text input extends the prefix with another embedding sequence intended to guide a specific query or completion.The additional sequence originates from the frozen language model’s embedding function.
  • A Additional text input: Additional text input produced meaningful output mainly with “generate metadata,” while often causing audio loss, hallucination, or fallback to frozen-language-model behavior.The Figure 6 examples are cherry-picked, and grounding with additional text input remains unclear.

B Inferring audio prefix

The paper interprets Pengi’s prefix embeddings by matching them to the most similar GPT-2 vocabulary tokens. The inferred audio prefixes contain audio-related words, but their structure is difficult to interpret because the mapping network must align inputs while guiding a fixed language model.

  • B Inferring audio prefix: Each prefix embedding is mapped to the highest-cosine-similarity token in the GPT-2 vocabulary to approximate a token-level interpretation.This is possible because the prefix and GPT-2 embeddings occupy the same latent space.
  • B Inferring audio prefix: In ESC50 examples, inferred audio prefixes contain salient audio-related words such as babies, thunder, and chicken that also appear in generated outputs.The correspondence provides qualitative evidence that some prefix content relates to the input audio.
  • B Inferring audio prefix: The interpreted prefix lacks clear structure because the mapping network must comprehend audio and text while guiding the fixed language model.The paper reports no interpretable insights from applying a similar method to the text-input prefix.
  • B Inferring audio prefix: Table 11 compares downstream-task performance across different text prompts, with prompt applicability varying by task.For example, question prompts apply to AQA, while emotion-recognition tasks use an emotion prompt.

D Constrastive Learning model details

The contrastive-learning setup trains audio and text encoders to align their embeddings, then uses the resulting audio encoder in Pengi. Experiments also examine encoder freezing and the role of text encoding and mapping.

  • Contrastive model: The contrastive model uses transformer-based HTSAT audio and CLIP text encoders, each followed by a trainable projection layer.Both encoders and projection layers are fine-tuned before the encoders are used in Pengi.
  • Contrastive model: The similarity matrix C is formed from batch audio and text embeddings Et and Ea, each in R^N×d.The embeddings are compared across a batch of size N.
  • Contrastive model: Training uses a symmetric cross-entropy loss applied along the text and audio axes of the similarity matrix.The loss uses the diagonal matches after softmax normalization.
  • Training setup: The experiments train models for 45 epochs with Adam, batch size 1536, on 20 V100 GPUs, using a base learning rate of 1e-4.Audio is sampled at 44.1 kHz and truncated to 7 seconds for HTSAT.
  • Ablations and evaluation: Pengi experiments compare a frozen audio encoder, the proposed architecture, and a mapper without the text encoder using task-specific evaluation metrics.Table 13 reports the frozen-encoder comparison, while Tables 14 and 15 cover the mapper ablation and CLAP* comparison.

G Different type of Pengi errors

Pengi errors fall into audio concept, hierarchy, and text-matching categories. These errors arise from incorrect audio representations, mismatched label granularity, or the choice of text embedding and similarity method.

  • Audio concept errors: Audio concept errors occur when the encoder misidentifies the underlying sound event.An example is predicting dog barking instead of door knocks with nearby cars.
  • Hierarchy errors: Hierarchy errors occur when a prediction is too general for the target domain’s required fine-grained class.The paper suggests improved prompts or log-likelihood scoring as possible remedies.
  • Text-matching errors: Text-matching errors depend on the text embeddings and similarity method used for close-ended tasks.Changing these components changes Pengi’s performance on those tasks.

H Constrastive Learning and Generative Pretraining

The comparison isolates generative pretraining by training CLAP* on the same data as Pengi. Pengi outperforms CLAP* on average and additionally supports open-ended audio tasks, while exceeding reported human performance on ESC50.

  • Experimental comparison: CLAP* is trained on the same 3.4M audio-text pairs as Pengi to separate data-scale effects from generative pretraining.The comparison targets whether generative pretraining benefits close-ended tasks beyond the effect of larger training data.
  • Results: Pengi’s generative pretraining outperforms contrastive CLAP* on average and supports Audio Captioning and Audio Question Answering.The result is reported across the evaluated downstream tasks.
  • Results: Pengi exceeds the reported 81% human performance on ESC50.The paper attributes the human comparison partly to differences in exposure and listening conditions.
Loading 2305.11834v2…