PaperlayerView exact revision on arXiv

Source-linked AI summary

Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation

Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, Zhou Zhao

arXiv:2305.18474v1cs.SDcs.LGcs.MMeess.AS

TL;DR

Text-to-audio systems face temporal misalignment, variable-length generation difficulties, and limited temporally aligned data. Make-An-Audio 2 combines LLM-parsed event order, structured text encoding, a temporal Transformer denoiser, and LLM-based data augmentation; it reports gains over baselines in objective and subjective metrics.

  • Problem

    Prior T2A methods can misalign or disorder events in complex captions, produce suboptimal variable-length audio, and rely on insufficient temporally aligned training pairs.

    Source: §1 · p. 2

  • Method

    Make-An-Audio 2 uses LLM-parsed event-and-order pairs, a structured-text encoder, a temporal feed-forward Transformer denoiser, and LLM-augmented audio-text data.

    Source: §1 · p. 2, §4.3 · p. 6

  • Results

    Make-An-Audio 2 surpasses baselines in objective and subjective metrics, with reported gains in temporal understanding, semantic consistency, and sound quality.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The authors report that the model better understands event order and generates variable-length audio without performance degeneration.

    Source: §6 · p. 10

    Speech synthesis is not yet supported and is left for future work.

    Source: E Limitations, future works and… · p. 17

Abstract

from arXiv · show

Large diffusion models have been successful in text-to-audio (T2A) synthesis tasks, but they often suffer from common issues such as semantic misalignment and poor temporal consistency due to limited natural language understanding and data scarcity. Additionally, 2D spatial structures widely used in T2A works lead to unsatisfactory audio quality when generating variable-length audio samples since they do not adequately prioritize temporal information. To address these challenges, we propose Make-an-Audio 2, a latent diffusion-based T2A method that builds on the success of Make-an-Audio. Our approach includes several techniques to improve semantic alignment and temporal consistency: Firstly, we use pre-trained large language models (LLMs) to parse the text into structured <event & order> pairs for better temporal information capture. We also introduce another structured-text encoder to aid in learning semantic alignment during the diffusion denoising process. To improve the performance of variable length generation and enhance the temporal information extraction, we design a feed-forward Transformer-based diffusion denoiser. Finally, we use LLMs to augment and transform a large amount of audio-label data into audio-text datasets to alleviate the problem of scarcity of temporal data. Extensive experiments show that our method outperforms baseline models in both objective and subjective metrics, and achieves significant gains in temporal information understanding, semantic consistency, and sound quality.

1 Introduction

Make-An-Audio 2 addresses temporal disorder, variable-length generation, and scarce temporally aligned audio-text data with a temporal-enhanced T2A framework. Experiments on AudioCaps and Clotho report gains over baselines in objective and subjective metrics.

1 Introduction

  • Prior T2A systems can misorder or omit sounds in complex captions, struggle with variable-length audio, and lack sufficiently rich temporally aligned training pairs.The cited passage connects variable-length shortcomings to fixed-length training and 2D structures that treat time and frequency equally.Source: §1 · p. 2
  • LLMs parse captions into ordered event pairs, while a structured-text encoder helps the diffusion process learn semantic alignment and event timing.This is intended to reduce the text encoder’s burden of recognizing events and their temporal information.Source: §1 · p. 2
  • A temporal feed-forward Transformer and 1D-convolution stacks replace 2D spatial structures, with variable-length audio supported during training.Source: §1 · p. 2
  • The method composes complex audio and structured captions from single-labeled samples, then uses an LLM to augment the captions into natural language.Source: §1 · p. 2
  • On AudioCaps and Clotho, Make-An-Audio 2 surpasses baselines in objective and subjective metrics, with gains in temporal understanding, semantic consistency, and sound quality.The paper also reports effective ablation results for its techniques.Source: §1 · p. 2

2 Related works

Related work covers generative approaches to images and audio, alongside language-model-based data augmentation. The cited methods vary in their generation strategies and ways of addressing limited or noisy data.

2.1 Text-to-image generative models

  • Text-to-image work includes autoregressive code generation and diffusion pipelines, with Latent Diffusion using a continuous VAE to compress images before diffusion.Source: §2.1 · p. 3

2.2 Text-to-audio synthesis

  • Text-to-audio systems use discrete-code diffusion, autoregressive waveform tokens, or latent diffusion; prior data strategies include pseudo-prompt enhancement and audio mixing.AudioGen mixes audio files and concatenates captions, while Make-An-Audio and TANGO propose distinct approaches to data scarcity.Source: §2.2 · p. 3

2.3 LLM-based data augmentation

  • LLM augmentation follows earlier word- and sentence-level methods; WavCaps uses ChatGPT to filter noisy web descriptions and generate captions from labels.The passage describes WavCaps as a large-scale, weakly labeled audio-captioning dataset.Source: §2.3 · pp. 3–4

3 Preliminary

The preliminary section outlines Make-An-Audio’s mel-spectrogram latent-diffusion pipeline and its pseudo-prompt captioning approach. It also notes limits of the generated captions.

3 Preliminary

  • Make-An-Audio converts waveforms to mel-spectrograms, then uses a VAE to encode compressed latents and reconstruct mel-spectrograms.The VAE objective combines reconstruction, GAN, and KL-penalty losses.Source: §3 · p. 4
  • A text encoder conditions latent diffusion, whose U-Net is trained to minimize mean squared error in noise space.The conditioning embedding enters through cross-attention; the objective is stated as optimizing the noise prediction.Source: §3 · p. 4
  • At inference, the diffusion network denoises sampled Gaussian noise into a latent, which the decoder and vocoder turn into audio; classifier-free guidance adjusts conditioning.As the guidance scale increases, the model reduces the probability of samples that do not use conditioning information.Source: §3 · p. 4
  • Pseudo-prompt enhancement uses captioning and audio-text retrieval models to construct captions for audio without natural-language annotations.The authors note that caption quality and template diversity are limited.Source: §3 · p. 4
  • The data-augmentation overview composes complex audio and structured captions from single-labeled audio and labels, then uses an LLM to generate diverse natural-language captions.Source: Figure 2 · p. 5

4 Make-An-Audio 2

Make-An-Audio 2 targets temporal modeling in text-to-audio generation with structured caption processing, data augmentation, and a Transformer-based denoiser. Its designs address temporal information and variable-length audio generation.

4.1 Motivation

  • A 1D-convolution-based audio VAE and feed-forward Transformer denoiser target variable-length generation, unlike prior 2D-convolution and spatial-transformer U-Net backbones.The design reflects that mel-spectrogram frequency positions are not spatially translation invariant.Source: §4.1 · p. 5

4.2 Temporal enhancement

  • LLMs parse natural-language captions into structured <event & order> pairs, linking each sound event to its corresponding order.The authors use LLM language understanding to provide temporal knowledge for the T2A model.Source: §4.2 · p. 5

4.3 Dual text encoders

  • The dual text encoders combine a main CLAP encoder with a temporal T5 encoder, whose weights are fine-tuned to capture event order.The main encoder is frozen, while the temporal encoder is fine-tuned on structured captions.Source: §4.3 · p. 6

4.4 LLM-based data augmentation

  • LLM-based data augmentation turns audio-label data into audio-text examples by composing tagged events and generating natural-language captions from structured captions.The process builds an event database from single-tag data, randomly mixes or concatenates two or three samples, and labels event timing as “all,” “start,” “mid,” or “end.”Source: §4.4 · p. 6

4.5 Transformer-based diffusion denoiser backbone

  • The feed-forward Transformer denoiser uses 1D convolution and a temporal Transformer to process audio latents, supporting variable-length audio generation.Its reported computation complexity drops from O((Ca/f × T/f)^2 × D) to O((T/f2)^2 × D).Source: §4.5 · pp. 6–7

5 Experiments

Make-An-Audio 2 is evaluated on audio quality, text-audio alignment, zero-shot performance, variable-length generation, and component analyses. The reported results favor its objective and human-rated quality, while analyses identify gains from temporal and encoder designs and sensitivity to vocoder mel processing.

5.1 Experimental setup

  • Training uses 0.92 million audio-text pairs spanning approximately 3.7K hours, with variable-duration batching and about 61K LLM-augmented pairs.The setup also truncates audio longer than 20 seconds and evaluates on AudioCaps-test and the zero-shot Clotho evaluation set.Source: §5.1 · p. 7

5.2 Main results

  • Make-An-Audio 2 scores better in FD, IS, KL, and FAD, with comparable CLAP scores, against baseline models on AudioCaps.Table 1 compares Make-An-Audio 2 with baseline T2A models on AudioCaps.Source: §5.2 · pp. 7–8, Table 1 · p. 8
  • MOS-Q is 78.31 and MOS-F is 75.63; Make-An-Audio 2 outperforms TANGO on MOS-F and raters prefer its naturalness and faithfulness.The human evaluation assesses audio quality and text-audio alignment faithfulness.Source: §5.2 · p. 8
  • On zero-shot Clotho-evaluation, Make-An-Audio 2 has significantly better results than TANGO and AudioLDM-L.Table 2 also compares Clotho-eval at 10 seconds and AudioCaps-test at 5 seconds.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 Analyses

  • Make-An-Audio 2 maintains high performance for variable-length generation, whereas TANGO and AudioLDM degrade when generating lengths unlike their 10-second training audio.The authors attribute this contrast to variable-length training and 1D-convolution/temporal-transformer components versus fixed-length training and 2D image-oriented processing.Source: §5.3 · p. 8
  • A classifier-free guidance scale of w = 4 yields the best performance in FD and FAD.The guidance-scale comparison is on AudioCaps-test.Source: §5.3 · p. 8, Table 3 · p. 9
  • Vocoder results favor BigVGAN’s mel-spectrogram extraction process; the authors attribute poor performance of two other vocoders to their shared mel-processing method.The compared vocoders include BigVGAN, AudioLDM’s HifiGAN, Make-An-Audio’s HifiGAN, and Diffsound’s MelGAN.Source: §5.3 · p. 9, Table 4 · p. 9

5.4 Ablation study

  • Ablations report gains from structured event-and-order inputs, dual text encoders, and replacing the 2D-convolutional VAE and U-Net with a 1D-convolutional VAE and feed-forward Transformer layers.The structured-input setting uses trainable T5; the dual-encoder comparison retains frozen CLAP to process the original caption.Source: §5.4 · pp. 9–10

6 Conclusions

The conclusion presents Make-An-Audio 2 as a temporal-enhanced text-to-audio model. It reports strong objective and subjective audio-generation quality alongside better handling of event order, variable lengths, and complex temporal relationships.

6 Conclusions

  • Make-An-Audio 2 combines LLM-extracted temporal information, a 1D-convolutional VAE, a feed-forward Transformer diffusion backbone, and LLM-based data augmentation.The conclusion says the model can generate variable-length audio without performance degeneration and handle complex temporal relationships and multiple concepts.Source: §6 · p. 10

Appendices

The supplied appendix passage contains only the paper title.

Appendices

  • The appendix passage supplies the title “Make-An-Audio 2: Temporal-Enhanced Text-to-Audio.”No appendix content is present in the supplied passage.Source: Appendices · p. 14

A Data details

The dataset statistics describe a large, varied audio-text collection and a filtering step to balance its contents.

A Data details

  • The dataset comprises 0.92 million audio samples totaling approximately 3.7K hours, spanning music, effects, voices, nature, and living sounds.For balance, 95% of AudioSet samples containing speech and music are filtered out.Source: A Data details · p. 14, Table 6 · p. 14

B Experimental details

The experimental details specify the VAE architecture and training setup, latent diffusion training settings, and tables listing component differences, backbone configurations, and parameter counts.

B Experimental details

  • The VAE replaces 2D convolutions with 1D convolutions and a spatial transformer with a temporal transformer, and is trained on AudioSet for 800k steps.It uses 8 NVIDIA A100 GPUs, batch size 32, and R1 regularization; Table 7 lists differences from the Make-An-Audio VAE.Source: B Experimental details · p. 14, Table 7 · p. 15
  • The latent diffusion model is trained for 1.2M steps on 8 NVIDIA A100 GPUs with batch size 32 and a 9.6 × 10−5 learning rate.Its backbone configurations are listed in Table 8.Source: B Experimental details · p. 14, Table 8 · p. 15
  • Table 9 displays the parameter counts for each Make-An-Audio 2 component.The supplied passage does not provide the counts themselves.Source: B Experimental details · p. 14

C.1 Subjective evaluation

This material describes the human evaluations, objective audio metrics, and ChatGPT prompt templates and examples used for evaluation and data preparation.

C.1 Subjective evaluation

  • Crowd-sourced MOS tests assess audio quality and text-audio faithfulness, with raters scoring audio quality and caption alignment on 20–100 scales.Faithfulness raters choose completely, mostly, or somewhat; the tests use Amazon Mechanical Turk, with participants paid $8 hourly.Source: §C.1 · p. 15

D ChatGPT prompts

  • ChatGPT prompts support temporal enhancement and caption augmentation; structured event-and-order pairs use “&” within pairs and “@” between pairs.Tables 10 and 11 show temporal-enhancement and data-augmentation examples, respectively.Source: Figure 3 · p. 16, D ChatGPT prompts · p. 16, Table 10 · p. 17

C.2 Objective evaluation

  • FAD is a reference-free perceptual distance between generated and ground-truth audio distributions, while CLAP measures audio-text alignment.FAD is adapted from FID; CLAP is adapted from CLIP score and closely correlates with human perception.Source: §C.2 · p. 16
  • FD measures distribution similarity, IS evaluates quality and diversity, and KL compares generated and ground-truth label distributions.FD, KL, and IS use PANNs on mel-spectrograms; FAD uses VGGish on raw waveforms, while CLAP measures audio-text alignment and is reference-free.Source: §C.2 · p. 16

E Limitations, future works and broader impact

The paper notes costs and limitations of its approach, identifies speech synthesis and optional structured inputs as future work, and flags possible broader social and ethical impacts.

E Limitations, future works and broader impact

Loading 2305.18474v1…