PaperlayerView exact revision on arXiv

Source-linked AI summary

FakeMusicCaps: a Dataset for Detection and Attribution of Synthetic Music Generated via Text-to-Music Models

Luca Comanducci, Paolo Bestagini, Stefano Tubaro

arXiv:2409.10684v2eess.AScs.SD

TL;DR

Music-focused audio-forensics research and paired text-music datasets are limited, despite the ease of using text-to-music models. The paper introduces FakeMusicCaps, a MusicCaps-based dataset generated by five TTM models, and reports initial detection and attribution results. The authors present it as an initial benchmark rather than a definitive account of the problem.

  • Problem

    Paired text-music datasets are limited, and the paper identifies a need to study detection and attribution of music generated by text-to-music models.

    Source: §I · p. 1

  • Method

    FakeMusicCaps replicates MusicCaps using captions to generate music with five TTM models, and the paper benchmarks closed- and open-set detection and attribution.

    Source: §I · p. 1

  • Results

    ResNet18 + Spec obtains the best closed-set performance across the reported metrics and the best open-set performance with both approaches, while SVM results are much worse than thresholding.

    Source: A. Closed-Set Performances · p. 3, B. Open Set Performances · p. 3

  • Takeaways & Limitations

    The authors present FakeMusicCaps as an initial benchmark for applying audio-forensics techniques to TTM detection and attribution.

    Source: §VI · p. 4

    The authors do not consider their results definitive, noting that the problem and TTM models are changing rapidly.

    Source: D. Discussion · p. 4

Abstract

from arXiv · show

Text-To-Music (TTM) models have recently revolutionized the automatic music generation research field. Specifically, by reaching superior performances to all previous state-of-the-art models and by lowering the technical proficiency needed to use them. Due to these reasons, they have readily started to be adopted for commercial uses and music production practices. This widespread diffusion of TTMs poses several concerns regarding copyright violation and rightful attribution, posing the need of serious consideration of them by the audio forensics community. In this paper, we tackle the problem of detection and attribution of TTM-generated data. We propose a dataset, FakeMusicCaps that contains several versions of the music-caption pairs dataset MusicCaps re-generated via several state-of-the-art TTM techniques. We evaluate the proposed dataset by performing initial experiments regarding the detection and attribution of TTM-generated audio.

I. INTRODUCTION

Text-to-music models are capable and easy to use, while music-focused audio-forensics research and available paired text-music datasets remain limited. The paper introduces FakeMusicCaps to support detection and attribution research, using five TTM models and examining closed- and open-set classification.

I. INTRODUCTION

  • TTM systems generate music from text using either language models that decode audio-codec tokens or latent diffusion models operating on compressed audio.Examples listed include MusicGen, AudioLDM2, and Mustango.Source: §I · p. 1
  • TTM models combine strong performance with ease of use, lowering the technical proficiency needed to use them.The authors note their uptake among the general public and interest from private industries.Source: §I · p. 1
  • The audio-forensics field has focused mainly on speech, while music-domain work has emphasized singing-voice detection rather than fake-music detection.A prior music deepfake approach studied which neural audio codec processed real tracks, not generated fake music.Source: §I · p. 1
  • Paired text-music datasets are scarce, and many TTM models lack publicly released code or weights.The passage identifies MusicCaps, Song Describer, and MusicBench as available paired datasets.Source: §I · p. 1
  • FakeMusicCaps replicates MusicCaps captions with five state-of-the-art TTM models to support detection and attribution benchmarks.The study evaluates closed-set classification and open-set classification that includes unseen generators from SunoCaps.Source: §I · p. 1
  • The authors focus on open-source TTMs, using commercial models only for open-set classification, unlike SONICS, which covers Suno and Udio for real/fake detection.The paper says open-source techniques may be more available to the research community for integration into FakeMusicCaps.Source: §I · p. 1

II. PROBLEM FORMULATION

The text-to-music attribution problem asks which generator produced an audio signal. The paper distinguishes closed-set cases, where all generators are known during training, from open-set cases, where some are encountered only at test time.

II. PROBLEM FORMULATION

  • Given audio x and a set of TTM generators, attribution trains a classifier to output probabilities that x came from each known generator.The problem formulation represents a TTM as generating waveform x from text representation τ.Source: §II · p. 2
  • Closed-set evaluation uses generators seen during both training and testing, whereas open-set evaluation includes generators unseen during training.The open-set setting requires specific classification strategies for unknown generators.Source: §II · p. 2
  • Fig. 1 schematically represents the text-to-music attribution problem.The passage identifies the figure's scope but does not specify its visual encodings.Source: Figure 1 · p. 2

III. FAKEMUSICCAPS DATASET

FakeMusicCaps is constructed by applying selected TTM architectures to MusicCaps captions. The dataset section introduces five models spanning autoregressive and diffusion-based approaches.

III. FAKEMUSICCAPS DATASET

  • The dataset uses five TTM architectures: MusicGen, MusicLDM, AudioLDM2, Stable Audio Open, and Mustango.The section presents the selected models before describing the generation procedure.Source: A. Considered Architectures · p. 2, §III · p. 2

A. Considered Architectures

  • TTM01-MusicGen is an autoregressive, single-stage transformer that decodes discrete audio tokens obtained through Encodec.The dataset uses the medium checkpoint with 1.5B parameters.Source: A. Considered Architectures · p. 2
  • TTM02-MusicLDM is a latent diffusion model operating on compressed audio representations extracted via HiFi-GAN.It adapts AudioLDM to music with beat-synchronous audio and latent mixup; its model was trained on the 455.6-hour Audiostock dataset.Source: A. Considered Architectures · p. 2
  • TTM03-AudioLDM2 is a latent diffusion model that compresses audio via a VAE and HiFiGAN and uses AudioMAE conditioning.The paper uses the audioldm2-music checkpoint, trained specifically for text-to-music generation.Source: A. Considered Architectures · p. 2
  • TTM04-Stable Audio Open is a latent-diffusion architecture using T5 as a text encoder and generating stereo audio at 44.1 kHz.It was trained on 7.3K hours of Creative Commons-licensed audio.Source: A. Considered Architectures · p. 2
  • TTM05-Mustango uses a music-domain-knowledge-informed UNet, MuNet, to inject concepts such as chords, beats, key, or tempo during reverse diffusion.The model was trained using the 53,168-track MusicBench dataset.Source: A. Considered Architectures · p. 2

B. Generation strategy

FakeMusicCaps pairs MusicCaps captions with generated ten-second tracks from five TTM models. The authors standardize the resulting audio format across tracks.

B. Generation strategy

  • The authors generated 27,605 ten-second tracks from MusicCaps captions using TTM01–TTM05, for almost 77 hours of audio.Tracks were converted to mono, downsampled to 16 kHz, and saved as 32-bit float WAV files.Source: B. Generation strategy · p. 2

IV. EXPERIMENTAL ANALYSIS

The experiments provide an initial validation of FakeMusicCaps using a disjoint test set and three neural-network baselines. The baselines operate on raw audio or log-spectrograms.

IV. EXPERIMENTAL ANALYSIS

  • The experimental analysis is a first validation of FakeMusicCaps for detection and attribution of TTM-generated music.Source: §IV · p. 2

A. Dataset

  • The test set contains 320 FakeMusicCaps tracks, selected to match SunoCaps captions and support open-set experiments with Suno-generated excerpts.Tracks were selected for each TTM model.Source: A. Dataset · p. 2

B. Baselines

  • The benchmark compares M5, a 0.5M-parameter raw-audio network, with RawNet2 on raw audio and ResNet18+Spec on log-spectrograms.ResNet18 was adapted to work with one-channel log-spectrograms.Source: B. Baselines · p. 2

C. Training

The models were trained to distinguish five known TTM models from real MusicCaps audio, using cross-entropy and Adam. Training stopped early when loss failed to improve.

C. Training

  • The six training classes comprise five known TTM models and real MusicCaps signals.Source: C. Training · p. 3
  • Training used cross-entropy loss and Adam with a learning rate of 1e −4, for at most 100 epochs with early stopping after 10 unimproved epochs.Source: C. Training · p. 3
  • Batch sizes were 32 for M5 and 16 for RawNet2 and ResNet18+Spec; ResNet18+Spec used a 512-point STFT with a 512-sample Hann window and 128-sample hop.Source: C. Training · p. 3

D. Classification Techniques

The paper tests closed-set attribution and introduces threshold and one-class-SVM approaches to open-set classification. In closed-set results, ResNet18+Spec performs best across the reported metrics.

D. Classification Techniques

  • Closed-set attribution selects the highest softmax score; open-set classification uses either a threshold on the two largest scores or a one-class RBF-kernel SVM.The threshold is 2, following.Source: D. Classification Techniques · p. 3

V. RESULTS

  • The experiments are preliminary and aim to demonstrate FakeMusicCaps' suitability for TTM detection and attribution.Source: §V · p. 3

A. Closed-Set Performances

B. Open Set Performances

The open-set experiments compare thresholding and one-class SVM. ResNet18+Spec performs best overall by the reported comparison, but its threshold approach misclassifies every unknown excerpt as real, while SVM results are much worse.

B. Open Set Performances

  • ResNet18+Spec performs best with both open-set methods, while SVM results are much worse than threshold results across the models.The relevant comparisons are reported in the threshold and SVM performance tables.Source: B. Open Set Performances · p. 3, Table II · p. 3, Table III · p. 3
  • With thresholding, ResNet18+Spec classifies known methods best but misclassifies all UNKNWN excerpts as real; M5 and RawNet2 also confuse unknowns with generated examples.M5 confounds unknowns with real and MusicGen examples; RawNet2 mostly confounds them with MusicGen.Source: B. Open Set Performances · p. 3, Table II · p. 3
  • With the SVM approach, models differ in behavior and mistake known techniques for unknown about half the time; RawNet2 has 0.82 accuracy on the unknown class.ResNet18+Spec also mistakes unknown examples for real ones.Source: B. Open Set Performances · p. 3, Table III · p. 3

C. Impact of window size

Window-length variation had modest effects across classification scenarios, although model-specific patterns emerged. The discussion presents the results as preliminary and identifies text-description and music-theory questions for future work.

C. Impact of window size

  • Accuracy shifts are not extreme across four windows (2.5–10 s): M5 improves from 7.5 to 10 s in closed-set and threshold open-set tests, while other trends are smaller or less clear.ResNet18+Spec shows no major improvements, RawNet2 a slight increase, and the SVM open-set trend is less clear.Source: C. Impact of window size · p. 3, Figure 3 · p. 4

D. Discussion

  • The authors frame the results as an initial, non-definitive approach, noting that apparent closed-set ease may not last as TTM quality rises.They particularly note increasing quality in commercial models.Source: D. Discussion · pp. 3–4
  • Future work could examine textual-description effects and music-theory approaches to detecting deepfakes, including generated music spliced into real tracks.Source: D. Discussion · p. 4

VI. CONCLUSION

FakeMusicCaps replicates MusicCaps using five state-of-the-art TTM models. The authors present it as an initial benchmark for detection and attribution, not a definitive result.

  • FakeMusicCaps replicates MusicCaps using five state-of-the-art TTM models and provides an initial benchmark for TTM detection and attribution, not a definitive result.Source: §VI · p. 4
Loading 2409.10684v2…