Source-linked AI summary
Adapting Frechet Audio Distance for Generative Music Evaluation
Azalea Gui, Hannes Gamper, Sebastian Braun, Dimitra Emmanouilidou
TL;DR
Evaluating the perceptual quality of generative music remains difficult, and FAD’s relationship to that quality is understudied. The paper tests FAD’s sensitivity to sample size, embeddings, and reference sets, and finds that embedding choice matters for predicting MusicCaps quality labels and listening-test ratings.
Problem
FAD is commonly used to evaluate generative music, but its relationship to perceptual quality is understudied and varied evaluation choices complicate comparisons.
Source: §1 · p. 1Method
The paper extrapolates resampled FAD scores to infinite sample size and evaluates embeddings and reference sets using MusicCaps labels and a listening test.
Source: §1 · p. 1, §3.3 · p. 2, §4.2 · p. 3Results
CLAP and L-CLAP outperform other embeddings overall for predicting MusicCaps quality labels, while GPT-4 quality labels correlate well with listening-test MOS.
Source: §4.2 · p. 3, §4.4 · p. 4Takeaways & Limitations
Per-song FAD may help identify outlier samples, including songs with little musical information, and predict perceptual quality across music sets and models.
Source: §1 · p. 1, §4.3 · p. 3FAD’s use as a perceptual-quality proxy assumes a high-quality reference, quality-relevant embeddings, a multinormal fit, and a meaningful single score for the test set.
Source: §2 · p. 1
Abstract
from arXiv · showhide
The growing popularity of generative music models underlines the need for perceptually relevant, objective music quality metrics. The Frechet Audio Distance (FAD) is commonly used for this purpose even though its correlation with perceptual quality is understudied. We show that FAD performance may be hampered by sample size bias, poor choice of audio embeddings, or the use of biased or low-quality reference sets. We propose reducing sample size bias by extrapolating scores towards an infinite sample size. Through comparisons with MusicCaps labels and a listening test we identify audio embeddings and music reference sets that yield FAD scores well-correlated with acoustic and musical quality. Our results suggest that per-song FAD can be useful to identify outlier samples and predict perceptual quality for a range of music sets and generative models. Finally, we release a toolkit that allows adapting FAD for generative music evaluation.
1. INTRODUCTION
Perceptual-quality evaluation remains challenging for generative music, and FAD’s relationship to perceived quality is uncertain. The paper examines FAD’s limitations and proposes improvements for evaluation.
1. INTRODUCTION
- Recent models increasingly generate audio directly rather than symbolic MIDI [1] [2] [3] [4] [5] [6] [7], but evaluating their perceptual quality remains challenging.Source: §1 · p. 1
- FAD is a popular reference-based metric, commonly calculated with MusicCaps and VGGish embeddings, but prior work questions its reliability for perceptual quality [12].Differences in sample sizes, reference sets, and embeddings may also make published scores difficult or impossible to compare.Source: §1 · p. 1
- The paper introduces FAD∞ to reduce sample-size bias, evaluates per-song FAD for outlier detection, and tests reference sets and embeddings against perceptual quality.It also releases a toolkit for music evaluation.Source: §1 · p. 1
2. BACKGROUND
FAD compares reference and test audio distributions in an embedding space, but its interpretation as a perceptual-quality proxy depends on several assumptions.
2. BACKGROUND
- FAD compares embedding distributions for reference and test audio, and its perceptual interpretation assumes a high-quality reference, quality-relevant embeddings, a suitable multinormal fit, and a meaningful set-level score.Source: §2 · p. 1
3. FAD FOR MUSIC EVALUATION
The paper examines reference-set quality, embedding choice, and sample-size bias as factors in adapting FAD for music evaluation.
3.1. Reference sets for quality estimation
- MusicCaps includes samples labeled low-quality and some nonmusic, while the proposed FMA-Pop and MusCC references span broad recordings and genres versus professionally mixed studio-quality music.Source: §3.1 · pp. 1–2
3.2. Audio embedding models for quality estimation
- Because embeddings trained with different objectives may capture different audio aspects and no preferred music-quality embedding is established, the study evaluates models spanning classifiers, audio–text, self-supervised, similarity, and codec objectives.Source: §3.2 · p. 2
3.3. Addressing sample size bias
- FAD generally decreases as sample size grows, but at different rates across models and embeddings; linear regression over resampled scores extrapolates an FAD∞ estimate.The figure compares EnCodec across test sets and multiple embeddings on MusicCaps, each with FMA-Pop as reference; prior evaluations may omit sample sizes or data.Source: §3.3 · p. 2, Figure 1 · p. 2
4. EXPERIMENTAL EVALUATION
Across four experiments, FAD performance depended on the embedding and reference set, while per-song scores showed promise for predicting quality and flagging outliers. Listening-test results supported meaningful correlations with acoustic and musical quality for appropriate choices.
4.1. Acoustic Sensitivity Test
- EnCodec and EnCodec-48k were particularly sensitive to distortion and low-pass filtering, while none of the tested embeddings was overly sensitive to reverberation.All models showed some sensitivity to pitch shifting.Source: §4.1 · p. 2, Figure 2 · p. 3
4.2. Predicting MusicCaps quality with per-song FAD
- CLAP and L-CLAP generally performed best at predicting MusicCaps quality labels, while EnCodec performed well for acoustic quality.MERT layer 4 performed similarly for musical quality; VGGish predicted musical-quality labels only slightly better than chance. MusicCaps as a reference reduced acoustic-quality prediction for CLAP and L-CLAP.Source: §4.2 · p. 3, Figure 3 · p. 3
4.3. Identifying outliers with per-song FAD
- Per-song FAD flagged synthetic low-information tones with MERT and heavily distorted or clipped samples with EnCodec.The excerpts selected by L-CLAP and VGGish also suggested that embedding and reference choices may favor particular genres or instruments.Source: §4.3 · pp. 3–4
4.4. Subjective evaluation
- CLAP and L-CLAP per-song FAD scores correlated better with listening-test quality ratings than other tested embeddings.MusCC appeared to improve musical-quality prediction over MusicCaps, especially for CLAP and L-CLAP; acoustic-quality differences between the references were less clear.Source: Figure 5 · p. 4, §4.4 · p. 4
5. CONCLUSION
The conclusion argues that carefully chosen embeddings and reference datasets can make FAD meaningfully correlate with subjective music quality. It also identifies per-song FAD as a possible aid for finding outliers.
5. CONCLUSION
- With an appropriate embedding and reference dataset, FAD scores meaningfully correlated with subjective acoustic and musical quality ratings.This conclusion draws on both subjective labels extracted from MusicCaps captions and a listening test.Source: §5 · p. 4
- Per-song FAD may help identify outliers, including samples with severe artifacts, atypical musical characteristics, or non-music signals.Source: §5 · p. 4
- The experiments support estimating and mitigating FAD sample-size bias and caution that classification-trained embeddings such as VGGish may not be optimal.The paper also releases an FAD toolkit for generative-music evaluation.Source: §5 · p. 4