Source-linked AI summary
Fréchet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, Matthew Sharifi
TL;DR
Music enhancement lacks a quantitative metric that reliably reflects perceived quality, while common signal metrics can require clean references and disagree with listeners. The paper introduces FAD by adapting FID to audio embeddings and validates it against controlled distortions and human ratings. FAD correlates more closely with human perception than the compared signal-based metrics, while its VGGish representation remains limited in temporal scope and input design.
Problem
Music enhancement is difficult to compare quantitatively by perceived output quality, and SDR may favor signal similarity over perceptual quality while requiring a clean reference.
Method
FAD compares statistics of VGGish embeddings from enhanced music with statistics from a large clean-music set using Fréchet distance between multivariate Gaussians.
Results
FAD correlated better with human ratings than SDR across a diverse set of artificial distortions.
Takeaways & Limitations
FAD can measure distortion intensity and is particularly useful for distortions that produce low SDR regardless of distortion intensity.
Takeaways & Limitations
The implementation uses 1 s windows, so FAD is unaware of long-distance temporal changes within a song.
Abstract
from arXiv · showhide
We propose the Fréchet Audio Distance (FAD), a novel, reference-free evaluation metric for music enhancement algorithms. We demonstrate how typical evaluation metrics for speech enhancement and blind source separation can fail to accurately measure the perceived effect of a wide variety of distortions. As an alternative, we propose adapting the Fréchet Inception Distance (FID) metric used to evaluate generative image models to the audio domain. FAD is validated using a wide variety of artificial distortions and is compared to the signal based metrics signal to distortion ratio (SDR), cosine distance and magnitude L2 distance. We show that, with a correlation coefficient of 0.52, FAD correlates more closely with human perception than either SDR, cosine distance or magnitude L2 distance, with correlation coefficients of 0.39, -0.15 and -0.01 respectively.
1 Introduction
Music enhancement lacks a reliable quantitative way to compare approaches by perceived output quality. The paper proposes FAD as a reference-free alternative based on embedding statistics and human-perception comparison.
- Music enhancement methods are difficult to compare quantitatively with respect to perceived output quality.
- SDR and SIR identify outputs closest to studio recordings but can rank less pleasing reconstructions higher because they ignore perceptual quality.
- SDR and SIR are full-reference metrics requiring access to the studio-recorded target.
- The paper proposes Fréchet Audio Distance (FAD), adapted from image-generation evaluation, for evaluating generated audio.
- FAD compares reconstructed-clip statistics with background statistics from a large studio-recorded music set and is compared with SDR against human ratings.
2 Related Work
Prior evaluation approaches include reference-based signal metrics, human judgments, and speech-specific automatic perceptual metrics. The paper adapts the image-based FID framework to music enhancement.
- Speech enhancement evaluation commonly uses direct signal comparisons, human evaluations, and signal-based heuristics designed to correlate with human scores.
- Reference-based metrics include cosine distance, L2 distance, SNR, SDR, and SIR, and can be applied to music enhancement without modification.
- Signal-level metrics do not necessarily predict how listeners perceive reconstructed music.
- Automatic metrics such as PESQ and STOI approximate human judgments but are designed specifically for speech quality.
- FAD adapts FID by comparing embeddings from generated audio with embeddings from a large background set using Fréchet distance between fitted Gaussians.
3 Fr´echet Audio Distance (FAD)
FAD measures how enhanced audio compares with clean studio music by comparing statistics of VGGish embeddings rather than individual reference clips. Its implementation uses Gaussian statistics over overlapping one-second windows, with known representation limits.
- FAD is motivated by disagreements between signal metrics and subjective judgments of whether enhanced audio sounds studio-quality.
- FAD compares embedding statistics from an evaluation set with statistics from a large clean-music set, making it reference-free.
- The method computes multivariate Gaussians for evaluation and background embeddings, then applies the Fréchet distance.
- VGGish supplies 128-dimensional embeddings from activations before its final classification layer.
- VGGish processes 1 s of log-mel features, and the implementation extracts windows every 0.5 s, creating 50% overlap.
- The authors note that ignoring phase and using mel-scaled bins may allow some distortions to go undetected.
4 Experimental Setup
The experiments validate FAD on controlled audio distortions, compare it with signal-based metrics, and collect pairwise human judgments of studio-like quality. Distortion intensity is varied systematically across a music evaluation set.
- Experimental design: The study computes clean-background statistics, applies controlled distortions to evaluation clips, and obtains an FAD score for each parameter configuration.
- Experimental design: Higher distortion intensity is expected to produce higher FAD scores for a given distortion function.
- Distortions: Gaussian noise varies σ and is added to the input signal, while pops alter a random p% of samples.
- Distortions: Mel encoding, frequency filtering, quantization, speed changes, pitch changes, reverberation, and Griffin-Lim reconstruction provide diverse distortion types.
- Distortions: All distortions are applied separately and in parallel to evaluation segments, producing scores for each tested configuration.
- Data: The dataset contains 600 hours of music: 540 hours for clean background statistics and 60 hours for metric evaluation.
- Evaluation metrics: Cosine distance, magnitude L2 distance, and SDR are computed alongside FAD for each distortion configuration.
- Human evaluation: Human raters compare pairs of 5 s clips, including clean audio as a pseudo-distortion, and may declare ties.
5 Results
FAD generally increases with distortion intensity and distinguishes distortions that signal-based metrics rate similarly or inconsistently. In human evaluations, FAD correlates more closely with perceived distortion severity than SDR, cosine distance, or magnitude L2 distance.
- 5 Results: FAD scores generally increase as distortion magnitude increases, while barely detectable Gaussian noise remains near clean audio’s score of 0.2.Very small Gaussian-noise distortions score 0.2–0.3, almost matching non-distorted clean audio.
- 5 Results: Similar FAD scores corresponded to similar subjective quality across distortion types, including Gaussian noise, pops, and 4-bit quantization.The authors observed comparable quality for Gaussian noise with standard deviation 0.031 and pops at 0.1, with 4-bit quantization slightly worse.
- 5 Results: FAD detected phase and mel-filter distortions despite embeddings using mel-scale magnitude spectrograms, with Griffin-Lim scores falling from 2.4 to about 0.31 as iterations increased.The Griffin-Lim score plateaued around 0.31 after 500 iterations; mel-filter scores were 2.4 with 64 bins and 4.3 with 32 bins.
- 5.1 Comparison to Signal Based Metrics: For time-displacing distortions, cosine distance remained 1 while FAD usually increased monotonically with distortion intensity; SDR was variable and generally low.This group included speed changes, pitch changes, and Griffin-Lim; SDR can still detect some differences because it allows time-invariant filter distortions up to T samples.
- 5.1 Comparison to Signal Based Metrics: FAD and magnitude L2 distance were individually correlated on a log scale but often disagreed about distortion intensity, especially for Griffin-Lim.Griffin-Lim magnitude L2 distances stayed between 0.4 and 0.6 while FAD scores varied more widely.
- 5.3 Human Evaluation: FAD correlated better with human distortion ratings than SDR, cosine distance, and magnitude L2 distance, with coefficients of 0.52, 0.39, −0.15, and −0.01 respectively.SDR performed poorly on several distortions, while cosine distance and magnitude L2 distance failed to compare different distortion types effectively.
6 Conclusion
The paper concludes that FAD is a reference-free metric for music enhancement that compares output embeddings with clean-music embeddings. Across diverse artificial distortions, FAD measures distortion intensity and correlates better with human ratings than SDR.
- FAD measures music-enhancement quality by comparing output-embedding statistics with statistics from a large set of clean music.
- FAD is reference-free: it requires only enhanced music output, not the original clean music or noise signal.
- FAD was useful for measuring distortion intensity across a large, diverse set of artificial distortions.
- FAD correlated better with human ratings than SDR when evaluating the quality of enhanced music.
7 Future Work
Future work extends FAD evaluation beyond the music-enhancement setting and beyond the tested distortions. The current implementation also cannot capture long-distance temporal changes because it uses one-second audio windows.
- FAD was evaluated for music enhancement, while its usefulness for other audio enhancement and generation algorithms remains a hypothesis.
- The evaluated distortions do not cover all distortions arising in real-world signals or enhancement, especially in other audio domains.
- Future evaluation should include additional distortions and combinations of distortions.
- The one-second embedding windows make FAD unaware of long-distance temporal changes within a song.
- Future implementations could replace VGGish inputs or use variable-length models producing one embedding per song.
A Window Step Size
FAD uses one-second embedding windows extracted at a step length selected for the evaluation. Reducing the step length generally changes scores little, so highly overlapping windows are usually unnecessary.
- VGGish requires one-second audio inputs, so continuous audio is partitioned into windows or sampled with a moving window every t seconds.
- The evaluated step-length results show that FAD scores change very little as the step length decreases.
- Highly overlapping segments are generally unnecessary for estimating FAD scores.
B Evaluation Set Size
FAD stability depends on the evaluation-set size because it estimates distances between multivariate Gaussians from embedding sets. About 300 clips produce usable results, while approximately 5,000 clips provide an ideal stable estimate.
- FAD can be strongly affected by embedding-set size because smaller sets yield less accurate multivariate Gaussian estimates.
- The experiment varied evaluation sets from 100 to 10,000 five-second clips and repeatedly recomputed FAD to measure variance.
- The index of dispersion D = σ^2/µ normalizes each distortion configuration’s variance by its mean.
- 100 clips, or 8 minutes 20 seconds of audio, are insufficient for a stable FAD score.
- About 5,000 clips, or around 7 hours of audio, would be ideal, while roughly 300 clips, or 25 minutes, provide usable results.
C Evaluated Distortion Parameter Configurations
The section catalogs the distortion configurations examined and identifies the subset evaluated by humans, alongside their worth, SDR, and FAD measurements.
- Table 1 lists all examined distortions and their parameter configurations.
- Table 2 lists the human-evaluated distortions and their parameter configurations.
- Table 2 reports each human-evaluated distortion’s Plackett-Luce worth value, SDR, and FAD scores.