Source-linked AI summary
Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound
Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Le, Nick Zacharov, Carleigh Wood, Ann Lee, Wei-Ning Hsu
TL;DR
Audio aesthetics are subjective, inconsistently measured, and difficult to assess with existing metrics across speech, music, and sound. The paper introduces four-axis annotation and trains non-intrusive per-item predictors, which support more nuanced assessment and improve generation quality when used as prompts while maintaining alignment metrics.
Problem
Audio aesthetics are subjective and costly to measure manually, while existing methods can be domain-specific, reference-dependent, or ambiguous and noisy.
Method
The paper defines four aesthetic axes, annotates speech, music, and sound, and trains Transformer-based predictors that output four per-item aesthetic scores.
Results
Adding aesthetic scores as prompts improved generation quality while maintaining the alignment metric across downstream TTS, TTA, and TTM tasks.
Takeaways & Limitations
Audiobox-Aesthetics provides open-source predictors and AES-Natural labels for automated audio-quality assessment and future research.
Takeaways & Limitations
Existing orthogonal quality measurements remain domain-specific and cannot be calibrated across different domains.
Abstract
from arXiv · showhide
The quantification of audio aesthetics remains a complex challenge in audio processing, primarily due to its subjective nature, which is influenced by human perception and cultural context. Traditional methods often depend on human listeners for evaluation, leading to inconsistencies and high resource demands. This paper addresses the growing need for automated systems capable of predicting audio aesthetics without human intervention. Such systems are crucial for applications like data filtering, pseudo-labeling large datasets, and evaluating generative audio models, especially as these models become more sophisticated. In this work, we introduce a novel approach to audio aesthetic evaluation by proposing new annotation guidelines that decompose human listening perspectives into four distinct axes. We develop and train no-reference, per-item prediction models that offer a more nuanced assessment of audio quality. Our models are evaluated against human mean opinion scores (MOS) and existing methods, demonstrating comparable or superior performance. This research not only advances the field of audio aesthetics but also provides open-source models and datasets to facilitate future work and benchmarking. We release our code and pre-trained model at: https://github.com/facebookresearch/audiobox-aesthetics
1 Introduction
Audio aesthetics are subjective and difficult to measure consistently, while existing quality metrics are often intrusive, domain-specific, or too coarse. The paper proposes four-axis, non-intrusive predictors trained across speech, music, and sound to provide more detailed automated assessment.
- Human-listener evaluations of audio quality are inconsistent and resource-intensive because aesthetics depend on perception and cultural context.
- Existing metrics include reference-dependent speech measures and statistic-based audio measures that cannot provide universal utterance-level assessment.PESQ and POLQA require ground-truth speech, while FAD compares reference and generated-audio embeddings but does not predict utterance-level quality.
- Overall quality scores such as MOS are ambiguous, noisy, and provide limited insight into which audio characteristics cause lower scores.The cited biases include corpus effects, range-equalizing bias, self-preference, and sensitivity to other samples in the test batch.
- The paper introduces four-axis aesthetic scores and trains four non-intrusive, utterance-level predictors for arbitrary audio.The training data spans speech, music, and sounds, and the authors release AES labels for AES-Natural alongside open-source predictors.
- The proposed predictors are evaluated against standard MOS and previous methods to assess their effectiveness across audio domains.
2 Aesthetics data annotation
The annotation framework decomposes audio aesthetics into four axes rated on a 1-to-10 scale, using diversified, normalized data and qualified raters. The resulting corpus covers roughly 500 hours and 97k samples, whose axis scores are generally weakly correlated.
- Annotation guidelines: MOS-like guidelines are vague because a single score combines technical quality and subjective preference, making results difficult to interpret.
- Annotation guidelines: The annotation scheme defines Production Quality, Production Complexity, Content Enjoyment, and Content Usefulness as separate aesthetic axes.The axes distinguish technical quality, scene complexity, subjective enjoyment, and usefulness as source material for content creation.
- Annotation procedure: Raters score all four axes from 1 to 10 using detailed guidelines and example audio samples with score guidance.
- Dataset construction: Samples span speech, sound effects, and music, with stratified metadata sampling, cross-modality shuffling, loudness normalization, and three ratings per item.
- Rater qualification: 158 raters were recruited after qualification requiring Pearson correlation > 0.7 with expert-labeled scores on production axes.
- Dataset scale and visualization: Approximately 500 hours and 97k audio samples were annotated, evenly split across speech, sound, and music.Most samples are 10–30 seconds long; Figure 1 reports percentages in score buckets by axis and modality.
- Axis relationships: Scores across different axes are usually not strongly correlated, supporting the need to decouple aspects of overall quality evaluation.Figure 2 reports Pearson correlations between the aesthetic axes.
3 Aesthetic score predictor model
Audiobox-Aesthetics uses a Transformer-based audio encoder to produce utterance-level aesthetic predictions across four axes. During inference, it applies the model to 10-second sliding windows and combines window predictions with length-based weights.
- The model uses a WavLM-based encoder with 12 Transformer layers and 768-dimensional hidden states.
- Learnable layer weights aggregate hidden representations across Transformer layers and sequence timesteps into one audio embedding.The embedding is projected through MLP blocks containing linear layers, layer normalization, and GeLU activation.
- Audiobox-Aesthetics outputs predicted aesthetic scores across four axes and trains against targets using combined MAE and MSE losses.
- Figure 3 summarizes the architecture together with its input and output types.
- During inference, the system predicts scores on 10-second sliding windows and computes their weighted average.Weights are proportional to each window’s length, so shorter final windows contribute less.
4 Objective evaluation
The evaluation compares Audiobox-Aesthetics predictors with established quality models across speech, sound, and music, using human annotations and correlations. Results show comparable speech performance, meaningful AES–OVL relationships, and a distinct production-complexity signal.
- Experimental settings: The speech evaluation compares three speech-specific predictors and the general PAM predictor against four independently trained Audiobox-Aesthetics models.Baselines include DNSMOS P.808, SQUIM-PESQ, UTMOSv2, and PAM; the AES models target production quality, production complexity, content enjoyment, and content usefulness.
- Speech: Audiobox-Aesthetics-PQ, -CE, and -CU achieve comparable performance to top speech-focused predictors on utterance-level PCC and system-level SRCC, including Chinese OOD speech.Audiobox-Aesthetics-PC does not correlate with speech quality, consistent with production complexity depending on the number of modalities.
- Sound and music: In music, GT-PQ, GT-CE, and GT-CU correlate highly with overall quality, whereas only GT-CE is strongly related to overall quality in sound.These patterns support using multiple axes because overall quality does not comprehensively capture production quality, enjoyment, usefulness, and related dimensions.
- Sound and music: Predicted scores follow the human-annotation pattern: each system is most related to its training target, while PAM-predicted scores strongly track sound and music enjoyment.Audiobox-Aesthetics models perform slightly worse than PAM in music, possibly because they were not trained with synthetic data.
- AES-Natural: AES-Natural contains 2950 audio samples with four-axis labels across 950 speech, 1000 music, and 1000 sound samples.The corpus spans varied audio types, qualities, and sampling rates, totals around 11.2 hours, and uses 10 trained annotators per sample.
- AES-Natural: Production complexity is decoupled from the other three AES axes, and its predictor achieves high accuracy, providing a distinct view of audio quality.The content-usefulness predictor remains the best choice for usefulness in an audio-type-agnostic scenario, while baselines show significantly worse robustness to unseen data.
5 Downstream tasks
The downstream experiments use Audiobox-Aesthetics-PQ to filter data or prompt generation across text-to-speech, text-to-music, and text-to-audio tasks. Prompting consistently improves subjective audio quality while preserving alignment better than filtering.
- Filtering strategy: Filtering removes samples below the 25th or 50th percentile of predicted aesthetic scores.The strategy retains only higher-scoring training data.
- Prompting strategy: Prompting adds rounded aesthetic scores to training prompts and uses fixed percentile-based quality strings during inference.Training explores rounding factors r = 2 and r = 5; inference uses the 50th, 75th, or 90th percentile.
- Experimental setup: Audiobox-Aesthetics-PQ predicts audio quality for filtering and prompting strategies across text-to-speech, text-to-music, and text-to-audio generation.Each scenario is evaluated across all three downstream tasks.
- Evaluation: Pairwise subjective evaluation samples 200 generated audios per test set, uses three listeners per pair, and reports bootstrap 95% confidence intervals.The intervals come from 1,000 bootstrap resamples.
- Results: Both filtering and prompting produce better audio quality than the baseline in every evaluated task and scenario.This result is reported from the pairwise comparisons in Table 9.
- Results: Prompting is preferred to filtering because filtering reduces training data and harms alignment metrics, whereas prompting preserves similar alignment to baseline.Subjective evaluation favors prompting over filtering across tasks.
6 Conclusion
The paper introduces audio aesthetic scoring across speech, music, and sound, using factorized axes and trained models. Downstream experiments show that prompting with aesthetic scores improves generation quality while maintaining alignment metrics.
- 6 Conclusion: The paper presents an approach to measuring aesthetic scores for speech, music, and sound.It factorizes aesthetic scores into different axes to reduce ambiguity relative to standard MOS scores.
- 6 Conclusion: Adding aesthetic scores as prompts improves generation quality while maintaining the alignment metric in TTS, TTA, and TTM tasks.The conclusion identifies these downstream applications as evidence for the approach’s utility.
A Aesthetics score annotation UI
Figure 4 shows the aesthetic score annotation UI.
- A Aesthetics score annotation UI: Figure 4 presents the aesthetic score annotation UI.The figure is identified as an interface for aesthetic score annotation.
- A Aesthetics score annotation UI: The figure is labeled as an annotation interface rather than an evaluation result.No quantitative outcome is stated in the figure passage.
- A Aesthetics score annotation UI: Use Figure 4 to inspect how aesthetic-score annotation is presented to annotators.The passage identifies the figure’s purpose but does not describe individual controls.
B Aesthetics score annotation guidelines
Tables 10 and 11 contain the audio aesthetics score annotation guidelines.
- A Aesthetics score annotation guidelines: Table 10 presents the audio aesthetics score annotation guidelines.The caption identifies the table’s content but does not specify individual axes or criteria.
- A Aesthetics score annotation guidelines: Together, Tables 10 and 11 should be read as the paper’s annotation-guideline material.The captions identify both tables as parts of the same guideline content.
- A Aesthetics score annotation guidelines: Table 11 continues the audio aesthetics score annotation guidelines.The continuation caption indicates that the guidelines span multiple table parts.