Source-linked AI summary
SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
Jixun Yao, Guobin Ma, Huixin Xue, Huakang Chen, Chunbo Hao, Yuepeng Jiang, Haohe Liu, Ruibin Yuan, Jin Xu, Wei Xue, Hao Liu, Lei Xie
TL;DR
Evaluating song aesthetics remains challenging because musical appreciation is subjective, multidimensional, and not fully captured by objective metrics. SongEval provides a large, expert-annotated benchmark of full-length songs across five aesthetic dimensions. Models trained on SongEval outperform existing objective audio metrics in predicting human-perceived musical aesthetics.
Problem
Song aesthetic evaluation remains underexplored because musical aesthetics are subjective and multidimensional, while existing objective metrics have limited ability to represent them.
Source: §1 · pp. 1–2Method
SongEval is an open-source dataset of full-length songs with professional aesthetic ratings across five dimensions, including coherence, memorability, vocal phrasing naturalness, structural clarity, and musicality.
Source: §1 · p. 2, §3.2 · p. 5, §3.3 · p. 5Results
Models trained on SongEval outperform existing objective audio metrics in predicting human-perceived musical aesthetics.
Source: §6 · p. 9Takeaways & Limitations
SongEval provides a benchmark resource for evaluating and comparing song-generation models using professionally informed, multidimensional song-level aesthetic ratings.
Source: §1 · p. 2The five aesthetic dimensions may overlap, making absolute separation between factors challenging despite annotation guidelines and expert training.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Aesthetics serve as an implicit and important criterion in song generation tasks that reflect human perception beyond objective metrics. However, evaluating the aesthetics of generated songs remains a fundamental challenge, as the appreciation of music is highly subjective. Existing evaluation metrics, such as embedding-based distances, are limited in reflecting the subjective and perceptual aspects that define musical appeal. To address this issue, we introduce SongEval, the first open-source, large-scale benchmark dataset for evaluating the aesthetics of full-length songs. SongEval includes over 2,399 songs in full length, summing up to more than 140 hours, with aesthetic ratings from 16 professional annotators with musical backgrounds. Each song is evaluated across five key dimensions: overall coherence, memorability, naturalness of vocal breathing and phrasing, clarity of song structure, and overall musicality. The dataset covers both English and Chinese songs, spanning nine mainstream genres. Moreover, to assess the effectiveness of song aesthetic evaluation, we conduct experiments using SongEval to predict aesthetic scores and demonstrate better performance than existing objective evaluation metrics in predicting human-perceived musical quality.
1 Introduction
Song generation increasingly produces full musical pieces, but judging their aesthetic quality remains difficult because musical appreciation is subjective and multidimensional. SongEval addresses this challenge with professionally annotated, multidimensional human ratings for song evaluation.
1 Introduction
- Full-length song generation must integrate vocals and accompaniment coherently, whereas many earlier studies focus on generating only one component.Recent systems generate full-length songs combining vocals and accompaniment, drawing increased interest from industry and academia.Source: §1 · pp. 1–2
- Objective metrics such as spectrogram distance, pitch accuracy, and embedding similarity capture signal-level or structural fidelity but miss subjective qualities such as emotional expression and musicality.The passage identifies this mismatch as a gap in song-generation evaluation.Source: Figure 1 · p. 2
- SongEval is introduced as an open-source dataset of professionally annotated songs, with ratings across five aesthetic dimensions.The dimensions are coherence, memorability, vocal breathing and phrasing naturalness, song-structure clarity, and overall musicality; the authors frame the ratings as expert consensus rather than personalized taste.Source: §1 · p. 2
2 Related Work
Subjective evaluation resources for music and audio remain limited, and existing datasets described here do not cover full-length songs with vocals and fine-grained aesthetic ratings. SongEval is presented as addressing that need.
2 Related Work
- Subjective datasets for music and audio remain limited, despite the development of human-annotated MOS datasets and prediction toolkits for speech synthesis.The speech-domain examples are cited as [21] [22] and [23] [24].Source: §2 · p. 3
- Table 1 compares SongEval with other subjective evaluation datasets, but the supplied caption does not report the table’s comparisons or values.The related-work discussion separately describes limitations of MusicEval and AES-Natural.Source: Table 1 · p. 3, §2 · p. 3
- MusicEval focuses on accompaniment generation, with about 16 hours of data and ratings for musical impression and prompt alignment, while AES-Natural spans 29 hours across speech, audio, and music but uses short music clips and three basic dimensions.The passage identifies the absence of full-length vocal songs and fine-grained aesthetic dimensions as limitations motivating SongEval.Source: §2 · p. 3
3 SongEval Dataset
SongEval is a benchmark dataset of full-length songs with expert aesthetic ratings. The section describes its collection pipeline, annotation dimensions, and dataset statistics.
3.1 Data Collection
- The collection pipeline generates lyrics and genre-aligned prompts, then uses five song-generation models to produce full-length songs, with both lyric-prompt and genre-only conditioning strategies.The prompts cover nine genres, and the lyrics and prompts include English and Mandarin; vocal range is used to remove low-quality outputs.Source: §3.1 · p. 4
3.2 Aesthetic Annotation
- Songs receive five-point ratings for coherence, memorability, vocal breathing and phrasing naturalness, song-structure clarity, and overall musicality.Each song is independently rated by four annotators with formal musical training; the dimensions cover continuity, memorable features, vocal delivery, structure, and listening enjoyment.Source: §3.2 · pp. 4–5
3.3 Dataset Statistics
- SongEval contains 2,399 full-length songs totaling approximately 140 hours, in English and Chinese and across nine common genres.Most Chinese songs last 2–6 minutes; English songs show a similar pattern, with some extending to 8 minutes.Source: §3.3 · p. 5
- The duration distribution is reported across languages and generation models, with DiffRhythm’s fixed 285-second output producing a concentration near four minutes.The dataset also includes a small “Other” subset of problematic cases and non-copyrighted real songs for reference and validation.Source: Figure 4 · p. 6, §3.3 · p. 6
4 Experimental Setup
The experiments train aesthetic-prediction models on SongEval, evaluate them on held-out songs, and assess prediction accuracy and ranking alignment with human ratings.
4.1 Datasets
- 2,199 songs were used for training, with 200 songs and 50 non-copyrighted real songs in the evaluation set.The evaluation set tests performance on unseen musical content.Source: §4.1 · p. 6
4.2 Models
- Four representative systems adapt speech and audio quality-assessment approaches to predict song aesthetic scores.The systems include MOSNet-based, LDNet-based, SSL-based, and UTMOS-based models.Source: §4.2 · pp. 6–7
4.3 Evaluation Metrics
- Four metrics assess alignment with human ratings: MSE for absolute error, and LCC, SRCC, and KTAU for correlation or ranking.Lower MSE indicates more accurate absolute predictions; SRCC evaluates ranking regardless of absolute values, while KTAU is more robust to ties and small rank differences.Source: §4.3 · p. 7
5 Experimental Results
SongEval-trained aesthetic predictors align more strongly with human ratings than the compared objective metrics, while SSL-based and UTMOS-based systems perform consistently well across aesthetic dimensions.
5.1 Aesthetic Evaluation
- SSL-based and UTMOS-based systems show consistently superior performance across dimensions, particularly coherence and structural clarity.Their score distributions are tighter and closer to human-annotated distributions across most dimensions; MOSNet-based and LDNet-based systems tend to produce narrower or biased distributions.Source: §5.1 · p. 8, Figure 5 · p. 8
- All evaluated models can reasonably predict multidimensional aesthetic scores, and the results support SongEval as a consistent training resource.The paper reports stable and interpretable scores across dimensions and styles and languages.Source: §5.1 · p. 8
5.2 Comparison with Other Objective Metrics
- The SongEval-trained UTMOS system correlates more strongly with human ratings than the compared objective metrics across all five dimensions, especially coherence and musicality.Table 4 reports Pearson correlations for musicality and the average across all aspects.Source: Table 4 · p. 9, §5.2 · p. 9
- The comparison treats objective metrics as complementary: Vocal Range detects singing but is less sensitive to memorability and phrasing naturalness.The paper also reports that Audiobox PC has significantly lower correlation across all five dimensions than the other Audiobox metrics.Source: §5.2 · p. 9
6 Conclusion
SongEval is presented as a benchmark for holistic song-aesthetic evaluation, with experiments showing that its trained models outperform existing objective audio metrics in predicting human-perceived aesthetics.
6 Conclusion
- SongEval contains 2,399 full-length songs totaling over 140 hours, with expert ratings across five dimensions, two languages, and nine genres.The ratings use a 1–5 scale and are provided by 16 professional annotators.Source: §6 · p. 9
7 Limitations and Future Work
The authors identify potential overlap among SongEval’s aesthetic dimensions and propose developing finer-grained automatic evaluation tools that generalize across styles, genres, and languages.
7 Limitations and Future Work
- The five dimensions may overlap in listener judgments, making absolute separation challenging despite annotation guidelines and expert training.The authors give coherence and structural clarity, and musicality and memorability, as possible pairs with high interdependence.Source: §7 · p. 9
- Future work aims to build more robust, fine-grained aesthetic predictors that generalize across musical styles, genres, and languages.Source: §7 · p. 9
A Supplementary Material
The supplemental material covers song characteristics, filtering, and annotation procedures.
A.1 We provide the characteristic details of the song generated by different systems.
- A.1 provides characteristic details of songs generated by different systems.Source: A Supplementary Material · p. 12
A.2. We describe the filter process of generated songs.
- A.2 describes the process used to filter generated songs.Source: A Supplementary Material · p. 12
A.3. We provide details of the annotation process.
- A.3 provides details of the annotation process.Source: A Supplementary Material · p. 12
A.1 Characteristic details of generated songs
The authors document audio-format characteristics across generation systems and standardize audio for downstream processing and model training.
A.1 Characteristic details of generated songs
- Audio formats were standardized and documented, with Table 5 summarizing each system’s sampling rate and channel configuration.Source: §A.1 · p. 12, Table 5 · p. 12
- YUE [16] produces mono-channel audio, which the authors duplicated to simulate stereo for uniform downstream processing and model training.Source: §A.1 · p. 12
A.2 Filter process of generated songs
Vocal-range analysis excludes generated samples without minimum vocal characteristics expected in sung performance.
A.2 Filter process of generated songs
- Samples lacking minimum expected singing characteristics, including instrumental-only tracks or speech-like readings, are excluded based on vocal-range analysis.The filter is intended to retain meaningful vocal content consistent with song structure and aesthetics.Source: §A.2 · p. 12
A.3 Details of annotation process
The annotation process used a web platform for full-song listening and five-dimension ratings, with trained annotators and third-party workflow oversight.
A.3 Details of annotation process
- A web platform combined audio playback with a visual spectrogram display for subjective annotation.Source: §A.3 · p. 12, Figure 6 · p. 13
- Annotators rated full songs across five aesthetic dimensions using sliders from 1 (very poor) to 5 (excellent).Source: §A.3 · p. 13
- An independent audio-annotation team managed the workflow, verified qualifications, monitored consistency, and trained annotators on the five criteria.Source: §A.3 · p. 13
- The process collected annotations for 2,399 songs, paying annotators $5 per song and incurring approximately $48,000 in song-generation and access costs.Source: §A.3 · p. 13