Source-linked AI summary
The GTZAN dataset: Its contents, its faults, their effects on evaluation, and its future use
Bob L. Sturm
TL;DR
GTZAN’s widespread use and integrity faults raise a central question: how interpretable and comparable are MGR evaluations based on it? The paper catalogs and analyzes its contents, surveys its use, and tests five MGR systems, finding that faults affect systems differently and can reverse comparisons. It concludes that GTZAN remains useful only when its contents are explicitly considered.
Problem
GTZAN is widely used for MGR evaluation, but repetitions, mislabelings, distortions, and limited scrutiny constrain the validity of conclusions drawn from it.
Method
The paper catalogs GTZAN’s contents and faults, surveys published uses, and experimentally evaluates five MGR systems under different fault conditions.
Results
Faults affect MGR systems by different magnitudes and can change statistically supported system rankings; removing repetitions changes NN’s normalized accuracy from about 0.6 to 0.63 and reduces MAPsCAT’s ST-to-AF accuracy by 25 points.
Takeaways & Limitations
GTZAN should not be banished, but evaluations must account for its musical content and faults when drawing conclusions.
Takeaways & Limitations
The paper is limited to GTZAN and does not make general conclusions about other datasets, MGR, or the possibility of creating faultless datasets.
Abstract
from arXiv · showhide
The GTZAN dataset appears in at least 100 published works, and is the most-used public dataset for evaluation in machine listening research for music genre recognition (MGR). Our recent work, however, shows GTZAN has several faults (repetitions, mislabelings, and distortions), which challenge the interpretability of any result derived using it. In this article, we disprove the claims that all MGR systems are affected in the same ways by these faults, and that the performances of MGR systems in GTZAN are still meaningfully comparable since they all face the same faults. We identify and analyze the contents of GTZAN, and provide a catalog of its faults. We review how GTZAN has been used in MGR research, and find few indications that its faults have been known and considered. Finally, we rigorously study the effects of its faults on evaluating five different MGR systems. The lesson is not to banish GTZAN, but to use it with consideration of its contents.
1 Introduction
GTZAN is the most-used public MGR evaluation dataset, yet its repetitions, mislabelings, and distortions have been insufficiently examined. This article catalogs those faults, surveys their use, tests their effects across MGR systems, and argues for careful rather than wholesale abandonment of GTZAN.
- Dataset use and contents: GTZAN appears in evaluations of 100 published MGR works and contains 1,000 half-minute excerpts labeled in ten categories.The dataset was not expressly created for MGR and historically lacked metadata identifying its contents.
- Dataset use and contents: Few researchers critically evaluated GTZAN’s contents, allowing its faults to remain undiscovered since its creation in 2002.
- Article contributions: The article extends prior work by cataloging GTZAN’s repetitions, mislabelings, and distortions, surveying its literature use, and measuring effects on MGR evaluation.
- Article contributions: The authors survey 100 GTZAN-using works, find few indications that its faults were considered, and measure artist repetition’s effect on MGR evaluation.
- Future use: GTZAN should not be discarded, but used with consideration of its contents and faults, which can support evaluation in music machine listening tasks.
- Article contributions: The study evaluates how GTZAN faults affect several MGR systems and estimates upper bounds for figures of merit of a “perfect” system.
- Scope and motivation: The article is delimited to GTZAN’s composition, faults, historical and contemporary MGR use, evaluation effects, and considered future use.
2 Analysis of GTZAN
The authors analyze GTZAN’s contents through metadata and user-applied tags, then formalize three dataset faults: repetitions, mislabelings, and distortions. Their analysis shows that category composition and labeling do not always align with genre assumptions, while repeated artists, recordings, and versions create additional structural concerns.
- Describing excerpts and categories: Top tags are defined as the tags contributing the majority of an excerpt’s total last.fm tag count.For example, “blues” and “blues guitar” account for 94.6% of the example’s total tag count.
- Describing excerpts and categories: GTZAN categories have broader and more varied meanings than their genre names alone suggest.The authors note that GTZAN Blues extends beyond canonical blues artists, while GTZAN Disco includes broader dance music from the seventies.
- Identifying faults: Repetitions: GTZAN contains exact, recording, artist, and version repetitions, including 50 exact and 21 recording repetitions.The authors also identify 13 version repetitions and find artist repetition across every category.
- Identifying faults: Repetitions: Artist repetition is substantial: all 100 Blues excerpts come from nine artists, while more than one-third of Reggae excerpts come from Bob Marley.These repetitions are identified using metadata and confirmed or characterized through the authors’ analysis.
- Identifying faults: Mislabelings: Mislabelings are formalized by comparing an excerpt’s tag profile with the top tags of its assigned and alternative categories.An excerpt is considered mislabeled when its assigned-category score is too low or another category’s score is sufficiently higher.
3 GTZAN in MGR Research
GTZAN has become central to MGR evaluation, appearing in 100 works, but studies use varied designs and rarely document listening for dataset faults. Reported classification accuracies are therefore difficult to interpret, especially because several results have been challenged or shown incorrect.
- Dataset prevalence: 100 MGR works use GTZAN, including 49 that use no other dataset, making it the field’s most-used public benchmark.GTZAN appears in more published MGR evaluations than any other publicly available dataset.
- Dataset scrutiny: Only five of the 100 works indicate that researchers listened to at least some GTZAN excerpts, while very few mention specific faults.Reported observations include repetitions, duplicates, and corrupted files, but systematic fault analysis is rare in the reviewed literature.
- Experimental designs: 96 of 100 GTZAN studies use Classify, usually reporting classification accuracy from stratified k-fold cross-validation, most often 10fCV.Other studies use Retrieve, Cluster, or Compose, and some use data splits, subsets, or augmented versions of GTZAN.
- Interpretation of results: About half of GTZAN studies use no other dataset, limiting conclusions about system performance on other datasets or in the real world.Most evaluations focus on classification accuracy as the figure of merit for comparing genre-recognition systems.
- Interpretation of results: Six reported 10-class classification results are incorrect or challenged, and all plotted results omit an artist filter, likely making them optimistic.The challenged results include studies affected by experimental mistakes, contradictions with other findings, or implausible system performance.
4 The faults of GTZAN and evaluation
The evaluation study tests how GTZAN’s repetitions, artist overlap, and other faults affect five MGR systems under different partitioning schemes. The results show that these faults affect systems unequally and can change both measured performance and conclusions about which system is better.
- 4.1 Method: The study evaluates nearest neighbor, minimum distance, minimum Mahalanobis distance, SRCAM, and MAPsCAT using common features or state-of-the-art feature representations.The classifiers are tested with four partitioning conditions: standard and artist-filtered cross-validation, each with and without identified repetitions and distortions.
- 4.2 Experimental results and discussion: GTZAN faults affect systems differently: removing repetitions reduces nearest neighbor’s performance, while minimum Mahalanobis distance is not similarly affected.Nearest neighbor’s mean normalized accuracy rises from about 0.6 to 0.63 when repetitions remain in the evaluation.
- 4.2 Experimental results and discussion: Artist filtering causes large performance decreases for all five systems, ranging from 7 normalized-accuracy points for minimum distance to 25 points for MAPsCAT.The artist-filtered partitions prevent artist overlap between training and test sets, exposing the optimistic effect of non-filtered evaluation.
- 4.2 Experimental results and discussion: System rankings depend on the partitioning: standard evaluation ranks SRCAM highest and minimum distance lowest, whereas artist-filtered evaluation changes the significant comparisons.MAPsCAT is significantly better than most systems under standard evaluation but is not significantly distinct from minimum distance or minimum Mahalanobis distance under artist filtering.
- 4.2 Experimental results and discussion: For SRCAM, removing repetitions and applying artist filtering substantially reduce class-level and overall metrics, including a 22-point, 28% decrease in estimated classification accuracy.Classes with more repetitions show larger recall changes, and several class F-scores decrease by at least 10%.
- 4.2 Experimental results and discussion: Correcting 59 suspected mislabelings produces further deterioration in SRCAM’s figures of merit and undermines an apparent ability to recognize classical music.The resulting confusions indicate that the apparent classical-genre strength is not supported by the corrected analysis.
5 Conclusions on the future use of GTZAN
GTZAN can still support research, but only when evaluations account for its musical contents and faults. Uncontrolled variables and limited size constrain what conclusions can be drawn from results.
- Future use of GTZAN: GTZAN cannot reliably determine which systems reproduce its labels or recognize real-world genres when its faults are considered.All 96 systems in Fig. 4 were evaluated using GTZAN in ST, preventing meaningful judgments of comparative performance or real-world genre recognition.
- Future use of GTZAN: Valid GTZAN evaluation depends on experimental designs that test whether system decisions and behaviors relate to musical content, not merely on selecting a figure of merit.The authors argue that analysis must account for GTZAN’s music and determine whether system behavior reflects the content behind its labels.
- Scope and limitations: 1,000 excerpts may be an insufficient random sample for learning genre characteristics, although larger datasets can contain similar faults.The paper treats GTZAN’s manageable, public, and well-studied status as useful despite this size limitation.
- Scope and limitations: No valid conclusion can come from an evaluation using uncontrolled independent variables, regardless of dataset size.The authors specifically identify Classify in GTZAN as an invalid evaluation under this condition.