Source-linked AI summary
The Jazz Transformer on the Front Line: Exploring the Shortcomings of AI-composed Music through Quantitative Measures
Shih-Lun Wu, Yi-Hsuan Yang
TL;DR
Transformer music models may produce compelling or coherent pieces, yet the authors report that generated music still falls short of human compositions. The paper applies a Transformer to Jazz with structural events and evaluates its output using music analyses and continuation prediction; results identify persistent structural and musical deficiencies, with some improvement from structure-related events.
Problem
Despite claims of compelling or coherent machine-generated music, the authors report that it still falls behind human compositions and seek to identify what goes wrong.
Source: §1 · p. 1Method
The paper trains a Transformer on WJazzD Jazz lead sheets with structure-related events and assesses generated music through event analyses and a continuation-prediction challenge.
Source: §1 · p. 1, §5.5 · p. 5, §7 · p. 6Results
Model (B) at loss 0.25 was closest to real data on most metrics, but generated music showed erratic events and lacked medium- and long-term repetitive structures.
Source: §6 · p. 6Takeaways & Limitations
The proposed metrics identify erratic pitch use, inconsistent rhythm and harmony, and missing repetitive structures, and offer quantitative measures for assessing future composition models.
Source: §7 · p. 6The continuation challenge treats high accuracy as evidence of sufficient overall Jazz understanding to distinguish correct from incorrect continuations.
Source: §5.5 · p. 5
Abstract
from arXiv · showhide
This paper presents the Jazz Transformer, a generative model that utilizes a neural sequence model called the Transformer-XL for modeling lead sheets of Jazz music. Moreover, the model endeavors to incorporate structural events present in the Weimar Jazz Database (WJazzD) for inducing structures in the generated music. While we are able to reduce the training loss to a low value, our listening test suggests however a clear gap between the average ratings of the generated and real compositions. We therefore go one step further and conduct a series of computational analysis of the generated compositions from different perspectives. This includes analyzing the statistics of the pitch class, grooving, and chord progression, assessing the structureness of the music with the help of the fitness scape plot, and evaluating the model's understanding of Jazz music through a MIREX-like continuation prediction task. Our work presents in an analytical manner why machine-generated music to date still falls short of the artwork of humanity, and sets some goals for future work on automatic composition to further pursue.
1. INTRODUCTION
The paper extends Transformer-based music generation to Jazz, using annotated structure in WJazzD while addressing technical challenges in representing its notes and chords. It also aims to examine how generated music falls short through objective metrics.
1. INTRODUCTION
- Using the Weimar Jazz Database (WJazzD) offers annotated structural events for Jazz composition, but requires handling short-note quantization and complex chord representations.The dataset differs from commonly used piano MIDIs.Source: §1 · p. 1
- The paper targets a gap between claims of compelling, coherent Transformer-generated music and evidence that machine compositions still lag humans, then asks what goes wrong.The authors point to their user study as evidence of the gap.Source: §1 · p. 1
- The paper aims to generate Jazz melody, chord progressions, and structure together, and to assess pitch use, rhythm, chord consistency, and structureness with objective metrics.The authors intend these analyses to identify problems in the generated music.Source: §1 · p. 1
2. RELATED WORK
Prior work spans computational analysis of Jazz performance, Transformer-based music generation, and objective evaluation of generated music. The paper positions its contribution against limited Jazz-composition work and metrics focused mainly on surface statistics.
2. RELATED WORK
- Transformer music models have used relative positional encoding, cross-domain pre-training, and event-token design; this work describes itself as the first Transformer attempt focused exclusively on Jazz composition.Prior work also investigated RNN-based lead-sheet composition and structure induction.Source: §2 · p. 2
- Earlier Jazz-composition efforts included a GAN trained on 44 lead sheets and an RNN variational auto-encoder for generating melody and chords.The GAN dataset comprised approximately 1,700 bars.Source: §2 · p. 2
- Most existing composition metrics emphasize surface statistics, while structureness indicators and a MIREX-like metric add piece-level structure and genre-understanding assessments.Computational studies of human Jazz performance include work using WJazzD to examine tonal-complexity evolution.Source: §2 · p. 2
3. THE JAZZ TRANSFORMER
The Jazz Transformer models Jazz lead sheets as event-token sequences using annotated WJazzD data, with note, meter, chord, and structural events. Transformer-XL addresses the memory and context-fragmentation challenges of processing long solos.
3.2 Data Representation
- The model represents each composition as an event-token sequence, extending a MIDI-derived representation to include WJazzD chord and structural events.Events are grouped into note-, metric-, chord-, and structure-related categories.Source: §3 · p. 2, §3.2 · p. 2
3.1 Dataset
- WJazzD contains 456 monophonic Jazz solos; restricting the data to 4/4 leaves 431 solos, with 5% held out for validation.The lead sheets include melody and beat tracks, with human-annotated and cross-checked notes, chords, metrical markings, and structure.Source: §3.1 · p. 2
3.2.1 Note-related Events
- Each melody note uses NOTE-VELOCITY, NOTE-ON, and NOTE-DURATION events; the 64th-note quantum covers 99.6% of notes.NOTE-DURATION values span 1–32 multiples of a 64th note; shorter notes are discarded and longer-than-half-note durations are clipped.Source: §3.2.1 · pp. 2–3, §3.2.3 · p. 3
3.2.2 Metric-related Events
- BAR and POSITION events represent bar and onset locations, while TEMPO-CLASS and TEMPO encode general speed and exact pace.Each bar has 64 subunits, and tempo events occur at every beat position.Source: §3.2.2 · p. 3
3.2.3 Chord-related Events
- The chord system decomposes 418 WJazzD chords into CHORD-TONE, CHORD-TYPE, and CHORD-SLASH events, reducing the unique chord-related events to 71.Together with manually constructed key templates, these events can represent the original chords and map them to notes for MIDI conversion.Source: §3.2.3 · p. 3
3.2.4 Structure-related Events
- PHRASE, MLU, PART, and REPETITION events mark phrase starts, midlevel units, and form-part boundaries to guide phrase breathing, short-term patterns, and repetition.PART and REPETITION events are placed at the beginning and end of form parts.Source: §3.2.4 · p. 3
3.3 Model and Training Setups
- Transformer-XL is used because its cached segment computations address memory constraints and context fragmentation when processing long solos.The two variants differ in whether they use structure-related events; both have 12 layers, 8 attention heads, and about 41 million parameters.Source: §3.3 · pp. 3–4
4. SUBJECTIVE STUDY
The blind listening study compared generated and real one-minute pieces across four rating aspects. Participants rated the generated compositions below the real pieces in every aspect, with statistically significant mean gaps.
4. SUBJECTIVE STUDY
- The blind test presented four one-minute pieces: two generated by Model (B) at loss 0.25 and two from real data.Model (A) was excluded to reduce participant burden, with the assumption that Model (B) was better.Source: §4 · p. 4
- The Jazz Transformer scored below human compositions on all four aspects, especially overall quality and structureness; one-tailed Z-tests found significant gaps for every aspect (p < 0.05).The aspects were overall quality, impression, structureness, and richness.Source: §4 · p. 4
5. OBJECTIVE EVALUATION METRICS
The section introduces objective measures for examining generated Jazz from event distributions, repeated structure, and continuation prediction. These metrics are intended to help investigate why generated compositions remain inferior to human-created music.
5. OBJECTIVE EVALUATION METRICS
- The subjective-study gaps between Model (B) and real WJazzD pieces were statistically significant for overall quality, impression, structureness, and richness (p < 0.05).The figure compares from-scratch compositions generated with structure-related events against real pieces.Source: Figure 2 · p. 4
5.1 Pitch Class Histogram Entropy
- Pitch-class entropy assesses tonality: a histogram dominated by a few pitch classes has low entropy, while scattered pitch use yields high entropy.The histogram is normalized across the 12 pitch classes for a period such as a bar.Source: §5.1 · p. 4
5.2 Grooving Pattern Similarity
- Grooving-pattern similarity compares binary vectors of note-onset positions within bars; higher similarity indicates more consistent rhythm, while lower similarity indicates more erratic rhythm.In this setting, each pattern is a 64-dimensional vector.Source: §5.2 · pp. 4–5
5.3 Chord Progression Irregularity
- Chord progression irregularity is the percentage of unique three-chord sequences in a piece; Jazz progressions built from repeated templates should not have very high irregularity.Two chord trigrams count as different if any chord differs.Source: §5.3 · p. 5
5.4 Structureness Indicators
- The fitness scape plot represents segment repetition as fitness values in a matrix, and the structureness indicator captures the strongest repeat within a selected duration interval.The paper uses short-, medium-, and long-term indicators to examine different duration ranges.Source: §5.4 · p. 5
5.5 MIREX-like Continuation Prediction Challenge
- At loss 0.25, Model (A) and Model (B) each reached their highest continuation-prediction accuracy: 80% and 83%, respectively.Each checkpoint answered 100 questions.Source: Figure 3 · p. 5
- The continuation challenge gives the model an eight-bar prompt and four eight-bar options, one correct and three randomly drawn from other pieces; it selects the option with the highest average generation probability.High accuracy is treated as evidence of enough overall Jazz understanding to distinguish correct from incorrect continuations.Source: §5.5 · p. 5
6. EXPERIMENT RESULTS AND DISCUSSIONS
Objective evaluations identify erratic musical events and a shortage of medium- and long-term repetition in generated compositions. Model (B) at loss 0.25 is closest to human compositions on most metrics, while performance worsens at overly low loss levels.
MIREX-like challenge
- MIREX-like continuation accuracy rises as training loss falls, peaks at loss 0.25, then drops at lower loss, suggesting possible overfitting.Both Model (A) and Model (B) show this trend across five checkpoints, each answering 100 multiple-choice questions drawn from held-out validation data.Source: §6 · p. 5
Objective evaluation
- Model (B) at loss 0.25 is closest to real data on most objective metrics, while its performance plunges at loss 0.1.The comparison includes Model (A) and other Model (B) checkpoints; short-timespan metrics H1, H4, and SI8 show a substantial lead over Model (A).Source: §6 · p. 6
- The comparison used 50 from-scratch, 32-bar compositions per checkpoint against 409 pieces in the training dataset.Objective evaluations used checkpoints at loss 0.8, 0.25, and 0.1 for Model (B), and 0.8 and 0.25 for Model (A).Source: §6 · p. 6
- Generated compositions show erratic musical events and lack medium- and long-term repetitive structures compared with real data.Higher H1 and H4 indicate unstable local pitch use; lower GS and higher CPI indicate less consistent rhythm and harmony, while SI15 indicators are significantly lower.Source: §6 · p. 6
- Model (B)’s performance worsens when training loss falls to an overly low level, consistent with the MIREX-like accuracy trend.The objective-evaluation comparison reports a plunge at loss 0.1 after the stronger performance at loss 0.25.Source: §6 · p. 6
Structureness
- In fitness scapes, Model (B)’s highest-scoring composition loses repetition traces beyond 10 seconds, unlike the human piece’s structures at 20–30 seconds.The selected compositions received structureness scores of 3.14 for Model (B) and 3.54 for the human piece; the human scape also shows two large section-like patterns.Source: §6 · p. 6, Figure 4 · p. 6
7. CONCLUSION AND FUTURE WORK
The paper presents the Jazz Transformer and objective metrics that expose shortcomings in generated music, including erratic pitch use, inconsistent rhythm and harmony, and missing repetition. Future work will examine links between these metrics and subjective ratings and pursue structure-inducing approaches.
Conclusion
- The Jazz Transformer incorporates structure-related events, and the proposed metrics identify erratic pitch use, inconsistent grooving and chord progression, and absent repetitive structures.The metrics are intended to support quantitative assessment alongside human evaluation, which the paper says has been relied on heavily.Source: §7 · p. 6
Future work
- Future work will study correlations between quantitative metrics and subjective evaluations and continue investigating ways to induce structure in machine-composed music.Possible directions include redesigning the Transformer to read structural information such as a fitness scape plot before composing at finer scales.Source: §7 · p. 6