Source-linked AI summary
EMOPIA: A Multi-Modal Pop Piano Dataset For Emotion Recognition and Emotion-based Music Generation
Hsiao-Tzu Hung, Joann Ching, Seungheon Doh, Nabin Kim, Juhan Nam, Yi-Hsuan Yang
TL;DR
EMOPIA addresses the shortage of emotion-labeled symbolic music data by providing a multi-modal pop piano dataset with audio, MIDI, and clip-level emotion annotations. The authors construct the dataset and prototype emotion classification and conditioned generation models, showing high classification accuracy and some target-emotion control in generated music.
Problem
Existing emotion-labeled music datasets generally provide audio only, while available MIDI datasets are few and small, limiting symbolic-domain emotion research.
Method
The authors build EMOPIA from piano performances, manually segment 387 songs into 1,087 clips, annotate four emotion classes, and evaluate classification and emotion-conditioned generation models.
Results
EMOPIA supports high-accuracy four-quadrant and valence-wise classification, while a Transformer-based model generates music with a target emotion to a certain degree.
Takeaways & Limitations
EMOPIA provides a multi-modal, piano-focused resource for clip-level emotion analysis and emotion-based symbolic music generation.
Takeaways & Limitations
Audio is shared only through YouTube links because of copyright restrictions, so availability depends on country-specific licenses and whether owners remove recordings.
Abstract
from arXiv · showhide
While there are many music datasets with emotion labels in the literature, they cannot be used for research on symbolic-domain music analysis or generation, as there are usually audio files only. In this paper, we present the EMOPIA (pronounced `yee-mò-pi-uh') dataset, a shared multi-modal (audio and MIDI) database focusing on perceived emotion in pop piano music, to facilitate research on various tasks related to music emotion. The dataset contains 1,087 music clips from 387 songs and clip-level emotion labels annotated by four dedicated annotators. Since the clips are not restricted to one clip per song, they can also be used for song-level analysis. We present the methodology for building the dataset, covering the song list curation, clip selection, and emotion annotation processes. Moreover, we prototype use cases on clip-level music emotion classification and emotion-based symbolic music generation by training and evaluating corresponding models using the dataset. The result demonstrates the potential of EMOPIA for being used in future exploration on piano emotion-related MIR tasks.
1. INTRODUCTION
EMOPIA addresses the shortage of sizable emotion-labeled symbolic music resources by providing a multi-modal, piano-focused dataset with clip-level annotations. Its design supports emotion recognition and emotion-conditioned generation in both audio and symbolic domains.
- Paper scope: The paper curates public emotion-labeled music datasets, details EMOPIA’s construction, and evaluates classification and symbolic-generation models as demonstrations of its potential.The prototype generation experiments include an RNN and a modified Transformer-based model conditioned on emotion.
- Dataset motivation and contribution: EMOPIA provides piano-focused audio and MIDI data with clip-level emotion labels, addressing the limited availability of emotion-labeled symbolic music datasets.Existing datasets largely provide audio only, while the few MIDI datasets are small; piano-only data also supports automatic transcription and reduces multi-instrument confounds.
- Dataset motivation and contribution: EMOPIA combines single-instrument, multi-modal, and clip-level annotation properties to support emotion recognition, emotion variation detection, and emotion-conditioned piano generation.The paper describes use cases in both audio and symbolic domains, including generating new compositions or variations of existing pieces.
- Dataset motivation and contribution: The dataset contains 1,087 emotion-labeled clips from 387 songs, with multiple clips per song and a fairly balanced distribution across four emotion classes.The clips are manually selected for emotion consistency and annotated using a four-class taxonomy derived from the Russell valence-arousal model.
- Resources: EMOPIA is publicly released through a Zenodo repository, with model code on GitHub and generated-piece examples on a demo webpage.The supplied passages identify these release locations but do not provide their URLs.
2. RELATED WORK
Prior work has studied emotion recognition from symbolic music and emotion-conditioned symbolic generation, but these efforts often rely on small datasets or address limited emotion dimensions. EMOPIA is positioned within this emerging line of research.
- Emotion Recognition in Symbolic Music: Symbolic music emotion recognition has used handcrafted harmony, rhythmic, and dynamic features, while comparisons have found MIDI features particularly effective for valence.The cited prior work includes analysis of 63 features from 83 classical MIDI files and comparisons among audio, lyric, and MIDI features.
- Emotion-conditioned Symbolic Music Generation: Prior emotion-conditioned symbolic generation studies have used small MIDI datasets, four-emotion generation, facial-expression conditioning, or arousal-conditioned style transfer.The reviewed approaches include LSTM and genetic-algorithm selection, chord-conditioned sequence-to-sequence generation, and semi-supervised FaderNets.
- Emotion-conditioned Symbolic Music Generation: Earlier generation work differs in whether it generates new pieces, modifies existing music, or conditions generation on valence, arousal, or other signals.The related work spans positive/negative generation, four-emotion generation, facial-expression conditioning, and arousal-only style transfer.
3. THE EMOPIA DATASET
EMOPIA is a multi-modal dataset of piano performances segmented into emotion-consistent clips and annotated using a four-quadrant valence-arousal taxonomy. It provides transcribed MIDI, multiple symbolic representations, metadata, and song-, phrase-, and clip-level information, although audio is shared only through YouTube links.
- 3.1 Song Selection and Segmentation: EMOPIA contains 387 piano solo performances manually segmented into 1,087 clips, with full-song, phrase, segmentation, and emotion-label information.The dataset covers approximately 11 hours of data.
- 3.1 Song Selection and Segmentation: The collection emphasizes professional piano performances spanning Japanese anime, Korean and Western pop covers, movie soundtracks, and personal compositions.Songs were selected for clear emotion, high quality, and limited post-production engineering.
- 3.1 Song Selection and Segmentation: The dataset uses manually selected emotion-consistent clips, automatic piano transcription, and symbolic representations including MIDI-like, REMI, and CP with an added emotion-token family.MIDI-like and REMI support classification experiments, while CP is used for emotion-conditioned generation.
- 3.2 Emotion Annotation: Russell’s 4Q assigns each clip to HVHA, HVLA, LVHA, or LVLA according to valence and arousal.The four classes correspond to the quadrants of Russell’s Circumplex model.
- 3.5 Dataset Availability: The dataset includes metadata, segmentation annotations, emotion labels, and transcribed MIDI, but audio is distributed through YouTube links because of copyright restrictions.Song availability therefore depends on regional licenses and whether owners remove recordings.
4. DATASET ANALYSIS
The dataset analysis links arousal to note-level activity and valence to major-minor key distributions. High- and low-arousal groups separate across several features, while key-distribution differences are larger across valence than arousal groups.
- Note Density, Length, and Velocity: Note density is measured as notes per beat, note length as average duration in beats, and velocity is taken directly from MIDI.These features are used as musical correlates of arousal.
- Note Density, Length, and Velocity: High-arousal clips generally have denser, louder, and shorter notes than low-arousal clips, while differences between quadrants sharing arousal are subtler.Q2 has more dynamics than Q1, and Q3 is not distinguishable from Q4 in note density.
- Key Distribution: Major-minor key distributions distinguish valence: high-valence quadrants skew toward major keys, whereas low-valence quadrants show the opposite trend.The analysis uses key information extracted with the Krumhansl-Kessler algorithm.
- Key Distribution: Key-histogram distances indicate that clips differing in valence are more separated than clips differing in arousal.The comparison uses symmetric Jensen-Shannon divergence.
5. MUSIC EMOTION RECOGNITION
EMOPIA supports four-way and binary emotion classification in symbolic and audio domains. Deep models generally outperform handcrafted-feature baselines, while MIDI representations perform better than audio for valence.
- Deep learning approaches generally outperform logistic regression with handcrafted features, except for audio-domain arousal classification.
- The evaluation measures four-way emotion accuracy plus separate arousal and valence accuracies using stratified 7:2:1 splits.
- MIDI-domain classifiers achieve higher valence accuracy than audio-domain classifiers, 0.883 versus 0.704.
- Among symbolic representations, MIDI-like tokens appear to outperform REMI for valence classification.
6. EMOTION-CONDITIONED GENERATION
The authors evaluate emotion-conditioned symbolic generation with Transformer and LSTM models using objective and human-rated metrics. The pretrained CP Transformer performs best on most metrics and shows partial human-perceived emotion control, especially for positive valence and high arousal.
- The models include an RNN-based LSTM approach and a pretrained, emotion-conditioned Compound Word Transformer fine-tuned on EMOPIA.
- The generation evaluation combines surface-level metrics, classifier-based emotion metrics, and a survey of 25 subjects rating five qualities on a five-point scale.
- The pretrained CP Transformer performs best on most surface-level, emotion-related, and subjective metrics in Table 6.
- Human ratings give the pretrained Transformer median scores of 4 for positive valence and high arousal, exceeding competing models in those conditions.
- The pretrained Transformer is not sufficiently effective for generating low-valence music according to subjective assessment.
7. CONCLUSION
The paper introduces EMOPIA as a multimodal pop-piano emotion dataset and demonstrates its use for recognition and emotion-conditioned symbolic generation. The authors identify further feature analysis and conditioning improvements as future work.
- EMOPIA combines piano audio, MIDI transcriptions, and four-class clip-level emotion annotations in a medium-scale public dataset.
- Prototype models achieve high accuracy for four-quadrant and valence-wise emotion classification, while a Transformer generates music with target emotion to a certain degree.
- Future work includes analyzing feature importance for emotion classification and incorporating important features into generation models.
- The authors release the dataset and implemented-model code for future emotion-related MIR research.