Source-linked AI summary
Predicting Music Valence and Arousal via Textual and Multimodal In-Context Learning in Large Language Models
Chi-En Dai, Chih-Pin Tan, Yi-Hsuan Yang
TL;DR
Valence remains challenging to predict from music, and direct zero-shot LLM/MLLM valence predictions are poor. The paper uses auxiliary-set demonstrations to refine a task-specific base model at inference time, reporting improved valence prediction with text-based and multimodal ICL.
Problem
Valence prediction from musical audio remains challenging because valence is abstract and context-dependent, while direct zero-shot LLM/MLLM performance on continuous valence–arousal prediction is largely unexplored.
Source: §1 · p. 1Method
The framework uses CH818 demonstrations to prompt LLMs or MLLMs to estimate residual offsets from a task-specific base model’s initial predictions, without fine-tuning.
Source: §1 · pp. 1–2, §3 · p. 3Results
ICL improves valence prediction, and text-based and multimodal architectures exhibit different optimal configurations.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that inference-time ICL with textual and multimodal demonstrations is effective for dimensional music emotion recognition.
Source: §5 · p. 4The experiments use the relatively small MER60 dataset, and the authors identify expanding the test data as future work.
Source: §5 · p. 4
Abstract
from arXiv · showhide
While arousal can be reliably predicted from musical audio, emotional valence remains difficult to model due to its subjective and context-dependent nature. In this work, we investigate the potential of in-context learning (ICL) with large language models to advance continuous valence and arousal prediction. Rather than predicting emotion scores directly, our framework leverages contextual demonstrations from an auxiliary dataset to refine the initial predictions produced by a small, task-specific base model. We systematically evaluate a text-based ICL variant using lyrics and generated audio captions, alongside a multimodal variant that incorporates raw audio signals. Using the entire MER60 dataset strictly as a held-out test set, our experiments demonstrate substantial performance gains, particularly for valence. Text-based ICL improves the valence $R^2$ from 0.110 to 0.427 with 80 demonstration shots, while the multimodal variant achieves 0.346 with only 10 shots. These findings suggest that training-free ICL offers an effective inference-time strategy for current language models to predict continuous music emotion.
1. INTRODUCTION
Valence remains difficult to predict from music, and direct zero-shot LLM and MLLM predictions perform poorly on it. This work tests whether inference-time demonstrations and a base-model anchor can improve continuous valence–arousal prediction.
1. INTRODUCTION
- GPT-5’s zero-shot valence R^2 was −0.801 and Qwen3-Omni 30B’s was 0.037, despite arousal R^2 scores of 0.699 and 0.764, respectively.The authors note that likely prior exposure to these older songs did not suffice for zero-shot valence reasoning.Source: §1 · p. 1
- The framework uses demonstrations from auxiliary CH818, disjoint from test-set MER60, to improve LLM and MLLM music-valence prediction without weight updates or task-specific fine-tuning.The approach operates entirely at inference time.Source: §1 · p. 1
- Text-based ICL tests lyrics and generated audio captions as musical-content surrogates, while multimodal ICL tests whether adding lyrics enhances raw-audio input.Lyrics are considered for their semantic and thematic context; captions can describe mood and musical features.Source: §1 · pp. 1–2
- Rather than predict absolute scores, the LLM or MLLM estimates a residual offset from the base model’s initial prediction, which is added to refine the scores.The base model is trained on CH818 to preserve cross-dataset evaluation.Source: §1 · p. 2
- The authors report that ICL improves valence prediction, with text-based and multimodal architectures exhibiting different optimal configurations.They investigate demonstration-shot count, model capacity, and whether gains differ between valence and arousal.Source: §1 · p. 2
2. RELATED WORK
Related work spans multiple emotion models and input modalities, while this study focuses on continuous perceived emotion from audio. The authors position continuous regression with LLM/MLLM demonstrations as relatively unexplored.
2. RELATED WORK
- Music emotion recognition research varies by categorical or continuous labels, perceived or felt emotion, and inputs such as audio, lyrics, or MIDI.This work focuses on continuous, perceived music emotion recognition from audio clips.Source: §2 · p. 2
- Deep-learning MER studies include convolutional and recurrent networks and pretrained backbones across different settings.The cited passage identifies these as prior architectural approaches.Source: §2 · p. 2
- CH818 and MER60 use expert-supervised annotations, and the study trains and demonstrates on CH818 while reserving all of MER60 for testing.CH818 covers 818 Chinese pop clips; MER60 contains 60 Western pop clips.Source: §2 · p. 2
- The authors contrast their continuous valence–arousal regression from audio with Yang et al.’s categorical emotion annotation for symbolic MIDI.They characterize ICL for continuous regression in audio and music as largely unexplored.Source: §2 · p. 2
3. METHODOLOGY
The method anchors each prediction in a base model’s valence–arousal estimate, then uses in-context demonstrations to predict a correction. It evaluates text-based and multimodal routes for providing musical context.
3. METHODOLOGY
- The second-stage LLM or MLLM predicts an offset to the base model’s valence–arousal estimate, adds it to that estimate, and clips the result to the valid range.The estimate is normalized to the 1–9 scale; the offset is used to correct rather than replace the base prediction.Source: §3 · p. 3
- At inference time, the method prepends N demonstrations to each test audio clip, or query.In Fig. 2, b denotes the base-model prediction and y denotes the ground-truth valence–arousal values.Source: §3 · p. 3, Figure 2 · p. 3
- Text-based ICL prompts the LLM with each demonstration’s base prediction, lyrics, audio description, ground-truth scores, and target offset.The model predicts the query clip’s calibration offset from the serialized demonstrations and query.Source: §3 · p. 3
- The text-based route transcribes lyrics with Whisper and generates English audio descriptions with Qwen2-Audio, covering mood, energy, melody and harmony, and expression.The descriptions are prompted to reflect listeners’ feelings.Source: §3 · p. 3
- Multimodal ICL replaces the audio description with the original audio while retaining the prompt structure and may interleave audio tokens with text.The MLLM processes clips through its native audio encoder, using acoustic and textual information.Source: §3 · p. 3
4. EXPERIMENT
The experiments evaluate a CH818-trained base model and test whether textual and multimodal in-context demonstrations improve its predictions on held-out MER60. Demonstrations improve valence prediction, with different input components and shot counts shaping the two ICL variants.
4. EXPERIMENT
- CH818 supplies base-model training and demonstrations, while the 60-clip MER60 dataset is held out for testing; ground-truth valence–arousal values average annotator ratings.CH818 clips are annotated by three music experts; MER60 clips are rated by 40 human subjects.Source: §4 · p. 3
4.1. Evaluation of the Base Model
- The CH818-trained base model has the best cross-dataset performance among deep learning models on MER60 and remains competitive with SVR.The comparison includes SVR, VGGish, and Music2Emo; all models generalize poorly from native validation data to MER60, particularly for valence.Source: §4.1 · pp. 3–4
4.2. Evaluation of Text-based ICL and MICL
- At 80 shots, combining the base prediction, audio description, and lyrics performs best; in MICL, the base prediction plus lyrics performs best at 10 shots.Audio descriptions are the most informative individual text-ICL input, while lyrics alone contribute little; adding descriptions slightly hurts MICL valence performance.Source: §4.2 · p. 4
- Text-based ICL improves with more demonstrations, while MICL improves with five shots but peaks at 10 and slightly degrades thereafter.The reported tables summarize results over five runs; their captions do not provide cell values here.Source: §4.2 · p. 4, Table 2 · p. 4, Table 3 · p. 4
- For 80-shot text ICL, replacing GPT-5 lowers valence R2 to 0.164, 0.285, and 0.317; for 10-shot MICL, smaller replacements lower it to 0.123 and 0.308.The authors report these results as suggesting that larger models generally perform better.Source: §4.2 · p. 4
5. CONCLUSION
The paper proposes textual and multimodal ICL for dimensional music emotion recognition, using demonstrations to predict offsets that refine a base-model prediction. Experiments on MER60 support the approach, while the authors identify the dataset's relative smallness and multimodal ICL improvement as future-work areas.
5. CONCLUSION
- The proposed ICL approach uses textual and multimodal demonstrations to predict offsets that refine a base-model prediction.The experiments use the relatively small MER60 dataset.Source: §5 · p. 4
- The authors report that experiments on high-quality but relatively small MER60 demonstrate the approach's effectiveness.Source: §5 · p. 4
- Future work will seek to improve multimodal ICL and expand the test data.Source: §5 · p. 4