Source-linked AI summary
Music Flamingo: Scaling Music Understanding in Audio Language Models
Sreyan Ghosh, Arushi Goel, Lasha Koroshinadze, Sang-gil Lee, Zhifeng Kong, Joao Felipe Santos, Ramani Duraiswami, Dinesh Manocha, Wei Ping, Mohammad Shoeybi, Bryan Catanzaro
TL;DR
Music Flamingo targets the persistent difficulty of understanding music beyond short, surface-level descriptions, especially across layered and culturally diverse songs. It combines large-scale layered data curation with an enhanced Audio Flamingo 3 backbone, theory-grounded reasoning training, and GRPO reinforcement learning, achieving state-of-the-art results across diverse music understanding and reasoning benchmarks. The paper also identifies remaining gaps in underrepresented traditions, specialized instrument skills, and broader musical-skill coverage.
Problem
Music understanding remains limited because music is layered and information-dense, while available data and annotations constrain models to surface-level descriptions and questions.
Method
The paper builds Music Flamingo by fine-tuning an enhanced Audio Flamingo 3 backbone on MF-Skills and using MF-Think followed by GRPO-based reinforcement learning for music reasoning.
Results
Music Flamingo achieves state-of-the-art results across diverse music understanding and reasoning benchmarks, including QA, reasoning, MIR, lyrics transcription, and captioning evaluations.
Takeaways & Limitations
Music Flamingo demonstrates movement from surface-level recognition toward layered, human-like perception of songs.
Takeaways & Limitations
Remaining limitations include underrepresented cultural traditions, specialized instrument skills, and incomplete coverage of musical skills.
Abstract
from arXiv · showhide
We introduce Music Flamingo, a novel large audio-language model designed to advance music (including song) understanding in foundational audio models. While audio-language research has progressed rapidly, music remains challenging due to its dynamic, layered, and information-dense nature. Progress has been further limited by the difficulty of scaling open audio understanding models, primarily because of the scarcity of high-quality music data and annotations. As a result, prior models are restricted to producing short, high-level captions, answering only surface-level questions, and showing limited generalization across diverse musical cultures. To address these challenges, we curate MF-Skills, a large-scale dataset labeled through a multi-stage pipeline that yields rich captions and question-answer pairs covering harmony, structure, timbre, lyrics, and cultural context. We fine-tune an enhanced Audio Flamingo 3 backbone on MF-Skills and further strengthen multiple skills relevant to music understanding. To improve the model's reasoning abilities, we introduce a post-training recipe: we first cold-start with MF-Think, a novel chain-of-thought dataset grounded in music theory, followed by GRPO-based reinforcement learning with custom rewards. Music Flamingo achieves state-of-the-art results across 10+ benchmarks for music understanding and reasoning, establishing itself as a generalist and musically intelligent audio-language model. Beyond strong empirical results, Music Flamingo sets a new standard for advanced music understanding by demonstrating how models can move from surface-level recognition toward layered, human-like perception of songs. We believe this work provides both a benchmark and a foundation for the community to build the next generation of models that engage with music as meaningfully as humans do.
1 INTRODUCTION
Music understanding remains difficult because music combines layered acoustic, structural, expressive, lyrical, and cultural information, while existing models and datasets often support only short, surface-level descriptions. Music Flamingo addresses this gap with richer data and reasoning-oriented training for more musician-like analysis.
- Motivation: The approach targets richer applications in music creation, recommendation, cross-cultural analysis, education, and interactive systems.
- Motivation: Existing music-language models often produce short, generic captions, misidentify basic attributes, or rely on text-derived knowledge instead of auditory analysis.
- Motivation: Music combines surface attributes, mid-level organization, and higher-level meaning, requiring models to connect these layers rather than recognize isolated properties.The paper identifies tempo, key, and timbre alongside harmony, form, rhythm, lyrics, affect, style, and cultural context.
- Contribution: Music Flamingo introduces an open large audio-language model designed to move music understanding beyond surface recognition toward trained-musician-like reasoning and interpretation.
- Contribution: MF-Skills provides 4M+ high-quality samples from long, multicultural songs with layered captions covering harmony, structure, timbre, lyrics, and cultural context.
2 RELATED WORK
Audio-language models have advanced across speech and sound, but scaling music understanding remains constrained by limited, imbalanced, and insufficiently diverse data. Related music research provides specialized MIR capabilities, yet broader music-language reasoning remains underdeveloped.
- Multimodal audio–language modeling: Audio-language models generally use encoder-only or encoder-decoder paradigms to support auditory retrieval, understanding, and generation tasks.
- Scaling music understanding: Music data has grown far more slowly than speech and environmental-sound data in large audio-language training mixtures.Audio Flamingo’s music component increased by approximately 10% from version 1 to 3, compared with much larger growth in other audio domains.
- Scaling music understanding: Existing music datasets and models are limited by low diversity, weak skill coverage, annotation difficulty, and reliance on private proprietary data.
- Music information retrieval and captioning: Music Information Retrieval has extensively studied key detection, chord recognition, tempo estimation, lyrics transcription, retrieval, classification, and captioning, often emphasizing instrumental music.
3 METHODOLOGY
Music Flamingo is built by strengthening Audio Flamingo 3, curating layered music data from diverse full-length songs, and applying supervised and reinforcement-learning post-training for step-by-step reasoning. Its data pipeline combines model-generated captions, MIR metadata, theory-grounded annotation, and refined existing datasets.
- Training Methodology: Music Flamingo first strengthens Audio Flamingo 3, fine-tunes it on music and related datasets, and then applies reinforcement learning to support step-by-step music reasoning.
- Building Foundational Music Understanding: MF-Skills addresses surface-level and classification-heavy prior datasets by representing music through layered captions and question-answer pairs designed for deliberate reasoning.
- Building Foundational Music Understanding: The curation pipeline collects approximately 3M full-length songs from diverse cultures, synthesizes initial segment captions, extracts MIR metadata, and generates detailed captions and QA.
- Building Foundational Music Understanding: Generated annotations cover tempo and key, instrumentation and production, lyrics and lyrical themes, song structure, dynamics, and other musical dimensions.
- Building Foundational Music Understanding: Existing datasets are refined by correcting metadata-related caption errors, adding vocal and lyrical information, and making multiple-choice questions less vulnerable to language priors and guessing.
- Post-Training with Reinforcement Learning: MF-Think supplies theory-grounded reasoning chains for supervised fine-tuning before GRPO optimization with rule-based rewards over sampled candidate responses.
4 EXPERIMENTS
Music Flamingo is evaluated against recent audio-language baselines across music retrieval, question answering, lyrics transcription, reasoning, and captioning benchmarks. It consistently achieves strong results, including higher structured-music performance and richer SongCaps descriptions than Audio Flamingo 3.
- Evaluation Setup: The evaluation compares Music Flamingo with recent SOTA audio-language models across a broad benchmark suite using accuracy, WER, human scores, and GPT-based evaluation.The benchmark set spans music information retrieval, question answering, lyrics transcription, reasoning, and SongCaps captioning.
- Music Understanding and Reasoning Evaluation: Music Flamingo consistently leads across music QA, reasoning, MIR, and lyrics-transcription benchmarks, including 76.83 accuracy on MMAU-Music and 65.6 and 74.58 on MMAU-Pro-Music and MuChoMusic.The reported results also include 80.76% accuracy on NSynth, 90.86% on Medley Solos DB, and lower Chinese and English lyrics-transcription WER than open and closed baselines.
- Music Understanding and Reasoning Evaluation: Removing reinforcement-learning fine-tuning with thinking traces reduces performance from 65.6 to 63.9 and from 74.58 to 69.5 on the two tougher music benchmarks.The comparison supports the reported value of step-by-step reasoning and exploration in those evaluations.
- SongCaps Captioning: On SongCaps, Music Flamingo scores 8.3 from human raters, 8.0 for correctness, and 8.8 for coverage, outperforming Audio Flamingo 3.SongCaps evaluates captions for culturally diverse songs using human judgments and LLM-as-a-judge assessments.
- Qualitative Evaluation: The experiments include qualitative assessment by trained music experts, with additional genre, popularity, and cross-cultural analyses reported in the appendices.These analyses complement the quantitative benchmark comparisons.
5 CONCLUSION, LIMITATIONS AND FUTURE WORK
The paper concludes that Music Flamingo combines layered music data, chain-of-thought reasoning, and reinforcement learning to achieve strong results across diverse music-understanding benchmarks. It also identifies cultural, task-specific, and skill-coverage gaps that remain to be addressed.
- Conclusion: Music Flamingo combines MF-Skills, MF-Think, improved training, and reinforcement learning to achieve state-of-the-art results across diverse music-understanding and reasoning benchmarks.The paper presents this combination as supporting movement beyond surface recognition toward layered perception of songs.
- Limitations and Future Work: The model remains limited on underrepresented or skewed cultural traditions, fine-grained piano technique recognition, and other specialized instrument skills.The paper calls for broader global music data and expanded coverage of musical skills.
A ETHICS STATEMENT
The ethics statement scopes the work to publicly available or research-licensed material and focuses analysis on musical attributes rather than identifying individuals. It addresses cultural bias, copyright risks, and possible misuse of singing-voice analysis.
- Data and Content: The study uses publicly available or research-licensed content, does not release copyrighted audio, stems, or lyrics, and substitutes non-copyrightable descriptors where needed.The work covers culturally diverse audio, music, and singing-voice understanding.
- Cultural Representation and Bias: Uneven representation across regions, languages, and genres can bias estimates or degrade performance on underrepresented traditions.The authors document dataset composition, emphasize vocal and multicultural evaluation, and caution against normative cross-cultural quality claims.
- Copyright and Content Ownership: The project avoids generative audio synthesis and reconstruction-enabling assets while providing guidance for filtering long verbatim lyric reproduction.The released outputs are textual QA, captions, and reasoning traces rather than synthesized audio.
- Privacy and Safety: Singing voices may encode sensitive traits, so the work focuses on musical attributes and releases only research artifacts while acknowledging profiling and surveillance risks.The stated safeguards include non-identifying metadata, documentation, and permitted-use licensing where possible.
B REPRODUCIBILITY STATEMENT
The paper describes a reproducibility package covering data, preprocessing, architectures, training, post-training, inference, and evaluation details. These disclosures are intended to provide the information needed to reproduce the reported results.
- Reproducibility Statement: The paper and appendix report dataset sources and splits, preprocessing, model architectures, training schedules, GRPO settings, inference settings, and benchmark protocols.The listed details include parameter counts, optimizers, learning-rate policies, batch sizes, gradient clipping, reward definitions, decoding constraints, and metrics.
C MUSIC FLAMINGO TRAINING DATASETS
Table 2 summarizes the datasets used to train Music Flamingo, including their size, audio-QA coverage, and training-stage epochs.
- Table 2 lists each training dataset alongside total hours, audio-QA pairs, and epochs used at each training stage.
D MUSIC FLAMINGO TRAINING DETAILS
The training-details section presents the model’s stage-specific configurations and directs readers to Table 3 for the settings.
- Training proceeds across multiple stages, with stage-specific configurations summarized in Table 3.
E USER STUDY ON MUSIC FLAMINGO
The user study compares Music Flamingo with Qwen3 Omni, GPT-4o, and Gemini 2.5 Pro on eight songs spanning English and Brazilian Portuguese. Music Flamingo performs best overall, while deeper contextual identification remains limited.
- The study uses eight songs, split evenly between English and Brazilian Portuguese and between highly popular and less-known artists.
- Music Flamingo performs best overall among the four compared models, although accurately identifying deeper context remains limited.
F COMPARATIVE ANALYSIS ACROSS SONGS FROM DIFFERENT CULTURES
Across culturally and linguistically diverse songs, Music Flamingo provides the strongest technical grounding and music-theoretic reasoning, while other models show complementary strengths in taxonomy or production narratives. Errors in genre or vocal-status recognition can propagate into inaccurate instrumentation, production, structure, and emotional interpretations.
- General technical characteristics (tempo, time signature, key): Music Flamingo consistently matched or closely tracked tempo and key, including relative-minor centers and felt double-time readings.Examples include Enamorados at metadata C major versus an A-minor harmonic center, Jim & Pam near 158 BPM in D, and Trava u doma at A minor and approximately 130 BPM.
- Emotional content and lyrics: Music Flamingo, Gemini, and GPT-4o generally provide coherent lyric-grounded emotional readings, while Qwen3’s instrumental misclassifications distort narrative and emotion.Emotion inferred from sonics without lyric anchoring loses nuance and can miss culture-specific references.
- Complex technical characteristics (chord progressions/voicings, song structure, production): Music Flamingo offers the deepest harmonic and structural analysis but sometimes over-specifies unsupported chords or percussion layers.Its conservative production notes are usually accurate, whereas Qwen3’s incorrect top-level premise can cascade into inapplicable structure and production claims.
- Failure-mode coupling: Genre misclassification can produce hallucinated instrumentation and production details, whereas canonical genre naming supports more accurate descriptions.For example, misreading Ai Se Eu Te Pego as forró prompted forró-typical percussion, while incorrect genre framing of La fille invited nonexistent drum programming.
- Model-specific observations (holistic): Gemini leads in canonical genre taxonomy, while GPT-4o is strongest for arrangement, dynamics, production prose, and lyric-grounded emotion.Qwen3-Omni can produce coherent mood and arrangement readings but remains inconsistent, including instrumental misclassifications of vocal songs.
- Model-specific observations (holistic): Music Flamingo delivers the most reliable technical grounding and music-theoretic reasoning across the culturally diverse song comparisons.It is strongest on tempo, key, section-level reasoning, harmonic intent, and deep structural analysis.
G LINEAR PROBING EXPERIMENTS WITH AUDIO ENCODERS
Linear probing reveals a trade-off across audio encoders: caption-trained Whisper-based encoders perform well on genre classification but less well on key classification, while MERT favors lower-level musical information. This gap motivates future encoders that preserve more low-level audio detail for music understanding.
- Encoder comparison: Whisper-based encoders achieve high genre-classification accuracy but comparatively low key-classification accuracy, whereas MERT shows the opposite pattern across higher- and lower-level tasks.The comparison is reported for Qwen2Audio, Audio Flamingo 3, and MERT encoders.
- Encoder comparison: Caption-trained Whisper representations may favor genre because captions mention genre more often than a song’s key.
- Implications: The observed accuracy gap is consistent with Music Flamingo outputs and motivates using an encoder that preserves more low-level audio information.The authors identify this as a direction for improving understanding of lower-level musical features.