PaperlayerView exact revision on arXiv

Source-linked AI summary

STEMMA: Song-to-Stem Multi-Audio Reasoning for Large Audio Language Models

Hoyeol Sohn, Wonil Kim, Keunhyoung Kim, Sangeun Kum, Taehyoung Kim, Dongjoo Moon, Theerasak Charoenchob, Teeratep Weerapang, Jongpil Lee, Juhan Nam

arXiv:2610.11884v1cs.SDeess.AS

TL;DR

Existing music LALMs and QA datasets provide limited evaluation of relationships among excerpts with known production provenance. STEMMA builds relation-first multi-audio QA and instruction-tuning data from catalog provenance; tuning improves multi-audio results, especially on structural relations, while retaining single-audio understanding.

  • Problem

    Existing LALMs and music QA datasets mostly handle recordings independently or compare tracks without known production relationships, limiting coverage of relations among songs, sections, and stems.

    Source: §1 · p. 1

  • Method

    STEMMA queries a catalog after specifying a target provenance relation, retrieving matching excerpts and hard negatives to build a benchmark and track-disjoint instruction-tuning set.

    Source: §1 · p. 1, §3.2 · p. 2, §3.3 · p. 2

  • Results

    Tuning improves every tested multi-audio benchmark for both backbones, with larger gains on structural relations; tuned models outperform cascades on STEMMA-Bench structural relations, while a cascade leads on Jamendo-MT-QA.

    Source: §5.1 · pp. 3–4

  • Takeaways & Limitations

    STEMMA provides a way to evaluate and train multi-audio reasoning over structural relations among sections and stems drawn from full songs.

    Source: §1 · p. 1

    Attribute relations remain difficult because fine-grained differences may be lost in captions and MIR estimates, and the reason they stay hard is open.

    Source: §5.2 · p. 4

Abstract

from arXiv · show

Music understanding often requires comparing excerpts and reasoning about relationships among songs, sections, and stems. However, existing large audio-language models (LALMs) and music question-answering datasets typically operate on single recordings or compare independently sampled tracks with no known production relationship. We introduce STEMMA, a multi-audio music question-answering framework built around production provenance: whether excerpts originate from the same track or section, and which stems belong to which mixtures. Because such relations are sparse under conventional audio-first sampling, STEMMA adopts a relation-first construction strategy: it first specifies a target relation and then queries the catalog for excerpts that satisfy it and hard negatives that do not. Labels are determined directly from catalog provenance rather than generated by a language model from metadata. We build STEMMA-Bench for evaluation and a track-disjoint training set, STEMMA-Instruct. Fine-tuning two LALMs on STEMMA-Instruct improves multi-audio reasoning, with the largest gains on structural relations directly determined by the catalog, while preserving single-audio music understanding.

1. INTRODUCTION

Existing music LALMs and QA corpora largely treat recordings independently, while music applications increasingly need comparisons across songs, sections, and stems. STEMMA uses known production provenance to construct multi-audio QA and training data for those relationships.

1. INTRODUCTION

  • Music retrieval, arrangement, and sample discovery require comparing excerpts across tracks and sections, but many conversational agents rely on tool outputs or stored attributes rather than joint audio reasoning [3, 4, 5, 6, 7].The needed cross-audio grounding localizes evidence in each input before inferring a relation across inputs.Source: §1 · p. 1
  • Prior multi-audio work extends comparison to independent track pairs, but does not systematically use known song–section–stem provenance.Performance also degrades as candidate sets grow.Source: §1 · p. 1
  • STEMMA specifies a provenance relation first, then queries matching examples and hard negatives to build STEMMA-Bench and track-disjoint STEMMA-Instruct.Fine-tuning two LALMs raises aggregate multi-audio accuracy while retaining single-audio music understanding.Source: §1 · p. 1

2. RELATED WORK

Prior music LALMs and multi-audio systems support music understanding or comparison, but existing comparative music QA uses independently sampled tracks. STEMMA instead uses sections and stems from the same song to ask structural questions.

Music understanding in LALMs

  • Music understanding LALMs include LLark, which unifies captioning, understanding, and reasoning over open-source music annotations; Music Flamingo scales the approach with synthetic supervision, and models are trained and evaluated one recording at a time.MOSS-Music is an open music audio-language model trained with a similar pipeline.Source: §2 · p. 1

Multi-audio grounding in LALMs

  • Multi-audio systems add paired or interleaved audio tasks, while MUGEN finds performance falls as candidate inputs increase and acoustic comparison is a bottleneck.Beyond Single-Audio and PolyAudio use targeted multi-audio instruction tuning for comparison and temporal reasoning.Source: §2 · p. 1

Comparative music QA

  • Jamendo-MT-QA compares independently sampled full-track pairs using questions generated from captions and metadata, whereas STEMMA’s same-song sections and stems enable structural questions.STEMMA’s relation-first pipeline retrieves tuples satisfying or violating catalog relations before rendering them as multi-audio instructions.Source: §2 · p. 1, Figure 1 · p. 2

3. RELATION-FIRST MULTI-AUDIO DATA

STEMMA constructs multi-audio QA by specifying a relation before sampling, organizing excerpts and their metadata in a catalog, and querying that catalog for examples. Its relation definitions control both tuple conditions and the QA format.

3.1. Why the relation comes first

  • Because same-track tuples are sparse under independent sampling, specifying a relation first retrieves valid examples directly and avoids rejected draws.Explicit hard-negative conditions control similarity to the positive, while catalog-derived labels and difficulty do not depend on language-model output; new tasks reuse the catalog and format.Source: §3.1 · p. 2

3.2. Building the catalog

  • The catalog represents each track as ordered sections and each section as a mixture or separated role, storing track, section, category, stem role, and musical attributes.When production metadata is unavailable, attributes can be estimated from full-song audio with MIR tools, allowing other full-song corpora to use the schema.Source: §3.2 · p. 2

3.3. Relation queries

  • Relations specify anchor, correct-candidate, and distractor sources plus positive and hard-negative conditions, using operators such as same, different, close, overlap, and greater.Hard negatives preserve selected context while violating the target relation; the nine relations pair conditions with units and candidate-QA questions.Source: §3.3 · p. 2, Table 1 · p. 3
  • Pair QA asks about two excerpts, while candidate QA presents an anchor with four audio candidates; the relation and QA format are independent.A catalog query generates each task tuple, answer, and answer position.Source: §3.3 · p. 2

4. EXPERIMENTAL SETUP

The experiments use catalogs built from authored production metadata and public full-song corpora with estimated fields, then evaluate track-disjoint instruction tuning on multi-audio and single-audio benchmarks.

Corpora and catalog construction

  • The training catalog combines 15,856 production songs with authored metadata, 69,297 JamendoMaxCaps songs, and 12,119 reusable Free Music Archive songs.Public-corpus fields are estimated from audio and stored in the same schema as authored fields, so each relation can be reused unchanged.Source: §4 · p. 3

Training data

  • STEMMA-Instruct contains 194,544 instances split evenly between Relation MultiQA and coreference, with one to five audio inputs per instance.Coreference uses labeled full songs and requires each song’s answer to remain associated with its matching audio label.Source: §4 · p. 3

Models and instruction format

  • Music Flamingo and MOSS-Music are fully fine-tuned for one epoch, extending their existing single-audio instruction tuning with labeled audio inputs and explicit response formats.Training uses batch size 128, learning rate 1 × 10−5, and two B200 GPUs.Source: §4 · p. 3

Evaluation

  • STEMMA-Bench evaluates 300 hand-checked candidate-QA items across 481 songs, including structural and musical-attribute relations, with stems authored rather than separated.Evaluation also reports MUGEN and Jamendo-MT-QA for multi-audio understanding, and MMAU-Pro, MMAU: Mini-Music, MMAR, and RUL-MuChoMusic for single-audio understanding.Source: §4 · p. 3

5. RESULTS

STEMMA-Instruct improves multi-audio benchmark performance for both backbones, with stronger gains on provenance-based structural relations than on musical attributes. Combining its two training roles yields the strongest multi-audio averages while single-audio performance remains above pretrained levels.

5.1. System-level comparison

  • Tuning improves every multi-audio benchmark for both backbones; structural-relation gains are 59.8 points for Music Flamingo and 23.6 for MOSS-Music.Source: §5.1 · p. 3
  • 0.8189 versus 0.5039 on STEMMA-Bench structural relations favors the best tuned model over the best cascade, while Jamendo-MT-QA favors cascades, 0.8716 versus 0.8622.Source: Table 2 · p. 4, §5.1 · p. 4

5.3. Training-mixture analysis

  • +18.29 and +3.50 points from Relation MultiQA alone exceed coreference-only gains of +5.55 and +0.84 for Music Flamingo and MOSS-Music, respectively.The two roles teach separating inputs and identifying their relationships.Source: Table 3 · p. 4, §5.3 · p. 4
  • +28.24 versus 23.84 for Music Flamingo and +6.88 versus 4.34 for MOSS-Music show that combined training beats either role alone by more than their gains add up.Single-audio accuracy stays above the pretrained score in every condition.Source: Table 3 · p. 4, §5.3 · p. 4

5.1. System-level comparison

  • Among pretrained systems, Qwen2.5-Omni leads STEMMA-Bench structural relations at 0.5354, ahead of MOSS-Music at 0.3543 and Music Flamingo at 0.2205.It is not music-specialized.Source: §5.1 · p. 4

5.2. Relation-level analysis

  • 59.8 points for Music Flamingo and 23.6 for MOSS-Music are the structural-relation tuning gains; attribute relations top out at 0.3526 against chance of 0.25.Music Flamingo reaches 0.8189, the highest structural-relation score in that column.Source: §5.2 · p. 4
  • Structural cascades reach 0.4094–0.5039, while attribute cascades reach 0.2254–0.2601; the fine-grained attribute differences may be lost in captions and MIR estimates.The passage says why attribute relations remain hard is open.Source: §5.2 · p. 4

6. CONCLUSION

STEMMA uses song–section–stem provenance to generate relation-first multi-audio question-answering and instruction-tuning data. Its combined training roles yield the highest multi-audio averages for both backbones while keeping single-audio averages above pretrained levels.

  • STEMMA uses song–section–stem provenance to generate relation-first multi-audio QA and instruction-tuning data.Source: §6 · p. 4
  • Combining Relation MultiQA with coreference yields the highest multi-audio average for both backbones while keeping their single-audio averages above pretrained levels.Source: §6 · p. 4
Loading 2610.11884v1…