PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't CLAP: Are Music-Text Models Bag-of-Words?

Yuan-Chiao Cheng, Alexander Lerch

arXiv:2609.30540v1cs.SDcs.CLeess.AS

TL;DR

The paper asks whether music-text models preserve instrument–attribute relations, a relevant question because CLAP is a predominant objective measure of prompt faithfulness. It tests original and attribute-swapped captions across contrastive models and a LALM, finding that contrastive models are insensitive to swaps and that the LALM’s advantage largely reflects language priors.

  • Problem

    It is unclear whether music-text models preserve which attributes belong to which instruments, despite CLAP’s use as an objective measure of prompt faithfulness.

    Source: §1 · p. 1

  • Method

    The authors swap one property between two instruments in captions and compare model choices across four contrastive models and a LALM, including text-only and prior-controlled tests.

    Source: §1 · p. 1, §2.3 · p. 3, §2.4 · p. 3

  • Results

    Four contrastive models show no reliable preference for original captions under attribute swaps, while Qwen2-Audio’s higher accuracy largely tracks text-only choices and falls near chance on prior-free onset-order audio.

    Source: §3.1 · p. 3, §3.2 · p. 4, §3.4 · p. 4

  • Takeaways & Limitations

    The findings indicate that CLAP and related metrics do not capture fine-grained musical meaning or attribute bindings in the tested captions.

    Source: §4 · p. 4

    The proposed explanations for the text-embedding failure—contrastive training objectives and short-phrase training data—are not tested in this study.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Text-to-music systems are assessed on audio quality and on how faithfully the music follows its prompt, and the CLAP score, the cosine similarity between a music-text model's audio and text embeddings, is the standard objective metric of faithfulness. We ask how accurately that score reflects the text: when an attribute is linked to an instrument (e.g., distorted guitar), does the text embedding capture that binding? To find out, we introduce an attribute swap perturbation: the caption of a real recording is edited by exchanging exactly one property, timbre, lead versus accompaniment, or order of first appearance, between two instruments. We then test four contrastive music-text models and one large audio-language model on whether the audio scores higher against the original caption than against the perturbed one. No contrastive model distinguishes the two captions reliably. The audio-language model does better, but further experiments show that its advantage rests largely on audio-agnostic language priors. Our results thus provide compelling evidence that the CLAP score and related metrics do not capture fine-grained musical meaning or attribute bindings; their representation is closer to a bag-of-words that leaves them insensitive to meaning-changing perturbations of the caption.

1. INTRODUCTION

CLAP is a predominant objective measure of prompt faithfulness, but it is unclear whether music-text models preserve which musical attributes belong to which instruments. The paper tests this with attribute swaps and reports that contrastive models are insensitive to the changes, while the LALM’s advantage largely reflects language priors.

1. INTRODUCTION

  • CLAP measures prompt faithfulness through audio–text embedding similarity, alongside subjective listening studies.Audio quality is often assessed with proxy metrics such as FAD and KAD.Source: §1 · p. 1
  • Prior work motivates testing attribute bindings: CLIP can ignore caption attributes, relations, and order, while audio-language models may rely on text priors.For music, it was unclear whether contrastive models share this shortcoming.Source: §1 · p. 1
  • The benchmark exchanges exactly one property between two instruments while preserving the other caption words, testing timbre, lead versus accompaniment, and first-appearance order.A correct result occurs when the audio is more similar to the original than to the swapped caption.Source: §1 · p. 1
  • The study introduces MASB, with 400 annotated recordings, and reports that contrastive models cannot reliably distinguish original from swapped captions, while the LALM advantage largely reflects language priors.The overview depicts recordings paired with original and swapped captions and evaluated by contrastive models and a LALM.Source: §1 · pp. 1–2, Figure 1 · p. 2

2. METHODOLOGY

The methodology tests whether music-text embeddings preserve caption attribute relations using annotated recordings, original–swapped caption pairs, and model comparisons. It also separates audio evidence from language priors through text-only choices and prior-balanced accuracy.

2. METHODOLOGY

  • The four experiments assess audio-caption similarity, the LALM’s language prior, text-embedding proximity, and a symmetric audio set that removes the prior.The methodology overview presents the recording and caption pair as inputs to contrastive models and a LALM.Source: §2 · p. 2, Figure 2 · p. 3

2.1. Dataset

  • The dataset contains 400 unmodified 10-second excerpts, each annotated for one property and rendered in two caption formats, yielding 800 caption pairs.MASB-Order adds 300 symmetric pairs from MoisesDB stems, where the two mixes differ in which instrument enters first.Source: §2.1 · p. 2

2.2. Caption construction and perturbation

  • Caption construction swaps the two instruments’ property slots, preserving the exact same words in the original and perturbed captions.The properties are timbre, lead versus accompaniment, and first-appearance order; results pool template and natural-caption versions.Source: §2.2 · p. 2

2.3. Models and scoring

  • Four contrastive models are evaluated by whether audio similarity is higher for the original caption, while the LALM is tested with audio and in a text-only baseline.The contrastive decision uses cosine similarity; chance accuracy is 0.5, and the text-only baseline selects between captions without audio while averaging caption orderings.Source: §2.3 · pp. 2–3, Figure 2 · p. 3

2.4. Separating text prior from audio evidence

  • The analysis separates language priors from audio evidence using agreement with text-only choices and prior-balanced accuracy across caption pairs where those choices are correct or incorrect.Cohen’s κ measures agreement beyond that predicted by the models’ accuracies; prior-balanced accuracy averages the two halves.Source: §2.4 · p. 3, Figure 3 · p. 3

3. EXPERIMENTS

The experiments test attribute-swap faithfulness through audio-caption similarity, language-prior controls, text-embedding proximity, and a symmetric audio test. Contrastive models show weak sensitivity to the swaps, while Qwen2-Audio’s stronger scores largely track text priors.

3.1. Exp. 1: CLAP score under attribute swap

  • Across properties, contrastive models’ mean accuracy ranges from 0.431 to 0.519, with no consistent preference for original captions or reliable prompt faithfulness.The corrected lead-role results for MuQ-MuLan (0.642) and MS-CLAP (0.630) were still too low to count as reliable.Source: §3.1 · p. 3

3.2. Exp. 2: Text prior versus audio evidence

  • Qwen2-Audio’s overall accuracy rises only from 0.644 without audio to 0.681 with audio, and its timbre accuracy falls from 0.682 to 0.639.On lead role and onset order, its audio choices mostly amplify text-only choices; on timbre, a small prior-balanced margin remains.Source: §3.2 · p. 4

3.3. Exp. 3: Single-modality text embedding proximity

  • For every encoder, median distance to the swapped caption (0.007 to 0.030) is below the paraphrase distance (0.062 to 0.122) and other-recording distance (0.37 to 0.71).The failure is already present in the text embedding before audio comparison; proposed explanations involving contrastive training and short training text are untested.Source: §3.3 · p. 4

3.4. Exp. 4: Symmetric audio onset-order test

  • With the text prior removed, every model scores near chance (0.497–0.538), and none reads which instrument starts first from the audio.The audio sets differ, however: MASB-Order uses MoisesDB stems with muting edits and template captions, while MASB uses real recordings with human annotations; this distribution shift might explain the gap.Source: §3.4 · p. 4

4. CONCLUSION

The results indicate that contrastive music-text models encode captions much like a bag of words, while Qwen2-Audio’s advantage rests largely on text priors. The findings warrant caution when treating CLAP as a measure of faithfulness to long, musically detailed captions.

4. CONCLUSION

  • Contrastive music-text models embed captions like a bag of words, with swaps barely moving text embeddings and no reliable original-caption preference.Qwen2-Audio scores higher, but its main competitive advantage stems from text priors.Source: §4 · p. 4
  • The results warrant caution before assuming CLAP faithfully encodes long-form, musically detailed captions.Source: §4 · p. 4
  • Qwen2-Audio’s higher scores largely reflect text priors rather than a consistent advantage from audio evidence.Source: §4 · p. 4

Related papers

Loading 2609.30540v1…