TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking

1. INTRODUCTION

• Different conditioning formats, architectures, and access modes complicate comparison because prompts, efficiency measurements, and evaluation metrics do not apply uniformly.A shared musical specification must be preserved while adapting its representation to each system; local and hosted systems also expose different efficiency measures.Source: §1 · p. 1
• TTM-Bench benchmarks heterogeneous systems through separate musical-content alignment and access-aware computational-efficiency workflows.Alignment uses semantic, genre, and musical-descriptor agreement with an aggregate score; efficiency covers generation time, resource use, and cost within observable execution boundaries.Source: §1 · p. 1
• A preliminary case study illustrates the framework across 13 open and commercial text-to-music systems.The case study is introduced as preliminary, not as a broad statistical evaluation.Source: §1 · p. 2

2. RELATED WORK

• TTM performance benchmarking spans automatic metrics, learned perceptual evaluators, computational-efficiency studies, and cross-system benchmarks.Source: §2 · p. 2
• Automatic metrics measure distinct properties, and their results depend on choices such as embedding model, reference data, preprocessing, and aggregation.Source: §2 · p. 2
• Human-annotated perceptual evaluators estimate alignment, preference, and musical quality, but depend on the coverage and potential biases of their training judgments.Source: §2 · p. 2
• Efficiency studies commonly report latency and real-time factor, but differing hardware and settings limit direct comparison, while memory, energy, and hosted-service cost are less consistently reported.Source: §2 · p. 2
• Existing cross-system approaches emphasize preferences or controlled evaluation, whereas TTM-Bench targets available systems with heterogeneous access and conditioning through component-level alignment and observable efficiency measures.Source: §2 · p. 2

3. METHOD

• Alignment compares each generation with a common reference-derived specification expressed in the conditioning format supported by its system.Semantic Alignment uses LAION-CLAP similarity; Genre Alignment combines genre overlap and classifier-score similarity; Descriptor Alignment combines timbre, BPM, and tempo-label similarity.Source: §3 · p. 2
• MCAS combines Semantic Alignment, Genre Alignment, and Descriptor Alignment with weights of 0.40, 0.35, and 0.25, respectively.The weights are empirically fixed, and sensitivity is tested with equal weighting, local perturbations, and leave-one-component-out configurations.Source: §3 · p. 2
• Efficiency measures respect access-mode boundaries: local execution reports latency, RTF, memory, GPU utilization, and GPU energy, while hosted services report client-observed latency, RTF, and listed API cost.RTF is measured latency divided by usable output duration; unavailable server-side resources are not estimated.Source: §3 · p. 3
• The case study evaluates 13 systems on instrumental generation using 100 reference tracks curated for varied conditions rather than statistical representativeness.HTDemucs extracts instrumental stems, and the descriptor pipeline derives 46 attributes per reference track from metadata, audio analysis, and controlled enrichment.Source: §3 · p. 3
• For each reference-system pair, five validated prompt variants and four 30-second generations per prompt yield 26,000 tracks, with scores averaged within pairs and across references.The protocol also uses paired reference-level bootstrap resampling for robustness and evaluates efficiency over 10 distinct inputs per system after warm-up.Source: §3 · p. 3

4. RESULTS

• Tested weighting alternatives preserve relative system-level MCAS results, but the three smallest pairwise MCAS contrasts have 95% bootstrap intervals that include zero.Agreement with the proposed weights has median/minimum Spearman ρ = 0.978/0.929 and Kendall τ = 0.923/0.821; small differences are therefore not robust to reference-set composition.Source: §4 · pp. 3–4
• Commercial systems outperform open models across all musical-content alignment measures, while component leaders differ between groups and alignment components show weak generation-level but higher system-level correlations.Generation-level correlations are weak (Pearson r = 0.217-0.281; Spearman ρ = 0.222-0.270), while correlations among system-level means are higher (Pearson r = 0.767-0.879; Spearman ρ = 0.687-0.896).Source: §4 · p. 4
• Among open models, ACE-Step leads latency, RTF, and energy, while MusicGen Large leads peak RAM and MusicLDM peak VRAM; among APIs, Stable Audio 2.5 leads latency and RTF, while Lyria 3 leads cost.Thus, no single system leads all efficiency measures, and faster generation does not necessarily mean lower cost.Source: Table 2 · p. 4, §4 · p. 4

• Across 13 TTM systems and 100 reference music tracks, higher musical-content alignment did not systematically coincide with lower computational demands.Source: §5 · p. 4
• Alignment components capture distinct generation-level information, supporting their individual reporting alongside aggregate MCAS.Source: §5 · p. 4
• Relative system-level MCAS results remained stable across tested weighting alternatives, but small differences between similarly scoring systems were sensitive to reference-set composition.Source: §5 · p. 4
• The findings are limited to the evaluated references, system versions, configurations, and access conditions.Source: §5 · p. 4
• MCAS is an experimentally defined aggregate whose weighting needs further validation against human judgments; future work will broaden references and study this relationship.Source: §5 · p. 4

• The study involved no human participants or animals, and therefore required no ethical approval.Source: §7 · p. 5