PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Text-to-Music Generation with Human Preference Rewards

Yonghyun Kim, Junwon Lee, Haiwen Xia, Yinghao Ma, Chris Donahue

arXiv:2606.21670v1cs.SDcs.AIcs.LG

TL;DR

Text-to-music systems commonly rely on FAD-CLAP and CLAP, leaving learned human-preference rewards underused in evaluation and optimization. This paper integrates TuneJury reward conditioning, reward-based sample selection, expert iteration, preference tuning, and inference post-processing, finding expert iteration dominant while inference-time score control saturates.

  • Problem

    The paper addresses the limited use of learned human-preference rewards alongside standard FAD-CLAP and CLAP metrics in text-to-music generation.

  • Method

    The pipeline uses TuneJury reward for training-time conditioning and sample selection, followed by reward-guided expert iteration, preference tuning, and inference post-processing.

  • Results

    −0.0362 FAD-CLAP is the dominant v1-chain improvement from expert iteration, while CRPO adds noise-level gain and the inference-time score scalar saturates.

  • Takeaways & Limitations

    For this submission, preference reward is most useful as a training-time signal and ranking criterion rather than as an inference-time control knob.

Abstract

from arXiv · show

We describe our entry to the efficiency track of the Academic Text-to-Music (ATTM) Grand Challenge at ICME 2026. Beyond the challenge protocol's FAD-CLAP and CLAP score, we add a learned human-preference reward from TuneJury, a twin pairwise ranker trained over open music-preference datasets. The reward serves both as a training-time conditioning signal and as a sample-selection criterion. The pipeline combines five engineering decisions on a 120M-parameter FluxAudio-S backbone, four at training time and one at inference: (i) training-time reward conditioning that doubles as an inference-time CFG axis, (ii) a sweep over five score-conditioning architectures, where training and inference use different variants, (iii) expert iteration on the top decile, (iv) a short preference-tuning pass (CRPO) for audio-text alignment, and (v) inference post-processing via joint CFG, source separation, and loudness normalization. Per-stage decomposition on 100 Song Describer prompts shows training-time reward conditioning as a functional conditioning axis, expert iteration as the dominant contributor, the preference-tuning pass adding only noise-level gain, and the inference-time score scalar already saturated by the end of the chain.

I. INTRODUCTION

The submission augments the ATTM efficiency-track evaluation with TuneJury human-preference rewards and combines five engineering decisions on a 120 M-parameter FluxAudio-S baseline. Reward conditioning supports both training and inference-time guidance, while expert iteration provides the dominant reported chain improvement.

  • Contribution: TuneJury supplies a learned human-preference reward used for per-clip training conditioning and selecting self-generated samples for expert-iteration fine-tuning.The reward is produced by a twin pairwise ranker trained over open music-preference datasets.
  • Pipeline: The pipeline applies five engineering decisions to the 120 M-parameter FluxAudio-S baseline, with four affecting training and one operating only at inference.The decisions cover reward conditioning, score-conditioning heads, expert iteration, CRPO preference tuning, and inference setup.
  • Reward conditioning: Score-conditioned variants improve FAD-CLAP by 0.025–0.040 absolute over the unconditional baseline, while null-score dropout enables reward as an inference-time CFG axis.The per-clip TuneJury score enters the backbone as a Fourier-embedded side input.
  • Expert iteration: −0.0362 FAD-CLAP is the dominant chain contribution from reward-guided expert iteration on the v1 chain.Samples are ranked using an equal-weight blend of ranker reward and CLAP-text similarity, and the top decile is used for fine-tuning.
  • Scope and limitations: The report is scoped to the engineering pipeline and uses a fixed single-value inference score scalar selected on SDD-100, without analyzing the score-response curve.The passage identifies reward-conditioned CFG extrapolation and its breakdown as questions for future work.

II. RELATED WORK

The pipeline builds on flow matching and classifier-free guidance, expert iteration, CRPO preference optimization, music representations, and open preference datasets. Its human-agent engineering loop follows established AI-driven research workflows.

  • Flow matching for audio generation: FluxAudio-S uses Flux-style flow matching, while classifier-free guidance combines conditional and unconditional inference passes to amplify conditioning.Flow matching learns a continuous-time velocity field whose Euler integration maps prior noise to data.
  • Self-improvement via expert iteration: Expert iteration alternates sampling and fine-tuning on top-quality samples; this pipeline uses one reward- and CLAP-text-filtered round.The formulation draws on ExIt and ReST.
  • Preference optimization: CRPO adapts direct preference optimization to text-to-music by constructing preference pairs with a CLAP-text scorer, and this pipeline uses the same procedure.DPO fits a policy directly to pairwise preferences without training a separate reward model.
  • Music representations and preference data: The ranker draws on contrastive and self-supervised music encoders and pools Music Arena, MusicPrefs, AIME, and SongEval with a RankNet pairwise logistic loss.Examples of encoders include LAION-CLAP-Music and MERT-v1-330M.
  • AI-driven research workflows: The human-agent loop follows AI-Driven Research for Systems: a human-defined objective is iterated against a programmatic evaluator.Related examples include FunSearch and AlphaEvolve, with full disclosure in AI Workflow Disclosure.

III. PROPOSED METHOD

The proposed method describes the backbone, score-conditioning head, and preference ranker, then details the training and inference pipelines. The deployed system uses v1 for Stages 1–2 and switches to v2 at Stage 3 through cross-loading.

  • Method overview: The method covers the backbone, score-conditioning head, preference ranker, training pipeline, and inference procedure.The pipeline is summarized in Fig. 1.
  • Method overview: The deployed system trains as v1 through Stages 1–2 and switches to v2 only at Stage 3 via cross-loading.The passage states that this design is justified in Section IV-B.
  • Score-conditioning architecture: Table I compares score-conditioning architectures on the Jamendo-100 validation set using CLAP, SCORE-r, and ∆OUT.SCORE-r measures Pearson correlation between input s and output reward, while ∆OUT measures the mean output-reward difference between s=+1.5 and s=−0.5.

A. Backbone · B. Score-Conditioning Head

The system uses the 120 M-parameter FluxAudio-S flow-matching transformer as its efficiency-track backbone, operating on 1D-Mel VAE latents with text and pooled LAION-CLAP conditioning. A learned reward scalar is added as a parallel conditioning input through a Fourier-feature and MLP pathway, with five injection strategies compared on Jamendo-100.

  • A. Backbone: FluxAudio-S is a 120 M-parameter Flux-style flow-matching transformer used as the generative backbone.The unconditional MeanAudio checkpoint is designated as the challenge’s efficiency-track baseline.
  • A. Backbone: The backbone operates on 1D-Mel VAE latents at 44.1 kHz for approximately 10-second clips.
  • A. Backbone: Text conditioning uses T5-Large cross-attention, while pooled LAION-CLAP features enter through adaptive layer normalization.
  • B. Score-Conditioning Head: The reward scalar s ∈R enters as a second conditioning input parallel to text.
  • B. Score-Conditioning Head: The scalar is mapped to a 448-d embedding through Fourier features and an MLP with a zero-initialized final projection.This initialization makes the generator identical to the unconditional backbone at the start of training.
  • B. Score-Conditioning Head: Five reward-injection strategies were compared on Jamendo-100, a 100-clip MTG-Jamendo holdout.
  • B. Score-Conditioning Head: InputAdd (v2) broadcasts the reward embedding to every audio latent at the input projection via z_i ← z_i + e_s.

C. Pairwise Preference Ranker · D. Training Pipeline

TuneJury is a twin pairwise ranker that converts audio and optional text into a quality score from multimodal embeddings and human A-vs-B preferences. The training pipeline uses instrumental MTG-Jamendo clips across score-conditioned SFT, expert iteration, and CRPO preference tuning, with approximately 40 GPU-hours of total compute.

  • C. Pairwise Preference Ranker: TuneJury maps an audio clip and optional text prompt to one quality scalar using CLAP-Music and MERT representations.Its released CLAP+MERT branches concatenate 512-dimensional audio, 1024-dimensional audio, and 512-dimensional text embeddings.
  • C. Pairwise Preference Ranker: The ranker uses RankNet pairwise logistic loss on human A-vs-B preferences from four sources, training an MLP on approximately 22K pairs.Approximately 2K pairs are held out, and the score head is 2048→1024→512→256→128→1 with BatchNorm, ReLU, and 0.5 dropout.
  • D. Training Pipeline: The generator pipeline has three stages: score-conditioned SFT, expert iteration, and CRPO, all trained on the same Demucs-separated instrumental MTG-Jamendo stem.Only data weighting and loss change between stages.
  • D. Training Pipeline: The dataset begins with approximately 55K MTG-Jamendo tracks, yielding approximately 535K ten-second clips with three ranker-derived reward columns.Using full-mix reward instead of instrumental-stem reward produced FAD-CLAP 0.515 versus 0.337 for instrumental-stem reward under otherwise identical SFT settings.
  • D. Training Pipeline: Stage 1 trains GlobalAdaLN score-conditioned SFT from scratch, while Stage 2 fine-tunes on the top decile of self-generated clips selected by reward and CLAP-text similarity.Stage 2 retains 64 of approximately 630 sampled clips, with reward mean +1.05, and oversamples them 5× into the training mixture.
  • D. Training Pipeline: Stage 3 switches to InputAdd, transfers all 203 shape-matched keys from the expert-iteration checkpoint, and applies CRPO to 2,000 preference pairs.Pairs match high-CLAP and low-CLAP generated samples under the same prompt; training uses β=2000, λFM=1.0, lr 10−6, and 5K updates.
  • D. Training Pipeline: The complete SFT, expert-iteration, CRPO, and ranker-training pipeline requires approximately 40 GPU-hours on one NVIDIA RTX A5000.This is the reported total compute for the full pipeline.

E. Inference Procedure

Inference uses joint classifier-free guidance on text and reward with fixed validation-selected scalars, then submits two seed-varied configurations sharing the same backbone and post-processing pipeline.

  • Joint classifier-free guidance: Joint classifier-free guidance lifts text and reward conditioning together relative to the doubly-unconditional baseline.The shared guidance scalar applies jointly to both conditioning signals.
  • Joint classifier-free guidance: w=4.0 and s=5.0 are fixed at inference, selected on the SDD-100 validation set rather than swept.The score scalar s=5.0 exceeds the training-time reward-score maximum of +2.76 on the train split.
  • Submitted configurations: Two submitted configurations share backbone weights and post-processing, differing only in inference random seed: 42 for Sub. 1 and 55 for Sub. 2.Both configurations use the same inference pipeline apart from the random seed.

IV. EXPERIMENTS

The experiments report internal validation on SDD-100 against the SDD-706 reference using FAD-CLAP and CLAP score under the official objective-metric protocol. They also define Reward evaluation and a shared single-value inference setup for the ablation results.

  • Experimental Setup: Internal validation uses SDD-100 prompts evaluated against the SDD-706 reference with FAD-CLAP and CLAP score.Both metrics use the LAION-CLAP-Music checkpoint.
  • Ablation: The cumulative ablation follows the deployed chain across N=100 SDD prompts, with paired-t improvement testing over the previous row.Sub. 2 is the Seed-55 sibling of Sub. 1 and uses the same chain.
  • Experimental Setup: FAD-CLAP is distribution-level, whereas CLAP score and Reward are per-prompt metrics supporting paired-t tests.Reward is the preference ranker’s mean output.
  • Experimental Setup: Unless noted otherwise, rows use s=5.0, w=4.0, 25 Euler steps, a prefix prompt, seed 42, and 3×mdx extraction.These settings define the shared single-value inference protocol.

A. Cumulative Stage Ablation · B. Cross-Mechanism Ablation

The cumulative ablation identifies expert iteration as the only added step with paired-t significance on CLAP score or Reward, while later steps remain within paired-t noise. The cross-mechanism ablation separately evaluates score-conditioning variants across checkpoints, weights, and forward mechanisms, including the submitted configurations.

  • A. Cumulative Stage Ablation: Step 2, expert iteration, is the only pipeline addition reaching paired-t significance on either CLAP score or Reward.Marginal contributions are measured against the preceding row, beginning with the unconditional FluxAudio-S checkpoint as a reference baseline.
  • A. Cumulative Stage Ablation: Steps 3 and 4 leave per-prompt distributions within paired-t noise of the previous row.This agrees with the reported cumulative-stage ablation conclusion.
  • A. Cumulative Stage Ablation: Row 0 uses the MeanAudio-released unconditional FluxAudio-S checkpoint generated without score conditioning as a reference baseline.Pipeline steps are added sequentially from Row 1 onward.
  • B. Cross-Mechanism Ablation: The score-conditioning mechanism is treated as a separable knob from the trained weights.The factorial design crosses checkpoint stage, weights version, and forward-mechanism version.
  • B. Cross-Mechanism Ablation: The cross-mechanism ablation runs an 8-cell factorial over {SFT-only, Chain-end}, {v1 weights, v2 weights}, and {v1 forward, v2 forward}.SFT-only is post-Stage-1, while Chain-end is post-Stage-2 before CRPO.
  • B. Cross-Mechanism Ablation: The evaluation also includes the two submitted configurations, with Hybrid defined as post-Stage-3 CRPO over Chain-end v1 →v2.Cross-loading uses state_dict.load(strict=False), and v1 and v2 share an identical 203-key parameter graph.

C. Inference-Time Score Sensitivity

The inference-time score scalar is active on the SFT-only checkpoint but saturated on the deployed Hybrid checkpoint. Across s ∈[0, 6], Hybrid outputs remain essentially unchanged, while SFT-only Reward tracks the scalar monotonically within its training range.

  • Inference-Time Score Sensitivity: On SFT-only, Reward rises from +0.16 to +0.47 across s ∈[0, 2], with Spearman ρ=1.0.The score scalar therefore tracks output Reward monotonically within the training range.
  • Inference-Time Score Sensitivity: On submitted Hybrid Sub. 1, Reward varies by only 0.04 across s ∈[0, 6], with Pearson r≈0, indicating a saturated inference knob.The deployed value is marked at s=5, and the Hybrid curve is essentially flat in both Reward and FAD-CLAP.
  • Inference-Time Score Sensitivity: The two checkpoints diverge sharply: Hybrid Reward already reaches +0.54 at s=0, whereas SFT-only responds throughout its training range.This contrast supports training-time score conditioning as an effective conditioning axis but shows little remaining inference-time sensitivity after the Hybrid pipeline.

D. Engineering Observations · V. CONCLUSION

The engineering observations show that reward conditioning matters primarily during training, mechanism transfer is asymmetric, and expert iteration dominates the chain. The conclusion frames these findings around a 40 GPU-hour ATTM entry using TuneJury reward for conditioning and sample selection.

  • D. Engineering Observations: Training-time score conditioning improves FAD-CLAP by 0.025–0.040 absolute over the unconditional baseline at SFT.The deployed Hybrid checkpoint has a flat s response across s ∈[0, 6], indicating saturated inference-time steering.
  • D. Engineering Observations: Mechanism transfer is asymmetric: v1 →v2 stays within 0.02 Reward of the v1-native cell, whereas v2 →v1 collapses.The reported reverse cross reaches FAD-CLAP ∼0.69 and Reward ∼−0.50.
  • D. Engineering Observations: Expert iteration is the dominant chain contributor on v1, changing FAD-CLAP by −0.0362, CLAP by +0.028, and Reward by +0.496.These deltas compare SFT-only with Chain-end on the v1 chain.
  • D. Engineering Observations: The v2 chain regresses under the same axis, with FAD-CLAP moving from 0.4442 to 0.4695 and Reward from +0.282 to +0.244.The passage attributes this to v2 expert iteration plus CRPO not converging cleanly under the available budget.
  • V. CONCLUSION: The submission used 40 GPU-hours on the 120 M-parameter FluxAudio-S baseline for the ICME 2026 ATTM Grand Challenge efficiency track.TuneJury supplied a learned human-preference reward used for both training-time conditioning and sample selection.
  • V. CONCLUSION: TuneJury reward conditioning is identified as a functional steering axis, with FAD-CLAP improving by 0.025–0.040 at SFT.The conclusion passage states that this effect is absorbed into the weights by chain-end.

AI WORKFLOW DISCLOSURE

The workflow used Claude Code CLI under human-directed development, with authors retaining responsibility for all substantive research choices, execution, validation, and supervision. The agent handled most line-level implementation and manuscript-production tasks, while authors reviewed commits and gated Overleaf pushes.

  • Tool: Claude Code CLI used Anthropic’s Claude Opus 4.6 and 4.7 through conversational natural-language requests, without an autonomous evaluation budget.Tool use was conversational rather than autonomous against a fixed objective.
  • Direction (human): The human authors proposed all architectural, training-data, evaluation, and post-processing choices, ran every job, and validated the results.The agent mainly mapped author-described procedures onto existing literature and suggested baseline hyperparameters during review.
  • Implementation (agent): The agent wrote most line-level code, drafted and edited the manuscript, maintained the bibliography, and tuned LaTeX layout.Implemented components included score-conditioning heads, expert-iteration sampling and filtering, the CRPO loop, post-processing scripts, evaluation harnesses, and figure source.
  • Supervision: The authors reviewed every commit, revised manuscript edits, and gated all Overleaf pushes.This supervision completed the human-agent split for transparency.
Loading 2606.21670v1…