MusicTwins: Training Audio-Language Models to Listen like Their Persona

1 INTRODUCTION

• Existing person-modeling approaches represent biography, views, personality, style, and behavior but do not explicitly model how individuals respond to music.Available music resources also do not evaluate single-listener writing style or musical perception, association, preference, affect, and behavior.Source: §1 · pp. 1–2
• Whether audio-language models can predict one person’s responses to textual and musical stimuli consistently with their broader identity remains underexplored.People differ in musical perception, associations, preferences, affect, and behavior, alongside background, views, personality, and writing style.Source: §1 · p. 2
• MusicTwins represents background, views, personality, and writing style alongside musical background and five dimensions of response to music.Related systems address selected persona traits or musical tasks, but not fidelity to a specific person’s broader identity across text and music.Source: §1 · p. 2
• MusicTwins combines persona evidence from real-world writing, links musical responses to recordings, and trains an audio-language model to draw on that representation.Its evaluation protocol tests agreement with evidence, generalization to new stimuli, and acknowledgment of unavailable information.Source: §1 · p. 2

2.1 Person Modeling in Response to Textual Stimuli

• Prior text-based person modeling uses profiles, writing, interviews, and surveys to represent traits and behavior; this work retains background, views, personality, and writing style.The review organizes persona evidence using dispositional traits, characteristic adaptations, and self-defining life narratives situated in social and cultural context.Source: §2.1 · p. 2

2.2 Person Modeling in Response to Musical Stimuli

• Musical responses vary with personality, self-views, cognitive ability, sociocultural background, age, expertise, familiarity, associations, emotional dispositions, listening motivations, and prior mood.These findings motivate retaining diverse person-specific evidence and evaluating fidelity to the same individual across textual and musical stimuli.Source: §2.2 · pp. 2–3

2.3 Personalized Music Systems

• Personalized music systems cover recommendation, emotion recognition, and listener behavior, often relying on selected characteristics or annotated musical attributes.Examples include systems using demographics, listening histories, affect labels, or combined audio, metadata, and conversation history.Source: §2.3 · p. 3
• Related work includes systems for individual-context inference, symbolic chord selection, and persona-conditioned singing critique, but none evaluates broader identity fidelity across textual and musical stimuli.The related-work review also situates these topics within a broader review of audio-language models and their training and evaluation resources.Source: §2 · p. 2, §2.3 · p. 3

3.1 Formulation and Design Rationale

• MusicTwins models what a target person would write in response to textual and musical stimuli, using structured evidence about how they would respond.The representation prioritizes dimensions supported by prior work and observable in available data, with emphasis on music.Source: §3.1 · p. 3

3.2 Dimensions of the Representation

• The representation covers background, views, personality, style, and a musical profile.These dimensions include biographical and sociocultural information; beliefs, interests, and preferences; character traits; and recurring features of written expression.Source: §3.2 · p. 3

3.3 Musical Profile

• The musical profile separates musical background from responses organized as perception, association, preference, affect, and behavior.Behavior is kept distinct because repeated listening may reflect purposes other than liking; situational context is optional and separate from the persona dimensions.Source: §3.3 · p. 3

3.4 Situational Context

• Situational context can include prior mood, concurrent activity, and listening purpose, and is included in the textual stimulus where applicable.Keeping context separate avoids treating behavior as direct evidence of other categories, since the same behavior may have different purposes.Source: §3.4 · p. 4, §3.3 · p. 4

4.1 System Architecture

• The architecture gives the model a persistent persona summary, retrieves narrower personal memories when relevant, and trains a shared model to apply persona evidence across individuals.Question–answer–audio tuples preserve the person’s wording with minimal editing for training.Source: §4.1 · p. 4, §4.2 · p. 4

4.2 Dataset Creation Pipeline

• The data pipeline summarizes writings for the system prompt, turns writings into searchable question-and-answer entries, and links music-related training questions to recordings.Questions refer to audio indirectly, while false-premise or unsupported-history questions receive brief refusals in the persona’s voice.Source: §4.2.1 · p. 4, §4.2.2 · p. 4, §4.2.3 · p. 4

4.2.4 Corpus

• The corpus contains 85 authors selected from the Reddit Language–Genre release, with up to 500 writings per author and musical identities matched to classical recordings.The original release analyzed over 500,000 non-music texts from nearly 5,000 authors.Source: §4.2.4 · p. 4

4.3 Memory

• Each persona’s memory stores retrieval entries with recordings, indexed using BM25, normalized dense text embeddings, and CLAP audio embeddings.The CLAP embeddings map audio and text into a shared space learned from audio-caption pairs.Source: §4.3 · p. 4

4.4.1 Learning Objectives and Evidence Distribution

• The four learning objectives are recall from memory, interpolation from the system prompt, extrapolation by listening without a direct precedent, and anti-hallucination through declining unsupported personal questions.Together, they teach use of relevant available evidence and refusal when evidence does not establish the answer.Source: §4.4.1 · p. 5
• The 1,276 question–answer pairs are divided into 416 extrapolation, 381 interpolation, 207 recall, 104 recall-and-interpolation, and 168 anti-hallucination pairs.The split is 1,060 training and 216 validation pairs, with no persona shared across splits; similar writings are clustered to reduce contamination.Source: §4.4.1 · pp. 5–6

4.4.2 Models and Training Procedure

• Training first uses supervised fine-tuning on recall, interpolation, and anti-hallucination examples, then adds extrapolation and more interpolation questions.The initial response format describes audio, assesses retrieved-memory relevance, cites supporting evidence, and answers in the persona’s voice.Source: §4.4.2 · p. 6
• The final training stage applies group-relative policy optimization, with an LLM judge scoring answers and additional personality and writing-style rewards for answers scoring at least 4 out of 5.The procedure samples eight responses per question and updates adapters using group-normalized advantages and a KL penalty.Source: §4.4.2 · p. 6

5.1 Evaluation Set

• The dedicated musical-response set contains 600 question–answer–audio tuples from ten held-out personas, covering all four objectives and five response dimensions.Most questions attach one recording; 64 attach two to four.Source: §5.1 · p. 6

5.2 Tasks and Metrics

• The protocol measures background correctness from memory-only evidence, correctness on view questions, and writing-style similarity across answerable questions.Style is average STAR similarity to each author’s centroid across retained writings.Source: §5.2 · p. 6
• Personality fidelity is 100 × (1 − MAE) between OCEAN scores from a 44-item interview and persona references, averaged across five traits and ten personas.Source: §5.2 · p. 6
• Musical-response performance equally weights recall, recall-and-interpolation, interpolation, extrapolation, and anti-hallucination, with each objective group contributing 20%.The first four use LLM-judged similarity rescaled to 0–100; anti-hallucination uses accuracy on the same scale.Source: §5.2 · p. 6
• Anti-hallucination accuracy tests whether responses reject unsupported premises or acknowledge missing information without fabricating personal details.Source: §5.2 · p. 6
• General music understanding is pooled accuracy on MuChoMusic, HumMusQA, and 206 music-only MMAR questions.Persona-related confidence intervals resample personas with their questions; benchmark intervals resample questions.Source: §5.2 · p. 6

6.1 Overall Results

• 5.7 score points (95% CI [3.8, 8.2]) higher overall and 11.2 score points (95% CI [8.1, 15.5]) higher on musical response versus the base model.Style rose from 77.9 to 80.5, and personality fidelity rose by 8.5 score points; background declined by 0.8, while music-understanding gains had confidence intervals including zero.Source: §6.1 · pp. 6, 8, Table 1 · p. 7

6.2 Validation across Training Stages

• Validation recall rose from 37.3 to 64.2, recall-and-interpolation from 57.4 to 88.9, and anti-hallucination accuracy from 35.7 to 85.7 after stage one.Extrapolation fell from 42.2 to 29.2, indicating a trade-off between explicit evidence use and answering without a direct precedent.Source: Figure 3 · p. 7, §6.2 · p. 8
• Stage two raised extrapolation from 29.2 to 41.1 and interpolation from 32.5 to 42.5, while the equal-weight mean stayed nearly unchanged at 60.1 to 60.0.Recall-and-interpolation and anti-hallucination declined during this stage.Source: Figure 3 · p. 7, §6.2 · p. 8

Third stage: refining responses through reinforcement

• GRPO achieved the highest equal-weight mean, rising from 60.0 to 67.9; interpolation reached 45.4, extrapolation 50.0, and anti-hallucination accuracy 92.9.Recall returned to 64.2 and recall-and-interpolation to 87.0; the passage characterizes this as a better balance across objectives.Source: Figure 3 · p. 7, Third stage: refining responses… · p. 8

6.3 Evaluating Person-Specific Responses to Music

• Persona conditioning narrows plausible answers to subjective questions about musical associations and affect, allowing comparison with a more narrowly defined reference.Source: §6.3 · p. 8

• The framework treats personas as stable despite changing taste, emotion, and behavior; evidence relies on proxies, one backbone, selected personas, and primarily classical music.Training used up to 30 seconds of audio per recording; broader genres and populations remain future work.Source: §7 · p. 8

8 CONCLUSION

• MusicTwins combines background, views, personality, and writing style with musical background and five dimensions of response: perception, association, preference, affect, and behavior.The representation is distributed across a persistent summary, per-persona memory, and a shared audio-language model trained on four objectives.Source: §8 · p. 8

A.1 Audio-Language Models and Music

• Existing music datasets and benchmarks assess general musical understanding but do not evaluate whether a model reproduces a specified listener’s responses or interpretations.Source: §A.1 · p. 15

B.1 Question–Answer Pair Requirements

• Question–answer pairs must attribute claims to the author’s writing, preserve wording and qualifications, ask natural questions, attach the needed audio, and ground annotations in the source.Recordings are referenced deictically rather than named, except in identification tasks.Source: §B.1 · p. 15

B.2 Anti-Hallucination Pairs

• Anti-hallucination probes ask about a persona’s history with recordings when the work, composer, or requested detail is unsupported, and include false-premise questions.An LLM creates the question and a brief refusal in the persona’s voice from verbatim writings.Source: §B.2 · p. 15

B.3 Synthetic Persona Example

• The synthetic persona example separates background, views, personality, style, and musical profile, while instructing the model to treat the profile as evidence and admit unsupported facts.The example musical profile includes playing history, listening habits, associations, and preferences.Source: §B.3 · pp. 15–16

B.4 Textual Stimuli and Example Answer

• The user input combines attached audio with a text block containing personal memories and the question; the worked example shows memory relevance and profile evidence supporting a first-person answer.The illustrated plain-memory format contains three entries; retrieval can supply up to ten.Source: §B.4 · p. 16

B.5 Example Question–Answer–Audio Tuples

• Example tuples illustrate extrapolation, interpolation, recall, and anti-hallucination, pairing audio and persona evidence with answers that retain the persona’s style.Examples include a preference between performances, an emotional response, a listening-history estimate, and refusal to accept an unsupported premise.Source: §B.5 · pp. 16–17

D RETRIEVAL SYSTEM

• Persona memory stores minimally edited writings, questions, keywords and entities, and associated audio; retrieval combines text matching with an audio extension and returns at most ten memories.Text retrieval uses BM25 and dense candidates; the audio extension adds CLAP-ranked candidates.Source: D RETRIEVAL SYSTEM · pp. 17–18

E.1 Background and Views

• Background and views are evaluated on 150 questions about facts removed from the persona prompt, with correctness averaged within each persona and equally across ten personas.The judge assigns a 1–5 correctness score.Source: §E.1 · p. 18

E.2 Personality

• Personality is assessed through 44 open-ended interview items per persona, converted to OCEAN ratings and scored as 100 × (1 − MAE) against dataset-derived references.Scores are averaged equally across five traits and ten personas.Source: §E.2 · p. 18

E.3 Style

• Style is measured by cosine similarity between answer embeddings and each author’s retained-writing centroid, using 4,798 writings that are not held out from persona construction.STAR embeddings are used, and the similarity calculation does not use an LLM judge.Source: §E.3 · p. 18

E.4 Musical Response

• Musical responses are scored for similarity of meaning on a 1–5 scale across learning objectives and five dimensions: perception, association, preference, affect, and behavior.The 550 answerable questions are grouped by objective and primary musical-response dimension.Source: §E.4 · p. 18

E.5 Anti-Hallucination

• Anti-hallucination accuracy measures whether responses reject unsupported premises or acknowledge missing personal information; accepted AI-identity denials mean this score does not itself measure persona fidelity.The task uses 50 probes.Source: §E.5 · p. 18

E.6 Music Understanding

• Music understanding is evaluated on 1,713 questions pooled across three benchmarks, with separate results for unconditioned inputs and inputs paired with a held-out persona and textual memories.The benchmarks are MuChoMusic, HumMusQA, and the music questions from MMAR.Source: §E.6 · pp. 18–19

E.7 Bootstrap Confidence Intervals

• Confidence intervals use 10,000 paired percentile resamples, sampling personas for persona-related tasks and questions within benchmarks for music understanding.With ten personas, coverage is approximate; intervals do not account for training-run variability or systematic judge bias.Source: §E.7 · p. 19

F LLM JUDGE PROMPTS

• Judge prompts specify content-focused scoring: musical responses are compared for meaning, while anti-hallucination answers are classified by whether they refuse unsupported personal claims without fabricating details.The musical-response rubric penalizes generic answers that omit the reference’s specific choice, detail, judgment, or reason.Source: §F.3 · p. 19, §F.4 · p. 20

G.1 Supervised Fine-Tuning

• Audio Flamingo Next is fine-tuned with 4-bit NF4 QLoRA while freezing the audio encoder and connector and applying rank-16 decoder LoRA adapters to attention and feed-forward projections.The adapters use α = 32 and dropout 0.05.Source: §G.1 · p. 20
• Both supervised fine-tuning stages span three-epoch learning-rate schedules, while the stage-one checkpoint is taken after three epochs and stage two uses all training questions after capping interpolation and excluding extrapolation in stage one.Stage one caps pure interpolation at 100 questions; examples with supporting memories appear twice with shuffled memory order.Source: §G.1 · p. 20
• The training loss weights mean cross-entropy over final-answer tokens, selected evidence-decision tokens, and the full response target by 1, 0.5, and 0.25, respectively.Inputs are limited to 8,192 text tokens and 60 seconds of audio per example, with at most 30 seconds per recording.Source: §G.1 · p. 20

G.2 Reinforcement Learning and Checkpoints

• The final model uses GRPO step 100, with one policy update per rollout batch, eight sampled responses per prompt, and a KL penalty of 0.02 against the frozen initial policy.Training uses four prompts per batch, temperature 1.0, and a 512-token generation limit.Source: §G.2 · p. 20
• The reward maps agreement scores from 1–5 to 0–1, rewards correct refusals, and adds 0.15 each for personality and style matches when agreement is at least 4.Truncated responses or responses without the required final-answer marker receive zero reward.Source: §G.2 · p. 20
• Figure 4 organizes training objectives by whether a question asks about a fact and where answer evidence lies; facts include documented preferences and other personal responses.The caption identifies the dimensions to compare, but does not state outcomes for the categories.Source: Figure 4 · p. 20