ChatMusician: Understanding and Generating Music Intrinsically with LLM

• Current LLMs inadequately represent long-term contrapuntal dependencies and connections between musical notes and text descriptions.Source: §1 · p. 1
• ChatMusician is an open-source LLM designed to integrate intrinsic musical abilities for symbolic music generation and understanding.Source: §1 · p. 1
• The authors introduce MusicTheoryBench, a college-level symbolic music-understanding benchmark covering music understanding and reasoning.Source: §1 · pp. 1–2
• The authors release MusicPile, a 4B-token music-language corpus, along with the benchmark, code, model, and demo.Source: §1 · p. 2

2.1 Issues in Music Generation and Understanding

• Music-generation systems can produce short, poorly structured works; repetition supports perceived musicality, yet rule-based patterns lack flexibility and deep-learning works may lack structure.Source: §2.1 · p. 2
• MusicTheoryBench contrasts with audio-focused benchmarks by testing verbal comprehension, advanced music theory, and symbolic music reasoning.Source: §2.1 · p. 2

2.2 Music Representations

• MIDI sequences can be lengthy and require segmentation that limits composition continuity, while tokenization can introduce quantization errors and unstable rhythms.Source: §2.2 · p. 2
• ABC notation is plain text with shorter sequences than MIDI; it encodes repetition and structure and avoids quantization issues.Source: §2.2 · pp. 2–3, Figure 2 · p. 3

2.3 LLMs for Complex Problem-solving Tasks in Non-language Domain

• Prior LLM work addresses non-language tasks including mathematical reasoning, code generation, and game decision-making.Source: §2.3 · p. 3

3.1 Language Corpora Curation

• MusicPile combines general, instruction, chat, and music-focused data to support conversational interaction and inject musical abilities into LLMs.General corpora include music-term-filtered text; music knowledge and summaries are derived from track metadata and GPT-4-generated material.Source: §3.1 · p. 3
• MusicPile’s overview distinguishes synthesized data from examples with new GPT-4-prompted rationales.The table caption defines the symbols for synthesis and newly curated rationales.Source: Table 1 · p. 4

3.2 Music Score Corpora Curation

• The collected score corpus spans music from different world regions and supports six score-generation tasks and two score-understanding tasks.Generation conditions include chords, melody, motifs, musical form, and style; understanding tasks extract motifs and forms from input scores.Source: §3.2 · p. 4, Figure 3 · p. 4, Table 2 · p. 5

3.3 MusicTheoryBench

• MusicTheoryBench assesses advanced LLM music understanding through separate music-knowledge and music-reasoning concepts.Music reasoning infers unannotated musical elements such as harmony, keys, and rhythms; knowledge covers musical commonsense including theory, history, instruments, and cultural context.Source: §3.3 · pp. 4–5
• The benchmark was crafted from college-level textbooks and exams, then reviewed by musicians and labeled into knowledge and reasoning subsets.Questions were manually compiled in JSON and ABC string formats; half were initially delivered in Chinese and translated into English.Source: §3.3 · p. 5
• The benchmark contains 372 four-option questions, including 269 music-knowledge questions, 98 music-reasoning questions, and five held out for few-shot evaluation.Reasoning questions require detailed analysis and multi-step inference involving elements such as chords, melodies, scales, and rhythms.Source: §3.3 · pp. 5–6

4.1 Training Settings

• ChatMusician-Base starts from LLaMA2-7B-Base weights and uses continual pre-training followed by fine-tuning.Training used LoRA adapters and a maximum sequence length of 2048.Source: §4.1 · p. 6

4.2 Data Settings

• Fine-tuning uses a 2:1 ratio of music scores to music knowledge and summary data, selected to balance music generation, understanding, and MMLU performance.The authors first trained on 78K samples for 10 epochs, then on 1.1M music-score samples for two epochs while maintaining the ratio.Source: §4.2 · p. 6

4.3 Evaluation and Baseline Systems

• Evaluation compares GPT-3.5, GPT-4, and LLaMA-2, using 5-shot MMLU, zero-shot MusicTheoryBench accuracy averaged across five option shuffles, and human judgment for musicality.Generation also uses phrase-level repetition, parsing success, and average percentile metrics to assess structure, format accuracy, and controllability.Source: §4.3 · p. 6

5.1 Music Understanding

• ChatMusician-Base scores 40.2 and ChatMusician 39.5 in music knowledge, above GPT-3.5 (31.2) and LLaMA2-7B-Base (33.3); ChatMusician scores 26.3 in reasoning, above GPT-4 (25.6).GPT-4 leads music knowledge at 58.2, while ChatMusician-Base reaches 27.1 in reasoning; ChatMusician’s music-knowledge score is about 0.7 points below its Base model.Source: §5.1 · pp. 6–7, Figure 5 · p. 7

5.2 Music Generation

• ChatMusician’s music-generation approach uses ABC notation and reports improved musicality while integrating up to six conditional generation tasks.The section also presents a generated score with phrase-level repetition, motifs, and variations echoing a motif.Source: §5.2 · p. 7, §5.2.2 · p. 8

5.2.1 Compression Ratio of ABC Notation

• ABC notation averages 288.21 tokens per song and 5.16 tokens per second, around 38% of MIDI-based representations, and has the best compression ratio.The authors suggest its text compatibility reduces training costs and learning complexity.Source: §5.2.1 · pp. 7–8, Table 4 · p. 8

5.2.2 Musicality

• 76% of ChatMusician’s generated samples contain repeat signs, and listeners prefer its music to GPT-4’s in 76% of pairwise judgments.Pairwise preferences for ChatMusician over GPT-4 and GPT-3.5 are significant (p = 2.7 × 10−6 and p = 3.8 × 10−10, respectively).Source: §5.2.2 · p. 8, Figure 6 · p. 8, Table 5 · p. 8

5.2.3 Controllability

• ChatMusician and GPT-4 each achieve ABC parsing success rates above 90%, versus 65.4% for GPT-3.5; ChatMusician also outperforms GPT-3.5 and GPT-4 on all five evaluated tasks.GPT-4’s low score on alphabetic form-and-motif generation is attributed to malformed ABC outputs.Source: §5.2.3 · p. 9, Table 6 · p. 9, Figure 8 · p. 9

5.3 Language Ability

• ChatMusician and ChatMusician-Base score higher on MMLU than LLaMA2-7B-Base, which the authors interpret as no compromise to general language ability.The authors say the results appear to indicate some enhancement of general language abilities.Source: §5.3 · pp. 9–10, Table 7 · p. 10

5.4 Memorization Effect of ChatMusician

• On 500 sampled training examples, exact full-sequence matches occur in 0.02% and partial matches sharing at least 80% of tokens occur in 0.24%.The partial-match threshold is at least 80% shared tokens.Source: §5.4 · p. 10

6 Conclusions

• The study introduces ChatMusician as an open-source LLM for music reasoning and composition, with notable performance on music and language benchmarks.The conclusion describes this as a step toward integrating musical creativity within language models.Source: §6 · p. 10
• The authors emphasize LLMs’ potential as tools for music understanding and creativity, while highlighting untapped possibilities in music and artificial intelligence.Source: §6 · p. 10
• The release includes MusicPiles, MusicTheoryBench, and ChatMusician as resources for further research.Source: §6 · p. 10

Limitation

• The current model predominantly generates Irish-style music, and hallucinations and limited diversity in handcrafted instructions constrain open-ended generation.Source: Limitation · p. 10

Ethics Statement

• Hallucinations could mislead learners in music education, while memorized training data raises possible copyright-infringement concerns.The authors plan plagiarism detection and further alignment strategies to address these risks.Source: Ethics Statement · p. 10

A Introduction to ABC Notation

• ABC notation is a text-based system designed for human readability and simple typing without specialized fonts or software.It is especially popular for folk and traditional tunes.Source: A Introduction to ABC Notation · p. 14
• ABC compactness reduces storage and transmission overhead, and its structured text can be tokenized into numerical sequences or embeddings.The appendix also describes ABC as scalable for data collection and suitable for generative models.Source: A Introduction to ABC Notation · p. 14
• ABC notation makes generated music human-readable for feedback and iterative refinement, and can be integrated with other data modalities.Source: A Introduction to ABC Notation · p. 14
• An ABC file places tune metadata in headers before notation that defines the melody; the example is a D-major waltz in 3/4 with two repeated parts.The example specifies an eighth-note default length.Source: A Introduction to ABC Notation · p. 14

1. Pitch, Note Value, and Notation System

• The listed theory scope spans pitch, note values, rhythm, intervals, triads, seventh chords, and tonal analysis of chord progressions.Source: §1 · p. 15, §3 · p. 15, §4 · p. 15

14. Chromatic Scale

• The appendix also lists chromatic scales, musical forms, counterpoint, melody, and musical styles and genres as theory topics.Source: §14 · p. 15, §15 · p. 16, §16 · p. 16

20. Analysis Techniques

• Analysis techniques, orchestration and instrumentation, and acoustics and sound are additional listed areas.Source: §20 · p. 16

• The held-out MusicTheoryBench examples are shown with the prompt used in five-shot evaluation.Source: C Examples used in 5-shot evaluation… · p. 16

D Music Instruction Dataset Curation

• The Irishman dataset provides music SFT data with a control code for overall structure and an ABC-notation score.Source: D Music Instruction Dataset Curation · p. 16
• Control codes encode section count, bar counts, and pairwise edit-distance similarity; for example, E:5 denotes similarity 0.5.For each later section, an E entry records its similarity to an earlier section.Source: D Music Instruction Dataset Curation · p. 16

D.0.1 Musical form analysis algorithm

• Musical-form analysis maps pairwise section similarities to identical, variation, or different labels using thresholds of 8 and 6.Scores at least 8 indicate identical sections; scores from 6 to under 8 indicate variation; scores under 6 indicate different sections.Source: §D.0.1 · p. 16
• The form algorithm converts the similarity labels into an alphabetic form, reusing letters for repeated sections and adding a prime for variations.The example produces ABB′C.Source: §D.0.1 · pp. 16, 18
• The resulting alphabetic forms are mapped to traditional, extended, or compound musical-form terms.Examples include binary and ternary forms, verse/chorus structures, and compound binary or ternary forms.Source: §D.0.1 · p. 18

D.0.2 Motif extraction algorithm

• Motif extraction first separates the sample into sections using control-code length information, then processes each section's token sequence.Source: §D.0.2 · p. 18
• The motif procedure removes bars, chords, annotations, decorations, and embellishments before counting token-sequence occurrences.Source: §D.0.2 · p. 18
• It returns the most frequent token sequence, using sequence length to break ties.Source: §D.0.2 · p. 18

E Settings of Data Mixture

• Setting 18 was selected for balanced music understanding, music generation, and language understanding; scaling it to 1.1M samples produced reported setting 21.The downsampled mixture experiments used 52k- or 78k-sample datasets to fit roughly one-day runs.Source: E Settings of Data Mixture · p. 18, Table 8 · p. 19

F Details of Average Percentile Score Metric

• Initial scores use task-specific measures: chord or melody edit distance, musical-form set differences, and, for motif conditioning, motif longest common subsequence.The alphabetic/terminology musical-form and motif-conditioned task combines musical-form difference with motif overlap.Source: F Details of Average Percentile Score… · p. 19
• Each task's initial scores are normalized to a common scale using their within-task percentiles.A percentile indicates the share of initial scores in that task below a sample's score.Source: F Details of Average Percentile Score… · p. 19
• The metric averages each model's percentile scores for each task and reports the results in Figure 8.Figure 8 presents average percentile scores for the tested models by task.Source: F Details of Average Percentile Score… · p. 19