Source-linked AI summary
Libretto: Giving LLM Agents a Sense of Musical Structure
Yichen Xu
TL;DR
Existing generative music systems produce audio that is difficult to inspect, edit, and diagnose structurally. Libretto provides an LLM-facing symbolic framework with readable representations, corpus-calibrated structural evaluation, retrieval, and revision, supporting multiple tasks with reported gains in gap filling and full-piece generation.
Problem
Audio-generation outputs expose limited note timing, voice assignment, phrase structure, harmonic motion, repetition, and local edit boundaries, while evaluation may not explain structural failures.
Method
Libretto converts music into an LLM-readable grammar and corpus-calibrated structural space, guiding an agent through retrieval, generation, diagnosis, and iterative revision.
Results
The loop raises gap-filling pass rate from 12% to 39% and full-piece pass rate from 62% to 94%, while retrieval triples full-piece generation from 25% to 75%.
Takeaways & Limitations
The same representation-evaluation loop supports gap filling, full-piece generation, gradual morphing, and educational music generation without training a new music model.
Takeaways & Limitations
The structural axes can be refined, expanded, and pruned, and the proposed reinforcement-learning direction remains open.
Abstract
from arXiv · showhide
Generative music systems can now produce impressive audio from text prompts, but audio outputs are difficult to inspect, edit, and diagnose as musical structure. We introduce Libretto, an agent-facing framework for symbolic music generation and revision. Libretto uses an LLM-native grammar with explicit onset slots, voices, and bar-level organization, then evaluates each piece in a corpus-calibrated statistical space over rhythm, harmony, melody, texture, form, and variation. The same structural axes support retrieval, diagnosis, copy-risk control, and iterative self-revision. Across gap filling, reference-guided full-piece generation, gradual morphing, and educational music generation, Libretto turns symbolic music from a raw token sequence into a measurable and editable object for language-model agents.
1 Introduction
Libretto addresses the difficulty of inspecting, evaluating, and revising generated music by representing symbolic pieces in an editable LLM-native grammar and a corpus-calibrated space of interpretable structural axes. An agent combines retrieval, measurement, and iterative self-improvement to support multiple symbolic-music applications, including longer-form multi-voice generation.
- Motivation: Generative audio systems make music creation accessible but produce outputs that are difficult to inspect as musical structure.Commercial systems can generate complete audio from short prompts, while research systems demonstrate strong text-conditioned audio generation.
- Motivation: Symbolic music provides an editable representation, but existing evaluation methods often fail to explain which structural property requires revision.Common measurements include human preference, likelihood, prompt adherence, contrastive audio-text alignment, and learned aesthetic prediction.
- Framework: Libretto represents pieces with explicit onset slots, voices, and bar-level organization, then measures rhythm, harmony, melody, texture, form, and variation in a corpus-calibrated statistical cloud.The grammar makes timing and structure readable and locally editable, while the structural axes describe each piece relative to existing music.
- Agentic composition: The composition agent retrieves musical concepts and examples, generates a candidate, evaluates structural axes, and revises it using musician-readable feedback.The system combines knowledge bases, retrieval, measurement, and iterative self-improvement.
- Applications: Libretto supports gap filling, reference-guided generation, gradual musical morphing, educational generation, and longer-form multi-voice pieces typically spanning around 100 bars.The framework targets specific music-theory concepts and extends beyond short fixed-length settings common in prior work.
2 Related Work
Related work spans symbolic music generation and representation, LLM-based symbolic composition, and text-conditioned audio generation with automatic evaluation. These directions establish relevant model families and comparison points while differing in how directly they expose musical structure for editing and diagnosis.
- Symbolic music generation and representations: Symbolic music generation has been studied across representations, model families, and evaluation protocols, with Music Transformer generating long-term structure, motif continuation, and coherent accompaniment.The paper points to Ji et al. [2023] for a broader survey and highlights relative self-attention as a mechanism used by Music Transformer.
- LLMs and agentic symbolic composition: Recent work investigates whether large language models can understand and generate symbolic music as text, including ChatMusician’s ABC-notation training and evaluation of generation and music-theory understanding.ChatMusician continues pretraining and fine-tuning LLaMA2 on ABC notation and music-language data, treating music as a second language.
- Text-to-audio generation, benchmarks, and evaluation: Text-conditioned audio generators offer an important comparison point, with MusicGen supporting text and melody conditioning but not directly exposing symbolic structure for editing or diagnosis.MusicGen models compressed audio tokens with a single-stage Transformer and produces strong audio outputs.
3 Methods
Libretto is an agent-facing text interface for symbolic music that represents pieces in an LLM-readable grammar and evaluates them in a corpus-calibrated structural space. The framework combines structural measurement, retrieval, copy-risk checks, and bounded revision across several music-generation tasks.
- Grammar: Libretto converts MIDI-like note structure into plain text with headers, voice declarations, bar blocks, chord labels, and slot-based note tokens.Tokens specify pitch, onset slot, and duration slot, using integer timing slots rather than floating-point time.
- Structural axes: Each piece receives a 29-axis fingerprint spanning rhythm, harmony, melody, texture, form, and within-song variation.Axes are computed from grammar tokens, pruned for redundancy, and converted to percentiles against a frozen 314-song corpus.
- Agent loop: Generation follows a bounded generate–measure–revise loop that parses candidates, computes fingerprints, checks copy risk, applies task gates, and returns musician-readable feedback.Feedback describes musical tendencies to adjust rather than exposing raw metric identifiers or exact numerical targets.
- Knowledge bases and retrieval: Retrieval grounds composition in corpus-attested concepts and examples, while a separate pedagogical theory knowledge base supports educational generation.The composing knowledge base covers harmony, groove, melody, voicing, form, and jazz-specific techniques.
- Copy and novelty: Libretto separates idiomatic similarity from direct copying by comparing bar-aligned onset–pitch pairs with retrieved examples, corpus matches, and hidden gap-filling answers.This note-level check is stricter than comparing chord labels or bar-level summaries.
- Applications: The shared framework supports gap filling, genre-conditioned full-piece generation, gradual morphing, and educational drills with task-specific structural, contextual, validity, and novelty requirements.Gap filling must fit local context and avoid copying; educational drills must satisfy constraints, detect the target concept, and differ from the shown example.
4 Experiments
Experiments validate Libretto’s grammar, structural axes, and calibrated gates before testing four applications. Results show that iterative revision and retrieval improve structural pass rates, while examples demonstrate readable diagnosis, stylistic morphing, and constrained education generation.
- Representation and measurement: The grammar is exact for pitch and voice, grid-faithful for timing, and explicit about abstracted musical information.Libretto makes note timing local, avoiding duration accumulation and downstream onset shifts at the cost of more characters.
- Representation and measurement: The 29-axis fingerprint captures weakly related structural dimensions and represents genre as a soft stylistic mixture rather than a hard category.Jazz and electronic occupy higher corpus-percentile regions on selected harmonic-complexity and self-similarity axes than folk and classical, with gaps of roughly 30–50 percentile points.
- Gate calibration: The structural, genre-fit, copy-risk, and novelty gates are calibrated against real or acceptable music to avoid rejecting naturally distinctive human music.The degeneracy gate permits a calibrated extreme-axis budget, while genre fit is calibrated per genre and copy risk uses real-song overlap statistics.
- Application outcomes: 39% versus 12% for gap filling and 94% versus 62% for full-piece generation are the loop’s pass rates with versus without iterative revision.Retrieval instead triples full-piece pass rate from 25% to 75%; in education, explicit scale, rhythm, and challenge constraints make retrieval add little.
- Qualitative demonstrations: A jazz continuation passes with copy risk 0.251, answer overlap 0.145, and beat alignment 98%, while examples also demonstrate gradual electronic-to-folk morphing and fresh theory-constrained drills.The education example uses E harmonic minor with 2.2% out-of-scale notes and copy-vs-shown score 0.065.
- Application outcomes: 33 of 51 gap-task songs improve, raising the pass count from 6 to 20, while full-piece revision changes six rows and makes five cross the pass gate.The loop acts selectively where structural gates expose a concrete failure rather than improving every piece indiscriminately.
5 Conclusion
Libretto makes symbolic music readable, measurable, and revisable for LLM agents by representing pieces in a corpus-calibrated structural space. Its structural axes support diagnosis, retrieval, and revision across multiple generation settings, while remaining open to refinement and optimization through agentic reinforcement learning.
- Conclusion: Libretto makes symbolic music readable, measurable, and revisable by an LLM agent.It presents an agent-facing framework for symbolic music generation and revision.
- Conclusion: A corpus-calibrated space over rhythm, harmony, melody, texture, form, and variation supports diagnosis, retrieval, and musician-readable revision.The framework avoids treating generation quality as a single subjective score.
- Conclusion: The same structural framework applies across gap filling, full-piece generation, gradual morphing, and educational music generation.Several directions remain open, including refining, expanding, or pruning structural axes while preserving meaningful variation and sufficient decorrelation.
- Conclusion: The feedback loop suggests agentic reinforcement learning that optimizes retrieval, editing, rewriting, and acceptance against structural rewards and copy-risk constraints.The passage identifies these actions as candidates for optimization in future music-generation systems.
Appendix · A Auxiliary results
The appendix presents auxiliary visual results that complement the main text, including genre fingerprints, application outputs, retrieval effects, axis profiles, and a gap-filling failure case. It also illustrates the Libretto grammar through real generated outputs.
- A Auxiliary results: Figure A.1 shows overlaid radar fingerprints across genres.
- A Auxiliary results: Figure A.3 shows how retrieval de-degenerates full-piece generation.
- A Auxiliary results: Figure A.4 visualizes per-piece axis profiles.
- A Auxiliary results: Figure A.5 presents an interpretable gap-filling failure case.
- A Auxiliary results: The appendix illustrates the Libretto grammar through three real generated outputs.
- A Auxiliary results: Figure A.2 summarizes representative generated outputs across four applications.
A. Education drill: single-voice pedagogy
The education drill presents a 20-bar, single-voice Piano excerpt in E minor on a 4/4, 16th-note grid at tempo 138. Its pedagogy makes harmonic-minor color, dominant-function harmony, and block-chord closure explicit in symbolic notation.
- Harmonic-minor pedagogy: The excerpt demonstrates E harmonic minor through the raised seventh D## and uses B7 as its dominant harmony.The passage explicitly identifies the raised seventh D## and the dominant B7.
- Format and setup: The drill is a 20-bar Piano-only excerpt in E minor, 4/4, at tempo 138 on a 16th-note grid.The structural header specifies one Piano voice, with the progression beginning in Em.
- Chordal closure: The closing material uses simultaneous Piano notes, including E4+B4+E5 and E4+G4+B4+E5, to represent block chords.The final notation shows stacked pitches with shared onsets and durations.
- Grid literacy: The notation exposes beat placement by mapping quarter-note beats to slots 1, 5, 9, and 13 in the 16th-note grid.The passage also defines >4 as a quarter note and >1 as a sixteenth note.
B. New generation: funk/soul multi-voice band
Libretto generates a 96-bar funk/soul piece in G minor with five declared voices and a 4/4, 102-BPM, sixteenth-grid structure. The excerpt demonstrates interlocking off-beat guitar and keyboard hits, syncopated bass, and lead lines over a Gm7–Cm7–D7 vamp.
- Piece realization: Bars 9–12 show four active voices interlocking over a Gm7–Cm7–D7 vamp.The excerpt contains off-beat guitar stabs, seventh-chord keyboard comping, syncopated bass with pickups, and a lead line.
- Piece specification: The generation specifies G minor, 4/4 meter, tempo 102, a sixteenth-note grid, 96 bars, and five voices: BASS, GTR, KEYS, HORNS, and LEAD.The declared voices are listed as BASS, GTR, KEYS, HORNS, and LEAD.
- Rhythmic mechanism: GTR and KEYS hits at slots 3, 7, 11, and 15 create the funk push between main beats.These positions are off-beat sixteenth-grid slots, and +-joined pitches encode chord voicings.
C. Gap filling: anchored multi-voice fill
Libretto fills a generated region within an existing song’s multi-voice structure using a 9-voice adaptive-grid grammar. Anonymized source-track names preserve the original voice set while keeping gap filling leakage-clean.
- Anchored multi-voice fill: Gap filling generates a region inside an existing song’s voice set while preserving its original multi-track structure.The excerpt uses a 9-voice adaptive-grid grammar.
- Anchored multi-voice fill: Anonymized labels such as Part2 and Part6 preserve voice structure while preventing leakage from real source-track names.These labels denote anonymized real source tracks used for leakage-clean gap filling.
- Anchored multi-voice fill: The representation specifies key, meter, tempo, adaptive grid, bar count, voices, and harmonic events including mid-bar changes.The example is in Bb major, 4/4, at tempo 200, with a 16th-note adaptive grid across 27 bars.
B Metric Definitions
The appendix defines Libretto’s structural axes, percentile fingerprint, copy-risk score, and calibrated gates from parsed grammar tokens. These quantities are descriptive rather than aesthetic and cover rhythm, harmony, melody, texture, form, and within-song variation.
- Notation and primitives: Each parsed note event records bar, onset, duration, MIDI pitch, pitch class, and voice, with population means, standard deviations, and normalized entropy used in metric definitions.The notation defines P as the set of parsed note events and D as the duration vector.
- Rhythm and harmony: Harmony metrics use duration-weighted pitch-class mass, half-bar chord prominences, bass-root transitions, Chord Change Rate, and diminished/augmented pitch-class color.Chord Change Rate counts adjacent nonempty half-bars with different chord prominences; pitch classes are interpreted modulo 12.
- Melody and form: Melody is selected from voices using mean pitch, density, and onset conditions, then characterized through pitch intervals, nonzero-motion indices, and Melody-Voice Range.The melody line takes the highest MIDI pitch at each selected melody-voice onset.
- Rhythm and harmony: Rhythm metrics include Onset Position Entropy over quarter-beat onset bins and Density Variability over per-bar note counts.Density Variability is defined as the standard deviation of bar note counts divided by their mean.
- Variation and calibration: Within-song variation compares windowed axis vectors against corpus standard deviations, while percentile fingerprints map axis values to a frozen 314-song corpus and flag coordinates at or beyond the 5th or 95th percentiles.An axis is a degenerate extreme when pct_a(P) ≤5 or pct_a(P) ≥95.
- Copy risk and gates: Generated pieces pass shared gates when n_extreme(P) ≤ C^1_g, fit(P, g) ≥ F_g, and copy risk(P) < T_g.The copy-risk representation uses rounded onset-and-MIDI-pitch pairs grouped by bar, and T_g is calibrated from real songs by genre.