Source-linked AI summary
AI Song Contest: Human-AI Co-Creation in Songwriting
Cheng-Zhi Anna Huang, Hendrik Vincent Koops, Ed Newton-Rex, Monica Dinculescu, Carrie J. Cai
TL;DR
Musicians’ needs when partnering with sophisticated generative models for songwriting remain less understood, despite advances in music generation. This paper studies 13 teams and 61 participants, finding that teams often combined smaller models and curated large volumes of generated material. Teams also faced challenges controlling co-creation across multiple models.
Problem
Relatively little is known about what musicians need when songwriting with sophisticated generative models, despite prior evidence that partnering with them can challenge users’ agency and control.
Source: §1 · p. 1Method
The researchers analyzed survey responses from 13 musician/developer teams (61 participants) who used their chosen AI technologies to create songs.
Source: §1 · p. 1Results
Teams often broke songwriting into smaller components, combining models and curating voluminous generated snippets, which added model-centric iteration alongside creative work.
Source: §1 · pp. 1–2Takeaways & Limitations
The findings inform human-AI system design for complex co-creation tasks such as songwriting.
Source: §7 · p. 6Teams still faced deeper challenges controlling co-creation when working across a plethora of models.
Source: §5 · pp. 4–5
Abstract
from arXiv · showhide
Machine learning is challenging the way we make music. Although research in deep generative models has dramatically improved the capability and fluency of music models, recent work has shown that it can be challenging for humans to partner with this new class of algorithms. In this paper, we present findings on what 13 musician/developer teams, a total of 61 users, needed when co-creating a song with AI, the challenges they faced, and how they leveraged and repurposed existing characteristics of AI to overcome some of these challenges. Many teams adopted modular approaches, such as independently running multiple smaller models that align with the musical building blocks of a song, before re-combining their results. As ML models are not easily steerable, teams also generated massive numbers of samples and curated them post-hoc, or used a range of strategies to direct the generation, or algorithmically ranked the samples. Ultimately, teams not only had to manage the "flare and focus" aspects of the creative process, but also juggle them with a parallel process of exploring and curating multiple ML models and outputs. These findings reflect a need to design machine learning-powered music interfaces that are more decomposable, steerable, interpretable, and adaptive, which in return will enable artists to more effectively explore how AI can extend their personal expression.
1. INTRODUCTION
The paper examines what musicians need when co-creating songs with an open-ended range of AI models, addressing limited evidence about how people interact with these systems. Across 13 teams and 61 participants, it finds that teams often combined smaller models and large-scale generation with curation, adding model-focused iteration alongside creative work.
Research gap
- The study addresses limited knowledge of musicians’ needs when songwriting with sophisticated models, including desires for agency, control, and authorship.Prior work found these desires even when composing short, two-bar counterpoints with a deep generative model.Source: §1 · p. 1
Motivation
- Songwriting offers a lens on co-creation because it interleaves text, melody, harmony, and audio, creating challenges across multiple media.Source: §1 · p. 1
Study and contributions
- 13 teams (61 participants) completed a survey-based study of how they used AI to create songs and what challenges and coping strategies they encountered.Teams could choose their AI algorithms while making a song under three minutes long.Source: §1 · p. 1
Findings
- Teams usually broke songwriting into smaller components, combining multiple generative models rather than relying on large end-to-end models.Some teams generated many snippets first, then curated them manually; AI also added model-centric iteration loops alongside creative work.Source: §1 · pp. 1–2
Contributions
- The paper contributes patterns in songwriting with generative models, an analysis of challenges and coping strategies, and recommendations for human-AI system design.Source: §1 · p. 2
2. RELATED WORK
Prior music-AI work has advanced composition assistance and deep music generation, while research on human interaction with these models remains limited. This paper extends that work by examining how people assemble a broad, open-ended set of real-world models and resources when songwriting.
AI for music
- Music-AI research has addressed composition tasks ranging from counterpoint and harmonization to infilling, chord exploration, semantic controls, and instrument building.The cited work also explores symbolic and acoustic manipulation of scores and sound.Source: §2 · p. 2
Deep music generation
- Modern deep-learning models can generate full scores and music coherent across local and distant regions, while song-form research tackles hierarchical and multi-track generation.Source: §2 · p. 2
Human-AI co-creation
- Despite model progress, relatively little research examines human interaction during co-creation; prior studies report steering difficulties and desires for creative freedom, agency, and authorship.This paper extends prior work focused largely on a single tool by studying people assembling a broad set of models, datasets, and technology.Source: §2 · p. 2
3. METHOD AND PARTICIPANTS
The researchers studied teams participating in the AI Song Contest and analyzed survey responses about their songwriting workflows. The teams chose their AI technologies, and researchers used inductive thematic analysis to identify patterns in the responses.
Contest setting
- The study took place during the first three months of 2020 at the AI Song Contest, where teams could use any AI technology to make a song under three minutes.Participants completed a survey after creating an audio file and consented to publication of survey data.Source: §3 · p. 2
Participants
- Thirteen teams comprising 61 people participated; teams ranged from 2 to 15 members, with a median size of 4.Nearly three quarters of teams had one or two experienced musicians; many members had backgrounds in both music and technology.Source: §3 · p. 2
Analysis
- Researchers used inductive thematic analysis of survey responses about AI use, system development, workflow integration, and trade-offs between AI and musician-composed song parts.One researcher identified relevant text, and two researchers collaboratively iterated on relationships and themes.Source: §3 · p. 2
4. HOW DID TEAMS CO-CREATE WITH AI?
Teams commonly built songs from smaller musical components, combining models and human work in varied ways. They generated and curated samples, and some teams also used AI output as material for active back-and-forth composition.
4.1 Leveraging modular musical building blocks
- Most teams broke composition into musical modules and combined smaller models; teams that tried end-to-end generation turned to this approach after control and quality difficulties.Teams used between one and six unique model types, with a median of four, across components such as lyrics, melody, harmony, bassline, drums, and synthesis.Source: §4 · pp. 2–3, §4.1 · p. 3
4.2 Combining building blocks
- Teams combined model outputs through manual stitching, heuristics, pipelines, or joint models, with pipelines allowing refinement between stages.Some teams layered melodic lines into harmony; others generated lyrics before conditioned melody, or generated chords before melody and bassline.Source: §4.2 · pp. 3–4
4.3 Generate then curate
- Teams often generated many samples and curated them manually or with an automated filtering step before human selection.One team generated over 450 melodies and basslines, and another generated 10K death metal lyric lines; the process could be painstaking but also exhilarating.Source: §4.3 · p. 4
4.4 Active co-creation
- Some teams used AI material as a foundation, seed, or prompt for further composition, while iterating between human ideas and model outputs.One participant described priming the AI with chord progressions, riffing on its output, and feeding new ideas back to the AI.Source: §4.4 · p. 4
5. TEAMS OVERCOMING AI LIMITATIONS
Teams worked around limited model control and awareness by steering generation, assembling local outputs into coherent song structures, and weighing models’ logistical costs alongside musical goals.
5.1 ML is not directly steerable
- Teams used fine-tuning, priming, and lyric prompts to steer model output toward desired styles, musical properties, or content.Examples include fine-tuning GPT-2 on lyrics in a target style and priming with sequences containing desired pitches or contours.Source: §5.1 · p. 5
- Interpolating between simple sequences in MusicVAE’s latent space unexpectedly yielded useful baseline sequences for one participant.The participant found this effect at high temperatures.Source: §5.1 · p. 5
5.2 ML is not structure-aware
- Because smaller models lack holistic song context, teams manually arranged their outputs and used musical knowledge to build song structure.One team curated chord progressions for song sections, then used a chord-conditioned model to generate melody and basslines.Source: §5.2 · p. 5
- Teams created section contrast by choosing continuations, varying generation temperature, or rewriting parts while conditioning melody and accompaniment on each other.One team also used section-specific miniVAEs and latent-space paths to connect distinctive sections and enable style transfer.Source: §5.2 · p. 5
5.3 ML setup can interfere with musical goals
- Model setup and customization—including checkpoint, script, and representation differences—added preprocessing work and influenced which models teams chose or abandoned.Larger teams sometimes explored multiple models in parallel to find what worked best.Source: §5.3 · pp. 5–6
6. DISCUSSION
The discussion calls for music models and interfaces that better fit musicians’ structures and workflows, while supporting the parallel creative and technical iteration that AI introduces.
6.1 Decomposeable and context-aware modeling
- Smaller models can be flexible but lack global context, while end-to-end models lack meaningful musical hierarchy; the authors suggest combining their strengths.They propose adding context-awareness and user-facing controls to smaller models, or giving end-to-end models intermediate representations aligned with familiar musical objects.Source: §6.1 · p. 6
6.2 AI-defined vs. User-defined building blocks
- Available AI tools can shape songwriting choices, so future model design should treat musicians’ existing workflows and building blocks as first-class principles.The authors note that some familiar structures, such as motif, verse, chorus, and coda, lack ready-made generative models or are not first-class building blocks in deep models.Source: §6.2 · p. 6
6.3 Support for parallel music and ML exploration
- Co-creation adds a parallel cycle of exploring and curating models to the creative flare-and-focus cycle, and this extra work often interferes with rapid iteration.The authors suggest integrating models into musicians’ workflows and surfacing effective model combinations or workflows to reduce model-based decision overhead.Source: §6.3 · p. 6
7. CONCLUSION
The study examined how people co-create songs with deep generative models and found that participants used workarounds to steer and assemble models toward creative goals.
7. CONCLUSION
- Participants used varied workarounds to steer and combine generative models toward their creative goals, informing recommendations for human-AI systems supporting complex co-creation.Source: §7 · p. 6