Source-linked AI summary
Rescuing Performance from the Demo: Co-Designing Drum Gesture Mappings with a Percussionist
Jordie Shier, Teresa Pelinski, Charalampos Saitis, Andrew Robertson, Andrew McPherson
TL;DR
Mapping systems can extend musical expression but may also constrain musicians’ practice, raising the risk of technological capture. The authors co-designed a drum gesture toolkit with a professional percussionist and recorded an album; they report a continuous gesture mapping method and identify knowing-when as tacit judgement in negotiating technology and music. They also leave open how to determine whether technological influence supports practice without access to a musician’s broader social context.
Problem
Mapping constraints can shape musical practice so strongly that musicians cannot recycle their existing skills, a risk the paper calls technological capture.
Source: §1 · p. 1Method
A researcher and professional percussionist co-designed a drum gesture toolkit over four months, using collaborative practice research and documenting and reflecting on the process.
Source: §1 · p. 1, §3.3 · p. 3Results
Takeaways & Limitations
The authors frame productive tension between musical goals and technology as a way to resist capture, while recognizing that this negotiation depends on knowing-when to attend to each.
Source: §2 · p. 1, §1 · p. 1, §5.2 · p. 6The authors cannot determine in isolation from the musician’s broader social and professional context whether technology’s aesthetic influence is a creative catalyst or capture.
Source: §5.2 · p. 6
Abstract
from arXiv · showhide
Augmenting instruments with sensors and neural network mappings is a well-explored digital musical instrument design approach. While augmentations can create new expressive opportunities, they also exert aesthetic influence and can constrain musicians' gestural language, which, if left unchecked, can lead to technological capture. To examine this, we conducted a study with a professional percussionist, co-developing a gesture mapping toolkit and recording a ten-track album. Drawing on the concept of productive dissonance, our study aimed to hold the musician's aesthetic in tension with technological constraints. This, along with a practice-based reflective approach, supported the development of a continuous gesture recognition method for percussive mapping and surfaced insights into the design process. We identify knowing-when as a form of tacit knowledge that supported productive dissonance, and raise an open question: absent a musician's broader social context, how do we know whether a technology's influence is genuinely supporting their practice?
1 INTRODUCTION
Sensor-to-synthesis mappings can extend existing musical skills, but their constraints may also shape performance so strongly that musicians cannot recycle those skills. The study co-designs a drum mapping toolkit and album with a professional percussionist to examine that risk and reflect on how to resist it.
1 INTRODUCTION
- Mapping constraints can overwhelm musical practice and prevent musicians from recycling their existing skills, a risk the paper calls technological capture.The authors also connect this risk to reification, where the mapping metaphor comes to define performance itself.Source: §1 · p. 1
- A researcher and professional percussionist worked together for four months to co-develop a drum gesture mapping toolkit and incorporate it into the musician’s practice.Creating and recording a performance with the mappings was a key goal, anchoring technical practice in music-making.Source: §1 · p. 1
- The study shares a Max4Live toolkit, a ten-track album, and a novel neural-network method for mapping continuous percussive gestures to synthesizer parameters in real time.Through documentation and reflection, the authors identify knowing-when: tacit judgement about when to attend to technology and when to attend to music.Source: §1 · p. 1
- The authors ask how to tell whether technology genuinely supports a musician’s practice when the musician’s broader social context is absent.They present this as an open question arising from the laboratory setting.Source: §1 · p. 1
2 “RESCUING THE PERFORMANCE FROM THE DEMO”
The paper asks whether musical performances with new technology serve the music or demonstrate the technology, and argues that the tension between these goals can be productive. It frames technological capture as a risk when musical practice comes to serve technology development instead of the other way around.
2 “RESCUING THE PERFORMANCE FROM THE DEMO”
- The performance-versus-demo tension can be productive, but becomes capture when technology development subordinates the aesthetic force of the music.The paper borrows “capture” from regulatory theory to describe practice coming to serve technology development rather than technology serving practice.Source: §2 · pp. 1–2
- DMI bottlenecks can steer musicians’ gestures through push effects that discard unrecognized gestures and pull effects toward convenient recognized ones.Jack et al. use “bottleneck” for DMI attributes that constrain gestural language [29].Source: §2 · p. 2
- Computationally efficient representations are often necessary, but become problematic when a reduced symbolic account of music becomes the basis for generating it.An onset detector, for example, presupposes that onset is musically meaningful; onsets may not be stable entities without such a detector.Source: §2 · p. 2
- Because onset-based representations are not universal to all music, the authors argue that technical systems should be situated within a well-defined musical aesthetic.They propose direct collaboration and co-design with a professional musician as a way to provide that aesthetic context.Source: §2 · p. 2
3 METHODOLOGY
The study treats collaborative design practice as its research method, documenting and reflecting on technical, aesthetic, and social dimensions as they evolve. A four-month collaboration with a professional percussionist moved from co-design sessions toward recording, while situated participant backgrounds and reflective documentation informed the process.
3.1 Study Co-Design
- The four-month study comprised an initial interview and co-design session, five iterative sessions, and two recording days focused explicitly on making music.Co-design sessions were spaced about two weeks apart and paired mapping development with short performances and reflection.Source: §3.1 · p. 2
3.2 The Participants-Collaborators
- The collaboration paired the researcher with Jem Doulton, a professional London percussionist whose work spans folk country to free improv.Doulton valued dexterity and hands-on playing, and recalled electronic pads gradually replacing his acoustic kit until he felt trapped and replaceable by a backing track.Source: §3.2 · p. 2, §3.2.1 · p. 2
3.2.1 The Musician
- Doulton’s emphasis on dexterity and hands-on playing, and his prior experience of feeling trapped by electronic pads, shaped the collaboration’s concern with technological capture.His practice spans folk country to free improv and he described its aesthetic as between rock and jazz.Source: §3.2.1 · p. 2
3.2.2 The Researcher
- The researcher brought computer-science and audio-engineering expertise alongside experience performing and producing electronic music with Ableton Live and synthesizers.That practice had informed his research on real-time percussion-to-synthesizer timbre mapping and digital musical instrument design.Source: §3.2.2 · p. 3
3 METHODOLOGY
- Collaborative design practice itself was the study’s principal research method, drawing especially on technical practice research and reflective design.This approach emphasizes documenting and reflecting on technical development as it unfolds alongside aesthetic and social dimensions.Source: §3.3 · p. 3
3.3 Documenting and Reflecting
- The team documented sessions and interviews with audio-video recordings, transcripts, thematic coding, version control, and a design journal.Reflection also drew on session reviews, musician prompts, and five interviews by the second author with the researcher; the authors revisited the documentation to articulate their insights.Source: §3.3 · p. 3
4 TOOLKIT AND PERFORMANCE
The toolkit extends existing percussive mapping tools with discrete hit classification, continuous gesture recognition, and timbre remapping, alongside performance devices. The collaboration produced a ten-track album whose recording sessions anchored the toolkit in musical performance.
4 TOOLKIT AND PERFORMANCE
- The toolkit builds on the researcher’s DataKnot-based tools and combines streaming-audio gesture recognition and mapping with performance tools such as looping and MIDI sequencing.The study’s technical starting point also drew on commercial and open-source percussive mapping systems.Source: §4.1 · p. 3, §4 · p. 3
4.1 Gesture Mapping Toolkit
- Gesture-mapping objects process streaming percussive audio into synthesizer parameters, while other toolkit objects support live performance structure.The toolkit overview distinguishes these two groups of objects.Source: §4.1 · p. 3
4.1.1 Discrete Gesture Recognition
- Discrete recognition detects a drum onset, extracts audio features, and classifies hit zones such as head versus rim with a lightweight MLP.The authors note that treating percussion as discrete events is a representational assumption that does not fit every playing technique or musical context.Source: §4.1.1 · p. 3
4.1.2 Continuous Gesture Recognition
- Continuous recognition maps sustained gestures rather than discrete strikes; the study used separate GRU-based systems for brush strokes and buzz rolls.The brush model used Mel-scaled spectral input after an NMF approach could not reliably exclude discrete strikes; the buzz-roll model used amplitude-envelope features to distinguish buzz strokes from regular double strokes.Source: §4.1.2 · pp. 3, 5
4.1.3 Timbre Remapping and Performance Devices
- The toolkit also includes timbre remapping from percussive input to synthesizer parameters, plus audio looping and melodic MIDI sequencing devices.The supplied overview identifies these as additional toolkit components.Source: §4.1.3 · p. 5
4.2 Musical Performance
- The collaboration recorded a ten-track album over two full days organized to replicate a studio session, with music-making as its output.The album is presented as an annotated portfolio with edited performance videos on an accompanying website.Source: §4.2 · p. 5
5 PRODUCTIVE DISSONANCE IN PRACTICE
The design arcs show how contact with the percussionist’s gestures exposed limits in computational representations, while performance repeatedly tested whether technical effectiveness or musical merit should guide decisions.
5.1.1 Continuous Gestures
- The musician’s brush exposed onset detection’s scope limit, and that friction drove continuous gesture recognition; a neural network expanded the mapping but imposed an artificial discrete/continuous boundary.The expanded system covered brushing, stick-tip drawing, and buzz rolls, while its discrete/continuous distinction remained an artifact of the classifier rather than the musician’s practice.Source: §5.1.1 · pp. 5–6
- Imprecise feature envelopes prevented a stable brushing loop, so the team dropped looping but retained real-time continuous mapping for live performance.Waveshaping alone could not resolve the temporal mismatch; composing music gave the collaborators a reason to address the limit and to judge when to keep pushing technically or switch to composition.Source: §5.1.1 · pp. 5–6
5.1.2 Cymbal Sensor
- The cymbal-mounted IMU triggered a chain of technical interventions that pulled focus from musical practice, so the team scrapped it before recording.The idea remained compelling, but it had originated technically rather than emerging from the musician’s practice, leaving no musical anchor to return to.Source: §5.1.2 · p. 6
5.2 Navigating Performance-Demo Ambiguity
- The study’s technology focus complicated whether classifier glitches reflected genuine aesthetic preference or accommodation to technical limits.The musician accepted occasional unintended notes as part of the mapping’s glitchiness, but the authors note that this aesthetic had not surfaced earlier and may have been a rationalization of failure.Source: §5.2 · p. 6
- Knowing-when meant judging a performance through either a technical or aesthetic lens; the authors could not assess technological influence apart from the musician’s broader social and professional context.When the algorithm produced errors, the musician judged the performance’s vibe intact, while attention had been pulled toward demonstrating technical effectiveness.Source: §5.2 · p. 6
6 DISCUSSION
The discussion traces how collaborative judgement can keep musical aims in tension with technical constraints, while asking whether an academic setting provides enough social context to assess technology’s aesthetic influence.
6.1 Knowing-When
- Knowing-when was tacit judgement developed through sustained collaboration, including when to push against representational limits, accept them, or evaluate aesthetically rather than technically.The researcher’s own aesthetic initiative also became part of the collaboration, raising the question of whether it occupied space that the musician’s broader social context might otherwise have filled.Source: §6.1 · p. 7
6.2 Productive Dissonance
- Classification can impose separable classes on gestures that may not have been separate in practice, and repeated pragmatic use can make those assumptions idiomatic and reified.The brush-and-onset mapping surfaced this tension; the resulting continuous-gesture method remained a classifier, but made representational assumptions more visible for conscious design.Source: §6.2 · p. 7
- Productive dissonance surfaced representational tensions and led to new representations supporting the musician’s gestural language and album composition, while leaving the laboratory’s social context in question.Source: §6.2 · p. 7
6.3 The Hermetic Laboratory
- The isolated academic setting made it difficult to distinguish a genuinely desired aesthetic influence from accommodation to technological constraints, and a technical intervention spiral likewise shifted toward technical aims.The authors suggest that working with a musician is necessary but insufficient to sustain productive dissonance.Source: §6.3 · p. 7
- Although the laboratory was porous to some social connections, sustained engagement with the musician’s broader network was limited, and the authors suggest deeper engagement in projects with real social commitments.Source: §6.3 · pp. 7–8
7 CONCLUSIONS
The paper asks whether technology-led performance can indicate technological capture, then reports a co-designed toolkit and album alongside an open question about judging aesthetic influence without broader social context.
7 CONCLUSIONS
- Technological capture may occur when a DMI’s constraints overwhelmingly influence musical practice and trap it in a prescriptive aesthetic range.Source: §7 · p. 8
- The collaboration partially developed a toolkit using RNNs to map continuous percussive gestures and produced a ten-track album presented as an annotated portfolio.Source: §7 · p. 8
- The reflective practice-based approach surfaced knowing-when as tacit knowledge and an open question about whether technology’s aesthetic influence supports the musician or is accepted under study conditions.The authors identify isolation from the musician’s broader social context as a limitation and suggest deeper engagement with that context as a promising approach.Source: §7 · p. 8
AUTHOR DECLARATIONS
The authors report ethical approval, data provenance, writing-tool use, and contributions.
AUTHOR DECLARATIONS
- The study received low-risk ethical approval; models were trained on author-recorded data, Claude Sonnet 4.6 edited author-written text, and ideas and research code came from the authors and collaborators.Source: AUTHOR DECLARATIONS · p. 8
A TOOLKIT TECHNICAL DETAILS
The section introduces the technical details of the final gesture-mapping toolkit and its overview table.
A TOOLKIT TECHNICAL DETAILS
- Table 1 overviews the objects included in the final gesture-mapping toolkit.Source: A TOOLKIT TECHNICAL DETAILS · p. 10
A.1 Discrete Gesture Recognition
The discrete gesture component detects drum strikes and classifies them, with the toolkit’s device supporting mappings from recognized gestures.
A.1 Discrete Gesture Recognition
- InputClassify performs discrete gesture recognition on drum audio.Source: §A.1 · p. 10
- Onset detection identifies strikes, which are classified as drum-head or rim gestures.Source: §A.1 · p. 10
- Figure 5 shows the Max4Live device for discrete gesture recognition.Source: Figure 5 · p. 10, §A.1 · p. 10
A.1.1 Method Overview
The method detects drum onsets, extracts audio features, and classifies them with a lightweight MLP; its training and outputs are configurable for performance use.
A.1.1 Method Overview
- The signal flow combines onset detection, audio-feature extraction, and classification with a lightweight MLP.Source: §A.1.1 · p. 10
- AmpSlice detects onsets when the difference between fast and slow amplitude envelopes exceeds a threshold, with sensitivity, lockout, and noise-floor controls.Source: §A.1.1 · p. 10
- A 256-sample window (≈5.8ms at 44.1kHz) yields LKFS loudness normalized for MIDI velocity, with spectral features optionally available for timbre remapping.Source: §A.1.1 · p. 10
- Classification uses 13 MFCC coefficients from a 40-band Mel representation, with summarized statistics yielding a 52-dimensional feature vector.Source: §A.1.1 · p. 10
- Training and validation data came from the musician’s co-design sessions; Python training used SMOTE to address class imbalance.Source: §A.1.1 · p. 10
- The MLP treats classification as regression, optimizing mean squared error with stochastic gradient descent; outputs can trigger events, send MIDI note and velocity, or provide spectral features.Source: §A.1.1 · p. 10
- The study focused on head/rim classification with a one-hidden-layer, two-neuron MLP; the device supports up to eight classes, while broader zone mapping was not explored in detail.Source: §A.1.1 · p. 10
A.2 Continuous Gesture Recognition
The toolkit extends drum mapping to sustained gestures, using gesture-specific audio representations and recurrent recognition alongside continuous control features. It also includes timbre-remapping and performance devices for live musical structure.
A.2 Continuous Gesture Recognition
- The continuous-gesture devices address sustained sounds such as brushing or drawing and buzz rolls or drags, and can extract continuous loudness for Ableton control.Source: §A.2 · p. 10
A.2.1 Input Representations
- Brush and drawing gestures use normalized Mel-spectrogram features, while buzz rolls use high-pass-filtered amplitude envelopes, including fast, slow, and difference signals.Source: §A.2.1 · p. 11
A.2.2 Neural Network Architecture and Training
- Recognition uses a GRU followed by an MLP, trained on manually labelled gesture regions from co-design recordings.Source: §A.2.2 · pp. 11–12
- Figure 7 presents example predictions for buzz-roll and brush/drawing models alongside audio spectrograms and ground-truth labels.Source: Figure 7 · p. 12, §A.2.2 · p. 12
A.2.3 Real-time Implementation and Mapping
- Real-time inference runs at 750Hz; predictions can be smoothed and thresholded, while continuous loudness can be shaped, adaptively scaled, and gated by detected gestures.Source: §A.2.3 · p. 12
A.3 Timbre Remapping
- TimbreRemapRun predicts synthesizer parameters from spectral features at detected onsets, then maps them in Ableton before triggering the synth with the input MIDI event.Source: §A.3 · p. 12
A.4 Performance Devices
- The Looper cycles through standby, recording, and playback in response to InputClassify triggers, enabling the musician to loop synth output and play over it.Source: §A.4 · pp. 12–13
- NoteSeq advances through pitch values on MIDI triggers, while PresetStep switches among configurable states that can select instrument chains for musical sections.Source: §A.4 · p. 13