Source-linked AI summary
Architecture and Affordances of PLAUD: Performative Latents and Unsupervised DDSP
Błażej Kotowski, Frederic Font
TL;DR
AI music systems are often developed around imitation and established control schemes, while PLAUD examines how architectural choices shape material engagement in live performance. Its neural synthesizer and Max for Live interface combine latent, synthesis-chain, and trajectory controls, demonstrating that expressive affordances can emerge directly from the architecture.
Problem
AI music systems are often developed for imitation with established semantic and compositional controls, motivating attention to material engagement with machine-learning artifacts in performance.
Method
The paper analyzes PLAUD’s architecture and affordances across a VAE-regularized NoiseBandNet model, latent smoothing, an optional transformer prior, and direct synthesis-chain interventions.
Results
PLAUD demonstrates that expressive affordances emerge directly from small-data training, variational latent structure, interpretable DDSP synthesis, and temporal tendencies encoded by the prior.
Takeaways & Limitations
Trajectory sampling supports live style transfer by combining one corpus’s temporal structure with another corpus’s timbral vocabulary.
Takeaways & Limitations
Computational cost limits simultaneous model use, especially with the prior, and remains the most pressing area for future development.
Abstract
from arXiv · showhide
PLAUD (Performative Latents and Unsupervised DDSP) is a neural synthesizer and Max for Live instrument for live electronic music, built on NoiseBandNet and trained on small personal sound corpora. We present its architecture, combining a variational DDSP synthesis model, latent smoothing, multi-scale spectral and adversarial losses, and an optional transformer prior, alongside a set of bending operations that intervene directly in the synthesis chain: component limiting, waveshaping, and prior feedback. The Max for Live interface exposes control generation, trajectory sampling, and modulation as primary modes of interaction. Throughout, we thread an affordance analysis arguing that the system's performative character follows from architectural decisions rather than being designed on top of them. The paper contributes both a technical account of the system and a situated affordance analysis of its role in live electronic music performance.
1 INTRODUCTION
PLAUD is a real-time generative audio system combining a DDSP synthesis engine, a Transformer-based prior, and a Max for Live interaction layer. The paper presents its technical architecture alongside an affordance analysis grounded in stage testing and first-person experimentation.
- System overview: PLAUD combines a DDSP-based synthesis engine for sonic vocabulary with a Transformer-based prior for temporal evolution, connected through a Max for Live instrument.The backend contains two loosely coupled neural networks trained on shared data.
- Performance-oriented development: PLAUD was developed for electronic music performance and tested on stage throughout the engineering process through first-person experimentation by the first author.These design decisions were motivated by material engagement with machine learning artefacts.
- Paper contribution: The paper details PLAUD’s technical architecture and threads an affordance analysis across the levels of the model.The analysis reflects on the engineering motivations behind the system.
- Affordance framework: The paper adopts a situated account of affordances in which artefacts request, demand, allow, encourage, discourage, or refuse specific interactions.This account measures affordances relative to the interacting animal rather than treating qualities as simply present or absent.
2 RELATED WORK
The related work situates PLAUD within neural audio synthesis, latent-space performance, deliberate generative-model divergence, and situated affordance analysis. It positions the system’s contribution against DDSP, RAVE, and frameworks for understanding how architectural decisions shape interaction.
- Neural audio synthesis: DDSP embeds classical signal-processing components in neural training loops, preserving interpretable internal structure that remains open to intervention.The paradigm spans instrument-oriented, sound-design, and vocoder applications, while NoiseBandNet departs from DDSP’s harmonic approach.
- Neural audio synthesis: Multi-scale adversarial losses have proven effective at recovering spectral detail that averaging-prone reconstruction objectives suppress.The passage specifically identifies Mel-GAN’s multi-scale discriminator as an example.
- Neural audio synthesis: RAVE combines a variational latent space, optional autoregressive prior, and real-time inference, making it a widely adopted starting point for neural instrument builders.Its streaming architecture and small training footprint support this adoption; PLAUD shares deployment infrastructure but departs at the synthesis level.
- Latent-space performance: Research on latent audio spaces examines traversal, artistic agency, sonic creativity, and the differing gestural affordances of models in sound design and performance.The cited work also identifies continuing problems in relating GAN latent spaces to real-time performance.
- Generative-model divergence: Active divergence frames deliberate out-of-distribution behaviour as compositional material and categorizes interventions at the data, network, and training levels.This provides vocabulary for practices involving the breakdown or misuse of generative models.
- Affordance analysis: The paper applies situated affordance analysis to examine how architectural and training-time decisions shape a neural synthesizer’s performance interactions.Its framework treats affordances as context-dependent relations in which artefacts can request, demand, allow, encourage, discourage, or refuse interactions.
3 SYSTEM ARCHITECTURE
PLAUD combines a variational DDSP synthesizer with an optional autoregressive prior, while its structured signal path, latent space, losses, smoothing, and prior establish distinct modes of performative control. Architectural choices make direct intervention, exploratory navigation, temporally complex modulation, richer corpus learning, and cross-corpus composition possible.
- Core architecture: PLAUD couples a DDSP-based neural synthesizer with a variational latent space and an optional autoregressive prior that together shape its sonic vocabulary and temporal tendencies.The synthesizer remains fully functional without the prior, which learns rhythmic patterns, envelopes, and longer-range dependencies from the corpus.
- Synthesis model: Fixed noise bands and predicted amplitudes keep the synthesis chain interpretable, enabling direct signal-path intervention rather than opaque manipulation.NoiseBandNet replaces harmonic oscillators with a filterbank of narrow noise components, preserving interpretability while avoiding a harmonicity bias.
- Latent control: A 4-dimensional variational latent space replaces hand-engineered loudness and centroid features, supporting exploratory navigation through representations discovered from complex data.For live performance, loudness and centroid are later reintroduced alongside two latent dimensions, providing minimal perceptual anchoring.
- Responsive temporal control: Regularisation and trajectory smoothing make latent control more continuously responsive while allowing simple modulation to produce temporally complex structures.At W = 32 and 44.1kHz, the control signal runs at 1378 Hz; smoothing shifts short-term temporal structure into the GRU, so output depends on the preceding trajectory.
- Corpus learning and prior generation: Adversarial fine-tuning preserves perceptual spectral detail in noisy material, while the optional transformer prior enables cross-corpus composition with controllable stochasticity.The prior operates autoregressively over discretised synthesizer controls, and softmax temperature balances reproduction of learned patterns against increasingly stochastic generation.
4 INTERFACE AND INTERACTION
PLAUD’s Max for Live interface exposes its latent space, autoregressive prior, and synthesis chain through controls for modulation, trajectory sampling, and direct synthesis intervention. Its performative affordances emerge from these architectural and interaction choices, which support exploration beyond the training data and familiar performance workflows.
- Interface organization: The Max for Live device exposes PLAUD’s latent space, autoregressive prior, and synthesis chain through functional control blocks and interaction modes.The interface includes model and preset management, synthesis controls, prior controls, trajectory sampling, posterior modifiers, bending parameters, and visualization.
- Interface organization: Up to sixteen parameter snapshots can be saved and interpolated, while four knobs control loudness, spectral centroid, and two latent variables.Audio-feature controls default to [0, 1], while latent-variable controls use [−1, 1].
- Affordances: Adjustable control ranges and bending operations let performers probe beyond the training distribution in controlled, musically legible ways.The interface encourages experimentation with out-of-distribution behavior and direct engagement with the network’s materiality.
- Control sources: External modulation and the autoregressive prior provide complementary sources for driving synthesizer inputs and generating control trajectories.Ableton Live sources include LFOs, sequencers, envelopes, and manual control, while the prior generates trajectories from learned temporal patterns and can be primed with manual modulation.
- Trajectory sampling: Trajectory sampling applies looping, slicing, transposition, variable speed, and direction to control trajectories rather than audio directly.Because the smoothed decoder depends on the trajectory leading to a latent point, playback speed and direction shape temporal structures rather than merely stretching or compressing sound.
- Bending operations: Bending operations intervene directly in synthesis and prior generation through component limiting, waveshaping, and prior feedback.These operations can create abrupt spectral discontinuities, transform noise toward sine and square waves, or destabilize prior predictions through adjustable feedback amount and delay.
5 PERFORMANCE REFLECTION
PLAUD’s performance affordances emerge through performance-specific models, expressive bending and trajectory sampling, while low fidelity and computational demands shape its constraints. Across live contexts, these properties make the instrument surprising, structured, and sometimes difficult to deploy.
- Performance-specific instruments: Quick training and small data requirements enable performance-specific instruments whose reactivity, timbral vocabulary, and character differ across self-curated datasets.The authors describe this variability as surprising and generative, with no two instruments being the same.
- Expressive bending: Bending operations support dramatic transitions from unstable sine tones and clicks to synthetically precise signals, with full spectral detail restorable by one knob movement.Component limiting reduces active components to a minimum, while waveshaping alters the resulting minimal signals.
- Reconstruction and expressivity: Low-fidelity reconstruction did not prevent expressivity: the synthesizer’s lo-fi output became part of its identity, while the prior did not achieve stable long-term generation.The passage notes that conventional evaluation criteria would often reject the reconstruction quality.
- Trajectory sampling: Trajectory sampling imposes structure on the prior’s chaotic tendencies and can combine pre-sampled rhythmic trajectories with generation from models trained on different material.The module also helps integrate PLAUD’s generation into more conventional grid-based setups.
- Computational limitation: Up to three model instances run with relative stability without the prior on a 2021 MacBook M1 Pro, but engaging it reduces capacity to one instance.This limitation has required performance-setup compromises, and some workshop participants’ hardware could not run the models at all.
6 CONCLUSION, LIMITATIONS AND FUTURE WORK
PLAUD’s expressive affordances emerge from architectural choices spanning data constraints, latent structure, DDSP interpretability, and prior dynamics. The paper identifies narrow audience and aesthetic commitments, computational and decoder limitations, and several directions for extending the system.
- Conclusion: PLAUD’s expressive affordances emerge directly from small-data constraints, variational latent structure, DDSP synthesis-chain interpretability, and temporal tendencies encoded by the prior.
- Conclusion: The system combines a VAE-regularised NoiseBandNet model, latent smoothing, adversarial training, an optional transformer prior, and synthesis-chain bending operations.
- Conclusion: Max for Live exposes trajectory sampling, modulation-based control, and the prior’s automatic steering mode, while the instrument’s performative character follows from architectural decisions.
- Limitations: PLAUD targets producers and performers prepared for RAVE, with noisy, textural, and unpitched material treated as instrument character rather than defects.
- Future work: Computational cost is the most pressing limitation; replacing the autoregressive GRU with causal convolutions could improve throughput and control over short-term context length.The prior’s resampling-rate trade-off also makes learning structure at low rates unfeasible.
- Future work: Alternative sinusoidal-plus-stochastic DDSP decoders could broaden timbral range, but unsupervised frequency learning remains difficult and requires more work before full-pipeline integration.
- Future work: The bending vocabulary could expand through spectral rolling, alternative component-subset selection modes, and harmonisation.
AUTHOR DECLARATIONS
The authors used Claude solely for language editing, while they developed all scientific content, interpretations, and conclusions.
- Claude was used to improve grammar and clarity, but the authors developed all scientific content, interpretations, and conclusions.