PaperlayerView exact revision on arXiv

Source-linked AI summary

PEACE: Joint Embeddings of DSP Effects Code and Audio

David Braun, Adam Finkelstein

arXiv:2610.03405v1cs.SD

TL;DR

No multimodal embedding links effected audio to the DSP code that produced it. PEACE pairs AFx-Rep audio with T5 and BoxGraph Faust-code encoders, yielding competitive mixed-length retrieval and gains over pretrained audio models on reverb.

  • Problem

    No multimodal embedding connects effected audio back to the code that produced it, unlike analogous image-text and audio-text embeddings.

    Source: §1 · p. 1

  • Method

    PEACE learns joint embeddings of Faust effect code and audio by pairing an AFx-Rep audio encoder with fine-tuned T5 and BoxGraph code encoders using SLAP's pairwise objective.

    Source: §1 · p. 1

  • Results

    T5 and BoxGraph are competitive on mixed-length retrieval, with tradeoffs elsewhere; scratch BoxGraph reaches 53.5% R@10 on reverb retrieval versus pretrained AFx-Rep's 28.3%.

    Source: §6 · p. 6, §5.3 · p. 6

  • Takeaways & Limitations

    Masking effect parameters lets audio queries retrieve chain topologies, with BoxGraph winning order-aware chain retrieval from audio.

    Source: §1 · p. 1, §6 · p. 6

    Out-of-domain evaluation covers only reverb, and the study lacks a user study.

    Source: §6 · p. 6

Abstract

from arXiv · show

This paper introduces PEACE, the first joint embedding of audio effect code and output audio. Building on SLAP's multimodal objective, we pair an AFx-Rep audio encoder with two code encoders for Faust, a functional language for audio signal processing. First, we evaluate a fine-tuned T5 transformer over Faust source code. Second, we evaluate a message-passing graph neural network over an intermediate representation of the Faust compiler, capturing both topology and UI parameters. We evaluate on audio-to-code retrieval, where masking UI parameters at inference yields embeddings that encode effect chain topology alone. When parameters are visible, the two code encoders tie on retrieval of mixed-length chains but have tradeoffs on single-effect galleries. With parameters fully masked, PEACE recovers ordered chain topology far above chance without the limitations of supervised methods. PEACE outperforms pretrained models on an out-of-distribution reverb retrieval benchmark and can improve frozen audio-only representations. Its dual understanding of topology and parameters lays the groundwork for music information retrieval systems that search, generate, and condition on DSP code.

1. INTRODUCTION

PEACE addresses the missing link between effected audio and the DSP code that produced it by jointly embedding audio and Faust code. Its code encoders support retrieval across effect chains, reverb audio, and masked chain topologies.

1. INTRODUCTION

  • PEACE addresses the absence of a multimodal embedding that links effected audio back to its producing code.Analogous joint embeddings for images and text, and audio and text, have supported retrieval, classification, and generation.Source: §1 · p. 1
  • Earlier approaches use text descriptions, labels, classification, or generation, while audio encoders do not provide program similarity or code-to-audio retrieval.The cited discussion notes that text descriptions can lose configuration detail, labels are costly, and direct parameter regression is ill-posed.Source: §1 · p. 1
  • Faust’s Block Diagram Algebra provides graph structure for encoding audio-processing programs, and Faust compiles to several widely used targets.The listed targets include C, C++, LLVM, JAX, and Wasm.Source: §1 · p. 1
  • PEACE applies the SLAP pairwise objective to an AFx-Rep audio encoder and two Faust code encoders: fine-tuned T5 and BoxGraph, a GNN over the compiler’s Block Diagram Algebra.The code modality is intended to capture both program structure and parameter values.Source: §1 · p. 1
  • T5 and BoxGraph roughly tie on mixed-length chain retrieval, while T5 generalizes better to longer chains and they split wins across single-effect types.The introduction also reports that BoxGraph outperforms T5 when retrieving chain topology from audio with effect parameters masked.Source: §1 · p. 1
  • On reverb retrieval from professional plugins unseen in training, PEACE—especially BoxGraph—outperforms pretrained audio models.Cross-modal SLAP training also improves retrieval with a frozen pretrained AFx-Rep encoder.Source: §1 · p. 1
  • The authors hope PEACE will lay groundwork for style transfer, dry-stem recovery, and DSP code generation.They also release the codebase, model weights, and an interactive embedding-exploration tool.Source: §1 · p. 1

2. RELATED WORK

Prior audio-effect work spans audio representations, chain estimation, and text guidance, while code-representation work includes joint embeddings for sound-generating programs. The paper situates PEACE against limitations in effect understanding, parameter sensitivity, and program comparison.

Audio Effect Representations

  • Audio foundation models can become invariant to effects used as training augmentations; AFx-Rep instead learns effect-aware representations, while Fx-Encoder++ extracts instrument-specific effects from mixtures.AFx-Rep is trained to classify the individual effect and parameter configuration applied to audio.Source: §2 · p. 2

Effect Chain Estimation

  • Effect-chain estimation has used supervised classification, graph generation or pruning, and iterative inversion; Okita and Katayose combine effect-set and dry-signal prediction with evolutionary search over order and parameters.Their search renders candidates and scores wet-signal reconstruction.Source: §2 · p. 2

Text-Guided Audio Effects

  • Text-guided methods optimize in CLAP space or predict parameters with language models, but joint language-audio embeddings can weakly encode perceptual timbre.LAION-CLAP reverses the human-labeled direction of “warm” for both EQ and reverb.Source: §2 · p. 2

Representation Learning for Code

  • Code-representation work includes graph-augmented and multilingual code models, while prior Sonic Pi embeddings focus on sound-generating programs and can be insensitive to parameter changes.The passage attributes the latter sensitivity issue to tokenizers fragmenting numbers; prior Faust work covered code generation and effect matching, not representation learning.Source: §2 · p. 2

3. METHOD

PEACE maps Faust code and wet audio into a shared space for cross-modal retrieval. Its SLAP objective trains complementary audio and code branches, while BoxGraph represents Faust programs as compositions of function-like boxes with topology and parameter values.

3. METHOD

  • Faust code and wet audio are mapped into a shared d-dimensional space, where queries retrieve code or audio embeddings by cosine similarity.Source: §3 · p. 2

3.1 Training Objective: SLAP

  • SLAP combines intermodal and intramodal cosine-distance losses, using online predictions against stop-gradient EMA target projections.The online branch adds a predictor; the EMA target branch does not.Source: §3.1 · p. 2

3.2 BoxGraph

  • BoxGraph represents Faust programs as a directed acyclic graph of boxes, with composition nodes joining children through Block Diagram Algebra operators.Boxes encode signal-processing functions with known input and output channel counts; the experiments use par and seq, leaving split, merge, and recurse for future work.Source: §3.2 · pp. 2–3
  • Node features encode box type, DSP identity, and parameter values; parameters can be masked during training and fully masked at inference to focus on effect selection and order.Continuous values are projected after remapping, while discrete values use per-parameter embeddings with a [MASK] option.Source: §3.2 · p. 3
  • BoxGraph passes position-aware messages through the graph and pools final features from seq and DSP nodes into its code embedding.The update uses the node’s initial feature and current messages rather than its previous hidden state; restricting the pooling set outperformed pooling all nodes or only the root.Source: §3.2 · p. 3

4. EXPERIMENTS

The experiments use rendered audio–Faust pairs and compare audio and code encoders under several initializations. Evaluation covers retrieval metrics, a room-impulse-response benchmark, and parameter-masked retrieval of ordered effect chains.

4.1 Dataset

  • Training, validation, and test sets contain 200K, 8,192, and 4,096 audio–code pairs, with dry audio drawn from nine training categories and four held-out test source types.The 21 Faust effects are rendered in chains of one to three sequential effects using sampled presets; wet audio is normalized to −18 LUFS.Source: §4.1 · p. 3

4.2 Experimental Setup

  • The setup compares three AFx-Rep initializations and two code encoders: BoxGraph and T5, with T5 fine-tuned or trained from scratch.BoxGraph design choices are selected using precomputed AFx-Rep latents before the configuration is fixed for remaining runs.Source: §4.2 · p. 4

4.3 Evaluation Metrics

  • Cross-modal retrieval is measured with Recall@1 and @10 plus mean and median normalized rank, alongside embedding-gap, modality-separability, and source-separability measures.Source separability uses source-type probes on audio embeddings; lower separability indicates greater content invariance.Source: §4.3 · p. 4

4.4 Room Impulse Response Retrieval Benchmark

  • The RIR benchmark tests whether a wet query retrieves its matching RIR from 1,831 candidates, using professional-plugin reverbs and vocal tracks.Baselines include pretrained AFx-Rep, Fx-Encoder++, and LAION-CLAP; PEACE is evaluated using latents and predictions.Source: §4.4 · p. 4

4.5 Chain Retrieval

  • Chain retrieval tests 420 ordered length-2 effect pairs with all parameter values masked, using 1,680 audio queries rendered with held-out presets.A supervised cross-entropy classifier is also evaluated on the same audio arm, but prior chain-classification methods produce labels rather than embeddings for topology retrieval.Source: §4.5 · p. 4

5. RESULTS

PEACE shows cross-modal and intramodal understanding across retrieval tasks, with performance varying by effect type. T5 generalizes better to longer chains, while BoxGraph leads on order-aware chain retrieval and performs strongly on reverb retrieval.

5. RESULTS

  • For AFx-Rep trained from scratch, fine-tuned T5 marginally leads Audio→Code retrieval while BoxGraph marginally leads Code→Audio; training T5 from scratch makes it underperform BoxGraph.The results suggest end-to-end training can partly overcome tokenization pitfalls when floats are rounded to two decimals.Source: §5 · pp. 4–5, Table 1 · p. 5
  • Source separability in the mid-50s, near the 50% chance level, indicates embeddings are largely invariant to source content.The paper describes this as a desirable property; T5 pairings have less separable modality embeddings than BoxGraph with AFx-Rep from scratch.Source: §5 · p. 5

5.1 Per-Effect Retrieval Results

  • On length-1 galleries, T5 marginally leads overall on R@1 and R@10, while BoxGraph wins R@10 on 11 of 21 effects, including several dynamics and nonlinear effects.T5 wins most time-based effects, including Chorus, Delay, Flanger, Phaser, ReverseEcho, and three of five reverbs.Source: §5.1 · p. 5, Table 2 · p. 5
  • Some effects impose a retrieval ceiling for particular inputs: Limiter may leave quiet audio unchanged, Gate may leave loud audio unchanged, and StereoWidth does not alter mono inputs.Source: §5.1 · p. 5

5.2 Chain Length Generalization

  • Beyond the training range of chain lengths 1–3, T5 retrieval degrades gradually while BoxGraph degrades much faster; both track closely through N ≤ 3 and peak at N = 3.Figure 3 plots retrieval against chain length N for a gallery of 1,024 and shades the training range.Source: §5.2 · p. 5, Figure 3 · p. 5

5.3 Reverb Retrieval Results

  • 53.5% R@10 for PEACE scratch BoxGraph qA exceeds pretrained AFx-Rep (28.3%), Fx-Encoder++ (11.6%), and LAION-CLAP (10.7%) on reverb retrieval from unseen professional plugins.With frozen AFx-Rep latents and BoxGraph, R@10 rises from 28.7% to 38.6%.Source: §5.3 · p. 6, Table 3 · p. 6

5.4 Chain Retrieval Results

  • 17.4% R@1 and 61.4% R@10 for AFx-Rep-BoxGraph beat T5’s 12.4% and 43.2% on ordered chain retrieval with all parameters masked.Counting any chain with the same effect set as correct raises BoxGraph to 27.2% and 67.4%, versus T5’s 20.7% and 49.1%.Source: Table 4 · p. 6, §5.4 · p. 6

6. CONCLUSION

PEACE’s T5 and BoxGraph encoders are competitive on mixed-length chain retrieval, with distinct strengths across retrieval settings. The conclusion also notes training-pipeline limits and outlines possible extensions and applications.

6. CONCLUSION

  • T5 and BoxGraph are competitive on mixed-length chains, while T5 generalizes better to longer chains and BoxGraph wins order-aware chain retrieval from audio.T5 has a minor lead on average within-effect retrieval, but BoxGraph wins about half of effect types; all PEACE models do well on intramodal reverb retrieval, particularly BoxGraph.Source: §6 · p. 6
  • Training uses randomly sampled parameters and presets, and source audio is not always fully dry, widening the sim-to-real gap.Within-effect retrieval variance motivates improvements in data quality and the audio encoder; out-of-domain evaluation covers only reverb, and the paper lacks a user study.Source: §6 · p. 6
  • BoxGraph could extend to new terminal DSP nodes and composition operators to represent richer DSP graphs beyond effect chains.The paper also notes that general-purpose code encoders such as multimodal LLM architectures may be favored by the Bitter Lesson.Source: §6 · p. 6
  • PEACE embeddings could support style transfer or multi-track mixing, and retrieval could supply candidate topologies for parameter search.The code encoder could also speed parameter search by reducing iterative audio rendering.Source: §6 · p. 6
Loading 2610.03405v1…