Source-linked AI summary
Evaluating Loss Functions in Differentiable Out-of-Domain Sound-Matching with Partial Parameter Distance
Amir Salimi, Daniel Penner, Kalvin Eng, Abram Hindle, Osmar R. Zaïane
TL;DR
OOD sound-matching lacks a reliable automatic evaluation when target and imitator synthesizers do not share a complete parameter space. The paper introduces PPD and tests four differentiable losses across controlled scenarios, finding synthesis-dependent rankings and agreement with listening tests in five of seven scenarios.
Problem
OOD sound-matching is underexplored because standard parameter loss requires a shared parameter space that mismatched target and imitator synthesizers lack.
Method
The paper applies PPD to critical parameters shared by mismatched synthesizers and validates automatic rankings with blinded listening tests across seven scenarios.
Results
Loss-function effectiveness remains synthesis-dependent, with SIMSE_Spec best for band-pass matching, DTW_Envelope for amplitude modulation and pulsating pitch, and JTFS for smooth pitch trajectories.
Takeaways & Limitations
PPD agrees with listening tests on the top-ranked loss in five of seven scenarios, supporting its use as an automatic diagnostic for OOD sound-matching.
Takeaways & Limitations
Selecting critical parameters requires domain knowledge and carries some subjectivity, so PPD is a per-scenario evaluation procedure rather than a universal metric.
Abstract
from arXiv · showhide
In out-of-domain (OOD) sound-matching, a synthesizer is optimized to mimic a sound it did not generate. OOD evaluation of loss functions is underexplored in part because the standard "parameter loss" metric requires a shared parameter space between target and imitator, which OOD settings lack. We introduce Partial Parameter Distance (PPD), which applies parameter loss only to the critical parameters that mismatched synthesizers share (e.g., filter cutoffs), enabling automatically evaluated OOD experiments; we verify its results with blinded listening tests. Across seven scenarios involving band-pass filtering, amplitude modulation, and pitch-bending, we evaluate four differentiable loss functions (SIMSE_Spec, L1_Spec, JTFS, DTW_Envelope). Loss-function effectiveness remains tightly coupled to the method of synthesis: SIMSE_Spec excels at filter-cutoff recovery, DTW_Envelope at amplitude-modulation recovery, and JTFS at smooth pitch trajectories. Parameter-based evaluation agrees with listening tests on the top-ranked loss function in five of seven scenarios, demonstrating its utility as a diagnostic tool.
1. INTRODUCTION
OOD sound-matching is practically relevant but difficult to evaluate automatically because target and imitator synthesizers may not share a complete parameter space. The paper introduces PPD-based controlled experiments to test whether loss-function rankings depend on synthesis method and align with listening judgments.
- Motivation: OOD sound-matching uses a different synthesis process or recording for the target, making reliable automatic evaluation a major challenge.In-domain matching uses the same synthesizer for target and imitator, whereas OOD matching does not.
- Research aim: The study re-examines prior in-domain findings that no universal audio loss function exists because performance depends on the synthesis method.It tests whether this synthesis-dependent behavior also appears under OOD conditions and verifies results with listening tests.
- Contribution: Partial Parameter Distance applies parameter loss to critical parameters shared by mismatched synthesizers, such as modulation rates or filter cutoffs.The approach evaluates progress on critical sound characteristics rather than requiring outright replication of the target.
- Contribution: The controlled framework uses partially overlapping synthesizer functionality and validates its automatic evaluation procedure against blinded listening tests.The authors propose the framework as a screening tool for certifying loss functions before deployment in more complex settings.
2. BACKGROUND
Prior sound-matching work defines differentiable optimization through audio or parameter losses, but systematic loss-function comparison under controlled OOD conditions remains limited. This paper addresses that gap with controlled minimal synthesizer pairs.
- Sound-matching foundations: Sound-matching optimizes synthesizer parameters so generated audio matches a target under a differentiable similarity measure.Audio-based losses compare acoustic representations such as spectrograms or amplitude envelopes, while parameter-based losses compare synthesizer parameters.
- Loss functions: Audio losses include spectrogram distances, amplitude-envelope distances, and JTFS, which applies wavelet transformations to time-frequency representations.These measures provide differentiable alternatives to direct parameter comparison.
- Research gap: Systematic comparison of loss functions under controlled OOD conditions is largely absent because most prior studies evaluate exclusively in-domain.Recent OOD work has emphasized estimator adaptation or reinforcement learning rather than controlled loss-function comparisons.
- Research gap: The paper compares losses across minimal OOD synthesizer pairs representing fundamental sound-design approaches.Their simplicity isolates the interaction between the loss function and the synthesizer.
3. METHODOLOGY
The methodology evaluates four differentiable losses across seven controlled scenarios, using optimization trials, PPD, bootstrapped rankings, and listening tests. PPD compares only normalized critical parameters shared by mismatched synthesizers.
- 3.1. Problem Setup: Four differentiable loss functions are evaluated across seven scenarios—six OOD and one in-domain—with at least 300 independent optimization trials per loss and scenario.Each trial samples target and imitator parameters uniformly, then performs 200 gradient-based optimization steps.
- 3.2. Loss Functions: The compared losses are L1_Spec, SIMSE_Spec, JTFS, and DTW_Envelope, spanning spectrogram, wavelet time-frequency, and amplitude-envelope representations.SIMSE_Spec is invariant to constant offsets in log-magnitude; JTFS captures wavelet-based time-frequency structure.
- 3.3. Evaluation Methods: Partial Parameter Distance applies P-Loss after optimization only to parameters with critical perceptual correspondence between the target and imitator synthesizers.The compared critical parameters are normalized to [0, 1], while selecting them requires domain knowledge and carries some subjectivity.
- 3.3. Evaluation Methods: Listening tests sample 40 target–imitator audio pairs per loss and scenario, with four blinded authors rating each pair on a 5-point Likert scale.The listening tests serve as the primary ground truth for verifying the automatic evaluation.
- Implementation: Optimization uses differentiable JAX functions, RMSProp with learning rate 0.045 for 200 iterations, and gradient clipping at an L2 norm of 1.The synthesizers are written in Faust and transpiled into JAX via DawDreamer; the pipeline is faster than real-time and parallelizable.
- 3.5. Bootstrapping and Ranking Procedures: Figure 1 displays bootstrapped 1/PPD or in-domain P-Loss distributions above mean Likert distributions, with colors and numbers indicating NPSK rank.Both axes are oriented so higher values are better; panel titles report ICC3k reliability.
- 3.5. Bootstrapping and Ranking Procedures: 1,000 bootstrap resampled means produce nonparametric performance distributions, and NPSK ranks statistically distinct distributions with rank 1 as best.Because PPD is a distance, the plotted automatic measure is 1/PPD so higher values match the direction of Likert ratings.
4. SCENARIOS AND RESULTS
Across controlled scenarios, loss-function performance depends on the synthesis method: spectrogram losses recover filter cutoffs, DTW_Envelope recovers modulation rates, and JTFS handles smooth pitch trajectories.
- Scenario design: The scenarios use simple synthesis methods and restrict searchable parameters to those dominating each sound’s perceptual identity.This isolates the interaction between loss function and synthesizer while keeping critical-parameter selection unambiguous.
- Band-pass matching: SIMSE_Spec ranks first for band-pass matching under both PPD and listening tests, although the remaining rankings disagree.The target and imitator share low- and high-pass cutoffs but use different source signals, making exact replication impossible.
- Amplitude modulation: DTW_Envelope ranks first by PPD in all three amplitude-modulation scenarios, accurately recovering modulation rate.Listeners agree in the non-overlapping-carrier case but prefer spectrogram losses in overlapping-carrier cases for better timbral similarity.
- Pitch-bending: JTFS ranks first in delayed and no-delay pitch-bending scenarios under both parameter-based and listening-test rankings.The result is consistent with JTFS’s reported agnosticism to small time and frequency shifts.
5. DISCUSSION
The discussion finds that synthesis-dependent loss behavior generalizes to OOD settings, while PPD is useful but must be interpreted alongside perception when multiple factors compete.
- Loss-function dependence: SIMSE_Spec is best for band-pass matching, DTW_Envelope for amplitude modulation and pulsating pitch, and JTFS for smooth pitch trajectories.L1_Spec ranks first in only one case, whereas SIMSE_Spec is never ranked last.
- Evaluation agreement: PPD agrees with listening tests on the top-ranked loss in five of seven scenarios.The two disagreements occur in overlapping-carrier AM, where PPD favors modulation-rate accuracy but listeners favor timbral similarity.
- Listening-test reliability: Listening-derived rankings are robust to leave-one-rater-out ablation, with unchanged top-ranked losses and mean rank correlation ρ = 0.98.Pooled inter-rater reliability is ICC3k = 0.86, p < 0.001.
- Practical implications: PPD is viable for OOD evaluation when critical shared parameters are perceptually dominant, but it is not intended for arbitrary synthesizer pairs.The discussion recommends pairing PPD with a small listening panel.