Source-linked AI summary
Sobolev Norms in Neural Embeddings Measure Audio Morphing Regularity
Théo Chasle Cauchy, Modan Tailleur, Barbara Pascal, Fanny Roche, Mathieu Lagrange
TL;DR
Evaluating audio morphing is challenging because it is perceptually defined, and existing metrics can focus on a single property or score adversarial trajectories favorably. The paper introduces SDIM, which compares trajectories with an ideal path using Sobolev distances in audio embeddings. Experiments report that SDIM distinguishes regular from adversarial trajectories, with performance depending on the embedding space.
Problem
Human evaluation of morphing is costly and listener-variable, while existing metrics may assess only specific properties and can score some adversarial trajectories favorably.
Source: §1 · p. 1Method
Results
SDIM computed in an appropriate space discriminates trajectories with different regularity and remains robust to specific adversarial trajectories, while recent, larger audio embeddings tend to yield higher regularity scores on controlled regular morphs.
Source: §7 · p. 4Takeaways & Limitations
SDIM jointly measures linearity and uniformity and highlights the importance of considering the overall trajectory when evaluating morphing regularity.
Source: §7 · p. 4The relationship between SDIM metrics and human perceptual judgments of morphing regularity remains for future work.
Source: §7 · p. 4
Abstract
from arXiv · showhide
Morphing has recently gained renewed interest with the emergence of generative models, particularly in audio and image generation. In musical sound synthesis, morphing can generate intermediate sounds between two targets, helping musicians and sound engineers explore new sounds with interesting perceptual properties. As morphing is inherently defined in perceptual terms, evaluating this task is challenging. In this work, we introduce Sobolev Distances to Ideal Morphing (SDIM), a novel objective metric to quantify the regularity of audio morphing trajectories in perceptually relevant audio embedding spaces. Leveraging a physics-based sound synthesizer, we evaluate the discriminative power of SDIM on controlled morphing trajectories with varying degrees of regularity and compare it with that of existing audio morphing metrics. Results show that, contrary to state-of-the-art metrics, the proposed metric reliably discriminates desirable trajectories from adversarial ones.
1. INTRODUCTION
Human evaluation of morphing is costly and can vary across listeners, while existing metrics may focus on one property and score adversarial trajectories favorably. The paper introduces SDIM and evaluates its ability to distinguish trajectory regularity.
1. INTRODUCTION
- Human-based evaluation requires many trajectory judgments and may vary substantially across subjects, motivating efficient objective metrics as complementary measures.The paper notes renewed interest in morphing alongside generative models in image and audio generation.Source: §1 · p. 1
- Existing audio morphing metrics may target only a specific property, such as smoothness, and can yield favorable scores for some adversarial trajectories.These limitations raise concerns about their robustness as unified measures of transitions between sounds.Source: §1 · p. 1
- SDIM compares an evaluated audio-embedding trajectory with an ideal morphing trajectory using Sobolev distances to quantify regularity.The paper evaluates the metric using a physical synthesis model and compares it with state-of-the-art morphing metrics.Source: §1 · p. 1
- SDIM is reported to resist subtly adversarial poor-quality trajectories that still score highly under state-of-the-art metrics; recent, deeper neural audio embeddings improve its ability to capture regularity.The paper also describes SDIM as jointly assessing linearity and uniformity.Source: §1 · p. 1
2. QUALITY OF MORPHING TRAJECTORIES
The section frames audio morphing as a perceptually gradual transition between source and target sounds, with an ideal path that is linear and regular in a perceptually correlated feature space. It describes existing quality criteria and metrics, then defines adversarial trajectory configurations that test whether metrics reflect regularity.
2. QUALITY OF MORPHING TRAJECTORIES
- A morphing trajectory is a sequence of sounds shifting perceptually from source to target; in a perceptually correlated feature space, the ideal is a regular linear interpolation called LUM.A realistic high-quality trajectory is expected to deviate only slightly from this ideal path.Source: §2 · p. 1
- The Null trajectory uses independent random audio samples and departs substantially from the ideal path, while the figure contrasts it with ideal and good-quality trajectories.Figure 1 encodes the ideal as solid green, the good-quality trajectory as solid blue, and Null as dotted red.Source: Figure 1 · p. 2, §2 · p. 2
- Correspondence, Smoothness, and Intermediateness describe sound-part matching, uniformly regular change, and intermediate perceptual properties, respectively; listening tests are costly and listener-variable.Existing metrics map sounds into embeddings and apply geometric criteria: Smoothness MF uses CLAP-distance correlation, while CDPAM-based metrics assess smoothness and intermediateness.Source: §2 · p. 2
- NUC preserves angular alignment but disrupts spacing regularity, whereas EQC keeps ideal source-distance progression while randomizing each intermediate point’s direction.Figure 2 depicts these as orange NUC and violet EQC trajectories.Source: Figure 2 · p. 2, §2 · p. 2
- The Correspondence SM metric uses MFCC embeddings to measure whether the midpoint leans toward source or target, but the paper cautions that this formulation may not align with perceptual correspondence.A value of 0 indicates that the midpoint is equidistant from source and target in the embedding space.Source: §2 · p. 2
3. SOBOLEV DISTANCES TO IDEAL MORPHING
The proposed SDIM approach relies on perceptually correlated embedding spaces to approximate properties that are not directly measurable. Sobolev distances compare both trajectory values and derivatives, and the two SDIM metrics distinguish value-based similarity from a derivative-inclusive measure of directness.
3. SOBOLEV DISTANCES TO IDEAL MORPHING
- The approach assumes that some embedding spaces correlate with human perception and capture much of Smoothness and Intermediateness, which are not directly measurable.The paper uses this hypothesis as the basis for comparing embeddings empirically.Source: §3 · p. 2
- Sobolev distances compare function values and derivatives through a chosen order, jointly reflecting whether a trajectory follows the ideal path and travels it at the same speed.For audio morphing, the trajectory parameter is one-dimensional and the embedding space may have multiple dimensions.Source: §3 · p. 3
- S0,2 measures value-based distance to LUM, while S1,2 also penalizes deviations through derivatives and quantifies trajectory directness.Small distances indicate that an evaluated trajectory shares LUM’s Smoothness and Intermediateness properties.Source: §3 · p. 3
4. PHYSICS-BASED IDEAL AUDIO MORPHING
A physics-based model of vibrating plates provides reference trajectories from wood-like to metal-like sounds. The model’s interpretable parameters include damping, which controls perceived material.
4. PHYSICS-BASED IDEAL AUDIO MORPHING
- LUM reference trajectories run from wood sounds to metal sounds, using a physical model designed to synthesize realistic plate sounds.The study notes that no inverse transform maps an ideal embedding-space trajectory back to sounds, making automated reference generation challenging.Source: §4 · p. 3
- The physical synthesizer models rectangular-drum vibration with a fourth-order partial differential equation and five interpretable parameters.These parameters control fundamental frequency, sound duration, frequency-dependent damping, overtone inharmonicity, and membrane aspect ratio.Source: §4 · p. 3
- Damping sets the modeled material: log(p) = log(0.15) yields metal-like sounds, while log(p) = log(2.0) yields wood-like sounds.The remaining synthesizer parameters are varied within specified admissible ranges, and the resulting metal and wood sound sets are perceptually validated by listening.Source: §4 · p. 3
5. REGULARITY OF NEURAL AUDIO EMBEDDINGS
The section compares audio embeddings by how regularly SDIM scores controlled ideal morphing trajectories. MERT representations, especially the largest model, receive the lowest normalized SDIM values.
5. REGULARITY OF NEURAL AUDIO EMBEDDINGS
- The choice of embedding matters because SDIM’s assessment depends on whether the space captures perceptual properties that should evolve linearly and uniformly.An embedding that captures subtle perceptual cues is expected to give ideal trajectories small S0,2 and S1,2 scores.Source: §5 · p. 3
- The comparison includes general-purpose MFCC, VGGish, and CLAP embeddings alongside music-specific CLAP and three MERT models.The MERT variants differ in model scale and training data, ranging from about 910 hours to 160,000 hours of music.Source: §5 · p. 3
- For each of 1,000 metal-to-wood anchor pairs, trajectories vary damping regularly while holding the other synthesizer parameters fixed.The anchors share frequency, duration, dispersion, and aspect-ratio parameters; SDIM values are normalized by their average on random Null trajectories.Source: §5 · p. 4
- Both SDIM metrics yield their lowest normalized values on LUM trajectories with MERT-v1-330M, which is then used in subsequent experiments.MFCC, VGGish, CLAP, and MS-CLAP instead produce large normalized SDIM values on these trajectories.Source: §5 · p. 4
6. SENSITIVITY TO TRAJECTORIES CONFIGURATIONS IN THE AUDIO DOMAIN
In the MERT embedding space, SDIM metrics are the only evaluated metrics reported as robust to all adversarial trajectory configurations. Angle indicators also distinguish the linearity of the constructed trajectory types.
6. SENSITIVITY TO TRAJECTORIES CONFIGURATIONS IN THE AUDIO DOMAIN
- The comparison covers ideal linear-uniform LUM, nonuniform NUC, nonlinear EQC, and random Null trajectories.NUC varies the synthesizer’s damping nonuniformly; EQC points are selected from 50,000 synthesized sounds to match the ideal trajectory’s source-distance progression in the embedding space.Source: §6 · p. 4
- LUM angles cluster near perfect alignment, NUC angles are slightly more dispersed, and EQC angles are highly dispersed, indicating nonlinearity.The histograms measure angles between the source-to-intermediate segment and the source-to-target direction across 11 intermediate points and 1,000 trajectories.Source: §6 · p. 4, Figure 3 · p. 4
- Only the SDIM metrics are robust to all adversarial configurations among the six metrics averaged over 1,000 realizations per trajectory type.Equivalent results were also obtained when the state-of-the-art metrics were computed in their original embedding spaces.Source: §6 · p. 4
7. CONCLUSION
The conclusion presents SDIM as a joint measure of trajectory linearity and uniformity and reports that it discriminates controlled trajectories, including adversarial ones. Human-judgment alignment remains future work.
7. CONCLUSION
- SDIM’s geometrical properties jointly measure trajectory linearity and uniformity in a single metric, intended for ranking sound-synthesizer designs.The conclusion describes these properties as theoretically grounded.Source: §7 · p. 4
- In experiments using a physics-based ideal morphing model, SDIM in an appropriate embedding discriminates trajectories with different regularity and remains robust to specific adversarial trajectories.The authors also report that recent and larger audio embeddings tend to yield higher regularity scores on controlled regular trajectories.Source: §7 · p. 4
- Future work will examine how SDIM relates to human perceptual judgments of morphing regularity.The authors also hypothesize that SDIM could extend to other regular, perceptually informed embedding spaces, including image and video spaces.Source: §7 · p. 4