PaperlayerView exact revision on arXiv

Source-linked AI summary

Sketch2Sound: Controllable Audio Generation via Time-Varying Signals and Sonic Imitations

Hugo Flores García, Oriol Nieto, Justin Salamon, Bryan Pardo, Prem Seetharaman

arXiv:2412.08550v2cs.SDeess.AS

TL;DR

Sketch2Sound addresses the challenge of creating temporally expressive sounds from text alone by conditioning a text-to-audio latent diffusion model on interpretable controls extracted from sonic imitations. Median-filtered controls provide adjustable temporal specificity, and experiments show control following with text adherence and audio quality comparable to a text-only baseline. The model offers sound artists a lightweight way to combine text semantics with gestural sonic input.

  • Problem

    Text descriptions do not directly provide the gestural and temporally aligned control needed for sound design, while vocal imitation communicates audio concepts without symbolic abstraction.

    Source: §I · p. 1

  • Method

    Sketch2Sound adds loudness, brightness, and pitch controls to a text-conditioned latent diffusion transformer, using median filtering to vary temporal specificity.

    Source: §I · pp. 1–2, B. Conditioning a latent audio DiT on… · p. 2

  • Results

    Sketch2Sound follows vocal-imitation control signals while retaining high text adherence and audio quality comparable to a text-only pretrained model.

    Source: §I · p. 2

  • Takeaways & Limitations

    Sound artists can combine the semantic flexibility of text prompts with the expressivity and precision of sonic gestures or vocal imitations.

    Source: §I · p. 2, §VI · p. 4

    Centroid control can entangle input room tone with generated audio, and dropping controls during quiet regions may fail for overcompressed inputs or loud room tones.

    Source: E. Limitations · p. 4

Abstract

from arXiv · show

We present Sketch2Sound, a generative audio model capable of creating high-quality sounds from a set of interpretable time-varying control signals: loudness, brightness, and pitch, as well as text prompts. Sketch2Sound can synthesize arbitrary sounds from sonic imitations (i.e.,~a vocal imitation or a reference sound-shape). Sketch2Sound can be implemented on top of any text-to-audio latent diffusion transformer (DiT), and requires only 40k steps of fine-tuning and a single linear layer per control, making it more lightweight than existing methods like ControlNet. To synthesize from sketchlike sonic imitations, we propose applying random median filters to the control signals during training, allowing Sketch2Sound to be prompted using controls with flexible levels of temporal specificity. We show that Sketch2Sound can synthesize sounds that follow the gist of input controls from a vocal imitation while retaining the adherence to an input text prompt and audio quality compared to a text-only baseline. Sketch2Sound allows sound artists to create sounds with the semantic flexibility of text prompts and the expressivity and precision of a sonic gesture or vocal imitation. Sound examples are available at https://hugofloresgarcia.art/sketch2sound/.

I. INTRODUCTION

Sketch2Sound addresses the temporal and gestural limitations of text-only sound design by combining text prompts with interpretable control signals extracted from sonic imitations. Its median-filtered controls support flexible temporal precision while retaining text adherence and audio quality.

  • Text-to-audio systems leave designers to modify generated sounds’ temporal characteristics for synchronization, unlike gestural Foley performance.Source: §I · p. 1
  • Vocal imitation provides a natural, non-symbolic way to communicate pitch, timing, and timbre, complementing language-based sound descriptions.Source: §I · p. 1
  • Sketch2Sound generates sounds from loudness, brightness, and pitch controls extracted from audio, including vocal imitations, alongside text prompts.Source: §I · pp. 1–2
  • Random median filtering varies control-signal temporal detail, letting artists trade strict imitation adherence against sound quality and text adherence.The method is intended to improve quality when vocal imitations cannot reproduce target sounds’ fine temporal behavior.Source: §I · p. 2
  • The approach is lightweight, requiring 40k fine-tuning steps and one linear layer per control, and can use arbitrary sonic imitations beyond voices.Source: §I · p. 2
  • Experiments show close control following from vocal imitations while maintaining high text adherence and audio quality comparable to a text-only pretrained model.Source: §I · p. 2

II. METHOD

The method conditions an audio latent diffusion model on interpretable, time-varying controls to create sound variations and new sounds through text-prompted sonic imitations.

  • Sketch2Sound conditions an audio latent diffusion model on interpretable, time-varying signals for expressive sound creation through text-prompted sonic imitations.Source: §II · p. 2

A. Time-varying control signals for sound objects

Sketch2Sound uses momentary audio descriptors as time-varying controls for sound generation, with additional control signals permitted beyond the core set.

B. Conditioning a latent audio DiT on time-varying control signals

Sketch2Sound adapts a pretrained text-to-sound latent diffusion transformer by projecting aligned control sequences into its noisy latents. Control dropout and guidance allow partial control and separate text-control strength, while median-filtered sketches improve the quality–adherence trade-off.

B. Conditioning a latent audio DiT on time-varying control signals

C. Creating sketchlike controls via control-rate filtering

Sketch2Sound makes control signals sketchlike by applying random median filters before model input, addressing temporal mismatch between vocal imitations and target sounds. During inference, artists can adjust control rate to trade fine temporal precision against text-prompt adherence.

III. EXPERIMENTAL DESIGN

The experiments evaluate Sketch2Sound on vocal-imitation sound synthesis using the VimSketch dataset and compare audio quality, text adherence, and control-signal adherence. Models are fine-tuned from a text-only checkpoint, except for the text-only baseline.

  • Models are fine-tuned for 40k steps from the text-only baseline checkpoint, while the text-only baseline requires no fine-tuning.Source: §III · p. 3
  • Evaluation uses approximately 12k VimSketch vocal imitations with text descriptions and reference sounds, generating 10k examples of up to 5 seconds per model variant.Source: §III · p. 3
  • Audio quality is measured with Frechét Audio Distance using VGGish and LAION-CLAP embeddings against a proprietary high-quality sound-effects reference set.Source: §III · p. 3
  • Text adherence is measured by CLAP embedding cosine similarity between generated audio and each target text prompt.Source: §III · p. 3
  • Control adherence uses L1 errors for loudness, centroid, pitch, chroma, and periodicity, with silence and unvoiced frames excluded as specified.Source: §III · p. 3

IV. EXPERIMENTS

The first experiment tests whether Sketch2Sound can synthesize from vocal imitations and text while maintaining audio quality, text adherence, and control adherence. It compares control configurations against a text-only baseline using a fixed inference filter size.

A. Control signals

  • Sketch2Sound is evaluated against a text-only baseline for audio quality, text adherence, and adherence to vocal-imitation controls.Source: A. Control signals · p. 3
  • Three fine-tuned variants use loudness, loudness plus centroid, or loudness plus centroid plus pitch as control signals.Source: A. Control signals · p. 3
  • All model variants use a fixed median-filter window size of 10 during inference.Source: A. Control signals · p. 3

B. Sketch type ablation

The ablation compares median filtering with no filtering and low-pass filtering, then examines how inference-time median-filter size changes the balance between sketchlike control and precision. Larger filters favor sketchlike inputs and potentially higher quality, while smaller filters preserve precision.

B. Sketch type ablation

  • Median-filter training is compared with no filtering and random low-pass filtering to remove fine temporal detail from control signals.Median-filter windows range from 1–25 control frames, while low-pass filters range from 5Hz–20Hz with a fixed 10Hz inference cutoff.Source: B. Sketch type ablation · p. 3

C. Inference-time control rates

  • Inference evaluates median-filter sizes of 1, 5, 10, 15, 20, and 25 to test whether control rate trades text adherence against fine-temporal control.Source: C. Inference-time control rates · p. 3
  • Larger inference-time median filters produce more sketchlike controls and can yield higher audio quality, whereas smaller filters are more precise but may reduce quality when vocal imitations are imprecise.Source: Figure 2 · p. 4

V. RESULTS AND DISCUSSION

Sketch2Sound’s controls substantially improve adherence to time-varying loudness, centroid, and pitch signals, while text adherence and audio quality remain near the text-only baseline. Median filtering improves quality and text adherence for sketchlike vocal imitations by trading off strict control matching.

A. Control signals

  • Control conditioning significantly improves adherence to the corresponding loudness, centroid, or pitch signal.A single loudness condition can also slightly improve adherence to other controls because these characteristics are correlated in natural sounds.Source: A. Control signals · p. 4
  • Text adherence and audio quality decrease slightly with controls, but the difference is near negligible versus the text-only baseline in most cases.Better-performed vocal imitations also lead to higher-quality generated sound effects.Source: A. Control signals · p. 4

B. Sketch type ablation

  • Median filtering improves audio quality and text adherence over no filtering and low-pass filtering for sketchlike vocal imitation controls.The method trades off control adherence because exact imitation can produce speechlike audio rather than sounds matching the text prompt.Source: B. Sketch type ablation · p. 4

C. Inference-time control rates

Sketch2Sound supports user-adjustable temporal detail in its controls and can use control curves to shape the semantics of generated sounds. Examples show loudness and pitch structure selecting contextually appropriate sound events under text prompts.

C. Inference-time control rates

  • Inference-time median-filter selection lets sound artists choose the temporal detail of control signals.The control-rate filtering method supports different-size median filters during inference.Source: C. Inference-time control rates · p. 4

D. The semantics of control curves are implicitly modeled

E. Limitations

A limitation is that centroid conditioning can transfer room tone from the input imitation into the generated audio. The conclusion nevertheless presents Sketch2Sound as a lightweight, temporally flexible tool for text- and control-conditioned sound generation.

E. Limitations

  • Centroid control can entangle the input imitation’s room tone with the generated audio.Dropping controls during quiet segments may help, but can fail for overcompressed inputs or loud room tones.Source: E. Limitations · p. 4

VI. CONCLUSION

  • Sketch2Sound combines text prompts with loudness, brightness, and pitch controls, using 40k fine-tuning steps and one linear adapter layer per control.The conclusion describes flexible temporal specificity and sonic-imitation-based generation for sound artists.Source: §VI · p. 4

Related papers

Loading 2412.08550v2…