Source-linked AI summary
SCAPES: Semantically Conditioned Autoregressive Prior for Environmental Sounds
Esteban Gutiérrez, Lonce Wyse, Frederic Font, Xavier Serra
TL;DR
Generative environmental sound synthesis often relies on large datasets and resource-intensive models, while semantic controls remain coarse for detailed waveform generation. SCAPES combines overlapping latent segmentation with a segment-autoregressive Continuous Normalizing Flow over continuous EnCodec representations. The model produces high-fidelity environmental textures with semantic adherence and long-term stability, while supporting continuous interpolation and lightweight, open experimentation.
Problem
Semantic control is intuitive for environmental sounds, but coarse descriptors do not provide the fine-grained structure needed to synthesize complex waveforms, encouraging large-scale resource-intensive systems.
Method
SCAPES uses overlapping audio segments, continuous EnCodec latents, and a Flow-Matching-trained Continuous Normalizing Flow applied autoregressively across segments with semantic conditioning.
Results
SCAPES produces high-quality stereo environmental textures with strong semantic adherence and long-term autoregressive stability across 10 acoustic categories using a 36-million-parameter architecture.
Takeaways & Limitations
SCAPES provides a lightweight framework for semantically controlled environmental sound synthesis and continuous interpolation between semantic states.
Takeaways & Limitations
SCAPES struggles with speech and complex polyphonic music because its 10 Hz control rate and limited memory buffer lack a global receptive field for long-term structure.
Abstract
from arXiv · showhide
As generative audio models grow in complexity, the computational and ecological costs of synthesizing everyday sounds have become increasingly prohibitive, often requiring industrial-scale resources and massive datasets. In this paper, we present SCAPES: a Semantically Conditioned Autoregressive Prior for Environmental Sounds. SCAPES is a lightweight, resource-efficient generative model designed to synthesize high-fidelity environmental textures through high-level semantic control. By operating on the continuous latent manifold of a neural audio codec, our approach bypasses the rigid structural constraints inherent to discrete tokenization. We propose a segmentation strategy that decomposes audio into overlapping segments, enabling a Continuous Normalizing Flow (CNF) to model the evolution of latent trajectories using Flow Matching. Our experiments demonstrate that a 36-million parameter instance of SCAPES can be trained on limited, uncurated datasets using a single consumer-grade GPU. Notably, convergence is achieved after training for approximately twice the source audio duration, yielding high-fidelity outputs with robust long-term stability and semantic consistency. Furthermore, we showcase the model's capacity for smooth semantic interpolation, providing a flexible and accessible tool for open research and creative sound design. Code, pretrained weights, audio examples, and an interactive demo are publicly available on our project page https://cordutie.github.io/projects/scapes.html
1. INTRODUCTION
Environmental sound generation benefits from semantic control, but mapping coarse descriptors to detailed waveforms has driven reliance on large, resource-intensive systems. SCAPES addresses this with a lightweight, semantically conditioned model operating autoregressively over continuous latent segments.
- Everyday listening prioritizes physical sources, materials, and semantic meaning over abstract properties such as pitch, loudness, and timbre.
- Environmental sounds span structured events, including keyboard strokes and applause, as well as stochastic textures such as rivers, fire, wind, and rain.
- Controllability remains challenging because environmental sounds are diverse and lack inherent structure, with available controls ranging from signal properties to semantic features.
- Semantic features are intuitive controls, but their coarseness makes mapping them to dense audio waveforms highly underconstrained.
- Large-scale text-to-audio systems dominate semantic control, making even simple environmental sounds resource-intensive and ecologically costly.
- SCAPES uses a small Continuous Normalizing Flow conditioned on continuous EnCodec latents, applying segment-level autoregression over prior latent states and CLAP embeddings.
2. BACKGROUND
The paper grounds SCAPES in continuous neural audio codecs and continuous-time generative flows. Its evaluation combines FAD and KAD across representation spaces to assess distributional similarity, semantic information, and texture statistics.
- Neural audio codecs compress high-fidelity audio into low-dimensional latent representations before reconstructing the waveform.
- EnCodec represents normalized 48 kHz audio in a compact 128-dimensional continuous latent space while retaining a scale factor for dynamic-range restoration.
- Continuous Normalizing Flows model transformations between distributions as continuous-time dynamics governed by a time-dependent vector field.
- Flow Matching trains a parameterized neural vector field to approximate conditional fields that transport samples from a base distribution toward target data.
- KAD provides a non-parametric alternative to FAD by comparing distributions in a kernel-induced feature space without restrictive Gaussian assumptions.
- Evaluation uses FAD with VGGish embeddings and KAD with CLAP and TexStat embeddings to cover baseline, semantic, and perceptual texture representations.
3. MODEL DESIGN
SCAPES models environmental sound as sequentially generated continuous latent atoms, combining local acoustic memory with fine-grained semantic conditioning. Overlapping segmentation, continuous-flow synthesis, and overlap-and-add reconstruction support coherent long-form generation.
- Data representation: SCAPES represents environmental audio as localized continuous EnCodec latent segments called atoms, each conditioned on preceding atoms and a semantic context embedding.The target atom is modeled from a temporal memory of previous atoms and high-level CLAP-derived context.
- Data representation: Continuous latent representations avoid the disjoint index structure of discrete tokenization and preserve the natural continuity of the audio signal.The framework is codec-agnostic when the codec provides continuous latents, and this work validates it with 48 kHz non-causal EnCodec.
- Data representation: Overlapping segments and a joint latent-scale representation address boundary artifacts caused by non-causal convolutions and segment-level normalization.Each atom contains 128 latent dimensions plus one broadcast global scale-factor dimension across N temporal frames.
- Data representation: Semantic annotations use CLAP embeddings from short target-centered context windows extended to 7 seconds through randomized overlapping repetition.This pairing provides finer temporal annotation while satisfying the CLAP implementation’s minimum input duration.
- Architecture and flow matching: A Transformer parameterizes the conditional vector field, using local self-attention, memory cross-attention, and semantic conditioning to generate target atoms.The model applies Conditional Flow Matching independently to latent and scale dimensions, with noise as the base distribution and previous atoms plus semantic context as conditioning.
- Inference and continuous synthesis: At inference, an ODE solver integrates the conditioned vector field from t = 0 to t = 1 for each atom, after which modified overlap-and-add reconstructs a continuous stream.A sliding memory buffer is updated recursively, and a 0.02 s crossfade helps maintain phase continuity between adjacent atoms.
4. TRAINING AND EVALUATION
SCAPES is evaluated on limited environmental-sound data for efficiency, generation quality, semantic adherence, long-term stability, and semantic interpolation. The results indicate resource-efficient training, strong distributional similarity and class adherence, stable long-term behavior, and smooth transitions between soundscapes, with objective evaluation of hybrid sounds remaining open.
- Dataset and training: Approximately 34 minutes of unedited Freesound audio spanning 10 environmental categories supported the evaluation.The dataset included sounds such as applause, bonfires, bubbling water, forests, helicopter rotors, keyboard strokes, rain, rivers, and wind.
- Dataset and training: A 36-million-parameter SCAPES model trained for 120 epochs on one RTX 4090 used under 8 GB VRAM and approximately one hour.Training used Conditional Flow Matching with a batch size of 32.
- Generation quality and resynthesis: SCAPES generally achieved superior distributional similarity to RAVE across FAD (VGGish), KAD (CLAP), and KAD (TexStat), despite RAVE’s strong signal reconstruction.The comparison was asymmetrical because RAVE learns its latent space from scratch, whereas SCAPES models distributions on a frozen neural audio codec.
- Generation quality and resynthesis: SCAPES showed more robust performance on highly noisy sounds, suggesting Flow Matching captures environmental texture statistics without an explicit reconstruction objective.The authors frame the explanation as a hypothesis about RAVE’s difficulty generalizing from the relatively homogeneous dataset.
- Semantic conditioning and class adherence: Generated samples were closer to reference data from their intended class than to other categories across VGGish, CLAP, and TexStat representations.The consistent diagonal structure indicates semantic adherence is also preserved in low-level perceptual texture statistics, not only in CLAP space.
- Semantic interpolation: Interpolations produced smooth transitions between related and unrelated environmental classes, including coherent hybrids between keyboard strokes and bonfire crackling.The keyboard-to-bonfire transition gradually changed rhythmic impulses into stochastic crackling while adding a low-frequency bonfire hum.
- Semantic interpolation: Objective evaluation of hybrid sounds remains an open challenge, although the interpolation results suggest generalization beyond explicitly represented training regions.The authors connect this capability to potential fine-grained creative sound design.
5. DISCUSSION
SCAPES emphasizes openness, adaptability, and semantic controllability through a lightweight framework for environmental sound synthesis. Its interpolation strategy supports transitions between semantic states, while its localized design limits performance on signals requiring long-term structure.
- Openness and adaptability: SCAPES offers an open-source, computationally efficient pipeline that can be adapted to domain-specific datasets using consumer-grade hardware.The framework requires no manual annotations and includes the core architecture, curated data, and pretrained weights.
- Controllability: SCAPES supports indefinite non-repeating textures, creative continuation, and semantic interpolation through several controllable generation modes.Its relaxed conditioning treats semantic prompts as directional guides for transitions rather than rigid instantaneous constraints.
- Semantic interpolation: Optimal Transport Flow Matching provides a regularized continuous vector field for traversing plausible intermediate states between sparse semantic clusters.The generative process follows smooth deterministic ODE trajectories while satisfying autoregressive memory and shifting semantic targets.
- Limitations: SCAPES struggles with speech and complex polyphonic music because its 10 Hz control rate and limited memory buffer lack the global receptive field for long-term structure.The authors describe these tasks as underspecified when predicting the next 0.1 seconds from local memory and high-level semantics.
6. CONCLUSIONS AND FUTURE WORK
SCAPES combines continuous latent representations, Continuous Normalizing Flows, and Flow Matching for lightweight environmental sound synthesis. The authors report high-quality, semantically adherent, stable textures and smooth interpolation, while future work targets conditioning flexibility and longer temporal context.
- Conclusions: SCAPES combines Continuous Normalizing Flows, Flow Matching, and continuous EnCodec latents into a lightweight environmental sound synthesis framework.
- Conclusions: A 36-million parameter architecture produces high-quality stereo textures with strong semantic adherence and long-term autoregressive stability across 10 acoustic categories.
- Conclusions: Operating on a continuous latent manifold enables SCAPES to interpolate smoothly between disparate semantic states.The paper identifies this capability as challenging for discrete token-based architectures.
- Future work: Future work may replace or augment CLAP conditioning with low-level acoustic features or human-made annotations.
- Future work: Future work may expand the localized receptive field through hierarchical attention or sparse transformers to address long-term dependencies in speech and music.