Source-linked AI summary
Rethinking Procedural Audio Pre-training: Source Scaling and Objective Adaptation
Jiajun Peng, Fengrui Liu, Xinyu Liu, Feng Liu
TL;DR
Procedural audio can support transferable representations, but how to scale procedural sources and whether natural-audio training choices transfer remain open questions. The paper varies formula-class coverage and rendering diversity in a controlled source and compares learning configurations across sources. It finds that scaling benefits depend on learner and task, while preferred masking differs by source, motivating source-aware pre-training.
Problem
Procedural audio is a viable representation-learning source, but how its structural and rendering dimensions should be scaled and whether natural-audio training choices transfer remain open questions.
Source: §1 · p. 1Method
The study varies formula-class coverage and within-class rendering diversity independently in FormulaBank and compares learners and sources under controlled pre-training conditions.
Source: §1 · p. 1Results
Scale benefits depend on learning formulation and downstream task, and the preferred masking regime is source-dependent.
Source: §1 · p. 2, §3.2 · p. 3, §3.4 · p. 4Takeaways & Limitations
Procedural pre-training calls for jointly considering source scaling and learning configuration rather than treating masking as a source-independent default.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Procedural audio has emerged as a viable source for transferable audio representation learning, but its design principles remain unclear.We revisit two questions: how a procedural source should be scaled, and whether training choices developed on natural audio should transfer unchanged to procedural data.Using a controlled source, we separate scale into formula-class coverage C and within-class rendering diversity I.Experiments with FDSL and AudioMAE show that these two forms of scale provide different benefits and depend on the learning formulation and downstream task. A matched AudioMAE study further shows that procedural audio favors low mask ratios (10%--25%), whereas AudioSet-28K favors 50%--75%. Shared-codebook analysis reveals lower patch diversity and stronger temporal predictability in procedural audio. These results motivate source-aware procedural pre-training, where source scaling and learning configuration are considered jointly.Code is available at https://github.com/Cross-Innovation-Lab/Formula-Bank.
1. INTRODUCTION
Prior work established that formula-driven and procedural audio can support transferable representation learning,,. This study asks how to scale procedural sources and whether natural-audio training choices transfer, finding that both scale benefits and preferred masking depend on the learner or source.
1. INTRODUCTION
- Earlier work showed representation learning from formula-generated patterns and labels and transferable representations from synthetic and procedural audio, which can complement recordings.Source: §1 · p. 1
- The study varies coverage and rendering diversity independently in FormulaBank and evaluates them across FDSL and AudioMAE with matched-data and matched-compute controls.A matched FormulaBank–AudioSet-28K comparison holds model, data size, optimization, mask-ratio grid, and downstream evaluation fixed while changing the pre-training source.Source: §1 · p. 1, Figure 2 · p. 2
- Procedural scale is multidimensional and learner-dependent, while the preferred masking regime is strongly source-dependent.These findings frame source composition and learning configuration as joint design choices.Source: §1 · p. 2
- Formula-class coverage and within-class rendering diversity provide distinct scale axes whose relative benefits depend on learning formulation and downstream task.The study asks whether procedural scale should be described by sample count alone or by these structural and rendering dimensions.Source: §1 · pp. 1–2
- FormulaBank supports heterogeneous audio pre-training paradigms, so its utility is not tied to a single learning objective.Source: §1 · p. 2
2. METHODOLOGY
The methodology builds a controlled procedural source that separates formula-class coverage from within-class rendering diversity. It evaluates these scaling dimensions across learning paradigms and relates masking behavior to shared-codebook patch statistics.
2.1. Controlled Procedural Source
- FormulaBank varies formula-class coverage and within-class rendering diversity independently, with each class sharing an underlying structure across acoustically varied renderings.The source is constructed from the AudioPG generator.Source: §2.1 · p. 2
2.2. Scaling Procedural Sources
- Fixed-data and fixed-compute controls test whether the value of coverage and rendering diversity depends on the learning formulation rather than additional training exposure.Source: §2.2 · p. 2
2.3. Generality and Source Adaptation
- AudioMAE, BYOL-A, SSAST, and PaSST are pre-trained on the same FormulaBank source, alongside a matched AudioMAE comparison that changes only the pre-training source.The comparison uses the same model, optimization, input representation, and downstream evaluation; r denotes the fraction of masked spectrogram patches.Source: §2.3 · pp. 2–3, Table 1 · p. 3
2.4. Source Statistics
- A shared codebook quantizes FormulaBank and AudioSet-28K patches to relate masking behavior to patch diversity and local temporal predictability.Lower H_t indicates stronger predictability between adjacent time patches.Source: §2.4 · p. 3
3. EXPERIMENTS
The experiments test procedural scaling across learning formulations, transfer across pre-training paradigms, and whether masking choices should change with the source. They find that source scale has learner- and task-dependent effects, while preferred masking regimes differ between FormulaBank and AudioSet-28K.
3.1. Experimental Setup
- The experiments evaluate classification accuracy on ESC-50, US8K, and SCv2, and mAP on FSD50K and AS-20K, with downstream models fully fine-tuned.AudioMAE uses 128-bin log-Mel spectrograms and nonoverlapping 16 × 16 time–frequency patches.Source: §3.1 · p. 3
3.2. Multidimensional Procedural Scaling
- The two scale dimensions are not equivalent: FDSL responds more clearly to broader formula-class coverage, while AudioMAE shows task-dependent saturation for coverage and broader improvements from rendering diversity.These trends persist under controlled budgets, supporting a multidimensional view of procedural scale that depends on learning formulation and downstream task.Source: §3.2 · p. 3, Figure 3 · p. 4
3.3. Generality Across Learning Paradigms
- AudioMAE, BYOL-A, SSAST, and PaSST all produce transferable representations from FormulaBank, evaluating source compatibility rather than ranking the methods.Each paradigm retains its native encoder and training recipe.Source: §3.3 · pp. 3–4, Table 2 · p. 4
3.4. Source-Dependent Masking and Source Statistics
- FormulaBank performs best at mask ratios of .10–.25, while AudioSet-28K favors .50–.75; the source-dependent shift is consistent across downstream tasks and repeated runs.The matched AudioMAE comparison holds the architecture, clip count, mask-ratio grid, and downstream evaluation fixed across sources.Source: §3.4 · p. 4, Figure 4 · p. 4
4. CONCLUSION
The results motivate source–objective co-design for procedural pre-training.
- Masking should be adapted to the statistics of the pre-training source rather than inherited as a source-independent default.Source: §4 · p. 4