Source-linked AI summary
Relational Synthesis: Structure-Mediated Concatenative Synthesis for Foley and Retrieval-Augmented Audio Generation
Keren Shao, Ayaka Kawano, Shlomo Dubnov
TL;DR
The paper asks whether a retrieved source can provide a Foley output’s acoustic identity while a separate reference controls its timing, without either waveform being persistently copied. It introduces relational synthesis, which uses reference-side relations to reorganize source grains. The reported results show improved reference-onset agreement and a complementary FO Event-L1 improvement, with source-carrier-level reference persistence and distributional checks.
Problem
Foley synthesis must coordinate acoustic identity and temporal structure, while retrieval-augmented audio can provide useful acoustic evidence but risks persistent copying when audio is directly conditioned.
Source: §1 · p. 1Method
Relational synthesis replaces pointwise reference-to-source grain matching with reference-side recurrence and directed amplitude relations that guide framewise selection from the source.
Source: §1 · p. 1Results
FO raises reference-onset agreement from .322 for Raw to .343 and lowers Event-L1 from .095 to .086; its reference persistence remains near source-carrier controls, while prompt and distributional checks change modestly.
Source: §3.3 · p. 4Takeaways & Limitations
Reference timing can improve without passing the reference waveform through the generator, while source-carrier controls indicate the source is not simply replayed locally.
Source: §3.3 · p. 4The formulation assigns acoustic identity to the retrieved source and temporal organization to the independent reference, while seeking to keep persistent local replication of both small.
Source: §1 · p. 1
Abstract
from arXiv · showhide
We ask: given a retrieved source audio $S$ and a separate reference audio $R$, can we synthesize novel audio $Y$ out of this pair $(S,R)$ such that $Y$ remains acoustically consistent with $S$, while not persistently copying segments of $S$ or $R$? The first clause is a well-known goal in Foley audio production, and the second is a well-known issue in neural RAG when $S$ and $R$ are naively injected into neural generators. We show that both clauses can be addressed simultaneously using a method we coin relational synthesis, a variation of concatenative synthesis where target cost is replaced by a relational Gromov-like structural cost. Rather than imitating the content of $R$, relational synthesis exploits it from the "other side of the hill": it transfers the temporal structure and directed amplitude motion of $R$ to reorganize and concatenate the grains of $S$ in a novel manner that protects $S$'s acoustic information. Our experiments show that relational synthesis integrates naturally with neural RAG and produces Foley audio that performs well on metrics measuring temporal agreement, acoustic fidelity, and leakage persistence, while maintaining distribution-level quality and text alignment.
1. INTRODUCTION
Foley synthesis must preserve a source’s acoustic identity while controlling how an event unfolds, but direct audio conditioning risks copying retrieved or reference material. Relational synthesis addresses this tension by reorganizing source grains according to reference-side temporal relations rather than matching each reference frame directly.
1. INTRODUCTION
- Foley requires control over both what a sound is and when and how it happens, while retrieval-augmented audio can supply acoustic evidence that is difficult to specify in text.Directly conditioning on retrieved or reference audio also risks letting that material persist as copied audio.Source: §1 · p. 1
- The target is an output that preserves the source’s acoustic identity, follows the reference’s temporal organization, and minimizes persistent local copying of either waveform.The paper assigns acoustic identity to S, temporal organization to R, and treats persistent replication as a separate constraint.Source: §1 · p. 1
- Pointwise matching creates a granularity dilemma: fine grains can follow timing but may mismatch local dynamics, whereas longer segments preserve source continuity but reduce temporal freedom and increase copying risk.Longer local matches may sacrifice framewise control and leave longer contiguous spans of the source in the output.Source: §1 · p. 1
- Relational synthesis retains framewise source selection but uses reference-side recurrence and directed amplitude relations to guide grain choices without requiring pointwise acoustic matching.The method specifies recurrence with CRG, FO, or VMO models and amplitude structure with directed amplitude motion, then solves grain selection with an annealed discrete solver.Source: §1 · p. 1
2. METHODS
Relational synthesis selects source grains frame by frame while using reference-side relations, rather than pointwise acoustic matching, to guide their arrangement. The section develops three recurrence-based relation models, adds directed amplitude-motion matching, and solves the resulting nonlocal objective with an annealed discrete search.
2. METHODS
- A framewise path selects source grains to realize reference times, while a Gromov-like cost couples within-domain relations instead of matching reference and source acoustics pointwise.The formulation uses reference-side structure, not a joint relational match between S and R.Source: §2 · pp. 1–2
2.1. Contextual Recurrence Graph
- CRG estimates recurrence from weighted similarities along local diagonals, capturing whether two times’ surrounding contexts evolve similarly.Its contextual recurrence graph uses a contextual radius and diagonal decay scale.Source: §2.1 · p. 2
2.2. Factor-Oracle Graph
- FO quantizes reference embeddings and assigns sparse positive relations to suffix-linked times whose longest repeated suffix reaches a minimum length of three.Negative relations retain CRG’s contextual-recurrence construction.Source: §2.2 · p. 2
2.3. Variable-Markov-Oracle Graph
- VMO jointly determines acoustic clusters and repeated context during oracle construction, selecting its clustering threshold by information-rate optimization.Unlike FO’s separate quantization and oracle construction, VMO uses a grid search over thresholds; the three models are parallel alternatives, not successive refinements.Source: §2.3 · pp. 2–3
2.4. Directed Amplitude Motion
- Directed amplitude motion complements recurrence relations by matching the relative extremeness of source and reference log-amplitude changes across time lags, without matching amplitudes directly.An insensitivity tolerance of ε = .1 is used in percentile space.Source: §2.4 · p. 3
2.5. Annealed Discrete Solver
- The recurrence and amplitude-motion objectives create a nonlocal, loopy interaction graph, so the authors use an annealed discrete search rather than standard Viterbi recursion.The search uses eight parallel restarts and twelve random-order coordinate sweeps, and does not guarantee a global optimum.Source: §2.5 · p. 3
3. EXPERIMENTS
The experiments evaluate relational synthesis on environmental-sound targets paired with vocal-imitation references, using separate measures for temporal transfer, acoustic fidelity, leakage, prompt alignment, and distributional quality. FO improves timing while preserving target-class acoustic fidelity and reducing local source persistence; directed amplitude motion strengthens temporal transfer across relational methods.
3.1. Setup
- Evaluation uses 248 distinct (S, R, T) triplets: same-class sources are sampled independently from Folds 3–5 as oracle retrieval, with Fold-2 targets and corresponding vocal-imitation references.The study uses 31 environmental-sound classes, 5-second mono inputs at 22.05 kHz, and a Stable Audio 3 Small-SFX generator without task-specific training or an added conditioning network.Source: §3.1 · p. 3
3.2. Metrics
- Metrics separate temporal agreement, target-class acoustic similarity, source and reference persistence, prompt alignment, and distributional quality; leakage similarities are diagnostic, not minimization objectives.Pair-level metrics average three neural seeds per triplet, while KL, Inception Score, and Fréchet distances are set-level measures.Source: §3.2 · p. 3, Table 1 · p. 4
3.3. Results and Discussion
- FO raises onset agreement from Raw's OR = .322 to OR = .343 and lowers Event-L1 from .095 to .086, while reducing local source persistence R2s_S from .353 to .299.FO's target CLAP changes from .552 to .551 and target log-mel similarity from .923 to .925; reference persistence is .144 versus .190 for Raw, while Direct Ref reaches .365.Source: §3.3 · p. 4
- With directed amplitude motion enabled, LR decreases from .110 to .097 for CRG, .096 to .086 for FO, and .096 to .087 for VMO; onset agreement (OR) rises in each pair.All six temporal deltas have paired 95% CIs excluding zero; acoustic and prompt alignment are generally preserved or improved, while leakage changes are comparatively small and set-level effects are mixed.Source: §3.3 · p. 4