Here the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generation

1 Introduction

• Existing VA models can generate two-channel audio without guaranteeing meaningful stereo spatial cues, and visual motion tracks do not directly specify auditory location.The proposed dynamic spatial correspondence links source motion with the evolution of perceived auditory location over time.Source: §1 · p. 2
• StereoBind combines VMB for source-aware motion binding, STE for absolute spatial conditioning, and RT-RoPE for relative motion modeling.The framework uses motion tracks to condition both video and stereo audio.Source: §1 · pp. 2–3
• StereoWorld-29K combines real-video spatialization and synthetic track-controlled scenes to provide scalable, interpretable supervision for source motion and stereo audio.The dataset addresses the scarcity of reliable stereo cues in existing large-scale VA datasets and the cost of calibrated real-world spatial recording.Source: §1 · pp. 2–3
• StereoWorldBench measures alignment between generated stereo audio and source motion, complementing conventional audiovisual quality metrics.The paper presents StereoWorldBench as a benchmark for evaluating dynamic spatial correspondence.Source: §1 · p. 3
• The paper presents StereoBind as the first track-conditioned stereo VA framework, with StereoWorld-29K and StereoWorldBench as the first large-scale dataset and benchmark for the task.The first-of-its-kind claims are qualified as being to the authors’ best knowledge.Source: Figure 1 · p. 2, §1 · p. 3

2.1 Audio-Video Joint Generation

• Joint VA methods progress from cross-modal attention and adapted image DiTs to dual-stream models, with other approaches adding spatiotemporal priors, modality-specific modeling, and synchronization-aware training.These advances target semantic correspondence and temporal synchronization, while motion-to-auditory-location alignment remains largely unexplored.Source: §2.1 · p. 3

2.2 Stereo Audio Generation

• Stereo audio work includes directional and multi-speaker control, visual guidance, scene geometry, and stereo understanding, yet dynamic correspondence between source motion and auditory cues remains underexplored.Examples include Spatial-Sonic, ISDrama, ViSAGe, FoleyDesigner [18], SonoWorld [14], and Sonic4D.Source: §2.2 · pp. 3–4
• Figure 2 depicts StereoWorld-29K construction through a VAS pipeline for source localization, tracking, and track-guided stereo rendering, and an SDS pipeline for diverse track-controlled scenes.The construction combines curated real-world video with synthetic audiovisual data.Source: Figure 2 · p. 4

3 StereoWorld-29K Dataset

• StereoWorld-29K combines natural real-world audiovisual content with synthetic scenes that provide more diverse, controllable source motion.The real-world branch has limited motion diversity, and both branches require reliable sound-source trajectories for spatial supervision.Source: §3 · p. 4

3.1 Data Curation

• Curated clips come from VGGSound, OpenHumanVid, FoleyBench, and LU-AVS, then undergo quality and motion-aware semantic filtering.The retained clips feature a spatially identifiable audio source whose motion is temporally coherent relative to the camera.Source: §3.1 · p. 4

3.2 Visual-Audio Spatialization Pipeline

• The VAS pipeline grounds audible sources, selects and tracks a matching visual instance, then uses estimated depth and camera parameters to render track-conditioned stereo audio.It back-projects the 2D track into camera-centered 3D, computes room impulse responses along the trajectory, and convolves them with mono source audio.Source: §3.2 · p. 5

3.3 Spatial Data Synthesis Pipeline

• The SDS pipeline first samples diverse spatial scene specifications and then converts them into prompts for VA synthesis.Stage 1 samples motion and semantic attributes and creates object-centric or human-centric scenes; Stage 2 uses separate prompt generators, with higher sampling temperature for diversity and lower temperature to preserve planned details.Source: §3.3 · p. 5

4.1 Problem Formulation and Overview

• StereoBind builds on LTX-2.3 and uses motion tracks to align visible-source trajectories with evolving perceived sound locations, without explicitly estimating ITDs or ILDs.Source: §4.1 · p. 6

4.2 Visual Motion Binding

• VMB uses learnable tokens to integrate the motion reference, target-video state, and audio stream, then supplies the fused representation to audio through cross-attention.Its block sits before Joint VA Attention in each transformer block.Source: §4.2 · pp. 6–7

4.3 Spatial Track Encoder

• STE shares temporally structured track features across blocks, then layer-specific modulation adapts VMB retrieval while gated residual updates preserve the pretrained audio residual pathway.Source: §4.3 · p. 7

4.4 Residual Track RoPE

• RT-RoPE resamples tracks to audio-token length and adds bounded, axis-specific residual phases to temporal RoPE; LoRA adapts audio attention, encoding signed relative displacement.Source: §4.4 · p. 8

5.2 Experimental Results.

• ILD-W 2.754, SELD-Acc 0.757, SMR-Err 14.86, AST-Ang 0.465, and AST-Cal 0.314; StereoBind outperforms all baselines on stereo spatial fidelity.It also achieves the best Motion Smoothness, Imaging Quality, Audio PQ, and AV Sync, while maintaining comparable Subject Consistency and audio quality.Source: Table 1 · p. 9, §5.2 · p. 10

5.1 Experimental Settings.

• The evaluation compares joint VA and V2SA baselines across visual quality, audio quality, and stereo spatial fidelity, using directional prompt descriptions for VA baselines.StereoBind is built on LTX-2.3 and trained on StereoWorld-29K; spatial metrics include ILD-W, SELD-Acc, SMR-Err, AST-Ang, and AST-Cal.Source: §5.1 · pp. 8–9

5.3 Ablation Study.

• On SWBench, full StereoBind achieves the strongest overall performance, and each tested factor contributes to stereo spatial fidelity.The ablations examine RT-RoPE, STE, VMB, and the number of VMB tokens.Source: Table 2 · p. 9, §5.3 · p. 10
• Removing RT-RoPE reduces AST-Ang and AST-Cal, indicating degraded learned spatial representations.The authors attribute the effect to the role of relative source displacement in capturing relative spatial dynamics.Source: §5.3 · p. 10
• Removing STE substantially increases SMR-Err and degrades AST-Ang and AST-Cal, underscoring the importance of absolute source-position information.Source: §5.3 · p. 10
• Removing VMB degrades overall spatial fidelity; performance largely saturates at 64 VMB tokens, while 128 yields marginal gains and higher memory overhead.The final model uses 64 tokens as a spatial-fidelity and computational-cost trade-off.Source: §5.3 · p. 10

• StereoBind extends stereo video-audio generation from semantic and temporal correspondence to dynamic spatial correspondence.Source: §6 · p. 11
• The framework combines VMB for cross-modal motion conditioning, STE for absolute spatial conditioning, and RT-RoPE for relative-motion modeling.Source: §6 · p. 11
• StereoWorld-29K provides training data and StereoWorldBench evaluates dynamic spatial correspondence; experiments report improved spatial consistency with visual source motion while preserving overall audiovisual quality.Future work will extend the approach to richer 3D scenes, multiple sound sources, and more complex source–listener dynamics.Source: §6 · p. 11

A Detailed Architecture of the Spatial Track Encoder

• The Spatial Track Encoder projects frame-wise 2D tracks, encodes temporal dependencies, and interpolates the resulting motion context to the audio-token sequence.The implementation uses 121 temporal positions and a latent dimension of 256.Source: A Detailed Architecture of the Spatial… · p. 15
• Track-Adaptive modulation produces layer-specific signals that let audio queries retrieve motion-aware visual context and control the strength of the spatial update.The adaptive table is added to projected track features for each Transformer block.Source: A Detailed Architecture of the Spatial… · p. 15

B Additional Details of Visual-Audio Spatialization Pipeline

• The VAS pipeline identifies a sound source, verifies candidate visual entities, extracts a motion track, and renders stereo audio using reconstructed scene geometry.Candidate verification evaluates semantic and action–acoustic consistency, temporal synchronization, and physical causal plausibility; the highest-confidence candidate is selected.Source: B Additional Details of Visual-Audio… · p. 15, §B.2 · p. 16, §B.3 · p. 16

B.1 Visual-Audio Source Discovery

• VAS source discovery uses a joint video-audio model to identify the dominant audible event and produce a grounding phrase for visual localization.The grounding phrase is used with GroundingDINO and SAM2 to obtain candidate boxes and masks.Source: §B.1 · p. 15, §B.2 · p. 15

B.3 Sound Source Tracking

• After candidate verification, AllTracker extracts a sparse source track, which is smoothed and used for both stereo-audio construction and motion conditioning.The pipeline combines VGGT-recovered geometry with a median-distance anchor of 1.5 m and two virtual receivers to generate time-varying stereo propagation paths.Source: §B.3 · p. 16, §B.4 · pp. 16–17

C Additional Details of Spatial Data Synthesis Pipeline

• SDS first samples a structured scene configuration and motion track, then turns those constraints into a generation prompt.The object-centric configuration specifies motion type, motion family, source extent, and semantic family; 70% of samples are dynamic, while static-left and static-right each account for 0.15.Source: C Additional Details of Spatial Data… · p. 17, §C.1 · pp. 17–18

C.2 Scene Distribution Bank

• The Scene Distribution Bank increases sampling probability for underrepresented categories and uses the five most recent accepted samples as negative diversity context.It tracks source and scene categories, semantic and motion families, and sound events.Source: §C.2 · p. 18

C.3 Object-Centric Scene Generation

• Object-centric synthesis builds a semantic blueprint from sampled constraints, while human-centric synthesis uses speaker profiles and environments with a speech planner.The object-centric blueprint is converted into an LTX-2.3-compatible prompt; the human-centric branch uses 47 profiles and 41 environments, with spoken content constrained to about 15 English words for a five-second clip.Source: §C.3 · p. 18, §C.4 · p. 18

D Distribution Analysis of StereoWorld-29K

• Dynamic samples are grouped into four motion levels by horizontal displacement, while static samples are classified as left, center, or right.The levels correspond to crossing one, two, three, or four horizontal regions; larger displacement generally produces more pronounced temporal stereo-cue changes.Source: D Distribution Analysis of… · pp. 19–20
• The dataset combines real-world and synthetic sources and covers a wide range of visible sound-producing semantic categories.The source composition and semantic-category distribution are shown in Figure 6.Source: D Distribution Analysis of… · p. 19, Figure 6 · p. 20

E Human Perceptual Evaluation

• Twenty-five volunteers compare outputs from five systems across 15 randomly sampled SWBench groups, rating immersion while wearing headphones.The comparison includes Ovi, MiniMax-H3, LTX-2.5, PrismAudio, and See2Sound; method order is randomized.Source: E Human Perceptual Evaluation · p. 20
• The five-point ratings jointly consider audiovisual correspondence, naturalness of perceived motion, stereo realism, and sense of presence.The scale ranges from very poor to excellent immersion.Source: E Human Perceptual Evaluation · pp. 20–21
• StereoBind achieves the highest mean perceptual score, and the authors report stronger perceived source-audio consistency and a more immersive audiovisual experience.Table 5 reports mean scores for overall immersion, rated on a five-point scale.Source: E Human Perceptual Evaluation · p. 21, Table 5 · p. 21

F Additional Qualitative Results

• Qualitative comparisons report that StereoBind generates spatially coherent stereo audio that follows visible-source motion and better preserves correspondence and temporal spatial transitions.Figures 8 and 9 compare StereoBind with open-source video-audio and video-to-stereo-audio models; some nearly uniform spectrograms correspond to stationary white noise.Source: F Additional Qualitative Results · pp. 21–22, Figure 8 · p. 22, Figure 9 · p. 23

G.1 Signed Inter-channel Level Difference

• SILD tracks signed left–right energy dominance over time, so a left-to-right source movement is expected to correspond to a transition from positive toward negative values.Values near zero indicate approximately balanced left–right energy.Source: §G.1 · pp. 23–24

G.2 Inter-channel Differential Spectrogram

• IDS displays the time–frequency magnitude of the difference between left and right channels, revealing where and when they diverge but not which channel dominates.A nearly duplicated mono signal yields little differential energy; compared methods share a color scale.Source: §G.2 · pp. 24–25

H.1 StereoWorldBench Sample Composition

• StereoWorldBench contains 95 prompt–track samples constructed independently from StereoWorld-29K, including 37 static and 58 dynamic examples.The dynamic set includes 27 left-to-right and 22 right-to-left tracks, plus 9 other non-static tracks.Source: §H.1 · p. 25
• The benchmark covers human, machine or object, and animal sources, with speech, dialogue, and broadcast audio forming its largest acoustic group.Human subjects account for 74 samples (77.89%); speech, dialogue, and broadcast audio account for 47 (49.47%).Source: §H.1 · pp. 25–26

H.2 Spatial-Audio Evaluation Metrics

• The metrics are complementary: they assess stereo energy balance, learned spatial representations, interaural-level cues, and alignment between audio localization and visible sources.The evaluation also specifies 100 ms windows and a 100 ms hop for audio resampling and segmentation.Source: §H.2 · pp. 26–28
• ILD-W compares reference and generated interaural-level-difference distributions across valid reference-active windows; lower distance indicates greater temporal consistency.ILD is computed over 1.7–4.6 kHz, with frequency bins weighted by combined binaural magnitude.Source: §H.2 · p. 26
• SELD-Acc tests whether predicted horizontal sound-source locations overlap visible-source intervals, but its reliability beyond speech and musical-instrument sounds is less established.A higher score indicates stronger spatial consistency; interpret it with the other spatial metrics.Source: §H.2 · p. 27
• SMR-Err compares generated and reference side-to-mid energy ratios to measure preservation of relative stereo energy; lower error indicates more faithful stereo balance.The mid component captures shared channel content, while the side component reflects inter-channel spatial variation.Source: §H.2 · pp. 27–28
• AST-Cal measures stereo-representation improvement over a duplicated-mono baseline by calibrating angular and distance similarities; higher scores indicate stronger preservation of spatial information.AST-Ang separately compares reference and generated direction representations by cosine similarity.Source: §H.2 · pp. 28–29

I Analysis of Visual Motion–Track Direction Inconsistency

• Contradictory numerical tracks and visual motion references degrade both signal-level and learned spatial metrics, but the degradation is moderate rather than a complete stereo collapse.The intervention indicates that stereo differences alone are insufficient; they must evolve consistently with visible-source motion.Source: I Analysis of Visual Motion–Track… · p. 29

J Future Work

• Multi-source stereo generation and integration of dynamic audiovisual correspondence into world models are identified as future directions.The proposed world-model direction includes predicting scene evolution, physical interactions, and viewpoint-dependent acoustic responses.Source: J Future Work · p. 29
• Generalization to unconstrained scenes remains a challenge, including camera motion, occlusion, off-screen sources, rapid dynamics, and reverberant environments.Source: J Future Work · p. 30