Supervising Sound Localization by In-the-wild Egomotion

1. Introduction

• The method is designed to address real-world localization limits in prior approaches, including reliance on synthetic data, specific binaural cues, or specific visual object categories.Natural recordings vary in microphone properties and often mix multiple sound sources, making simulation and learning-signal extraction difficult.Source: §1 · p. 1
• Camera motion during periods when a sound source remains approximately stationary provides cross-modal supervision for learning stereo sound localization from unlabeled in-the-wild videos.Multi-view geometry estimates camera pose, and the audio model is trained to make predictions consistent with that motion.Source: §1 · p. 1
• The approach combines weak visual motion supervision with traditional interaural intensity difference cues, without requiring joint training of visual and audio models.It uses off-the-shelf multi-view geometry and is described as handling diverse motions and camera translation.Source: §1 · pp. 1–2
• StereoWalks provides in-the-wild stereo sound, egomotion, and human-provided sound-direction labels, and is presented as the first dataset of this kind to the authors’ knowledge.The contribution summary describes it as a dataset for learning localization from unlabeled audio-visual signals.Source: §1 · p. 2
• The authors report better performance than previous methods on their proposed dataset and performance comparable to other methods on existing real-world benchmarks.The introduction also reports that the method outperforms a method using stronger supervision [13].Source: §1 · p. 2

2. Related Work

• Prior audio-visual spatial learning uses sight-sound correspondence for scene analysis and related tasks, while Chen et al. jointly learn source direction and camera pose using multi-view audio-visual data.The paper instead estimates camera motion with off-the-shelf multi-view geometry and uses it to supervise audio learning from natural video; used simple simulated scenes for training.Source: §2 · p. 2
• Existing spatial-audio datasets include synthetic multichannel data and real recordings, but many are limited in duration or sound type, lab-recorded, or require labor-intensive annotation.StereoWalks is collected from diverse internet videos and uses camera motion as free supervision, without specialized multi-microphone or Aria-glasses hardware.Source: §2 · pp. 2–3
• Other audio-visual learning work studies semantic correspondence, temporal alignment, scene analysis, sound separation, or visual spectrograms; this paper focuses on using visual egomotion to supervise sound localization.Source: §2 · p. 3
• Camera-motion estimation work develops methods for feature matching, visual SLAM, and robust pose estimation; this paper uses estimated camera poses as supervision for audio.Source: §2 · p. 3

3.1. The StereoWalks dataset

• StereoWalks includes 20 hours of filtered YouTube stereo video with an iPhone subset and labeled validation/test angles; two additional datasets use iPhone and in-ear microphones.The additional datasets are Stereo-Fountain and Binaural-Fountain.Source: §3.1 · p. 3

3.2. Learning audio localization from egomotion

• The model predicts azimuth over angle categories from −90° to 90°, assuming sound sources lie on one side.The method focuses on azimuth because elevation and distance are challenging to perceive from audio alone.Source: §3.2 · pp. 3–4
• A pretrained camera-pose model supplies binary rotation-direction and front-back translation labels that constrain compatible angle pairs across two audio segments.The rotation label indicates clockwise versus counterclockwise motion; the translation label indicates forward camera motion.Source: §3.2 · p. 4
• Interaural intensity differences provide complementary supervision by penalizing angle predictions inconsistent with whether sound is on the camera’s left or right.The left-right cue is derived from the loudness difference between stereo channels.Source: §3.2 · pp. 4–5
• The objective combines rotation, translation, and binaural losses, with weights controlling the relative contributions of the visual-motion losses.The constrained losses sum cross-entropy over angle labels allowed by the corresponding visual or binaural cue.Source: §3.2 · p. 5

4.1. Datasets

• StereoWalks includes YouTube videos with camera motion and stereo sound, with YT-Stereo-iPhone identified as a likely-iPhone subset; the study also constructs HM3D-SS 2.0 simulated data with moving sources and egomotion.The simulated audio uses binaural room impulse responses and scene-based train, validation, and test splits.Source: §4.1 · p. 5

4.2. Implementation details

• The angle model uses a ResNet-18 on four-channel spectrograms for 32-class prediction, while camera motion is estimated with SuperGlue and Perspective Fields.For Ours-full, λ1 is 0.9 and λ2 is 1; training runs for 100 epochs on an 80GB A100 GPU.Source: §4.2 · p. 5

4.3. Evaluation

• Evaluation uses angle MAE and 2-way or 8-way accuracy, comparing chance, IID-direct, GTRot, supervised variants, and real-world self-supervised methods; ablations remove individual loss components.The tested variants include rotation, translation, and pseudo-binaural loss combinations.Source: §4.3 · pp. 5–6

4.4. Results

• Ours-full outperforms baselines across in-the-wild datasets, achieves comparable best performance on YT-Stereo-iPhone and Stereo-Fountain, and gives comparable results on In-the-wild audio after fine-tuning.The comparisons are reported in Tables 2 and 3.Source: §4.4 · p. 6, Table 2 · p. 6, Table 3 · p. 6

4.5. Analysis

• With limited simulated data, Ours-full surpasses prior methods in Overlap and Overlap Intermittent, while GTRot leads in One Source; with more simulated data, their performance is comparable.The table defines intermittent sources as silent half the time and overlap as multiple sound sources.Source: §4.5 · p. 7, Table 4 · p. 7
• Egomotion-only supervision matches performance in Rotation&Translation with 0–2 m camera translation, but distant sources make camera shifts less impactful; rotation plus pseudo-binaural cues captures distant localization, and the overlap setting remains comparable to the best results.The analysis evaluates different motion and distance settings, including overlapping sources.Source: §4.5 · p. 8
• Front/back accuracy is higher on Binaural-Fountain than Stereo-Fountain; the authors suggest the headset’s ear or head cues may explain the difference.The comparison uses recordings made beside the same fountain with an iPhone 13 Pro and a Sennheiser AMBEO Smart Headset.Source: §4.5 · p. 8, Table 6 · p. 8

5. Conclusion

• The method and dataset target in-the-wild sound localization, with supervision from visual camera motion and binaural audio cues.Source: §5 · p. 8
• Experiments on in-the-wild and simulated data find that egomotion supervision aids sound localization.Source: §5 · p. 8
• Low-quality stereo audio and overlapping sounds complicate curation, which may overlook brief sound effects in unlabeled monocular videos.Source: §5 · p. 8
• Future work is anticipated to address these limitations with more visual and contextual data.The authors frame this work as a step toward sound localization in challenging real-world conditions and multimodal methods that learn geometry from unlabeled data.Source: §5 · p. 8

A.1. Dataset

• The section provides further information about the dataset collection procedure.Source: §A.1 · p. 12

A.1.1. Internet data collection

• YT-stereo is collected by querying YouTube for walking-tour videos and excluding portrait-mode or mono-audio recordings.Source: §A.1.1 · p. 12
• Stable sources are selected because their angular velocity should be much slower than the camera's; the process selects 20 hours of training video.Source: §A.1.1 · p. 12
• The YT-Stereo-iPhone validation angles range from -180 to 180 degrees without filtering.Source: Figure 5 · p. 12

A.1.2. Dataset Filtering

• YT-Stereo-iPhone training data are unfiltered, while validation and test sets are filtered for stable sources and clear camera motion.The described selection uses 5-second clips every 3 seconds, SuperGlue motion estimates, and camera rotations of at least 10 degrees; a separate stable-source filter uses a 15-degree threshold and samples rotation angles for a relatively uniform distribution.Source: §A.1.2 · p. 12
• The YT-Stereo-iPhone validation set comprises 100 manually selected clips drawn from 1,600 filtered clips; Fig. 5 shows the angle distribution of that validation set.Source: §A.1.2 · p. 12
• IID pseudo-labels indicate left or right according to which side is louder, and filtering discards clips whose camera-rotation direction differs from IID-change direction.Source: §A.1.2 · p. 12
• Fountain recordings use 15 selected positions around the fountain, with approximately 10 minutes recorded at each position; the Binaural-Fountain clips are randomly split into train, test, and validation sets.Binaural-Fountain recordings used an iPhone 13 Pro and a Sennheiser AMBEO Smart Headset; Stereo-Fountain recordings used an iPhone 13 Pro.Source: §A.1.2 · pp. 12–13
• Table 7 compares dataset length, source visibility, motion types, and source types per 5-second clip; Table 8 summarizes classification accuracy for RWAVS and Stereo-Music.Source: Table 7 · p. 13, Table 8 · p. 13
• Stereo-Music uses a stationary iPhone recording a mobile phone playing music, with data randomly split across train, test, and validation.Source: §A.1.2 · p. 13
• RWAVS records indoor and outdoor scenes with a binaural microphone and moving devices, unlike ReplayNVAS, which has a constant environment and viewpoint.RWAVS scene recordings range from 10 to 25 minutes.Source: §A.1.2 · p. 13
• The simulated set contains 50,000 audio-visual pairs from 200,000 viewpoints and approximately 30 hours of synthesized data, split across 81/9/10 scenes.Moving sound is represented by sources at different positions; binaural audio is rendered by convolving binaural RIRs with LibriSpeech samples.Source: §A.1.2 · p. 13

• The questionnaire asks annotators to locate sources in five-second clips and allows an unknown response.Source: §A.1.3 · p. 13
• Direction labels use 90 for left, -90 for right, 0 for in front, and 180/-180 for behind; only distinguishable sources need labeling.Source: §A.1.3 · p. 13
• Annotators also record source categories and movement, and can skip audio that is too low quality.Source: §A.1.3 · p. 14

A.1.4. Additional scenes

• The added data include labeled iPhone recordings around a fountain and music player, Sennheiser binaural recordings around a fountain, and randomly split RWAVS scenes.Source: §A.1.4 · p. 14
• 50% accuracy on the supervised 4-classification RWAVS task and 28% on Stereo-Music show limited success when training on each dataset alone.Source: §A.1.4 · p. 14

A.2.1. Estimate camera motion

• Camera-motion estimates accumulate rotations and translations between frames, including estimates at three- and six-frame intervals, with proportional adjustment for gaps.Source: §A.2.1 · p. 14

A.2.2. Training details

• Training uses AdamW at learning rate 0.0001 with cosine annealing and early stopping; each model takes about two hours, with one run per setting using the same seed.Source: §A.2.2 · pp. 14–15