Source-linked AI summary
Self-supervised Moving Vehicle Tracking with Stereo Sound
Chuang Gan, Hang Zhao, Peihao Chen, David Cox, Antonio Torralba
TL;DR
The paper addresses vehicle localization in a visual reference frame without relying on visual input at inference time, avoiding labor-intensive audio-box annotation. It trains a stereo-sound student from a vision teacher using unlabeled audio-visual videos, and reports effective localization, including under poor lighting, while noting sensitivity to video-segment length.
Problem
The task is to localize moving vehicles in a visual reference frame from stereo sound and camera metadata without manually annotating audio-to-box correspondences.
Method
A vision teacher and stereo-sound student use synchronized unlabeled videos to transfer vehicle-location knowledge across modalities during training.
Results
The proposed system outperforms audio-only baselines, maintains robust tracking under poor lighting, and can localize multiple vehicles despite reduced performance.
Takeaways & Limitations
Stereo auditory localization can operate without visual input at test time and can augment visual tracking in poor-light conditions.
Takeaways & Limitations
Localization improves with longer video segments, but performance stabilizes for segments longer than 1 second.
Abstract
from arXiv · showhide
Humans are able to localize objects in the environment using both visual and auditory cues, integrating information from multiple modalities into a common reference frame. We introduce a system that can leverage unlabeled audio-visual data to learn to localize objects (moving vehicles) in a visual reference frame, purely using stereo sound at inference time. Since it is labor-intensive to manually annotate the correspondences between audio and object bounding boxes, we achieve this goal by using the co-occurrence of visual and audio streams in unlabeled videos as a form of self-supervision, without resorting to the collection of ground-truth annotations. In particular, we propose a framework that consists of a vision "teacher" network and a stereo-sound "student" network. During training, knowledge embodied in a well-established visual vehicle detection model is transferred to the audio domain using unlabeled videos as a bridge. At test time, the stereo-sound student network can work independently to perform object localization us-ing just stereo audio and camera meta-data, without any visual input. Experimental results on a newly collected Au-ditory Vehicle Tracking dataset verify that our proposed approach outperforms several baseline approaches. We also demonstrate that our cross-modal auditory localization approach can assist in the visual localization of moving vehicles under poor lighting conditions.
1. Introduction
The paper targets vehicle localization in a visual reference frame using stereo sound and camera metadata, learned from unlabeled audio-visual videos. Its cross-modal system transfers visual detection knowledge to an audio student, enabling inference without visual input.
- Problem and motivation: Stereo sound provides spatial cues that can support localizing moving vehicles in a visual reference frame without visual input.The system uses stereo audio and camera metadata to recover vehicle coordinates.
- Potential applications: The approach is intended for lower-cost, lower-power, privacy-preserving traffic monitoring and for augmenting visual tracking in poor lighting.The camera is required only during a brief training phase for the traffic-monitoring use case.
- Self-supervised learning: Self-supervision avoids labor-intensive manual associations between vehicle bounding boxes and corresponding audio.The method exploits naturally synchronized audio and visual streams in unlabeled videos.
- Proposed framework: A vision teacher transfers object-detection knowledge to a stereo-sound student during training.The teacher predicts vehicle boxes, while the student learns to map audio signals to those coordinates.
- Inference: At test time, the student predicts moving-vehicle coordinates from stereo sound and camera metadata alone.The system is designed to operate independently of visual input after training.
2. Related Work
Related work spans sound localization, audio-visual localization, self-supervised learning, and student-teacher knowledge transfer. Unlike prior approaches requiring visual and audio inputs together or transferring only class-level information, this work targets region-level object localization from audio at test time.
- Sound localization: Sound localization includes active techniques such as echolocation and passive methods using microphone arrays and beam-forming.Passive systems estimate source location from timing differences across microphones.
- Audio-visual localization: Audio-visual localization methods identify image pixels associated with objects producing particular sounds.Recent approaches learn visual-sound correlations and apply network localization techniques to images.
- Cross-modal learning: Self-supervised audio-visual learning exploits implicit labels in unlabeled data to transfer knowledge between modalities.The related student-teacher paradigm trains a student network to mimic teacher outputs.
- Distinction from prior work: This work differs from prior cross-modal methods by transferring richer region-level object location information rather than only class-level information.It also seeks localization within a visual reference frame using audio alone at test time, whereas earlier work used both visual and audio inputs.
3. Approach
The approach uses synchronized unlabeled video and audio to transfer vehicle-localization knowledge from a fixed vision teacher to a stereo-audio student. The student learns bounding-box prediction and feature alignment during training, then localizes vehicles from sound and camera metadata alone.
- Synchronized vision and sound in unlabeled video provide self-supervision for learning vehicle bounding-box localization.
- Network Architectures: The system combines a YOLOv2 vision teacher with a stereo-audio student network.
- Network Architectures: Each 1-second segment uses its middle frame for YOLOv2 pseudo-labels, while stereo spectrograms and camera metadata feed the audio student.
- Transfer of Knowledge from Vision to Sound: The audio student predicts bounding-box coordinates, IoU, and class probabilities through an H×W×125 output.
- Transfer of Knowledge from Vision to Sound: Feature alignment uses paired and unpaired examples so student representations approach the teacher’s for paired inputs while remaining discriminative for negatives.
- Transfer of Knowledge from Vision to Sound: At test time, the audio network operates independently, and temporal smoothing links predictions into vehicle tracks.
4. Experiments
The experiments evaluate cross-modal auditory localization on a newly collected dataset recorded across diverse streets and camera configurations. The study also examines robustness under poor lighting and other challenging scenes.
- The study evaluates the proposed method on a newly collected Auditory Vehicle Tracking dataset and compares it with competing baselines.
- Evaluation: The experiments assess whether auditory localization is more robust than visual tracking under poor lighting and examine cross-scene generalization.
- Dataset: The dataset uses a portable smartphone and stereo microphone setup, with a wide-angle lens to increase the camera field of view.
- Dataset: Videos were recorded on 15 streets while varying camera height, pitch, and orientation to capture diverse views.
- Dataset: The recorded conditions include single-car and two-car cases with stereo audio sampled at 48 kHz.
4.2. Experimental Setup
The evaluation uses held-out video segments, manually labeled testing boxes, and Average Precision plus center-distance metrics. Samples are one-second stereo-audio/video segments represented by 187×80 time-frequency features.
- Data Split: The data split contains 3,329 training clips, 415 validation clips, and 423 testing clips.
- Data Split: The split yields 227,810 training samples, 27,779 validation samples, and 28,672 testing samples after one-second segmentation.
- Evaluation Metrics: Testing boxes are labeled with Amazon Mechanical Turk and used only for evaluation, not model training.
- Evaluation Metrics: Performance is reported with AP at IoU 0.5 and 0.75, averaged AP from IoU 0.5 to 0.95, and center distances on x and y coordinates.
- Input Representation: Each sample contains 24 frames and one second of stereo sound, converted into an 187×80 Mel-frequency time-frequency representation.
4.3. Baselines
The cross-modal auditory localization system outperforms audio-only baselines when trained with object detection loss and feature alignment, with further gains from tracking post-processing. Spectrograms and stereo sound are important for localization, while performance remains below the oracle vision system.
- The proposed method outperforms all audio-only baselines when trained with both object detection loss and feature alignment.
- Tracking post-processing increases AP and produces more consistent, smoother tracking.
- The oracle vision-based YOLOv2 achieves 79.54% AP@0.5, while StereoSoundNet retains an approximately 20% performance gap.
- Spectrogram v.s. Raw Waveform: Spectrogram representations clearly outperform raw waveforms for cross-modal auditory localization.
- Mono Sound v.s. Stereo Sound: Mono sound performs significantly worse than stereo sound in AP, indicating that stereo input provides stronger localization supervision.
- Tracking Performances: Tracking post-processing improves MOTA and identity switches, while also reducing fragments, false positives, and false negatives.
4.5. Performances Under the Poor Lighting
Under poor lighting, the daytime-trained StereoSoundNet maintains robust tracking without fine-tuning, whereas the visual tracking system fails in these scenarios. The evaluation uses five nighttime videos with key-frame localization labels.
- The evaluation uses five nighttime videos with object localization labeled on key frames.
- StereoSoundNet trained on daytime data is applied to nighttime scenes without fine-tuning.
- Under poor lighting, cross-modal auditory localization maintains robust tracking performance while the visual tracking system fails.
- Table 4 evaluates auditory vehicle localization under poor lighting using Average Precision, where higher values are better.
4.6. Generalization on Novel Scenes
The auditory object detection system is evaluated on novel scenes, with 10 scenes used for training and 5 disjoint scenes for testing. Camera height and angle metadata improve generalization, although performance remains below that in seen scenarios.
- The evaluation splits videos from 15 scenes into 10 training scenes and 5 disjoint testing scenes.Testing camera shots may differ from those used during training.
- Camera height and angle metadata significantly improve generalization to novel scenes.The metadata provides prior knowledge about the camera position when visual reference frames were captured.
- Novel-scene performance still has considerable gaps compared with performance on seen scenarios.
4.7. Visualization
Visualizations show StereoSoundNet operating across scenes and camera angles using stereo sound and camera metadata, while also revealing both auditory failure cases and situations where it succeeds despite visual tracking failures.
- StereoSoundNet performs robustly across different scenes and camera angles using only stereo sound and camera metadata.
- The system has failure cases involving fast-moving vehicles and noisy sounds such as construction, wind, and sediment.
- StereoSoundNet succeeds when visual localization fails because of occlusion, backlighting, reflection, or poor lighting.These examples support its potential to assist visual object localization under less-than-ideal image quality.
5. Conclusion
The work introduces a stereo-sound auditory vehicle localization system trained through vision–sound knowledge transfer and evaluates it on a new dataset. The approach also shows potential for augmenting visual tracking under poor-light conditions, while future work targets more simultaneous scenes and object types.
- The study introduces an Auditory Vehicle Tracking dataset containing over 3000 video clips.It also provides an automatic quantitative method for evaluating models and results.
- A student-teacher network transfers knowledge from vision-based object detection to the sound modality.
- The auditory vehicle tracking algorithm demonstrates potential to augment visual tracking under poor-light conditions.
- Future work includes extending the approach to more simultaneous scenes and more kinds of objects.