PaperlayerView exact revision on arXiv

Source-linked AI summary

The Audio-Visual BatVision Dataset for Research on Sight and Sound

Amandine Brunetto, Sascha Hornauer, Stella X. Yu, Fabien Moutarde

arXiv:2303.07257v3cs.RO

TL;DR

BatVision addresses the shortage of large-scale real data for robotic echolocation and audio-visual 3D scene understanding. It records robot-carried chirps and binaural echoes alongside synchronized RGB-D images across varied real environments, and shows that audio-only depth prediction and simulation-developed methods work on the dataset. The dataset is nevertheless constrained by real-world noise and fixed emitter–microphone geometry.

  • Problem

    Large-scale real audio-visual data for robotic echolocation and 3D scene understanding is lacking, limiting simulation-to-reality adaptation and robustness to real background sounds.

  • Method

    The authors collect robot-carried chirps and binaural echoes synchronized with RGB-D images while traversing varied real campus spaces.

  • Results

    Audio-only depth prediction achieves good results, and a state-of-the-art method developed for simulated data performs similarly on BatVision without further adjustments beyond spectrogram resolution.

  • Takeaways & Limitations

    BatVision provides large-scale real audio-visual data for robotic echolocation, depth prediction, and broader audio-visual scene-understanding research.

  • Takeaways & Limitations

    Real recordings may contain environmental noise, and the fixed close emitter–microphone arrangement prevents independently varying their positions.

Abstract

from arXiv · show

Vision research showed remarkable success in understanding our world, propelled by datasets of images and videos. Sensor data from radar, LiDAR and cameras supports research in robotics and autonomous driving for at least a decade. However, while visual sensors may fail in some conditions, sound has recently shown potential to complement sensor data. Simulated room impulse responses (RIR) in 3D apartment-models became a benchmark dataset for the community, fostering a range of audiovisual research. In simulation, depth is predictable from sound, by learning bat-like perception with a neural network. Concurrently, the same was achieved in reality by using RGB-D images and echoes of chirping sounds. Biomimicking bat perception is an exciting new direction but needs dedicated datasets to explore the potential. Therefore, we collected the BatVision dataset to provide large-scale echoes in complex real-world scenes to the community. We equipped a robot with a speaker to emit chirps and a binaural microphone to record their echoes. Synchronized RGB-D images from the same perspective provide visual labels of traversed spaces. We sampled modern US office spaces to historic French university grounds, indoor and outdoor with large architectural variety. This dataset will allow research on robot echolocation, general audio-visual tasks and sound phænomena unavailable in simulated data. We show promising results for audio-only depth prediction and show how state-of-the-art work developed for simulated data can also succeed on our dataset. Project page: https://amandinebtto.github.io/Batvision-Dataset/

I. INTRODUCTION

BatVision addresses the lack of large-scale real audio-visual data for robotic echolocation and 3D scene understanding by pairing recorded chirp echoes with synchronized RGB-D imagery in varied environments.

  • Research gap: Large-scale real data is missing for audio-visual 3D scene understanding, limiting simulation-to-reality adaptation and robustness to real-world background sounds.Existing audio-visual recordings are sparse because real-world collection requires complex setups and long sessions.
  • Dataset contribution: The dataset covers typical office scenes alongside corridors and open spaces, establishing real-world settings for audio-visual and echolocation research.BV1 and BV2 provide complementary campus environments with substantial architectural variety.
  • Motivation: Sound can complement cameras and LiDAR under occlusion, reflections, smoke, darkness, and low-light conditions, while supporting bat-like 3D scene understanding.Prior work predicted depth from audible chirp echoes, motivating more data in realistic robotic scenarios.
  • Dataset contribution: BatVision records forward-facing chirps, binaural echoes, and synchronized RGB-D images while traversing complex real campus spaces.The dataset spans varied materials, room shapes, and objects, with data from UC Berkeley and Mines Paris.

II. RELATED WORK

Related datasets provide simulated or narrowly controlled acoustic and audio-visual data, whereas BatVision targets large-scale real audio-visual 3D scene understanding in varied public spaces.

  • Acoustic Room Impulse Response Datasets: Existing real RIR datasets use high-quality measurements but commonly restrict scenes to rectangular rooms, cuboid objects, or simple materials.BatVision instead samples public spaces selected for varied materials, shapes, and architectural properties.
  • Audio-Visual Simulation: Simulation benchmarks such as SoundSpaces enable scalable audio-visual research, but models can overfit simulator characteristics and lack a comparably sized real dataset.Bridging simulation and reality remains a significant challenge for audio-based 3D scene understanding.
  • Real-World Audio-Visual Datasets: Prior real audio-visual datasets support activities, sound events, vehicle detection, or passive distance estimation, but often lack sufficient environmental spatial ground truth for 3D reconstruction.BatVision pairs forward-facing imagery and depth with recorded echoes for audio-visual spatial analysis.
  • Real-World Audio-Visual Datasets: BatVision complements related datasets by collecting active, humanly audible chirp interactions with scene materials in complex environments.Its recordings are smaller than EGO4D but better suited to the paper’s echolocation and scene-understanding tasks.
  • Real-World Audio-Visual Datasets: Recent chirp-based datasets study occupancy or depth but remain smaller or simpler, while BatVision targets varied real scenes for depth prediction and occupancy mapping.The reviewed datasets include 5,000-sample recordings and a 1,478-sample single-room setup, whereas BatVision is designed for broader scene variety.

A. Dataset Overview

BatVision combines robot-carried binaural audio, RGB-D imagery, and depth labels from varied Berkeley and Mines Paris environments, with splits and recording choices designed for realistic spatial prediction.

  • Dataset Overview: BatVision spans varied architectural styles, room shapes, materials, and acoustic impressions across UC Berkeley and Mines Paris sites.The collection includes lecture halls, corridors, offices, and cobblestone paths.
  • Dataset Overview: The dataset records humanly audible 20 Hz–20 kHz chirps, binaural echoes, and synchronized 1280x720 RGB-D images from a forward-facing robot perspective.Each instance contains one synchronized chirp and RGB-D image; audio is recorded at 44.1 kHz with 24 bits.
  • Dataset Overview: BV2 has a more long-tailed instance-depth distribution and more complex pixel-wise depth structure than BV1.The distributions reflect the broader scene variety in BV2.
  • Dataset Overview: BV1 contains 52,220 instances collected across two floors of a Berkeley institute, while BV2 contains 3,120 instances from Ecole des Mines Paris with large visual and acoustic variety.BV2 provides monocular RGB images and depth from an Intel RealSense D455 camera and records echoes with a binaural 3Dio Free Space microphone.
  • Dataset Overview: BV1 audio is truncated to 72.5 ms for echoes from up to 12 m, whereas BV2 retains 0.45 s recordings to capture reflections from objects up to 75 m.The different windows reflect the smaller spaces in BV1 and larger spaces in BV2.
  • Dataset Overview: BV2 uses time-based trajectory splits of 1,911 training, 625 validation, and 584 test instances to avoid neighboring poses across sets.This makes the split harder than random assignment, though easier than separating complete rooms as in BV1.
  • Dataset Overview: Performance on outdoor, carpeted, or stone-corridor features may require domain adaptation when those features are absent from training data.The authors observed outdoor reconstruction performance was acceptable but sub-par.

B. Limitations

Real-world recording introduces environmental noise, hardware and format variation, and motion-related data constraints that bound how BatVision data should be used.

  • B. Limitations: Real-room recordings may contain noise from nearby streets and busy academic institutions, requiring models to learn robustness to these noise profiles.This differs from simulated data, where such recording-context noise is absent.
  • B. Limitations: Because the robot carries the emitter and microphone together, the dataset changes echoes rather than independently varying emitter and microphone positions as some RIR methods require.This constrains direct comparison with approaches needing independently positioned recording hardware.
  • B. Limitations: Motion filtering leaves few instances from the same pose, although retained examples support noise-model learning and can be removed using optical-flow thresholding.Users can trade off noise-model data against stricter pose filtering.
  • B. Limitations: Different hardware for BV1 and BV2 produces different formats, while shorter BV1 audio, purple image discoloration, and incomplete speaker frequency coverage add modality-specific artifacts.The camera issue affects some bright Mines Paris images, and the consumer-grade speaker cannot produce the full frequency spectrum.

IV. DEPTH PREDICTION ON BATVISION DATA

The authors evaluate audio-only and audiovisual depth prediction on BatVision, finding that simulated-data methods transfer to real recordings while preserving broad scene geometry but struggling with fine structures.

  • Beyond Image to Depth: The Beyond Image to Depth predictions recover visible layouts and obstacles, including free space between desks from audio alone, but fine structures such as cables remain difficult.The evaluation covers BatVision V1 and V2 test sets in meters.
  • Limitations: Fine-structure prediction may be limited by coarser real-world depth ground truth, whose stereo measurement accuracy depends on object distance.This limitation affects depth labels from both ZED and RealSense cameras.
  • Beyond Image to Depth: Comparable results show that Beyond Image to Depth [19] transfers from simulation to BatVision with the same hyperparameters.The method requires only spectrogram-resolution changes when switching to real data.
  • Results: The study compares BatVision and simulated Replica and Matterport data using RMSE, reporting similar V1 and Matterport difficulty and a slight audio-only U-Net advantage over [19].Table II reports results for audiovisual and audio-only settings.
  • U-Net Baseline: An audio-only U-Net predicts free space, obstacles, and general room layout on both BatVision datasets without additional feature or GAN enhancements.The baseline uses spectrograms as input and is trained separately on BV1 and BV2.
  • U-Net Baseline: The U-Net reconstructs chairs, corridors, and obstacles across varied materials, while outdoor predictions systematically underestimate depth.The outdoor bias is attributed to the predominance of indoor data, and finer structures are also lost on BV1.

V. CONCLUSION

The BatVision dataset provides large-scale real audio-visual data for studying sound-based scene understanding. An audio-only baseline and a state-of-the-art method trained on the dataset achieve good depth-prediction results.

  • Conclusion: BatVision supports research on audio-visual scene understanding by providing large-scale real data for improving task performance and discovering new uses of sound.The dataset is intended to support research on perceiving environments acoustically, like a bat.
  • Conclusion: The paper presents an audio-only depth-prediction baseline and obtains good results when training a state-of-the-art approach on BatVision.These findings support depth prediction from sound alone or alongside vision.
Loading 2303.07257v3…