PaperlayerView exact revision on arXiv

Source-linked AI summary

STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events

Kazuki Shimada, Archontis Politis, Parthasaarathy Sudarsanam, Daniel Krause, Kengo Uchida, Sharath Adavanne, Aapo Hakala, Yuichiro Koyama, Naoya Takahashi, Shusuke Takahashi, Tuomas Virtanen, Yuki Mitsufuji

arXiv:2306.09126v2cs.SDcs.CVcs.MMeess.ASeess.IV

TL;DR

Real-scene SELD datasets lacked video, limiting evaluation of audio-visual SELD. This paper introduces the task and STARSS23, and reports that visual input benefits localization and detection of body-related classes but performs worse for non-body-related classes.

  • Problem

    Real-scene SELD data such as STARSS22 lacked video, while audio-visual localization datasets generally lacked multichannel spatial labels or focused on speech.

    Source: §1 · p. 2

  • Method

    The paper proposes per-frame activity and DOA estimation from aligned multichannel audio and video, and introduces STARSS23 with spatiotemporal sound-event annotations.

    Source: §1 · p. 2

  • Results

    Visual input improves localization and detection for body-related classes but performs worse for non-body-related classes; the reported evaluations demonstrate benefits of visual object positions.

    Source: §4.4 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    STARSS23 provides a real-scene dataset for audio-visual SELD research, with results supporting visual object positions for body-related classes.

    Source: §4.4 · p. 9, §6 · p. 10

    Practical visual features for non-body-related classes and more sophisticated fusion than concatenation remain to be developed.

    Source: §5 · p. 10

Abstract

from arXiv · show

While direction of arrival (DOA) of sound events is generally estimated from multichannel audio data recorded in a microphone array, sound events usually derive from visually perceptible source objects, e.g., sounds of footsteps come from the feet of a walker. This paper proposes an audio-visual sound event localization and detection (SELD) task, which uses multichannel audio and video information to estimate the temporal activation and DOA of target sound events. Audio-visual SELD systems can detect and localize sound events using signals from a microphone array and audio-visual correspondence. We also introduce an audio-visual dataset, Sony-TAu Realistic Spatial Soundscapes 2023 (STARSS23), which consists of multichannel audio data recorded with a microphone array, video data, and spatiotemporal annotation of sound events. Sound scenes in STARSS23 are recorded with instructions, which guide recording participants to ensure adequate activity and occurrences of sound events. STARSS23 also serves human-annotated temporal activation labels and human-confirmed DOA labels, which are based on tracking results of a motion capture system. Our benchmark results demonstrate the benefits of using visual object positions in audio-visual SELD tasks. The data is available at https://zenodo.org/record/7880637.

1 Introduction

Existing SELD work had real-scene audio and DOA labels but lacked video, while audio-visual localization datasets did not provide the spatial sound-event information needed for this task. The paper proposes audio-visual SELD and introduces STARSS23 as a dataset for studying it.

1 Introduction

  • STARSS22 supported audio-only SELD in natural scenes but lacked video, while most other SELD datasets used synthetic audio.STARSS22 included overlapping and moving events with natural temporal and DOA distributions.Source: §1 · p. 2
  • Audio-visual SELD uses aligned multichannel audio and video to estimate each target class’s framewise activity and DOA.The proposed task draws on the visual correspondence between sound events and their source objects.Source: §1 · p. 2, Figure 1 · p. 2
  • Existing audio-visual localization datasets generally lack multichannel spatial labels or address speaker DOA rather than varied sound events.Some multichannel audio-visual datasets have DOA labels, but their evaluation focuses on speech rather than events such as footsteps or musical instruments.Source: §1 · p. 2
  • STARSS23 combines multichannel audio, video, and framewise activity, DOA, and distance annotations across 13 sound-event classes.Its development set includes over seven hours of recordings with 57 participants in 16 rooms, guided by instructions intended to encourage event activity and content diversity.Source: §1 · pp. 2–3

2 Related Work

Prior work developed SELD methods and datasets, but resources for evaluating audio-visual SELD on natural sound events remained limited. Related audio-visual datasets often lacked multichannel spatial labels or focused on speech localization.

2 Related Work

  • SELD methods progressed from separate activity and DOA outputs to ACCDOA and Multi-ACCDOA approaches for handling overlapping events.Multi-ACCDOA uses auxiliary duplicating permutation invariant training to handle overlaps from the same class.Source: §2 · p. 3
  • STARSS22 provides natural-scene audio with temporal and DOA labels, whereas earlier spatial datasets used isolated or limited simultaneous events.STARSS22 does not release video, so it cannot evaluate audio-visual SELD.Source: §2 · p. 3
  • Audio-visual source-localization datasets lack multichannel recordings, and multichannel audio-visual DOA datasets have focused on speakers rather than varied sound events.Other multichannel audio-visual datasets often have no DOA labels because they target self-supervised learning or non-localization tasks.Source: §2 · pp. 3–4

3 STARSS23 Dataset

STARSS23 provides multichannel audio, video, and spatiotemporal annotations for natural sound scenes. Its construction combines guided recordings with manual event annotation and motion-capture positions, and its analysis describes event coverage, polyphony, and DOA distributions.

3.1 Overview

  • STARSS23 extends STARSS22 with aligned video, about 2.5 additional hours in the development set, and source-distance annotations.The dataset supports training and evaluation of audio-visual SELD systems and includes an evaluation set without publicly available annotations.Source: §3.1 · p. 4

3.2 Data Construction

  • Recordings in Tampere and Tokyo used participant-improvised scenes guided by instructions, with 13 target sound-event classes.The selected classes reflect events adequately captured in the scenes and conform to the AudioSet ontology.Source: §3.2 · p. 5
  • The 32-channel recordings are converted to FOA and MIC formats, while manually annotated event activity is paired with motion-capture positions and confirmed against audio-video views.Temporal annotations have 100-millisecond resolution, and clips that fail confirmation are annotated again.Source: §3.2 · pp. 5–6

3.3 Data Analysis

  • The dev-set-train analysis characterizes frame coverage, polyphony, and DOA, finding frequent speech, music, and domestic sounds alongside class-specific spatial patterns.Speech azimuth appears broadly distributed, while footsteps occur at lower azimuths than -10 degrees.Source: §3.3 · p. 6, Table 2 · p. 6, Figure 3 · p. 6

4 Benchmark

The benchmark evaluates an audio-visual SELD system on STARSS23 against matched audio-only systems using joint activity and DOA metrics. Results favor audio-visual input for body-related events, while full 13-class results vary by class and audio format.

4.1 Audio-Visual SELD System

  • The system encodes detected-object boxes along azimuth and elevation, fuses visual and audio embeddings, and predicts multi-ACCDOA vectors whose length represents activity and direction represents DOA.The visual embedding is repeated along the audio time axis before concatenation with the audio embedding.Source: §4.1 · pp. 7–8

4.2 Evaluation Metric

  • Evaluation uses ER20° and F20° for location-aware detection, plus localization error and recall; the latter three metrics are macro-averaged across classes.A detection counts as a true positive when class labels match and angular difference is below 20°.Source: §4.2 · p. 8

4.3 Experimental Setting

  • The experiments compare audio-visual and audio-only systems in FOA and MIC formats, including a five-class body-related evaluation; person boxes are used as visual input.The development training split is used for training and the development test split is held out for validation.Source: §4 · p. 6, §4.3 · pp. 8–9

4.4 Experimental Results

  • FOA systems perform better than MIC systems; within FOA, audio-visual input has slightly worse location-aware F-score but lower localization error and comparable localization recall.MIC shows a similar trend of lower localization error for the audio-visual system.Source: §4.4 · p. 9
  • In FOA, audio-visual input raises average location-aware F-score for five body-related classes but lowers it for non-body-related classes.The authors suggest person bounding boxes contribute to body-related performance and might limit performance on other classes.Source: §4.4 · p. 9
  • The audio-visual system outperforms the audio-only system on all metrics for the five body-related classes.The paper attributes the improvement to visual location information of people.Source: §4.4 · p. 9

5 Discussion on Limitations and Future Work

The discussion identifies data limitations and open modeling challenges for audio-visual SELD, alongside potential uses for STARSS23 beyond its original task.

5 Discussion on Limitations and Future Work

  • Deploying audio-visual SELD may raise privacy concerns if recorded people have not consented to raw visual capture.The authors also note that improved SELD capability could empower surveillance without appropriate regulation.Source: §5 · p. 10
  • STARSS23’s imbalance and long tail make rare categories difficult to learn when the dataset is used for training.The same distribution can make STARSS23 appropriate for evaluation because it reflects real soundscapes.Source: §5 · p. 10
  • STARSS23 is relatively small as training data.The discussion suggests simulation, separate audio-branch training, or unlabeled audio-visual data as possible resources.Source: §5 · p. 10
  • Visual features for non-body-related classes and a model structure beyond concatenation remain open ways to improve audio-visual SELD performance.People’s visual locations improve performance in body-related classes; a cited recent system used decision-level fusion with keypoints such as feet or telephones.Source: §5 · p. 10
  • STARSS23 may support audio-visual sound source localization and cross-modal retrieval, as well as evaluation of large multimodal models extended to multichannel audio.Its spatial annotations and multichannel audio extend beyond stereo.Source: §5 · p. 10

6 Conclusion

The paper introduces audio-visual SELD and STARSS23, evaluates audio-visual systems against audio-only systems, and reports benefits from visual object positions. It also points to further research uses for the dataset.

6 Conclusion

  • The paper broadens SELD to audio-visual tasks and presents STARSS23 with multichannel audio, video, and spatiotemporal annotations from natural sound scenes.Quantitative evaluations compare audio-visual SELD systems with audio-only systems and demonstrate benefits from visual object positions.Source: §6 · p. 10

Appendix

  • The appendixes cover data construction, analysis, experiments, competition, social impact, personal data handling, and Datasheets for Datasets questions.Source: Appendix · p. 14

A Data Construction

STARSS23 recordings use guided but improvisational scenes, with source annotations built from tracking and video, and data supplied in a frame-based format.

A.1 Sound Scene Recording

  • Recording instructions specify duration, participants, props, active classes, and a scene outline, while leaving many scene details to participants.An example prompts three people to talk, play cards, and respond to a ringing phone.Source: §A.1 · p. 14
  • Scenes combine target events, unannotated directional interference, and natural background noise; target classes include diverse sounds and may group subclasses.The paper provides more specific AudioSet-related labels for encountered sounds in some target classes.Source: §A.1 · p. 14

A.2 Annotation

  • Spatial annotations use tracked source and participant markers, with fixed offsets from head tracking used to estimate mouth, feet, and hand positions for body-related events.When mocap tracking misses cannot be interpolated, annotators use 360-degree video, and visually check interpolated annotations.Source: §A.1 · p. 14, §A.2 · pp. 14–15
  • Table 5 relates target classes to specific AudioSet classes, with untabulated target classes mapped one-to-one to similarly named AudioSet classes.Source: Table 5 · p. 15

A.3 Data Format

  • The data format records frame, active class, source index, azimuth, elevation, and distance; frames have 100-millisecond resolution, angles are in degrees, and distance is in centimeters.The source index identifies a source across events, while source index and distance are additional information that can be used during training.Source: §A.3 · p. 15

B Data Analysis

Event duration varies across classes, with several useful contrasts reported in the duration analysis.

B.1 Duration

  • Phone and bell have similar median durations, but phone has a longer box because calls repeat until answered.Source: §B.1 · p. 15
  • Speech durations span a wide range, while laughter events are shorter than speech events.Source: §B.1 · p. 15
  • Domestic sounds, music, musical instruments, and faucet events tend to last longer, while door and knock events are relatively short.Source: §B.1 · p. 15

B.2 DOA

DOA patterns vary with source mobility: human-produced sounds are dispersed, while fixed sources cluster at discrete room positions. The evaluation metrics jointly assess detection and spatial localization, including performance beyond the 20° threshold.

B.2 DOA

  • Human-produced classes have dispersed DOAs, while fixed-position sources such as doors, knocks, and faucets cluster at discrete points.Music and bells resemble fixed sources; domestic sounds, phones, and musical instruments can resemble speech because their sources may be still or moving.Source: §B.2 · pp. 15–16

C.1 Evaluation Metric

  • The location-aware F-score counts detections as true positives only within 20° of the reference, while misses and extraneous predictions contribute false negatives and false positives.Predictions are spatially matched to references using the Hungarian algorithm before applying the threshold.Source: §C.1 · p. 16
  • Class-aware localization error averages angular error among matched true positives, and localization recall measures the fraction of reference events detected.Both metrics are macro-averaged and unthresholded; classes with no true positives receive a 180° localization error.Source: §C.1 · p. 16

C.2 Experimental Setting

The experiments combine multichannel audio features with person detections from video, then select and evaluate trained models using aggregated SELD error. CNN-based visual features had lower training loss but worse validation results than object detection features.

C.2 Experimental Setting

  • Audio input uses seven channels—four amplitude spectrograms and three IPDs—from 1.27-second signals, with 20-ms frames, 10-ms hops, and 257 frequency bins.The model processes 128 frames at inference and shifts the input by 120 frames.Source: §C.2 · p. 16
  • Video features use YOLOX person bounding boxes because only the person class was stably detected in preliminary STARSS23 experiments.Training uses Adam with learning rate 0.001 and weight decay 10^-6; models are checked every 1,000 iterations through 20,000, and the best aggregated SELD-error model is selected.Source: §C.2 · p. 17
  • CNN-based visual features had lower training losses but worse validation results than person-detection features.The authors suggest bounding boxes provide enough information about human positions for body-related classes, while richer CNN features may require more training data to generalize.Source: §C.2 · p. 17
  • The model has 0.8 million parameters, and the reported scores and error bars average five experiments.Training takes around six hours on a single GPU.Source: §C.2 · p. 17

C.3 Experimental Results

Audio-visual SELD performs better on body-related classes for F-score and localization error across the reported formats, although non-body-related F-scores are worse. STARSS23 also supports comparable challenge evaluation, while its limited event diversity constrains coverage of application-specific scenes.

C.3 Experimental Results

  • Audio-visual SELD has higher F-scores for body-related classes and lower localization error for those classes in both FOA and MIC formats, but lower F-scores for non-body-related classes.MIC F-scores show a trend similar to FOA; localization recall shows no significant trend between formats.Source: §C.3 · p. 17

D Competition

  • STARSS23 serves as the DCASE 2023 SELD Task development and evaluation dataset, enabling direct comparison with challenge submissions using evaluation recordings.The paper's audio-visual SELD implementation is the challenge's baseline method for its audio-visual track.Source: D Competition · p. 17

E Social Impact

  • STARSS23 supports audio-only and audio-visual SELD research, but its sound-event diversity is limited and cannot capture conditions in many application-specific scenes.Its recordings from two geographically distant sites also offer opportunities for cross-regional evaluation.Source: E Social Impact · p. 18

F Personal Data Handling

Participants received an explanation of recording and data release, including potential privacy risks, before signing consent. They were instructed to avoid personal information and keep conversations generic.

F Personal Data Handling

  • Before recording, participants were told the study purpose, recording and release procedures, and potential personally identifiable information risks, including raw speech and blurred faces.The explanation used written text and verbal description, with time for questions; participants signed consent after understanding the purpose and risks.Source: F Personal Data Handling · p. 18
  • Participants were instructed not to reveal personal information and to limit conversations to generic topics.These instructions were part of the recording procedures described before consent.Source: F Personal Data Handling · p. 18
  • The dataset was recorded with more than 50 voluntary participants, and the authors state that Institutional Review Board approvals were in place.Consent was obtained after the explanation and question-and-answer period.Source: F Personal Data Handling · p. 18

G Data Sheet

This datasheet documents STARSS23’s composition, collection, processing, uses, and release, alongside information on earlier versions, proposed extensions, and figures illustrating label distributions and SELD results.

G.1 Motivation

  • STARSS23 was created for audio-visual SELD, providing aligned multichannel audio, video, and spatiotemporal annotations to fill a gap in existing datasets.The dataset was created by Creative AI Laboratory at Sony Group and Audio Research Group at Tampere University; Google funded collection and annotation at Tampere University.Source: §G.1 · p. 18

G.2 Composition

  • STARSS23 provides temporal-activation, DOA, and source-distance labels for 13 target sound-event classes.The development set contains about 7 hours 22 minutes across 168 annotated clips; the evaluation set has about 3.5 hours across 79 clips and no annotations. Audio and video are converted to 4-channel 24 kHz and 1920×960 equirectangular formats, respectively, and faces are blurred; 12 development recordings lack video.Source: §G.2 · p. 19

G.3 Collection Process

  • The dataset is collected through improvised participant scenes using a 32-channel microphone array, 360° camera, motion-capture system, and wireless microphones.Recordings took place from September 2021 to April 2022 and November 2022 to February 2023; the team obtained IRB approval, blurred faces, and had participants sign consent forms after explaining risks.Source: §G.3 · pp. 20–21

G.4 Preprocessing/Cleaning/Labeling

  • Audio and video are resized, sound-event activation and DOA are annotated and checked by listening and viewing, and raw recordings are retained for future use.Python is used for audio conversion, RICOH THETA and FFmpeg for video conversion, Audacity and REAPER for temporal-activation annotation, and Motive for tracking results.Source: §G.4 · p. 22

G.5 Uses

  • The paper is the dataset’s first scientific use, and the authors identify potential applications including audio-only SELD, speaker DOA estimation, source localization, cross-modal retrieval, and distance estimation.The authors also provide a repository for the SELD system.Source: §G.5 · p. 22

G.6 Distribution

  • STARSS23 is publicly distributed through Zenodo under the MIT License.Its DOI is 10.5281/zenodo.7709051.Source: §G.6 · p. 23

G.7 Maintenance

  • Sony Group and Tampere University support and maintain the dataset, with updates announced on its website.Changes are announced on the Zenodo record, and updates are synced on the website.Source: §G.7 · p. 23
  • Older dataset versions remain in Zenodo when changes are made.Source: §G.7 · p. 24
  • Proposed dataset extensions or contributions can be discussed with the authors, validated, and released as a new Zenodo version with an announcement.Source: §G.7 · p. 24
  • Figure 7 visualizes the distribution of azimuth and elevation labels, with each scatter-plot point representing a frame.The top and right plots are stacked histograms.Source: Figure 7 · p. 25
  • Figure 8 shows an object detector detecting people but missing a sink slightly right of center.Source: Figure 8 · p. 26
  • Figure 9 compares per-class audio-visual and audio-only SELD performance and marks classes with zero localization recall as N/A.Bars are omitted for those classes because localization-error statistics are less meaningful when there are no true-positive outputs.Source: Figure 9 · p. 27
Loading 2306.09126v2…