PaperlayerView exact revision on arXiv

Source-linked AI summary

AV-QTSS: Class-Query Target Sound Spatialization in Multi-Source 360$^\circ$ Videos

Qingyu Luo, Peng Zhang, Wenwu Wang, Philip J. B. Jackson

arXiv:2610.34401v1eess.AScs.SD

TL;DR

Existing audio-visual spatialization renders the scene as a whole, leaving selected-source output unresolved even when a user wants to control an individual event. AV-QTSS combines a class-conditioned separator, dense visual localizer, and FOA renderer; experiments report improved localization and spatial accuracy, with real-recording transfer after adaptation.

  • Problem

    Existing audio-visual spatialization targets whole-scene soundtracks, while source-selective spatialization must extract the queried sound and bind it to its visual source.

    Source: §1 · p. 1

  • Method

    AV-QTSS uses a 13-class query, monaural mixture, and 360° video to generate the queried source alone in FOA using a query-conditioned separator, dense localizer, and renderer.

    Source: §1 · p. 1

  • Results

    Experiments show improved localization and spatial accuracy over established methods and ablated variants, with real-domain transfer after adaptation.

    Source: §4 · p. 4

  • Takeaways & Limitations

    The demonstrated scope is source-specific FOA rendering for closed-set classes active in the clip, with real-domain transfer following adaptation.

    Source: §4 · p. 4

    The formulation assumes closed-set class queries active in the clip, cannot separate same-class instances, and may confuse semantically close classes.

    Source: §4 · p. 4

Abstract

from arXiv · show

Spatial audio is central to immersive 360$^\circ$ media, yet existing audio-visual spatialization renders the scene as a whole, leaving no control over an individual source. We introduce Audio-Visual Query-conditioned Target Sound Spatialization (AV-QTSS): given a monaural multi-source mixture, a 360$^\circ$ video and a class query from the 13-class DCASE sound-event ontology, it generates the first-order Ambisonics (FOA) signal of the queried source alone. The omnidirectional mixture contains no explicit spatial cues, so direction has to be inferred from the video, where several objects may sound at once. We propose a query-conditioned dense binding localizer that predicts a spherical probability map over the whole scene, coupled to the renderer so both are optimized jointly. Experiments with one to four concurrent sources show improved localization and spatial accuracy relative to established methods and ablated variants, and the localizer transfers to real 360$^\circ$ recordings after adaptation.

1. INTRODUCTION

Existing audio-visual spatialization reconstructs a scene-level soundtrack, while source-selective spatialization must extract a queried sound and bind it to the correct visual source. AV-QTSS generates the queried source alone in FOA from a monaural mixture, 360° video, and a class query.

1. INTRODUCTION

  • Existing methods target scene-level soundtracks, not independently usable spatial outputs for selected sources.Source-selective spatialization must also keep the extracted sound, bound visual source, and rendering direction consistent in multi-source scenes.Source: §1 · p. 1
  • AV-QTSS uses a 13-class DCASE query, monaural mixture, and 360° video to generate the queried source alone in FOA.Its query-conditioned Separator, dense Localizer, and map-conditioned Renderer connect extraction, visual binding, and spatial rendering.Source: §1 · p. 1, Figure 2 · p. 2
  • The paper presents a purpose-built 80-hour corpus and reports improved localization and spatial accuracy, with real-domain transfer after adaptation.The stated contributions include a Query-conditioned Dense Binding Localizer for multi-source scenes.Source: §1 · p. 2

2. METHOD

AV-QTSS processes a monaural mixture, 360° video, and class query to produce the queried source in FOA. Its separator extracts the omnidirectional target channel, while a dense query-conditioned map guides a renderer that is jointly optimized with the localizer.

2. METHOD

  • AV-QTSS separates the queried omnidirectional channel, localizes it on a per-frame spherical map, and renders the directional FOA channels.The Localizer uses five video frames; the Renderer preserves the separated W channel while predicting Y, Z, and X.Source: §2 · p. 2

2.1. Query-Conditioned Separator

  • AudioSep is adopted for class-query separation because the number of active sources is unknown and a fixed-source-count separator would not fit the task.Frozen CLAP class prototypes receive a learned zero-initialized residual to address the fixed query vocabulary and reverberant omnidirectional target channel.Source: §2.1 · p. 2

2.2. Query-Conditioned Dense Binding Localizer

  • The QDB Localizer fuses a query-specific score with the AVS sounding prior to predict the queried source’s probability over the spherical grid.The query score distinguishes the target from other sounding objects, while the prior drives down silent cells.Source: §2.2 · p. 2

2.3. FOA Renderer

  • The Renderer conditions query-specific time-frequency masks on the dense map, and joint optimization lets the FOA loss shape the map.The localization objective includes map cross-entropy, expected-direction cosine loss, and a penalty on regions associated with other active sources.Source: §2.3 · p. 3

3. EXPERIMENT

The experiments pair an 80-h source-supervised synthetic corpus with real STAIRS26 recordings used for localization transfer, then evaluate full rendering and localization. AV-QTSS achieves the strongest overall balance and lowest overall angular error, while adaptation improves localization on STAIRS26.

3.1. Datasets

  • The 80-h AV-QTSS-Syn corpus supplies monaural mixtures and reverberant per-source targets, while STAIRS26’s 7.5 h of real video supports localization only.STAIRS26 lacks source-wise waveform and FOA references.Source: §3.1 · p. 3

3.2. Experimental Setup

  • Training adapts the separator in stages, jointly optimizes the Localizer and Renderer, and updates only the QDB Localizer on STAIRS26.The full system uses separated-target estimates when jointly training the Localizer and Renderer.Source: §3.2 · p. 3
  • Evaluation combines directional-channel reconstruction and DOA errors with MOS-C and MOS-S ratings from 11 participants on five decoded clips per method.MOS-C assesses content correctness and MOS-S perceived spatial accuracy.Source: §3.3 · p. 3

3.3. Results

  • Ours achieves the lowest overall angular error and strongest overall balance across the three groups on AV-QTSS-Syn.The comparison covers separator, renderer, and localizer variants.Source: Table 1 · p. 4, §3.3 · p. 4
  • On synthetic subsets, No-Class beats No-Binding, while class embeddings yield gains that grow with K; the ranking persists after adaptation.STAIRS26 accuracy is not monotonic in K because its subsets differ in scene and class content.Source: §3.3 · p. 4, Table 2 · p. 4
  • Without adaptation, the QDB Localizer stays between 11.1% and 32.8% Acc@30°; on STAIRS26, adaptation at least doubles Acc@30° in every subset, although absolute accuracy remains low at K=4.Source: §3.3 · p. 4
  • With the scene and mixture fixed, extracted content and rendered energy follow the queried source in synthetic and real examples.For the synthetic scene, SI-SDR is 9.6 dB for the water tap and 6.2 dB for the bell.Source: §3.3 · p. 4, Figure 3 · p. 4

4. CONCLUSION

AV-QTSS renders a queried source in FOA from a mono mixture and 360° video, using an 80-h corpus and a QDB Localizer jointly optimized with the Renderer. The paper reports improved localization and spatial accuracy, with transfer to real recordings after adaptation.

4. CONCLUSION

  • AV-QTSS renders a user-specified source in FOA from a mono mixture and 360° video, using an 80-h corpus for per-source supervision and a QDB Localizer coupled to the Renderer for joint optimization.Source: §4 · p. 4
  • Experiments report improved localization and spatial accuracy over established methods and ablated variants, plus real-domain transfer after adaptation.Source: §4 · p. 4
  • The formulation assumes closed-set class queries active in the clip, cannot separate same-class instances, and may confuse semantically close classes; future work targets open-vocabulary queries, higher polyphony, and realistic corpora.Source: §4 · p. 4

Related papers

Loading 2610.34401v1…