PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-agent Auditory Scene Analysis: Improved Localization Speed and Robustness by Multi-beamformed Speech Quality Feedback

Caleb Rascon

arXiv:2610.00538v1eess.AScs.AIcs.MA

TL;DR

Linear auditory-scene pipelines are sensitive to localization errors, while earlier quality-feedback optimization is slow amid variable single-window estimates. This work tests multiple locations with quality estimates from the same window and combines them by weighted averaging, yielding faster, more robust, and more stable correction in real-life acoustic scenarios.

  • Problem

    Earlier quality-feedback mechanisms rely on single estimates that vary across windows, making optimization slow and difficult, while linear ASAs are sensitive to localization errors.

    Source: §1 · pp. 2–3, §2 · p. 7

  • Method

    The quality-estimation agent evaluates a set of locations using one audio window, and the optimization agent updates location through a quality-weighted average.

    Source: §5 · p. 17

  • Results

    Across real-life acoustic scenarios, the proposed mechanism is more robust to larger localization errors, more stable, faster, and less complex than previous mechanisms.

    Source: §5 · p. 18

  • Takeaways & Limitations

    Interaction between the quality-estimation and optimization agents produces a robust optimization process, and the complete ASA remains able to run in real time.

    Source: §5 · p. 18

    The quality-estimation agent responds more slowly, and Squim assumes complete recordings and needs 3.0 s for moderately stable results.

    Source: §5 · p. 18

Abstract

from arXiv · show

A real-time auditory scene analyzer (ASA) aims to carry out the tasks of locating, separating and classifying the sound sources present in a given acoustic environment. Recently, an effort has been made into modelling an ASA as a multi-agent system, with each one of its agents performing one of the aforementioned tasks and communicating their results to the rest of their peer agents. These communication routes are used as feedback loops to fix local errors at a global level, providing robustness while reducing local complexity. An example of the benefits of this approach is the optimization of speech quality by correcting in real-time the estimated location of the speech source of interest. However, their optimization speed has been shown to be considerably slow. One possible reason is that it solely relies on a series of single quality estimations (provided by a reference-free quality estimator model) that vary considerably from one window to the next, which results in a difficult search space to optimize. In this work, a new optimization mechanism is proposed that instead relies on a series of sets of quality estimations over a range of locations, providing a clearer view of the search space, simplifying its optimization. The proposed ASA now has a considerably smaller optimization time, is more accurate, and is more stable when being evaluated in real-life acoustic scenarios to correct higher levels of localization errors, all while being less complex than previous efforts. The only trade-off is that there is an increase in the response time of the quality estimation agent, but the complete ASA is still able to run in real-time. The performance shown in this work again shows the benefits of modelling an ASA as a multi-agent system.

1 Introduction

The proposed multi-agent ASA uses sets of speech-quality estimates across locations to simplify localization correction, improving robustness, stability, speed, and complexity relative to previous mechanisms. It addresses the difficulty of correcting localization errors when single quality estimates vary across audio windows.

1 Introduction

  • Linear ASA pipelines let localization errors affect later tasks, prompting more complex localization techniques that increase computation and overall response time.Source: §1 · p. 2
  • A multi-agent ASA assigns localization, separation, and classification to separate agents that exchange results as feedback, providing local error robustness while reducing global complexity.Source: §1 · p. 2
  • The focused feedback loop uses separation quality to find a corrected source location, connecting the separation agents with localization, quality-estimation, and optimization agents.Source: §1 · p. 2
  • Earlier approaches use single quality estimates and Adam-based optimization, but are slow, variable, moderately complex, and reliable over a relatively small range of localization errors.The approach added memory of the best location and improved robustness, with varying success across acoustic scenarios.Source: §1 · p. 3
  • The proposed mechanism uses sets of quality estimates to correct a wider range of localization errors with faster, more stable, robust results and less complexity than previous mechanisms.The estimates are combined through a simple weighted average.Source: §1 · p. 3

2 Background of Multi-agent Auditory Scene Analysis

Prior systems correct localization by using separated-speech quality as feedback, but their quality estimates vary across windows and their optimization can be slow. The section describes the earlier smoothing and optimization methods and their trade-offs.

2 Background of Multi-agent Auditory Scene Analysis

  • The feedback approach corrects location by optimizing separated-speech quality, because an incorrect direction degrades speech after pre-beamforming.Source: §2 · p. 3
  • Squim estimates signal-to-distortion ratio from 3-second speech windows, and the estimates vary between windows; exponential smoothing with α = 0.9 was found to balance smoothness and responsiveness.Source: §2 · p. 4
  • The [11] Adam-based method updates source direction using the quality gradient, but its objective is only close to the required smoothness and often overshoots the correct location.Source: §2 · pp. 4–5
  • The method stores the best location and quality, then resets after iterations without improvement to restart from that local optimum.Source: §2 · p. 5
  • The mechanism reduces localization errors, improves consistency and successful-run rates over, but does not shorten optimization time and is more complex.Source: §2 · pp. 6–7
  • Squim assumes a complete recording, while window-by-window estimates can fluctuate even when true quality is unchanged, making optimization difficult and slow at the low update factors used.Source: §2 · p. 7

3 Proposed Optimization Mechanism

The proposed mechanism evaluates multiple beamformed locations using the same audio window, then uses quality-weighted averaging to update the estimated location. This provides a clearer, overlapping view of the search space while keeping the ASA within real-time operation.

3 Proposed Optimization Mechanism

  • Each iteration estimates quality at multiple locations within a range, using the same audio window to provide a clearer view of the search space.Source: §3 · p. 7
  • The method uses beamformer output rather than the full enhancement output because the latter produced a flat search space and required more computing resources; PHASE and MVDR beamformers were evaluated.MVDR was also evaluated because it is less prone to phase-induced artifacts in real-life reverberant scenarios.Source: §3 · p. 7
  • Quality estimates weight their corresponding locations in the update, while θrange and θsteps jointly configure the mechanism’s update behavior.Source: §3 · pp. 7, 9
  • Testing more locations increases quality-estimation response time, but Squim’s reported 0.0538–0.0704-second response for 3.0-second windows leaves time for several tests while remaining below the real-time threshold.Source: §3 · p. 8
  • The weighted average acts as simplified interpolation across a location range; overlapping ranges reduce overshooting and guide updates toward the correct location.Source: Figure 3 · p. 9, §3 · p. 9
  • The quality-estimation agent supplies sets of estimates and the optimization agent computes their weighted average, so optimization emerges from their interaction rather than residing in either agent alone.Source: §3 · p. 9

4 Evaluation and Results

The evaluation characterizes optimization settings and compares the recommended configuration across real-life acoustic scenarios. The balanced configuration remains accurate and stable across the tested conditions, while usually optimizing in under 15 seconds.

4.1 Material

  • The evaluation used AIRA recordings from Office A and Department Store, extending tests beyond prior parameter characterization in the nearly noise- and reverberation-free Anechoic Chamber.The proposed mechanism was characterized in Office A and then evaluated with varying speech-source counts in Office A and Department Store.Source: §4.1 · p. 10

4.2 Methodology

  • Repeatable real-time evaluations replayed recordings through JACK in 1024-sample, 48 kHz windows, repeated each 30-second recording four times, and applied an initial artificial localization error.The quality estimator received a 16 kHz stream; its agent stored the latest 3.0 seconds of audio, while the optimizer waited 0.25 seconds before requests.Source: §4.2 · pp. 10–11

4.3 Variables and Metrics

  • The evaluation varied beamformer, quality measure, θrange, and θsteps, using ten runs per combination and RMS, stability, optimization-time, latency, and good-run metrics.RMS and stability used the last three quarters of each run; good runs were defined by RMS below 5°.Source: §4.3 · pp. 11–12

4.4 Quality Type and Beamforming Technique

  • STOI provided the best mean accuracy across tested configurations, and MVDR considerably outperformed PHASE in good-run percentage when using STOI.The evaluation therefore used STOI and MVDR in subsequent tests.Source: Table 1 · p. 12, §4.4 · p. 12, Table 2 · p. 13

4.5 Characterization of Optimization Configurations

  • Most configurations with θrange ≥ 10° and θsteps ≥ 3 had consistently high good-run percentages, while higher values of both parameters correlated with accuracy.Faster optimization instead correlated with lower θrange and higher θsteps, revealing conflicting speed and accuracy tendencies.Source: Figure 4 · p. 13, §4.5 · pp. 13–14, Figure 5 · p. 13
  • The balanced [θrange = 20.0, θsteps = 5] configuration combines medium accuracy and stability with the second-lowest optimization time and one of the lowest quality-estimation latencies.The proposed mechanism reached the correct-location range considerably faster than, with less variation between runs and lower optimization complexity.Source: Table 3 · p. 14, §4.5 · pp. 15–16
  • The characterization selected configurations meeting good-run percentage ≥1.0, RMS below 3.0°, and optimization time under 15.0 s; the [25.0, 7] and [25.0, 5] settings emphasized accuracy/stability and speed, respectively.The [25.0, 7] setting was most accurate and stable but had high quality-estimation latency and the second-highest optimization time; [25.0, 5] was fastest but least accurate and stable.Source: Table 3 · p. 14, §4.5 · pp. 14–15

4.6 Performance in Different Acoustic Settings

  • RMS stayed below 5° across acoustic scenarios, localization errors, and source counts; optimization time stayed below 15.0 s except with four sources and θerror = 25°.Accuracy fell by only ≈1° per additional source, while varying θerror with the same source count did not alter RMS accuracy.Source: Table 4 · p. 16, §4.6 · p. 17

5 Conclusion

The paper replaces single-estimate quality feedback with multiple location-steered estimates and a weighted-average optimizer. In real-life acoustic scenarios, the mechanism is reported to improve speed, stability, robustness to larger localization errors, and complexity, with higher quality-estimation response time as a trade-off.

5 Conclusion

  • Earlier localization-separation feedback used Squim to optimize separation quality, but was reported as slow and relatively complex.The feedback loop corrects localization error by optimizing estimated speech quality.Source: §5 · p. 17
  • The new mechanism estimates quality at several locations per iteration and uses a weighted average to interpolate within a location range.The optimization behavior emerges from interaction between the quality-estimation and optimization agents; the range size and number of estimates can calibrate it.Source: §5 · pp. 17–18
  • The proposed mechanism is faster, more stable, more robust to larger localization errors, and less complex than previous mechanisms across real-life acoustic scenarios.The evaluation included varying noise and reverberation levels and varying numbers of sound sources.Source: §5 · p. 18
  • The quality-estimation agent has considerably higher response time, but the complete ASA still meets real-time operation requirements.The increased response time affects optimization time, while separation latency is not affected by the parallel multi-agent system.Source: §5 · p. 18
  • Squim response time also affects optimization time; it assumes complete recordings and requires substantial data for moderately stable results.The paper reports 3.0 s as a balance between stability and latency and suggests exploring other models or modifying Squim to consider causality.Source: §5 · p. 18
Loading 2610.00538v1…