Source-linked AI summary
Robust Localization and Tracking of Simultaneous Moving Sound Sources Using Beamforming and Particle Filtering
Jean-Marc Valin, François Michaud, Jean Rouat
TL;DR
The paper tackles robust localization and tracking of simultaneous moving sound sources for mobile robots. It combines an eight-microphone frequency-domain steered beamformer with particle filtering, demonstrating real-time tracking at distances up to seven meters. The system also supports robot motion control using sound direction.
Problem
Mobile robots need to localize and track simultaneous moving sound sources in real-life environments.
Method
The system combines an eight-microphone frequency-domain steered beamformer with probabilistic post-processing and particle-filter tracking.
Results
The system localizes and tracks simultaneous moving sound sources in noise and reverberation at distances up to seven meters.
Takeaways & Limitations
The system provides acoustic information for robot interaction and can support sound separation and speech recognition.
Takeaways & Limitations
The system cannot detect pure tones, and its delay model assumes no diffraction.
Abstract
from arXiv · showhide
Mobile robots in real-life settings would benefit from being able to localize and track sound sources. Such a capability can help localizing a person or an interesting event in the environment, and also provides enhanced processing for other capabilities such as speech recognition. To give this capability to a robot, the challenge is not only to localize simultaneous sound sources, but to track them over time. In this paper we propose a robust sound source localization and tracking method using an array of eight microphones. The method is based on a frequency-domain implementation of a steered beamformer along with a particle filter-based tracking algorithm. Results show that a mobile robot can localize and track in real-time multiple moving sources of different types over a range of 7 meters. These new capabilities allow a mobile robot to interact using more natural means with people in real life settings.
1 Introduction
Sound localization gives robots auditory information that complements other sensors, but reliable three-dimensional localization is difficult with only two microphones. The paper addresses this with an eight-microphone, robust beamforming and probabilistic tracking approach for simultaneous moving sources.
- Sound localization determines sound-source coordinates relative to a point in space.
- Auditory sensing can complement vision because it is omnidirectional, works in darkness, and is not limited by walls.
- Two microphones provide limited localization, including azimuth-only estimates, front-back ambiguity, and reduced precision along the microphone axis.
- Arrays with more than two microphones support azimuth and elevation localization while reducing uncertainty from noise, reverberation, and imperfect microphones.
- The proposed system combines an all-directions frequency-domain beamformer with probability-based post-processing to localize multiple moving sources robustly.The method performs localization in one step rather than estimating TDOAs before position conversion.
2 Related work
Earlier artificial-audition work largely focused on localization with two microphones, while tracking methods generally assumed a single source. This paper extends particle filtering to multiple sound sources by addressing source-observation assignment.
- Artificial-audition systems for mobile robots can localize, separate, and process sound sources for information such as speech recognition.
- Most prior mobile-robot localization work used two microphones, limiting elevation estimation and often leaving front-back ambiguity unresolved.
- Eight-microphone approaches improved robustness by combining time-delay and direction-of-arrival estimation in a single step, but did not track simultaneous sources.
- Kalman- and particle-filter methods generally assumed one source because assigning localization observations to individual sources was difficult.
- The paper applies particle filtering to multiple sound-source tracking by solving the source-observation assignment problem.
3 System Overview
The localization system combines a microphone array, a steered-beamformer localization stage, and a particle-filter tracker. The array uses up to eight omnidirectional microphones whose relative positions must be known.
- The system consists of a microphone array, a memoryless steered-beamformer localizer, and a particle-filter tracker.
- The array contains up to eight omnidirectional microphones mounted on the robot.
- Microphone placement is flexible, provided the relative microphone positions are known to approximately 0.5 cm accuracy.
- The beamformer steers across all possible directions to maximize output energy and provide the initial localization.
4 Localization Using a Steered Beamformer
The system searches a spherical direction grid with a frequency-domain steered beamformer, refining beamformer outputs through spectral weighting and probabilistic tracking. It handles multiple candidate sources by iterative localization and particle-based state estimation.
- Delay-And-Sum Beamformer: The delay-and-sum beamformer searches directions by maximizing output energy after aligning microphone signals according to their arrival delays.For a single source, correctly aligned delays make the signals add constructively and maximize energy.
- Spectral Weighting: Whitening sharpens cross-correlation peaks, but equal frequency-bin weighting can reduce noise robustness and hinder narrow-band voice detection.The method therefore introduces an SNR-based weighting mask and a reverberation-related weighting modification.
- Direction Search on a Spherical Grid: A uniform spherical grid contains 2,562 points, with each associated region covering about 2.5° around its center.The grid is created by recursively subdividing an initial icosahedral grid into smaller triangles.
- Direction Search on a Spherical Grid: Under a far-field assumption, the direction search uses precomputed microphone-pair TDOAs and evaluates beamformer energy across the spherical grid.The assumption is exact only for an open array, although experiments show the approximation works with obstacles inside a closed array.
- Multiple-Source Localization and Tracking: Multiple-source localization repeats the search after removing each located source’s contribution, while always seeking four sources and using particle filtering to handle false detections.The particle-based tracker predicts source states, associates beamformer peaks probabilistically, updates particle weights, estimates locations, and resamples when needed.
- Direction Refining: Optional three-dimensional local refinement improves near-field direction accuracy, but the estimated distance is too unreliable to provide a good distance estimate.The refinement evaluates five distances ranging from 50 cm to 5 m.
5 Particle-Based Tracking
The system combines instantaneous frequency-domain beamformer observations with particle filtering to track multiple sound sources over time. Probabilistic post-processing models source dynamics, assigns observations, suppresses false detections, and produces smoothed trajectories.
- Particle-filter tracking: The particle state contains three position and three derivative dimensions, while unit-sphere and tangential-speed constraints leave four degrees of freedom.The sampling importance resampling algorithm operates on these constrained source-direction states.
- Particle-filter tracking: Particle filtering integrates steered-beamformer measurements over time to track an arbitrary, non-constant number of sound sources.Each source is represented by weighted particles, and its estimated location is the weighted mean of particle positions.
- Source dynamics: The excitation-damping predictor models stationary, constant-velocity, and accelerated sources using two parameters per source dynamic.The three parameterizations are (α = 2, β = 0.04), (α = 0.05, β = 0.2), and (α = 0.5, β = 0.2), respectively.
- Observation assignment: Each potential source is classified as a false detection, an existing tracked source, or a new source, under one-to-one observation-to-source assignments.Assignment probabilities combine observation likelihoods with prior probabilities for new sources and false detections.
6 Results
The evaluation uses an eight-microphone array mounted on the Spartacus robot in two configurations and tests it in rooms with different reverberation conditions. The environments differ in reverberation decay settings.
- Robot and array configurations: Eight omni-directional microphones are mounted on the Spartacus robot in two array configurations, C1 and C2.C1 is an open array using microphones arranged on the summits of a 16 cm cube.
- Evaluation environments: Experiments use a medium-size room with 350 ms reverberation time and a hall with 1.0 s reverberation time.The environments are labeled E1 and E2, respectively.
- Evaluation environments: The reverberation decay γ is set to 0.65 in E1 and 0.85 in E2, while other parameters remain unchanged across tasks, configurations, and environments.
6.1 Characterization
The system was characterized for detection reliability and localization accuracy across microphone configurations, distances, sound types, and source heights. It achieved near-perfect reliability with C1 at seven meters, while C2 performance depended on sound type and remained human-comparable in accuracy.
- Detection reliability: Detection reliability counted localizations within 10° and separately tracked reflections and other errors.The evaluation used sounds at distances from 1 m to 7 m, with measurements normalized by sounds played.
- Experimental setup: The evaluation used hand claps, a spoken sentence, and 100 ms white-noise bursts at three source heights.Sources were tested at heights of 0.1 m, 1 m, and 1.4 m.
- Detection reliability: C1 achieved near-perfect detection reliability even at 7 m, whereas C2 reliability depended on sound type.Only hand claps could not be reliably detected beyond 1 m with C2.
- Localization accuracy: Localization accuracy was comparable to or better than human accuracy, whose reported range was 2–4° under similar conditions.Accuracy was evaluated at 1 m and 3 m using separate azimuth and elevation root mean square errors.
6.2 Source Tracking
The particle-filter system tracked multiple moving sources with stationary or moving robots, including sources whose trajectories intersected. Tracking degraded gracefully as microphones were removed, but four microphones could not reliably track more than two sources.
- 6.2.1 Moving Sources: The estimated trajectories of four moving speakers remained consistent in both environments, with only one brief false detection in E1.The result used delayed estimation of 500 ms.
- 6.2.2 Moving Robot: With a moving robot and two talking people, estimated source trajectories stayed consistent in both environments, with one brief false detection in E1.The robot passed between the speakers and then made a half-turn left.
- 6.2.3 Sources with Intersecting Trajectories: The particle filter kept tracking two sources as they intersected, despite the beamformer typically seeing only one source when they were close.The prediction step imposed inertia on source trajectories.
- 6.2.4 Number of Microphones: Tracking degraded gracefully from eight to four microphones, but four microphones could not reliably track more than two sources.Seven microphones made little difference compared with the eight-microphone baseline.
6.3 Localization and Tracking for Robot Control
The robot was controlled in real time using the direction of a detected speaker while moving between environments. Tracking ran at 25 Hz, although wall reflections in corridors sometimes produced false sources.
- Robot control: The robot followed the longest-present speaking source, moving forward when it was within 10° ahead and turning toward it otherwise.The controller used only the source direction.
- Robot control: The system controlled the robot between E1 and E2 through corridors and an elevator while a person spoke from 1–3 m away.The test used normal speaking intensity.
- Robot control: Tracking data were provided at 25 Hz with no additional estimator delay, but corridor reflections were sometimes mistaken for real sources.Considering only the oldest source reduced problems from reflections and noise sources.
7 Conclusion
The paper concludes that an eight-microphone system can localize and track simultaneous moving sound sources in noise and reverberation up to seven meters. It also demonstrates real-time robot control from sound direction and identifies speech-recognition applications.
- Conclusion: An eight-microphone system localized and tracked simultaneous moving sound sources amid noise and reverberation at distances up to 7 m.The approach combined a frequency-domain steered beamformer with particle-filter tracking for multiple sources.
- Conclusion: The robot was controlled in real time using only sound direction, while multiple-source localization could support sound separation and speech recognition.The paper frames these capabilities as useful information from the robot’s acoustic environment.