Source-linked AI summary
Perception-Inspired Bayesian Causal Fusion for Audiovisual Source Localization
Kyung Yun Lee, Sungnyun Kim, Sebastian J. Schlecht, Tae-Hyun Oh, Vesa Välimäki
TL;DR
Multimodal fusion is useful when modalities share a source, but audible sources can be outside a camera’s limited field of view, making unrelated visual cues harmful to localization. The paper uses Bayesian causal inference to gate precision-weighted fusion over visible candidates, operating post-hoc on frozen audio and visual models. CAUSAL improves on-screen DOAE while limiting off-screen degradation, without joint network retraining.
Problem
Fusion can help when modalities share a latent source, but limited visual coverage means audible sources may be invisible and unrelated visual cues can mislead localization.
Source: §1 · p. 1Method
The method infers a common-cause posterior over visual candidates and uses it to gate precision-weighted fusion of frozen audio and visual estimates.
Source: §1 · p. 2, §2.1 · p. 2Results
CAUSAL improves on-screen DOAE by 2.75° while limiting off-screen degradation to +0.26° and reducing eligible-event error by 0.91°.
Source: §3.4 · p. 4Takeaways & Limitations
The post-hoc gate needs no multimodal training, and improved audio or visual models can be substituted directly.
Source: §4 · p. 4Under the common-cause model, audio and visual observations are treated as noisy measurements of the same source direction.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
Multimodal fusion promises more accurate perception but only when the modalities share a common cause. When they do not, the second modality carries no information about the target, and fusing it can only corrupt the estimate. We cast this whether-to-fuse decision as Bayesian causal inference, following the optimal-observer model of human multisensory perception, and implement it as a plug-and-play layer on top of frozen audio and visual models for sound event localization and detection. The model infers a common-cause posterior over visible candidates, then gates precision-weighted fusion accordingly. Fusing unconditionally more than doubles the direction error, whereas the causal gate improves on-screen localization while limiting off-screen degradation, without any joint network retraining.
1. INTRODUCTION
Multimodal fusion helps when modalities carry complementary information about a shared state, but can harm localization when a visible cue has no common cause with the sound. This paper frames whether to fuse as Bayesian causal inference and applies the resulting gate post-hoc to frozen models for sound event localization and detection.
1. INTRODUCTION
- Fusion can reduce localization error when a source is visible, but an unrelated visual entity can pull the estimate away from an audible source outside the camera’s field of view.The paper notes this setting also applies to robots, wearable devices, and AR headsets pairing forward-facing cameras with multi-channel microphones.Source: §1 · p. 1
- Existing joint networks lack an explicit fusion gate, while reliability-weighting methods do not infer whether modalities share a cause.Related approaches require end-to-end training on mixed on/off-screen data and target source separation or localization heatmaps rather than spatial event estimation.Source: §1 · p. 1
- The optimal-observer account first infers whether cues share a common cause, then combines them by precision when they do and segregates them otherwise.Without a shared cause, the second modality carries no information about the target and integration can degrade the estimate.Source: §1 · p. 1
- CAUSAL applies Bayesian causal inference post-hoc to frozen audio and visual models, extending the two-cue azimuth model to multiple visual candidate rays on the sphere.It evaluates common-cause classification and localization on the DCASE 2025 stereo on/off-screen dataset and a constructed four-channel version for full-sphere assessment.Source: §1 · pp. 1–2
2. METHOD
The method casts audiovisual localization as optimal-observer causal inference: infer whether the observations share a source, then estimate direction. It extends this gate to multiple visual candidates and model-averages candidate-conditioned estimates.
2.1. Optimal observer model
- When cues share a cause, the observer integrates them; otherwise, vision carries no information about the audio source, so the optimal estimate weights integration by the common-cause posterior.Full integration and segregation are the p=1 and p=0 cases.Source: §2.1 · p. 2
2.2. Audio and visual observations
- Audio directions and compatible visual detections become spherical observations, with C=0 representing an off-screen, occluded, or undetected source.Bounding-box centers are projected to unit camera rays; detector labels are mapped to compatible audio-event classes before candidate selection.Source: §2.2 · p. 2
2.3. Angular likelihood
- Candidate visual directions receive a von Mises–Fisher likelihood centered on the audio direction under a common cause, versus a uniform-over-solid-angle likelihood without one.Audio and visual localization uncertainty is estimated from training-event great-circle RMS errors.Source: §2.3 · p. 2
2.4. Causal posterior and fusion
- The model allocates the visible-event prior equally among candidates, computes a common-cause posterior, then continuously model-averages candidate estimates on the sphere without a threshold.As the common-cause posterior approaches zero the estimate reduces to audio; near one it approaches precision-weighted combination with the dominant candidate.Source: §2.4 · p. 3
3. EXPERIMENTS AND RESULTS
Experiments use two DCASE2025 dataset configurations to evaluate common-cause classification, localization, and comparison with a jointly trained baseline. CAUSAL discriminates proxy labels and improves localization, including against the stereo AUDIO baseline, while limiting off-screen degradation.
3.1. Setup
- Two DCASE2025 configurations are evaluated: stereo audio with perspective video and a four-channel microphone counterpart for full-sphere evaluation.The microphone directions are rotated into the camera viewing coordinate frame.Source: §3 · p. 3, §3.1 · p. 3
3.3. Common-cause classification
- 0.920 AUROC and 0.847 AP: the continuous posterior discriminates on-screen from off-screen events against the detector-independent on-screen label.Against the common-cause proxy, CAUSAL achieves AUROC 0.924 and AP 0.805; neither proxy directly annotates causal structure.Source: Table 1 · p. 3, §3.3 · p. 4
- 0.093 Brier score and 0.060 ECE: CAUSAL’s posterior is calibrated against the on-screen proxy.Against the common-cause proxy, the corresponding values are 0.085 and 0.048.Source: Table 1 · p. 3, §3.3 · p. 4
3.1. Setup
- No ground-truth common-cause label exists, so the posterior is assessed against an on-screen label and a stricter detection-based proxy.The stricter proxy shares the detector and semantic mapping used to construct the candidate set.Source: §3.1 · p. 3
3.2. Compared methods
- Four strategies vary whether the common-cause posterior gates fusion: AUDIO, FORCED, GATE-20, and CAUSAL.AUDIO and FORCED are full segregation and integration; GATE-20 uses a fixed 20° threshold, while CAUSAL infers the posterior and model-averages.Source: §3.2 · p. 3
3.4. Localization
- 2.75°: CAUSAL improves on-screen DOAE, limits off-screen degradation to +0.26°, and reduces eligible-event error by 0.91°.FORCED fusion roughly doubles eligible-event error by pulling off-screen events toward unrelated detections.Source: Table 2 · p. 4, §3.4 · p. 4
3.5. Comparison with DCASE 2025 Task 3 baseline
- 2.81°: CAUSAL reduces on-screen DOAE relative to AUDIO; AV-BASE’s on-screen DOAE is 19.13°, versus 16.59° for AUDIO and 13.78° for CAUSAL.CAUSAL attains on/off-screen accuracy of 0.76 vs. 0.74 without joint training.Source: Table 3 · p. 4, §3.5 · p. 4
4. CONCLUSION
The conclusion presents Bayesian causal fusion as a post-hoc gate for frozen audio and visual models, improving on-screen localization while limiting off-screen harm without multimodal retraining.
4. CONCLUSION
- Across two datasets, causal fusion improves on-screen DOAE over audio-only; on stereo, it also beats jointly trained audiovisual systems in on-screen localization and on/off-screen classification.The plug-and-play gate requires no multimodal training.Source: §4 · p. 4
- Without the gate, off-screen fusion drastically raises direction error because the visual channel is conditionally independent of the source.The gate uses precision-weighted cue combination to capture on-screen benefit while containing off-screen harm.Source: §4 · p. 4
- The gate relies on an open-vocabulary detector and manual binary class mapping; failures reduce candidates and default to audio-only, forgoing visual benefit.The authors suggest soft compatibility scores from an audio-language model such as CLAP as an extension.Source: §4 · p. 4