Source-linked AI summary
Adaptive Evidence Weighting for Audio-Spatiotemporal Fusion
Oscar Ovanger, Levi Harris, Timothy H. Keitt
TL;DR
Fixed heterogeneous predictors are difficult to combine when evidence reliability varies and generative posteriors are unavailable. FINCH uses bounded per-sample log-linear gating of contextual evidence alongside a pretrained audio classifier, retaining an audio-only fallback. Across bioacoustic benchmarks, it improves over audio-only and fixed-weight fusion, with the strongest reported CBI result reaching 3% accuracy for the contextual prior alone versus 0.38% random guessing as a supported context-strength comparison.
Problem
Fixed predictors from different evidence sources are often unavailable for joint retraining, while discriminative outputs alone do not directly provide the true posterior.
Method
FINCH applies bounded per-sample log-linear gating to contextual evidence while keeping the audio and contextual predictors fixed, with ω(x, s) = 0 recovering audio-only classification.
Results
FINCH consistently outperforms audio-only and fixed-weight fusion baselines across evaluated bioacoustic benchmarks and subsets.
Takeaways & Limitations
Selective contextual integration can improve robustness under weak and heterogeneous contextual signals while preserving an explicit audio-only fallback.
Takeaways & Limitations
The study's gating mechanism uses a compact set of uncertainty- and confidence-based summary statistics, leaving alternative features and more expressive gating architectures unexplored.
Abstract
from arXiv · showhide
Many machine learning systems have access to multiple sources of evidence for the same prediction target, yet these sources often differ in reliability and informativeness across inputs. In bioacoustic classification, species identity may be inferred both from the acoustic signal and from spatiotemporal context such as location and season; while Bayesian inference motivates multiplicative evidence combination, in practice we typically only have access to discriminative predictors rather than calibrated generative models. We introduce \textbf{F}usion under \textbf{IN}dependent \textbf{C}onditional \textbf{H}ypotheses (\textbf{FINCH}), an adaptive log-linear evidence fusion framework that integrates a pre-trained audio classifier with a structured spatiotemporal predictor. FINCH learns a per-sample gating function that estimates the reliability of contextual information from uncertainty and informativeness statistics. The resulting fusion family \emph{contains} the audio-only classifier as a special case and explicitly bounds the influence of contextual evidence, yielding a risk-contained hypothesis class with an interpretable audio-only fallback. Across benchmarks, FINCH consistently outperforms fixed-weight fusion and audio-only baselines, improving robustness and error trade-offs even when contextual information is weak in isolation. We achieve state-of-the-art performance on CBI and competitive or improved performance on several subsets of BirdSet using a lightweight, interpretable, evidence-based approach. Code is available: \texttt{\href{https://anonymous.4open.science/r/birdnoise-85CD/README.md}{anonymous-repository}}
1. Introduction
FINCH addresses fusion of fixed discriminative predictors when evidence reliability varies across inputs and generative posteriors are unavailable. It adaptively weights contextual evidence while retaining an interpretable audio-only fallback.
- Motivation: When generative distributions are unavailable, discriminative audio and contextual predictors cannot directly compute the true posterior.Log-linear fusion provides a practical posterior-combination surrogate using available predictor outputs.
- Motivation: Fixed or globally learned fusion weights can fail because evidence-source reliability varies across samples.This variation can cause degraded performance or pathological dominance by one source.
- FINCH: FINCH combines fixed audio and spatiotemporal predictors through an adaptive log-linear framework with per-sample contextual weighting.The framework preserves product-of-experts structure without retraining or recalibrating the base predictors.
- FINCH: Setting ω(x, s) = 0 exactly recovers the audio-only classifier, while bounding ω(x, s) limits contextual influence.The normalized fused posterior is obtained with a softmax over classes.
- Evaluation: Adaptive weighting consistently outperforms fixed-weight fusion and audio-only baselines, especially when contextual information is heterogeneous or weak in isolation.The evaluation combines a state-of-the-art acoustic classifier with a structured spatiotemporal prior for bioacoustic species classification.
2. Related Work
Prior work includes ensemble, log-linear, mixture-of-experts, uncertainty-based, and bioacoustic approaches to combining predictive evidence. FINCH differs by adaptively modulating contextual evidence from fixed independently trained predictors while preserving modularity.
- Classifier Fusion: Classical ensemble methods combine multiple classifiers but commonly rely on shared data assumptions, classifier diversity, error correlation, or tuned weights.These approaches include averaging, voting, Bayesian combination, bagging, boosting, and stacking.
- Classifier Fusion: Logarithmic opinion pools and product-of-experts models provide log-linear or multiplicative mechanisms for combining predictive distributions.Product-of-experts posteriors sharpen when experts agree and diffuse when they disagree.
- Adaptive Fusion: Most prior fusion methods use fixed or globally learned weights, whereas FINCH weights contextual evidence per sample.This targets nonstationary expert reliability while retaining log-linear interpretability.
- Adaptive Fusion: Unlike typical mixture-of-experts systems, FINCH keeps constituent predictors fixed and independently trained and uses gating to modulate contextual strength rather than select experts.Its multiplicative probability-space formulation is motivated by conditional independence of evidence sources given the label.
- Reliability Estimation: Uncertainty and confidence methods commonly identify unreliable predictions or abstain, while FINCH continuously modulates contextual contribution within probabilistic fusion.This distinguishes FINCH from selective classification based on abstention.
- Bioacoustic Context: Bioacoustic systems combine audio with location or season through ecological priors, metadata constraints, feature concatenation, or joint multimodal predictors.Joint models are more expressive but require paired supervision and become necessary when conditional dependence is strong.
- FINCH: FINCH combines a pretrained audio classifier with a structured spatiotemporal prior through bounded per-sample gating while preserving modular pretraining benefits.The approach targets settings where context is informative but heterogeneous.
3. Method
FINCH combines audio and spatiotemporal evidence in log-space, using a bounded, sample-dependent gate to modulate contextual influence while retaining an audio-only fallback.
- FINCH represents audio, spatiotemporal context, and their fused classifier as pθ(y | x), pψ(y | s), and pω(y | x, s).
- The fused log-score adds the audio posterior to a gated contextual log-prior, then applies a softmax over classes.The gate ω(x, s) controls contextual influence per sample.
- The gate uses uncertainty, confidence, and contextual-structure features processed by a two-layer MLP with sigmoid scaling.The resulting weight lies in [0, ωmax].
- A variance regularizer encourages non-constant gating so the model learns genuine per-input adaptivity rather than globally suppressing context.The regularizer is kept small and selected with validation-based model selection.
- FINCH recovers the audio-only classifier when ω(x, s) = 0 and bounds contextual influence through a constrained maximum weight.This preserves an explicit fallback and limits pathological contextual dominance.
- The framework assumes conditional independence as motivation for multiplicative fusion and freezes the audio classifier so prediction changes arise from fusion.
4. Experimental Setup
The experiments evaluate FINCH on CBI and BirdSet using benchmark-specific spatiotemporal predictors, a frozen pretrained audio representation, and staged fusion training.
- Datasets: Experiments use the CBI and BirdSet large-scale bioacoustic benchmarks.CBI contains 264 species with audio, location, and date metadata.
- Spatiotemporal predictors: CBI uses an AdaSTEM prior queried by location and date, whereas BirdSet uses a metadata-only MLP trained on its training split.
- Audio model: The audio branch uses frozen BEATs representations from NatureLM-Audio followed by a linear classification head.The embeddings have dimension 4096 and are mean-pooled over time.
- Training procedure: Training proceeds from audio-only classification to fixed-weight fusion and finally adaptive gating fusion.The adaptive gate is initialized to match the fixed-weight solution before learning sample-dependent weights.
- Training procedure: The staged design separates representation learning from evidence fusion while keeping the audio encoder frozen throughout.
5. Results
FINCH improves benchmark performance by adaptively weighting contextual evidence, while preserving an audio-only fallback when context is weak or unreliable. Across CBI, BirdSet, and test-time examples, the results support selective, input-dependent fusion over fixed weighting.
- CBI: 0.826 test accuracy: FINCH improves CBI performance from 0.806 for audio-only to 0.826, while fixed-weight fusion reaches only 0.808.The spatiotemporal prior scores 0.030 alone, indicating that the gain comes from selective integration rather than context-only prediction.
- CBI: FINCH outperforms both audio-only models and joint audio–context systems under the same CBI linear-probe protocol.The method adds only a lightweight gating network on top of frozen predictors.
- BirdSet: FINCH matches or improves strong audio-only baselines across BirdSet retrieval (AUROC), detection (cmAP), and classification (Top-1 accuracy).Improvements vary across subsets and metrics, while adaptive weighting avoids degrading performance relative to audio-only models despite a simple metadata-only MLP prior.
- Qualitative analysis: In representative test-time cases, FINCH uses different gating weights to amplify reliable context or suppress an acoustically dominant but ecologically implausible class.The examples show recovery of correct predictions when context is highly specific and when both marginal predictors fail in isolation.
6. Discussion
FINCH adaptively weights contextual evidence, improving fusion under weak or heterogeneous signals while retaining an audio-only fallback and bounded contextual influence. Its broader applicability is tempered by approximate independence and practical gating and prior limitations.
- FINCH consistently outperforms standalone predictors and fixed-weight fusion across large-scale bioacoustic benchmarks.
- Per-sample integration, rather than global modality calibration, drives the reported gains.
- The audio-only classifier remains an explicit fallback, while bounded gating suppresses unreliable contextual evidence.
- Adaptive fusion corrects audio errors when context is reliable and avoids pathological contextual domination when it is not.
- The lightweight gating design uses uncertainty and confidence summaries pragmatically, leaving alternative features and more expressive architectures unexplored.
- The spatiotemporal prior achieves roughly 3% accuracy versus approximately 0.38% for random guessing across 264 classes, indicating weak standalone context.
- False positives are reduced but not eliminated because the interpolating prior assigns nonzero probability to sparsely observed regions.
- FINCH applies beyond bioacoustics when predictors provide complementary, approximately independent evidence about a shared target.
Impact Statement
FINCH offers a general adaptive fusion framework for heterogeneous predictors, with potential relevance to ecological monitoring and decision-support systems. Its use in conservation or policy settings still requires human validation because automated predictions are not definitive evidence of species presence.
- FINCH adaptively fuses heterogeneous probabilistic evidence under approximate conditional independence across multiple domains.
- Improved acoustic-context integration may support more accurate large-scale analyses of species distributions and temporal trends.
- Automated predictions should not be treated as definitive evidence of species presence without appropriate human validation in conservation or policy settings.
- Bounded contextual influence and a primary-evidence fallback are designed to mitigate unreliable or biased auxiliary models in real-world decision-support systems.