Source-linked AI summary
Signal-Independent and Signal-Dependent Neural Ambisonic Matrix Encoding for Arbitrary Arrays with Variable Microphone Counts
Shichao Hu, Zhiheng Jin, Chunyang Xu, Mengyao Zhu
TL;DR
Many neural Ambisonic encoders require fixed microphone counts, limiting their fit to devices with different configurations. This paper proposes shared, masked-attention SI and SD matrix encoders for variable-count arrays. Both outperform Static LS across evaluated conditions, with SD achieving stronger overall reconstruction and binaural-cue metrics than SI.
Problem
Many neural Ambisonic encoders require fixed microphone counts, limiting deployment across devices with different microphone configurations and adaptation to changing available channels.
Source: §1 · p. 1Method
The paper uses shared microphone-wise processing and masked self-attention for variable-count matrix encoding, with SI conditioned on ATFs and SD additionally incorporating microphone signals.
Source: §1 · p. 1, §2.2 · p. 2Results
Both SI and SD improve all five metrics over Static LS, and SD achieves the best group means.
Source: §4.1 · p. 3Takeaways & Limitations
The results support generalization across microphone counts, source counts, and source domains, with SD achieving better overall reconstruction and binaural-cue metrics than SI across evaluated conditions.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Recent neural Ambisonic encoders accommodate diverse array geometries, yet many existing neural encoders require a fixed microphone count because the number of microphone channels is embedded in the network architecture. This requirement limits deployment across devices with different microphone configurations and adaptation to changes in available channels. To address this limitation, we investigate Transformer-based matrix encoding for arbitrary microphone arrays with variable microphone counts. This is achieved through shared microphone-wise processing and masked self-attention that models inter-microphone relationships across variable-size arrays. Within this framework, we consider signal-independent (SI) encoding, which predicts encoding matrices from array transfer functions, and introduce a signal-dependent (SD) extension that additionally incorporates the observed microphone signals. Both models are trained on simulated scenes using LibriSpeech sources and extensively evaluated under changes in source type, unseen microphone counts, and increased source counts beyond those used during training. Both SI and SD outperform conventional least-squares (LS) encoding in aggregate reconstruction performance across the evaluated conditions. SD consistently achieves stronger overall performance than SI. These results demonstrate that the proposed framework enables array-agnostic Ambisonic encoding while retaining generalization across microphone counts and acoustic source conditions.
1. INTRODUCTION
Ambisonic encoders must accommodate diverse microphone layouts and counts, but many neural encoders retain fixed channel counts. This study proposes shared, attention-based SI and SD matrix encoders and evaluates their generalization beyond training conditions.
1. INTRODUCTION
- Different device designs and sensor failures can change microphone layouts and available inputs, motivating a device-independent Ambisonic front end.Ambisonics represents sound fields through spherical-harmonic coefficients, decoupling reproduction from capture geometry.Source: §1 · p. 1
- Classical LS filters accommodate diverse geometries but require recomputation when arrays change and remain fixed during inference, unlike signal-dependent encoding.Parametric signal-dependent encoding can improve bandwidth and spatial resolution, but relies on scene-model assumptions and accurate spatial parameter estimation.Source: §1 · p. 1
- Existing neural encoders for unseen layouts still retain fixed microphone counts, while diffusion approaches can incur inference costs from repeated network evaluations [14] [15].Other approaches support different counts or generate encoders offline, addressing different aspects of geometry adaptation and source-domain generalization.Source: §1 · p. 1
- Both SI and SD improve over Static LS under evaluated unseen conditions, and SD further improves overall FOA reconstruction over SI.The models use shared microphone-wise processing and masked self-attention; evaluation includes unseen microphone counts, denser mixtures, and non-speech sources.Source: §1 · p. 1
2. METHODS
The methods combine an analytical LS encoder with learned residual corrections. SI predicts corrections from array transfer functions, whereas SD also uses observed microphone signals; shared processing and masking support variable microphone counts.
2.1. Baseline Method
- The baseline maps microphone STFTs to total-scene FOA coefficients using a regularized LS encoder derived from array transfer functions and FOA steering responses.The directional grid uses 96 approximately uniformly distributed Fibonacci samples.Source: §2.1 · pp. 1–2
2.2. Signal-independent neural encoding
- SI uses a shared ATF encoder, masked Transformer, and microphone-wise decoder to predict residual coefficients from each microphone’s directional responses.Microphone signals apply the predicted matrix and supervise FOA reconstruction, but are not predictor inputs.Source: §2.2 · p. 2
2.3. Signal-dependent neural encoding
- SD predicts a residual correction to the LS encoding matrix from observed microphone spectra and array transfer functions.Its audio context queries directional ATF features, which are fused with audio features before a shared head predicts four complex residual coefficients per microphone.Source: §2.3 · p. 2
2.4. Variable-count processing and masking
- Shared microphone-wise processing, masked self-attention, and shared coefficient heads allow variable microphone counts without changing network parameters.Microphone count sets token-sequence length rather than feature width; padding masks exclude absent microphones, whose signals and output coefficients are zeroed.Source: §2.4 · p. 2
3. EXPERIMENTAL SETUP
The experiments compare LS, SI, and SD on simulated LibriSpeech scenes, then test generalization across microphone counts, source counts, and source domains. Training uses a shared reconstruction objective, with FOA and binaural measures for evaluation.
3.1. Compared methods
- Analytical LS, ATF-only SI, and signal- and ATF-conditioned SD use the same array ATFs, microphone recordings, and FOA targets; neural models are evaluated without target-domain fine-tuning.Both neural models are trained with LibriSpeech sources.Source: §3.1 · p. 2
3.2. Data and generalization tests
- Tests separately assess seen and unseen microphone counts and source counts on LibriSpeech, then compare LibriSpeech with unseen ESC-50 and white-noise sources.The microphone tests use P = 4, 6 versus P = 5, 7; source-count tests use K = 1, 2 versus K = 5, 7. Cross-domain tests use P = 4, 6 and K = 5, 7.Source: §3.2 · p. 3
3.3. Training settings
- Both models minimize the same multi-resolution complex STFT MAE objective.Preliminary experiments favored it for spectral reconstruction, coherence, and SI-SDR; adding magnitude-based terms reduced magnitude error but degraded other metrics.Source: §3.3 · p. 3
3.4. Evaluation metrics
- FOA reconstruction is evaluated with waveform SI-SDR, complex spectral MSE, and coherence; binaural errors are computed from decoded target FOA references.The binaural reference is not measured binaural ground truth.Source: §3.4 · p. 3
4. RESULTS AND DISCUSSION
Across microphone counts, source counts, and source domains, SI and SD improve on Static LS, with SD generally strongest. Reconstruction becomes harder as source counts rise, and white-noise performance remains limited.
4.1. Microphone-count generalization
- For unseen P={5,7}, SI and SD beat Static LS by 1.34 and 1.75 dB SI-SDR, while SD exceeded SI by 0.41 dB.For seen P={4,6}, SD exceeded SI by 0.38 dB; higher absolute scores for unseen counts do not imply an intrinsic advantage because the groups use different configurations.Source: §4.1 · p. 3, Table 1 · p. 4
- Below approximately 200 Hz, both neural encoders have their largest absolute MSE reductions versus Static LS, and SD further improves on SI.Lower absolute errors at high frequencies may partly reflect lower speech energy rather than better relative reconstruction accuracy.Source: Figure 3 · p. 4, §4.1 · p. 4
4.2. Source-count generalization
- As K rose from {1,2} to {5,7}, SI-SDR and coherence fell for all methods, but SI and SD still beat Static LS in all five measures.SI's SI-SDR gains over Static LS were 1.10 and 0.99 dB for seen and unseen source counts; SD's were 1.47 and 1.26 dB, respectively, while SD had higher SI-SDR in both groups.Source: §4.2 · p. 4
4.3. Cross-source-domain generalization
- Across LibriSpeech, ESC-50, and white noise, SD achieves the best mean SI-SDR gains over Static LS of 1.26, 0.95, and 0.24 dB, respectively; all methods have negative white-noise SI-SDR.Source: §4.3 · p. 4
4.2. Source-count generalization
- SI uses the same linear encoding matrix for a fixed ATF across source counts, whereas SD depends on observed signals and retains an advantage at unseen counts.The comparison does not isolate the contribution of signal dependence alone.Source: §4.2 · p. 4
5. CONCLUSION
The conclusion presents ATF-conditioned SI and SD encoders that support variable microphone counts and generalize across tested counts and source domains. SD achieves better overall reconstruction and binaural-cue metrics than SI, while white-noise reconstruction remains challenging.
5. CONCLUSION
- ATF-conditioned SI and SD encoders use shared processing and masked self-attention to support variable microphone counts.Both operate beyond the training microphone counts without retraining or architectural changes.Source: §5 · p. 4
- Both models generalize across microphone counts, source counts, and source domains, while SD achieves better overall reconstruction and binaural-cue metrics than SI.White-noise reconstruction remains challenging.Source: §5 · p. 4
- Future work will compare additional SD networks and generative methods, evaluate measured arrays and real recordings, and explore binaural-aware training and perceptual evaluation.Source: §5 · p. 4