PaperlayerView exact revision on arXiv

Source-linked AI summary

EigeNet: Geometry-Informed Multi-Modal Learning for Few-shot Novel View RIR Prediction

Chong Jing, Zitong Lan, Junan Zhang, Zhizheng Wu

arXiv:2605.28101v1cs.SDcs.AIcs.MM

TL;DR

Few-shot novel-view RIR prediction must infer spatially varying acoustics from sparse observations while handling multi-view spatial-temporal structure and geometry-acoustic relationships. EIGENET combines CVAT with geometry-informed modulation and auxiliary power-spectrum supervision, achieving state-of-the-art performance across simulated and real-world datasets and sim-to-real generalization.

  • Problem

    Existing methods struggle with high-fidelity acoustic-field estimation because they inadequately capture multi-view spatial-temporal relationships and explicitly model geometry-acoustic correlations.

  • Method

    EIGENET combines a Cross-view Alternate-attention Transformer with geometry-informed modulation and auxiliary power-spectrum supervision for few-shot novel-view RIR prediction.

  • Results

    EIGENET achieves state-of-the-art performance across simulated and real-world datasets, with reported robustness under sparse references and consistent gains across network architectures.

  • Takeaways & Limitations

    The experiments support combining data-driven modeling with geometry-informed physical priors and multi-task supervision for few-shot novel-view RIR prediction.

Abstract

from arXiv · show

Predicting spatially varying Room Impulse Response (RIR) from sparse observations is a critical but highly challenging inverse problem for immersive spatial audio rendering. In this work, we present EIGENET, a geometry-informed multi-modal framework for few-shot novel view RIR prediction. At its core is a Cross-view Alternate-attention Transformer that iteratively refines local intra-view acoustic structures and global cross-view spatial relationships. We empirically demonstrate that this architecture is capable of making full use of the multi-view multi-modal context while performing spatial-temporal reasoning for RIR prediction. Inspired by acoustic ray tracing, we design a geometry-informed modulation block to formulate the connection between geometric features and RIR power spectrum. In the mean time, an auxiliary loss is introduced to transform the single-target waveform prediction into a multi-task learning framework. Through ablation studies, we demonstrate that this design yields consistent performance gains regardless of the underlying backbone, thereby confirming its foundational utility and architecture-agnostic generalizability for RIR prediction task. Evaluated on both simulated and real-world benchmarks, EIGENET achieves both state-of-the-art performance in few-shot novel view RIR prediction and sim-to-real generalization. Codes and checkpoints are available on https://github.com/FEAfeatherTHER/EigeNet.

I. INTRODUCTIO

EIGENET addresses few-shot novel-view RIR prediction by combining alternate attention for multi-view spatial-temporal reasoning with geometry-informed acoustic modulation and auxiliary power-spectrum supervision.

  • Existing methods struggle to capture both temporal structure within individual RIRs and spatial relationships among sparse multi-view observations.
  • Black-box approaches often fail to explicitly model correlations between room geometry and acoustic patterns, limiting physical interpretability in complex configurations.
  • CVAT alternates intra-view and cross-view attention to model local temporal structure and global spatial relationships in multi-view multi-modal context.
  • The geometry-informed modulation block uses geometric features to explicitly modulate acoustic representations, inspired by the relationship between geometry and RIR power spectrum.
  • The auxiliary power-spectrum loss converts single-target RIR prediction into multi-task learning and improves cross-room generalizability while supporting physically plausible synthesized RIRs.
  • Experiments report consistent gains across network architectures and state-of-the-art performance on AcousticRooms and Hearing-Anything-Anywhere, including robustness with sparse references.

II. RELATED WORK

The paper situates EIGENET among multimodal, differentiable-rendering, and alternate-attention approaches, then outlines its geometry encoding, modulation, and CVAT-based prediction pipeline.

  • Few-shot novel-view RIR methods use multimodal inputs, weighted reference-RIR combinations, or generative modeling to predict acoustic responses in unseen views.
  • Differentiable-rendering methods combine physical acoustic knowledge with neural networks but may depend on approximations that do not hold universally.
  • Alternate-attention alternates frame-wise and global-wise self-attention, making it suited to multi-view context and previously evaluated for novel-view image prediction.
  • EIGENET receives source, receiver, reference-RIR, and panorama-depth information to estimate a target RIR at a novel source position.
  • Reference and target geometric-acoustic tokens are concatenated and processed jointly by CVAT to predict the target RIR.
  • A geometric encoder produces geometric tokens, while a modulation block conditions target acoustic tokens on geometry and predicts multi-octave power spectrum.

C. Encoders

The encoders represent room geometry and reference or target RIR information as compatible tokens, while geometry modulates target acoustic tokens and supports multi-octave spectrum prediction.

  • Geometry encoder: Depth maps and source–receiver coordinates are projected into 3D, converted into two propagation-related tensors, and patchified into geometry tokens.The tensors are formed by subtracting source and receiver coordinates from 3D pixel coordinates.
  • Acoustic encoder: Reference RIRs are encoded with pretrained 16 kHz DAC continuous latent representations before vector quantization.DAC was selected after evaluating its reconstruction quality on the AcousticRooms dataset.
  • Acoustic encoder: Target-view time sequences are projected with an MLP from sinusoidal positional encoding and then modulated using geometry-informed features.The resulting target representation T0 is converted into target acoustic tokens by the modulation block.
  • Geometry-informed modulation: The modulation block uses a DiT layer with adaptive normalization to combine target representations with geometry tokens and produce target acoustic tokens.This design operationalizes the relationship between room geometry and the RIR power spectrum.
  • Geometry-informed modulation: The auxiliary spectrum branch regresses a 7-band multi-octave spectrum spanning 63 Hz to 4000 Hz instead of the original STFT spectrum.The bands are 63, 125, 250, 500, 1 kHz, 2 kHz, and 4 kHz.
  • Geometry-informed modulation: A combined MRSTFT and EDC loss is used to prevent temporal collapse during spectrum prediction.The spectrum loss is defined as LMRSTFT(Ŝ0, S0) + LEDC(Ŝ0, S0).

1) Token Organization:

Each view is organized as a sequence containing geometry and acoustic tokens, and the transformer alternates local within-view processing with global cross-view exchange.

  • Token Organization: Geometry tokens are prepended to their corresponding acoustic tokens for every reference and target view.Each view token has shape Vi ∈ R^(1+n)×f.
  • Token Organization: The unified sequence preserves both geometric information and the temporal structure of acoustic tokens across views.Reference views are indexed by i ∈ {1, ..., N}, while i = 0 denotes the target view.
  • Token Organization: Local attention operates independently within each view, allowing acoustic tokens to interact with their corresponding geometry token.This restriction preserves within-view temporal structure.
  • Token Organization: The design alternates local and global attention to refine local acoustic structure while integrating spatial relationships across views.This creates complementary intra-view and cross-view dependencies for target RIR prediction.

4) Global Attention (Cross-view):

Global attention exchanges information across the unified multi-view sequence, enabling reference acoustic cues to inform target prediction while later features are projected and decoded into waveforms.

  • Global Attention (Cross-view): Global attention operates over all view tokens, enabling information exchange across reference and target views.The global operation complements local within-view attention by exposing tokens to cross-view context.
  • Global Attention (Cross-view): Cross-view exchange allows the model to aggregate acoustic cues from reference views when inferring the target view.The target prediction therefore uses information distributed across the available viewpoints.
  • Global Attention (Cross-view): Stacked Alternate-attention blocks iteratively refine intra-view representations and integrate cross-view spatial information.Each block applies a local attention layer followed by a global attention layer.
  • Global Attention (Cross-view): Final local and global features are concatenated and projected into the hidden space of the DAC latent representation.The linear projection maps the combined features back to the DAC latent dimensionality.
  • Global Attention (Cross-view): The predicted latent representation is passed through the frozen DAC decoder to reconstruct the target waveform.The prediction head supervises the RIR signal in the waveform domain and targets fine-grained temporal and spectral characteristics.

B. Metrics.

Evaluation uses three acoustic RIR metrics and compares EigeNet with sampling, interpolation, nearest-neighbor, and learning-based baselines across simulated and real settings.

  • Metrics: The evaluation reports EDT, C50, and T60 as common measures of RIR prediction quality.EDT measures initial 5 dB decay, C50 compares early and late energy, and T60 measures 60 dB cumulative attenuation after an initial 5 dB drop.
  • Metrics: T60 is approximated by multiplying estimated T20 by 3 for numerical stability.Predicted and ground-truth RIR T60 values are compared under this procedure.
  • Baselines: The baselines include Random Across Rooms, Random Same Room, Linear Interp., and Nearest Neighbor (KNN).Linear interpolation uses reference distances, while KNN selects the closest reference RIR.
  • Baselines: xRIR is the primary learning-based baseline and is retrained on the AR dataset at 16 kHz for 10 epochs.The comparison follows xRIR’s released configuration and scripts.
  • Baselines: Diff-RIR is compared only on HAA because it is a physics-based differentiable rendering method introduced with that dataset.FLAC is excluded because it is a concurrent work.

D. Implementations.

EIGENET predicts 16 kHz, 0.5-second RIRs using DAC acoustic tokens and a multi-view transformer, then demonstrates strong simulated and real-world performance across sparse reference settings.

  • Implementation: Each 16 kHz RIR contains 25 acoustic tokens with 1024 dimensions after DAC tokenization.The predicted RIR length is 8000 samples, corresponding to 0.5 seconds.
  • Implementation: The Cross-view Alternate-attention Transformer uses six 16-head blocks with 1024-dimensional features, while depth maps are encoded by a Vision Transformer.Depth tensors are divided into 16 × 32 patches before processing.
  • Implementation: Training uses waveform EDC and power-spectrum losses, with weights warmed up over the first 2000 steps and optimization performed for 10 epochs on eight H100 GPUs.The loss weights are λ_EDC = 1 and λ_spectrum = 0.01.
  • AcousticRooms results: EIGENET achieves the lowest error across all metrics and reference counts on AcousticRooms, including the extremely sparse K=1 setting.It outperforms both xRIR and classical interpolation baselines.
  • AcousticRooms results: On a challenging Cafe scene, EIGENET better preserves early-reflection peaks and reduces high-frequency artifacts in the late STFT tail than xRIR.The qualitative behavior is consistent with the quantitative gains reported for AcousticRooms.
  • Sim-to-real results: Across four HAA rooms and reference settings, EIGENET outperforms all baselines except Classroom with four reference views.In dampenedBase, its T60 error is 46 ∼49%, compared with xRIR’s approximately 191 ∼254%.

3) Scaling Behavior with Reference Count K:

EIGENET remains more accurate than xRIR as reference views vary from K=1 to K=8 on simulated and real data, although HAA scaling is less monotonic, potentially because of dataset bias.

  • Scaling setup: Scaling curves average errors across rooms on AcousticRooms and HAA test splits for K ∈{1, 4, 8}.The evaluated metrics are EDT, C50, and T60 errors, with lower values indicating better performance.
  • Real-data scaling: The HAA comparison spans four acoustic environments with 8:2 train-test splits and includes Diff-RIR at K=12.Metrics are EDT, C50, and T60 errors, with lower values preferred.
  • Simulated-data scaling: On AcousticRooms, EIGENET outperforms xRIR at every reference count and starts with a markedly lower error at K=1.The performance gap persists or widens as the number of references decreases.
  • Real-data scaling: On HAA, EIGENET’s advantage over xRIR holds across reference counts, but the expected monotonic improvement with increasing K weakens.The authors attribute this trend to potential dataset bias.

1) Influence of the Geometric Inputs:

Ablations show that geometric inputs and Alternate-attention are important for multi-view RIR prediction, while masking probes reveal distinct dependence on acoustic and geometric context, with T60 as an exception.

  • Influence of geometric inputs: Removing either panoramic depth or coordinate locations causes severe broadband degradation across all reference counts, especially for EDT and C50.Multi-octave analysis likewise supports the necessity of comprehensive geometric inputs.
  • Impact of attention mechanisms: Alternate-attention outperforms Cross-attention broadly and improves C50 and T60 over Self-attention, while maintaining comparable EDT.Across octave bands, it consistently leads on C50 and has advantages on EDT and mid-to-high-band T60.
  • Masking probes: Cross-attention and Self-attention rely heavily on acoustic tokens, whereas geometric-token growth provides only marginal improvement in their masking probes.When geometric and acoustic reference sets match, additional acoustic tokens produce further metric improvement.
  • Masking probes: Alternate-attention improves with more complete multi-view multi-modal context and performs worse when only partial geometric context is available.The probes indicate that it does not function as a mere soft interpolator.
  • Masking probes: T60 breaks the general masking pattern because it is a diffuse-field quantity and weakly position-dependent.The paper connects this behavior to estimating T60 from existing reference acoustic tokens using the Sabine equation.

3) Impact of the Modulation Block:

The modulation block is evaluated with different power-spectrum targets and across multiple attention mechanisms. Results show that the 7-band multi-octave target provides a cleaner supervision signal, while modulation consistently improves performance and works best with Alternate-attention.

  • Target choices: The ablation compares a 7-band multi-octave power spectrum with no power-spectrum loss and a full 513-bin STFT spectrum.The full STFT variant uses a 1024-sample window.
  • Target choices: Full STFT supervision improves EDT and C50 but weakens T60 because its dense frequency representation is difficult to regress.The 7-band multi-octave reduction instead provides a cleaner, physics-informed signal.
  • Attention mechanisms: The modulation block is evaluated with Alternate-attention, Self-attention, and Cross-attention, each with and without modulation under identical configurations.Experiments cover reference counts K ∈ {1, 4, 8}.
  • Attention mechanisms: Across all attention mechanisms and reference counts, variants with modulation uniformly outperform their counterparts.The gain is most dramatic for Cross-attention, while Self-attention also improves clearly.
  • Attention mechanisms: Alternate-attention with modulation consistently achieves the lowest errors across all metrics and reference counts.This combination is the paper’s proposed model and remains strongest despite modulation benefiting every attention type.

V. CONCLUSIONS

EIGENET combines data-driven modeling with geometry-informed physical priors for few-shot novel view RIR prediction. Its Cross-view Alternate-attention Transformer and geometry-informed modulation block achieve state-of-the-art performance across simulated and real-world datasets.

  • V. CONCLUSIONS: EIGENET integrates data-driven modeling with geometry-informed physical priors for few-shot novel view RIR prediction.The framework uses a Cross-view Alternate-attention Transformer and a geometry-informed modulation block under a multi-task learning paradigm.
  • V. CONCLUSIONS: Extensive experiments show state-of-the-art performance across both simulated and real-world datasets.Additional experiments provide empirical evidence for the interpretability and necessity of the proposed design.
Loading 2605.28101v1…