PaperlayerView exact revision on arXiv

Source-linked AI summary

EigeNet: Geometry-Informed Multi-Modal Learning for Few-shot Novel View RIR Prediction

Chong Jing, Zitong Lan, Junan Zhang, Zhizheng Wu

arXiv:2605.28101v1cs.SDcs.AIcs.MM

TL;DR

Few-shot novel-view RIR prediction must infer spatially varying acoustics from sparse observations while handling multi-view spatial-temporal structure and geometry-acoustic relationships. EIGENET combines CVAT with geometry-informed modulation and auxiliary power-spectrum supervision, achieving state-of-the-art performance across simulated and real-world datasets and sim-to-real generalization.

  • Problem

    Existing methods struggle with high-fidelity acoustic-field estimation because they inadequately capture multi-view spatial-temporal relationships and explicitly model geometry-acoustic correlations.

    Source: §I · p. 1

  • Method

    EIGENET combines a Cross-view Alternate-attention Transformer with geometry-informed modulation and auxiliary power-spectrum supervision for few-shot novel-view RIR prediction.

    Source: §I · p. 2, A. Problem Definition · p. 2

  • Results

    EIGENET achieves state-of-the-art performance across simulated and real-world datasets, with reported robustness under sparse references and consistent gains across network architectures.

    Source: §I · p. 2, §V · p. 11

  • Takeaways & Limitations

    The experiments support combining data-driven modeling with geometry-informed physical priors and multi-task supervision for few-shot novel-view RIR prediction.

    Source: §V · pp. 10–11

Abstract

from arXiv · show

Predicting spatially varying Room Impulse Response (RIR) from sparse observations is a critical but highly challenging inverse problem for immersive spatial audio rendering. In this work, we present EIGENET, a geometry-informed multi-modal framework for few-shot novel view RIR prediction. At its core is a Cross-view Alternate-attention Transformer that iteratively refines local intra-view acoustic structures and global cross-view spatial relationships. We empirically demonstrate that this architecture is capable of making full use of the multi-view multi-modal context while performing spatial-temporal reasoning for RIR prediction. Inspired by acoustic ray tracing, we design a geometry-informed modulation block to formulate the connection between geometric features and RIR power spectrum. In the mean time, an auxiliary loss is introduced to transform the single-target waveform prediction into a multi-task learning framework. Through ablation studies, we demonstrate that this design yields consistent performance gains regardless of the underlying backbone, thereby confirming its foundational utility and architecture-agnostic generalizability for RIR prediction task. Evaluated on both simulated and real-world benchmarks, EIGENET achieves both state-of-the-art performance in few-shot novel view RIR prediction and sim-to-real generalization. Codes and checkpoints are available on https://github.com/FEAfeatherTHER/EigeNet.

I. INTRODUCTIO

EIGENET addresses few-shot novel-view RIR prediction by combining alternate attention for multi-view spatial-temporal reasoning with geometry-informed acoustic modulation and auxiliary power-spectrum supervision.

  • Existing methods struggle to capture both temporal structure within individual RIRs and spatial relationships among sparse multi-view observations.Source: §I · p. 1
  • Black-box approaches often fail to explicitly model correlations between room geometry and acoustic patterns, limiting physical interpretability in complex configurations.Source: §I · p. 1
  • CVAT alternates intra-view and cross-view attention to model local temporal structure and global spatial relationships in multi-view multi-modal context.Source: §I · pp. 1–2
  • The geometry-informed modulation block uses geometric features to explicitly modulate acoustic representations, inspired by the relationship between geometry and RIR power spectrum.Source: §I · p. 2
  • The auxiliary power-spectrum loss converts single-target RIR prediction into multi-task learning and improves cross-room generalizability while supporting physically plausible synthesized RIRs.Source: §I · p. 2
  • Experiments report consistent gains across network architectures and state-of-the-art performance on AcousticRooms and Hearing-Anything-Anywhere, including robustness with sparse references.Source: §I · p. 2

II. RELATED WORK

The paper situates EIGENET among multimodal, differentiable-rendering, and alternate-attention approaches, then outlines its geometry encoding, modulation, and CVAT-based prediction pipeline.

  • Few-shot novel-view RIR methods use multimodal inputs, weighted reference-RIR combinations, or generative modeling to predict acoustic responses in unseen views.Source: A. Few-shot Novel View RIR Prediction… · p. 2
  • Differentiable-rendering methods combine physical acoustic knowledge with neural networks but may depend on approximations that do not hold universally.Source: B. Differentiable rendering of… · p. 2
  • Alternate-attention alternates frame-wise and global-wise self-attention, making it suited to multi-view context and previously evaluated for novel-view image prediction.Source: C. Alternate-attention · p. 2
  • EIGENET receives source, receiver, reference-RIR, and panorama-depth information to estimate a target RIR at a novel source position.Source: A. Problem Definition · p. 2
  • Reference and target geometric-acoustic tokens are concatenated and processed jointly by CVAT to predict the target RIR.Source: Figure 1 · p. 3, B. Model Overview · p. 3
  • A geometric encoder produces geometric tokens, while a modulation block conditions target acoustic tokens on geometry and predicts multi-octave power spectrum.Source: B. Model Overview · p. 3

C. Encoders

The encoders represent room geometry and reference or target RIR information as compatible tokens, while geometry modulates target acoustic tokens and supports multi-octave spectrum prediction.

Geometry encoder

  • Depth maps and source–receiver coordinates are projected into 3D, converted into two propagation-related tensors, and patchified into geometry tokens.The tensors are formed by subtracting source and receiver coordinates from 3D pixel coordinates.Source: 1) Geometric · p. 3

Acoustic encoder

  • Reference RIRs are encoded with pretrained 16 kHz DAC continuous latent representations before vector quantization.DAC was selected after evaluating its reconstruction quality on the AcousticRooms dataset.Source: 2) Acoustic · p. 3
  • Target-view time sequences are projected with an MLP from sinusoidal positional encoding and then modulated using geometry-informed features.The resulting target representation T0 is converted into target acoustic tokens by the modulation block.Source: 2) Acoustic · p. 3, D. Geometry-informed Modulation Block · p. 4

Geometry-informed modulation

  • The modulation block uses a DiT layer with adaptive normalization to combine target representations with geometry tokens and produce target acoustic tokens.This design operationalizes the relationship between room geometry and the RIR power spectrum.Source: D. Geometry-informed Modulation Block · p. 4
  • The auxiliary spectrum branch regresses a 7-band multi-octave spectrum spanning 63 Hz to 4000 Hz instead of the original STFT spectrum.The bands are 63, 125, 250, 500, 1 kHz, 2 kHz, and 4 kHz.Source: D. Geometry-informed Modulation Block · p. 4
  • A combined MRSTFT and EDC loss is used to prevent temporal collapse during spectrum prediction.The spectrum loss is defined as LMRSTFT(Ŝ0, S0) + LEDC(Ŝ0, S0).Source: D. Geometry-informed Modulation Block · p. 4

1) Token Organization:

Each view is organized as a sequence containing geometry and acoustic tokens, and the transformer alternates local within-view processing with global cross-view exchange.

Token Organization

4) Global Attention (Cross-view):

Global attention exchanges information across the unified multi-view sequence, enabling reference acoustic cues to inform target prediction while later features are projected and decoded into waveforms.

Global Attention (Cross-view)

  • Global attention operates over all view tokens, enabling information exchange across reference and target views.The global operation complements local within-view attention by exposing tokens to cross-view context.Source: 4) Global Attention (Cross-view) · p. 4
  • Cross-view exchange allows the model to aggregate acoustic cues from reference views when inferring the target view.The target prediction therefore uses information distributed across the available viewpoints.Source: 4) Global Attention (Cross-view) · p. 4
  • Stacked Alternate-attention blocks iteratively refine intra-view representations and integrate cross-view spatial information.Each block applies a local attention layer followed by a global attention layer.Source: 5) Alternating Structure · p. 4
  • Final local and global features are concatenated and projected into the hidden space of the DAC latent representation.The linear projection maps the combined features back to the DAC latent dimensionality.Source: F. Prediction Head · p. 4
  • The predicted latent representation is passed through the frozen DAC decoder to reconstruct the target waveform.The prediction head supervises the RIR signal in the waveform domain and targets fine-grained temporal and spectral characteristics.Source: F. Prediction Head · p. 4

B. Metrics.

Evaluation uses three acoustic RIR metrics and compares EigeNet with sampling, interpolation, nearest-neighbor, and learning-based baselines across simulated and real settings.

Metrics

  • The evaluation reports EDT, C50, and T60 as common measures of RIR prediction quality.EDT measures initial 5 dB decay, C50 compares early and late energy, and T60 measures 60 dB cumulative attenuation after an initial 5 dB drop.Source: B. Metrics · p. 5
  • T60 is approximated by multiplying estimated T20 by 3 for numerical stability.Predicted and ground-truth RIR T60 values are compared under this procedure.Source: B. Metrics · p. 5

Baselines

  • The baselines include Random Across Rooms, Random Same Room, Linear Interp., and Nearest Neighbor (KNN).Linear interpolation uses reference distances, while KNN selects the closest reference RIR.Source: C. Baselines · p. 5
  • xRIR is the primary learning-based baseline and is retrained on the AR dataset at 16 kHz for 10 epochs.The comparison follows xRIR’s released configuration and scripts.Source: C. Baselines · p. 5
  • Diff-RIR is compared only on HAA because it is a physics-based differentiable rendering method introduced with that dataset.FLAC is excluded because it is a concurrent work.Source: C. Baselines · p. 5

D. Implementations.

EIGENET predicts 16 kHz, 0.5-second RIRs using DAC acoustic tokens and a multi-view transformer, then demonstrates strong simulated and real-world performance across sparse reference settings.

Implementation

  • Each 16 kHz RIR contains 25 acoustic tokens with 1024 dimensions after DAC tokenization.The predicted RIR length is 8000 samples, corresponding to 0.5 seconds.Source: D. Implementations · p. 5
  • The Cross-view Alternate-attention Transformer uses six 16-head blocks with 1024-dimensional features, while depth maps are encoded by a Vision Transformer.Depth tensors are divided into 16 × 32 patches before processing.Source: D. Implementations · p. 5
  • Training uses waveform EDC and power-spectrum losses, with weights warmed up over the first 2000 steps and optimization performed for 10 epochs on eight H100 GPUs.The loss weights are λ_EDC = 1 and λ_spectrum = 0.01.Source: D. Implementations · p. 5

AcousticRooms results

  • EIGENET achieves the lowest error across all metrics and reference counts on AcousticRooms, including the extremely sparse K=1 setting.It outperforms both xRIR and classical interpolation baselines.Source: 1) Performance on AcousticRooms Dataset · p. 5
  • On a challenging Cafe scene, EIGENET better preserves early-reflection peaks and reduces high-frequency artifacts in the late STFT tail than xRIR.The qualitative behavior is consistent with the quantitative gains reported for AcousticRooms.Source: 1) Performance on AcousticRooms Dataset · p. 6, Figure 2 · p. 6

Sim-to-real results

  • Across four HAA rooms and reference settings, EIGENET outperforms all baselines except Classroom with four reference views.In dampenedBase, its T60 error is 46 ∼49%, compared with xRIR’s approximately 191 ∼254%.Source: 2) Sim-to-real Performance on… · p. 6

3) Scaling Behavior with Reference Count K:

EIGENET remains more accurate than xRIR as reference views vary from K=1 to K=8 on simulated and real data, although HAA scaling is less monotonic, potentially because of dataset bias.

Scaling setup

Real-data scaling

  • The HAA comparison spans four acoustic environments with 8:2 train-test splits and includes Diff-RIR at K=12.Metrics are EDT, C50, and T60 errors, with lower values preferred.Source: Table III · p. 7

Simulated-data scaling

  • On AcousticRooms, EIGENET outperforms xRIR at every reference count and starts with a markedly lower error at K=1.The performance gap persists or widens as the number of references decreases.Source: 3) Scaling Behavior with Reference… · p. 7

Real-data scaling

  • On HAA, EIGENET’s advantage over xRIR holds across reference counts, but the expected monotonic improvement with increasing K weakens.The authors attribute this trend to potential dataset bias.Source: 3) Scaling Behavior with Reference… · p. 7

1) Influence of the Geometric Inputs:

Ablations show that geometric inputs and Alternate-attention are important for multi-view RIR prediction, while masking probes reveal distinct dependence on acoustic and geometric context, with T60 as an exception.

Influence of geometric inputs

  • Removing either panoramic depth or coordinate locations causes severe broadband degradation across all reference counts, especially for EDT and C50.Multi-octave analysis likewise supports the necessity of comprehensive geometric inputs.Source: 1) Influence of the Geometric Inputs · pp. 7–8

Impact of attention mechanisms

  • Alternate-attention outperforms Cross-attention broadly and improves C50 and T60 over Self-attention, while maintaining comparable EDT.Across octave bands, it consistently leads on C50 and has advantages on EDT and mid-to-high-band T60.Source: 2) Impact of Different Attention… · p. 8

Masking probes

  • Cross-attention and Self-attention rely heavily on acoustic tokens, whereas geometric-token growth provides only marginal improvement in their masking probes.When geometric and acoustic reference sets match, additional acoustic tokens produce further metric improvement.Source: 2) Impact of Different Attention… · p. 9
  • Alternate-attention improves with more complete multi-view multi-modal context and performs worse when only partial geometric context is available.The probes indicate that it does not function as a mere soft interpolator.Source: 2) Impact of Different Attention… · p. 9
  • T60 breaks the general masking pattern because it is a diffuse-field quantity and weakly position-dependent.The paper connects this behavior to estimating T60 from existing reference acoustic tokens using the Sabine equation.Source: 2) Impact of Different Attention… · p. 9

3) Impact of the Modulation Block:

The modulation block is evaluated with different power-spectrum targets and across multiple attention mechanisms. Results show that the 7-band multi-octave target provides a cleaner supervision signal, while modulation consistently improves performance and works best with Alternate-attention.

Target choices

Attention mechanisms

  • The modulation block is evaluated with Alternate-attention, Self-attention, and Cross-attention, each with and without modulation under identical configurations.Experiments cover reference counts K ∈ {1, 4, 8}.Source: 3) Impact of the Modulation Block · p. 10, Table VI · p. 10
  • Across all attention mechanisms and reference counts, variants with modulation uniformly outperform their counterparts.The gain is most dramatic for Cross-attention, while Self-attention also improves clearly.Source: 3) Impact of the Modulation Block · p. 10
  • Alternate-attention with modulation consistently achieves the lowest errors across all metrics and reference counts.This combination is the paper’s proposed model and remains strongest despite modulation benefiting every attention type.Source: 3) Impact of the Modulation Block · p. 10, Figure 10 · p. 10

V. CONCLUSIONS

EIGENET combines data-driven modeling with geometry-informed physical priors for few-shot novel view RIR prediction. Its Cross-view Alternate-attention Transformer and geometry-informed modulation block achieve state-of-the-art performance across simulated and real-world datasets.

V. CONCLUSIONS

  • EIGENET integrates data-driven modeling with geometry-informed physical priors for few-shot novel view RIR prediction.The framework uses a Cross-view Alternate-attention Transformer and a geometry-informed modulation block under a multi-task learning paradigm.Source: §V · p. 10
  • Extensive experiments show state-of-the-art performance across both simulated and real-world datasets.Additional experiments provide empirical evidence for the interpretability and necessity of the proposed design.Source: §V · p. 11

Related papers

Loading 2605.28101v1…