PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Neural Acoustic Fields

Andrew Luo, Yilun Du, Michael J. Tarr, Joshua B. Tenenbaum, Antonio Torralba, Chuang Gan

arXiv:2204.00628v2cs.SDcs.CVcs.LGcs.ROeess.AS

TL;DR

Spatial auditory representations have lagged behind learned visual representations, despite acoustic cues informing perception of sources and environments. Neural Acoustic Fields represent scene acoustics continuously using impulse responses and local geometric conditioning; they predict plausible reverberations at unseen locations and can support visual learning and scene-structure inference.

  • Problem

    Learned implicit representations have advanced visual modeling, but spatial auditory representations remain less developed, despite acoustic cues conveying source and environmental information.

    Source: §1 · p. 1

  • Method

    NAFs encode impulse responses in the time-frequency domain and condition predictions on learned local geometry at emitter and listener locations.

    Source: §3.2 · p. 4, §3.3 · p. 5

  • Results

    NAFs predict plausible environmental reverberations at unseen scene locations, outperform baselines in acoustic modeling, and their learned representations may aid visual learning and scene-structure inference.

    Source: §1 · p. 2, §4.3 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    The learned acoustic representations may facilitate audio-visual cross-modal learning and be used to infer scene structure.

    Source: §5 · p. 10

    NAFs generalize within a single scene but do not currently generalize to multiple scenes.

    Source: §5 · p. 10

Abstract

from arXiv · show

Our environment is filled with rich and dynamic acoustic information. When we walk into a cathedral, the reverberations as much as appearance inform us of the sanctuary's wide open space. Similarly, as an object moves around us, we expect the sound emitted to also exhibit this movement. While recent advances in learned implicit functions have led to increasingly higher quality representations of the visual world, there have not been commensurate advances in learning spatial auditory representations. To address this gap, we introduce Neural Acoustic Fields (NAFs), an implicit representation that captures how sounds propagate in a physical scene. By modeling acoustic propagation in a scene as a linear time-invariant system, NAFs learn to continuously map all emitter and listener location pairs to a neural impulse response function that can then be applied to arbitrary sounds. We demonstrate that the continuous nature of NAFs enables us to render spatial acoustics for a listener at an arbitrary location, and can predict sound propagation at novel locations. We further show that the representation learned by NAFs can help improve visual learning with sparse views. Finally, we show that a representation informative of scene structure emerges during the learning of NAFs.

1 Introduction

Spatial acoustic cues convey source direction, distance, and environmental properties, but acoustic representations pose challenges absent from visual learning. Neural Acoustic Fields model scene acoustics continuously and support prediction at novel emitter–listener locations.

1 Introduction

  • Spatial acoustic cues convey source direction and distance without visual confirmation, help estimate surrounding-environment properties, and contribute to realism in games and simulations.Acoustic cues complement visual observations in everyday perception.Source: §1 · p. 1
  • NAFs represent scene acoustics implicitly, addressing prior approaches’ handcrafted parameterizations that prevent application to arbitrary scenes.The paper introduces a generic implicit neural representation of scene acoustics.Source: §1 · pp. 1–2
  • Acoustic impulse responses can exceed 10,000 time-domain values, while ray-tracing for novel emitter–listener locations requires intractable computation.These are the two challenges identified for learning acoustic neural representations.Source: §1 · p. 2
  • NAFs encode impulse responses in the time-frequency domain and condition decoding on learned local geometry at emitter and listener locations.This decomposition transfers local information from training pairs to novel emitter–listener combinations.Source: §1 · p. 2
  • NAFs outperform baselines in modeling scene acoustics and their learned structure can improve novel-view generation and support scene-structure inference.The introduction reports these outcomes without specifying benchmark values.Source: §1 · p. 2

2 Related Work

Prior sound-field methods balance efficiency and fidelity through handcrafted encodings, while learned approaches have often fixed either listener or emitter position. NAFs use an implicit representation and local grids to model acoustics across arbitrary emitter and listener locations.

Audio Field Coding

  • Earlier learned sound-field approaches use either a stationary listener or emitter, whereas NAFs enable querying arbitrary emitter and listener locations.Handcrafted encodings have often prioritized efficiency over full-fidelity reproduction.Source: §2 · p. 2

Implicit representations

  • NAFs leverage local grids as an inductive bias for generalizing to novel inputs, with a shared feature grid queried at listener and emitter positions.The architecture also embeds position, frequency, and time, and predicts magnitude and phase.Source: §2 · p. 3, Figure 2 · p. 3

Audio-Visual Learning

  • Related audio-visual work models reverberations or sounds from images, while NAFs use geometric features learned from acoustic fields to improve 3D view generation.The cited approaches model the STFT using convolution or implicit functions.Source: §2 · p. 3

3 Methods

The methods represent scene acoustics as impulse responses that can be queried for emitter–listener locations. NAFs predict these responses in a time-frequency representation and use local geometric features to support generalization to unseen location combinations.

3.1 Background on the Propagation of Sound

  • For a fixed location pair, an impulse response is the sound pressure induced at the listener by an impulse at the emitter; convolving it with an emitted waveform gives the listener response.Figure 3 visualizes test-set impulse-response predictions and audio reverberated with predicted and ground-truth responses.Source: §3.1 · p. 3, Figure 3 · p. 4

3.2 Neural Acoustic Fields

  • A neural field maps listener position, head orientation, ear, and emitter position to a time-domain impulse response.The paper models directional listeners using their 3D orientation and a binary left/right ear index.Source: §3.2 · p. 4
  • Because direct waveform prediction is difficult, NAFs predict an impulse response’s STFT log-magnitude and instantaneous-frequency components, whose structure is more learnable.The phase is represented through instantaneous frequency derived from unwrapped phase differences over time.Source: §3.2 · p. 4

3.3 Generalization through Local Geometric Conditioning

  • NAFs use a learned 2D grid of spatial latents, queried at emitter and listener positions, as context for predicting impulse responses at unseen location combinations.The grid features are learned jointly with the NAF parameters.Source: §3.3 · p. 5
  • A single latent grid serves emitter and listener locations, reflecting acoustic reciprocity: swapping omnidirectional emitters and listeners does not change the impulse response.The authors report that this shared-grid design outperforms a separate dual-grid implementation.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate NAFs on acoustic-field reconstruction, cross-modal visual learning, and scene-structure inference, with acoustic modeling tested at seen and unseen locations.

4.1 Setup

  • The evaluation uses six synthetic Soundspaces scenes and a recorded cuboidal-room MeshRIR scene, with a 10% random RIR test split and 200 training epochs per scene.Soundspaces includes multi-room, non-rectangular, and rectangular layouts; MeshRIR is monaural and varies emitter height.Source: §4.1 · p. 6

4.2 Architecture Details

  • NAFs use a restricted Soundspaces input parameterization and combine interpolated local-grid features with coordinate, orientation, ear, time, and frequency embeddings to predict STFT magnitude and phase.The grid features are jointly learned through differentiable interpolation; MeshRIR additionally allows three-dimensional emitter positions and omits orientation and binaural embeddings.Source: §4.2 · pp. 6–7

4.3 Evaluation of Neural Acoustic Fields

  • NAFs outperform interpolation baselines on unseen impulse responses across six scenes and perform better on spectral and T60 metrics in MeshRIR; human raters preferred NAFs in 82.38% of responses.The comparison reports a fraction of the storage cost and no sound-energy leakage found with linear interpolation. Sharing local geometric features also performs better despite using fewer learnable parameters, while geometric conditioning improves reconstruction increasingly as training data decreases.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 Cross-modal learning

  • Jointly training NeRF and NAF with a shared local grid improves visual PSNR and SSIM on 50 test images, with larger gains for sparser training views.The qualitative comparison reports fewer floaters in free space; NAF contributes less when visual training data is sufficient.Source: §4.4 · pp. 8–9, Table 3 · p. 8, Figure 6 · p. 8

4.5 Inferring scene structure

  • A linear layer decodes nearest-wall distance from NAF latents more successfully than from MFCC features, whose decoding fails.NAF features are smooth and semantically meaningful, separate multi-room layouts, and achieve higher explained variance on test locations.Source: §4.5 · pp. 9–10, Table 4 · p. 9, Figure 8 · p. 9

5 Discussion

NAFs represent scene acoustics and support unseen-location reverberation prediction, audio-visual learning, and scene-structure inference. The current method generalizes within a single scene, while broader access to spatial audio and increased gaming time are possible societal effects.

5 Discussion

  • The method currently generalizes within a single scene; generalization to novel scenes remains future work.The authors suggest multimodal inputs as one possible route to synthesizing fields from few-shot visual or acoustic input.Source: §5 · p. 10
  • Spatial audio may broaden deployment and equitable access, while more believable gaming audio could increase time spent gaming.Both societal effects are framed as possibilities.Source: §5 · p. 10
  • NAFs represent scene acoustics and support prediction of plausible reverberations at unseen locations, audio-visual learning, and scene-structure inference.The authors describe the representation as compact, continuous, and differentiable.Source: §5 · p. 10

A Additional Visualization of Rooms

Additional visualizations show NAF-predicted loudness across four rooms. The maps reflect room geometry and source distance.

A Additional Visualization of Rooms

B Additional Visualization on Real-World Data

On the real-world MeshRIR dataset, NAFs predict a smoothly varying acoustic field from discrete training samples, unlike interpolation methods that introduce artifacts.

B Additional Visualization on Real-World Data

C Additional quantitative results

The additional MeshRIR results report better performance than a kernel-ridge baseline and low error on an interaural localization-related measure. NAFs can be queried at arbitrary emitter and receiver positions, unlike the baseline's per-emitter models.

C Additional quantitative results

D Architecture and Training Details

The section compares three local-geometry conditioning designs and details the network, input encodings, and training procedures used to learn impulse responses.

D Architecture and Training Details

  • Sharing emitter and listener feature grids performs better than using separate grids, while a third variant omits local-geometry conditioning.The separate-grid model samples emitter and listener positions from different grids; the shared-grid model samples both from one grid.Source: D Architecture and Training Details · p. 17
  • Emitter, listener, phase, and time inputs use sinusoidal embeddings; orientation and left/right are retrieved, then added as 512-dimensional conditioning vectors at intermediate layers.The orientation and left/right embeddings have shapes R^(7×4×512) and R^(7×2×512), respectively.Source: D Architecture and Training Details · pp. 17–18
  • Each network has eight fully connected layers and a two-layer skip connection added to the fourth intermediate layer, with 512 features and Leaky ReLU activation.The no-conditioning model's grid is initialized across the scene bounding box, with points 0.25 m from their nearest neighbors and 64 features per point.Source: Figure A5 · p. 17
  • For each scene, the log-spectrogram is normalized using per-time/frequency means and standard deviations, and the instantaneous-frequency phase representation is normalized before training.Ten sine and cosine frequencies encode position, phase, and time; maximum frequencies are 27 Hz for position and 210 Hz for time and frequency.Source: D Architecture and Training Details · p. 18
  • Training impulse responses are zero-padded to each scene's maximum length, with stochastic padding applied at 0.1 probability to focus modeling on early regions; inference runs to that maximum duration.Training samples can have different lengths because the implicit function is trained on individual time-frequency coordinates.Source: D Architecture and Training Details · p. 18

E Dataset Visualization

The visualization shows sparse emitter-listener probes in a room and contrasts interpolation with the denser acoustic fields recovered by NAF.

E Dataset Visualization

  • Occlusion and room geometry mean even small emitter or listener movements can change results; nearest-neighbor and linear interpolation perform poorly, while NAF recovers substantially denser fields.Figure A6 shows the room structure and probe locations used in training.Source: Figure A6 · p. 18

F Storage Comparison

The appendix compares spatial-acoustic storage and compression baselines, then tests time-domain learning and grid regularization as alternatives.

F Storage Comparison

G Details of the compression baselines

  • AAC-LC and Opus serve as lossy audio-coding baselines, with bitrates chosen to approximate NAF representation sizes where the encoders allow.The methods were selected for widespread usage, strong bitrate quality, and open-source implementations.Source: G Details of the compression baselines · p. 19

G.1 AAC baseline

  • AAC-LC uses a combined stereo bitrate of 24 kBit/s in constant-bitrate mode; lower rates sometimes caused encode/decode failures.The implementation uses ffmpeg 5.0's open-source "aac" encoder.Source: §G.1 · p. 19

G.2 Opus baseline

  • Opus uses 12kBit/s for stereo in constrained variable-bitrate mode, with complexity 10 and music mode.The setup uses opustools 0.2 backed by libopus 1.3.1.Source: §G.2 · p. 19

H Alternative Neural Representations

I L2 regularized grid in NeRF

Loading 2204.00628v2…