PaperlayerView exact revision on arXiv

Source-linked AI summary

CTAN: Cycle-Temporal Attention Network for Embodied Audio-Visual Navigation

Teng Liu, Yinfeng Yu

arXiv:2609.17420v1cs.MMcs.AIcs.SDeess.SP

TL;DR

Audio-visual navigation needs stronger semantic interaction between heterogeneous visual and acoustic features than simple multimodal aggregation provides. CTAN uses AVRCA for bidirectional cycle-consistent enhancement and TCMM for historical-context integration, achieving superior performance over prior methods on Replica and Matterport3D across SR, SPL, and SNA.

  • Problem

    Simple multimodal aggregation fails to capture deep semantic relationships between vision and audio, underusing complementary information and weakening robust navigation in complex environments.

  • Method

    CTAN combines AVRCA, which uses bidirectional cycle consistency for semantic enhancement, with TCMM, which adaptively integrates enhanced features and historical context.

  • Results

    CTAN achieves superior success rate, success weighted by path length, and scene navigation accuracy over previous audio-visual navigation methods on Replica and Matterport3D.

  • Takeaways & Limitations

    The framework supports stable semantic associations and persistent spatial awareness, particularly when sounds are unseen or acoustic perception is unreliable.

Abstract

from arXiv · show

Audio-visual embodied navigation equips robots with the capability to infer the locations of sound sources by integrating visual inputs and acoustic information (e.g., depth observations and binaural audio cues). The core challenge lies in establishing effective semantic interactions across heterogeneous modalities (which exhibit distinct feature distributions). Existing feature fusion strategies, however, often rely on simple multimodal aggregation and therefore fail to capture the underlying geometric and semantic relationships, leading to information degradation in complex environments. To overcome these limitations, this work presents the Cycle-Temporal Attention Network (CTAN), a framework designed for active semantic-enhanced fusion (rather than straightforward multimodal combination). Specifically, the proposed Audio-Visual Reconstruction Cross-Attention (AVRCA) module employs a bidirectional cycle-consistency constraint (between visual and acoustic representations) to reinforce the spatial semantic attributes of both modalities, thereby facilitating more robust cross-modal interaction. Additionally, we design a Temporal Cross-Modal Memory (TCMM) mechanism to dynamically integrate real-time enhanced multimodal features with historical context, reducing performance drops caused by auditory dead zones. Experimental results obtained on the Replica and Matterport3D benchmarks indicate that the proposed approach achieves superior performance over previous audio-visual navigation methods in terms of success rate (SR), success weighted by path length (SPL), and scene navigation accuracy (SNA).

I. INTRODUCTION

Audio-visual navigation requires meaningful semantic interaction between vision and audio, but simple feature combination underuses complementary information and weakens robustness. CTAN addresses this with bidirectional semantic enhancement and temporal memory, and reports stronger navigation across Replica and Matterport3D.

  • Existing methods often rely on simple feature combinations that miss deep vision-audio semantic connections and underuse complementary information.Direct fusion also struggles to maintain durable interactions when modalities are disturbed or incomplete.
  • CTAN combines AVRCA for bidirectional semantic enhancement with TCMM for historical-context integration, targeting stable cross-modal associations in complex environments.AVRCA uses cycle consistency, while TCMM maintains continuous perceptual representation.
  • TCMM adaptively fuses instantaneous enhanced perceptions with historical context to construct persistent spatial representations and mitigate auditory dead zones.The temporal interaction is intended to support stable, coherent policy execution in complex indoor environments.
  • CTAN consistently surpasses existing cross-modal fusion methods on Replica and Matterport3D in navigation efficiency and success rate.The supplied result passage identifies both benchmarks and reports superior performance, without giving numerical values.

II. RELATED WORK

Audio-visual navigation research has progressed from multimodal policy learning and structured spatial memory toward richer semantic and temporal modeling. Recent methods address cross-modal alignment, disturbance robustness, and unreliable acoustic observations, while TCMM dynamically preserves useful historical context.

  • SoundSpaces established real-scanned Replica and Matterport3D environments with high-fidelity acoustic simulation for embodied audio-visual navigation.Early studies connected multimodal perception with navigation policies and showed auditory guidance can aid navigation in unseen 3D environments.
  • Prior work explored hierarchical policy optimization, audio-visual maps, local geometric memory, semantic label embeddings, oracle guidance, adversarial learning, and omnidirectional perception.These methods progressively enrich spatial representation, semantic correspondence, multimodal alignment, and disturbance robustness.
  • Existing temporal approaches aggregate historical information with recurrent networks or transformers, but may fail to recover context in auditory dead zones.TCMM instead uses a dynamic context buffer with adaptive historical-feature selection to preserve temporal continuity.

III. METHODOLOGY

CTAN encodes depth and binaural spectrogram observations, enhances each modality through bidirectional cross-attention with cycle-consistency regularization, and fuses the result into a spatial representation. The fused embedding is then prepared for temporal sequence modeling through TCMM.

  • III. METHODOLOGY: The model separately encodes depth images and binaural spectrograms with CNN encoders before cross-modal enhancement.Each encoder uses three convolutional layers followed by a 512-unit fully connected layer with ReLU activation.
  • B. Cross-Modal Fusion: AVRCA performs bidirectional cross-modal translation, using each modality as a query to attend to the other and produce audio-augmented visual and visual-augmented audio features.This replaces passive concatenation with adaptive aggregation of relevant cross-modal context.
  • B. Cross-Modal Fusion: Cycle consistency reconstructs the original modality representations from translated features, with an MSE loss constraining the cyclic reconstructions.The constraint is introduced to prevent trivial solutions or attention collapse and preserve modality-specific structure.
  • III. METHODOLOGY: The enhanced modality representations are concatenated and mapped by an MLP to produce the current-step spatial representation, which feeds TCMM sequence modeling.The resulting context-rich vector is the input to subsequent temporal processing.

C. Adaptive Temporal Reasoning

TCMM combines current multimodal perception with temporally selected historical context, then adaptively gates the two signals before producing a normalized state for policy learning.

  • C. Adaptive Temporal Reasoning: TCMM uses current perception as a query over a positional-encoded sliding memory to extract relevant historical context.The memory contains the previous k fused embeddings, with k = 10 in the implementation.
  • C. Adaptive Temporal Reasoning: An adaptive gate balances the current embedding and contextual cue to form a temporal-aware state.The gate is computed from their concatenation using an MLP and Sigmoid activation.
  • C. Adaptive Temporal Reasoning: Layer normalization produces the final state representation, which is then passed to a GRU and Actor-Critic network for policy learning and value estimation.The normalization is used to ensure training stability.

A. Experimental Setup

Experiments evaluate CTAN in Habitat using SoundSpaces audio rendering on Replica and Matterport3D, with navigation assessed by SPL, SR, and SNA under depth-based evaluation.

  • A. Experimental Setup: Experiments use Replica and Matterport3D environments with SoundSpaces-based acoustic simulation in Habitat.The setup uses room impulse responses and material-specific properties generated through bidirectional path-tracing.
  • A. Experimental Setup: The evaluation tables report overall navigation performance and component-removal ablations on Replica and Matterport3D.The ablation tables examine individual components, with “w/o” denoting removal of the corresponding component.
  • A. Experimental Setup: SPL measures path efficiency, SR measures target-reaching success, and SNA combines goal attainment with final orientation accuracy.The three metrics are reported as percentages under the depth setting.

B. Performance Comparison

CTAN is evaluated on unseen environments with heard and unheard sound settings, and it outperforms SoundSpaces in path efficiency and success rate across both benchmarks.

  • B. Performance Comparison: Evaluation covers heard and unheard sounds in unseen environments, testing cross-modal generalization without access to memorized spatial layouts.Heard sounds share training and testing categories, whereas unheard sounds are not observed during training.
  • B. Performance Comparison: CTAN improves SPL over SoundSpaces by 2.5% and 1.9% for heard sounds, and by 4.9% and 5.5% for unheard sounds on Replica and Matterport3D.These gains indicate more efficient path planning in the reported comparisons.
  • B. Performance Comparison: CTAN improves SR over SoundSpaces by 1.1% and 2.6% for heard sounds, and by 6.3% and 4.2% for unheard sounds on Replica and Matterport3D.The reported unheard-sound comparisons involve unseen acoustic conditions.

C. Ablation Study

Ablations show that AVRCA, TCMM, and cycle consistency each contribute to CTAN’s navigation performance, with distinct effects across evaluation settings and environments.

  • C. Ablation Study: Removing any CTAN component generally degrades performance under both Heard and Unheard settings on Replica and Matterport3D.The ablated components are AVRCA, TCMM, and the cycle-consistency loss Lcyc.
  • C. Ablation Study: 6.5% SR drop follows AVRCA removal under Replica’s Unheard setting, highlighting the importance of cross-modal semantic enhancement for novel sound sources.Removing AVRCA also causes a noticeable decline in SR and SPL.
  • C. Ablation Study: CTAN’s representative trajectories use shorter paths and higher SPL than the compared navigation behavior, while SoundSpaces shows wandering and backtracking.The supplied trajectory caption defines higher SPL and shorter blue trajectories as better performance.
  • C. Ablation Study: SPL falls from 27.4% to 21.9% when TCMM is removed under Matterport3D’s Unheard setting, indicating temporal memory’s value in large-scale environments.The decrease is larger than under the Heard setting, consistent with preserving historical context over long trajectories.
  • C. Ablation Study: Removing Lcyc consistently decreases SPL across both datasets and evaluation settings, supporting its role as a regularizer for stable audio-visual associations.The decline is smaller than that caused by removing the architectural modules.

D. Qualitative Analysis

Qualitative analyses show that CTAN produces efficient trajectories and dynamically balances immediate perception with historical memory during navigation.

  • Navigation trajectories: CTAN generally stays closer to theoretical shortest paths than SoundSpaces, which exhibits inefficient exploration, local wandering, and repeated backtracking.The trajectory comparison covers representative episodes on Replica and Matterport3D.
  • Dynamic Gating Analysis: TCMM’s memory and perception weights vary across sampled action steps, reflecting adaptive rather than fixed use of historical context and current observations.Figure 3 visualizes normalized weights from the TCMM gating values gt and 1 −gt.

V. CONCLUSIONS

CTAN addresses difficult audio-visual navigation through semantic cross-modal enhancement and temporal memory, achieving consistent gains on Replica and Matterport3D.

  • V. CONCLUSIONS: CTAN combines AVRCA for visual-auditory semantic correspondence with TCMM for retaining useful historical information during navigation.These designs target challenging cross-modal interaction and incomplete acoustic observations in complex 3D environments.
  • V. CONCLUSIONS: CTAN consistently improves navigation success, path efficiency, and scene adaptability on Replica and Matterport3D, particularly for unseen sounds and large-scale indoor environments.The conclusion identifies dynamic scenarios and richer sensory information as future directions.
Loading 2609.17420v1…