PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformer-Based Token Fusion and Dynamic Graph Planning for Audio-Visual Navigation

Shaohang Wu, Yinfeng Yu

arXiv:2609.17421v1cs.AIeess.SP

TL;DR

Audio-visual navigation remains vulnerable to incomplete or misleading visual perception and inefficient collision-based recovery. TDGP combines transformer-based token fusion for high-level multimodal decisions with a collision-penalty graph planner for low-level replanning. Experiments report stronger performance on Replica and MP3D, with sound enhancement improving generalization in unheard acoustic scenarios.

  • Problem

    Audio-visual navigation systems lack adaptive correction and replanning under incomplete or misleading visual perception, while collision-based compensation is unsafe and inefficient.

  • Method

    TDGP decouples high-level perception from low-level control using transformer-based audio-visual token fusion and a collision-penalty planner that removes collided edges and applies penalties.

  • Results

    TDGP outperforms baseline models on Replica and MP3D, and sound enhancement improves generalization in unheard acoustic scenarios.

  • Takeaways & Limitations

    The model supports robust local decision-making and adaptive path replanning under perceptual uncertainty, especially in unheard acoustic scenarios.

Abstract

from arXiv · show

Audio-Visual Navigation (AVN) requires an agent to localize and navigate toward a continuously vocalizing target relying solely on visual observations and acoustic cues. Currently, systems lack the ability to adaptively correct and replan when faced with incomplete or misleading visual perception. Furthermore, relying on physical collisions to compensate for missing visual information results in inefficient and unsafe navigation, whereas existing methods are overly dependent on passive visual perception. To address these issues, we propose the Transformer-based Token Fusion and Dynamic Graph Planning (TDGP) model, which incorporates high-level perception layers and leverages the Transformer model to fuse multimodal cues for precise local planning. Next, a low-level planning layer is designed that uses physical collision penalties to remove edges that collide with the map in real time and apply corresponding penalties, forcing the agent to automatically re-plan to compensate for the lack of visual information. Experiments show that our TDGP model outperforms baseline models on the Replica and Matterport3D (MP3D) datasets, and that the model's sound enhancement strategy significantly improves generalization in unheard acoustic scenarios.

I. INTRODUCTION

Audio-visual navigation combines visual and auditory cues, but passive vision can be incomplete or misleading, while collision-based recovery is unsafe and inefficient. TDGP addresses these limitations by coupling transformer-based audio-visual fusion with collision-penalty planning and reports state-of-the-art performance on Replica and MP3D.

  • I. INTRODUCTION: Passive visual perception is limited by narrow viewpoints, missing global context, transparent obstacles, and abnormal lighting that can create false passability signals.These failures can lead traditional systems to plan through apparently traversable but obstructed regions.
  • I. INTRODUCTION: Physical collisions provide geometric information when visual perception is unreliable, but frequent collisions are unsafe and inefficient for navigation.The paper therefore treats collision feedback as information to incorporate into planning rather than relying only on post-collision recovery.
  • I. INTRODUCTION: The Tokenization Fusion Mechanism in High-Level Transformers performs deep alignment between audio and visual features.The mechanism is designed to improve multimodal integration for navigation decisions.
  • I. INTRODUCTION: The Collision-Penalty Path Planner removes collided edges from the navigation map and applies a penalty, enabling automatic replanning.This mechanism is intended to mitigate the risks and inefficiencies associated with physical collisions.
  • I. INTRODUCTION: TDGP decouples high-level perception and decision-making from low-level path planning to support local navigation under perceptual uncertainty.Its high-level transformer fuses audio-visual cues, while the low-level planner handles path selection.
  • I. INTRODUCTION: TDGP achieves state-of-the-art performance on Replica and MP3D, while sound enhancement improves generalization in unheard acoustic scenarios.The paper evaluates the model across these two datasets and emphasizes robustness to unseen acoustics.

A. Hierarchical Reinforcement Learning and Planning for Audio-Visual Navigation

The method uses hierarchical reinforcement learning with a high-level policy that updates local goals from current maps and sounds, and a low-level planner that executes them. TDGP implements this hierarchy through transformer token fusion, GRU temporal decision-making, and local target selection on a 3 × 3 grid.

  • A. Hierarchical Reinforcement Learning and Planning for Audio-Visual Navigation: The high-level policy is invoked at every environmental timestep, allowing local goals to update from the latest maps and sounds.This differs from approaches that update hierarchical goals less frequently.
  • A. Hierarchical Reinforcement Learning and Planning for Audio-Visual Navigation: TDGP consists of the Tokenization Fusion Mechanism in High-Level Transformers and the Low-level Collision-Penalty Path Planner.These modules form a two-layer policy that separates navigation decisions from path planning.
  • A. Hierarchical Reinforcement Learning and Planning for Audio-Visual Navigation: Table I reports experiments trained on heard and unheard telephone splits from the Replica and Matterport3D datasets.The table covers both dataset environments and acoustic conditions.
  • A. Hierarchical Reinforcement Learning and Planning for Audio-Visual Navigation: The fusion mechanism extracts visual and acoustic features, converts them into tokens, and uses a Transformer to integrate information across modalities.A learnable [CLS] token aggregates global sequence information for multimodal decision-making.
  • A. Hierarchical Reinforcement Learning and Planning for Audio-Visual Navigation: A GRU processes the fused global context with the previous hidden state and outputs probabilities over nine local navigation targets on a 3 × 3 grid.The resulting local target is selected by the high-level policy for downstream planning.

B. Low-level Collision-Penalty Path Planner

The CPP converts high-level local goals into atomic actions while maintaining a dynamically updated navigation graph. Collision signals remove blocked edges and, together with collision penalties, encourage safer replanning.

  • B. Low-level Collision-Penalty Path Planner: CPP transforms each local goal into an atomic action using a dynamic graph of navigable nodes and valid edges.The planner maps the goal to the nearest graph node, computes a shortest path, and derives the next action from the next node and agent orientation.
  • B. Low-level Collision-Penalty Path Planner: The planner uses Dijkstra’s algorithm to obtain a shortest path from the current node to the goal node before selecting the next node and atomic action.The local 3×3 goal grid is transformed from egocentric to global coordinates and mapped to the nearest navigation node.
  • B. Low-level Collision-Penalty Path Planner: A collision removes the attempted edge from the graph at the next time step, allowing the planner to refine future paths from physical feedback.The collision indicator identifies whether the attempted transition failed, and the corresponding edge is removed from the edge set.
  • B. Low-level Collision-Penalty Path Planner: The reward combines target, proximity, and collision terms to penalize unsafe interactions and incentivize proactive avoidance during training.The proximity reward is positive near the target and negative when the agent is far away; the collision term addresses the efficiency limits of relying only on collisions.

C. Audio Enhancement

The audio enhancement strategy augments training episodes with noise, additional target-location sounds, and spectral masking. These augmentations are probabilistically activated to improve robustness in complex acoustic environments.

  • C. Audio Enhancement: Training probabilistically applies one or more audio augmentations at each episode’s start to improve policy robustness and generalization.The strategy can add random non-target noise, mix a second sound at the target, and apply temporal and frequency spectral masks.

A. Experimental Setup

Experiments use SoundSpaces 1.0 with Replica and MP3D indoor-scene datasets under heard and unheard acoustic conditions. The MP3D unheard setting is identified as the most demanding and primary evaluation focus.

  • A. Experimental Setup: Experiments run on SoundSpaces 1.0, which combines Habitat-based visual rendering with physics-based audio simulation.The platform is used for both training and evaluation in audio-visual navigation.
  • A. Experimental Setup: Replica contains 18 scanned indoor scenes, whereas MP3D contains 85 larger and more diverse environments for evaluating generalization.Scenes are split into training, validation, and testing groups: 9/4/5 for Replica and 73/11/18 for MP3D.
  • A. Experimental Setup: Evaluation covers heard and unheard acoustic settings, with unheard MP3D serving as the most demanding and primary focus.The dataset splits support testing across different indoor scenarios and acoustic conditions.
  • A. Experimental Setup: The environment visualizations show RGB images, depth inputs, top-down planning maps, shortest paths, agent positions, and fields of view.The maps depict the planning space and the green shortest path for each scene example.

B. Performance of the Transformer-based Token Fusion and Dynamic Graph Planning model

TDGP outperforms SoundSpaces across Replica and MP3D navigation metrics while combining multimodal fusion with collision-penalized planning. Its components are complementary, though projected-map noise can trade path efficiency for success rate.

  • Model Performance: TDGP improves SPL, SR, and SNA over SoundSpaces on Replica and MP3D in both heard and unheard scenarios.Replica gains are 14.8%, 7.3%, and 25.6% in heard conditions and 7.0%, 9.1%, and 9.5% in unheard conditions; MP3D gains are 8.3%, 6.0%, and 22.9% heard and 14.2%, 8.7%, and 15.5% unheard, respectively.
  • Limitations: CPP uses collision penalties and graph-edge removal to support replanning, but projected 2D-map noise can preserve success rate at the expense of SPL.The limitation is most relevant to large, complex environments such as MP3D.
  • Model Performance: TDGP’s two-layer design separates high-level perception and low-level planning to address incomplete multimodal perception and navigation instability.TFM handles cross-modal fusion, while CPP modifies the navigation graph after collisions.
  • Ablation Studies: The complete model outperforms configurations without TFM or CPP on SR and SPL, supporting complementary roles for multimodal fusion and hierarchical planning.Removing TFM replaces Transformer fusion with concatenation, while removing CPP produces unstable navigation in the ablation study.

C. A Study on Ablation with Audio Enhancement

Audio enhancement improves TDGP’s generalization in unseen acoustic scenarios. On MP3D unseen data, the reported SR rises from 42.2% to 46.0%.

  • Audio Enhancement: Audio enhancement raises MP3D unseen-dataset SR from 42.2% to 46.0%, a 3.8% increase.The comparison is presented as evidence of robustness in noisy or unseen acoustic scenarios.
  • Audio Enhancement: The enhancement strategy is evaluated through ablations with and without complex scenarios, while navigation metrics vary with training steps.Table III concerns complex scenes used during training but not evaluation; Fig. 5 tracks metric changes during training.

D. Qualitative Analysis

Qualitative trajectories show TDGP guiding the agent toward targets on top-down maps, while training curves indicate faster convergence than SoundSpaces. The broader evaluation also reports effectiveness in path efficiency and unheard-acoustic generalization.

  • Qualitative Trajectories: TDGP trajectories effectively guide the agent to the target in qualitative top-down-map analyses.Figures 3 and 4 present representative navigation trajectories.
  • Training Dynamics: TDGP achieves higher SPL with fewer training steps than SoundSpaces, indicating faster convergence.The comparison is based on the training-step analysis shown in Fig. 5.
  • Overall Evaluation: Across Replica and Matterport3D, the evaluation reports effectiveness in path efficiency and generalization, especially in unheard acoustic scenarios.This conclusion summarizes the reported scope of the experiments rather than a specific qualitative trajectory.
Loading 2609.17421v1…