PaperlayerView exact revision on arXiv

Source-linked AI summary

ADAPT: Audio-Depth Action Planning with Transition-Aware Rewards for Lights-Out Industrial Navigation

Wenjun Huang, Shenghao Fu, Yian Jin, Yang Ni, Yifan Li, Lichi Li, Xingjian Diao, Pengfei Zhang, Yutong Song, Hanning Chen, Jiarui Cui, Tianyu Yang, Shilong Li, Yu Kong, Mohsen Imani

arXiv:2610.15956v1cs.AI

TL;DR

RGB-based navigation depends on illumination, leaving VLA agents poorly suited to lights-out industrial settings; spatial audio and depth provide complementary target-direction and geometry cues. ADAPT combines these modalities and uses TARS for action-aware reward shaping, and it consistently outperforms baselines on seen and unseen scenes.

  • Problem

    RGB-based VLA navigation assumes sufficient illumination, while lights-out factories may operate with little or none; spatial audio and depth are complementary, but their joint use for VLA navigation remains largely unexplored.

    Source: §1 · pp. 1–2

  • Method

    ADAPT adapts an LMM to jointly use spatial audio and depth for discrete navigation actions, with TARS measuring remaining executable actions over a position–heading graph.

    Source: §1 · p. 2

  • Results

    ADAPT consistently outperforms the baselines across seen and unseen scenes.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The results demonstrate the potential of spatial audio and depth for reliable RGB-free navigation in lights-out industrial environments.

    Source: §7 · p. 8

    The study evaluates ADAPT in controlled, high-fidelity simulation rather than on a physical robot, which cannot capture all physical microphone and depth-sensor effects.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Vision-language-action models for embodied navigation typically rely on RGB observations, limiting their applicability in lights-out industrial environments where visible illumination may be unavailable. We study illumination-independent navigation using spatial audio for target localization and depth for obstacle-aware geometry. To support this setting, we introduce DarkNav-Bench, a benchmark with 24 industrial scenes and four progressively challenging acoustic conditions. We further propose ADAPT, an audio-depth action-planning framework equipped with spatial-audio perception and optimized for closed-loop navigation through reinforcement learning (RL). Our key RL contribution, transition-aware reward shaping (TARS), defines a pose-aware shortest-action potential over a position-heading graph. Unlike conventional geodesic-progress rewards, TARS accounts for both feasible movements and in-place rotations, providing action-consistent feedback for policy optimization. Experiments show that ADAPT consistently outperforms training-based navigation baselines and zero-shot multimodal models on both seen and unseen scenes. On unseen environments, it achieves 90.6% success under Clean conditions and retains 60.0% success under the most challenging Localized Noise setting. These results demonstrate the potential of spatial audio and depth as complementary modalities for reliable RGB-free navigation in lights-out industrial environments.

1 Introduction

RGB dependence limits VLA navigation in lights-out factories, while spatial audio and depth offer complementary directional and geometric cues. The paper introduces DarkNav-Bench and ADAPT, including TARS for action-consistent reward shaping.

1 Introduction

  • RGB-based VLA agents assume sufficient, consistent illumination, a constraint that conflicts with lights-out factories and can make these agents unreliable or unusable.Maintaining lighting for robotic perception adds energy consumption, while reducing illumination degrades RGB-based navigation.Source: §1 · p. 1
  • Spatial audio indicates target direction, while depth captures local geometry; their joint use for VLA navigation remains largely unexplored.Source: §1 · p. 2
  • DarkNav-Bench evaluates spatial-audio-and-depth navigation in lights-out industrial scenes under progressively challenging acoustic conditions.ADAPT uses a lightweight spatial-audio adapter and trains the model to reason jointly over audio and depth for discrete navigation actions.Source: §1 · p. 2
  • TARS defines a pose-aware potential over a position–heading action graph, measuring remaining executable actions so rotations receive feedback based on their effect on navigation progress.Unlike geodesic progress, it accounts for both forward movement and in-place rotations.Source: §1 · p. 2

2 Task Formulation

The task is sequential navigation to an acoustic target while avoiding obstacles, using spatial audio and egocentric depth without RGB or visible illumination. The policy chooses among discrete movement, rotation, and stopping actions from its available interaction context.

2 Task Formulation

  • The agent navigates toward an active acoustic target while avoiding obstacles, using spatial audio and egocentric depth but no RGB observations or visible illumination.Simulator state includes robot pose, target location, and environment geometry, but is not exposed to the policy.Source: §2 · pp. 2–3
  • Four-channel FOA audio provides target-direction cues, while egocentric depth captures local geometry for collision-aware motion.Source: §2 · pp. 2–3
  • The discrete action space includes moving forward, turning left, turning right, and stopping; policy decisions use current observations and may include past actions.Source: §2 · p. 2
  • An episode ends when the agent reaches the target and stops or the maximum navigation horizon is reached.Success is determined by whether target distance meets the stated threshold.Source: §2 · p. 3

3 Benchmark Construction

DarkNav-Bench adapts industrial simulation environments for RGB-free navigation using spatial audio and depth. Its 24 scenes and cumulative acoustic settings support evaluation across increasing interference and unseen layouts.

3 Benchmark Construction

  • The benchmark retains IndustryNav’s cluttered layouts, dynamic agents, collision-aware interaction, and egocentric depth, replacing visual navigation with an audio-depth task.RGB may be rendered for visualization and debugging but is never provided to the agent.Source: §3 · p. 3
  • A stationary acoustic emitter continuously plays a chirp that is spatialized relative to the agent, providing an evolving directional cue as the agent moves and rotates.Source: §3 · p. 3
  • Four cumulative settings progress from Clean to Propagation, Diffuse Noise, and Localized Noise, adding geometry-dependent effects, diffuse background noise, and localized sound sources.Localized sources include machinery and sounds from moving robots or equipment, creating spatially structured distractors.Source: §3 · p. 3
  • Diffuse background noise uses real machine recordings, with interference strength controlled by signal-to-noise ratio; lower SNR means stronger interference.Source: §3 · p. 3
  • DarkNav-Bench has 24 industrial scenes: 10 are used as seen scenes for training data collection and 14 remain unseen for scene-level generalization tests.The sampling protocol uses 12,000 training episodes, 3,000 seen-scene test episodes, and 4,200 unseen-scene test episodes across all four difficulty levels.Source: §3 · pp. 3–4

4 Method

ADAPT uses a three-stage pipeline: spatial-audio adaptation, multimodal supervised fine-tuning, and PPO reinforcement learning for closed-loop navigation.

4.1 Spatial Audio Adaptation

  • Spatial-audio adaptation adds an SAA and auxiliary direction-regression head to address audio encoders that are not optimized for directional cues.The head provides explicit direction supervision during adaptation and is discarded afterward; the trained SAA is retained for navigation training.Source: §4.1 · p. 4

4.2 Multimodal Supervised Fine-Tuning

  • Multimodal supervised fine-tuning initializes the policy from expert demonstrations by predicting each expert action as a short text response.The teacher-forced loss applies only to action-answer tokens.Source: §4.2 · pp. 4–5

4.3 Reinforcement Learning with Transition-Aware Reward Shaping

  • PPO fine-tunes the SFT policy for closed-loop behavior, using a backbone-shared actor–critic with a gradient-decoupled critic.The actor reuses the multimodal backbone; the critic estimates state value from a stop-gradient representation.Source: §4.3 · p. 5
  • TARS shapes rewards with a pose-aware action-distance potential over a position–heading graph, accounting for feasible movements and orientation-dependent costs.Its potential measures the minimum executable actions remaining to reach the goal, so it distinguishes helpful from counterproductive in-place rotations.Source: §4.3 · p. 5

4.4 Bounded Action History

  • ADAPT can condition decisions on the eight most recent actions, serialized in the prompt without adding trainable parameters.The same action-history interface is used during training and inference.Source: §4.4 · p. 5

5 Experimental Results

ADAPT outperforms training-based baselines and non-oracle zero-shot LMMs across seen and unseen scenes, with goal-directed trajectories. Ablations and acoustic analyses examine the roles of its components and performance under challenging conditions.

5 Experimental Results

  • Evaluation uses SR, SPL, SNA, and FDG; zero-shot baselines without oracle information receive a relaxed stopping protocol favoring their success registration.The protocol registers success automatically upon entering the goal radius.Source: §5 · p. 6

5.1 Main Results

  • ADAPT consistently outperforms training-based baselines and non-oracle zero-shot LMMs across seen and unseen scenes.In Figure 3, ADAPT reaches the target while competing methods stagnate or deviate from effective routes.Source: §5.1 · pp. 6–7, Figure 3 · p. 7

5.2 Discussion

  • Table 3 ablations show complementary contributions from SAA and TARS, with the largest TARS gains when action history is absent.Short action-history conditioning further improves robustness without an additional memory module.Source: §5.2 · p. 7, Table 3 · p. 7
  • Under acoustic difficulties, DoA estimation varies non-monotonically: Prop performs best overall, while stronger DN interference degrades fine-grained estimation.Median error near 10° alongside MAE above 25° indicates a heavy-tailed pattern; the analysis fixes SNR to isolate noise strength.Source: §5.2 · pp. 7–8, Table 4 · p. 8
  • Under DN, navigation performance degrades as diffuse-noise level increases, with severe low-SNR conditions limiting robustness.SR, SPL, and SNA decline while FDG increases as SNR falls.Source: §5.2 · p. 8, Table 5 · p. 8

6 Related Work

Prior navigation and VLA work often depends on RGB, including audio-navigation approaches that use sound alongside vision. ADAPT instead combines spatial audio and depth for lights-out action planning.

Vision-Language-Action Models

  • Recent VLA models generate actions from multimodal observations, but predominantly rely on RGB and focus on visually grounded manipulation.ADAPT extends this action-generation paradigm to audio-depth perception when reliable visual observations are unavailable.Source: §6 · p. 8

Audio-Visual Navigation; Acoustic-Guided Navigation and Active Acoustic Sensing

  • Audio-visual navigation and audio-goal methods use spatial audio but generally retain vision as a primary perceptual modality.ADAPT combines spatial acoustic guidance for the goal with depth observations for obstacle geometry in lights-out settings.Source: §6 · p. 8

Acoustic-Guided Navigation and Active Acoustic Sensing

  • Active acoustic sensing such as VisualEchoes uses emitted chirps and echoes to infer spatial properties, unlike ADAPT’s audio-goal and depth-based planning.Source: §6 · p. 8

7 Conclusion

The conclusion presents DarkNav-Bench and ADAPT as an audio-depth approach to lights-out industrial navigation, combining spatial-audio adaptation with TARS for closed-loop progress. It reports consistent gains over baselines on seen and unseen scenes and identifies SAA and TARS as beneficial in ablations.

  • DarkNav-Bench and ADAPT target audio-depth navigation in lights-out industrial environments.Source: §7 · p. 8
  • ADAPT consistently outperforms baselines across seen and unseen scenes, while ablations confirm benefits from SAA and TARS.Source: §7 · p. 8
  • The framework combines spatial-audio adaptation with TARS, which measures navigation progress over a position–heading action graph.Source: §7 · p. 8

Limitations

The study evaluates ADAPT in high-fidelity simulation rather than on physical robots, and the benchmark uses controlled acoustic signals. Its training also assumes access to privileged simulator information, although that information is withheld from the deployed policy.

  • Evaluation is simulation-only, which cannot capture all physical microphone and depth-sensor effects; physical-robot evaluation remains future work.Source: Limitations · p. 9
  • The benchmark uses the same periodic target chirp and representative equipment loops, unlike real settings that may include intermittent targets, unfamiliar signals, and similar-sounding distractors.Source: Limitations · p. 9
  • Expert demonstrations and TARS use privileged simulator information during training, although the deployed policy never receives those variables.Source: Limitations · p. 9

Ethical Considerations

The paper frames audio-depth navigation as a simulation-based research framework, not a deployment-ready or safety-certified system. Its task formulation keeps privileged simulator state separate from the policy, which acts from spatial audio and depth observations.

Ethical Considerations

  • Navigation results in simulation do not guarantee safe factory operation; physical deployment requires extensive validation and privacy protections for potentially sensitive acoustic data.Source: Ethical Considerations · p. 9

A Task Formulation Details

  • The simulator state includes robot position and heading, target position, and surrounding geometry, but the policy does not directly observe these privileged variables.Source: A Task Formulation Details · p. 12
  • The policy's spatial-audio input is four-channel first-order ambisonics, while depth provides distances to visible surfaces and local traversability cues.Source: A Task Formulation Details · p. 12
  • The task requires approaching the acoustic target while avoiding collisions and unnecessary actions, and an episode ends on a successful stop or at the action limit.Source: A Task Formulation Details · p. 12

B Benchmark Construction Details

DarkNav-Bench extends industrial Unity environments with spatial audio and acoustic scene modeling while keeping RGB unavailable to the agent. It uses discrete motion primitives and layered acoustic conditions, and its shared actor–critic architecture reuses multimodal representations while separating value-learning gradients from the policy backbone.

B Benchmark Construction Details

  • DarkNav-Bench extends IndustryNav with spatial audio and acoustic scene modeling while retaining its industrial layouts, dynamics, action space, and depth sensing; RGB is rendered only for visualization and debugging.Source: B Benchmark Construction Details · p. 13

Navigation Dynamics and Action Geometry

C.1 Backbone-Shared Actor–Critic

  • The actor and critic reuse one multimodal backbone representation, but value-loss gradients are detached from the shared backbone to retain computational efficiency without critic-driven representation updates.Source: §C.1 · pp. 17–18

C.2 Action-Space Potential Construction

TARS computes shortest remaining action counts on a position–heading graph, giving orientation-sensitive reward feedback that geodesic progress misses. PPO optimization combines policy and value objectives with exploration and SFT anchoring.

C.2 Action-Space Potential Construction

  • TARS builds a position–heading state graph with collision-free forward edges and unit-cost forward, left, and right actions, then uses reverse breadth-first search to compute minimum actions to the goal.Goal-region states start at N(s) = 0, irrespective of orientation; the graph uses 20 heading bins with 18° rotations.Source: §C.2 · p. 18
  • For continuous simulator states, TARS looks up the minimum action count among nearby traversable grid states with the same discretized heading, falling back to the nearest traversable location.The action distance and occupancy map are used only to construct training reward and are not given to the policy.Source: §C.2 · p. 18

C.3 PPO Optimization

  • PPO uses generalized advantage estimation, a clipped policy objective, a clipped value objective, and an entropy bonus to encourage exploration.The advantage estimate bootstraps at nonterminal transitions and is masked at terminal transitions.Source: §C.3 · p. 19
  • SFT anchoring regularizes PPO toward the frozen SFT policy over the restricted navigation action set, while allowing policy preferences and representations to adapt to closed-loop experience.The KL constraint strength is controlled by β.Source: §C.3 · pp. 19–20

D Evaluation Metrics

The evaluation metrics separate task completion, geometric route efficiency, discrete-action efficiency, and final proximity. The training passages describe spatial-audio adaptation, multimodal supervised fine-tuning, closed-loop PPO fine-tuning, and action-history context.

D Evaluation Metrics

  • SR, SPL, SNA, and FDG jointly assess task completion, geometric path efficiency, discrete-action efficiency, and final distance to the goal.The paper argues that a strong policy should perform well across these measures rather than optimize just one.Source: D Evaluation Metrics · pp. 20–21

E.1 Spatial-Audio Adaptation

  • Spatial-audio adaptation trains the proposed SAA and an auxiliary direction-prediction head while freezing the pretrained audio and vision encoders, projectors, and base language model.The auxiliary head supplies directional supervision during adaptation and is then discarded; the trained SAA is retained.Source: §E.1 · p. 21

E.2 Multimodal Supervised Fine-Tuning

  • Multimodal SFT predicts navigation actions from spatial audio and egocentric depth, further optimizes SAA, and adds LoRA adapters to the LMM decoder while keeping the pretrained towers and base weights frozen.LoRA is applied to the decoder’s self-attention projections.Source: §E.2 · p. 21

E.3 PPO Fine-Tuning

  • PPO fine-tuning starts from the SFT checkpoint and uses closed-loop Unity interaction to optimize long-horizon navigation under the RL reward.Training first warms up the critic with the actor frozen, then jointly updates actor and critic; the procedure retains demonstrated behavior while adapting to policy-induced states.Source: §E.3 · pp. 22–23

E.4 Action-History Context

  • Action-history prompts contain the eight most recent actions in chronological order, with unavailable entries represented by none at episode start.History contains actions only, not previous audio, depth, collision outcomes, or privileged state.Source: §E.4 · p. 23

F Zero-Shot Multimodal Baselines

This section evaluates general-purpose multimodal models in observation-only and oracle-assisted zero-shot settings, contrasting direct audio-depth navigation with action reasoning using privileged target information. It presents navigation comparisons across ADAPT, training-based baselines, and Qwen Omni variants, reporting step counts, final distance to goal, and path length.

Zero-Shot without Oracle Information

  • Observation-only zero-shot receives current FOA audio, depth, and eight recent commanded actions, but no prior observations or ground-truth navigation state.The action history is only a control prior: it records commands, not outcomes, and is intended to reduce repeated turns or left–right oscillation.Source: F Zero-Shot Multimodal Baselines · p. 23
  • Observation-only agents choose forward, left, or right without stop, and succeed upon entering the target region, a relaxed protocol that isolates target-direction recovery and obstacle-aware navigation.This favorable diagnostic does not require inferring when to stop from acoustic amplitude.Source: F Zero-Shot Multimodal Baselines · p. 23

Zero-Shot with Oracle Information

  • Oracle-assisted zero-shot supplies ground-truth target bearing and distance, approximately isolating action reasoning, but is diagnostic rather than directly comparable because ADAPT does not receive that privileged information.The gap between oracle-assisted and observation-only performance is intended to indicate whether zero-shot performance is more limited by spatial perception or downstream action reasoning.Source: F Zero-Shot Multimodal Baselines · pp. 23–24

Closed-Loop Evaluation

  • Both settings are evaluated closed-loop: each single predicted action is executed before the next observation, with no future observations, transition outcomes, or privileged simulator state revealed.The evaluation uses the same navigation dynamics and maximum episode horizon as the other methods.Source: F Zero-Shot Multimodal Baselines · p. 24

Evaluated Models

  • Qwen2.5-Omni and Qwen3-Omni receive FOA channels as four separate mono inputs in fixed ACN/N3D order, alongside egocentric depth, under a common prompt and action-output protocol.The FOA channels are not downmixed to mono or converted to binaural audio; decoding and response processing are identical, with no model-specific navigation heuristics or post-processing.Source: F Zero-Shot Multimodal Baselines · pp. 23–24

F Zero-Shot Multimodal Baselines

  • ADAPT recorded 41 steps, FDG: 0.5 m, path: 33.2 m, versus MAGNet’s 100 steps, FDG: 1.1 m, path: 32.7 m and SoundSpaces’ 86 steps, FDG: 1.4 m, path: 69.4 m.SAVi recorded 100 steps, FDG: 10.8 m, path: 35.9 m.Source: F Zero-Shot Multimodal Baselines · p. 28
  • Qwen 2.5-Omni without oracle information recorded 100 steps, FDG: 31.4 m, path: 2.6 m; with oracle information it recorded 100 steps, FDG: 31.6 m, path: 0.0 m.Qwen 3-Omni with oracle information recorded 100 steps, FDG: 22.1 m, path: 11.7 m; without oracle information it recorded 100 steps, FDG: 24.5 m, path: 12.1 m.Source: F Zero-Shot Multimodal Baselines · p. 28
  • ADAPT recorded 38 steps, FDG: 0.8 m, path: 27.0 m; MAGNet, SAVi, and SoundSpaces recorded FDG values of 25.8 m, 20.0 m, and 12.6 m, respectively.Their reported steps and path lengths were MAGNet: 100 steps, path: 37.4 m; SAVi: 100 steps, path: 18.1 m; SoundSpaces: 100 steps, path: 73.4 m.Source: F Zero-Shot Multimodal Baselines · p. 28
  • Qwen 3-Omni with oracle information recorded 34 steps, FDG: 1.0 m, path: 28.5 m; without oracle information it recorded 100 steps, FDG: 29.0 m, path: 8.0 m.In the same displayed comparison, Qwen 2.5-Omni without oracle information recorded 100 steps, FDG: 29.1 m, path: 0.0 m; with oracle information it recorded 100 steps, FDG: 27.7 m, path: 1.5 m.Source: F Zero-Shot Multimodal Baselines · p. 28
  • ADAPT recorded 76 steps, FDG: 1.0 m, path: 43.2 m; MAGNet recorded 100 steps, FDG: 28.5 m, path: 38.9 m, while SAVi recorded 60 steps, FDG: 0.4 m, path: 39.3 m.SoundSpaces recorded 100 steps, FDG: 22.0 m, path: 30.0 m.Source: F Zero-Shot Multimodal Baselines · p. 28
  • ADAPT recorded 41 steps, FDG: 1.4 m, path: 30.1 m; MAGNet recorded 85 steps, FDG: 0.8 m, path: 46.6 m, and SoundSpaces recorded 100 steps, FDG: 8.9 m, path: 57.0 m.SAVi recorded 100 steps, FDG: 15.3 m, path: 51.8 m.Source: F Zero-Shot Multimodal Baselines · p. 28

Related papers

Loading 2610.15956v1…