PaperlayerView exact revision on arXiv

Source-linked AI summary

AsyncVLA: An Asynchronous VLA for Fast and Robust Navigation on the Edge

Noriaki Hirose, Catherine Glossop, Dhruv Shah, Sergey Levine

arXiv:2602.13476v1cs.ROcs.LG

TL;DR

Large robotic foundation models offer rich vision-language understanding but impose computational and communication delays that challenge navigation. AsyncVLA pairs remote high-level VLA guidance with a reactive onboard Edge Adapter, and reports over 40% better performance than synchronous baselines in dynamic navigation, including with delays up to 6 seconds.

  • Problem

    Large robotic foundation models incur substantial computational cost and latency, while mobile navigation also faces dynamic obstacles, limited onboard compute, and remote communication delays.

    Source: §I · p. 1

  • Method

    AsyncVLA separates remote VLA semantic guidance from a lightweight onboard Edge Adapter and trains them end-to-end with reactive trajectories up-weighted.

    Source: §I · p. 2

  • Results

    AsyncVLA outperforms synchronous baselines by over 40% in dynamic navigation scenarios, with control delays fluctuating up to 6 seconds.

    Source: §I · p. 2

  • Takeaways & Limitations

    AsyncVLA maintains robust navigation in dynamic and partially observable indoor environments while compensating for network and inference latencies.

    Source: §VII · p. 8

    AsyncVLA depends on open-source base VLAs that can be finetuned end-to-end, and available navigation datasets contain few dynamic interactions.

    Source: §VII · p. 8

Abstract

from arXiv · show

Robotic foundation models achieve strong generalization by leveraging internet-scale vision-language representations, but their massive computational cost creates a fundamental bottleneck: high inference latency. In dynamic environments, this latency breaks the control loop, rendering powerful models unsafe for real-time deployment. We propose AsyncVLA, an asynchronous control framework that decouples semantic reasoning from reactive execution. Inspired by hierarchical control, AsyncVLA runs a large foundation model on a remote workstation to provide high-level guidance, while a lightweight, onboard Edge Adapter continuously refines actions at high frequency. To bridge the domain gap between these asynchronous streams, we introduce an end-to-end finetuning protocol and a trajectory re-weighting strategy that prioritizes dynamic interactions. We evaluate our approach on real-world vision-based navigation tasks with communication delays up to 6 seconds. AsyncVLA achieves a 40% higher success rate than state-of-the-art baselines, effectively bridging the gap between the semantic intelligence of large models and the reactivity required for edge robotics.

I. INTRODUCTION

AsyncVLA addresses the latency and deployment constraints of large robotic foundation models with an asynchronous large-model/ lightweight-policy design. It positions this approach against prior efficiency and hierarchical VLA work, emphasizing navigation with dynamic obstacles and network delays.

I. INTRODUCTION

  • Large VLA inference can lengthen control cycles, while mobile robots face partial views, dynamic obstacles, limited onboard compute, and additional remote-workstation communication latency.Remote inference can also constrain operational duration through power consumption when performed onboard.Source: §I · p. 1
  • AsyncVLA uses a lightweight inner-loop Edge Adapter, 100× smaller than the VLA, to rapidly incorporate recent observations while the large outer-loop model provides semantic guidance.Training up-weights dynamic interactions and first trains the adapter before end-to-end finetuning aligns both policies.Source: §I · p. 2
  • AsyncVLA outperforms synchronous baselines by over 40% in dynamic navigation scenarios, including under control delays fluctuating up to 6 seconds.The framework pairs remote high-level guidance with a reactive onboard policy.Source: §I · p. 2

II. RELATED WORK

  • Prior efficiency approaches reduce model size, quantize weights, accelerate action decoding, or add action heads, but do not decouple an onboard policy from an off-board VLA to address network delays of several seconds.The cited approaches chiefly address inference delays on the order of hundreds of milliseconds.Source: §II · p. 2
  • Prior fast-slow VLA work largely targets manipulation with delays up to about 300 ms and limited predictable motion, whereas AsyncVLA targets unpredictable navigation dynamics and network latency around 5 seconds.AsyncVLA exposes the adapter to stale VLA observations alongside current observations and up-weights dynamic-obstacle interactions.Source: §II · pp. 2–3

III. AN ASYNCHRONOUS VLA FOR NAVIGATION

AsyncVLA combines a large base VLA with a lightweight onboard Edge Adapter for reactive navigation. The adapter adjusts delayed VLA guidance using current observations to produce actions for the robot.

III. AN ASYNCHRONOUS VLA FOR NAVIGATION

  • AsyncVLA integrates a lightweight action head, reactive-data rebalancing, and end-to-end training to combine pretrained visual-language understanding with rapid control from recent observations.The system is designed to retain semantic understanding while enabling quick reactive control.Source: §III · p. 3

A. Edge Adapter Architecture

  • The Edge Adapter uses delayed action-token embeddings from the base VLA and current observations to adapt actions to the robot’s present environment.The base VLA runs on a workstation while the adapter runs on the robot’s onboard controller.Source: A. Edge Adapter Architecture · p. 3, Figure 2 · p. 3
  • The 76M-parameter Edge Adapter processes final-layer action features from the base VLA, whose observations may be delayed by fluctuating network and workstation latency.The token embeddings carry the base model’s semantic and visual information while keeping the adapter lightweight.Source: A. Edge Adapter Architecture · p. 3
  • The adapter’s ViT receives projected base-VLA action tokens plus embeddings for the current image and a six-channel image from the delayed observation.The images are resized to 96×96, and only the current-image output token feeds the action head, preventing delayed embeddings from directly determining the generated action.Source: A. Edge Adapter Architecture · pp. 3–4

B. Up-weighting Reactive Trajectories

AsyncVLA emphasizes training examples where actions change during a chunk, then trains the Edge Adapter and base VLA together to better align their asynchronous outputs. Its objective combines action imitation and smoothing.

B. Up-weighting Reactive Trajectories

  • Training up-weights sequences with sudden intra-chunk action changes, identified by comparing current and delayed action references and selecting samples whose final-pose distance exceeds a threshold.Such changes capture reactive behaviors that are relatively rare in the pretraining data.Source: B. Up-weighting Reactive Trajectories · p. 4

C. End-to-end Training

  • The base VLA supplies action-token embeddings conditioned on a goal, and the Edge Adapter maps those embeddings to an action chunk.Goals may be specified as an image, 2D pose, or language prompt; the experiments focus on pose- and language-conditioned cases.Source: C. End-to-end Training · p. 4
  • Training first initializes the base VLA from pretrained weights and trains the adapter and projector, then fine-tunes the entire policy end-to-end to align both components.The first stage keeps the base VLA parameters frozen; the second includes them in the trainable parameter set.Source: C. End-to-end Training · p. 4
  • The training objective sums an action-chunk imitation loss and an action-smoothing objective.The trainable parameter set expands to include the base VLA in the second training stage.Source: C. End-to-end Training · p. 4

IV. IMPLEMENTATION AND SYSTEM DESIGN

AsyncVLA builds on OmniVLA and compresses its action-token representations for an onboard Edge Adapter. A lightweight visual encoder and action head support low-latency action generation.

A. Network architecture

  • OmniVLA provides the base vision-based navigation policy, interpreting goal modalities with large-scale visual encoders and a language model.Its supported goal modalities include 2D goal poses, language instructions, and egocentric goal images.Source: A. Network architecture · p. 4
  • OmniVLA action-token embeddings encode action, semantic, and visual information, but their 8×4×4096 dimensions would increase onboard network size and computation.Source: A. Network architecture · pp. 4–5
  • A token projector with two MLP ResNet blocks compresses each action token from 4×4096 to 1024 dimensions before Edge Adapter input.The compression also enables efficient transmission of token embeddings from the workstation to the onboard controller.Source: A. Network architecture · p. 5
  • The Edge Adapter uses EfficientNet-B0 features from 96×96 images and a four-layer MLP action head to predict an N-step action chunk.Source: A. Network architecture · p. 5

B. Training dataset

AsyncVLA is trained on a mixture of navigation datasets, with sampling adjustments intended to emphasize reactive behavior. During inference, the remote base VLA and onboard Edge Adapter operate asynchronously, with buffered observations used to align delayed guidance.

B. Training dataset

  • Training combines GNM, LeLaN, and SACSoN, which respectively aggregate public robot datasets, include in-the-wild language-conditioned data, and capture pedestrian scenes.Source: B. Training dataset · p. 5

C. Training details

  • Training uniformly samples the three datasets, up-weights half of each batch using a 1.0 m threshold, and prioritizes pedestrian samples from SACSoN.For OmniVLA-initialized AsyncVLA, five H200 GPUs use an effective batch size of 180 with LoRA training approximately 5% of model parameters; other hyperparameters follow OmniVLA defaults.Source: C. Training details · p. 5

D. System integration for inference

  • At inference, the workstation runs the base VLA while the onboard controller runs the Edge Adapter; images travel to the workstation and returned action-token embeddings include timestamps.Source: D. System integration for inference · p. 5
  • The controller matches returned embeddings to buffered past images by timestamp, then uses a PD controller to turn generated 2D poses into velocity commands.Source: D. System integration for inference · p. 5

V. EXPERIMENTAL SETUP

The evaluation covers real-world goal-pose and language-conditioned navigation, including varied environments and language commands outside the training distribution. Experiments also include pedestrian interactions and measure workstation-to-robot delays across environments.

A. Navigation tasks

  • Goal-pose trials direct the robot to targets 12–30 meters away in indoor and outdoor settings; language-conditioned trials use diverse instructions and targets 5–20 meters away.Source: A. Navigation tasks · pp. 5–6
  • Evaluation uses ten challenging, cluttered environments with two trials per environment, including pedestrian interactions in at least one trial each.Pedestrian interactions are kept consistent across methods for fair, reproducible comparisons.Source: A. Navigation tasks · p. 5
  • The experimental setup places the Edge Adapter on the robot and the base VLA on a remote workstation; Fig. 4 measures delays across four environments using distinct colors.Source: Figure 3 · p. 6, Figure 4 · p. 6
  • Half of the language-command trials use out-of-distribution prompts specifying how to navigate toward a goal, and behavior is assessed for instruction compliance.Source: A. Navigation tasks · p. 6

B. Mobile Robot Platforms

Experiments use the Vizbot mobile robot, with onboard Edge Adapter control and a remote workstation running the base VLA. The study measures variable WiFi delay between image transmission and return of action-token embeddings.

B. Mobile Robot Platforms

  • Vizbot carries fisheye and tracking cameras plus wheel-velocity sensors; its external tracking camera estimates robot pose and goal location for pose-conditioned navigation.Source: B. Mobile Robot Platforms · p. 6
  • The Edge Adapter runs on a 30 W Jetson Orin onboard, while the base VLA runs on an RTX 4090 workstation and sends token embeddings over WiFi.Source: B. Mobile Robot Platforms · p. 6
  • Measured image-to-embedding delay varies from 0.28 to 6.0 seconds across the experimental environments.Source: B. Mobile Robot Platforms · p. 6

C. Baselines

The evaluation compares AsyncVLA with lightweight and large-model baselines, then tests whether end-to-end training matters and whether navigation remains robust under large delays.

C. Baselines

  • The baselines include OmniVLA, OmniVLA-edge, an Edge Adapter trained without end-to-end finetuning, and AsyncVLA kept entirely on the workstation.The workstation comparison isolates the effect of network latency; the no-E2E baseline keeps the foundation model frozen while training the Edge Adapter.Source: C. Baselines · pp. 6–7
  • OmniVLA-edge is a 108M-parameter onboard policy running at 6 Hz, while 8.26B-parameter OmniVLA runs on a workstation at up to 5 Hz.OmniVLA is also the pretrained VLA used in AsyncVLA; its generated action pose chunks are followed at 10 Hz over 8 steps.Source: C. Baselines · p. 6

VI. EVALUATION

  • The evaluation asks whether AsyncVLA beats latency-unaware VLA baselines, needs end-to-end training, and remains robust to large navigation delays.The first two questions are addressed through baseline comparisons; the delay question is evaluated separately.Source: §VI · p. 7

A. AsyncVLA vs. baselines

AsyncVLA outperforms the reported baselines in pose-conditioned navigation and preserves language-following performance. Its advantage over OmniVLA grows as artificial workstation latency increases, while it remains robust to delays up to 6 seconds.

A. AsyncVLA vs. baselines

  • Without end-to-end training, the Edge Adapter fails to learn a good mapping from the base VLA, and overall performance drops relative to AsyncVLA..Source: A. AsyncVLA vs. baselines · p. 7
  • At a maximum speed of 0.3 m/s, AsyncVLA yields to the faster pedestrian and safely resumes its goal path, while workstation OmniVLA collides.Source: Figure 5 · p. 7, A. AsyncVLA vs. baselines · p. 7
  • AsyncVLA follows out-of-distribution language instructions comparably to OmniVLA and substantially better than OmniVLA-edge.Source: Figure 6 · p. 7, A. AsyncVLA vs. baselines · p. 7
  • AsyncVLA maintains robust navigation across modalities despite WiFi delays of up to 6 seconds, guiding the robot to goals without collision.The Edge Adapter uses recent observations while retaining high-level guidance from the base VLA.Source: A. AsyncVLA vs. baselines · p. 7
  • 85% of trials reached specified goals without collisions, a 40% advantage over state-of-the-art baselines; those baselines averaged 15% more time to reach the goal.The cluttered indoor trials included pedestrians actively interfering with the robot; the comparison table defines success rate as reaching the goal without collisions.Source: A. AsyncVLA vs. baselines · p. 7, Table I · p. 8

B. Quantitative analysis for time delay

  • At 0.2, 2.0, and 5.0 seconds of artificial workstation latency, AsyncVLA's advantage over OmniVLA widens; Fig. 7 measures success, goal time, and static collisions.Fig. 8 shows AsyncVLA maintaining similar goal-directed trajectories while OmniVLA deviates and collides with static obstacles.Source: B. Quantitative analysis for time delay · pp. 7–8, Figure 7 · p. 8

VII. DISCUSSION AND FUTURE WORK

AsyncVLA combines a pretrained VLA for high-level visual and semantic understanding with an onboard Edge Adapter for adaptive action inference. It reports stronger navigation performance than baselines and identifies constraints on model availability and dynamic-interaction training data.

VII. DISCUSSION AND FUTURE WORK

  • The framework pairs a pretrained VLA's visual and semantic understanding with an onboard Edge Adapter for fast, adaptive action inference.It is designed for static and dynamic obstacles in partially observable indoor environments and to compensate for network and inference latency.Source: §VII · p. 8
  • AsyncVLA outperforms OmniVLA and OmniVLA-edge in collision avoidance, language-instruction adherence, and goal-reaching efficiency, including under latency up to 5 seconds.It also generalizes to out-of-distribution language instructions.Source: §VII · p. 8
  • AsyncVLA is limited to base VLAs with open-source weights that can be finetuned end-to-end, and available navigation datasets contain few dynamic interactions.Future work includes exploring other sources of dynamic interactions, such as human video.Source: §VII · pp. 8–9

APPENDIX

The appendix describes the training objective and asynchronous inference pipeline. Training combines action imitation and regularization, while timestamped observations and waypoint tracking support onboard control.

A. Training Objective

  • The objective sums imitation and action-smoothness losses, with imitation supervision covering local poses and relative pose changes for rapid adjustment.The local pose teaches coarse movement direction; relative-pose supervision encourages rapid responses to dynamic changes.Source: A. Training Objective · p. 12
  • For LeLaN, which lacks raw action references, training penalizes only the final generated action to keep it near the language-specified target pose.The action-delta objective regularizes changes across the generated actions.Source: A. Training Objective · p. 12

B. Inference details

  • The robot timestamps captured images, and the workstation returns action-token embeddings with the original timestamp so the edge controller can retrieve the matching observation.A buffer stores observed images and timestamps for matching.Source: B. Inference details · p. 12
  • A PD controller follows the waypoint trajectory and publishes velocity commands at 10.0 Hz, with baseline trajectories updated when new outputs arrive.The controller reaches a future waypoint and derives linear and angular velocity commands.Source: B. Inference details · p. 12

C. Evaluation details

The evaluation covers pose-conditioned navigation in varied environments, including pedestrian interactions, and defines success by collision-free arrival within a time limit. Results include low-frequency goal-reaching and qualitative action adjustments around pedestrians.

C. Evaluation details

  • The pose-conditioned evaluation used ten challenging environments, two trials per environment, and pedestrian interactions in one trial; success required collision-free arrival within 1 meter in 120 seconds.The same pedestrian interactions were used across methods for comparison.Source: C. Evaluation details · p. 12

D. Breakdown of quantitative analysis

E. Visualization of action chunk from AsyncVLA

  • Across three visualized scenes, the Edge Adapter deviated from delayed-observation action chunks to move toward goals and avoid pedestrians visible in current observations.The base VLA's observation may not include pedestrians that appear in the Edge Adapter's current observation.Source: E. Visualization of action chunk from… · p. 13, Figure 9 · p. 13
Loading 2602.13476v1…