PaperlayerView exact revision on arXiv

Source-linked AI summary

TIC-VLA: A Think-in-Control Vision-Language-Action Model for Robot Navigation in Dynamic Environments

Zhiyu Huang, Yun Zhang, Johnson Liu, Rui Song, Chen Tang, Jiaqi Ma

arXiv:2602.02459v2cs.RO

TL;DR

Robots must follow language instructions while controlling continuously in dynamic environments, but VLA systems often assume temporally aligned reasoning and action. TIC-VLA models delayed semantic states and latency explicitly, trains under asynchronous delays, and reports robust navigation in simulation and real-world deployment. The paper also introduces DynaNav for realistic evaluation.

  • Problem

    VLA navigation models often assume temporally aligned reasoning and control, despite delayed VLM inference and continuous real-time control requirements.

  • Method

    TIC-VLA uses a delayed semantic-control interface and latency-consistent imitation-learning and reinforcement-learning training to model asynchronous reasoning during action generation.

  • Results

    TIC-VLA demonstrates robust navigation under realistic inference latency and consistent improvements over prior navigation and language-guided baselines in simulation and real-world settings.

  • Takeaways & Limitations

    TIC-VLA advances real-time language-guided robot navigation by explicitly addressing inference latency and limited onboard computation.

  • Takeaways & Limitations

    The system is not fully optimized for runtime efficiency, real-world evaluation remains limited in scale, and extension beyond navigation is future work.

Abstract

from arXiv · show

Robots in dynamic, human-centric environments must follow language instructions while maintaining real-time reactive control. Vision-language-action (VLA) models offer a promising framework, but they assume temporally aligned reasoning and control, despite semantic inference being inherently delayed relative to real-time action. We introduce Think-in-Control (TIC)-VLA, a latency-aware framework that explicitly models delayed semantic reasoning during action generation. TIC-VLA defines a delayed semantic-control interface that conditions action generation on delayed vision-language semantic states and explicit latency metadata, in addition to current observations, enabling policies to compensate for asynchronous reasoning. We further propose a latency-consistent training pipeline that injects reasoning inference delays during imitation learning and online reinforcement learning, aligning training with asynchronous deployment. To support realistic evaluation, we present DynaNav, a physics-accurate, photo-realistic simulation suite for language-guided navigation in dynamic environments. Extensive experiments in simulation and on a real robot show that TIC-VLA consistently outperforms prior VLA models while maintaining robust real-time control under multi-second reasoning latency. Project website: https://ucla-mobility.github.io/TIC-VLA/

1. Introduction

TIC-VLA addresses the temporal mismatch between delayed vision-language reasoning and continuous real-time control in dynamic environments. It exposes semantic staleness to the controller, trains under realistic delays, and evaluates the resulting system in simulation and real-world navigation.

  • Motivation: Existing VLA systems often assume reasoning and control are temporally aligned, although VLM inference can take seconds while control runs at tens of hertz.This mismatch is especially pronounced on mobile robots with limited computation.
  • Motivation: Prior navigation systems either abstract away embodiment and timing or pause execution during reasoning, while asynchronous designs may still treat semantic outputs as temporally fresh.These approaches do not explicitly account for inference latency in dynamic environments.
  • TIC-VLA: TIC-VLA exposes delayed semantic representations, latency metadata, and ego-motion offsets to the real-time action policy through a delayed semantic-control interface.The interface distinguishes TIC-VLA from systems that merely separate slow reasoning from fast control.
  • Training: The latency-consistent training pipeline injects inference delays during imitation learning and reinforcement learning to align policy learning with asynchronous deployment.This training strategy targets robust navigation under variable delays.
  • Evaluation: DynaNav provides realistic rendering, dynamic human agents, physics-based execution, and diverse indoor and outdoor scenarios for data collection, online reinforcement learning, and benchmarking.The suite supports realistic and reproducible evaluation of language-guided navigation.
  • Results: Experiments in simulation and real-world deployment demonstrate robust navigation under realistic inference latency, with consistent improvements over prior navigation and language-guided baselines.The paper reports evaluation in both simulated and real-world environments.

2. Related Work

Prior work advances learning-based, vision-language, and dynamic-environment navigation, but existing VLA navigation methods generally underrepresent inference latency. TIC-VLA instead combines asynchronous reasoning and control with a physics-accurate benchmark for language-guided navigation in human-centric environments.

  • Learning-based Visual Navigation: Learning-based visual navigation increasingly uses diffusion policies, imitation and reinforcement learning, and world modeling instead of traditional map-based pipelines.These approaches target navigation in complex environments without relying on maps.
  • VLA for Navigation: VLA navigation methods span direct action prediction, intermediate linguistic actions, and language-informed trajectory planning.Examples include NaVid, NaVILA, and TrackVLA.
  • VLA for Navigation: StreamVLN, MobileVLA, and dual-system approaches address temporal reasoning and real-time execution through asynchronous inference.These systems balance deliberative reasoning with reactive control.
  • VLA for Navigation: Most existing VLA navigation models implicitly assume negligible inference latency, rely on powerful GPUs, or block execution during reasoning inference.TIC-VLA differs by explicitly modeling latency as part of control.
  • Navigation in Dynamic Environments: Dynamic-environment navigation research incorporates social cues, human intent, and socially compliant behavior into visual or lightweight policy representations.Representative systems include Social-LLaVA, Narrate2Nav, and Vi-LAD.
  • Navigation in Dynamic Environments: DynaNav provides a physics-accurate simulation suite for language-guided navigation in human-centric environments.It complements prior work by emphasizing realistic dynamic-environment evaluation.

3. Method

TIC-VLA treats delayed semantic reasoning as part of real-time navigation, coupling cached semantic states and latency metadata with current observations. Its asynchronous policy and three-stage latency-consistent training pipeline support action generation under variable reasoning delays.

  • Think-in-Control VLA: TIC-VLA models inference delay explicitly, conditioning control on delayed semantic representations, latency metadata, ego-motion, current visual observations, and robot state.The delayed semantic-control interface lets the policy reinterpret semantic information produced from an earlier world state.
  • VLM Semantic Reasoning: The VLM reasons over observations anchored at t −∆t, producing scene understanding, object identification, intent prediction, and future waypoints relative to inference start.Effective latency combines VLM inference time and elapsed time since the last completed reasoning update.
  • Latency-Aware Action Policy: The action policy combines visual and cached semantic features through cross-attention Transformer layers, while state and latency metadata receive learned positional encodings.The resulting action-query representation is mapped through an MLP to generate action outputs.
  • Latency-Aware Action Policy: At each control timestep, the high-frequency action policy uses current observations and the most recent cached VLM state while asynchronous inference proceeds.The policy outputs a short horizon of continuous actions that are integrated into a trajectory and used to select a target point.
  • Latency-Consistent Training Pipeline: During reinforcement learning, a value network receives current image tokens, goal position, and robot state, while rewards include target reaching, goal progress, and collision terms.The policy uses a Gaussian action distribution whose mean is derived from the predicted trajectory and whose standard deviation is learned during training.

4. Experiments

Experiments evaluate TIC-VLA in simulated and real-world dynamic navigation, including latency robustness, interface design, and deployment constraints. TIC-VLA shows strong closed-loop performance while asynchronous reasoning remains active under delayed inference.

  • Experimental Setup: The evaluation compares TIC-VLA with point-goal, VLN, and VLA navigation baselines using NE, SR, SPL, and CR.Point-goal baselines receive privileged goal information, whereas TIC-VLA uses egocentric observations and language instructions.
  • Simulation Testing: After RL fine-tuning, TIC-VLA achieves the highest success rate and lowest collision rate on DynaNav.It also outperforms prior VLN and stronger VLA baselines, while NavDP retains lower navigation error and higher SPL with direct point-goal supervision.
  • Simulation Testing: RL fine-tuning maintains higher success rates across increasing VLM latency, while collision rates remain relatively insensitive to inference delay.The IL-based action expert declines more noticeably as latency increases, whereas the RL-fine-tuned policy is more robust.
  • Interface and Latency Ablation: KV-cache semantic features combined with latency-aware modeling and training achieve the best overall interface performance.Waypoint-based guidance performs worse because it is sparse and may be inconsistent with local observations.
  • Real-world Testing: On a Unitree Go2, TIC-VLA reaches 0.85 success after RL fine-tuning, up from 0.70, and maintains 0.75 success on Jetson Orin NX deployment.The real-world evaluation covers indoor and outdoor scenes with pedestrians, obstacles, clutter, and uneven terrain.
  • Ablation Study: Test-time reasoning improves navigation, reducing NE from 14.23 to 10.55 while increasing SR from 40.00 to 55.29 and SPL from 34.22 to 50.29.Disabling reasoning increases the forward rate from 0.5 Hz to 4 Hz but produces weaker task completion and navigation progress.
  • Ablation Study: The 3-second action prediction horizon achieves the best overall performance among TIC-VLA variants without RL fine-tuning.The 1-second horizon has the lowest collision rate, while the 5-second horizon underperforms because longer prediction can reduce near-term accuracy.

5. Conclusions

TIC-VLA addresses the temporal mismatch between slow semantic reasoning and real-time control through latency-aware training and a delayed semantic-control interface. Simulation and real-world results show consistent improvements over prior navigation and language-guided baselines, while the system retains several deployment and scope limitations.

  • TIC-VLA addresses the temporal mismatch between slow semantic reasoning and real-time control using a delayed semantic-control interface and latency-aware training.
  • Simulation and real-world results show consistent improvements over prior navigation and language-guided baselines.
  • Runtime efficiency is not yet fully optimized, leaving room for faster inference and deployment.
  • Real-world evaluation remains limited in scale, and extending beyond navigation to robotic manipulation remains future work.

Impact Statement

TIC-VLA addresses inference latency and limited onboard computation for real-time robot navigation, with potential relevance to service robotics, logistics, and assisted mobility. Safety-critical deployment still requires human oversight because perception, reasoning, and control failures can create risks in shared human spaces.

  • TIC-VLA may support service robotics, logistics, and assisted mobility, especially on edge platforms.
  • Shared human-space deployment can involve misinterpreted instructions, collisions, and socially inappropriate behavior.
  • Human oversight remains necessary in safety-critical settings because perception, reasoning, and control failures persist.

A. Model Details

The model combines a vision-language backbone, shared visual encoding, delayed semantic guidance, latency metadata, and a Transformer-based action expert. It predicts short-horizon action chunks that are integrated into continuous trajectories, while a value network supports learning.

  • The VLM uses InternVL3-1B and processes four frames spanning a nine-second temporal window for reasoning and waypoint generation.Frames are sampled at three-second intervals, with three historical frames and one current frame.
  • A shared vision encoder provides current visual tokens to the action expert while VLM observations remain temporally lagged because of inference latency.
  • The Transformer action expert conditions on current visual tokens, proprioception, delayed VLM guidance, and latency metadata.
  • Latency metadata includes estimated displacement during inference, represented by ∆x, ∆y, ∆θ, and the time delay ∆t.
  • Cross-attention outputs are converted into action chunks, which are integrated into continuous trajectories over a three-second horizon.The horizon contains T = 30 action chunks, each parameterized by (dx, dy, dθ).
  • The value network combines current visual features with robot state and relative goal position for value estimation.

B. Training Details

Training constructs temporally standardized navigation episodes, generates language and reasoning annotations, and trains the action policy with explicitly delayed VLM information. Imitation learning and online reinforcement learning both expose the policy to asynchronous reasoning conditions and latency metadata.

  • Data Processing: Raw sequences are segmented into 20-second episodes and downsampled to 10 Hz using egocentric RGB images and robot pose trajectories.
  • Data Processing: Each annotated sample combines a history window, current frame, and future trajectory represented with robot-centric relative motion offsets.
  • VLM Supervised Fine-Tuning: SFT generates long-horizon navigation instructions from five representative frames and future trajectory offsets for each episode.
  • VLM Supervised Fine-Tuning: Concise reasoning traces are generated from past trajectories, historical observations, current observations, and future observations.
  • VLM Supervised Fine-Tuning: The VLM is trained to output either reasoning followed by waypoints or waypoint-only predictions, supporting explicit reasoning when needed and efficiency otherwise.
  • Imitation Learning with Delayed Inference: Imitation learning samples inference delays and conditions the action policy on delayed VLM states, reasoning annotations, guidance waypoints, and latency metadata.
  • Imitation Learning with Delayed Inference: The imitation-learning pipeline integrates predicted actions into trajectories and updates the policy by minimizing imitation loss.
  • Online Reinforcement Learning: Online reinforcement learning runs VLM inference asynchronously, retrieves delayed hidden states and latency metadata, and optimizes a stochastic Gaussian policy with PPO.

C. Benchmark Details

DynaNav evaluates language-conditioned navigation across four physics-based environments with controlled variation in crowd density, instructions, robot platforms, and time limits. The benchmark spans hospital, office, warehouse, and outdoor scenarios with distinct spatial and dynamic challenges.

  • DynaNav contains 85 episodes across Hospital, Office, Outdoor, and Warehouse environments.
  • Task difficulty varies pedestrian density from 0 to 200 agents, instruction complexity, and time limits while maintaining consistent goals within each task.
  • Tasks share semantic objectives while varying crowd density, robot platform, and timeout constraints to test dynamic-obstacle robustness, instruction following, and cross-robot generalization.
  • Hospital Environment: Hospital episodes emphasize narrow passages, occlusions, and socially constrained motion amid dense pedestrian traffic.
  • Office Environment: Office episodes require multi-step spatial instructions across branching rooms, corridors, elevators, and landmark-based goals.
  • Warehouse Environment: Warehouse episodes stress disambiguation of repeated structures and precise rack locations under pedestrian interference, with density varying from 0 to 60.
  • Outdoor Environment: Outdoor episodes involve long-distance navigation over uneven terrain and open spaces, with 100 to 200 pedestrians and a relaxed success threshold.
  • Figure 9 organizes instructions and navigation scenarios across the four environments, highlighting differences in layouts, landmarks, and human density.

D. Experiment Details

Experiments combine teleoperated demonstrations, asynchronous online reinforcement learning, latency injection, and multiple action-policy baselines in a reproducible physics-based simulator. Deployment and rollout procedures maintain high-frequency control while semantic reasoning runs asynchronously.

  • Demonstrations are collected by keyboard teleoperation across four DynaNav environments with dynamic humans and natural-language task descriptions.
  • Online reinforcement learning trains TIC-VLA on three tasks across office, hospital, and warehouse environments at 10 Hz.
  • Training injects random VLM inference delays averaging approximately 50 environment steps, or 5 seconds, before outputs reach the policy.
  • The reward assigns wg = 400 for reaching the goal, wp = 5 for progress, wc = −100 for collisions, and ws = −0.1 for speed penalty.
  • The behavior-cloning baseline uses demonstrations without latency or delayed-perception modeling and receives privileged relative-goal information.
  • Action Policy Architectures: The diffusion action policy denoises fixed-length 30-step action chunks using 100 noise steps and five-step DDIM inference.
  • Action Policy Architectures: The flow-based policy models conditional action-chunk distributions with a normalizing flow conditioned on vision features and cached VLM hidden states.
  • Simulation runs at 30 Hz while the policy acts at 10 Hz, applying each action for three simulation frames before termination on goal reaching or timeout.

E. Additional Results

Additional results show TIC-VLA sustaining coherent, adaptive navigation under delayed perception in simulation and outperforming prior methods in real-world testing. Ablations identify latency-efficiency trade-offs in the VLM backbone and favor a query-based policy and six-layer action expert.

  • Simulation Testing: TIC-VLA generates coherent semantic reasoning and stable low-level control across outdoor, hospital, and warehouse simulations with delayed perception.
  • Simulation Testing: In simulation, the robot adapts to dynamic agents by yielding, adjusting speed, and selecting appropriate turning maneuvers.
  • Real-world Testing: TIC-VLA consistently outperforms prior methods across indoor and outdoor real-world navigation tasks measured over five trials per task.
  • Influence of Action Policy: The query-based Transformer policy achieves the best results across all metrics, while diffusion and flow-matching policies provide no performance benefit.
  • Simulation Testing: Figure 11 reports additional qualitative simulation results for outdoor, hospital, and warehouse settings.
  • Influence of VLM Backbone: InternVL3 achieves the best trade-off between representational capacity and inference efficiency compared with SmolVLM2 and Qwen2.5-VL.
  • Influence of Action Expert Architecture: The six-layer action expert performs strongest overall; twelve layers worsen success rate, SPL, and collision rate despite slightly lower navigation error.
  • Sensitivity to Odometry Drift: TIC-VLA remains stable under typical odometry drift and succeeds in a substantial portion of severe stress-test episodes, though accumulated localization errors degrade performance.
Loading 2602.02459v2…