PaperlayerView exact revision on arXiv

Source-linked AI summary

TrackVLA: Embodied Visual Tracking in the Wild

Shaoan Wang, Jiazhao Zhang, Minghan Li, Jiahang Liu, Anqi Li, Kui Wu, Fangwei Zhong, Junzhi Yu, Zhizheng Zhang, He Wang

arXiv:2505.23189v1cs.ROcs.CV

TL;DR

Embodied visual tracking must combine target recognition with trajectory planning under occlusion and dynamic scenes, while existing methods commonly separate these functions. TrackVLA unifies them through a shared LLM backbone with task-specific recognition and planning heads, achieving strong zero-shot benchmark performance and real-world generalization at 10 FPS.

  • Problem

    Embodied visual tracking requires accurate recognition and trajectory planning under severe occlusion and high scene dynamics, while existing approaches typically separate these capabilities.

  • Method

    TrackVLA jointly trains recognition and tracking using shared token encoding and LLM forwarding, with a language-modeling head and an anchor-based diffusion head.

  • Results

    TrackVLA achieves SOTA zero-shot performance on Gym-UnrealCV, outperforms EVT-Bench baselines, and robustly tracks unseen objects in novel real-world environments at 10 FPS.

  • Takeaways & Limitations

    TrackVLA supports zero-shot deployment across synthetic and real-world tracking settings while maintaining high-frequency inference and strong generalization.

  • Takeaways & Limitations

    TrackVLA relies solely on egocentric observation and a waypoint controller, limiting its field of view and motion flexibility.

Abstract

from arXiv · show

Embodied visual tracking is a fundamental skill in Embodied AI, enabling an agent to follow a specific target in dynamic environments using only egocentric vision. This task is inherently challenging as it requires both accurate target recognition and effective trajectory planning under conditions of severe occlusion and high scene dynamics. Existing approaches typically address this challenge through a modular separation of recognition and planning. In this work, we propose TrackVLA, a Vision-Language-Action (VLA) model that learns the synergy between object recognition and trajectory planning. Leveraging a shared LLM backbone, we employ a language modeling head for recognition and an anchor-based diffusion model for trajectory planning. To train TrackVLA, we construct an Embodied Visual Tracking Benchmark (EVT-Bench) and collect diverse difficulty levels of recognition samples, resulting in a dataset of 1.7 million samples. Through extensive experiments in both synthetic and real-world environments, TrackVLA demonstrates SOTA performance and strong generalizability. It significantly outperforms existing methods on public benchmarks in a zero-shot manner while remaining robust to high dynamics and occlusion in real-world scenarios at 10 FPS inference speed. Our project page is: https://pku-epic.github.io/TrackVLA-web.

1 Introduction

Embodied visual tracking requires tightly coupled target recognition and trajectory planning, especially under occlusion and dynamic scenes. TrackVLA unifies these capabilities and achieves strong zero-shot and real-world generalization.

  • Embodied visual tracking requires persistent target following and is a fundamental embodied-AI capability demanded in robotics.
  • Severe occlusion and highly dynamic scenes make the coupled requirements of target recognition and trajectory planning particularly challenging.
  • Existing methods decouple recognition and planning into detection and planning models, but remain limited to category-level tracking in relatively open areas.
  • TrackVLA jointly trains recognition and trajectory planning through a shared token-encoding and LLM-forwarding mechanism with task-dependent decoding heads.Recognition uses a language modeling head, while planning uses an anchor-based diffusion head for waypoint trajectories.
  • 855K recognition samples and 855K robot-tracking samples are collected across varying difficulty levels using public datasets and EVT-Bench.
  • TrackVLA achieves zero-shot SOTA performance on Gym-UnrealCV and robust tracking of unseen objects in novel real-world environments at 10 FPS.It also significantly outperforms baselines on EVT-Bench involving detailed language input and crowded environments.

2 Related Works

Related work spans embodied visual tracking, embodied navigation, and vision-language-action models. TrackVLA addresses the dynamic, occluded tracking setting within this broader landscape.

  • Embodied Visual Tracking: Embodied visual tracking requires continuously pursuing dynamic targets using visual observations, with accurate recognition and trajectory planning.
  • Embodied Visual Tracking: Many tracking methods decouple perception and planning into separate modules, a design associated with error accumulation.
  • Embodied Visual Tracking: Figure 2 depicts TrackVLA receiving video and language instruction and producing either a tracking trajectory or a recognition answer.
  • Embodied Navigation: Embodied navigation includes language-, object-, and question-answering subtasks, but most existing tasks target static indoor environments.
  • Embodied Navigation: TrackVLA focuses on embodied visual tracking as navigation in highly dynamic and occluded environments, requiring identification and continuous tracking of a moving target.
  • Vision-Language-Action Models: Vision-language-action models extend pretrained vision-language models with action generation and have been applied to manipulation and navigation.

3 Method

TrackVLA unifies target recognition and trajectory planning through shared visual-language processing with task-specific decoding. Its anchor-based diffusion action model generates waypoint trajectories efficiently from learned trajectory anchors.

  • Task Formulation: The embodied visual tracking formulation maps language instructions and egocentric RGB frame sequences to linear and angular velocity actions.Success requires maintaining a 1–3 m following distance while facing the target.
  • TrackVLA Architecture: TrackVLA uses a parallel prediction branch to jointly support trajectory planning and target recognition.Both tasks share token encoding and LLM forwarding, while decoding differs by task.
  • Observation Encoding: TrackVLA compresses visual observations with grid pooling, using fine-grained features for the latest frame and coarse-grained tokens for historical observations and recognition.Fine-grained features contain 64 tokens, while coarse-grained features contain 4 tokens.
  • LLM Forwarding: Visual features are projected into the LLM latent space, combined with language tokens, and processed through a shared LLM before task-specific decoding.Recognition uses an autoregressive language-modeling head; tracking passes the predicted hidden state to the action model.
  • Anchor-based Diffusion Action Model: The anchor-based diffusion action model clusters training trajectories into anchors, perturbs them with Gaussian noise, and denoises them into waypoint trajectories.Predefined anchors provide coarse trajectories and reduce denoising iterations, yielding a 5× speedup over vanilla diffusion policies.
  • Training Objective: TrackVLA jointly optimizes trajectory regression and anchor-score prediction with a tracking loss combined with text prediction loss.The overall objective is L = Ltrack + αLtext, and the action model uses a Diffusion Transformer with two denoising steps.

4 Data Collection

TrackVLA is trained and evaluated with paired tracking and recognition data spanning synthetic scenes, humanoid targets, and open-world visual question answering. EVT-Bench provides diverse simulated tracking episodes and separates training from testing by avatars and scenes.

  • Training Data: TrackVLA training uses 855K embodied visual tracking samples and 855K video-based recognition samples.The two datasets are empirically combined at a 1:1 ratio for training.
  • Embodied Visual Tracking Data: The tracking data comes from a custom Habitat 3.0 simulator with automatically generated humanoid avatars and natural walking behaviors.Avatars use randomized SMPL-X shapes and textures, while ORCA provides dynamic collision avoidance; walking speeds range from 1.0 m/s to 1.5 m/s.
  • EVT-Bench: EVT-Bench contains 100 humanoid avatars, 804 HM3D and MP3D scene environments, and 25,986 episodes.Training and testing splits contain no overlapping avatars or scenes.
  • Benchmark Tasks: EVT-Bench evaluates single-target, distracted, and ambiguity tracking with progressively more demanding target-identification requirements.The tasks range from simple follow instructions to fine-grained descriptions and visually identical distractors.
  • Recognition Data: Because tracking samples are synthetic, TrackVLA adds 855K recognition samples comprising 362K human-recognition and 493K open-world VQA samples.These samples extend recognition beyond the synthetic tracking environments.
  • Recognition Data: Human-recognition VQA samples are built from SYNTH-PEDES by composing 1–3 human images over diverse backgrounds with textual attribute and spatial descriptions.The descriptions include individual attributes, relative positions, and related identity information.

5 Experiments

TrackVLA is evaluated across embodied tracking, recognition, real-world deployment, and design ablations. It shows strong zero-shot performance and generalization, while balanced multimodal data and the anchor-based action model improve results.

  • Experiments assess embodied tracking, target recognition, and the effectiveness of proposed designs using Gym-UnrealCV and EVT-Bench.
  • Quantitative Comparison: TrackVLA successfully tracks targets throughout Single Target and Unseen Objects tasks over 500 steps and 100 episodes in zero-shot Gym-UnrealCV evaluation.
  • Quantitative Comparison: 3.25% higher EL and 12.3% higher SR than EVT are reported for the challenging Distractor task on Gym-UnrealCV.
  • Quantitative Comparison: TrackVLA significantly outperforms existing approaches across STT, DT, and AT on EVT-Bench, although performance drops from STT to distracted and ambiguous tracking.
  • Quantitative Comparison: 10 FPS inference matches SoM + GPT-4o recognition performance while running approximately 100× faster than GPT-based baselines.
  • Ablation Study: Increasing training-sample scale consistently improves DT performance, while a 1:1 tracking-to-recognition ratio performs best.
  • Ablation Study: The anchor-based diffusion action model outperforms autoregressive, MLP, and vanilla diffusion-policy baselines across all metrics, with larger DiT backbones improving performance.

6 Conclusions

TrackVLA is a VLA model for embodied visual tracking that jointly handles trajectory and text outputs. Joint training on tracking and recognition data supports strong simulated performance and zero-shot real-world deployment.

  • TrackVLA outputs both tracking trajectories and text-based responses within a unified VLA model.
  • The model is jointly trained on embodied visual tracking data and open-world recognition data to learn synergy between the two modalities.
  • The dataset contains 855K embodied visual tracking samples and 855K open-world recognition samples.
  • Extensive experiments demonstrate state-of-the-art simulation performance and strong generalization enabling zero-shot deployment in real-world scenarios.

7 Limitations

The reported limitations concern observation coverage and motion control, while the supplied implementation and benchmark passages describe training, inference, and simulation procedures.

  • Egocentric-only observation limits TrackVLA to a narrow field of view, typically 90° FOV.Panoramic or multi-view inputs are proposed to improve tracking robustness.
  • The current approach uses only a waypoint controller and lacks a more flexible local motion controller.The authors suggest that adding such a controller could improve movement speed and expand reachable areas.
  • Training and Inference: TrackVLA uses a two-stage training pipeline that first aligns visual embeddings with the LLM and then jointly trains the projector, LLM, and action model.
  • Training and Inference: Inference resizes frames to 224×224, uses a special [Track] token, performs single-step LLM autoregression, and applies diffusion to trajectory anchors.
  • Benchmark Procedure: EVT-Bench trajectories include a start point, 0–2 intermediate waypoints, and an endpoint, with consecutive-waypoint distances exceeding dmin = 3 m.
  • Benchmark Procedure: Episodes terminate when the target reaches its destination or when the agent collides with the humanoid.

C.3 Metric Definitions

The evaluation uses success, tracking-quality, and collision metrics, alongside environment settings spanning basic, obstructed, urban, and difficult-terrain scenarios.

  • Metrics: Success Rate counts episodes where the agent remains oriented toward the target at a safe distance of 1–3 meters.
  • Metrics: Tracking Rate measures the proportion of steps during which the agent successfully tracks the target, defined as TR = S/L.
  • Metrics: Collision Rate measures the proportion of episodes terminated by collisions between the agent and the target humanoid avatar.
  • Evaluation Settings: Gym-UnrealCV evaluation allows up to 500 steps per episode and defines success as keeping the target within a 90-degree, 750-cm tracking region throughout.Failure occurs when the target remains outside the region for more than 50 consecutive steps.
  • Evaluation Settings: The environments range from SimpleRoom to parking lots, urban scenes, and Snow Village with occlusion, low light, reflective surfaces, uneven terrain, or backlighting.

D.5 Experiment Results

TrackVLA generalizes across unseen environments, distractor-heavy scenes, and unseen object categories, while the experiments also examine action-model and temporal-design choices.

  • Benchmark Results: TrackVLA achieves state-of-the-art performance across five unseen environments and passes all test cases without simulator training data.This evaluates zero-shot transfer across varied lighting, viewpoints, and scene layouts.
  • Benchmark Results: The unseen-environment results report Average Episode Length and Success Rate as the two metrics in each Table 6 cell.
  • Benchmark Results: TrackVLA achieves state-of-the-art performance across distraction scenarios requiring it to follow the first observed target despite identical distractors.
  • Benchmark Results: TrackVLA successfully tracks unseen horse, dog, sheep, and pig categories in SimpleRoom, demonstrating generalization to novel object types.
  • Ablations: Removing history observations causes a significant performance drop, while a history window of 32 is selected as optimal.
  • Ablations: Predicting 10 waypoints yields the best overall performance among the tested waypoint lengths.

F.4 Human Recognition Dataset

The human recognition dataset combines multiple data types and evaluates how recognition diversity and background variation affect performance.

  • Dataset Design: Human recognition data are categorized as Single Human, Multiple Human, and Same Human, representing one person, 2–3 different individuals, and two identical individuals.
  • Dataset Design: Including each human recognition data type improves recognition performance.
  • Dataset Design: Replacing human recognition images with plain white backgrounds causes a notable performance drop.
  • Deployment: The robotic platform uses a Unitree GO2 quadruped with an Intel RealSense D455 camera and transmits RGB observations to a remote server over the Internet.The camera provides 640×480 RGB frames with a 90° horizontal field of view.

H Real-world Experiments

Real-world experiments compare TrackVLA with a commercial tracking drone across increasing difficulty, including occlusion and fast target motion.

  • Experimental Setup: The comparison with DJI Flip tests easy, medium, and hard tracking scenarios, each repeated 10 times.Difficulty increases from open outdoor environments to wall occlusions and high-speed target motion.
  • Results: Both TrackVLA and DJI Flip achieve a 100% success rate in the Easy setting.
  • Results: As difficulty increases, DJI Flip’s performance drops significantly below TrackVLA’s performance.Representative cases show TrackVLA succeeding where DJI Flip fails.
  • Results: The real-world visualization includes tracking under occlusion and fast target motion, where TrackVLA outperforms existing commercial tracking drones.
Loading 2505.23189v1…