PaperlayerView exact revision on arXiv

Source-linked AI summary

ReferTrack: Referring Then Tracking for Embodied Visual Tracking

Hanjing Ye, Tianle Zeng, Jiazhao Zhang, Shaoan Wang, Zibo Zhang, Weisi Situ, Yuchen Zhou, Yonggen Ling, Hong Zhang

arXiv:2607.20061v1cs.RO

TL;DR

Embodied visual tracking must identify a language-specified target and plan motion using onboard vision, but existing reasoning is weakly aligned with image-space detections. ReferTrack selects the target from indexed bounding boxes, tracks it with temporal bbox cues, and achieves state-of-the-art single-view performance, including 89.4%, 73.3%, and 74.1% success rates across EVT-Bench splits.

  • Problem

    Embodied visual tracking requires jointly identifying a language-specified target and planning collision-free following motions from onboard vision.

  • Method

    ReferTrack selects an indexed image-space bounding box, then predicts waypoints using queued historical boxes encoded as TVBI tokens and co-training on Refer-QA.

  • Results

    89.4%, 73.3%, and 74.1% success rates establish state-of-the-art single-view performance on EVT-Bench’s single-target, distracted, and ambiguity splits.

  • Takeaways & Limitations

    Explicit image-space referring enables strong single-view tracking performance, matching or surpassing several multi-camera baselines on identification-heavy tasks.

  • Takeaways & Limitations

    ReferTrack assumes that an unobserved historical target can be represented by a deterministic zero-valued bounding-box sentinel.

Abstract

from arXiv · show

Embodied visual tracking (EVT) requires a mobile agent to continuously follow a specific target described in natural language using only onboard vision. While recent vision-language-action (VLA) policies unify target identification and trajectory planning, their chain-of-thought (CoT) reasoning often operates in abstract spatial latents that are difficult to supervise and weakly aligned with explicit image-space detections. To address this, we introduce ReferTrack, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera. Our model first selects the target from an indexed set of bounding boxes, then decodes tracking waypoints conditioned on this image-grounded decision. To preserve target motion cues over time, ReferTrack maintains a sliding-window queue of previously selected bounding boxes, injecting their geometric features into the visual history via temporal-viewpoint-bbox indicator (TVBI) tokens. We further enhance target identification by co-training on a custom Refer-QA dataset. On EVT-Bench, ReferTrack achieves state-of-the-art single-view performance with success rates of 89.4%, 73.3%, and 74.1% on the single-target, distracted, and ambiguity tracking splits, respectively -- matching or even surpassing several multi-camera baselines on identification-heavy tasks. Finally, real-world deployments on legged and humanoid robots validate its robust sim-to-real transfer capabilities. Code is available at https://github.com/MedlarTea/referTrack.

1 Introduction

ReferTrack frames embodied visual tracking as a referring-then-tracking problem, selecting a target from image-space bounding boxes before planning its trajectory. On EVT-Bench’s single-forward-view setting, it achieves state-of-the-art performance across tracking splits and surpasses several multi-view baselines on identification-heavy tasks.

  • Problem setting: Embodied visual tracking requires following a natural-language-specified target using only onboard vision.Success depends on identifying the instructed pedestrian and planning collision-free motions that maintain an appropriate following distance.
  • Prior approaches: Recent vision-language-action models unify target identification and trajectory planning within a single next-token prediction policy.Many co-train EVT with other navigation tasks to learn general navigational priors and obtain baseline tracking capabilities.
  • ReferTrack: ReferTrack selects one indexed image-space bounding box as the target, converting identification into a supervised multiple-choice problem aligned with visual-language grounding.This referring-then-tracking formulation replaces reliance on abstract spatial codes with explicit egocentric-view detections.
  • Results: On EVT-Bench’s single-forward-view setting, ReferTrack achieves state-of-the-art performance across the single-target, distracted, and ambiguity tracking splits.Its single-camera results surpass several reported multi-view baselines and reinforcement-learning-refined methods on identification-heavy splits.

2 Related Work

Related work spans vision-language navigation, embodied visual tracking, and language-grounded object identification. Existing approaches range from modular perception-control pipelines to end-to-end multimodal models and target-conditioned visual memory.

  • Vision-Language Navigation: Recent vision-language navigation models fine-tune vision-language models end-to-end on navigation trajectories, following earlier approaches that prompted off-the-shelf LLMs with textual scene descriptions.The passage contrasts early text-based prompting with trajectory-based end-to-end training.
  • Vision-Language Navigation: After Refer-CoT identifies a target, historical bounding boxes are queued and integrated into the visual stream as target-conditioned memory for continuous tracking.The mechanism converts generic spatiotemporal indexing into memory conditioned on the selected target.
  • Embodied Visual Tracking: Embodied visual tracking requires mobile agents to continuously pursue dynamic, language-specified targets, traditionally using separate perception and reinforcement- or imitation-learning control modules.The modular design offers interpretability but leaves recognition errors able to compound into planning.
  • Referring, Grounding, and Catalog-Based Identification: Referencing and grounding methods localize language-described objects, while multimodal models increasingly output bounding boxes as tokens or select among region proposals.This line of work connects natural-language descriptions with explicit image regions.

3 Method

ReferTrack grounds embodied visual tracking in an indexed image-space target decision before trajectory prediction. It preserves target geometry across a sliding visual history with TVBI tokens and trains referring, navigation, and question-answering capabilities jointly.

  • Model Overview: ReferTrack uses a dual-branch architecture that encodes forward-view history, indexed pedestrian candidates, and language before sequential referring and trajectory prediction.Visual, catalog, and language tokens are concatenated and passed through a large language model in two stages.
  • Refer-CoT and Trajectory Prediction: The model selects one indexed bbox or ⟨NO EXIST⟩ as a compact, image-space Refer-CoT decision aligned with the language instruction.Target selection is a strictly one-token classification decision over registered candidate tokens, with ⟨NO EXIST⟩ indicating an unobservable target.
  • Observation Encoding: ReferTrack encodes visual history with SigLIP and DINOv2 features, grid-pooled fine and coarse tokens, and a sliding window of the latest H frames.TVBI tokens inject normalized target boxes into historical frame groups, while current-frame fine tokens remain TVI-only.
  • Referred-Target Bbox Queue: A FIFO queue stores preceding referred-target boxes for future TVBI encoding, uses ground-truth boxes with occasional incorrect-index noise during training, and updates autoregressively at inference.Unobserved historical targets use [0, 0, 0, 0] as an absence sentinel that distinguishes missing geometry from visible target motion.
  • Refer-CoT and Trajectory Prediction: The selected Refer-CoT token conditions action-token generation, whose dedicated MLP head decodes M tracking waypoints.This separates image-grounded target identification from downstream motion planning.
  • Training Objective: ReferTrack jointly fine-tunes the LLM, vision modules, and action modules using a weighted sum of navigation trajectory, Refer-CoT, and auxiliary losses.The trajectory loss supervises waypoint prediction with Mean Squared Error, while the Refer-CoT term supervises the ground-truth target index.

4 Experiments

ReferTrack is evaluated on EVT-Bench in Habitat 3.0 under a single forward-view protocol across single-target, distracted, and ambiguity tracking tasks. It achieves strong benchmark performance with a compact supervised-only policy, while ablations and real-world deployments assess its referring components and transferability.

  • Main results: 89.4% SR, 92.5% TR, and 1.6% CR on STT; 73.3% SR, 81.8% TR, and 7.6% CR on DT; and 74.1% SR, 85.7% TR, and 7.7% CR on AT.The evaluation uses only the front RGB camera for target identification and tracking.
  • Main results: ReferTrack attains the best single-view SR/TR/CR on STT, showing that its referring interface preserves standard tracking stability.The policy uses a compact 4B-parameter backbone, 1.3M navigation samples, and supervised fine-tuning without reinforcement learning.
  • Ablation studies: 81.5% SR and 84.7% TR with ground-truth target boxes exceed the full model’s 73.3% SR and 81.8% TR, while the expert policy reaches 85.1% SR.This oracle variant bypasses Refer-CoT and evaluates motion planning under perfect target identification.
  • Ablation studies: Removing TVBI lowers SR/TR from 73.3/81.8 to 70.4/80.8, while removing both Refer-CoT and TVBI causes a larger drop to 55.7% SR and 71.4% TR.The results indicate that explicit image-space target selection provides the primary robustness under distractors, with TVBI further stabilizing tracking.
  • Real-world deployment: 10.6 Hz average loop frequency and 12 ms target detection support deployment on Unitree Go2 and Unitree G1 using a single forward-facing camera.Go2 follows pedestrians through cluttered obstacles, while G1 maintains the referred target under multi-person interference.

5 Conclusion

ReferTrack is a referring-then-tracking VLA policy for embodied visual tracking with a single forward-facing camera. It identifies the instructed target through indexed bounding-box selection, preserves motion cues with a sliding-window queue and TVBI tokens, and co-trains on Refer-QA.

  • ReferTrack performs embodied visual tracking with a referring-then-tracking VLA policy using a single forward-facing camera.
  • A single Refer-CoT token selects the instructed target through compact, index-based bounding-box selection before waypoint prediction.
  • A sliding-window bounding-box queue injects target motion cues into visual history through temporal-viewpoint-bbox indicator (TVBI) tokens.
  • ReferTrack co-trains on a custom Refer-QA dataset that uses the same indexed catalog to enhance target identification.

A Training Details · A.1 Expert Tracking Data Curation

ReferTrack’s expert tracking data are curated from EVT-Bench in Habitat 3.0 using a simulator-state oracle that plans local target-following waypoints. Successful, collision-free episodes provide downsampled samples containing robot-target motion and visual target boxes, while failures are retained separately for inspection.

  • A Training Details: Expert trajectories come from the EVT-Bench training split in Habitat 3.0.
  • A.1 Expert Tracking Data Curation: A custom oracle controller accesses simulator states for both the robot and instructed humanoid target.
  • A.1 Expert Tracking Data Curation: At each simulation step, the controller queries Habitat’s geodesic shortest-path planner from the robot to the target.
  • A.1 Expert Tracking Data Curation: The returned path is densified and used to select a local lookahead waypoint for tracking.
  • A.1 Expert Tracking Data Curation: Rollouts record per-step robot and target poses, expert base velocities, and target bboxes extracted from panoptic observations.
  • A.1 Expert Tracking Data Curation: Episodes are accepted only when Habitat’s following metric remains successful without collision or prolonged target loss.
  • A.1 Expert Tracking Data Curation: Failed rollouts are stored separately for inspection, while successful trajectories are downsampled to 330K samples for STT and distracted tracking.
  • A.1 Expert Tracking Data Curation: The oracle demonstrates backing up, waypoint chasing around large turns, rear-side following, and stopping when the target stops.

A.2 Refer-QA Data Generation

Refer-QA supervision is generated from SYNTH-PEDES by compositing captioned pedestrian crops into non-overlapping scenes and training the model to select the queried indexed bounding box or ⟨NO EXIST⟩. Each candidate is paired with a catalog ID, image-space bounding box, and caption, while absent captions use an all-zero box.

  • Data synthesis: Refer-QA samples composite 2–3 pedestrian crops onto resized background-image regions using random scales from [0.75, 1.5] while rejecting overlaps.The construction uses the same SYNTH-PEDES dataset as prior EVT methods for fair comparison.
  • Candidate supervision: Each pedestrian receives a random catalog ID from 0 to 19 with its image-space bbox and caption, plus one absent negative caption represented by an all-zero bbox.Valid pedestrians are shuffled into the indexed navigation catalog, with a fixed virtual ⟨NO EXIST⟩ slot appended.
  • Candidate supervision: The training prompt asks the model to find a caption in the video and answer with object indexes from the indexed candidate catalog.Examples supervise selection of the queried indexed bbox or ⟨NO EXIST⟩.

A.3 Optimization Settings · B Inference Details

ReferTrack uses a two-stage supervised fine-tuning strategy, beginning with projector-only training and continuing with joint navigation and Refer-QA training. The specified optimization includes stage-specific learning rates, a 1:1 data ratio, 20K steps, and a global batch size of 256.

  • A.3 Optimization Settings: Stage 1 freezes the language model and vision encoders while training only the vision projector Pvision for one epoch.Stage 1 uses general QA datasets [38] [43].
  • A.3 Optimization Settings: 1 × 10−4 is the Stage 1 learning rate for vision-projector training.This learning rate applies during the single-epoch Stage 1 training on general QA datasets.
  • A.3 Optimization Settings: Stage 2 jointly trains the model on navigation and Refer-QA tasks at a 1:1 data ratio.The joint training follows TrackVLA++ [2].
  • A.3 Optimization Settings: 20K steps and a global batch size of 256 define the Stage 2 training schedule.These settings are specified for the joint navigation and Refer-QA training stage.
  • A.3 Optimization Settings: 2 × 10−5 is the learning rate assigned to the LLM during Stage 2 optimization.The LLM learning rate is specified separately from the remaining trainable modules.
  • A.3 Optimization Settings: 1 × 10−4 is the learning rate for remaining trainable modules, including the projectors and action head.This rate is distinct from the 2 × 10−5 learning rate used for the LLM.

B.1 Robot Platform

ReferTrack was evaluated on both quadruped and humanoid robot platforms in real-world deployments. Each robot used a single forward-facing Intel RealSense D455 camera and portable Wi-Fi, with the model deployed on a remote high-performance server.

  • B.1 Robot Platform: ReferTrack was tested on a Unitree Go2 quadruped and Unitree G1 humanoid, each equipped with a single forward-facing Intel RealSense D455 camera and portable Wi-Fi.The model was deployed on a remote high-performance server.

B.2 Inference Pipeline

ReferTrack deploys as a remote-GPU WebSocket service that processes streamed camera frames and language instructions, returning a selected target slot and trajectory. The pipeline uses a single forward-facing Intel RealSense D455 camera and latency-oriented engineering optimizations.

  • Remote inference: ReferTrack streams JPEG-compressed RGB frames and language instructions to a remote GPU server, which returns the selected target slot and predicted trajectory.The server decodes frames, updates the detector/tracker, and builds an indexed bounding-box catalog.
  • Robot platforms: Both robot platforms use a single forward-facing Intel RealSense D455 camera and portable Wi-Fi to communicate with the remote server.The deployment architecture places high-performance inference on the remote server.
  • Latency optimization: Latency is reduced by one-time checkpoint loading, optional torch.compile warm-up, and parallel DINO and SigLIP feature extraction.Each robot connection has an independent streaming session, while current-frame features are computed in separate Python threads on separate CUDA str…
Loading 2607.20061v1…