PaperlayerView exact revision on arXiv

Source-linked AI summary

VLX-Go: Robust Embodied Tracking through Multi-Stage Learning and On-Policy Refinement

Kyusong Lee, Heting Ying, Tiancheng Zhao

arXiv:2610.19361v1cs.CVcs.CLcs.RO

TL;DR

Following a designated person among distractors remains challenging because shared spaces introduce competing candidates. VLX-Go uses staged multimodal learning before tracking-specific waypoint prediction; on-policy refinement further improves distractor-tracking outcomes.

  • Problem

    Following an isolated person leaves unresolved the challenge of distinguishing the intended target from competing people in shared spaces.

    Source: §1 · pp. 1–2

  • Method

    VLX-Go aligns visual and language modalities, develops multimodal representations, and then specializes the policy for tracking and waypoint prediction.

    Source: §1 · p. 2

  • Results

    On EVT-Bench, distractor-tracking and ambiguous-tracking success reached 58.22% and 60.93%; DAgger added 2.42 points to DT success and reduced collision rate by 1.99 points.

    Source: §9 · p. 11, §8 · p. 11

  • Takeaways & Limitations

    The results support staged multimodal learning as a promising direction for distractor tracking and motivate further study of on-policy refinement in shared spaces.

    Source: §8 · p. 10, §9 · p. 11

    Real-world tracking-quality observations are qualitative, not a controlled user study, and the relationship between benchmark DT scores and user experience remains unquantified.

    Source: §8 · p. 11

Abstract

from arXiv · show

Embodied visual tracking requires a robot to identify a designated target from visual observations and control its own motion to follow that target through an environment. Following an isolated person is a comparatively simple demonstration of embodied visual tracking, but following the intended person among distractors remains challenging. Existing methods exhibit a substantial drop in success from single-target to distractor tracking, motivating a focus on the shared spaces that person-following robots must navigate. We present \method{}, a vision--language waypoint policy built around the hypothesis that visual and language representations should be aligned before tracking-specific motion is learned. The policy combines complementary frozen visual encoders, multiscale temporal context, a transformer backbone, and a lightweight head for robot-centric waypoint prediction. Its training progresses through vision--language alignment, multimodal reasoning mid-training, and tracking specialization on 19,532 oracle episodes. Compared with 41.54\% distractor-tracking success for our earlier OmTrackVLA model, the complete multi-stage recipe achieves 55.80\%. We then explore on-policy refinement using filtered oracle labels for states visited by the policy. This preliminary study adds 2.42 percentage points of distractor-tracking success, reaching 58.22\%, while reducing collision rate by 1.99 pts. The final policy records the highest distractor- and ambiguous-tracking success rates among the compared methods. These results support staged multimodal learning as a promising direction for tracking among distractors and motivate further study of on-policy refinement for person following in shared spaces.

1 Introduction

VLX-Go addresses the challenge of following a designated person among distractors with a vision–language waypoint policy and staged training. The complete recipe improves distractor-tracking success over the earlier baseline, and preliminary DAgger refinement adds further success.

1 Introduction

  • Distractor tracking tests whether a robot can follow the intended person amid competing candidates, a challenge not captured by isolated-person demonstrations.Reported methods show a substantial STT-to-DT success drop; for example, Uni-NaVid falls from 25.70% to 11.30%.Source: §1 · pp. 1–2
  • Short-horizon robot-centric waypoints make the link between multimodal understanding and local control explicit, while the low-level controller handles execution.Following requires combining target identity with current spatial configuration and recent motion.Source: §1 · p. 2
  • Training progresses from vision–language adapter alignment to multimodal reasoning mid-training and tracking specialization, followed by DAgger with filtered privileged oracle labels.The tracking data comprise 19,532 oracle episodes.Source: §1 · p. 2
  • 58.22% DT success follows DAgger refinement, up from 55.80% with the multi-stage policy and 41.54% with the earlier OmTrackVLA baseline.The preliminary refinement also reports fewer collisions, without specifying the amount here.Source: §1 · p. 2
  • VLX-Go combines frozen DINOv3 and SigLIP features, multiscale temporal context, a transformer, and a lightweight decoder for robot-centric waypoints.The policy predicts a short-horizon trajectory that can be passed to a low-level controller.Source: §1 · p. 2
  • The report evaluates tracking across conditions and emphasizes episode success and collision avoidance, with interaction diagnostics for further development.The introduction identifies service robots, guidance, and human–robot collaboration as relevant settings.Source: §1 · pp. 1–3

2 Related Work

Related work spans embodied tracking benchmarks, language-conditioned robot policies, staged multimodal learning, and visual representation methods. VLX-Go connects visual history and language to waypoint prediction for closed-loop tracking.

2 Related Work

  • EVT-Bench evaluates language-conditioned tracking in STT, DT, and AT settings, progressively stressing persistent tracking, distractor rejection, and semantic target disambiguation.Because the robot controls its camera, uncertainty about whom to follow changes its trajectory.Source: §2 · p. 3
  • Language-conditioned robot policies progress from grounding language proposals in affordances with SayCan and perceptual input with PaLM-E to RT-2's vision–language–action formulation.Related embodied tracking work also includes active tracking, visual servoing, and visual-foundation-model policies.Source: §2 · p. 3
  • OmChat [12] motivates separating modality alignment from later multimodal learning, while VLX-Go ultimately specializes the representations for tracking-specific waypoint prediction.OmChat combines projector alignment, progressive multimodal pretraining, and visual instruction tuning.Source: §2 · p. 3
  • TrackVLA specializes VLA learning for EVT, TrackVLA++ [8] adds spatial reasoning and target memory, and OmTrackVLA is VLX-Go's waypoint-policy predecessor.VLX-Go instead investigates staged multimodal training followed by oracle-supervised on-policy data aggregation with DAgger.Source: §2 · p. 3
  • VLM-FO1 [21] and VLX-Seek 1.5 address region-level grounding, whereas VLX-Go maps visual history and language to robot-centric waypoints for closed-loop tracking.VLX-Go pairs DINOv3's dense spatial structure with SigLIP's language-aligned semantics [9].Source: §2 · pp. 3–4

3 Task Formulation

The task formulation conditions a robot's tracking policy on a target description and a temporal window of monocular images. The policy predicts future robot-centric waypoints that a low-level controller turns into motion.

3 Task Formulation

  • At time t, the policy receives a natural-language target description and a temporal window of monocular RGB observations.The history contains H frames.Source: §3 · p. 4
  • The policy predicts K future robot-centric waypoints, forming a local trajectory for the low-level controller to execute.Evaluation reports success rate and collision rate, while tracking rate describes adherence to benchmark tracking conditions.Source: §3 · p. 4

4 VLX-Go

VLX-Go maps language-conditioned visual history to robot-centric waypoint sequences using complementary frozen encoders and a staged training pipeline.

4.2 Complementary Frozen Visual Encoders

  • DINOv3 dense visual features and SigLIP image–language semantics are spatially aligned and concatenated into 1536-D visual features.Both visual towers remain frozen, and their outputs are precomputed in FP16.Source: §4.2 · p. 4

4.3 Multiscale Temporal Tokenization

  • With 32 frames, history uses 2 × 2 pooling while the current frame uses 8 × 8 pooling, retaining temporal context while reserving detail for control.The resulting visual representation contains 31 coarse historical grids and one finer current-frame grid.Source: §4.3 · p. 4, Figure 1 · p. 5

4.1 Architecture Overview

  • Frozen DINOv3 and SigLIP features, language tokens, and a learnable action query feed a transformer whose action state produces eight robot-centric waypoints.The visual features are pooled into 188 tokens before projection into the transformer backbone.Source: §4.1 · p. 4, Figure 1 · p. 5

4.4 Projection, Temporal Indicators, and Action Query

  • A shared projector and temporal/view indicators combine visual tokens with instruction tokens and a learnable action query, distinguishing frame age and stream scale.The transformer processes this multimodal sequence, and the final hidden state at the action position is read for decoding.Source: §4.4 · p. 5

4.5 Waypoint Planner

  • A three-layer MLP maps the action state to robot-centric waypoints, trained with masked waypoint regression.The default planner predicts eight waypoints.Source: §4.5 · p. 5, Figure 1 · p. 5

4.6 Three-Stage Training and DAgger

  • Training proceeds from adapter-only alignment to backbone–adapter VQA mid-training, then tracking SFT of the backbone, adapter, and waypoint head, followed by DAgger on filtered policy rollouts.The visual encoders remain frozen throughout; DAgger mixes oracle demonstrations with filtered, oracle-labeled policy rollouts.Source: §4.6 · p. 6, Table 1 · p. 6

5 Oracle Data Collection and Supervision

The paper builds tracking supervision from a scripted simulator oracle and converts recorded trajectories into robot-centric waypoint examples.

Oracle trajectories

  • A scripted Habitat-Sim oracle uses ground-truth robot and human positions to generate following commands for STT, DT, and AT settings.The collection provides expert behavior-cloning supervision from simulator state.Source: §5 · p. 7

Recorded supervision

  • Each step records egocentric RGB, the target description, robot and target positions, and the expert action; the policy receives RGB history and language, not privileged state.Observations and state logs are aligned at a 10 Hz control rate; privileged simulator state constructs supervision.Source: §5 · p. 7

Training-data scale

  • The Stage 3 oracle collection contains 19,532 episodes, with training and benchmark evaluation episodes disjoint across HM3D and Matterport3D scenes.Scene identifiers may recur across task splits, so scene counts are reported separately by task.Source: §5 · p. 7

Trajectory supervision

  • Preprocessing pairs each target description with 31 past RGB observations and the current frame, producing eight future robot-centric waypoint targets at 0.1-second intervals.The episode count describes the source collection, not the number of training windows; the oracle also labels policy-visited states during DAgger.Source: §5 · p. 7

6 Experiments

Experiments compare VLX-Go on EVT-Bench’s single-target, distractor, and ambiguous tracking tasks, then evaluate whether DAgger refinement improves policy-visited-state supervision. The refined policy has the highest listed success rates on distractor and ambiguous tracking, while evaluation variability warrants caution about small differences.

6.1 Setup

  • EVT-Bench evaluates STT, DT, and AT on 1,405 episodes per task; DT tests identity preservation among distractors, while AT tests ambiguity among visually similar candidates.Benchmark evaluation episodes are disjoint from training episodes.Source: §6.1 · p. 7

6.2 Comparison with Prior Methods

  • The three-stage model reaches 55.80 SR and 12.10 CR on DT versus TrackVLA’s 57.60 SR and 5.80 CR, and 60.64 SR and 9.40 CR on AT.On AT, its success is 10.44 points above TrackVLA’s, while its collision rate is 7.70 points lower.Source: §6.2 · p. 8
  • Against OmTrackVLA, the three-stage policy raises DT success from 41.54% to 55.80% (+14.26 points), while DT CR rises from 11.31% to 12.10%.The DAgger-refined policy reaches 58.22% DT success and 10.11% CR, improving both metrics over OmTrackVLA.Source: §6.2 · p. 8

6.3 On-Policy Data Aggregation

  • During DAgger rollouts, the learner’s command is executed and privileged oracle commands become labels; retained samples are framed, close-range, or recovery states.Recovery samples cover up to two seconds after target loss, following at least one second of framed target visibility.Source: §6.3 · p. 9
  • DAgger fine-tuning uses 390,000 samples: 20% filtered rollout data and 80% oracle demonstrations, with each source balanced across tasks.Latest-policy samples are used first; training runs for one epoch.Source: §6.3 · p. 9
  • Repeated evaluation changed outcomes in approximately 23% of episodes, so small success-rate differences are interpreted cautiously without an aggregate uncertainty interval.The evaluation uses 1,405 episodes per task.Source: §6.3 · p. 9
  • DAgger raises DT SR from 55.80% to 58.22% (+2.42 percentage points) and lowers CR from 12.10% to 10.11% (−1.99 points); its SR is highest listed on DT and AT.Table 4 covers the full DT and AT validation sets, with 1,405 episodes per task.Source: §6.2 · p. 8, §6.3 · p. 10, Table 4 · p. 10

7 Collision Analysis

Collision analysis finds that DAgger’s DT success gain accompanies fewer contacts, concentrated in forward motion. The comparison uses one evaluation per model and classifies contacts by recent forward velocity, not responsibility.

Protocol and scope

  • The comparison uses one evaluation per model on the same 1,405 episode IDs per task; forward-motion contacts are classified by recent mean velocity of at least 0.15 m/s.The remaining category includes slow, stopped, and reverse motion and does not establish responsibility for contact.Source: §7 · p. 10

Collision outcomes

  • On DT, success rises from 55.80% to 58.22% as collisions fall from 170 to 142 and CR drops from 12.10% to 10.11%.Forward-motion collisions fall from 88 to 56, including a drop from 62 to 34 after step 30; below-threshold collisions change from 82 to 86.Source: §7 · p. 10

Interpretation

  • The observed collision reduction is concentrated in forward motion during distractor tracking, while the logs do not identify which person was hit.Improved visual and language context or close-range control are possible explanations, not isolated causes.Source: §7 · p. 10

8 Discussion

The discussion presents staged multimodal learning as promising for distractor tracking, while treating on-policy gains and real-world observations as preliminary evidence.

Multimodal learning for tracking among distractors

  • DT success rises from 41.54% with OmTrackVLA to 55.80% with the complete recipe, consistent with multimodal preparation benefiting tracking among competing candidates.The discussion notes that this comparison does not isolate the contribution of alignment or individual stages.Source: §8 · p. 10

Tracking rate and practical following quality

  • High tracking rate alone does not capture following quality: recovery history, obstacle detours, and avoiding contact can matter, while following distance was not always constant.These observations motivate emphasizing episode success and collision rate; they do not establish that lower tracking rate is inherently preferable.Source: §8 · p. 11

Deployment experience motivates prioritizing DT

  • Real-world observations suggest higher DT success provided better user experience, but this was qualitative rather than a controlled user study.The relationship between benchmark DT scores and user experience remains to be quantified.Source: §8 · p. 11

Promising preliminary on-policy refinement

  • DAgger adds 2.42 percentage points of DT success and reduces collision rate by 1.99 points; Table 6 separates forward-motion collisions from all remaining collisions.Forward-motion collisions use a mean forward-velocity threshold of 0.15 m/s before contact.Source: §8 · p. 11, Table 6 · p. 11

9 Conclusion

VLX-Go combines staged multimodal learning with waypoint prediction and on-policy data aggregation for embodied tracking. On EVT-Bench, it achieves strong distractor and ambiguous tracking results, while further DAgger investigation remains future work.

VLX-Go and its training recipe

  • VLX-Go connects RGB history and language to robot-centric waypoint trajectories using frozen visual features, multiscale temporal context, a transformer, and a lightweight head.Its recipe includes vision–language alignment, multimodal reasoning mid-training, tracking-specific waypoint supervision, and filtered-oracle on-policy data aggregation.Source: §9 · p. 11

Evaluation

  • On EVT-Bench, distractor and ambiguous tracking have success rates of 58.22% and 60.93% and collision rates of 10.11% and 10.39%, respectively.The conclusion highlights the distractor-tracking results as evidence for staged multimodal learning and preliminary on-policy refinement.Source: §9 · p. 11

Future work

  • Future work will test additional DAgger rounds, demonstration–rollout balance, label-filtering rules, target retention, collision avoidance, and transfer to real shared spaces.The conclusion frames these as directions for further investigation.Source: §9 · p. 12

Related papers

Loading 2610.19361v1…