PaperlayerView exact revision on arXiv

Source-linked AI summary

TrAct: Bridging Robot Control and Visual Prediction with Visual Tracks

Zhi Cao, Howard Ji, Kevin Zhang, Kuangzhi Ge, Li Fei-Fei, Jiajun Wu, Huang Huang

arXiv:2608.24101v1cs.RO

TL;DR

Robot actions are embodiment-specific and weakly aligned with image-space changes, limiting their use as world-model conditioning signals. TrAct instead uses visual tracks as an intermediate interface: VLAT proposes paired actions and tracks, TWM predicts their visual outcomes, and VLAC selects the best outcome. TrAct improves success from 27% to 55% in simulation and from 49% to 76% on real-world tasks compared with π0.5, while TWM improves video prediction over AWM.

  • Problem

    Robot actions are embodiment-specific and only weakly aligned with image-space visual changes, limiting their effectiveness as conditioning signals for robot world models.

  • Method

    TrAct jointly predicts candidate actions and visual tracks, conditions a world model on the tracks, and uses a vision-language reward model for action selection.

  • Results

    27% to 55% and 49% to 76%: TrAct improves success over π0.5 on LIBERO-INTEGRAL and real-world tasks, while TWM consistently improves future prediction over AWM.

  • Takeaways & Limitations

    Visual tracks provide an effective bridge between embodiment-specific control and embodiment-agnostic visual prediction, supporting stronger generalization.

Abstract

from arXiv · show

Robot actions are inherently embodiment-specific and only weakly aligned with image-space visual changes, limiting their effectiveness as conditioning signals for robot world models. In contrast, visual tracks provide an embodiment-agnostic representation of how task-relevant points move through a scene, offering dense image-space guidance for accurate and spatially precise future video prediction. Building on this observation, we propose TrAct, a world-model-based robot decision-making framework that uses visual tracks as an intermediate interface between control and prediction. TrAct consists of three components: a Vision-Language-Action-and-Track model (VLAT) that jointly predicts candidate actions and corresponding visual tracks from the current observation and language instruction; a track-conditioned world model (TWM) that predicts future visual outcomes conditioned on the proposed tracks; and a vision-language reward model (VLAC) that scores the predicted outcomes. At inference time, VLAT generates candidate action-track pairs, TWM rolls out their visual consequences, and VLAC selects the track whose predicted outcome best satisfies the instruction; the action paired with the selected track is then executed by the robot. Experiments on the proposed LIBERO-INTEGRAL benchmark and real-world Franka manipulation show that TrAct improves success rates from 27% to 55% in simulation and from 49% to 76% on real-world tasks compared with the strong VLA baseline $π_{0.5}$. Furthermore, TWM consistently improves video prediction quality over the action-conditioned world model (AWM). These results demonstrate that visual tracks provide an effective shared interface between robot control and visual prediction, enabling more accurate world modeling and stronger robot generalization.

1 Introduction

TrAct addresses the weak alignment between embodiment-specific robot actions and image-space visual changes by using visual tracks as an interface between policy learning and world-model decision making. Its VLAT, TWM, and VLAC components jointly propose, predict, and evaluate action–track candidates.

  • Motivation: Robot actions are embodiment-specific and weakly aligned with image-space changes, making them limited interfaces between control and visual prediction.Action-conditioned world models must infer dense visual futures from sparse, robot-specific commands.
  • Approach: VLAT jointly predicts candidate actions and corresponding visual tracks from observations and language instructions.This produces paired candidates for subsequent predictive evaluation.
  • Decision making: TWM predicts future visual outcomes from proposed tracks, while VLAC scores those outcomes against the task instruction.The selected track outcome determines which paired robot action is executed.
  • Results: 27% to 55% and 49% to 76%: TrAct improves success over π0.5 on LIBERO-INTEGRAL and real-world tasks, respectively.The framework is evaluated in simulation and on physical robots, while TWM improves future prediction over action-conditioned alternatives.
  • Approach: TrAct uses visual tracks as a shared intermediate representation between robot policies and world models.The framework is designed to improve both planning and future prediction.

2 Related Work

Prior work uses world models and visual representations to connect robot decisions with predicted scene evolution, but action conditioning remains poorly aligned with image-space motion. TrAct builds on visual tracks as dense, embodiment-agnostic guidance for prediction and control.

  • Embodied Video Generation World Models: Robot world models evaluate candidate decisions, but action conditioning remains poorly aligned with image-space change.Actions are embodiment-specific and their visual effects depend on scene geometry, object configuration, and contact dynamics.
  • Embodied Video Generation World Models: Visual tracks directly specify point motion and provide dense spatial guidance for video prediction.This contrasts with models that may ignore erroneous actions and hallucinate success from visual priors.
  • Visual Tracks and Embodiment-Agnostic Action Representations: Structured motion representations such as 2D point tracks and optical flow separate motion from visual rendering.Prior work decodes these representations into executable actions for transfer across embodiments and from human video to robots.
  • Vision-Language-Action Models and Closed-Loop Control: TrAct extends closed-loop VLA control by using visual tracks to connect policy outputs with predictive world-model evaluation.The framework combines candidate generation, visual rollout, and reward-guided selection rather than relying only on open-loop action generation.

3 Methods

TrAct generates paired action–track candidates, predicts their visual consequences with a track-conditioned world model, and selects among them using a vision-language reward model. VLAT jointly predicts actions and tracks, while TWM encodes tracks as spatial controls for multi-view video generation.

  • Inference Pipeline: TrAct decomposes decision making into candidate generation, track-conditioned visual rollout, and reward-based outcome scoring.The reward model is implemented with InternVL2 following the VLAC framework.
  • Policy Architecture: VLAT modifies a flow-matching VLA policy to jointly output executable robot actions and 2D point tracks.The policy is built on π0.5 and produces action–track pairs for candidate evaluation.
  • Policy Architecture: VLAT is trained with a joint flow-matching objective over ground-truth actions and tracks.Tracks include gripper mesh vertices and uniformly sampled background and object points, with different point sets across camera views.
  • Track-Conditioned World Model: TWM augments Stable Video Diffusion with ControlNet to encode predicted track coordinates as spatial control maps.Agent-view and wrist-view tracks use separate channels, enabling view-specific motion while maintaining cross-view temporal consistency.
  • Training: TWM and AWM are pretrained on 76K DROID trajectories and finetuned on the same LIBERO and real-world datasets for fair comparison.The shared training setup supports comparison between track and action conditioning.
  • Inference: At inference, VLAT proposes diverse candidates, TWM rolls out each track-conditioned video, and the reward model scores each rollout before execution.Temperature-Scaled Resampling controls candidate diversity, and the selected pair’s action is executed by the robot.

4 Simulation Experiments

Simulation experiments evaluate TrAct on standard LIBERO and the harder LIBERO-INTEGRAL benchmark, including distribution and cross-embodiment shifts. Track-conditioned prediction improves video quality and supports stronger task success, with TrAct reaching 0.55 average success on LIBERO-INTEGRAL.

  • Benchmark Setup: LIBERO-INTEGRAL combines robustness variations and UR5 cross-embodiment transfer to evaluate generalization beyond near-saturated standard LIBERO tasks.The benchmark contains 20 tasks spanning object, camera, task, robot-initialization, and cross-embodiment variations.
  • Video Prediction: TWM outperforms AWM on all five video metrics across both camera views, producing more spatially accurate and temporally coherent predictions.In simulation, TWM achieves PSNR 24.51 vs. 15.12 on agent view and reduces LPIPS from 0.438 to 0.106.
  • Task Success: 98.3% average success is achieved by TrAct on the standard LIBERO suites, compared with 96.8% for π0.5 and 98.0% for VLAT.The small gains reflect near-saturated performance on the standard tasks.
  • Task Success: 0.55 average success is achieved by TrAct on LIBERO-INTEGRAL, improving over π0.5 at 0.27, VLAT at 0.44, and VLAT+AWM at 0.49.TrAct records the highest success rates for Object 0.60, Camera 0.60, RobotInit 0.65, and Cross-Embodiment 0.50 variations.
  • Generalization: Track conditioning yields consistent gains over action conditioning across Object, Camera, RobotInit, and Cross-Embodiment shifts.The reported comparisons are 0.60 vs. 0.50, 0.60 vs. 0.45, 0.65 vs. 0.55, and 0.50 vs. 0.44, respectively.

5 Physical Robot Experiments

Physical experiments evaluate TrAct on unseen Franka Panda manipulation tasks under regular and visually shifted backgrounds. TWM improves future-video quality over AWM, while TrAct achieves the strongest downstream success through track-conditioned action selection.

  • Physical Setup: The physical setup uses a Franka Emika Panda with agent-view and wrist-view RGB cameras and 400 demonstrations across four training tasks.Evaluation uses five unseen manipulation tasks under regular and visually distinct backgrounds.
  • Video Prediction: TWM outperforms AWM on all five video metrics across both camera views in the real-world evaluation.On wrist view, LPIPS improves from 0.372 to 0.225 and FVD from 246 to 140.
  • Robot Results: TrAct averages 0.76 success versus 0.66 for VLAT+AWM across the five unseen physical tasks.Both world-model variants improve over VLAT alone at 0.55.
  • Robustness: Under unseen backgrounds, VLAT+AWM drops to 0.4 while TrAct remains at 0.7.The gap also appears without domain shift on cabinet closing, where TrAct scores 0.8 versus 0.6.
  • Robustness: Track conditioning is consistently stronger than action conditioning, with similar +20 pp gains reported on the greenblock and pouring tasks.The results suggest that track conditioning remains more robust to background changes than action-conditioned video generation.

6 Conclusion

TrAct uses visual tracks as a shared interface between robot policy learning and world-model-based decision making. Its inference loop proposes action–track candidates, predicts their visual consequences, and executes the action whose rollout best satisfies the instruction.

  • TrAct uses visual tracks as a shared interface between robot policy learning and world-model-based decision making.
  • VLAT jointly predicts actions and visual tracks, while TWM conditions future prediction on tracks rather than raw actions.
  • At each decision step, VLAT proposes K diverse action–track candidates and TWM imagines their visual consequences.
  • VLAC selects the action chunk whose predicted rollout best satisfies the instruction, then the robot executes it.

B Pretraining Data and Unified Track Representation

TrAct represents visual motion with fixed, embodiment-agnostic track slots that can be jointly supervised across heterogeneous robot and human datasets. Tracks receive consistent supervision, while action supervision is retained only where executable robot action labels exist.

  • Unified Track Representation: Fixed view-specific track slots represent possible gripper, hand, background, or scene-point tracks across heterogeneous datasets.Unavailable track types and camera streams are masked rather than assigned embodiment-specific meanings.
  • Unified Track Representation: DROID and EgoDex activate different combinations of gripper, hand, and background-point tracks according to their cameras and annotations.DROID uses seven gripper points per view, while EgoDex uses fourteen hand points plus twenty-five background points.
  • Pretraining Objective: The training objective adds action loss only for robot data, controlled by whether robot action annotations are available.
  • Pretraining Objective: Tracks provide a common supervision signal across datasets, whereas actions are supervised only when their control format is valid and executable.EgoDex supervises tracks only; robot datasets supervise both tracks and actions.
  • World-Model Comparison: TWM predictions are more spatially accurate and motion-coherent than AWM predictions across agent-view and wrist-view cameras.AWM more often shows spatial drift, distorted object shapes, and less coherent motion, especially in wrist view.

E Physical Experiment Setup

The physical experiments train on four Franka manipulation tasks and test five unseen tasks, including task and background shifts. Qualitative comparisons also examine track- and action-conditioned predictions across agent-view and wrist-view cameras.

  • Experiment Setup: Training uses 400 demonstrations at 15 Hz across four tasks, while evaluation uses five manipulation tasks unseen during training.
  • Experiment Setup: The evaluated shifts include new objects, altered cabinet states, and changed pouring configurations.
  • Qualitative Comparison: TWM produces more spatially accurate and motion-coherent frames than AWM across both agent-view and wrist-view cameras.
  • Experiment Setup: The physical setup uses base and wrist-view cameras, trains on four tasks, and tests five out-of-distribution tasks with an unseen background for each task.
  • Benchmark Context: LIBERO-INTEGRAL contains robustness and cross-embodiment tasks designed to evaluate generalization under distribution and embodiment shifts.Its cross-embodiment setting replaces the Franka Panda with a UR5 robot.

F.2 Per-Task LIBERO-INTEGRAL Results

TrAct achieves the strongest aggregate LIBERO-INTEGRAL performance across robustness and cross-embodiment settings, with the best or tied-best result on 18 of 20 tasks.

  • Aggregate Results: 0.55 average success rate is achieved by TrAct on LIBERO-INTEGRAL, exceeding π0.5, VLAT, and VLAT+AWM by 0.28, 0.11, and 0.06 absolute points.
  • Evaluation Coverage: The per-task evaluation covers object swaps, object, task, and camera variations, robot-initialization changes, and UR5 cross-embodiment transfer.
  • Per-Task Results: TrAct achieves the best or tied-best performance on 18 out of 20 LIBERO-INTEGRAL tasks.

F.3 Statistical Robustness and Candidate Scaling

Three-seed evaluation supports the robustness of track conditioning, while candidate scaling improves selection success with diminishing returns and higher rollout cost.

  • Statistical Robustness: Three independent LIBERO-INTEGRAL seeds yield non-overlapping 95% confidence intervals for TrAct and VLAT+AWM.The evaluation keeps checkpoints, candidate counts, and selection hyperparameters fixed across seeds.
  • Candidate Scaling: Success improves monotonically as sampled candidates increase from K=5 to K=20, but gains diminish.The increase is 2 points from K=5 to K=10 and 1 additional point from K=10 to K=20.
  • Candidate Scaling: K=5 recovers most of the selection benefit and is a reasonable operating point when inference latency matters.World-model rollout cost grows with K, whereas K=20 is used for the strongest main simulation setting.

G Larger Pretraining Mixture Experiments

TrAct+ tests whether expanding the real-world pretraining mixture improves performance on difficult UR5 cross-embodiment tasks, with results showing a higher average success rate.

  • Training Mixture: TrAct+ expands pretraining with 76K DROID, 60K BridgeData V2, and approximately 300K processed EgoDex clips.The larger mixture increases embodiment diversity and manipulation data coverage beyond the main model’s training mixture.
  • Results: 0.38 average success rate is achieved by TrAct+, up from 0.32 for TrAct on five difficult UR5 cross-embodiment tasks.π0.5, TrAct, and TrAct+ are compared under the same evaluation protocol.
  • Evaluation Setting: The five evaluated tasks are selected from LIBERO-INTEGRAL’s most challenging UR5 cross-embodiment tasks for TrAct.The supplied rollout figures illustrate additional simulation, UR5, and task-specific examples.
Loading 2608.24101v1…