PaperlayerView exact revision on arXiv

Source-linked AI summary

TraceGen: World Modeling in 3D Trace Space Enables Learning from Cross-Embodiment Videos

Seungjae Lee, Yoonkyo Jung, Inkook Chun, Yao-Chih Lee, Zikui Cai, Hongjia Huang, Aayush Talreja, Tan Dat Dao, Yongyuan Liang, Jia-Bin Huang, Furong Huang

arXiv:2511.21690v1cs.ROcs.CVcs.LG

TL;DR

Learning robot tasks from few demonstrations is difficult, and videos of humans or other robots are hard to reuse across embodiment, camera, and scene differences. TraceGen predicts motion in 3D trace-space and is pretrained on TraceForge’s heterogeneous video corpus. It reports few-shot success in robot-to-robot and human-to-robot transfer, with inference substantially faster than video-generation-based world models.

  • Problem

    Learning new robot tasks from few demonstrations remains difficult, while abundant videos from other embodiments are hard to reuse across embodiment, camera, and scene differences.

    Source: §1 · p. 2

  • Method

    TraceGen predicts future scene-level motion in 3D trace-space and is pretrained on heterogeneous human and robot videos converted into consistent traces by TraceForge.

    Source: §1 · p. 2

  • Results

    80% success across four tasks with five target-robot videos and 67.5% real-robot success from five uncalibrated human demos; inference is 50–600× faster than video-generation-based world models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results suggest that reasoning in trace-space provides an effective inductive bias for cross-embodiment manipulation.

    Source: §6 · p. 9

    Zero-shot generation is not fully reliable for novel embodiments or unseen environments and can yield physically infeasible trajectories.

    Source: §7 · p. 9

Abstract

from arXiv · show

Learning new robot tasks on new platforms and in new scenes from only a handful of demonstrations remains challenging. While videos of other embodiments - humans and different robots - are abundant, differences in embodiment, camera, and environment hinder their direct use. We address the small-data problem by introducing a unifying, symbolic representation - a compact 3D "trace-space" of scene-level trajectories - that enables learning from cross-embodiment, cross-environment, and cross-task videos. We present TraceGen, a world model that predicts future motion in trace-space rather than pixel space, abstracting away appearance while retaining the geometric structure needed for manipulation. To train TraceGen at scale, we develop TraceForge, a data pipeline that transforms heterogeneous human and robot videos into consistent 3D traces, yielding a corpus of 123K videos and 1.8M observation-trace-language triplets. Pretraining on this corpus produces a transferable 3D motion prior that adapts efficiently: with just five target robot videos, TraceGen attains 80% success across four tasks while offering 50-600x faster inference than state-of-the-art video-based world models. In the more challenging case where only five uncalibrated human demonstration videos captured on a handheld phone are available, it still reaches 67.5% success on a real robot, highlighting TraceGen's ability to adapt across embodiments without relying on object detectors or heavy pixel-space generation.

1. Introduction

TraceGen addresses the difficulty of learning new robot tasks from scarce demonstrations when videos from other embodiments are hard to reuse across camera and scene differences. It learns motion in 3D trace-space from a large heterogeneous video corpus and reports few-shot success in both robot-to-robot and human-to-robot settings.

Motivation

  • Robots need task-specific demonstrations that are slow and costly to collect, while human videos are abundant but differ in embodiment, camera, and scene.Pixel-space and language-token world models also have control-relevant limitations and expensive inference.Source: §1 · p. 2

Key insight

  • Trace-space represents scene-level motion as 3D trajectories, retaining geometric structure while discarding appearance and backgrounds.The representation is intended to support reuse across embodiments, environments, and tasks.Source: §1 · pp. 2–3

Approach

  • TraceGen predicts future motion directly in trace-space, learning a transferable motion prior from human and heterogeneous robot videos without object detectors or heuristic filtering.The approach avoids heavy generative rendering and adapts with only a few warm-up videos.Source: §1 · pp. 2–3

Data engine

  • TraceForge converts heterogeneous videos into consistent 3D traces, producing 123K videos and 1.8M observation–trace–language triplets.Its pipeline compensates for camera motion and retargets speed across embodiments.Source: §1 · pp. 2–3, Figure 2 · p. 2

Results in low-data regimes

  • 80% success across four tasks with five target-robot videos and 67.5% real-robot success from five uncalibrated human demos; inference is 50–600× faster than video-generation-based world models.The human demonstrations were recorded with a handheld phone in a different scene.Source: §1 · p. 3

2. Related Work

Embodied world models predict future pixels, language tokens, or motion traces, each with different control-relevant trade-offs. TraceGen extends trace prediction to scene-level 3D motion and complements implicit world models, while trace representations can feed downstream planning, tracking, and policy learning.

2.1. Embodied World Models

  • Pixel generators spend capacity on irrelevant backgrounds and textures and can hallucinate geometry, while language-token outputs lack the resolution for fine object motion.Figure 3 illustrates geometry or affordance hallucination and inadequate token motion detail.Source: §2.1 · p. 3, Figure 3 · p. 3
  • Prior trace models largely use static in-lab data and 2D traces; some 3D variants track only objects, requiring detection and filtering while missing robot motion.These limitations can yield error cascades and an incomplete physical representation.Source: §2.1 · p. 3
  • TraceForge supports in-the-wild training, and TraceGen predicts scene-level 3D trajectories of robots and objects without heuristic filtering or bounding boxes.The training data is reported as 15× larger than prior image–trace–language triple data [62].Source: §2.1 · p. 3
  • Implicit world models shape latent dynamics without decoding future traces, but typically lack metrically consistent scene-level 3D motion; TraceGen explicitly models that motion and can complement them.The passage notes that implicit objectives could in principle be layered onto TraceGen’s encoder.Source: §2.1 · p. 3

2.2. Trace for Robot Manipulation

  • Predicted traces can support planning, tracking-based execution, policy supervision, or observation; this work uses a basic tracking controller as a minimal execution demonstration.Developing more sophisticated policies is left for future work.Source: §2.2 · p. 4

TraceForge: Dataset Construction

TraceForge turns heterogeneous human and robot videos into trace-annotated data, using event segmentation, 3D tracking, camera-frame alignment, and temporal normalization to build consistent training examples.

TraceForge: Dataset Construction

  • TraceForge curates 123K episodes and about 1.8M observation–trace–language triplets from eight sources spanning human, single-arm, and bimanual robot demonstrations.The pipeline converts heterogeneous videos into trace annotations paired with multimodal observations and language.Source: TraceForge: Dataset Construction · p. 4

3.1. Event Chunking and Instruction Generation

  • Event chunks use available start–end indices or motion-based frame selection, and each chunk receives three VLM-generated instruction variants.The variants are an imperative, a multi-step decomposition, and a natural-language request; existing human-written instructions are retained and augmented.Source: §3.1 · p. 4

3.2. 3D Point Tracking with Camera Pose and Depth Prediction

  • For each event chunk, a 20 × 20 keypoint grid is tracked, with each 3D trace represented by image-plane coordinates and corresponding depth.This screen alignment supports co-training and consistent supervision across 2D and 3D modalities; camera pose and depth are estimated to reconstruct traces.Source: §3.2 · pp. 4–5

3.3. World-to-camera Transformation

  • Traces are transformed into the reference camera frame, then represented as screen-aligned pixel coordinates paired with camera-coordinate depth.The reference frame maintains point-of-view consistency across time.Source: §3.3 · p. 5

3.4. Speed Retargeting

  • Speed retargeting resamples each 3D path to a fixed length L by normalized cumulative arc length while preserving its relative motion profile.Uniform resampling targets align trace lengths across embodiments without distorting local velocity patterns.Source: §3.4 · p. 5

TraceGen: Architecture and Training

TraceGen predicts future scene motion in 3D trace-space using multimodal visual and language conditioning. Its flow-based decoder learns keypoint increments through a stochastic-interpolant framework and generates trajectories by ODE integration.

4.1. Multi-Encoder Feature Extraction

  • Three frozen visual encoders process RGB geometry, RGB semantics, and depth, while a frozen T5-base encoder represents task instructions.DINOv3 supplies spatially aware features, SigLIP semantically aligned features, and a learnable stem adapts depth maps to SigLIP's input space.Source: §4.1 · p. 6, §4.2 · p. 7
  • The visual streams are concatenated and projected to 768 dimensions, then combined with text tokens as conditioning for the trace decoder.The conditioning input contains visual and text tokens; training updates the fusion layer and decoder while keeping encoders frozen.Source: §4.1 · p. 6, §4.2 · p. 7

4.2. Flow-based Trace Decoder

  • The decoder adapts a 3D transformer to predict scene traces from 20×20 spatial keypoints tracked across 32 future timesteps, with conditioning injected through Adaptive LayerNorms.The keypoints are processed in 2×2 patches, yielding 10×10 spatial tokens per timestep.Source: §4.2 · p. 6
  • Rather than predict absolute grid coordinates, TraceGen predicts keypoint increments that implicitly capture the scene's 3D motion.The trace is represented as increments between temporal observations of a reference grid.Source: §4.2 · p. 6
  • The model learns a conditional velocity field over an interpolation between noise and ground-truth traces, using a linear schedule that yields constant velocity.At test time, it generates trajectories with 100-step ODE integration using multimodal vision-language conditioning.Source: §4.2 · pp. 6–7

5. Experiments

TraceGen is evaluated for real-world manipulation, human-to-robot transfer, and the contribution of cross-embodiment pretraining. It combines high few-shot success with faster inference than video-based world models.

5. Experiments

  • Experiments test real-world manipulation on four Franka Research 3 tasks, using one RGB-D frame and a language instruction to predict a trace that inverse kinematics converts to joint commands.The evaluations address effectiveness and efficiency, human-to-robot transfer, and the role of pretraining and warm-up.Source: §5.1 · p. 7, §5 · p. 7

5.1. Performance and Efficiency Comparison in Real-World Experiment

  • 80% success across four tasks after five target-robot videos, with TraceGen achieving the best success–efficiency combination against video and trace-based baselines.The evaluation uses a five-video warm-up despite variations in object layouts and initial robot poses.Source: Figure 7 · p. 8, §5.1 · p. 8
  • TraceGen runs 3.8× faster than trace-generation baselines and over 50× faster than large video-generation models; NovaFlow (Wan2.2) requires more than 600× longer inference time, highlighting pixel-space scaling difficulty for real-time robotics.Source: §5.1 · p. 8

5.2. Human–Robot Skill Transfer

  • 67.5% success across four tasks followed fine-tuning on five uncalibrated handheld human videos, while the From Scratch model achieved 0%.The videos varied in viewpoints, backgrounds, and object layouts; no target-robot demonstrations were used.Source: Figure 8 · p. 8, §5.2 · p. 8

5.3. Role of Pretraining and Warmup

  • With five target-robot videos, the pretrained model achieves 80% overall success versus 25% from scratch; with fifteen videos, the rates are 82.5% and 30%, respectively.The results indicate that most performance comes from pretraining, while warm-up aligns motion priors with task-specific configurations.Source: §5.3 · p. 8, Table 1 · p. 9
  • Under the same five-video warm-up, SSV2 pretraining yields 25% success and Agibot pretraining yields 45%, both below full cross-embodiment pretraining.The compared sources are human hand-centric SSV2, robot-centric Agibot, and the full cross-embodiment dataset.Source: Table 2 · p. 9, §5.3 · p. 9

6. Conclusion

TraceGen predicts future manipulation motion in compact 3D trace space, and TraceForge converts heterogeneous videos into consistent trace annotations. The paper reports strong real-world success with few demonstrations and faster inference than video-based approaches.

6. Conclusion

  • With 5–15 demonstrations, TraceGen achieves 80–82.5% success on real-world tasks while running 50× faster than video-based approaches.The conclusion presents trace-space reasoning as an effective inductive bias for cross-embodiment manipulation, with computational and sample efficiency.Source: §6 · p. 9
  • TraceForge processes heterogeneous video sources into consistent 3D traces by compensating for camera motion and normalizing embodiment-specific speeds.The resulting training corpus comprises 123K episodes.Source: §6 · p. 9
  • TraceGen represents manipulation as 3D traces of scene points, providing a unified motion representation across human hands and robot arms.The model predicts future motion in compact 3D trace space rather than pixel space.Source: §6 · p. 9

7. Limitations and Future Work

The paper identifies limits in trajectory generation, demonstration quality, and the scope of tested embodiments and tasks. It also notes that some noisy demonstrations remain after filtering.

7. Limitations and Future Work

  • Alternative interpolation schedules and explicit control of trajectory modes for ambiguous tasks remain unexplored.The current approach uses linear interpolation with ODE integration and samples diverse trajectories through different noise initializations.Source: §7 · p. 9
  • Some source videos contain inefficient or corrective motions, and noisy demonstrations remain despite additional filtering.These demonstrations introduce suboptimal supervision signals.Source: §7 · p. 9
  • Zero-shot generation is not fully reliable for novel embodiments or unseen environments, and plausible trajectories can be physically infeasible.Fine-grained manipulation trajectories may also lack enough detail for precise robot execution; the limits of trace-space abstraction for very different robot types remain untested.Source: §7 · p. 9

B. 3D Trace Extraction Accuracy of TraceForge

TraceForge’s 3D trace extraction is checked against robot end-effector trajectories from forward kinematics. Across nine teleoperated episodes, the reported endpoint error is below 2.3 cm on every axis.

B. 3D Trace Extraction Accuracy of TraceForge

  • The accuracy check is intended to establish whether TraceForge’s reconstructed 3D trajectories reflect real robot motion and provide training supervision.The authors characterize the resulting supervision as centimeter-level motion accuracy.Source: B. 3D Trace Extraction Accuracy of… · p. 13
  • TraceForge’s traces are compared with forward-kinematics ground-truth end-effector trajectories across nine teleoperated episodes.For each episode, the predicted grid-point path whose first-frame 2D projection is closest to the end-effector is treated as the corresponding trace.Source: B. 3D Trace Extraction Accuracy of… · p. 13
  • Sub–2.3 cm endpoint error on all axes is reported across episodes lasting 13–24.5 seconds, with an average displacement of 70.96 cm.Table 3 reports absolute endpoint error along the x, y, and z axes in camera coordinates.Source: Table 3 · p. 13, B. 3D Trace Extraction Accuracy of… · p. 13

C. Model Training Details

TraceGen uses DINOv2 and SigLIP for RGB features, a depth encoder for geometry, and T5 for text. Its training freezes the encoders while learning fusion and trace prediction; evaluation also uses a pixel-wise depth correction procedure.

C.1. Encoder Freezing Strategy

  • Freezing DINOv2, SigLIP, and T5 leaves the fusion layer and decoder trainable, retaining pretrained representations while reducing trainable parameters, accelerating training, and improving generalization.The retained representations include visual geometry and semantics, vision-language alignment, and linguistic task representations.Source: §C.1 · p. 14

D.1. Evaluation setup

  • Evaluation uses one RGB-D observation and a text instruction to predict trajectories, then converts them to joint-space commands through inverse kinematics after camera-to-base transformation.The Franka Research 3 performs four tasks; prediction uses 100-step ODE integration, with guidance for Clothes and Brush but not Block and Ball.Source: §D.1 · p. 14

D.2. Depth rescaling

  • A pixel-wise depth-rescaling map is smoothed with Gaussian blur to correct trace z values, because a single median-based scale often errs when depth varies substantially.The procedure compares predicted and sensor depth maps across pixels; the median-based approach was found inadequate in these settings.Source: §D.2 · p. 14

D.3. Implementation details for the baselines

The baseline implementations adapt their input processing and components for comparison with TraceGen. The accompanying failure analysis categorizes trial outcomes and visualizes how attempts progress into different failure modes.

D.3. Implementation details for the baselines

  • 3DFlowAction’s single-image detector can miss the target or include the gripper, so its filtering pipeline uses minimally moving image sequences and manually checked bounding-box alignment.The sequence helps identify the gripper and match the filtered region to the ground-truth object mask.Source: §D.3 · p. 14
  • NovaFlow removes grasp proposal because the tasks do not involve grasping, and replaces MegaSaM with the shared fine-tuned VGGT depth and pose predictor for comparison.The robot starts already holding the object; the replacement predictor is described as similarly accurate and faster by avoiding expensive 3D optimization.Source: §D.3 · pp. 14–15
  • Wan 2.2 latency uses multiple GPUs because it cannot fit on one A5000, while Veo 3.1 latency uses average API response time; both choices favor the baselines.Other baseline runtimes were measured on an NVIDIA RTX A5000.Source: §D.3 · p. 16

E. Failure modes analysis

  • Failure analysis categorizes every executed trial across four tasks and four model configurations, then uses Sankey diagrams to show transitions from attempts to outcome modes.The diagrams expose early errors such as incorrect approach and later ones such as partial completion or final-alignment drift.Source: E. Failure modes analysis · pp. 16–17

E.1. Warmup data

The experiments use a lightweight warmup stage to translate embodiment-agnostic traces into the target robot’s action space, with robot-domain and human-to-robot regimes.

E.1. Warmup data

  • Robot-to-robot warmup provides five in-domain robot demonstrations per task, with object layouts and initial robot poses differing from evaluation.This setup prevents warmup clips from simply memorizing evaluation configurations.Source: §E.1 · p. 17
  • Human-to-robot warmup provides five uncalibrated handheld-phone videos per task, each 3–4 seconds long and differing from robot evaluation in background, lighting, embodiment, and object placement.The figures show the first and final frames of each demonstration across Clothes, Block, Brush, and Ball.Source: §E.1 · p. 17
  • The robot warmup figure shows the first frame of each demonstration above its final frame for all four tasks.The task set is Clothes, Block, Brush, and Ball.Source: Figure 14 · p. 17

E.2. Long horizon experiments

The long-horizon Sorting experiment tests whether TraceGen can compose placing behaviors across four sequential subtasks. The reported comparison favors pretrained TraceGen over training from scratch on later steps.

E.2. Long horizon experiments

  • Sorting requires four consecutive placements: left trash to red, pink and blue LEGO blocks to green, then right trash to red.Five human teleoperation demonstrations of the full process are segmented into four primitive subtasks for warmup.Source: §E.2 · p. 17
  • A predefined scripted grasp policy picks up each object and starts TraceGen’s placing behavior from a consistent home configuration.TraceGen models the 3D trace component and does not include an external grasping module.Source: §E.2 · pp. 17–18
  • Pretrained TraceGen maintains high success on later subtasks, while the scratch model’s success degrades markedly over time; the table reports per-step rates across 10 rollouts.The pretrained model occasionally fails the first trash placement; the table orders subtasks from left to right in temporal sequence.Source: Table 4 · p. 18, §E.2 · p. 18
  • The authors conclude that pretrained motion priors support stable composition of primitive placing behaviors and mitigate error accumulation, despite no explicit optimization for extended planning.They attribute reliable skill stitching over longer horizons to the compact 3D trace representation.Source: §E.2 · p. 18
Loading 2511.21690v1…