Source-linked AI summary
RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches
Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, Priya Sundaresan, Peng Xu, Hao Su, Karol Hausman, Chelsea Finn, Quan Vuong, Ted Xiao
TL;DR
Robot policies still struggle to generalize to new tasks, especially when language does not describe motion-specific similarities. RT-Trajectory conditions an imitation policy on coarse trajectory sketches, achieving broader task generalization than the evaluated conditioning baselines while retaining visual guidance and flexibility.
Problem
Robot policy generalization to new tasks remains challenging, particularly when semantically unseen instructions describe motions similar to training data.
Method
RT-Trajectory trains a policy with hindsight 2D trajectory sketches that encode end-effector motion and gripper interactions, with sketches supplied by people or automated methods.
Results
67% success rate across seven manipulation skills never seen during training, outperforming three baseline methods.
Takeaways & Limitations
Trajectory sketches provide visual task guidance while preserving flexibility for handling ambiguous scenes and generalizing to novel semantics.
Takeaways & Limitations
The current approach assumes a stationary robot using only end-effector motions and does not yet support whole-body mobile manipulation.
Abstract
from arXiv · showhide
Generalization remains one of the most important desiderata for robust robot learning systems. While recently proposed approaches show promise in generalization to novel objects, semantic concepts, or visual distribution shifts, generalization to new tasks remains challenging. For example, a language-conditioned policy trained on pick-and-place tasks will not be able to generalize to a folding task, even if the arm trajectory of folding is similar to pick-and-place. Our key insight is that this kind of generalization becomes feasible if we represent the task through rough trajectory sketches. We propose a policy conditioning method using such rough trajectory sketches, which we call RT-Trajectory, that is practical, easy to specify, and allows the policy to effectively perform new tasks that would otherwise be challenging to perform. We find that trajectory sketches strike a balance between being detailed enough to express low-level motion-centric guidance while being coarse enough to allow the learned policy to interpret the trajectory sketch in the context of situational visual observations. In addition, we show how trajectory sketches can provide a useful interface to communicate with robotic policies: they can be specified through simple human inputs like drawings or videos, or through automated methods such as modern image-generating or waypoint-generating methods. We evaluate RT-Trajectory at scale on a variety of real-world robotic tasks, and find that RT-Trajectory is able to perform a wider range of tasks compared to language-conditioned and goal-conditioned policies, when provided the same training data.
1 INTRODUCTION
RT-Trajectory addresses the challenge of generalizing robot policies to new tasks by using coarse trajectory sketches as an expressive, practical conditioning modality. The framework uses trajectory labels that capture motion similarities and supports human or model-generated task specifications.
- Generalist robot policies must generalize beyond training objects, scenes, and motions because exhaustive robotic datasets are impractical.
- Language conditioning improves command generalization but may lack the specificity needed to describe novel motions, while one-hot conditioning misses task similarities.
- RT-Trajectory uses coarse 2D trajectories as a middle ground between expressive task specification and ease of use.The labels can be extracted automatically from demonstrations and can reflect similarities between different robot motions.
- Trajectory sketches provide a human-interpretable, richly expressive conditioning signal for policies across object-manipulation tasks.They can be drawn by humans or generated with modern image-editing models.
2 RELATED WORK
Prior robot-learning work studies semantic, visual, and other distribution shifts, while this paper focuses on task generalization through new combinations of states and actions or wholly unseen motions. Existing conditioning representations differ in how specifically they define goals and trajectories.
- Generalization in Robot Learning: Prior studies examine generalization to novel semantics, objects, visual shifts, action consistency, and combinations of distribution shifts.
- Generalization in Robot Learning: This paper additionally studies task generalization involving new combinations of seen states and actions or wholly unseen states and motions.
- Policy Conditioning Representations: Policy-conditioning methods vary along axes of goal specification and whether they condition on full trajectories or only end states.
- Policy Conditioning Representations: Language conditioning under-specifies end states, whereas goal-image conditioning can over-specify them with irrelevant scene details.
3 METHOD
RT-Trajectory trains a transformer-based imitation policy on hindsight trajectory sketches derived from demonstrations. Its representations combine projected end-effector motion with temporal, interaction, and optional height information, while inference sketches can come from people or automated generators.
- Overview: The method reuses demonstration data by extracting hindsight trajectory labels and trains a transformer-based policy conditioned on 2D trajectory sketches.The policy uses imitation learning with an RT-1-style architecture.
- Hindsight Trajectory Labels: Hindsight labels are built from projected 2D end-effector trajectories, color grading for temporal motion, and interaction markers.
- Hindsight Trajectory Labels: Interaction markers identify key gripper-closing and gripper-opening time steps on the projected end-effector path.
- Trajectory Representations: RT-Trajectory (2D) combines temporal trajectory information with gripper interaction markers, while RT-Trajectory (2.5D) additionally encodes end-effector height.
- Inference Trajectory Generation: At inference, trajectory sketches can be generated from human drawings, human videos, LLM-written code, or image-generation models.
4 EXPERIMENTS
The experiments test RT-Trajectory on seven unseen skills, diverse trajectory-generation methods, and realistic distribution shifts. RT-Trajectory substantially outperforms language- and goal-conditioned baselines and supports prompt-based behavioral adaptation.
- 4.1 EXPERIMENTAL SETUP: The evaluation uses 73K demonstrations covering 8 seen skills and 542 tasks, then tests seven unseen skills involving novel objects, workspaces, motions, and manipulation settings.The unseen skills include combining seen motions, precise placement, unseen heights and containers, towel folding, and swivel-chair interaction.
- 4.2 UNSEEN TASK GENERALIZATION: 67% success for RT-Trajectory (2.5D) and 50% for RT-Trajectory (2D) exceed RT-1 (16.7%), RT-2 (11.1%), and RT-1-Goal (26%).The 2.5D variant helps on tasks where height information reduces ambiguity, including picking-height inference.
- 4.3 DIVERSE TRAJECTORY GENERATION METHODS: Trajectory sketches generalize beyond manual drawings to human videos, LLM-generated waypoints, and image-generation models, despite differences from training hindsight trajectories.Human-video sketches are more squiggly, LLM trajectories are precise and linear, and image-model sketches are noisy.
- 4.4 EMERGENT CAPABILITIES: Changing coarse trajectory prompts reproducibly changes behavior modes, enabling visual prompt engineering without retraining or collecting more data in some fixed-scene cases.This provides a practical way to query a policy with alternative trajectory prompts when a scenario fails.
- 4.4 EMERGENT CAPABILITIES: RT-Trajectory performs tasks in new buildings and rooms with novel backgrounds, lighting, objects, layouts, and furniture geometries, with little to moderate prompt engineering.The evaluated cases combine visual distribution shifts with novel motion requirements.
5 CONCLUSION AND LIMITATIONS
RT-Trajectory uses 2D trajectory sketches to support generalization to unseen manipulation skills, achieving higher success than prior methods. The approach remains limited to stationary robots using end-effectors and does not yet encode stricter spatial constraints.
- 67% success on 7 unseen manipulation skills surpassed the best prior state-of-the-art result of 26%.The evaluation benchmarked RT-Trajectory against three baseline methods.
- RT-Trajectory conditions policies on 2D trajectory sketches to generalize across novel tasks and motions beyond training data.The sketches provide visual guidance while preserving flexibility in ambiguous scenes and novel semantics.
- The current system assumes a stationary robot and useful manipulation performed only with the end-effector.Extending the approach to mobile manipulation and whole-body control remains future work.
- Unseen skills generally show large variance in the relative height of their first gripper interactions compared with similar training trajectories.The comparison uses a self-query baseline with relative difference 0.0.
- Unseen-skill trajectories generally have larger Fréchet distances from their most similar training trajectories than training-skill queries.This indicates greater motion dissimilarity under the motion-similarity analysis.
- Future trajectory-sketch systems could support spatial regions where guidance is enforced more strictly, such as avoiding fragile objects.
A EXPERIMENT DETAILS
The experiment details distinguish seen training tasks from unseen evaluation tasks and document how task and scene counts are represented. Language instructions are used only for language-conditioned baselines.
- Seen training tasks are listed with descriptions, example language instructions, and counts of distinct tasks per skill.For example, separate object-specific tasks count as distinct tasks.
- Unseen evaluation tasks are listed with descriptions, example language instructions, and counts of evaluation scenes.The language instructions apply only to language-conditioned baselines.
A.3 QUANTITATIVE RESULTS FOR UNSEEN TASKS
The unseen-task quantitative-results section reports success rates for tasks conditioned on human-drawn trajectory sketches.
- Table 4 reports success rates for unseen tasks conditioned on human-drawn sketches.
B IMPLEMENTATION DETAILS FOR DIFFERENT INPUT MODALITIES
Human-drawn trajectory sketches are collected through a GUI that overlays 2D paths, interaction markers, and height annotations on the robot’s camera image. The evaluation pipeline can also search over sketch prompts and derive sketches from human videos.
- Human-drawn trajectory sketches: Users draw 2D trajectory curves over the current robot camera image through a graphical interface.The resulting sketch is made height-aware through the interface.
- Human-drawn trajectory sketches: Users add gripper opening or closing markers and assign height values to selected trajectory points.Heights for unannotated points are interpolated from annotated points.
- Prompt engineering: Prompt engineering explores trajectory-sketch prompts with a held-out 2.5D policy and saves the first successful sketch for each scene.If every attempt fails, the last attempted sketch is saved instead.
- Video-derived trajectories: Human videos are converted into trajectory inputs by detecting 21 hand landmarks, lifting them to 3D with depth, and interpolating end-effector poses.Key frames for grasping and releasing are manually annotated.
B.4 IMPLEMENTATION DETAILS FOR RT-1-Goal
RT-1-Goal uses the same network architecture as RT-Trajectory but replaces the trajectory sketch with a goal image for conditioning.
- RT-1-Goal uses a goal image instead of a trajectory sketch while retaining RT-Trajectory’s network architecture.
C MOTION DIVERSITY ANALYSIS
The motion-diversity analysis compares robot end-effector trajectories using the order-preserving, parameterization-independent Fréchet distance. Each waypoint is an end-effector center position measured with Euclidean distance, excluding orientation and interaction actions.
- The analysis measures similarity between end-effector motion trajectories with the Fréchet distance.It finds the minimum leash length for agents traversing two curves while preserving temporal order.
- Fréchet distance is suitable here because it preserves trajectory order while remaining independent of parameterization.
- Each trajectory consists of waypoint sequences compared recursively using a pointwise distance measure such as Euclidean distance.
- Each waypoint is the sensed end-effector center position, and the distance measure is Euclidean distance.Orientation and gripper opening or closing are not included in the similarity measure.
C.2 ADDITIONAL SAMPLES OF TRAJECTORY SIMILARITIES
Figure 14 provides additional examples comparing evaluation trajectories with their most similar trajectories from the training dataset.
- Figure 14 shows evaluation trajectories alongside their most similar trajectories from the training dataset.
D ADDITIONAL VISUALIZATION
The additional visualizations illustrate RT-Trajectory’s rollouts across unseen skills, generated trajectory prompts, realistic visual shifts, prompt engineering, retry behavior, and height-aware disambiguation. Together, they show qualitative task execution, prompt sensitivity, recovery after failure, and the role of depth information.
- RT-Trajectory is reported to handle simultaneous visual shifts involving new buildings, backgrounds, distractors, lighting, objects, furniture textures, heights, and geometries.These evaluations were conducted in realistic settings rather than controlled laboratory environments.
- Height-aware color grading in RT-Trajectory (2.5D) reduces the ambiguity of 2D sketches that cannot distinguish deeper from higher arm motion.The comparison shows 2.5D lifting the object where 2D moves it deeper.
- Additional examples visualize new skills, image-generated trajectories, and emergent behavior across articulated objects, circuitous motions, and novel rooms.
- Example rollouts cover seven unseen skills, including folding, drawer movement, chair picking, and swiveling.
- Image-generation models produce trajectory sketches for tasks such as opening a drawer, placing an orange, and moving a can near a bottle.
- Suboptimal trajectory prompts can fail, while improving the trajectory prompt under identical initial scene conditions can produce success.For elevated placements, the prompt should first draw a higher peak before descending to the target.
- RT-Trajectory can recover from an unsuccessful drawer-opening attempt by retrying with a different grasp location.In the illustrated case, it switches from the handle to the drawer edge.