PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents

Tongsheng Ding, Zhen Luo, Yixuan Yang, Boyu Wang, Luyang Xie, Jinyu Yang, Feng Zheng

arXiv:2608.00486v1cs.CV

TL;DR

Fine-grained language-to-motion supervision is scarce, while existing object-trajectory predictors often require privileged inputs or costly video-generation pipelines. DreamTraj addresses this by reading 6-DoF motion from intermediate representations of a frozen video diffusion model using one RGB image and an instruction, achieving state-of-the-art translation and rotation prediction and running 4.6× faster than generate-then-extract.

  • Problem

    Object-trajectory prediction lacks fine-grained language-to-motion supervision, while existing predictors require privileged inputs or costly video-based perception.

  • Method

    DreamTraj predicts 6-DoF trajectories from one RGB image and an instruction by decoding motion from intermediate features of a frozen image-to-video diffusion model.

  • Results

    DreamTraj achieves state-of-the-art translation and rotation prediction while delivering a 4.6× end-to-end speedup over generate-then-extract pipelines.

  • Takeaways & Limitations

    DreamTraj provides an efficient single-image, language-conditioned route to object-level 6-DoF motion prediction without privileged geometric inputs.

Abstract

from arXiv · show

Accurate prediction of object trajectories during manipulation is essential for closing the perception-action loop. Progress is limited on two fronts: available datasets lack fine-grained language-to-motion annotations, and existing predictors either rely on privileged inputs such as video, depth, or CAD models, or recover motion from fully generated videos through costly, error-prone perception pipelines. We close the supervision gap with the MOVE dataset, 5,038 object-centric egocentric trajectories, each paired with a fine-grained natural-language instruction rather than a coarse verb-noun label. We further propose DreamTraj, which predicts a 6-DoF object trajectory from a single RGB image and a task instruction, requiring no video, depth, or CAD model at inference: rather than generating a video, it reads motion from the internal representations of a frozen image-to-video diffusion model at an early denoising step. A lightweight flow-matching Reader decodes query-key attention tracks and pooled hidden states into relative 6-DoF poses. To our knowledge, this is the first approach to directly decode object 6-DoF trajectories from intermediate video diffusion representations rather than generated pixels. DreamTraj sets a new state of the art on both translation and rotation against forecasters that consume multi-frame or privileged inputs, and runs 4.6x faster than generate-then-extract pipelines.

1 Introduction

DreamTraj addresses the difficulty of obtaining language-grounded 6-DoF object trajectories by introducing the MOVE dataset and predicting motion from a single RGB image and instruction. It decodes trajectories from intermediate representations of a frozen video diffusion model, achieving state-of-the-art translation and rotation prediction without explicit video generation.

  • Motivation: Object-centric 6-DoF trajectories provide an executable interface for anticipating manipulated-object motion, enabling advance planning, object-level imitation, and action verification.
  • Problem: 3D trajectory supervision is difficult and expensive because capturing metric pose sequences requires depth, CAD models, multi-view rigs, or markers, whereas 2D observations and language are abundant.
  • Dataset: 5,038 human-inspected 6-DoF object trajectories in the Move dataset are paired with fine-grained language instructions for precise language-to-motion grounding.
  • Approach: DreamTraj predicts trajectories from a single RGB frame and task instruction without privileged geometric inputs, explicit video generation, or pixel-space extraction.
  • Approach: At an early denoising step, a lightweight flow-matching readout decodes query–key attention tracks and pooled intermediate features from a frozen image-to-video diffusion backbone.
  • Results: DreamTraj achieves state-of-the-art translation and rotation prediction while being substantially more efficient than generate-then-extract pipelines.

2 Related Work

Prior work faces supervision and inference constraints: large video corpora lack continuous object poses, precisely annotated datasets require specialized sensing, and motion predictors typically rely on privileged inputs or costly video generation and parsing.

  • Datasets: Large-scale egocentric corpora provide thousands of video hours but only verb–noun tags, while precise pose datasets require RGB-D sensors, Aria glasses, or laboratory rigs.These collections trade annotation quality against accessibility and do not provide continuous object pose in the cited examples.
  • Motion Prediction: Motion-prediction interfaces generally assume privileged inference inputs, including goal images, query points, RGB-D, video context, object meshes, scene geometry, or an initial pose.Point-track and flow methods still require lifting 2D tracks to 3D, while 6-DoF forecasters consume additional scene or object information.
  • Video-Based Pipelines: Generate-then-extract methods synthesize and fully decode future videos before recovering motion from pixels through inverse dynamics, dense flow, policies, or pose-tracking cascades.The cited pipelines include video-conditioned policies and generator–filter–pose-tracker cascades, all of which denoise and decode a full video first.

3 Dataset Construction

The MOVE dataset is constructed by prioritizing supervision quality: a three-phase pipeline filters raw egocentric clips, adds object-centric language annotations, and augments real trajectories with screened generated futures. It produces 5,038 final instances from 7,246 raw clips while preserving object-specific 6-DoF motion supervision.

  • Pipeline: The three-phase pipeline compresses 7,246 raw clips into 2,975 high-quality real trajectories and 5,038 final instances after augmentation.The phases are selection, object-centric re-annotation, and generative augmentation.
  • Selection: Selection retains clips with meaningful manipulation, one fine-grained instruction, a groundable object, and a gap-free pose track.Clips containing multiple or partial actions are temporally cropped rather than discarded.
  • Annotation: Every retained clip is re-annotated with the manipulated object, verb, and motion manner, then human-corrected against the video and crop boundaries.An AWQ-quantized Qwen3.5-27B drafts the labels from sampled frames before human correction.
  • Generative augmentation: Generative augmentation uses Wan2.2 futures screened against recovered tracks, adding 2,063 generated trajectories and bringing MOVE to 5,038 instances.Each generated trajectory inherits its source clip’s object and instruction.
  • Trajectory extraction: Generated-video labels use DA3 metric depth and SAM 3D Objects meshes to avoid translation scale ambiguity and unavailable or imprecise CAD models.The pipeline grounds the object with GroundingDINO and SAM2 and tracks pose by render-and-compare with FoundationPose.

4 Method

DreamTraj predicts a frame-0-anchored relative 6-DoF trajectory from one RGB frame and an instruction by reading motion and scene signals from an early latent of a frozen Wan2.2 image-to-video diffusion model. A lightweight flow-matching Reader jointly decodes these signals into pose tokens, then restores metric scale only at the output.

  • Overview: DreamTraj takes one RGB frame and instruction and predicts the manipulated object’s 6-DoF trajectory over the next 49 frames in the initial camera frame.The design separates a frozen dynamics-supplying backbone from a small trained Reader.
  • Trajectory representation: The trajectory is represented with scale-free relative displacements, log-depth ratios, and rotations; metric scale is restored only at output from an input-frame depth estimate and camera intrinsics.Depth enters only during final metric conversion, using a sensor reading when available or a monocular estimate otherwise.
  • Frozen diffusion backbone: The frozen Wan2.2 backbone is read at step 16 of its 40-step denoising schedule, where the motion signal saturates.Stride-2 flow caching recomputes the guidance-combined flow every other step and reuses it between steps.
  • Motion readout: Query–key attention tracks provide object motion by matching frame-0 query patches against every patch in every frame, avoiding chained drift.The object is grounded once in the input frame with GroundingDINO and SAM2, and matching uses head-averaged cosine similarity.
  • Motion readout: Mean-pooled hidden states from blocks 0, 19, and 39 provide scene context, yielding 13 vectors with 128 context dimensions plus 4 object-motion dimensions.The hidden states are pooled over a 4 × 4 anchor grid and projected to 128 dimensions using in-fold PCA.
  • Reader: A 7.1M-parameter flow-matching DiT jointly denoises all 13 relative 9-D pose tokens, rather than accumulating motion frame by frame.It uses self-attention over pose tokens, cross-attention to context tokens, conditional flow matching, and 20 Euler integration steps at inference.

5 Experiments

Experiments evaluate DreamTraj on 5,038 MOVE trajectories with source-clip-separated five-fold cross-validation, measuring translation and rotation errors under protocol-matched comparisons. DreamTraj outperforms retrained forecasters, reduces inference cost versus generate-then-extract pipelines, and uses selected diffusion readouts and complementary feature inputs.

  • Evaluation protocol: 5,038 samples are evaluated with five-fold cross-validation, assigning each recorded clip and its generated derivatives to the same fold.This prevents generated twins of held-out clips from appearing in training.
  • Evaluation protocol: 6.4 cm ADE, 9.5 cm FDE, and 24.6° rotation error are achieved by the deployed Reader over all out-of-fold samples.Translation uses anchor-relative displacement over 13 pose tokens, while rotation uses SO(3) geodesic angle; errors are per-clip medians.
  • Accuracy comparison: DreamTraj leads EgoScaler on every metric despite its depth and initial-pose inputs, with the result surviving input-convention and timestamp stress tests.DreamTraj’s inference window is adapted to EgoScaler’s timestamps and horizon for the comparison.
  • Accuracy comparison: ObjectForesight trails DreamTraj on both translation and rotation after retraining on MOVE, while each baseline is evaluated in its own protocol-defined block.The comparisons include ObjectForesight and EgoScaler, both retrained on MOVE; EgoScaler starts from its released 7B checkpoint.
  • Readout selection: 48% margin growth occurs from k=14 to k=16, after which later steps add under 3%, selecting k=16 as the earliest post-switch readout.The jump occurs at Wan’s high- to low-noise expert switch, with paired t=12.6 over 955 clips.
  • Readout selection: Block 28, head 32 achieves the highest attention-track margin, 0.191 over 955 clips, and is used for reading the object track.The selection comes from evaluating all 40×40 block-head pairs.

6 Conclusion · Technical Appendix

The conclusion presents DreamTraj as a single-RGB-frame, language-conditioned method that reads 6-DoF motion from frozen video-diffusion representations without explicit video synthesis. It also identifies MOVE as the supervision resource and summarizes the technical appendix’s supporting materials.

  • 6 Conclusion: DreamTraj predicts 6-DoF object trajectories from a single RGB frame and language instruction.It directly reads motion from intermediate representations of a frozen image-to-video diffusion model.
  • 6 Conclusion: DreamTraj bypasses explicit video synthesis by reading motion from intermediate diffusion representations.
  • 6 Conclusion: MOVE supplies 5,038 object-centric egocentric trajectories paired with fine-grained instructions.The dataset addresses the supervision gap identified for this task.
  • 6 Conclusion: DreamTraj beats prior forecasters on translation.
  • Technical Appendix: The technical appendix reports the full experimental configuration and cross-validation protocol.
  • Technical Appendix: It documents baseline reproduction details and dataset statistics.
  • Technical Appendix: It also provides supporting ablations and additional qualitative results.

A Full Experimental Configuration … A.3 Optimisation

The experimental configuration fixes the diffusion backbone, readout channels, Reader architecture, optimization schedule, and conditional flow-matching objective for reproducible trajectory prediction. The Reader alone is trained, with pose-specific auxiliary losses and the anchor frame excluded from optimization.

  • A Full Experimental Configuration: The configuration specifies every setting required to reproduce the reported tables.These details supplement the facts given in Sec. 5.1 of the main paper.
  • A.1 Backbone and Readout: Wan2.2-I2V-A14B is the fixed dual-expert mixture-of-experts image-to-video diffusion transformer backbone.The sampler, guidance scale, and seed remain fixed across experiments, so variation comes from Reader training alone.
  • A.1 Backbone and Readout: Four q·k-cell geometry channels encode normalized centroid coordinates, log-spread ratio, and normalized in-plane rotation.The centroids are frame-0-relative and normalized by image resolution; rotation is normalized by 90◦.
  • A.2 Reader Architecture: The Reader is the only trained component: a holistic flow-matching DiT jointly denoising all 13 pose tokens.Its 7.08M parameters comprise 6.88M in the DiT body and 0.20M in the instruction encoder.
  • A.3 Optimisation: 5,799 steps define the deployed Reader snapshot at the end of epoch 200, using a fixed 200-epoch budget rather than validation-based early stopping.Each epoch contains 29 optimizer steps at batch size 128, and Table 4 ablations use the same schedule.
  • A.3 Optimisation: The objective combines conditional velocity-field flow matching with endpoint, SO(3) chordal, and translation smoothness terms weighted 0.1, 0.1, and 0.05.Frame 0 is pinned to the anchor pose in noise and target, then excluded from the loss.

A.4 Metric Definitions … B.2 Group Construction

The appendix defines robust trajectory metrics and reports DreamTraj results using five-fold, source-clip-grouped cross-validation. It also separates one-off preprocessing from inference cost and prevents related recorded and generated samples from crossing folds.

  • A.4 Metric Definitions: All reported errors are per-clip medians, and translation is computed after mapping scale-free tokens to metric units.This prevents a small number of catastrophic clips from dominating an error column.
  • A.4 Metric Definitions: ADE and FDE compare anchor-relative predicted and ground-truth positions in centimetres, making results independent of reference-frame choice.Rotation is measured as an SO(3) geodesic angle in degrees, averaged over the clip and reported at the final token.
  • A.4 Metric Definitions: Depth correlation uses detrended log-depth ratios and reports the absolute Pearson correlation of residuals as a median over clips.Detrending removes shared monotone camera-distance drift before correlation is computed.
  • A.5 Compute: Reader training for all five folds takes 16 minutes on one RTX PRO 6000, while backbone preprocessing is excluded from the measured inference budget.Preprocessing requires one backbone forward per clip to the read step with stride-2 flow caching.
  • B Cross-Validation Protocol: Every DreamTraj number is an out-of-fold prediction produced under five-fold cross-validation.The protocol is specified because related MOVE samples must not straddle fold boundaries.
  • B.1 Why Grouping Is Necessary: Individual-sample splitting would place near-duplicates sharing a frame, instruction, and object instance across folds, overstating generalization.Generated samples arise from the same recorded clip under multiple sampling seeds and have their own generated motion supervision.
  • B.2 Group Construction: Folds are assigned by source clip: recorded and generated samples sharing an identifier form one group, and groups are dealt round-robin into five folds.Groups are permuted once under a fixed seed, and every sample inherits its group’s fold.
  • B.2 Group Construction: 5,038 samples—2,975 recorded and 2,063 generated—are distributed across 2,982 groups, with no group spanning multiple folds.Groups contain one to nine samples, so fold sample counts are not exactly equal.

B.3 Evaluation … C.3 Generate-then-Extract Cascade

Evaluation uses pooled out-of-fold predictions across all 5,038 samples, with filtering performed before grouped folds are assigned. Baselines are retrained or instantiated under controlled protocols, including matched inputs, horizons, timestamps, hardware, and diffusion settings.

  • B.3 Evaluation: 5,038 samples are each scored exactly once by a Reader trained on the other four folds, with errors reported as pooled per-clip medians.The five held-out prediction sets are pooled after cross-validation.
  • B.3 Evaluation: Baselines use the same fold assignment when retrained, and the released mapping from sample identifiers to folds makes the split reproducible.This prevents comparisons from being confounded by different splits.
  • B.4 Filtering Precedes the Split: 2,975 recorded clips and 2,063 generated futures remain after pool-wide filtering, forming the 5,038-sample dataset before any fold is created.Recorded candidates decrease from 7,246 to 2,975, while generated futures decrease from 7,815 to 2,063.
  • B.4 Filtering Precedes the Split: Filtering decisions are made before grouped folds exist, ensuring human screening cannot use held-out boundaries or leak information across folds.The grouping procedure is applied only after the full pool has been screened.
  • C Baseline Reproduction: Both accuracy baselines are retrained on MOVE under the same fold assignment as DreamTraj, with implementation changes documented for auditability.Neither released code ran unmodified on the dataset.
  • C.1 ObjectForesight: ObjectForesight receives multi-frame video, a CAD mesh, and three ground-truth context poses, while DreamTraj receives none of these privileged inputs.The comparison deliberately preserves the baseline’s three inputs.
  • C.1 ObjectForesight: ObjectForesight is evaluated with separately trained short and whole-action models, each compared against DreamTraj at matching timestamps.The native configuration predicts 8 steps at stride 4 over 1.17 s; the longer configuration predicts 13 steps at stride 8.
  • C.2 EgoScaler: EgoScaler is retrained from its released 7B checkpoint to predict 20 steps using depth, the initial 6-DoF pose, and text; its scale units require correction for meaningful comparison.Its published monocular depth pipeline is not metrically calibrated on egocentric footage.

D Dataset Statistics … D.4 Symmetry Handling

The dataset comprises 5,038 trajectories spanning multiple source corpora, with augmentation deepening existing object–instruction coverage rather than adding semantic categories. Fine-grained instructions are human-corrected, and rotational symmetry is handled by supervising only observable rotation components.

  • D Dataset Statistics: The dataset-statistics section provides per-corpus composition and coverage details beyond the main paper’s dataset description.
  • D.1 Composition: 5,038 trajectories are drawn from six source corpora, with recorded data dominated by HOI4D and HOT3D.The listed corpora are HOI4D, TACO, HOT3D, OakInk2, H2O, and FPHA.
  • D.1 Composition: 2,063 generated trajectories descend from 544 source clips, requiring grouped folds because up to nine samples can share a first frame and instruction.Each source clip contributes between one and eight surviving generations.
  • D.2 Coverage: Augmentation preserves exactly 14 object categories, 11 verbs, and 92 object–verb pairs while deepening coverage within existing semantics.Generated trajectories inherit their source clip’s object and instruction.
  • D.2 Coverage: Translation and rotation magnitudes are only loosely coupled, including near-pure translations and substantial reorientations at small displacement.This variation prevents strong performance on both metrics by predicting one magnitude from the other.
  • D.3 Instruction Annotation: Every retained clip receives a fresh object-centric instruction annotation drafted from sampled frames and corrected by a human against the video.The human also adjusts the temporal crop to retain exactly one complete action.
  • D.3 Instruction Annotation: The resulting annotations are full phrases rather than verb–noun tags, and generated samples inherit their source instruction verbatim.This preserves feature alignment rather than creating new semantic coverage.
  • D.4 Symmetry Handling: For rotationally symmetric objects, flagged clips project target rotations onto the observable subspace before loss computation to avoid supervising unobservable motion.The ambiguity concerns rotation about the symmetry axis, which is unobservable from RGB.

E Additional Ablations … F.1 Extended Comparison

The additional ablations validate DreamTraj’s readout design, identifying informative attention cells, a specific denoising step, and a sensitive pooled-hidden-state block. Extended qualitative comparisons use the main paper’s identical protocol on more clips.

  • E Additional Ablations: The ablations examine which attention cell to read, when to read it, and which blocks provide pooled scene context.These experiments supply supporting evidence for the readout choices reported in Sec. 5.6.
  • E.1 The Permutation Control: 0.43 correlation remains against an unrelated clip’s ground truth for a smooth shared temporal profile, motivating permutation-controlled evaluation.Manipulation trajectories commonly rise during pickup and carrying before falling during placement.
  • E.2 Head Selection: 0.192 is the top margin score among 40 × 40 block-head pairs, with block 35 / head 16 at 0.184 and block 32 / head 26 at 0.163.Only three blocks carry a usable object track; the scan covers 955 clips.
  • E.2 Head Selection: The margin-selected block 28 / head 32 also yields the lowest ADE after retraining Readers on each candidate.The training-free criterion and end-task ranking agree.
  • E.3 Readout Step: 48% is the margin gain between k=14 and k=16, after which every later step adds under 3%.The jump coincides with the high- to low-noise expert switch at t = 900; k=16 is selected as the earliest post-switch step.
  • E.4 Pooled-Hidden Block Selection: Replacing block 19 with adjacent block 20 degrades all four metrics when retraining the Reader under the grouped five-fold protocol.The comparison uses 5,036 of 5,038 samples because every block must be available for every sample.
  • F.1 Extended Comparison: Figure 7 extends the main paper’s qualitative comparison to more clips under the identical protocol.This provides the extended qualitative results reported in F.1.

F.2 Beyond the Training Distribution

DreamTraj is qualitatively evaluated beyond the training distribution on unseen scenes, objects, and actions. The evaluation spans phone photographs and a video-game frame, but lacks ground-truth trajectories and therefore contributes no quantitative results.

  • Generalization beyond the training distribution: DreamTraj is applied to scenes drawn from outside all six source corpora, including phone photographs of everyday desks and a video-game frame.Figure 8 evaluates generalization beyond the training distribution using scenes, objects, and actions absent from every training corpus.
  • Generalization beyond the training distribution: The evaluation covers five object types: a shampoo bottle, keyboard, cup, game controller, and figurine.These objects are distributed across the phone photographs and video-game frame.
  • Generalization beyond the training distribution: Because none of the examples has a ground-truth trajectory, the results are qualitative and excluded from quantitative comparisons.The figure presents qualitative predictions rather than measured trajectory accuracy.
Loading 2608.00486v1…