TAPESTRY: A Single Backbone for Video Generation and Robot Control

Motivation

• Video models contain physical-interaction priors difficult to learn from limited robot demonstrations, motivating their transfer to policies that generalize across tasks, environments, and embodiments.The paper frames that transfer as a central question for robot learning.Source: §1 · p. 1

Related approaches

• Video-rollout pipelines compound errors across generation stages and require costly frame generation, while latent-feature policies use action heads trained from scratch.Shared-backbone approaches also face tradeoffs: training without a pretrained video prior, costly inference tokens, or a separately trained action module.Source: §1 · pp. 1–2

Approach

• TAPESTRY treats each action timestep as a video-VAE patch token, allowing actions to use the pretrained backbone’s dynamics priors without pixel-space video generation.A linear projection maps actions to VAE patch dimensions before the shared patchifying convolution and transformer process them.Source: §1 · p. 2

Contributions

• The shared backbone is trained with video, joint video/action, and action-only diffusion objectives, then predicts action chunks in one DDIM step without video generation at inference.The contributions also include an asymmetric attention mask for joint training and real-world bimanual evaluation.Source: §1 · p. 2

2 Method

• The policy maps a current RGB observation and language goal to an action chunk, using video and robot-interaction data for joint backbone training and deploying without video generation.Video data supplies video-denoising training; robot data supports joint or action-only denoising.Source: §2 · p. 2

2.1 Action Tokenization

• Each action timestep becomes one VAE-patch token through a linear projection, affine-free LayerNorm, and the video backbone’s shared patchifying convolution.The LayerNorm matches per-token mean and variance to the approximately unit-normal VAE latent statistics; the resulting action sequence has one token per timestep.Source: §2.1 · p. 3

2.2 Training Objectives

• The same DiT trains in video, joint video/action, and action-only modes, with a shared clean text/observation prefix and mode-specific noised tokens and losses.Joint training couples video and action timesteps; its asymmetric mask lets action tokens attend to video while preventing video tokens from attending to actions.Source: §2.2 · p. 4

2.3 Inference

• At test time, the action pathway takes one DDIM step from pure noise conditioned on observation and text, producing an action chunk without sampling video or using a VAE decoder.Under the stated schedule, the one-step output approximates the conditional mean of the action chunk.Source: §2.3 · p. 4

2.4 Implementation

• The implementation uses CogVideoX-2B, video data from robot and human datasets, and in-domain bimanual teleoperation as the only source of action labels.Training runs use 32 H200s, with 75k pretraining steps followed by 25k fine-tuning steps.Source: §2.4 · pp. 4–5

Frequency-space mismatch

• Action chunks’ f95 is 4–6× lower than natural video, creating a frequency-prior mismatch when the video backbone denoises actions.f95 is the frequency below which 95% of signal energy lies.Source: §3 · p. 5, Figure 2 · p. 6
• At t = 99, teacher-forced action outputs reach 1.71× the ground-truth jitter band despite in-distribution noisy inputs.The rising jitter is in the model’s outputs, not merely a consequence of feeding it off-manifold inputs.Source: §3 · p. 5

Single-step inference

• At zero terminal SNR, v-prediction makes the single-step target independent of noise, so inference from t = T sidesteps the low-t mismatch.An epsilon-dependent terminal target—such as epsilon prediction, non-zero terminal SNR, or flow-matching at σmax = 1—restores jitter.Source: §3 · p. 5

Jitter across denoising steps

• Iterative DDIM compounds per-step residuals into larger action error, while the output diverges from in-distribution statistics after a second update.The single-step output from t = T matches in-distribution statistics; scheduler choice does not change the multi-step pattern.Source: §3 · p. 5, Figure 2 · p. 6

4.1 LIBERO task generalization

• 76% held-out LIBERO success puts TAPESTRY 6 points above Cosmos Policy and DreamZero (both 70%) and 15 above AMPLIFY.TAPESTRY wins decisively on Object and Spatial, while Cosmos Policy and DreamZero lead on LIBERO-10 and DreamZero is slightly ahead on Goal; from-scratch training reaches 58%.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 Real-world bimanual manipulation

• On real-world tasks, TAPESTRY reaches 93 ± 5% in-distribution success and 67 ± 12% out-of-distribution success, completing the novel object-composition task.The in-house demonstrations cover three bimanual tasks; OOD tests probe object placement, motion primitives, and object composition, with 10 rollouts per trained task and 5 per OOD condition.Source: §4.2 · pp. 6–7, Table 2 · p. 8, Table 3 · p. 8

4.3 Inference efficiency

• TAPESTRY produces a 16-action chunk in 84 ms using 9 GB on an RTX 5090, while DreamZero takes about 3.7 s with caching and checkpointing.That is approximately 44× slower than TAPESTRY; DreamZero’s released code takes about 20 s per chunk on the same GPU.Source: §4.3 · pp. 7–8

4.4 Ablations

• At a fixed 5-epoch budget on LIBERO, TAPESTRY reaches 58%; removing the video objective gives 0%, while training from scratch gives 13%.Separate video and action objectives reach 36%, removing affine-free LayerNorm reaches 33%, and bypassing the patchifying convolution with direct projection diverges.Source: §4.4 · p. 8

Video generation and inverse dynamics

• Video-rollout methods such as UniPi [1], AVDC [29], and DreamGen [2] recover actions through inverse dynamics, adding a second generative stage and full-video inference cost.Their inference cost scales with video-token count, and pseudo-action fidelity is bounded by video quality and inverse-dynamics accuracy.Source: §5 · p. 8

Feature-based action heads

• Feature-head methods including VidMan [30], VPP [3], mimic-video [4], and Video Policy [31] use backbone features but train a separate action head from scratch.The action tokenization is decoupled from the pretrained backbone’s input distribution.Source: §5 · p. 8

Shared video-action backbones

• Shared-backbone methods differ in pretraining, action representation, and deployment cost; some generate video or use a separate action module.Examples include UWM [5], Cosmos Policy [7], DreamZero [8], Fast-WAM [9], and GigaWorld-Policy.Source: §5 · p. 8

TAPESTRY

• TAPESTRY sends actions through the backbone’s pretrained patchifier and unembedding, then drops video tokens and performs one denoising step at deployment.Its action pathway adds only W, and the single-step setup is supported under zero-terminal-SNR v-prediction.Source: §5 · p. 9

6 Conclusion

• The policy runs about 44× faster than optimized DreamZero on the same GPU and improves on the strongest published baseline on LIBERO held-out task generalization.It runs on a consumer-grade RTX 5090; single-step DDIM inference uses one forward pass from pure noise under a zero-terminal-SNR v-prediction schedule.Source: §6 · p. 9
• TAPESTRY combines a VAE-patch action tokenizer, three-objective training on one shared backbone, and single-step DDIM inference without video generation at inference.The tokenizer reuses the VAE patchifying convolution and matches the per-token mean and variance of its latents.Source: §6 · p. 9
• Single-step inference returns one deterministic conditional-mean action chunk, so multimodal or highly stochastic tasks may need iterative refinement or another remedy.The single-step result depends on the backbone's zero-terminal-SNR v-prediction schedule; transfer to rectified-flow backbones remains open, and per-embodiment action heads are unexplored beyond the 20D bimanual space.Source: §6 · p. 9

Video Generation with Inverse Dynamics

• Video-generation approaches predict a full rollout and recover actions with a separate inverse-dynamics module or optical-flow correspondences.Their inference cost scales with video tokens and denoising steps, and pseudo-action fidelity depends on video quality and inverse-dynamics accuracy.Source: A Extended related work · p. 13

Conditioning on Video-Model Latents

• Latent-conditioning approaches extract video-model features at one noise level, then use a separately trained action head, leaving action tokenization decoupled from pretrained inputs.The cited examples include intermediate partially denoised latents and parallel video and action branches.Source: A Extended related work · p. 13

Video Generation Models as a Shared Action Backbone

• Shared-backbone approaches route video and action tokens through one transformer, but differ in pretraining, deployment token cost, and whether pretrained weights act directly on actions.UWM trains from scratch; Cosmos Policy and DreamZero emit video at inference; Fast-WAM uses a separate action expert; GigaWorld-Policy adds new projections and an iteratively sampled head.Source: A Extended related work · p. 13
• TAPESTRY tokenizes actions through the same video patchifying convolution and drops video tokens at deployment, reducing the inference budget from about 10^3 video tokens to 16 action tokens.Its only added action module is a projection with 1280 parameters.Source: A Extended related work · p. 13

B Per-subtask real-world results

• Table 4 reports cumulative in-distribution per-subtask success counts out of 10 rollouts per cell, with failures propagating to every later subtask in that rollout.Table 4 final-subtask counts agree with Table 2; the corresponding out-of-distribution cells in Table 5 are out of five rollouts and agree with Table 3.Source: B Per-subtask real-world results · p. 13, Table 4 · p. 14
• The real-world experiments use a 20-DoF bimanual YAM platform equipped with wrist and third-person cameras.The setup is used for the experiments in §4.2.Source: Figure 3 · p. 14

C Ablation details

• Without action-only training, the joint video+action model fails under action-only inference; the separate-objectives and full-recipe variants score 36% and 58%.Those two rows share data, initialization, and hyperparameters, differing only in objective; the joint video+action row reaches 78% with video generation at inference.Source: C Ablation details · p. 14

• Visual conditioning uses random-resize crops, color jitter, and Gaussian blur; dropout 0.1 is applied to conditioning and action-denoising pathways.Source: D Hyperparameters and training details · p. 14
• The DreamZero baseline full-finetunes Wan2.2-TI2V-5B on 32 H200s, using 100k LIBERO steps or 50k real-world steps.Real-world training uses 25k pretraining and 25k in-domain fine-tuning steps.Source: D Hyperparameters and training details · p. 14
• TAPESTRY uses 1.9k H200-hours for real-world training and 2.5k on LIBERO, versus DreamZero’s 2.4k and 1.1k, respectively.Cosmos Policy uses 7.1k H200-hours on LIBERO.Source: D Hyperparameters and training details · p. 14
• Table 5 covers three OOD conditions—novel object placement, motion primitive, and object composition—with five rollouts per condition.The scenes include a croissant on an oven door, pouring a golf ball, and placing a Lego brick on a boomerang.Source: Table 5 · p. 15, Figure 4 · p. 15

E Dataset details

• The mixture contains about 673 training hours; its 6.02-hour in-domain bimanual teleoperation slice drives half of every gradient step.Cross-embodiment video is used only for video diffusion, while in-domain target-platform data supports video and action objectives.Source: E Dataset details · p. 15
• Cross-embodiment robot data and human or internet video are routed to video mode; target-task human videos are also collected to bias the prior toward deployment scenes.Tests of BridgeData v2 and AgiBot-World action labels found that video-only routing gave better in-domain action performance.Source: E Dataset details · p. 15
• Captions are capped at 80 tokens, covering approximately 95.8% of mixture captions; selected video sources are filtered for static scenes, text overlays, and extreme aspect ratios.Failed in-domain teleoperation episodes are excluded; AgiBot-World and BridgeData v2 are used as released.Source: E Dataset details · pp. 15–16
• Table 6 reports that the five-epoch full recipe reaches 58% held-out success versus 13% from scratch, rising to 76% versus 58% at convergence.Source: Table 6 · p. 16

F Action-space alignment

• Actions use a shared 20D space with per-arm position, 6D rotation, and gripper aperture; cross-embodiment poses are transformed into the target frame and gripper signals normalized.BridgeData v2 trajectories randomly assign their single arm to left or right, masking the unused arm; in-domain positions and rotations are chunk-relative.Source: F Action-space alignment · p. 16
• Action chunks contain 16 steps, about 0.53 seconds of motion; cross-embodiment streams were resampled to 30 Hz for action-transfer experiments, while final models use those sources as video only.Translational dimensions are percentile-clipped and scaled during training, then returned to their original range at inference.Source: F Action-space alignment · p. 16

G.1 Derivation: 1-step DDIM under zero-terminal-SNR v-prediction

• With zero terminal SNR, the noisy input becomes pure noise and one DDIM step reduces to −vθ, yielding the conditional mean of the action-chunk distribution given conditioning.Source: §G.1 · p. 18

G.2 Determinism of vθ at t = T

• At t = T, Varϵ vθ is 3.5×10^-6 per coordinate, five to six orders of magnitude below its value at t < T.The measured collapse matches the prediction that the terminal v-target decouples from ϵ.Source: §G.2 · p. 18, Figure 5 · p. 18

G.3 Teacher-forced vs. iterative inference

• Teacher-forced jitter rises from 0.92× to 1.71× the ground-truth band as t decreases, while iterative jitter grows to about 8× the band.Teacher-forced inputs are in-distribution at each timestep; Table 9 reports the per-step sweep against a GT band of 0.166 ± 0.027.Source: §G.3 · p. 19, §G.6 · p. 19, Table 9 · p. 20

G.4 Alternative explanations ruled out

• Removing shared AdaLN coupling or decoupling modality timesteps does not remove iterative jitter growth; DDIM, UniPC, and a zero-shift schedule likewise retain it for multi-step inference.On the robot, jitter appears from K = 2 and grows with K unless smoothed; filtered multi-step inference was not usable on the final model.Source: §G.4 · p. 19, §G.5 · p. 19

G.7 Wan rectified-flow comparison

• Wan rectified flow retains ϵ-dependence at its noise terminus: Varϵ vθ is 1.17, versus 3.5×10^-6 for TAPESTRY, and its converged single-step jitter is about 59× its ground-truth band.The figure contrasts the 64-seed terminal traces for the two parameterizations.Source: §G.7 · p. 20, Figure 7 · p. 21

G.8 Conditional-mean reduction

• TAPESTRY’s single-step output is a conditional mean rather than a sample, yet the two-croissant probe succeeded in 8/10 rollouts, matching the single-croissant control, with every reach committing to one object.Source: §G.8 · p. 20, §G.9 · p. 21

G.10 Spectral comparison: natural video vs. action chunks

• Natural-video f95 is 4–6× action-chunk f95, and roughly one-third of video energy exceeds action f95; low-t denoising can add out-of-distribution high frequencies to actions.The comparison uses pixel-space video; because the VAE is a temporal low-pass filter, the high-frequency contrast is conservative for latent space.Source: §G.10 · pp. 21–22, Table 10 · p. 22