PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboTTT: Context Scaling for Robot Policies

Yunfan Jiang, Yevgen Chebotar, Ruijie Zheng, Fengyuan Hu, Yunhao Ge, Jimmy Wu, Tianyuan Dai, Scott Reed, Li Fei-Fei, Yuke Zhu, Linxi "Jim" Fan

arXiv:2607.15275v1cs.ROcs.AIcs.LG

TL;DR

Robot policies have mostly used short visuomotor contexts, limiting evidence for exploiting long histories. RoboTTT integrates Test-Time Training with robot foundation models to scale context to 8K timesteps, achieving 79% average task completion, 87% higher than a single-step baseline.

  • Problem

    Robot foundation models mostly use short visuomotor contexts, leaving open how policies can learn from and exploit arbitrarily long contexts.

  • Method

    RoboTTT integrates Test-Time Training with robot foundation models, using fast weights and sequence action forcing with truncated backpropagation through time for long-context modeling.

  • Results

    79% average task completion, 87% higher than the single-step GR00T N1.7 baseline, with RoboTTT achieving the most fully successful trials on every task.

  • Takeaways & Limitations

    RoboTTT shows steady closed-loop gains as pretraining context scales to 8K timesteps, supporting context length as a scaling axis for robot foundation models.

  • Takeaways & Limitations

    Scaling training context length increases training cost, and RoboTTT does not handle every deployment failure mode.

Abstract

from arXiv · show

Recent robot foundation models operate with single-step or short-history visuomotor context. We introduce Test-Time-Training Robot Policies (RoboTTT), a robot model and training recipe that scale visuomotor context to 8K timesteps, three orders of magnitude beyond state-of-the-art policies, without growing inference latency. At this context length, we unlock new robot capabilities: one-shot in-context imitation from human video demonstrations, on-the-fly policy improvement, robustness to perturbations, and stronger performance on multi-stage, long-horizon tasks. We also observe, for the first time, steady gains in closed-loop performance as pretraining context length scales. At its core, RoboTTT integrates Test-Time Training into robot foundation models such as Vision-Language-Action policies, yielding a sequence model whose recurrent state consists of fast weights, parameters updated by gradient descent during both training and inference, compressing histories into weight space and retrieving contextual information for long-context conditioning. To scale training context length, the recipe combines sequence action forcing with truncated backpropagation through time. On challenging real-robot manipulation tasks, RoboTTT improves overall performance by 87% over the single-step context baseline and fully completes a five-minute, ten-stage assembly task, which no baseline ever does. RoboTTT trained with 8K-timestep context outperforms the same model pretrained with 1K timesteps by 62%, suggesting context length as a new scaling axis for robot foundation models. Videos are available at https://research.nvidia.com/labs/gear/robottt/

1. Introduction

RoboTTT addresses the short visuomotor context of current robot foundation models by scaling context to 8K timesteps without increasing inference latency. It integrates test-time training with fast-weight recurrent states to support long-context conditioning and constant-cost inference.

  • Contribution: 8K timesteps—three orders of magnitude beyond state-of-the-art robot foundation models—are reached without growing inference latency.The resulting model is called RoboTTT-8K.
  • Motivation: Long visuomotor context supports one-shot in-context imitation from a single human video demonstration.The motivation contrasts this capability with the single-step or short-history context used by most state-of-the-art robot foundation models.
  • Method: RoboTTT integrates Test-Time Training into Vision-Language-Action policies, using fast weights updated by gradient descent during both training and inference as recurrent state.Unlike slow weights, fast weights continue adapting during deployment.
  • Method: Fast weights retain salient features while discarding redundant observations and actions, and propagating them over time keeps inference cost constant.This contrasts with Transformer inference, whose cost grows with history even when using a KV cache.

2. Preliminaries

The preliminaries define Test-Time Training as a mechanism that updates fast weights during training and inference to encode and retrieve contextual information. They also formalize robot sequence models as policies conditioning on rollout history and current observations, with long-context scaling targeting the history length.

  • Test-Time Training Mechanism: TTT updates fast weights during both training and inference, unlike slow weights, which update only during training and remain frozen at inference.Fast weights dynamically model contextual information, while slow weights are the model parameters.
  • Test-Time Training Mechanism: The TTT update step encodes contextual information into fast-model parameters, and the apply step retrieves it for downstream prediction.The projection matrices and fast-weight initialization are learned with the outer task loss.
  • Robot Sequence Models: A robot trajectory consists of image, proprioception, and action chunk tuples across timesteps, with language omitted for simplicity.The trajectory is represented as ξ = {(o_t, q_t, A_t)}^T_{t=1}.
  • Robot Sequence Models: Robot sequence policies condition actions on rollout history and the current observation, while long-context policies aim to scale the history length.The policy is written as π(A_t|ξ_<t, o_t, q_t).

3. Method: Test-Time-Training Robot Policies

RoboTTT integrates test-time-training layers into robot foundation models, using fast weights to process information across timesteps while attention remains within each timestep. Its training recipe combines sequence action forcing with TBPTT, and supports context-only learning from demonstrations or failures for in-context adaptation.

  • Architecture: RoboTTT adds TTT layers after DiT attention, so attention processes each timestep while TTT layers process information across the trajectory.The model predicts H-step action chunks from vision-language, proprioception, and noised action tokens spanning T timesteps.
  • Architecture: Learned tanh gates initialize near zero, preserving pretrained model capabilities while allowing TTT contributions to increase during training.Each DiT layer learns α initialized to 0.001, weighting the TTT output before adding it to the attention output.
  • Training recipe: Sequence action forcing independently samples the diffusion noise level for each action chunk, avoiding instability caused by one shared noise level across the sequence.The method uses flow matching with independently sampled noise levels because shared noise can make entire sequences uniformly easy or hard to learn.
  • Training recipe: TBPTT enables arbitrarily long training contexts under fixed memory by truncating gradients at segment boundaries while carrying fast weights across the full sequence.Memory depends on segment length rather than total sequence length, and the initialization W0 receives gradients through the first segment.
  • Contextual adaptation: Masked flow-matching losses let demonstrations or failures update fast weights as context, while robot actions or human corrections provide imitation targets.For video imitation, the video loss is masked and the robot trajectory is trained conditionally; DAgger Distillation maps failures to corrective actions.

4. Experiments

Experiments on three long-horizon bimanual assembly tasks show that RoboTTT improves task completion over short-context and recurrent-memory baselines, while longer pretraining context steadily improves closed-loop performance. Long-context conditioning also enables one-shot imitation and stronger recovery from perturbations.

  • Main evaluation: RoboTTT reaches 79% average task completion, 87% higher than GR00T N1.7’s 42% and 41% higher than GDN’s 56%.It records the most fully successful trials on every task and completes Gear Bot in 2 of 10 trials, while no baseline does.
  • Context-length scaling: RoboTTT reaches 71.5% at 8K pretraining timesteps, 63% higher than the same model at 1K timesteps (43.9%).Performance scales steadily from 128 to 8K timesteps, surpasses the best short-context baseline from 1K onward, and shows no saturation; GDN does not benefit from longer context.
  • One-shot imitation: RoboTTT completes six of ten one-shot Circuit assemblies from an in-context human video, whereas GDN fails entirely.The result indicates that RoboTTT can retrieve the demonstrated configuration through gradient-descent-updated fast weights, unlike the recurrent-memory baseline.
  • Perturbation robustness: RoboTTT recovers in 15/20 roof-perturbation trials versus 13/20 for GDN and at most 10/20 for short-context baselines.For tire perturbations, both RoboTTT and GDN recover in 18/20 trials, showing within-episode conditioning benefits from visuomotor context.
  • Ablations: Removing sequence action forcing significantly hurts closed-loop performance, while the TTT-linear variant remains 27% worse than the MLP fast model.The findings support sequence action forcing and expressive nonlinear fast models as important components of the training recipe.
  • Ablations: Adding action tokens produces a 23% relative improvement, and adding register tokens yields a further 18% relative improvement for the TTT MLP.Register tokens do not help GR00T N1.7, suggesting their benefit depends on TTT temporal modeling.

5. Related Work

Prior robot policies use short contexts, autoregressive histories, or recurrent architectures with deployment tradeoffs, while TTT offers fast-weight adaptation for continuous contextual storage. RoboTTT applies TTT layers to visuomotor policies and addresses history-induced spurious correlations by dynamically retaining relevant information in parameter space.

  • Long-Context Policies: Most robot foundation models use single-step or short-history context, typically only the current observation or 2–8 consecutive observations.Some methods extend observation windows through trajectory inputs, token compression, or past-action prediction.
  • Long-Context Policies: Autoregressive rollout-history models capture long-context dependencies but incur decoding latency that grows linearly with context length, whereas RNN policies provide constant inference complexity.The passage contrasts these approaches with traditional recurrent architectures such as LSTMs.
  • Long-Context Policies: Prior methods mitigate history-induced spurious correlations through context summarization, auxiliary objectives, or selective context bypassing.These correlations arise when policies overfit to past actions implicitly encoded in past observations.
  • Test-Time Training: TTT updates a small subset of parameters, called fast weights, with a self-supervised objective during training and inference to continuously store and retrieve contextual information.Recent work improves TTT optimization, online learning objectives, and integration with language and vision models.
  • Test-Time Training: Unlike robotics works that fine-tune entire models using extra test-task data, RoboTTT builds visuomotor policies on TTT layers with fast weights.The closest cited setting equips VLMs with fast weights for robotic value-function adaptation, rather than visuomotor policy construction.

6. Limitations and Conclusion

RoboTTT scales robot-policy visuomotor context to 8K timesteps by integrating Test-Time Training into robot foundation models, enabling several long-context capabilities. The work also identifies increased training cost, the need for robotics-oriented TTT objectives, and unresolved deployment failures as limitations and future-work directions.

  • Limitations: Scaling training context length increases training cost, motivating future adoption of newer Test-Time Training techniques such as TNT.The passage identifies training cost as the first limitation and TNT as a possible future direction.
  • Limitations: Future work could explore robotics-oriented objectives for the Test-Time Training layers.The paper describes its integration of Test-Time Training as principled while pointing to robotics-oriented objectives as an open direction.
  • Limitations: Although RoboTTT substantially improves task performance, it does not handle every failure mode encountered in deployment.The passage states that unresolved deployment failures remain a limitation.
  • Conclusion: RoboTTT scales robot-policy visuomotor context to 8K timesteps by integrating Test-Time Training into robot foundation models.The method is presented as both a robot model and a training recipe.
  • Conclusion: RoboTTT enables one-shot imitation from in-context human video demonstrations, on-the-fly policy improvement, robustness to external perturbations, and stronger closed-loop performance on multi-stage, long-horizon tasks.These capabilities arise at the 8K-timestep context length.

A. Model Architecture, Training, and Deployment Details · A.1. Model Architecture

RoboTTT augments pretrained GR00T N1.7 with test-time-training layers integrated into its Diffusion Transformer action head. These layers add fast models updated by gradient descent at test time, increasing the model from 538M to 690M parameters.

  • A.1. Model Architecture: RoboTTT is instantiated on pretrained GR00T N1.7, using Eagle as the VLM backbone and a Diffusion Transformer as the action head.The architecture adds a TTT layer to each of the DiT’s 16 layers.
  • A.1. Model Architecture: 538M original DiT parameters increase to 690M after adding roughly 10M-parameter TTT layers across 16 DiT layers.Each TTT layer contains a two-layer MLP fast model with GeLU activation.
  • A.1. Model Architecture: Each TTT layer updates its fast model by standard gradient descent at test time.The fast model is a two-layer MLP with GeLU activation.

A.2. Training · A.3. Deployment

RoboTTT is pretrained on long-trajectory tabletop robot and egocentric human-video data with sequence-modeling layers updated while GR00T N1.7’s other components remain frozen. Deployment uses a four-camera YAM bimanual robot setup at 30 Hz with 480p RGB observations.

  • A.2. Training: Pretraining mixes tabletop bimanual robot data with egocentric human video data curated to emphasize long trajectories.The trajectory-length distribution is reported in Fig. A.1.
  • A.2. Training: Pretraining updates only newly added TTT or GDN sequence-modeling layers, while post-training fine-tunes all parameters.The remaining GR00T N1.7 components are frozen during pretraining.
  • A.2. Training: 16 NVIDIA GB200 GPUs are used per pretraining run, and 8 GPUs are used per task-specific post-training run.The runs use NVIDIA GB200 GPUs.
  • A.2. Training: 1K context length and per-device batch size of 1 are used for the reported training setup.Models are pretrained for 30K steps and post-trained for 20K steps.
  • A.2. Training: AdamW with weight decay 1 × 10−5 is used for all models.Pretraining uses WSD with peak learning rate 2 × 10−5, while post-training uses cosine scheduling with peak learning rate 5 × 10−5.
  • A.3. Deployment: Deployment runs on the YAM bimanual tabletop robot with four RealSense D405 cameras streaming 480p RGB observations.The cameras are positioned top, bottom, left wrist, and right wrist.
  • A.3. Deployment: 30 Hz control frequency is achieved on a workstation equipped with an NVIDIA RTX 5090 GPU.Inference runs on the workstation during deployment.

B. Task Definition and Experiment Details

This section defines the tasks and presents the experiment details.

  • The section covers task definitions and experiment details.

B.1. Task Definition

The benchmark defines multi-stage robot assembly tasks, including Pup Go Car, Gear Bot, and Circuit, scored on a [0, 1] task-completion scale using task-specific rubrics. Rubrics award partial credit for intermediate assembly steps and, for Circuit, require correct assembly order.

  • Scoring: Tasks are scored on a [0, 1] task completion scale using task-specific rubrics.The rubrics define partial-credit milestones for task progress.
  • Pup Go Car: Pup Go Car requires roof and wheel assembly with screwing, drilling, bimanual handoffs, and a car flip.The robot places and tightens the roof, flips the car, inserts and tightens a wheel, and transfers the drill between hands.
  • Gear Bot: Gear Bot requires installing gears and wheels on both chassis sides, flipping the chassis twice, attaching the robot head, and driving it by remote.The task assembles the entire toy model through multiple installation, flipping, insertion, and remote-control stages.
  • Circuit: Circuit involves roughly 80 configurations of two- or three-piece component assemblies, sometimes followed by powering the circuit on.Scoring varies by component count and switch or press-button presence, with no partial credit for an incorrect assembly order.

B.2. Experiment Details

Experiments compare RoboTTT with matched GR00T and GDN baselines and evaluate all models on YAM bimanual tabletop robots under reproduced initial object placements. Rollout counts vary by task and setting because some evaluations take substantially longer.

  • Baselines: GR00T N1.7 and GR00T N1.7 Hist. use the official implementation, with GR00T N1.7 Hist. extended for history-frame input.For GDN, each TTT layer is replaced by a matched Gated DeltaNet layer while preserving layer placement, gating, and parameter count.
  • Evaluation: 20 rollouts evaluate the Pup Go Car and Circuit tasks, while 10 evaluate the Gear Bot task and Circuit under one-shot human-video evaluation.Models are deployed on YAM bimanual tabletop robots, with initial object placements recorded and reproduced to ensure identical starting conditions.
Loading 2607.15275v1…