Source-linked AI summary
Humanoid Locomotion as Next Token Prediction
Ilija Radosavovic, Bike Zhang, Baifeng Shi, Jathushan Rajasegaran, Sarthak Kamat, Trevor Darrell, Koushil Sreenath, Jitendra Malik
TL;DR
The paper asks whether powerful models can be learned for sensory and motor representations beyond language. A causal transformer autoregressively predicts complete sensorimotor trajectories and can learn from incomplete trajectories by masking missing modalities. A full-sized humanoid walked in the real world zero-shot, and offline-data policies were comparable to state-of-the-art reinforcement-learning approaches in tested settings.
Problem
The paper asks whether powerful models can be learned for sensory and motor representations beyond language.
Source: §1 · p. 1Method
A causal transformer autoregressively predicts complete sensorimotor trajectories and can learn from incomplete trajectories by masking missing modalities.
Source: §1 · p. 1, §3.2 · p. 4Results
Takeaways & Limitations
The findings suggest generative modeling of large sensorimotor trajectory collections as a promising path toward challenging real-world robot control tasks.
Source: §1 · p. 3The approach assumes that a dataset of sensorimotor trajectories is given.
Source: §3 · p. 3
Abstract
from arXiv · showhide
We cast real-world humanoid control as a next token prediction problem, akin to predicting the next word in language. Our model is a causal transformer trained via autoregressive prediction of sensorimotor trajectories. To account for the multi-modal nature of the data, we perform prediction in a modality-aligned way, and for each input token predict the next token from the same modality. This general formulation enables us to leverage data with missing modalities, like video trajectories without actions. We train our model on a collection of simulated trajectories coming from prior neural network policies, model-based controllers, motion capture data, and YouTube videos of humans. We show that our model enables a full-sized humanoid to walk in San Francisco zero-shot. Our model can transfer to the real world even when trained on only 27 hours of walking data, and can generalize to commands not seen during training like walking backward. These findings suggest a promising path toward learning challenging real-world control tasks by generative modeling of sensorimotor trajectories.
1. Introduction
The paper asks whether generative modeling can learn sensorimotor representations and applies autoregressive transformer prediction to humanoid trajectories. It reports zero-shot real-world walking and simulation performance comparable to reinforcement-learning approaches in tested settings.
1. Introduction
- The authors frame sensorimotor modeling as a largely unexplored extension of generative modeling, following positive results for visual autoregressive modeling and manipulation representations.They ask whether language-style modeling can learn powerful sensory and motor representations.Source: §1 · p. 1
- A causal transformer autoregressively predicts complete sensorimotor sequences, modeling their joint data distribution rather than only conditional actions.Training uses simulated trajectories from reinforcement-learning policies, a model-predictive controller, motion capture, and YouTube videos; test-time actions are executed autoregressively while sensory predictions are ignored.Source: §1 · pp. 1, 3
- The authors argue that joint prediction can provide a richer world model, use noisy or imperfect trajectories, and learn from trajectories with missing information.Source: §1 · p. 1
- The framework learns from incomplete trajectories by predicting available information and replacing missing sensory or motor tokens with learnable mask tokens.This permits use of human videos, which provide movement observations but not motor commands or complete sensory inputs.Source: §1 · p. 1
- The policy walked zero-shot across San Francisco locations, while offline-data policies were comparable to state-of-the-art reinforcement learning in tested settings.The deployments covered different surfaces, including walkways, concrete, asphalt, tiled plazas, and sanded roads.Source: Figure 1 · p. 2, §1 · p. 3
2. Related Work
Related work spans autoregressive generative modeling across modalities, transformer applications in robotics, and established approaches to humanoid locomotion. This paper distinguishes its approach through autoregressive modeling of diverse trajectories for real-world humanoid walking.
Generative modeling
- Autoregressive transformers have been applied to pixels, language, and multimodal data, with prior work reporting scaling behavior and in-context learning (19) (3).The paper explores these models for real-world humanoid locomotion.Source: §2 · p. 3
Transformers in robotics
- Robotics transformer studies include language-conditioned behavior cloning (38) (2), embodied-data language models (10), reinforcement learning (33), masked sensorimotor prediction (32), and goal-conditioned demonstrations (1).The paper focuses instead on autoregressive modeling of diverse trajectories.Source: §2 · p. 3
Humanoid locomotion
- Prior humanoid locomotion work includes model-based control (34) (18), optimization-based dynamic motions (22), and simulation reinforcement learning that enabled real-world locomotion (33).Unlike that prior learning-based work, this paper uses autoregressive modeling rather than reinforcement learning.Source: §2 · p. 3
3. Approach
The approach models sensorimotor trajectories autoregressively and extends training to incomplete data through masking and modality-aligned prediction. A causal transformer supports both training on mixed data and action prediction at inference.
3.1. Objective
- Each trajectory is tokenized, and the model learns its autoregressive density by minimizing negative log-likelihood using a constant-variance Gaussian and mean-squared token error.The authors choose regression over binning or vector quantization for simplicity after finding it works reasonably well.Source: §3.1 · pp. 3–4
3.2. Missing modalities
- Incomplete trajectories are represented by inserting mask tokens for missing actions or other modalities, while ignoring losses at masked positions.This lets trajectories without action data be processed in the same format as complete observation-action sequences.Source: §3.2 · p. 4
3.3. Aligned prediction
- For each input token, the model predicts the next token from the same modality rather than predicting without regard to modality.Source: §3.3 · p. 4
3.4. Joint training
- Joint training on complete and incomplete trajectories works comparably to pre-training on noisy or incomplete data before training on complete trajectories.The authors use joint training in most experiments for simplicity.Source: §3.4 · p. 4
3.5. Model architecture
- The vanilla transformer embeds modality tokens with shared-across-time projections and positional embeddings, then uses causal attention to predict states and actions.Masked positions receive no loss, and the model is trained with both complete and incomplete data.Source: §3.5 · pp. 4–5
3.6. Model inference
- At inference, the model conditions on past observed observation-action pairs, predicts actions autoregressively, and discards predicted observations in favor of the robot’s actual observations.The observed observation and predicted action are appended to the history for the next prediction.Source: §3.6 · p. 5
4. Dataset
The dataset combines complete robot sensorimotor trajectories with action-free controller and human-motion trajectories, enabling training from sources with different available information.
4.1. Neural network trajectories
- 10k trajectories of 10 seconds each provide complete observations and ground-truth actions from a reinforcement-learning policy.Commands cover forward, sideways, and turning velocities on flat ground, without domain randomization.Source: §4.1 · p. 5
4.2. Model-based trajectories
- Two sets of 10k 10-second controller trajectories provide action-free observations, including data with randomized leg length, step clearance, and floor bounciness.The controller’s torque outputs do not match the model’s joint-position action space; commands span forward, sideways, and turning velocities from −1.0 to 1.0.Source: §4.2 · pp. 5–6
4.3. Human motion capture trajectories
- About 1k standing, walking, and running trajectories from human motion capture are retargeted to robot poses using inverse kinematics.The optimization tracks human keypoint locations while regularizing joint velocity and smoothness, subject to posture and velocity constraints.Source: §4.3 · p. 6
4.4. Trajectories from YouTube videos
- YouTube videos are converted into noisy robot trajectories by reconstructing 3D human motion, retargeting it with inverse kinematics, and filtering for low optimization cost.The reconstructed human poses are estimated from video before retargeting.Source: §4.4 · p. 6
5. Experiments
The experiments evaluate humanoid locomotion in simulation and on Digit hardware, including outdoor deployment across varied San Francisco locations and surfaces.
5.1. Experimental setup
- Digit is a 1.6 m, 45 kg humanoid with 30 degrees of freedom, 20 actuated, whose high dimensionality and four-bar linkage make fast optimization challenging.The paper presents trajectory-collection learning as particularly relevant to this challenge.Source: §5.1 · pp. 6–7
- The trajectory-adherence setup compares desired and actual paths for the proposed policy and an RL policy at 0.5 m/s forward speed across yaw commands from −0.4 to 0.4 rad/s.The robot starts at the origin; Figure 5 plots dotted desired paths and solid actual paths.Source: Figure 5 · p. 7
- The transformer uses four self-attention layers with four heads each, 192 hidden dimensions, and a 16-step input context.The model applies LayerNorm before attention and ReLU after the MLP.Source: §5.1 · p. 7
5.2. Real-world deployment
- The full-sized Digit robot walked zero-shot at varied San Francisco locations over one week, traversing walkways, concrete, asphalt, tiled plazas, and dirt roads.The authors describe city deployment as less forgiving and requiring consistent performance.Source: §5.2 · p. 7
5.3. Evaluation Metrics
The section defines tracking and next-token prediction errors as complementary measures of locomotion performance, and reports that the two metrics consistently predict it.
5.3. Evaluation Metrics
- The authors report that tracking and prediction error can consistently predict locomotion performance.Figure 6 also presents tracking-error comparisons against a state-of-the-art benchmark and the improvement from adding action-free trajectories to action-labeled RL data.Source: §5.3 · p. 7, Figure 6 · p. 7
- Tracking error measures command following through position deviation from the ideal base trajectory during 10-second simulated walks.Commands use sampled forward and angular velocities with zero lateral velocity; the robot starts from rest.Source: §5.3 · p. 7
- Prediction error is the held-out next-token loss on state-action trajectories, combining state and action prediction errors.The validation trajectories come from the RL policy and are separated from training data.Source: §5.3 · p. 7
5.4. Comparison to the state of the art
The proposed policy tracks commands better than the RL comparator across tested turns, while prediction error is strongly correlated with tracking performance across models.
5.4. Comparison to the state of the art
- The policy outperforms the RL controller at all tested turning speeds and tracks straight-line walking nearly perfectly.In the quantitative comparison, it consistently has lower mean position tracking error across commanded yaw velocities, despite training on trajectories from that RL policy.Source: §5.4 · pp. 7–8
5.5. Prediction error correlates with performance
- Across 14 models, prediction error and task tracking error correlate with Pearson r = 0.87, indicating that lower validation prediction error likely accompanies more accurate command following.Figure 7 plots the two errors for the models.Source: §5.5 · p. 8, Figure 7 · p. 8
5.6. Gait quality
The policy retains a smoother, more cyclic gait than the RL policy, walks backward under an unseen command, and benefits from training on incomplete trajectories.
5.6. Gait quality
- At a heading velocity of 0.5 m/s, the policy’s left-knee phase portrait has fewer irregularities and a smoother, cyclic gait than the RL policy.The policy retains the RL policy’s overall phase-portrait shape while showing fewer aberrations.Source: Figure 8 · p. 8, §5.6 · p. 8
5.7. Generalization to unseen commands
- The robot walks backward at speeds up to 0.5 m/s under negative heading commands, although backward walking was absent from action-labeled training data.It does so without falling.Source: §5.7 · p. 8
5.8. Training with action-free data
- Joint training on complete and incomplete trajectories consistently improves performance over training only on complete trajectories.Incomplete sources include YouTube human videos without action labels.Source: §5.8 · p. 8
5.9. Scaling studies
Position tracking error decreases with more training trajectories and larger models; longer context windows also produce better policies.
Training data
- More training trajectories reduce position tracking error.This is a positive signal for increased performance with larger datasets.Source: §5.9 · p. 9
Context length
- Larger context windows, tested at 16, 32, and 48 steps, produce better policies.The authors suggest this reflects in-context adaptation that improves with scale.Source: §5.9 · p. 9
Model size
- Tracking error decreases monotonically as model size increases from 1M to 2M to 8M parameters.The models vary in embedding dimension, attention heads, and transformer blocks.Source: §5.9 · p. 9
5.10. Ablation studies
The ablations compare tokenization, prediction alignment, training schedule, and prediction targets, finding benefits from modality alignment and state-action prediction alongside trade-offs in tokenization.
Concatenated vs. separate tokens
- Separate observation and action tokens yield lower tracking error, while concatenated tokens yield lower prediction error; overall performance is comparable.Separate tokens double input length and add computation overhead.Source: §5.10 · p. 9
Modality-aligned vs. non-aligned prediction
- Modality-aligned prediction performs better than non-aligned prediction.Without alignment, predicted observations feed action prediction at test time, unlike the real observations used during training, causing a discrepancy that can introduce action-prediction error.Source: §5.10 · p. 9
Joint training vs. staged training
- Pre-training on state prediction followed by fine-tuning on complete data with action prediction performs similarly to joint training.The staged approach also yields a reasonable locomotion policy.Source: §5.10 · p. 9
State-action prediction vs. action-only prediction
- Predicting both states and actions improves trajectory-tracking performance over predicting actions alone.The authors hypothesize the additional learning signal supports richer world representations.Source: §5.10 · p. 9
6. Discussion
The paper presents self-supervised humanoid locomotion from diverse sensorimotor trajectories and demonstrates zero-shot real-world walking by a full-sized humanoid.
6. Discussion
- The approach trains a model on sensorimotor trajectories from neural network policies, model-based controllers, human motion capture, and YouTube videos.The paper presents this as a self-supervised approach to real-world humanoid locomotion.Source: §6 · p. 9
- The model enables a full-sized humanoid to walk zero-shot in the real world.The authors suggest generative modeling of large trajectory collections as a promising path toward challenging real-world robot control tasks.Source: §6 · p. 9
- The paper frames generative modeling of large trajectory collections as a promising path for learning challenging real-world robot control tasks.Source: §6 · p. 9