PaperlayerView exact revision on arXiv

Source-linked AI summary

VioLA: Learning Generalist Humanoid Control Policies from Human Data

Mert Albaba, Jens Beißwenger, Anna Manasyan, Daniel Marta, Michael J. Black, Wieland Brendel, Andreas Krause, Georg Martius, Martin Riedmiller

arXiv:2610.12435v1cs.ROcs.LG

TL;DR

Humanoid instruction following requires coordinated whole-body control, while robot demonstrations for training are limited. VioLA predicts body and hand motion latents shared across human and robot demonstrations, then uses pretrained controllers to execute them. On a real robot, it achieves 100% locomotion success and 88.6% manipulation success without task-specific fine-tuning.

  • Problem

    Humanoid generalist policies must coordinate whole-body movement, but robot demonstrations are limited and human recordings do not contain robot joint commands.

    Source: §1 · p. 2

  • Method

    VioLA predicts body and hand motion latents in shared human–robot spaces, with paired pretrained controllers converting those predictions into robot commands.

    Source: §1 · p. 2

  • Results

    On a real Unitree G1 without task-specific fine-tuning, VioLA achieves 100% locomotion success and 88.6% manipulation success.

    Source: §1 · p. 3, §5 · p. 12

  • Takeaways & Limitations

    Human recordings can teach VioLA which movements an instruction requires without a robot performing each demonstration.

    Source: §5 · p. 12

    Human supervision requires annotated body and hand motion, and human-then-mixed training improves locomotion while robot-only training achieves higher manipulation success.

    Source: §5 · p. 12

Abstract

from arXiv · show

Teaching a humanoid to follow instructions with its whole body runs into two obstacles. Its action space is large and tightly coupled: legs, arms, and fingers must move together while the robot keeps its balance, which makes joint-level actions hard to learn. And humanoid demonstrations are scarce, so current humanoid generalist policies do not follow new instructions out of the box and are fine-tuned on teleoperated demonstrations of each task before deployment. Human demonstrations exist in far larger numbers, but a person's motion is not a robot command. We remove both obstacles by changing what the generalist policy predicts. We introduce VioLA, a generalist humanoid policy that predicts body and hand motion latents instead of joint commands. A pretrained body- and hand-controller execute these latents on the robot. Their corresponding motion encoders map human and robot motion into the same latent spaces. A human recording is therefore labeled in the policy's action space, and the training demonstration pool contains 140.6 million frames, 93.2% of them human. As a result, VioLA follows locomotion instructions on the real robot zero-shot, without task-specific fine-tuning, reaching 100% success where GR00T N1.7 and $\Psi_0$ reach 16.7% and 0%, respectively. It also reaches 88.6% manipulation success without task-specific fine-tuning. The same approach works across two VLA and one world-action model backbones. A generalist policy trained on human demonstrations alone performs locomotion tasks on the real robot zero-shot. Code and checkpoints will be released.

1 INTRODUCTION

Humanoid instruction following must coordinate the whole body while learning from a limited supply of robot demonstrations. VioLA predicts body and hand motion latents shared by human and robot encoders, so human recordings can supervise its policy.

Approach

  • VioLA predicts body and hand motion latents, while pretrained controllers convert them into coordinated robot commands and human and robot encoders provide targets in the same action spaces.This lets the policy select which motions to make while the frozen controllers determine how to execute them.Source: §1 · p. 2

Human supervision

  • The training pool contains 140.6 million frames, 93.2% from human recordings, which are converted into the policy’s action space by motion encoders.The encoders and controllers remain frozen during policy training.Source: §1 · pp. 2–3

Results

  • 100% locomotion success and 88.6% manipulation success on a real Unitree G1 without task-specific fine-tuning for either task.The manipulation tasks include closing a laptop, hanging up a coat, and rotating a chair.Source: §1 · p. 3

Scope

  • The same recipe is tested with two VLA and one world-action model backbone, and a policy trained exclusively on human demonstrations shows zero-shot robot locomotion.The paper’s contributions also include comparisons of VLA and world-action backbones and of human and robot supervision.Source: §1 · p. 3

2 RELATED WORK

Prior work connects robot policies to language, motion priors, or human demonstrations in different ways. VioLA uses shared motion-latent action outputs for both body and hands, directly executable through paired pretrained robot decoders.

Generalist robot policies

  • Generalist policies extend from robot-arm VLAs to world-action models that add future-video prediction, while humanoid generalists such as GR00T and Ψ0 use task-specific teleoperation to adapt to target tasks.DreamZero jointly predicts video and actions, while DiT4DiT uses video-model features to condition action prediction.Source: §2 · p. 3

VioLA

  • VioLA’s body and hand predictions are both motion latents, and pretrained paired robot decoders turn them into joint commands.Paired encoders supply targets from both human and robot demonstrations.Source: §2 · p. 3, Figure 2 · p. 4

Motion priors and hierarchical control

  • Humanoid motion-prior systems learn or execute motion latents, but LeVERB’s hardware experiments replay latent sequences generated in simulation.SONIC pairs motion encoders with a body controller that executes their latents.Source: §2 · p. 3

Learning from human demonstrations

  • Human-data methods variously use physical shadowing, reconstructed motion, shared pose supervision, human-centric wrist and hand actions, or human tokens that are not passed to a robot controller.VioLA instead labels original human images with the same latent commands its frozen robot decoders execute, without a corresponding physical robot demonstration.Source: §2 · p. 4

3 VIOLA

VioLA predicts shared body and hand motion latents as its action space, with pretrained encoders and decoders kept frozen while the policy learns. Human and robot demonstrations are labeled in that space, and deployment uses the decoders to execute predictions continuously.

3 VIOLA

  • The method’s central design is to use paired human–robot motion latent spaces as the policy’s action space, leaving pretrained encoders and decoders frozen and updating only the generalist policy.The decoders produce robot joint commands from the predicted body and hand motions.Source: §3 · p. 4

3.1 A SHARED LATENT ACTION SPACE

  • Each action concatenates a body and hand latent, and the policy predicts chunks of successive actions from an egocentric image, instruction, and robot proprioception.Predicting both latents together lets reaching and finger motion be timed to one another.Source: §3.1 · p. 5

3.2 LEARNING FROM HUMAN AND ROBOT DEMONSTRATIONS

  • Human and robot motion encoders map demonstrations into the same target space, so both sources train the same prediction head with the same loss.Human proprioception is supplied by tracking the demonstrated motion with the frozen body controller in simulation; demonstrations without hand annotations supervise only body predictions.Source: §3.2 · p. 5

3.3 FROM PREDICTED MOTIONS TO CONTINUOUS EXECUTION

  • At deployment, the body controller maps body latents and recent robot state to joint commands, while the hand decoder maps hand latents to finger commands.Body and hand commands update at 50 Hz as the body controller reads the changing robot state.Source: §3.3 · p. 6
  • Real-time chunking conditions each new motion-latent prediction on the latents scheduled to execute before inference finishes, so it continues the ongoing motion across a chunk switch.The paper measures continuity across chunk switches in Section 4.7.Source: §3.3 · p. 6

4 EXPERIMENTS

Experiments evaluate VioLA without task-specific fine-tuning on held-out episodes, laboratory settings, and object instances, while the evaluation task families overlap the training data. Results cover locomotion, manipulation, motion-token predictions, human supervision, backbone choice, and continuity between action chunks.

4.1 SETUP

  • Evaluation uses five trials per task; task families overlap training, but episodes, the laboratory environment, and object instances are held out.Zero-shot means deployment without task-specific fine-tuning.Source: §4.1 · p. 6, §4 · p. 6

4.2 ZERO-SHOT LOCOMOTION

  • VioLA completes 30/30 locomotion and 31/35 manipulation trials; GR00T N1.7 records 5/30 locomotion and 0/35 manipulation successes, while Ψ0 records 0/30 and 3/35.The trials are conducted without task-specific fine-tuning.Source: §4.2 · p. 6, Figure 3 · p. 7, §4.3 · p. 7

4.3 ZERO-SHOT MANIPULATION

  • Every VioLA variant fails all five bottle-carrying trials; including that task lowers manipulation success from 31/35 (88.6%) to 31/40 (77.5%).Failures occur at different stages, including grasp timing, the transition from grasping to walking, and upright release.Source: §4.3 · p. 8

4.4 PREDICTING MOTIONS, NOT JOINTS

  • Chair rotation uses an average of 60 body and 31 hand tokens, while laptop closing uses 82 body and 43 hand tokens over roughly ten seconds.The frozen controllers continue updating joints at every control step.Source: §4.4 · pp. 8–9, Figure 5 · p. 9

4.5 LEARNING FROM HUMAN DEMONSTRATIONS

  • Human-only VioLA succeeds in 12/30 locomotion trials; human-then-mixed training reaches 30/30, compared with 24/30 for robot-only training after 200k updates.Human-only training transfers behaviors to the robot without robot demonstrations in generalist-policy training.Source: §4.5 · p. 9, Figure 6 · p. 10
  • Robot-only training reaches 33/35 manipulation successes (94.3%), versus 31/35 (88.6%) for human-then-mixed; the mixed schedule helps locomotion, while robot-only training does better on shampoo grasping.The two variants match on the other five manipulation tasks.Source: §4.5 · pp. 9–10

4.6 COMPARING POLICY BACKBONES

  • GR00T and DiT4DiT VioLA each succeed in 30/30 locomotion trials; on manipulation, GR00T reaches 31/35, π0.5 16/35, and DiT4DiT 9/35.The variants use the same training data and budget without task-specific fine-tuning.Source: Figure 7 · p. 10, §4.6 · p. 10

4.7 CONTINUITY ACROSS MOTION CHUNKS

  • RTC reduces the largest body-latent step change from 0.332 to 0.064 for Stand Still and from 0.455 to 0.070 for Raise Arm over the displayed 5.1-second recordings.Without RTC, sharp spikes recur near chunk switches; with RTC, changes there stay close to between-switch variation.Source: §4.7 · p. 11, Figure 8 · p. 11

5 LIMITATIONS AND CONCLUSION

On a real Unitree G1, VioLA achieves zero-shot locomotion and manipulation success, while its limitations include dependence on pretrained motion modules, annotation needs, and evaluation scope.

5 LIMITATIONS AND CONCLUSION

  • VioLA depends on the movements its pretrained body and hand modules can execute, and its hand decoder lacks contact-force feedback.The trials show remaining difficulties in reaching to grasp objects and retaining them through task completion; contact feedback is proposed as one improvement direction.Source: §5 · p. 12
  • Human-motion supervision requires reliable body and hand motion annotations, and human-then-mixed training improves locomotion while robot-only training achieves higher manipulation success.How to retain the locomotion benefit while improving object interaction remains open.Source: §5 · p. 12
  • The main evaluation covers thirteen tasks on one Unitree G1, with five trials per task; broader testing across tasks, environments, and embodiments is needed.Source: §5 · p. 12
  • The policy predicts body and hand movements from images, instructions, and proprioception, while frozen pretrained modules turn predictions into joint commands.Human recordings can provide action targets without a robot performing each demonstration.Source: §5 · p. 12
  • VioLA achieves 100% locomotion success and 88.6% manipulation success on a real Unitree G1 without task-specific fine-tuning.GR00T performs best among the tested backbones, particularly for manipulation.Source: §5 · p. 12

A.1 MOTION REPRESENTATION AND DATA CONVERSION

The appendix specifies the motion codes and demonstration conversion used to train VioLA, including closed-loop construction of body targets and a staged data-sampling schedule.

A.1 MOTION REPRESENTATION AND DATA CONVERSION

  • Each encoder produces a 64-coordinate motion code from a reference window, while policy chunk length determines how many successive codes it predicts.At 50 Hz, robot and hand encoders use ten frames spaced five frames apart; the human body encoder uses ten consecutive frames.Source: §A.1 · p. 17
  • Training examples concatenate body and hand encoder outputs as targets and pair the target chunk with an image, instruction, and robot state.Human body motion is SMPL-based and hand motion is fitted MANO; human examples use simulated robot states, while robot examples retain recorded proprioception.Source: §A.1 · p. 17
  • Body targets for human and robot demonstrations are generated by tracking reference motion with the frozen controller in simulation, recomputing codes from the simulated robot’s current heading.The simulator advances after the encoder computes a code and the controller decodes it into joint commands.Source: §A.1 · p. 17
  • Robot hand targets use commanded finger motion where available, but hand proprioception uses measured motion; examples without hand annotations contribute only to body loss.Commanded fingers may close farther than an object permits, and motion targets are computed before generalist-policy training.Source: §A.1 · p. 17
  • The body decoder uses a 64-coordinate code and ten frames of proprioceptive history to produce 29 joint-position targets, with control at 50 Hz.Four 200 Hz physics steps occur per control update.Source: §A.1 · p. 17
  • The converted pool combines EgoSuite human recordings with four robot corpora; human data comprise 64.6% of episodes and 93.2% of frames, with all data converted to 50 Hz.Humanoid Everyday, PSI, and UnifoLM-WBT contain real-G1 demonstrations; LeVERB contains simulated G1 demonstrations.Source: Table 1 · p. 18, §A.1 · p. 18
  • VioLA trains for 150k human-only steps, then 50k steps sampling each of five corpora uniformly; this gives human data an expected 80% of examples.Robot-only variants train on the four robot corpora, while the human-only variant uses human demonstrations throughout.Source: §A.1 · p. 18

A.2 HAND MOTION INTERFACE

A pretrained paired hand encoder–decoder maps human and robot hand motion into shared codes that the generalist policy learns to predict.

A.2 HAND MOTION INTERFACE

  • The paired hand module maps human and robot hand motion into a shared 64-coordinate space, with Dex3 and Inspire decoders producing hardware-specific finger trajectories.The hand encoders and decoders remain frozen during policy training; the policy predicts their codes rather than learning the hand prior.Source: §A.2 · p. 18

A.3 BACKBONE AND ACTION-EXPERT INSTANTIATIONS

VioLA uses three backbone configurations with shared training data and budget, but different action-chunk lengths and implementation details.

A.3 BACKBONE AND ACTION-EXPERT INSTANTIATIONS

  • The GR00T, π0.5, and DiT4DiT variants use the same training data and budget and each predict 128 coordinates per action step.Each step contains 64 body and 64 hand coordinates; GR00T predicts 40 successive steps and the other variants predict 50.Source: §A.3 · p. 18
  • π0.5 receives body and finger state plus a delayed hand code that is zeroed with probability 0.5 during training.The 50-tick delay prevents the code’s reference window from exposing future targets; the code can also be zeroed at deployment.Source: §A.3 · p. 18
  • GR00T combines a Cosmos-Reason2 vision-language backbone with a flow-matching action transformer and receives body, finger, and gravity state.Its state dropout is 0.2.Source: §A.3 · p. 18
  • The schedule samples human-only data for 150k steps, then samples all five corpora uniformly for 50k steps.The human recordings provide 93.2% of available frames, while sampling probabilities determine their training exposure.Source: Figure 9 · p. 19
  • DiT4DiT uses Cosmos-Predict2.5-2B features to condition its action transformer, with video and motion flow-matching losses during training.Training video supervision uses the current frame and four future frames; deployment has only the current observation and predicts future video features.Source: §A.3 · p. 19

A.4 FLOW-MATCHING CONVENTIONS AND MASKS

This section specifies flow matching for motion-latent prediction, including how masked targets enter the loss and how the world-action model retains video prediction.

A.4 FLOW-MATCHING CONVENTIONS AND MASKS

  • Inference starts from noise at τ = 1 and integrates the learned velocity backward toward a clean action chunk at τ = 0.Training mixes each target chunk with noise and trains the action expert to predict the difference, treating encoder codes as continuous coordinates.Source: §A.4 · p. 19
  • DiT4DiT retains its future-video prediction objective alongside motion prediction, with human future frames supervising video and encoded body and hand motion supervising action.The video and motion losses are summed.Source: §A.4 · p. 19
  • Losses exclude invalid frames and missing hand targets; π0.5 averages annotated coordinates per step, while GR00T and DiT4DiT average valid coordinates across the batch.For fully annotated batches the reductions are equivalent up to GR00T’s 10^-6 denominator offset; with missing hand annotations, they weight steps and coordinates differently.Source: §A.4 · pp. 19–20
  • All VioLA backbone variants predict the same 128 body and hand coordinates, while chunk length and state conditioning follow each backbone.The table caption specifies that targets are not normalized.Source: Table 2 · p. 20
  • Backbones use different flow-matching time samplers; GR00T’s opposite time direction is equivalent to Equation 6 after reversing time and velocity sign.Each backbone keeps its own time sampler and numerical solver settings.Source: §A.4 · p. 20

A.5 DECODER EXECUTION

This appendix section specifies decoder execution and evaluation procedures, then documents object-transport failures, timing, chunk-transition measurements, and motion-token calibration.

A.5 DECODER EXECUTION

  • The body controller updates every 20 ms; the hand decoder reconstructs ten frames, and execution smooths frame five before converting it to finger commands.For the G1’s Inspire hands, the decoder produces 12 finger targets; Dex3-trained policies convert measured Inspire motion into the policy’s Dex3 state representation.Source: §A.5 · p. 20

A.6 EVALUATION PROTOCOL

  • Evaluation uses locally available objects and five trials per task; success requires completing the instruction, with only a score of one counting as success.Evaluation tasks overlap the training task families, but episodes, laboratory environment, and object instances are held out; zero-shot means no task-specific fine-tuning.Source: §A.6 · pp. 20–21

A.7 FAILURES IN GRASPING AND OBJECT TRANSPORT

  • 5/5 bottle-carrying trials fail for every VioLA variant; including this separate task lowers VioLA manipulation success from 31/35 (88.6%) to 31/40 (77.5%).The variants fail at different stages, including grasping, beginning to walk, and stable release.Source: §A.7 · p. 21

A.8 EXECUTION TIMING

  • Timing summaries use 161 GR00T, 83 π0.5, and 57 DiT4DiT VioLA episodes, averaging prediction delay and chunk-arrival intervals per episode.The table reports means with one standard error; the archive includes additional runs beyond scored trials.Source: §A.8 · pp. 21–22, Table 3 · p. 22

A.9 MEASURING CHUNK-TRANSITION CONTINUITY

  • Four selected 5.1-second traces compare continuous body latents at 20-ms steps; RTC and non-RTC runs have slightly different prediction delays and chunk intervals.Mean delay is 138.9 ms with RTC versus 154.1 ms without for Stand Still, and 130.0 versus 155.3 ms for Raise Arm; these are four examples, not trial averages.Source: §A.9 · p. 22

A.10 MOTION TOKEN COUNTS AND CALIBRATION

  • Motion-token counting groups body and hand predictions separately by grid-based distance to retained representatives, which remain original continuous predictions.A prediction joins its nearest retained representative within threshold ρ; otherwise it becomes a new representative.Source: §A.10 · p. 22
  • Calibration selects the largest tested threshold passing the tracking-error reference in at least 95% of episodes; grouping measures local decoded-pose differences, not accumulated execution error.The body reference is 2.57 cm and the hand reference is 1.44 cm; hand calibration uses Dex3 fingertip proxies rather than the deployed Inspire trajectory.Source: §A.10 · p. 23
  • 8 body and 32 hand thresholds satisfy mean-error budgets in all 84 calibration, 21 held-out, and three chair-alt transfer episodes.The finer thresholds reduce mean episode-level wrist p95 from 5.09 to 3.03 cm and worst-finger p95 to 0.31 cm for chair rotation and laptop closing.Source: §A.10 · p. 23

Related papers

Loading 2610.12435v1…