WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time

• Human videos provide a scalable steering interface, but existing approaches often require robot data, human-side annotations, retargeting, or large-context conditioning.Source: §1 · p. 2
• Extensive experiments show that WAM-TTT consistently outperforms in-context-learning-based human-video conditioning baselines.Source: §1 · p. 2
• WAM-TTT adapts world action models at deployment time from raw human demonstrations without requiring robot actions.Source: §1 · p. 2
• The method absorbs human videos into an adaptive memory through self-supervised video prediction instead of treating them as trajectories to imitate.Source: §1 · p. 2
• Paired human-robot meta-training aligns human visual cues with robot behaviors through a key–value memory reconstruction objective.Source: §1 · p. 2
• WAM-TTT keeps the pretrained WAM frozen during deployment, avoiding full-model fine-tuning while supporting efficient and reusable steering toward new task variants.Source: §1 · p. 2

• World-action models couple future visual prediction with action generation, grounding policies in imagined state transitions rather than purely reactive action prediction.Source: §2 · pp. 2–3
• Existing world-action models primarily emphasize pretraining and do not adequately address steerability during deployment.Source: §2 · p. 3
• Test-time training stores current-sequence information in adaptive parameters, while WAM-TTT calibrates a branch so human-video features become residual skill memory inside the WAM.Source: §2 · p. 3
• Unlike robotics adaptation methods based on robot observations, rewards, or rollouts, WAM-TTT uses human demonstrations as its test-time information source.Source: §2 · p. 3
• WAM-TTT transfers skills from small unlabeled human-play videos without converting them into robot actions or explicit poses, generated demonstrations, robot-context examples, interaction rollouts, or deployment-time fine-tuning.Source: §2 · p. 3

3.1 Architecture

• The base LDA world-action model couples a video expert for visual latent tokens with an action expert for robot action tokens through joint attention.Source: §3.1 · p. 4
• TTT residual branches are added only to the video expert, where context produces Keys and Values and current video tokens produce Queries for fast-weight processing.Source: §3.1 · p. 4

3.2 Human-Robot Meta-Training

• Fast weights adapt through inner SGD on human video prediction and key–value reconstruction objectives, while the outer objective remains the paired robot WAM multitask loss.Source: §3.2 · pp. 4–5
• During meta-training, synchronized human tokens form Keys and Values while robot video tokens form Queries, enabling memory reconstruction to align human cues with robot behavior.Source: §3.2 · p. 4
• Human and robot episodes are synchronized by matching normalized robot episode phase to the nearest frame in the paired human video.Source: §3.2 · p. 5

3.3 Test-Time Training from Human Video

• At deployment, only video-side fast weights update from a small batch of action-free human videos; WAM parameters, slow projections, initialization, and action-expert parameters remain frozen.Source: §3.3 · p. 5
• The adapted fast weights are fixed during robot rollout, so deployment-time skill absorption requires no gradient step on the WAM.Source: §3.3 · p. 5

4.1 Experimental Setup

• The evaluation spans three robot embodiments, nine manipulation tasks, and Orig. versus New settings, with New combining changes in lighting, table height, and objects.Progress is averaged over 25 trials per task-setting cell.Source: §4.1 · p. 5, §4 · p. 5

4.2 Compared with baselines

• Qualitative rollouts pair robot filmstrips on unseen tasks with the human demonstrations used as deployment-time Key/Value.The human video is shown on the left and the robot rollout on the right.Source: Figure 4 · p. 6, §4.2 · p. 7
• 46.2% average progress across nine unseen-household tasks exceeds LDA by 13.7 points and WAM-ICL by 39.1 points.WAM-TTT wins seven tasks outright and ties on Flip Steak, while Stamp Paper is the exception.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 Ablation Study

• The ablation protocol isolates deployment human-video adaptation, TTT fast-weight memory, and human-robot meta-training with paired data.Comparisons include WAM-TTT versus w/o TTT, WAM-LoRA, w/o Meta Training, and w/o Memory Recon.Source: §4.3 · p. 7, Table 2 · p. 8

4.4 Generalization Preservation

• Generalization is tested after adapting on target-task human videos and then perturbing lighting, object position, and embodiment-related appearance.The protocol compares WAM-TTT with LDA, π0.5, and WAM-ICL.Source: §4.4 · pp. 7–8
• WAM-TTT maintains strong performance across all tested perturbation types while keeping the WAM backbone and action expert frozen.The comparison is designed to distinguish fast-weight adaptation from in-context conditioning using the same human demonstration.Source: §4.4 · p. 8

Conclusion

• WAM-TTT adapts only video-side fast weights at test time, using self-supervised video prediction and key–value memory reconstruction while the WAM and slow projections remain frozen.Meta-training uses paired human-robot demonstrations to prepare the memory interface for control.Source: §5 · p. 8
• The recipe yields deployment-time skill absorption without gradient updates to the WAM and matches or exceeds online-adaptation baselines on a real-robot manipulation suite.This conclusion is reported for the paper’s Section 4 evaluation.Source: §5 · p. 8

Limitations

• Meta-training assumes paired human and robot episodes cover the same skill-phase distribution; misalignment can degrade the inner adaptation signal without being flagged by the loss.This is identified as the first stated caveat.Source: §5 · p. 8
• Adaptation is bounded by fast-weight expressiveness and fixed slow projections, with weaker adaptation expected as deployment tasks drift farther from the meta-training pairing distribution.The boundary of this degradation has not been characterized empirically.Source: §5 · p. 8
• The deployment interface accepts only egocentric human RGB frames and does not use hand-pose, contact, or 3-D scene cues.The paper identifies these cues as useful in related work.Source: §5 · p. 8

Future Direction

• The meta-training/test-time interface could extend beyond human Key/Value to auxiliary modalities with phase-pairable training signals.The outlook proposes parallel fast-weight branches under the same loss-only-then-residual regime.Source: §5 · p. 9

Stream integration

• TTT modifies only the video stream; the action expert remains untouched, so human-video adaptation enters through the modality naturally supervised by action-free human observations.Hatted block outputs receive a video residual, whereas the action output is passed directly to the next block.Source: A Meta-training algorithm · p. 14

Runtime memory

• The fast-weight MLP stores human keys and values parametrically, replacing an explicit episode-length key–value cache with an attention-like readout.The readout applies θO fW(θQ(zr)) and injects the result into the video stream as a residual.Source: A Meta-training algorithm · pp. 14–15

Memory objective

• Key–value reconstruction trains the fast weights to map synchronized human keys to values, providing the memory behavior needed for robot-query retrieval.In the linear special case, this objective defines a kernel-free, softmax-free linear-attention readout; the deployed nonlinear MLP is its smooth learned analogue.Source: A Meta-training algorithm · pp. 14–15

Meta-training

• The inner loop updates fast weights using human video prediction and key–value reconstruction, while the outer loop trains slow projections, initialization, and WAM parameters through robot multitask loss.The robot loss retains the underlying video- and action-side diffusion targets, with the TTT residual shifting the robot video stream.Source: A Meta-training algorithm · pp. 14–15

Test-time adaptation

• At deployment, the same combined objective adapts only fast weights from action-free human videos, while the WAM, slow projections, and initialization remain frozen.Human video prediction and memory reconstruction are both computable without robot-side supervision, preserving the intended human key–value readout.Source: A Meta-training algorithm · pp. 14–15

Hyperparameters

• Table B.1 is identified as the source of the main WAM-TTT hyperparameters.The supplied passage names the table but does not provide its parameter values.Source: B Hyperparameters and datasets · p. 15, Table B.1 · p. 16

Meta-training algorithm

• Algorithm A.1 phase-aligns each robot timestep with a nearest-phase human frame before running inner adaptation on the combined human-side objective.Each iteration computes human video prediction and per-layer memory reconstruction losses, then updates all fast-weight layers simultaneously.Source: B Hyperparameters and datasets · p. 16
• After adaptation, the robot forward pass applies TTT residuals to the video stream, and the outer loss backpropagates through the analytical inner update into WAM and slow parameters.The outer objective is the robot multitask loss, while deployment freezes the learned slow components and adapts only fast weights.Source: B Hyperparameters and datasets · p. 16

Embodiments and datasets

• The dataset contains 2,286 paired robot–human episodes across nine manipulation tasks, distributed across Unitree G1, Galbot gripper, and Galbot sharpa embodiments.Counts are 600, 544, and 1,142 episodes respectively; robot data come from a standardized cubicle and human videos from actual household evaluation scenes.Source: B Hyperparameters and datasets · p. 16

Data collection

• The benchmark uses a 5×2 task layout, with the long-horizon Multi-step Steak represented in two panels and the same task ordering for robot and human collection.Human demonstrations are egocentric GoPro recordings without hand-pose, joint-angle, or motion-retargeting annotations.Source: Figure B.1 · p. 17, Figure B.2 · p. 17, B Hyperparameters and datasets · p. 17

Orig. results

• 61.1% Orig. progress makes WAM-TTT the top method, exceeding no-TTT LDA at 50.2%, WAM-ICL at 48.4%, π0.5 at 31.8%, EGOSCALE at 30.1%, and WAM-COTRAIN at 29.8%.The full table reports Orig. and New progress across all nine tasks, with cells averaged over 25 trials.Source: C Full main results: Orig. and New… · p. 17, Table C.1 · p. 18

Orig. → New transfer

• WAM-TTT preserves 76% of its Orig. average under New, combining the highest Orig. score with the highest meaningful retention under household-scene shift.The retention analysis derives New/Orig ratios and absolute gaps from the per-method nine-task averages.Source: C Full main results: Orig. and New… · p. 18
• WAM-ICL retains only 15% and loses 41.3 points under the household shift, showing fragile transfer for in-context human-video conditioning.Its long-context conditioning that helps in distribution becomes a liability when visual statistics change.Source: C Full main results: Orig. and New… · p. 18

Form comparison

• WAM-COTRAIN has 85% retention but starts at 29.8%, so its high ratio is described as largely artifactual and its New performance remains second-worst after WAM-ICL.The comparison distinguishes relative retention from absolute task performance.Source: C Full main results: Orig. and New… · p. 18
• Among human-data methods, only WAM-TTT achieves both strong in-distribution accuracy and robust OOD transfer through fast-weight adaptation that preserves the pretrained backbone.Co-training sacrifices Orig. quality, whereas in-context conditioning collapses under scene shift.Source: C Full main results: Orig. and New… · p. 18

D Per-task progress definitions

• Per-task progress sums predefined milestone weights to 1.0, averages over 25 trials per task-setting cell, and reserves full score for satisfying the final goal.Scores are automatically derived from robot end-effector and known scene-object poses.Source: D Per-task progress definitions · p. 19
• Critical completion milestones receive most of the weight, including successful stamping at +0.50, pouring at +0.60, flipping at +0.45, and placing the third cup at +0.36.Smaller weights capture preliminary reaches and grasps that provide signal when trials stall before completion.Source: D Per-task progress definitions · pp. 19–20

E.1 Qualitative rollout gallery in unseen household, office, and kitchen scenes

• Figure E.1 shows representative WAM-TTT rollouts in unseen household, office, and kitchen scenes, including six benchmark tasks and three additional in-the-wild behaviors.Rows use five evenly spaced keyframes, and all rollouts come from the Table C.1 checkpoint after in-scene human-video TTT adaptation without per-task hyperparameter sweeps.Source: §E.1 · p. 21, Figure E.1 · p. 22

E.2 Direct lab-scene generalization without scene-specific human data

• Without scene-specific human videos, WAM-TTT retains useful behavior in an unseen lab scene with simultaneous lighting, tablecloth, object-instance, and target-pose shifts.The evaluation uses only meta-trained slow projections and initialization, isolating direct transfer from deployment-time human-video adaptation.Source: §E.2 · pp. 22–23, Figure E.2 · p. 23
• The lab-scene rollouts cover Stamp Paper, Transfer Bottle, Swap Place, Flip Steak, Multi-step Steak, and Pyramid Stacking across the two Galbot embodiments.The reported behavior transfers despite the joint out-of-distribution shift, indicating that meta-training calibration is not tied to cubicle statistics.Source: §E.2 · p. 23

E.3 Robustness across six axes of in-scene distribution shift

• Only test-time fast weights are updated, so human-side domain shift rewrites the adaptive memory without overwriting the pretrained policy or its visual-action prior.The WAM parameters, slow projections, and initialization parameters remain frozen during adaptation.Source: §E.3 · pp. 24–25
• WAM-TTT avoids co-training and in-context conditioning failure modes by isolating human-video adaptation in a residual fast-weight branch.The supplied passages contrast this design with backbone-updating co-training and conditioning streams exposed directly to perturbed observation statistics.Source: §E.3 · pp. 24–25
• The evaluated shifts include novel objects, changed lighting, raised tables, repositioned chassis with clutter, unfamiliar stamp affordances, and out-of-distribution target placement.These perturbations span both visual-scene statistics and manipulation geometry or affordances.Source: §E.3 · p. 24
• WAM-TTT evaluates object, lighting, height, chassis, affordance, and object-position shifts using perturbed in-scene human videos without checkpoint switching or per-axis hyperparameter tuning.The same checkpoint is used across all six axes, and each axis perturbs the deployment scene and paired human-side videos together.Source: §E.3 · pp. 23–24, Figure E.3 · p. 25

E.4 Data-ratio ablation

• 74.1 versus 73.7 on the 3-task average shows that (100, 100) matches (200, 0) at equal total budget, while (10, 190) falls to 51.4.The comparison indicates that paired human data can halve robot teleoperation cost without sacrificing performance, but cannot replace robot grounding entirely.Source: §E.4 · p. 26, Table E.1 · p. 26
• Adding 100 paired human episodes raises the 3-task average from 59.5 to 74.1 (+14.6 pts) at fixed robot budget.Doubling human episodes further to (100, 200) gives 73.3, marginally below the deployed (100, 100) setup.Source: §E.4 · pp. 25–26

E.5 Model-architecture ablation

• The architecture ablation varies only the VLM conditioning backend while retaining the full meta-training and test-time TTT pipeline.This isolates the contribution of VLM presence, pretraining, and joint adaptation.Source: §E.5 · p. 26, Table E.2 · p. 26
• Removing the VLM costs 28 points, removing VLM pretraining costs 20 points, and freezing the pretrained VLM costs 46 points on Table Bussing.The reported values are 72 for DiT-only, 80 for no-pretrain, and 54 for frozen VLM, versus 100 for the full model.Source: §E.5 · p. 26, Table E.2 · p. 26

E.6 Action-pseudolabel ablation

• WAM-TTT instead uses action-free human videos, with video-generation loss driving test-time updates because no robot action stream is available.This preserves the noise-tolerant human-side supervision used by the deployed design.Source: §E.6 · pp. 27–28
• The alternative pipeline estimates wrist pose, fits MANO, retargets fingertip and palm targets, and produces embodiment-matched pseudo-qpos for FD supervision.The FD objective predicts the next frozen-DINOv3 feature from the current human observation and pseudo-action.Source: §E.6 · p. 27
• 72.3 to 28.9 (−43.4 pts) shows that adding MANO-retargeted pseudo-actions and the FD loss uniformly harms the 4-task average.The comparison uses the deployed VG ONLY design against VG + FD with pseudo-actions under the same evaluation protocol.Source: §E.6 · pp. 27–28, Table E.3 · p. 28
• FD costs 41.4 points on Transfer Bottle, 66.7 on Table Bussing, 39.9 on Deliver Drink, and 25.5 on Swap Place.The damage is attributed to embodiment-mismatched actuation signals and residual single-view retargeting noise, including on the dexterous Swap Place robot.Source: §E.6 · p. 28