HarmoWAM: Harmonizing Generalizable and Precise Manipulation via Adaptive World Action Models

Motivation

• Existing WAM paradigms trade off exploration and precision: Imagine-then-Execute approaches near-perfect OOD object approach but can fall to 60% manipulation success, whereas Joint Modeling exceeds 80% precision near targets but stays below 60% for OOD approach.The contrast reflects world-model generalization versus an exploration space constrained by SFT data.Source: §1 · p. 2

Method

• HarmoWAM uses world-model spatio-temporal conditions to coordinate a predictive expert for temporally coherent action generation with a reactive expert that infers actions from predicted observations.The unified design targets both generalizable transit and fine-grained manipulation.Source: §1 · pp. 2–3
• The Process-Adaptive Gating Mechanism predicts task stage from visual observations and switches experts across transit and interaction phases.A process-aware dataset built from keyframe extraction trains the gate to route control according to interaction events.Source: §1 · pp. 2–3

Evaluation

• 33% and 29% average OOD gains over leading VLA and WAM methods, respectively, establish HarmoWAM’s strong zero-shot generalization across three variation types.In ID settings, it also exceeds leading VLA and WAM methods by 15% and 11%, respectively.Source: §1 · p. 2
• Experiments cover six real-world tasks, including four single-arm and two dual-arm tasks, under ID and three OOD scenarios varying background, object position, and object instances.The evaluation also examines long-horizon deployments while preserving both transit generalization and manipulation precision.Source: §1 · pp. 2–3

Vision-Language-Action Models

• VLA research has increasingly adopted continuous generative policies and flow-based action generation, but current models remain heavily constrained by their training-data distribution.The passage identifies costly data requirements for improving generalization.Source: §2 · p. 3

World Action Models

• Imagine-then-Execute methods predict future visual observations before deriving actions, using video dynamics to provide foresight for downstream decisions.Examples include DreamGen and WoW [13].Source: §2 · p. 3
• Joint Modeling methods combine video and action generation, using predictive video representations as latent supervision or action conditioning.Examples include VPP, Video2Act, Genie Envisioner, Video Policy, UVA, UWM, Cosmos policy [26], and Motus [4].Source: §2 · p. 3

HarmoWAM

• HarmoWAM exploits both explicit and implicit world-model priors to unify generalizable transit and precise manipulation in an end-to-end WAM.This positions the method as a synthesis of the two established WAM paradigms.Source: §2 · p. 3

3.2 Motivation

• Imagine-then-Execute generalizes transit but lacks interaction precision, whereas Joint Modeling achieves precise manipulation but degrades in OOD transit, motivating a unified framework.The Imagine-then-Execute baseline has interaction success below 75% in ID and 55% in OOD, while Joint Modeling’s OOD transit success drops to 32% despite 95% average OOD interaction success when initialized near the target.Source: §3.2 · p. 4

3.3 HarmoWAM Framework

• HarmoWAM uses a world model to provide explicit future videos and implicit latent representations that condition predictive and reactive Transformer-based action experts.The predictive expert uses current latent features for structured, temporally coherent actions, while the reactive expert combines predicted frames with future latent dynamics for action inference.Source: §3.3 · pp. 4–5, Figure 2 · p. 5
• The predictive expert generates fine-grained actions from latent dynamics, while the reactive expert fuses predicted visual features and latent representations before decoding actions.The reactive pathway concatenates pooled latent features with visual features and applies an Orientation Decoder; the resulting inverse dynamics is informed by spatio-temporal knowledge.Source: §3.3 · p. 5

3.4 Process-Adaptive Gating Mechanism

• Process-Adaptive Gating predicts whether an observation belongs to interaction or transit and routes control to the predictive expert above 0.5, otherwise selecting the reactive expert.Interaction labels are derived from gripper-state changes or task-specific end-effector height thresholds; the predictive expert focuses more on manipulated objects, whereas the reactive expert attends to the gripper and surrounding task-relevant environment.Source: §3.4 · p. 6

3.5 Downstream Training Recipe

• Training first fine-tunes the world model on task-specific visual dynamics, then freezes it while optimizing the predictive expert, reactive expert, and gating network with separate losses.The world model uses conditional Flow Matching; the predictive expert uses diffusion denoising, the reactive expert uses Smooth L1 supervision, and the combined objective weights the reactive and gating losses by 0.1 and 0.05.Source: §3.5 · p. 6

4.2 In-Domain Results

• HarmoWAM achieves 89% in in-domain evaluations, outperforming π0.5 at 74%, Cosmos-Policy at 78%, and Wan+AnyPos at 67%.Its standard deviation under repeated evaluation is ±3%.Source: §4.2 · p. 7

4.3 Generalization

• Across OOD settings, evaluation covers unseen backgrounds, object positions, and manipulated objects, with representative configurations shown in Figure 4.The experiments use six real-world tasks and randomized tabletop object positions.Source: §4.1 · p. 7, §4.3 · p. 8, Figure 4 · p. 9
• HarmoWAM reaches 81% under unseen backgrounds, 85% with unseen objects, and mitigates the transit–interaction trade-off in spatially unseen regions through coordinated experts.Under unseen backgrounds, it outperforms Wan+AnyPos at 53% and π0.5 at 60%; under unseen positions, π0.5 and Cosmos-Policy fall to 32% and 26%.Source: §4.3 · p. 8

4.4 Ablation Study

• Removing either action expert sharply reduces OOD performance, with the reactive expert supporting exploration and the predictive expert supporting precise manipulation.Removing the reactive expert lowers position-OOD success to 14%, while removing the predictive expert yields 56% position-OOD and 60% object-instance-OOD success.Source: §4.4 · p. 9
• Process-Adaptive Gating outperforms output-averaging alternatives, while removing world-model video latents substantially degrades both experts.Averaging causes a 46% position-OOD drop and Keyframe-Based Averaging a 31% drop; removing latents reduces reactive OOD average to 54% and predictive ID performance from 95% to 62%.Source: §4.4 · p. 9, Figure 5 · p. 9

5 Conclusion

• HarmoWAM unifies predictive and reactive control under shared spatio-temporal priors to achieve broad exploration and fine-grained interaction in one closed-loop policy.The conclusion reports consistent superiority across diverse tasks and unseen environments.Source: §5 · pp. 9–10

A Real-World Set-up

• The real-world platform uses Franka Research 3 manipulators, multi-view Intel RealSense cameras, UMI grippers, and synchronized proprioceptive feedback.The single-arm setup uses three cameras, while the dual-arm setup uses one global and two wrist views.Source: A Real-World Set-up · p. 15, Figure 6 · p. 15

Single-Arm Tasks

• The dataset includes four single-arm tasks: picking fruit, stacking cans, pouring Coke, and writing “Yes”.These tasks test grasping, stable placement, precise alignment, rotational control, and sequential writing.Source: Single-Arm Tasks · pp. 15–16

Dual-Arm Tasks

• The two dual-arm tasks require flower handover and vase insertion, or placing items into a bag followed by zipper closure.Put Items to Bag and Zip is the longest-horizon task, averaging approximately 400 steps.Source: Dual-Arm Tasks · p. 16

C.1 Baselines.

• Baselines span VLA policies, Imagine-then-Execute WAMs, and Joint Modeling WAMs, including π0.5, QwenVLA-OFT, Wan+AnyPos, VPP, and Cosmos-Policy.Wan+AnyPos combines Wan2.2-TI2V-5B future-video prediction with AnyPos inverse dynamics.Source: §C.1 · p. 16
• All methods use the same real-world demonstrations, task instructions, control spaces, and testing protocol, except Wan+AnyPos uses a third-view camera perspective.Each method is evaluated with the corresponding single-arm or dual-arm action dimensionality.Source: §C.1 · p. 16

C.2 Training Protocol.

• The world model is initialized from Wan2.2-TI2V-5B, pretrained on broad robot datasets, and fully fine-tuned for task-specific video prediction.The frozen world model then supplies explicit and implicit video conditions to the action experts and gating network.Source: §C.2 · pp. 16–17
• The predictive expert uses diffusion denoising, the reactive expert uses Smooth L1 regression, and the gating network uses binary cross-entropy.Single-arm and dual-arm action spaces contain 7 and 14 dimensions, respectively.Source: §C.2 · p. 17

C.3 Out-of-Domain (OOD) Scenario Construction

• OOD construction tests unseen distractors and lighting, spatially disjoint target regions, and substitute objects differing in appearance and geometry.These settings assess visual robustness, spatial generalization, and semantic or geometric adaptation during interaction.Source: §C.3 · p. 17

D.1 Additional Preliminaries

• Inverse Dynamics Models translate predicted future video trajectories into executable action sequences in the Imagine-then-Execute paradigm.The IDM conditions on future frames from the predicted trajectory.Source: §D.1 · p. 17

D.2 Motivation Experiments Details

• Joint Modeling often fails during OOD transit, whereas Imagine-then-Execute reaches target vicinity but can produce inaccurate grasping, placement, or stacking actions.The evaluation separates transit from interaction to measure generalization and precision independently.Source: §D.2 · pp. 18–19
• The study evaluates two real-world tasks across ID and background, position, and semantic OOD settings using separate transit and interaction measures.Joint Modeling interaction is initialized near the target object to decouple precision from transit failures.Source: §D.2 · p. 18

D.3 Process-Adaptive Gating Details

• The gating pipeline labels interaction segments around gripper-state changes and task-specific end-effector height events, with other frames treated as transit.For dual-arm tasks, either arm can trigger the interaction label and its surrounding temporal window.Source: §D.3 · pp. 19–20
• The gating classifier achieves 96.95% average frame-level accuracy on 1,637 held-out test frame pairs, supporting adaptive switching between experts.Labels use a 20-frame window before and after each key interaction event.Source: §D.3 · p. 20

E.1 Additional Task Execution Visualizations.

• Complete execution visualizations show smooth, continuous trajectories during can alignment, pouring, and flower insertion.These examples emphasize phases requiring fine-grained control.Source: §E.1 · p. 20

E.2 Additional Generalization Visualizations.

• Additional examples cover all three OOD dimensions: unseen background, unseen position, and unseen object semantics.The visualizations present complete examples under each distribution shift.Source: §E.2 · p. 20

F.1 In-Domain(ID) Stage-Wise Results.

• HarmoWAM maintains 85% average success on dual-arm tasks, outperforming the next-best baseline by 12 percentage points across complex collaborative procedures.It maintains 70% in final flower insertion and 85%, 80%, and 80% across the final three bag-and-zip stages.Source: §F.1 · p. 21, Table 6 · p. 21
• HarmoWAM reaches a 91% average success rate across all single-arm tasks, outperforming the next-best baseline, Cosmos-Policy, by 10 percentage points.Source: §F.1 · p. 21

F.2 Out-of-Domain(OOD) Stage-Wise Results.

• Under background variation, HarmoWAM averages 83% on single-arm and 76% on dual-arm tasks, outperforming the strongest baselines by 21 and 15 percentage points.It reaches 65% in the second stacking stage and 60% in final flower insertion, exceeding the best baselines.Source: §F.2 · pp. 21–22, Table 7 · p. 22, Table 8 · p. 22
• Under position variation, HarmoWAM reaches 83% on single-arm and 70% on dual-arm tasks, versus 51% and 45% for the best baselines.Its stage-wise results include 85% and 70% on Stack Coke Cans and 90%, 85%, and 65% on Put Flowers in Vase.Source: Table 7 · p. 22, Table 8 · p. 22, §F.2 · p. 23
• Under semantic variation, HarmoWAM achieves 87% on single-arm and 81% on dual-arm tasks, compared with 66% and 53% for the strongest baselines.The advantage is especially visible in contact-sensitive stages involving stacking and flower insertion.Source: Table 7 · p. 22, Table 8 · p. 22, §F.2 · p. 23

F.3 Additional Ablation

• Five denoising steps achieve 85% success without reducing inference speed, while more steps provide limited gains and lower inference frequency.Three steps produce 80% success with blurred predictions; 10 and 50 steps reach 85% and 87%, respectively.Source: §F.3 · p. 23, Table 9 · p. 23

G Failure Case Analysis

• Tilted stacking occurs when small front-back placement errors destabilize the released upper object, especially under difficult third-view visual estimation.Overall approach and stacking intent can remain correct despite inaccurate relative displacement estimates.Source: G Failure Case Analysis · p. 23
• Flower insertion can fail after successful handover and transport when grasp-induced pose variations cause the stem to contact the vase rim.The task combines bimanual handover, relative-pose recovery, and tight-tolerance insertion.Source: G Failure Case Analysis · pp. 23–24
• Zipper pulling may fail when the small zipper pull slips because gripper friction or contact stability is insufficient for sustained constrained force.A slight loss of contact can interrupt the pulling process and prevent closure.Source: G Failure Case Analysis · p. 24

H Broader Impact

• Deployment beyond controlled laboratories still requires safety guardrails, uncertainty-aware control, human supervision, and hardware-level constraints.The discussion notes risks from inaccurate predictions, OOD observations, and ambiguous instructions.Source: H Broader Impact · p. 25

• A fixed world-model generation horizon requires downstream tasks to preserve the same future-video horizon for alignment with learned spatio-temporal dynamics.This limits flexibility when tasks require different prediction horizons.Source: I Limitations · p. 25