PaperlayerView exact revision on arXiv

Source-linked AI summary

DyWA: Dynamics-adaptive World Action Model for Generalizable Non-prehensile Manipulation

Jiangran Lyu, Ziming Li, Xuesong Shi, Chaoyi Xu, Yizhou Wang, He Wang

arXiv:2503.16806v2cs.ROcs.AI

TL;DR

Existing non-prehensile manipulation policies rely heavily on multi-view geometry and pose tracking and generalize poorly across changing physical conditions. DyWA jointly predicts future states and adapts dynamics from historical trajectories. It improves simulation success over baselines and reaches 68% average success in real-world experiments across varied geometries and physical conditions.

  • Problem

    Existing methods depend on multi-view cameras and precise pose tracking and struggle to generalize across variations in object mass and table friction.

    Source: §1 · p. 2

  • Method

    DyWA jointly predicts future states while adapting dynamics from historical observation-action trajectories through a dynamics-adaptive world action model.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    31.5% improvement in success rate over baselines in simulation and 68% average success rate in real-world experiments.

    Source: §1 · p. 2, §4.1 · p. 6

  • Takeaways & Limitations

    DyWA generalizes across object geometries, adapts to table friction, and remains robust on half-filled water bottles and slippery objects.

    Source: §1 · p. 2

    The point-cloud-only visual input struggles with symmetric objects and with transparent or specular objects whose depth is incomplete.

    Source: §5 · p. 8

Abstract

from arXiv · show

Nonprehensile manipulation is crucial for handling objects that are too thin, large, or otherwise ungraspable in unstructured environments. While conventional planning-based approaches struggle with complex contact modeling, learning-based methods have recently emerged as a promising alternative. However, existing learning-based approaches face two major limitations: they heavily rely on multi-view cameras and precise pose tracking, and they fail to generalize across varying physical conditions, such as changes in object mass and table friction. To address these challenges, we propose the Dynamics-Adaptive World Action Model (DyWA), a novel framework that enhances action learning by jointly predicting future states while adapting to dynamics variations based on historical trajectories. By unifying the modeling of geometry, state, physics, and robot actions, DyWA enables more robust policy learning under partial observability. Compared to baselines, our method improves the success rate by 31.5% using only single-view point cloud observations in the simulation. Furthermore, DyWA achieves an average success rate of 68% in real-world experiments, demonstrating its ability to generalize across diverse object geometries, adapt to varying table friction, and robustness in challenging scenarios such as half-filled water bottles and slippery surfaces.

1. Introduction

Non-prehensile manipulation extends robotic handling beyond grasping but remains difficult under partial observation and changing physical conditions. DyWA addresses these limits by adapting dynamics from historical trajectories while jointly predicting actions and future states.

1. Introduction

  • Non-prehensile manipulation includes pushing, sliding, toppling, and flipping for objects that are difficult or inefficient to grasp.Source: §1 · p. 1

2.1. Non-prehensile Manipulation

  • Planning-based methods require object properties or CAD models, whereas learning-based point-cloud methods support broader unseen-object generalization but commonly retain camera and tracking requirements.Source: §1 · p. 2, §2.1 · p. 3

1. Introduction

  • Existing methods depend on multi-view geometry and precise pose tracking, while often failing to generalize across object mass and table-friction variations.Source: §1 · p. 2
  • Single-view distillation suffers because partial observability omits geometry, Markovian models average across dynamics, and latent-action supervision misses contact dynamics.Source: §1 · p. 2
  • DyWA combines historical-trajectory dynamics adaptation, joint action and future-state prediction, and FiLM conditioning into a world action model.Source: §1 · p. 2
  • 31.5% improvement in success rate over baselines demonstrates the method’s simulation advantage across camera-view and pose-tracking settings.Source: §1 · p. 2
  • 68% average real-world success rate shows generalization across object geometries and table friction, including half-filled bottles and slippery objects.Source: §1 · p. 2

2.2. World Models

  • World models predict future states conditioned on actions; DyWA uses the world action model framing to jointly model actions and future states for manipulation.Source: §1 · p. 2, §2.2 · p. 3

3. Method

DyWA distills a privileged teacher into a vision-based student for 6D non-prehensile rearrangement. Its world action model predicts actions and task states, adapts dynamics from trajectory history, conditions features with FiLM, and executes variable-impedance actions after randomized simulation training.

3.1. Task Formulation

  • The task is 6D object rearrangement toward a target pose using partial point clouds, joint states, and end-effector poses.Source: §3.1 · p. 3

3.2. Pipeline Overview

  • The student is distilled from a privileged state-based RL teacher but uses only limited observations feasible for real-world deployment and transfers zero-shot after training.Source: §3.2 · pp. 3–4

3.3. World Action Model

  • A world action model jointly predicts the robot action and corresponding future task state, using object-centric state prediction rather than high-dimensional visual forecasts.Source: §3.3 · p. 4
  • The model encodes partial point clouds, proprioception, and a transformed goal point cloud before producing actions and next states.Source: §3.3 · p. 4

3.4. Dynamics Adaptation

  • Historical observation-action sequences are processed into an adaptation embedding supervised by privileged geometry and physics representations.Source: §3.4 · p. 4
  • FiLM converts the dynamics embedding into feature-wise scaling and shifting, allowing early world-model layers to adapt representations to varying dynamics.Source: §3.4 · pp. 4–5

3.5. Action Space with Variable Impedance

  • Variable impedance control executes end-effector residuals and joint-space impedance parameters, with inverse kinematics producing commanded joint positions.Source: §3.5 · p. 5

3.6. Training Protocol

  • The objective sums imitation, world-model, and adaptation losses, while training uses PPO, DAgger, domain randomization, and perturbations for sim-to-real robustness.Source: §3.6 · p. 5

4. Experiments

Experiments benchmark DyWA in simulation, analyze its components, test real-world generalization, and demonstrate applications with language-conditioned and pre-grasping manipulation.

4.1. Benchmarking Tabletop Non-prehensile Rearrangement in Simulation

  • 31.5% improvement in success rate over baselines across simulation evaluation tracks, with the largest gap under unknown states and single-view observations.The benchmark varies camera views, pose availability, object geometry, and dynamics properties including mass, friction, and restitution.Source: §4.1 · pp. 5–6, Table 1 · p. 6

4.2. Ablation Study

  • 73.3% success rate with both World Modeling and Dynamics Adaptation, compared with 59.9% when their complementary combination is absent.World Modeling or Dynamics Adaptation alone improves over naive DAgger by only 1.7% or 5.7%, respectively.Source: §4.2 · p. 6, Table 2 · p. 6
  • 82.2% success rate with World Modeling, Dynamics Adaptation, and FiLM, while FiLM raises the corresponding RMA result from 65.6% to 70.0%.The ablation evaluates FiLM as the conditioning bridge between adaptation embeddings and the world action model.Source: §4.2 · p. 7

4.3. Real-World Experiments

  • 68% average real-world success rate versus 36% for CORN across diverse objects without external pose tracking.The evaluation includes slippery objects and non-uniform mass distributions such as a half-filled bottle.Source: §4.3 · p. 7, Table 3 · p. 7
  • Dynamics Adaptation maintains consistent success rates and stable execution times across progressively different surface-friction conditions.Without dynamics adaptation, performance degrades and execution times become erratic across the tested tablecloths.Source: §4.3 · p. 7, Table 4 · p. 7

4.4. Applications

  • VLM integration enables natural-language goal specification, including directing an electric-drill grip toward a person’s hand.The system also supports continuous non-prehensile rearrangement and pre-grasping reorientation for thin cards and broad cracker boxes.Source: §4.4 · pp. 7–8

5. Conclusion, Limitations, and Future Works

DyWA jointly predicts future states and adapts dynamics from historical trajectories to improve generalizable non-prehensile manipulation. The method reduces reliance on multi-camera setups and pose tracking but remains limited by point-cloud-only visual input.

  • DyWA jointly predicts future states while adapting dynamics from historical trajectories for generalizable non-prehensile manipulation.Source: §5 · p. 8
  • Point-cloud-only input creates ambiguity for symmetric objects and incomplete depth for transparent or specular objects.The paper identifies additional appearance information as a promising direction.Source: §5 · p. 8

Supplementary Material

The supplementary material includes figures showing the real-world experiment setup and the objects used in those experiments.

6. Real-world Setup

The real-world setup uses a Franka Panda robot arm and an Intel RealSense D435 camera to capture point clouds.

  • A Franka Panda robot arm executes actions while an Intel RealSense D435 camera captures point clouds.The setup uses color-based segmentation followed by depth-based back-projection to obtain a single-view object point cloud.Source: §6 · p. 12

7. RL-based Teacher Policy

The teacher policy uses a reward design combining success, reaching, contact, and energy terms, alongside separate modality encoders and an MLP policy network. The architecture analysis reports that DyWA primarily benefits distillation rather than improving the RL teacher itself.

7.1. Reward Design

  • The reward combines task success, goal reaching, contact induction, and energy penalty terms.Source: §7.1 · p. 12
  • Dense reaching and contact rewards use potential-function formulations based on object-goal and gripper-object distances.The object-goal distance uses a bounding-box-based metric, while the gripper-object distance uses the object center of mass and gripper tip.Source: §7.1 · p. 12
  • The energy penalty is scaled using joint torques and joint velocities.Source: §7.1 · p. 12

7.2. Architecture

  • Table 5 reports success rates for the RL-based teacher policy.Source: Table 5 · p. 12
  • The RL teacher uses separate encoders for each modality and an MLP-based policy network.Source: §7.2 · p. 12
  • Figure 9 presents the simulation environment setup.Source: Figure 9 · p. 13
  • World modeling and FiLM leave teacher performance nearly unchanged, suggesting DyWA primarily improves distillation and imitation-loss optimization.The analysis attributes the unchanged teacher performance to the RL policy reaching its current architectural upper bound.Source: §7.2 · p. 13

8. Alternative of Dynamics Factor Conditioning

The section compares cross-attention with FiLM for conditioning the student policy on dynamics factors. FiLM is supported as the more effective and lightweight conditioning mechanism.

8. Alternative of Dynamics Factor Conditioning

  • Cross-attention conditioning significantly degraded student-policy performance compared with FiLM.The comparison concerns alternative dynamics-conditioning methods.Source: §8 · p. 13, Table 6 · p. 13
  • The authors hypothesize that cross-attention is more sensitive to distribution shifts and may require architectural changes or extensive data augmentation.Source: §8 · p. 13
  • FiLM provides a lightweight method for integrating adaptation embeddings into the world-action model.Source: §8 · p. 13

9. Simulation Assets and Setup

The simulation benchmark uses sampled DexGraspNet objects with stabilized initial poses and separate training and test object sets. The section also documents the associated policy and teacher hyper-parameters.

9. Simulation Assets and Setup

  • 323 DexGraspNet objects are used for training and 10 objects for testing in the simulation benchmark.Training and test objects are shown in Figures 10 and 11.Source: §9 · p. 13, Figure 10 · p. 14, Figure 11 · p. 14
  • Stable initial poses are generated by dropping objects under randomized orientations and retaining poses that remain stationary with the center of mass inside the support polygon.At least 128 initial candidates are collected per object before equivalent orientations are pruned.Source: §9 · p. 13
  • The benchmark section includes separate hyper-parameter specifications for the student training algorithm, student encoder and policy, and teacher PPO algorithm.These specifications are listed in Tables 7, 8, and 9.Source: Table 7 · p. 13, Table 8 · p. 13, Table 9 · p. 13

10. Vision Encoder

The vision encoder uses a simplified PointNet++ architecture to process point clouds through hierarchical grouping, local neighbor aggregation, and residual MLP processing.

10. Vision Encoder

  • The student and teacher policies use simplified PointNet++ point-cloud encoders with different numbers of fixed-scale grouping layers.The student uses two grouping layers, while the teacher uses one.Source: §10 · p. 14
  • Each grouping layer selects key points by farthest point sampling and forms local groups using K-nearest neighbors.Grouped points are processed by per-point and global MLP layers to generate features.Source: §10 · p. 14
  • After hierarchical grouping, concatenated point features pass through residual MLP layers to produce enriched output points.Source: §10 · p. 14

11. Hyper-parameters

The section provides implementation references for the policy and teacher policy, including their hyper-parameters and qualitative simulation outcomes.

11. Hyper-parameters

  • Tables 7–10 demonstrate the hyper-parameters of the policy and the teacher policy.Source: §11 · p. 14
  • Figure 4 presents qualitative results in the simulation.Source: Figure 12 · p. 15
Loading 2503.16806v2…