PaperlayerView exact revision on arXiv

Source-linked AI summary

In-Context Learning World Models for Robot Dual-Arm Manipulation of Soft Garment with Partial Observations

Hengyi Sim, Quentin Rouxel, Dennis Hong, Darwin Caldwell, Fei Chen

arXiv:2610.30988v1cs.RO

TL;DR

Soft-object manipulation is challenging because its states are high-dimensional, observations can be partial, and dynamics depend on material and geometry, while real-world data collection is costly. The paper combines generative state reconstruction and in-context action-conditioned dynamics prediction in a Transformer world model trained on simulated interactions. It reports accurate reconstruction and prediction alongside real-world folding across varied garments and materials without test-time updates.

  • Problem

    Soft-object manipulation involves high-dimensional states, partial observations, and material-dependent dynamics, while collecting varied real-world interaction data is expensive and labor-intensive.

    Source: §I · pp. 1–2

  • Method

    A Transformer world model trained on randomized simulated interactions combines generative full-state reconstruction with action-conditioned dynamics prediction using test-time context.

    Source: §I · p. 2, A. State Estimation Model · p. 4, B. Dynamics Prediction with In-Context… · p. 5

  • Results

    The reported results include accurate and stable dynamics prediction, correspondence-preserving state estimates, and real-world folding across garment and material variants without test-time updates.

    Source: §VI · pp. 9–10, §VII · p. 11

  • Takeaways & Limitations

    Real-world results across cloths, shirts, trousers, materials, and two robot embodiments support sim-to-real transfer, while multi-stage folding beyond two-fold training demonstrations indicates longer-sequence generalization.

    Source: §VI · p. 10

    Errors can accumulate in multi-stage folding beyond the two-fold training demonstrations.

    Source: §VI · p. 10

Abstract

from arXiv · show

Robust manipulation of soft objects, such as garments, is a critical skill for service and household robots to adopt, but remains challenging due to its high-dimensional state space, and material-dependent deformations. Humans address this challenge by using mental world models that "imagine" the shape and material properties from partial observations, which enables them to predict how actions affect future states. As such, embedding such world models in robots allows for better generalization in manipulating a wide variety of garments in the real world. However, learned world models typically require large amounts of real world data that is strenuous to collect, and a mechanism to infer latent properties that govern dynamics. We present a Soft-Object World Model that performs generative full-state estimation and action-conditioned dynamics prediction using a transformer backbone trained on large-scale simulated soft object interactions, with measures to mitigate the sim-to-real gap. By learning to infer latent shape and material cues through in-context learning over a variable-length partial point-cloud history, the model generalizes across robot embodiments, material parameters, garment types and shapes, and manipulation coordination modes without weight updates. The world model is then integrated with a Goal-Conditioned action planner for motion synthesis. We validate the approach in simulation and real world for bimanual garment folding, reporting improvements in reconstruction fidelity, prediction accuracy, and task success.

I. INTRODUCTION

Soft-object manipulation is difficult because states are high-dimensional, dynamics depend on material and geometry, and observations are partial. This work develops a Transformer world model trained on simulated interactions and evaluates its reconstruction, prediction, and real-world garment-folding capabilities.

I. INTRODUCTION

  • Soft objects combine high-dimensional state spaces, nonlinear material-dependent dynamics, and partial observations, while collecting varied real-world interaction data is expensive and labor-intensive.Self-occlusion and folds can hide task-critical regions, requiring plausible unseen-geometry inference.Source: §I · pp. 1–2
  • The proposed Transformer world model reconstructs full object states from partial observations and predicts future deformations conditioned on actions and past estimated states.It trains on a large, diverse Isaac Lab corpus with randomized material parameters to address data limitations and improve sim-to-real transfer.Source: §I · p. 2
  • Variable-length in-context sequences let the model infer material and shape cues at inference without fine-tuning, supporting adaptation to geometry and material variation.The model uses estimated states from short demonstrations as context.Source: §I · p. 2
  • The work evaluates reconstruction, multi-step action-conditioned prediction, and bimanual garment manipulation, and reports transfer across garment categories and material variants without test-time parameter updates.The experiments include real-world garment-manipulation tasks.Source: §I · p. 2

II. RELATED WORK

Prior soft-object manipulation methods include trajectory-constrained skills, model-free policies, empirical dynamics models, and graph-based approaches. This work is situated among world models that use point clouds and generative estimation to support prediction under partial observations.

A. Soft Object Manipulation

  • Trajectory-planned methods restrict learning to grasp selection or predefined skills, while model-free RL and imitation learning map perceptual inputs directly to actions.Model-free methods do not explicitly represent soft-object dynamics or latent geometry and material properties; model-based methods need accurate state estimates and extensive data.Source: A. Soft Object Manipulation · p. 2

B. Soft Object World Models

  • Empirical-data and graph-based dynamics models have been applied to soft objects, but may require domain knowledge and carefully crafted inductive biases that restrict scalability and adaptability.Point clouds offer geometric detail, temporal correspondence, and sim-to-real transfer, while generative state estimation addresses ambiguity from partial observations.Source: B. Soft Object World Models · pp. 2–3

C. In-Context Learning

The framework uses estimated state histories and test-time context to infer manipulation-relevant properties, then reconstructs full point-cloud states from partial observations. Its state estimator uses conditional flow matching to generate states from a learned distribution.

C. In-Context Learning

  • In-context learning adapts a model to new instances without weight updates by conditioning on short histories or exemplar interactions.For soft objects, estimated states from partial point-cloud demonstrations provide context for inferring latent material and shape priors.Source: C. In-Context Learning · p. 3

III. PROBLEM FORMULATION

  • The task is to infer a dense 3-D object state from partial observation history and predict multi-step future states conditioned on actions and test-time context.At inference, the dynamics model uses estimated context and state history, a query action trajectory, and a binary grasp mask to score candidate futures.Source: §III · pp. 3–4

IV. METHODOLOGY

  • The world model first estimates full states from observation and context point clouds, then predicts future states conditioned on candidate action plans.The method uses Transformers to handle high-dimensional point-cloud representations and variable-length conditioning.Source: §IV · p. 4

A. State Estimation Model

  • The state estimator uses conditional flow matching to reconstruct partial observation and context point clouds, integrating a learned vector field from a source sample to generate object states.The target samples are expert-demonstration point-cloud states, and the model learns a straight-line vector field for the probability path.Source: A. State Estimation Model · p. 4
  • A Vision Transformer processes point-cloud patches, with adaptive normalization and cross-attention conditioning; interpolation and an MLP decode the representation into 3-D coordinates.Farthest-point sampling supplies patch centers, and Voronoi cells define the patches.Source: A. State Estimation Model · pp. 4–5

B. Dynamics Prediction with In-Context Learning

The dynamics model predicts future soft-object states from action trajectories, estimated state history, and context. Training on randomized materials with a consistent interaction pattern supports test-time material adaptation through context, without weight updates.

B. Dynamics Prediction with In-Context Learning

  • The dynamics Transformer predicts multi-step future states from robot actions, estimated context, and state history, with variable lengths for context, history, and prediction horizon.It predicts per-step deformation added to the last history state and uses Fourier features to encode continuous action coordinates.Source: B. Dynamics Prediction with In-Context… · p. 5
  • Randomized-material demonstrations with a consistent pick-and-place pattern train the model to associate material-dependent deformations with latent material properties.At inference, context modulates rollouts without weight updates; prediction improves with matching-material context and degrades with mismatched context.Source: B. Dynamics Prediction with In-Context… · p. 5
  • Cross-attention and adaptive normalization fuse action and partial point-cloud embeddings, which the policy query uses to produce context-aware future-state predictions.Training masks partial point-cloud conditioning to emulate variable amounts of available history.Source: B. Dynamics Prediction with In-Context… · p. 5

C. Goal-Conditioned Flow-Matching Action Generation

A goal-conditioned generative policy proposes diverse, task-relevant action trajectories, and the world model evaluates them to select a trajectory predicted to move the garment toward its target.

C. Goal-Conditioned Flow-Matching Action Generation

  • The goal-conditioned policy generates diverse, smooth, geometrically meaningful, and task-relevant trajectories for the world model to evaluate against the target.It is trained on the same demonstrations as the world model.Source: C. Goal-Conditioned Flow-Matching… · p. 5
  • Each proposal is a relative 6-DoF Cartesian end-effector displacement trajectory, transformed into absolute poses from the selected grasp points.This makes each trajectory start from the grasp locations and avoids evaluating proposals with implausible initial motions.Source: C. Goal-Conditioned Flow-Matching… · p. 5

1) Grasp-Point Selection and Direction Prior:

The planner biases grasp selection and trajectory generation toward garment regions needing the most movement, then uses predicted outcomes to select a candidate action.

1) Grasp-Point Selection and Direction Prior:

  • Per-point displacement between current and target garment states assigns higher grasp probability to regions more likely to need manipulation.The selected grasp points’ target-to-current displacement provides a coarse motion direction that biases the flow-matching source while preserving proposal diversity.Source: 1) Grasp-Point Selection and Direction… · pp. 5–6

2) Flow-Matching Policy and World-Model Selection:

  • A reused diffusion-transformer backbone learns a conditional flow-matching policy over demonstrated action trajectories, conditioned on current state, target state, and grasp masks.At inference, the learned flow is integrated from independently sampled structured sources to produce K proposals.Source: 2) Flow-Matching Policy and… · p. 6
  • For each proposal, the context-conditioned world model predicts a future garment trajectory, and the planner compares its predicted terminal state with the target using MSE.The comparison is over N garment points.Source: 2) Flow-Matching Policy and… · p. 6
  • Only the action trajectory with the best predicted target match is executed, narrowing evaluation to informed proposals.The passage states this reduces unnecessary computation and enables faster action selection.Source: 2) Flow-Matching Policy and… · p. 6

A. Setup

The evaluation combines Isaac Lab simulation with real-world tests, including two distinct robot platforms. Simulation data randomize garment properties and bimanual folding trajectories, while real-world comparisons use geometric and pointwise metrics.

A. Setup

  • The study evaluates the method in Isaac Lab simulation and demonstrates sim-to-real transfer in real-world experiments.Source: A. Setup · p. 6
  • Training data comprise 100,000 randomized folding demonstrations per soft object, each spanning 110 timesteps.Material properties, trajectories, and bimanual grasp configurations are randomized.Source: A. Setup · p. 6
  • Real-world evaluation uses X-trainer and the 42-DoF Tiangong humanoid, with different grippers and no platform-specific model modifications or retraining.The action representation uses relative 6-DoF end-effector displacements, and the world model operates on garment point clouds.Source: A. Setup · p. 6
  • Evaluation compares observed garment points with simulated particles using MSE for pointwise correspondence and CD and EMD for global shape similarity.Source: A. Setup · p. 6

B. State Reconstruction

The section evaluates state reconstruction from partial observations and action-conditioned dynamics prediction. Flow Matching performs best overall for reconstruction, while the supervised Transformer best captures the simulated dynamics; autoregressive errors rise with prediction horizon.

B. State Reconstruction

  • Estimated shapes from partial observations closely resemble the corresponding real-world soft objects, suggesting recovery of global geometry despite occlusion.Source: B. State Reconstruction · p. 7
  • Flow Matching provides the best overall state-reconstruction performance, with the lowest MSE and competitive CD and EMD, while PCN has the lowest CD and EMD.PCN's shirt-category MSE is approximately 20 times greater than Flow Matching's; pointwise correspondence matters for subsequent dynamics prediction.Source: B. State Reconstruction · p. 7

C. Dynamics Prediction

  • Dynamics training pairs a target demonstration with an independent context demonstration sharing its material parameters, then predicts the target's future states.Simulation sequences are subsampled at uniformly spaced five-frame intervals.Source: C. Dynamics Prediction · pp. 7–8
  • The MSE-trained Transformer achieves the lowest dynamics-prediction loss across both object categories, although CD and EMD tradeoffs vary by category.Its predictions are more physically plausible than the GNN's, and the study attributes its advantage over generative objectives to deterministic, fully observable, Markovian simulated dynamics.Source: C. Dynamics Prediction · p. 8
  • All models' autoregressive errors increase with prediction horizon, and the GNN degrades fastest.On average, the MSE-trained Transformer and GNN perform more strongly, consistent with their deterministic supervised formulations.Source: C. Dynamics Prediction · p. 8

D. Ablation Studies

The ablations test whether world-model proposal evaluation improves goal-directed actions and whether context demonstrations convey material information. Evaluation lowers terminal-state error across garment categories, and matched material context improves prediction relative to mismatched context.

D. Ablation Studies

  • The evaluator compares a policy executing a candidate trajectory without evaluation against one that selects among proposals using predicted terminal-state error.The study runs 100 simulated trials per garment category and also reports one real-world trial.Source: D. Ablation Studies · p. 8
  • The World Model Evaluator lowers terminal-state error across garment categories, with roughly 2x the performance of the unevaluated policy.It rolls out candidate proposals and selects the one predicted to reach the goal most closely.Source: D. Ablation Studies · p. 8, Figure 7 · p. 8

E. In-Context Learning Material Identification

F. Real World Rollouts

Real-world rollouts evaluate the method across garment types, materials, sizes, and manipulation coordination modes, using demonstrations for in-context conditioning without test-time updates. Long-horizon folds and transfer to a humanoid robot further probe generalization.

F. Real World Rollouts

  • Each of 60 rollouts used four teleoperated demonstrations for in-context conditioning, with no gradient-based parameter updates during evaluation.The trials included bimanual coordinated, bimanual sequential, and single-arm manipulation.Source: F. Real World Rollouts · p. 9
  • Success rates across all 60 real-world trials are reported in Table III.Trials covered flat cloth, shirts, and trousers, with two material and size variants per category.Source: F. Real World Rollouts · p. 9, Table III · p. 9

G. Real World Long Horizon Folding

  • Shirt folds required four sequential folds, two beyond the two-fold sequences used for training; trousers were evaluated with two-fold sequences.The paper reports satisfactory performance for the trousers and presents qualitative long-horizon results in Fig. 10.Source: G. Real World Long Horizon Folding · p. 9
  • The method transferred from a 7-DOF bimanual robot with parallel grippers to a 42-DOF humanoid with dexterous hands.The authors attribute cross-embodiment transfer to the embodiment-agnostic action-space design.Source: G. Real World Long Horizon Folding · p. 9

VI. DISCUSSION

The discussion reports accurate state reconstruction, action-conditioned prediction, and real-world garment folding, while identifying the model’s scope and open challenges. The conclusion emphasizes generalization across garments and embodiments, with dynamic interactions and closed-loop replanning left for future work.

VI. DISCUSSION

  • The supervised Transformer achieved the lowest MSE and slower autoregressive error growth than generative and GNN baselines.The discussion distinguishes this fit to deterministic, fully observable, Markovian dynamics from generative models’ possible suitability for multimodal tasks.Source: §VI · p. 9
  • Fig. 10 compares goal-conditioned policy performance with and without the world model and shows multi-stage folds for shirts and trousers.The caption also includes target states for each manipulation.Source: Figure 10 · p. 10
  • Real-world results show high success across cloths, shirts, trousers, materials, and two robot embodiments without test-time updates.The discussion attributes this support for sim-to-real transfer to point-cloud representation, simulation randomization, and relative Cartesian actions.Source: §VI · p. 10

VII. CONCLUSION

  • Successful multi-stage folds beyond the two-fold training demonstrations indicate longer-sequence generalization, although prediction and execution errors can accumulate.The conclusion identifies closed-loop state estimation and replanning as future work to reduce accumulated errors.Source: §VI · p. 10, §VII · p. 11
  • The study focuses primarily on quasi-static folding; future work will examine more dynamic interactions and increasingly ambiguous future states.The conclusion also identifies closed-loop state estimation and replanning as important for longer-horizon manipulation.Source: §VI · p. 10, §VII · p. 11
Loading 2610.30988v1…