PaperlayerView exact revision on arXiv

Source-linked AI summary

MeshPriorDiT: Hierarchical Modeling for Action-Conditioned Cloth Dynamics

Zihang Wang, Jianming Hu, Shang Su, Hao Huang, Mengkai Shi, Jun Gao, Shuo Feng

arXiv:2608.26766v1cs.RO

TL;DR

Action-conditioned cloth dynamics models must preserve local physical plausibility while coordinating motion across distant regions. MeshPriorDiT combines a topology-aware mesh prior with a generative residual, reducing aggregate Global MSE by 43.42% versus GNN-Only and 75.03% versus DiT-DDPM across three 15-step tasks.

  • Problem

    Cloth dynamics prediction must maintain locally plausible deformations and coordinated motion across distant regions during multi-step rollouts.

  • Method

    MeshPriorDiT combines an action-conditioned topology-aware mesh prior for local motion with a Residual DiT that generates coordinated corrections under grasp constraints.

  • Results

    43.42% and 75.03%: across three 15-step tasks, MeshPriorDiT reduces aggregate Global MSE versus GNN-Only and DiT-DDPM, respectively, while maintaining comparable Edge-strain MSE to GNN-Only.

  • Takeaways & Limitations

    MeshPriorDiT maintains effective residual correction over long autoregressive rollouts while balancing trajectory accuracy and local structural consistency.

  • Takeaways & Limitations

    The setup assumes a fixed grasp mask and uses K = 3 and H = 5 prediction settings.

Abstract

from arXiv · show

Action-conditioned cloth dynamics prediction requires both locally plausible deformation and long-range coordination. Existing approaches largely follow two paradigms. Mesh-based GNNs capture local physical responses through material connectivity. However, their finite message-passing range limits coordination between topologically distant regions, while autoregressive rollouts tend to accumulate prediction errors. Transformer-based dynamics models capture long-range interactions through global attention, but often operate without explicit material connectivity and must learn local topological responses directly from data. We propose MeshPriorDiT, a hierarchical dynamics model that decomposes future cloth motion into a structured mesh prior and a generative residual. An action-conditioned mesh GNN first predicts multi-step vertex displacements, yielding a reference trajectory that respects material topology and grasp constraints. Conditioned on historical states, planned actions, and the mesh prior, a Residual DiT then uses conditional flow matching to jointly generate the residual motion not captured by the prior. The generated residual is further rescaled and decoded using material adjacency to coordinate corrections across neighboring vertices. We evaluate MeshPriorDiT on 15-step autoregressive rollouts across three cloth manipulation tasks. Averaged over the three tasks, MeshPriorDiT reduces average Global MSE by 43.42% relative to the GNN-Only baseline and by 75.03% relative to the DiT-DDPM baseline, while maintaining a favorable Edge-strain MSE comparable to that of GNN-Only.

Introduction

MeshPriorDiT combines a topology-aware mesh prior with a globally coordinated generative residual for action-conditioned cloth dynamics. Across three manipulation tasks, it improves long-horizon accuracy while preserving favorable structural quality.

  • Introduction: Existing GNNs encode material topology but have limited distant communication, while global-attention models capture cross-region motion without guaranteeing consistent neighboring-vertex responses.The paper motivates combining explicit topology with global interaction modeling to preserve both local plausibility and long-range coordination.
  • Introduction: MeshPriorDiT decomposes future cloth trajectories into an action-conditioned mesh prior and a generative residual.The mesh branch predicts multi-step vertex displacements under grasp constraints, while the residual branch corrects the prior.
  • Introduction: The mesh prior captures local physical propagation through fixed material connections, whereas the Residual DiT models long-range coordination, complex deformation, and rollout errors.Magnitude scaling and topology-aware decoding coordinate residual corrections across neighboring vertices.
  • Introduction: 43.42% lower average Global MSE than GNN-Only and 75.03% lower than DiT-DDPM over 15-step rollouts, while maintaining comparable Edge-strain MSE to GNN-Only.The evaluation covers three cloth manipulation tasks and multi-step autoregressive rollouts.

Related Work

Prior cloth-dynamics work spans local graph simulators, globally attentive generative models, and hybrids that combine geometry with sequence modeling. MeshPriorDiT differs by separating topology-respecting action-conditioned propagation from residual correction under explicit grasp constraints.

  • Graph-based simulators: Graph-based simulators model relational dynamics through message passing, with MeshGraphNets predicting mesh-vertex state increments over material edges.Multiscale extensions shorten propagation paths using coarse graphs, hierarchical representations, or pooling.
  • Cloth-specific models: Cloth-oriented graph and physics-guided models incorporate observations, material variation, contact, or self-supervised deformation objectives.Examples include Cloth-Splatting, AdaptiGraph, and physics-guided garment simulators.
  • Generative models: Generative cloth models capture distant interactions through Transformer attention or diffusion, but must infer material neighborhoods and control boundaries without explicit structural inputs.Flow Matching and rectified flow learn continuous velocity fields along probability paths.
  • Hybrid and residual approaches: MeshPriorDiT separates local physical propagation from residual detail, unlike prior hybrid models that jointly predict the complete state.Its graph rollout supplies the structured component, while residual modeling handles corrections.
  • Hybrid and residual approaches: Existing residual methods improve rollouts or model deviations from coarse priors, but do not correct action-conditioned cloth-mesh rollouts under explicit grasp constraints.MeshPriorDiT addresses this setting using a graph rollout as a structural prior.

Method

MeshPriorDiT predicts action-conditioned cloth motion by combining a topology-constrained mesh prior with a generative residual model. The residual is topology-decoded and composed with the prior while anchoring grasped vertices during autoregressive rollout.

  • Inputs and representation: The model uses the latest K = 3 mesh states, H = 5 planned end-effector displacements, and a fixed grasp mask to predict future vertex motion.The cloth is represented by a fixed template mesh whose edges encode material connections and whose vertices have 3D positions.
  • Architecture: MeshPriorDiT decomposes future cloth motion into an autoregressive mesh prior over material topology and a Residual DiT generated through conditional flow matching.The model conditions Residual DiT on the mesh prior and combines generated residuals with the prior trajectory.
  • Motion prediction: The Mesh Prior autoregressively predicts H-step displacements, while Residual DiT generates displacement residuals conditioned on historical states, planned actions, and the prior trajectory.Future positions are recovered by cumulatively applying predicted displacements from the current state.
  • Topology-aware decoding: A row-normalized Jacobi update coordinates position residuals over material neighborhoods, scales them by γ, and composes them with the prior while anchoring grasped vertices.Predicted positions are fed back as history during long-horizon autoregressive rollout.

Action-Conditioned Mesh Dynamics Prior

The mesh prior predicts action-conditioned cloth motion over fixed material topology using deformation-aware node and edge features. Its constrained rollout preserves free-vertex graph predictions, enforces grasp motion exactly, and provides a topology-aware motion prior for the Residual DiT.

  • Topology-aware mesh prior: The mesh prior propagates action-conditioned motion over fixed material topology using directed edges for each undirected material connection.Each connection is represented in both directions from the two most recent states.
  • Topology-aware mesh prior: Edge features combine current three-dimensional deformation with fixed two-dimensional material relations.They include relative positions, distances, and template-coordinate differences between connected vertices.
  • MeshGraphNets dynamics: Encoded node and edge features pass through residual MeshGraphNets blocks whose decoder predicts normalized discrete accelerations for integration.Separate MLP encoders produce latent node and edge representations before L residual message-passing blocks and node decoding.
  • Action constraints: The planned action overwrites candidate predictions at grasped vertices, preserving graph predictions at free vertices while enforcing prescribed grasp motion exactly.This constrained update is applied recursively to generate the prior rollout.
  • Training and rollout: The mesh prior is trained independently with single-step acceleration supervision and supplies a topology-aware structured motion prior rather than the final prediction.The full model uses three historical states, while the second-order mesh update uses the two most recent states.

Residual DiT

Residual DiT generates normalized residual cloth motion conditioned on history, actions, grasp constraints, and a mesh rollout prior. A fixed adjacency-based decoder then coordinates neighboring corrections while preserving commanded grasp motion.

  • Residual target and conditioning: Residual DiT uses the mesh rollout as a reference to jointly model residuals between ground-truth future increments and the H-step mesh prior.The prior defines the residual target and supplies step-wise increments, cumulative displacements, and grasp indicators as conditioning tokens.
  • Residual generation: Conditional Flow Matching trains Residual DiT to predict a velocity field for normalized residual generation from constrained Gaussian noise.The model is conditioned on historical states, planned actions, the mesh prior, and the grasp mask; inference projects states and velocities onto free vertices.
  • Spatial coordination: A fixed, parameter-free decoder uses material adjacency to smooth position residuals and enforce spatial consistency across neighboring vertices.This decoder addresses abrupt per-vertex corrections that are not explicitly penalized by the per-vertex Flow Matching objective.
  • Spatial coordination: The decoder’s γ controls residual magnitude and λ controls spatial coordination, while grasped vertices follow the commanded trajectory exactly.Setting γ = 0 recovers the Mesh Prior, whereas γ = 1 and λ = 0 use the full unsmoothed residual.
  • Qualitative rollouts: Figure 2 shows six-snapshot autoregressive rollouts for corner-to-side folding, diagonal folding, and vertical lifting, comparing predicted and ground-truth boundaries.Grasped vertices and planned actions are separately indicated in the visualizations.

Long-Horizon Autoregressive Rollout

MeshPriorDiT performs long-horizon rollouts by recursively applying an H-step predictor across W non-overlapping windows. After the first window, it relies solely on its own predictions while recomputing the mesh prior, residual, and grasp constraints.

  • Long-Horizon Autoregressive Rollout: For rollout length T = WH, MeshPriorDiT recursively applies its H-step predictor over W non-overlapping windows.This defines the long-horizon autoregressive rollout structure.
  • Long-Horizon Autoregressive Rollout: After each window, the K most recent predicted states provide history for the next action block.Only the first window uses observed states; later windows use model predictions without future ground truth or teacher forcing.
  • Long-Horizon Autoregressive Rollout: The mesh prior, residual, and grasp constraints are recomputed in every window.This recomputation accompanies the prediction-only autoregressive rollout after the initial window.

Experiments · Experimental Setup

The experiments evaluate MeshPriorDiT against structured and generative baselines through four research questions spanning accuracy, long-horizon rollouts, mesh-prior effects, and residual-decoding trade-offs. The setup uses disjoint SoftGym data splits, validation-selected operating points, and physical-space Global and Edge-strain MSE metrics.

  • Experiments: The experiments address whether MeshPriorDiT improves over GNN-based and DDPM-based baselines, retains advantages in long-horizon rollouts, and benefits from mesh priors and residual decoding.A fourth question examines how residual gain and topology-aware decoding affect the trade-off between trajectory metrics.
  • Experimental Setup: The dataset contains 500,000 SoftGym cloth-dynamics windows from disjoint train, validation, and test trajectory splits.The splits yield 400,000/50,000/50,000 windows with three observed and five target frames, while three manipulation tasks are approximately balanced within each split.
  • Experimental Setup: MeshPriorDiT is compared with GNN-Only and reproduced UniClothDiff, denoted DiT-DDPM, using the same mesh prior and rollout protocol for GNN-Only.GNN-Only removes residual correction with γ = 0; validation selects Balanced (γ = 0.5, λ = 0.8) and MinMSE (γ = 1.0, λ = 0.8) operating points.
  • Experimental Setup: The Mesh Prior is independently trained with 15 message-passing blocks and hidden dimension 128, while the frozen-prior Residual DiT uses flow matching during training.Training uses 16 NVIDIA A100 GPUs, global batch size 2,048, AdamW, and BF16 mixed precision; inference uses a 50-step Heun solver.
  • Experimental Setup: Evaluation restores predictions to physical coordinate space and reports Global MSE and Edge-strain MSE, with lower values indicating better performance.Relative improvements are computed from original unrounded evaluation records.
  • Experimental Setup: Qualitative 15-step autoregressive rollouts are included alongside the quantitative evaluation.The rollout visualizations are provided in Figure 2.

Q1: Comparison with Baselines · Q2: Long-Horizon Autoregressive Rollout

MeshPriorDiT outperforms GNN-Only and DiT-DDPM across evaluated rollout horizons, with the mesh prior and Residual DiT providing complementary structured prediction and correction. In 15-step autoregressive rollouts, both configurations maintain lower errors than the baselines, although residual magnitude creates a trade-off between coordinate accuracy and local mesh quality.

  • Q1: Comparison with Baselines: Across 5-, 15-, 20-, and 35-step rollouts, MeshPriorDiT-MinMSE reduces Global MSE relative to GNN-Only by 49.59%, 43.94%, 43.49%, and 60.50%, respectively.Table 1 reports Global MSE across rollout horizons and Edge-strain MSE on the test set; coordinate errors use 10^-5 units and Edge-strain MSE uses 10^-3 units.
  • Q1: Comparison with Baselines: At 35 steps, Balanced achieves the lowest Edge-strain MSE, reducing it by approximately 11.1% relative to GNN-Only, while MinMSE achieves the lowest coordinate error.MinMSE’s Edge-strain MSE is slightly higher than Balanced’s, indicating a trade-off between trajectory accuracy and local mesh quality.
  • Answer to Q1.: MeshPriorDiT outperforms both GNN-Only and DiT-DDPM across all evaluated rollout horizons.The results support complementarity between a topology-grounded mesh prior and Residual DiT corrections.
  • Q2: Long-Horizon Autoregressive Rollout: Errors from GNN-Only and DiT-DDPM progressively accumulate during autoregressive rollout, whereas Balanced and MinMSE maintain substantially lower errors.Each predicted five-step window is fed back as historical input to the next window.
  • Q2: Long-Horizon Autoregressive Rollout: 43.42% and 43.94% lower average Global MSE than GNN-Only, and 75.03% and 75.26% lower than DiT-DDPM, are achieved by Balanced and MinMSE, respectively, at 15 steps.Figure 3 reports per-step Global MSE averaged over 60 test trajectories overall and 20 per action.
  • Q2: Long-Horizon Autoregressive Rollout: Relative to GNN-Only, Balanced reduces H15 Global MSE by 51.89% and 56.98% on Corner-to-Side Fold and Vertical Lift, while MinMSE reduces it by 55.85% and 68.86%.On Diagonal Fold, where GNN-Only is already strong, residual correction offers limited benefit and can slightly degrade aggregate error.
  • Answer to Q2.: MeshPriorDiT’s advantages over both baselines persist throughout autoregressive rollout, showing that Residual DiT corrects accumulated coordinate drift after repeated feedback.This persistence is reported for both MeshPriorDiT configurations.

Q3: Roles of the Mesh Prior and Residual DiT

The complete MeshPriorDiT model outperforms both DiT-DDPM and GNN-Only, while its mesh prior and Residual DiT divide responsibilities between physically structured motion prediction and targeted correction. The mesh prior captures primary topology-constrained motion, and the residual branch reduces localized errors in folding and highly deformable regions.

  • Roles of the Mesh Prior and Residual DiT: MeshPriorDiT outperforms both DiT-DDPM and GNN-Only, with the complete model improving further over GNN-Only through residual compensation.GNN-Only already benefits from mesh topology, local connectivity, and grasp constraints as physical inductive biases.
  • Roles of the Mesh Prior and Residual DiT: Figure 4 shows that Residual DiT substantially shrinks and localizes high-error regions around complex deformations, correcting deviations in the mesh prior.Before correction, larger GNN-Only errors concentrate near the grasped vertex and along the downward folding region; the mesh prior preserves the overall boundary and primary folding structure.
  • Roles of the Mesh Prior and Residual DiT: The mesh prior captures primary motion and local physical propagation under topological constraints, while Residual DiT provides targeted corrections in folding and highly deformable regions.Together, the branches form a complementary hierarchical prediction mechanism.

Q4: Balancing Trajectory Accuracy and Mesh Quality

MeshPriorDiT balances trajectory accuracy and mesh quality by tuning residual gain γ and topology-aware smoothing strength λ. Moderate smoothing improves both Global MSE and Edge-strain MSE, while excessive smoothing degrades performance.

  • Q4: Balancing Trajectory Accuracy and Mesh Quality: At γ = 1, setting λ = 0.8 reduces Global MSE from 1.525 to 1.119 and Edge-strain MSE from 18.655 to 4.460.Figure 5 reports these improvements under the selected gain and smoothing configuration; stronger smoothing causes slight degradation.
  • Q4: Balancing Trajectory Accuracy and Mesh Quality: Topology-aware residual smoothing reduces Neighbor Variation, Edge-strain MSE, and Coordinate MSE by 80.4%, 78.1%, and 12.3%, respectively.The smoothing step applies residual correction over fixed template adjacency to improve local consistency without altering mesh topology.
  • Q4: Balancing Trajectory Accuracy and Mesh Quality: Increasing γ lowers Global MSE but amplifies local edge strain, whereas λ suppresses high-frequency residuals and moderately improves both metrics.The Balanced configuration prioritizes the trade-off between trajectory accuracy and mesh quality, while MinMSE prioritizes the lowest coordinate error.

Conclusion

MeshPriorDiT decomposes action-conditioned cloth dynamics into a topology-aware mesh prior and a generative residual. Across three tasks at 15-step rollouts, it substantially lowers aggregate Global MSE against both GNN-Only and DiT-DDPM.

  • Conclusion: MeshPriorDiT decomposes cloth dynamics into a topology-aware mesh prior and a generative residual.The mesh GNN captures structured local motion, while Residual DiT corrects remaining trajectory errors, particularly in highly deformable regions.
  • Conclusion: The mesh GNN captures structured local motion, while Residual DiT corrects remaining trajectory errors in highly deformable regions.
Loading 2608.26766v1…