PaperlayerView exact revision on arXiv

Source-linked AI summary

DynaSlots: Slot-Centric 3D Dynamics Pretraining for Robot Manipulation

Meng Li, Lijiang Chen, Qi Zhao, Jun Jiang, Longhao Zou, Shuchang Lyu, Guangliang Cheng

arXiv:2610.32330v1cs.RO

TL;DR

Existing dynamics-aware pretraining compresses scene transitions into a global latent action, despite manipulation’s localized changes. DynaSlots learns temporally consistent slots and per-slot dynamics without action, segmentation, or reconstruction labels. It consistently outperforms scene-level baselines across simulation and real-world manipulation benchmarks.

  • Problem

    Dynamics-aware pretraining models transitions with a global scene feature and latent action, although manipulation changes can be localized and involve distinct regional transitions and interactions.

    Source: §I · p. 1

  • Method

    DynaSlots decomposes point-cloud frames into temporally consistent slots and learns per-slot latent actions, forward dynamics, and cross-slot relations without action, segmentation, or reconstruction labels.

    Source: §I · pp. 1–2

  • Results

    DynaSlots consistently outperforms scene-level baselines across simulation and real-world manipulation benchmarks.

    Source: §V · p. 7

  • Takeaways & Limitations

    The paper’s results support slot-level dynamics modeling as a useful 3D pretraining approach for manipulation.

    Source: §V · p. 7

Abstract

from arXiv · show

Robot manipulation involves localized changes: picking up a cup moves the cup while leaving the surrounding scene largely unchanged. Yet existing 3D visual pretraining methods, including recent dynamics-aware variants, model state transitions at the scene level, compressing the motion of everything in a scene into a single global latent action. This conflates semantically distinct transitions in scenes with multiple moving parts, obscures dependencies between changing regions, and leaves the learned representation vulnerable to task-irrelevant scene content. We introduce DynaSlots, an action-free, label-free pretraining framework that decomposes each point cloud frame into a small set of temporally consistent slots and models dynamics at this per-slot level rather than the scene level. A temporal slot propagation module promotes consistent slot assignments across frames; an inverse dynamics model then extracts one latent action per slot from feature differences, and a diffusion-based forward dynamics model, paired with a self-supervised interaction head, predicts each slot's future state while learning pairwise proximity-and-motion relations. Experiments on Meta-World, Adroit, and a RealMan RM65-B robot demonstrate that DynaSlots outperforms prior scene-level baselines. It achieves superior success rates in simulation and exhibits strong robustness to scene clutter and unseen objects in real-world manipulation. Code is available at \href{https://zhucezhanghao12345-source.github.io/DynaSlots-Anonymous/}{https://zhucezhanghao12345-source.github.io/DynaSlots-Anonymous/}.

I. Introduction

Existing dynamics-aware pretraining models transitions at the scene level, although manipulation often changes only localized regions. DynaSlots instead learns slot-level dynamics without action, segmentation, or reconstruction labels.

I. Introduction

  • Scene-level dynamics modeling compresses the whole point cloud into one feature and latent action, with potential semantic aliasing, implicit interaction dependencies, and distractor sensitivity.The passage illustrates the mismatch with a moving milk carton and an otherwise stationary table.Source: §I · p. 1
  • DynaSlots decomposes frames into temporally consistent slots, extracts one latent action per slot, and models cross-slot dynamics and relations.Its temporal propagation promotes correspondence; a diffusion forward dynamics model and interaction head capture slot futures and pairwise relations.Source: §I · pp. 1–2
  • The framework requires neither action labels, segmentation ground truth, nor reconstruction supervision.The paper presents it as a label-free, slot-based dynamics-aware 3D pretraining framework.Source: §I · pp. 1–2
  • Extensive experiments demonstrate DynaSlots’ effectiveness for robotic manipulation.The supplied introduction claim does not specify benchmark values or comparisons.Source: §I · p. 2

II. Related Work

Prior 3D pretraining emphasizes spatial structure or cross-modal alignment, while dynamics-aware and slot-based work supplies related components. DynaSlots integrates per-slot latent-action learning with temporal correspondence and relational dynamics on 3D point clouds.

A. 3D Visual Pretraining

  • Point-MAE, Point-BERT, and related methods learn point-cloud structure or cross-modal alignment, rather than action-dependent temporal transitions.The passage contrasts masked modeling and cross-modal objectives with DynaSlots’ per-slot dynamics signal.Source: A. 3D Visual Pretraining · p. 2

B. Dynamics-Aware Pretraining and Latent Actions

  • AFRO is the closest baseline, learning inverse and forward dynamics in a global 3D feature space; DynaSlots assigns latent actions to corresponding slots and learns cross-slot relations.Other dynamics-aware approaches use distinct signals, including visual transitions, vision-action histories, end-effector actions, and diffusion-based point-cloud prediction.Source: B. Dynamics-Aware Pretraining and… · p. 2

C. Slot-Based Representation Learning

  • Slot-based video and control methods model temporal dynamics or interactions, but DynaSlots integrates these ideas with per-slot latent actions on unordered 3D point clouds.Its learned slots are soft scene decompositions and need not correspond one-to-one with physical objects; pretraining needs no action labels, segmentation labels, or reconstruction targets.Source: C. Slot-Based Representation Learning · p. 2

III. Methodology

DynaSlots learns dynamics-aware representations from unlabeled coordinate-only point-cloud sequences by binding points into temporally consistent soft slots. A Point Transformer tokenizes each frame, and the resulting slot representations support per-slot dynamics pretraining and downstream policy conditioning.

III. Methodology

  • Coordinate-only point-cloud sequences are tokenized and bound into K=6 temporally consistent soft slots before per-slot dynamics modeling.The inverse dynamics model extracts latent actions from slot differences, the diffusion forward dynamics model predicts future slot states, and an interaction head estimates pairwise proximity-and-motion relations.Source: §III · pp. 2–3, Figure 2 · p. 3

A. Point Cloud Tokenization

  • A shared Point Transformer processes the full, ungrouped point set, lifting points to 256-dimensional embeddings before projecting each point to a 64-dimensional token.The passage specifies no farthest-point sampling or neighbor grouping; the online and EMA target encoders share the architecture and process frames with shared weights.Source: A. Point Cloud Tokenization · p. 3

B. Temporal Slot Propagation

Temporal slot propagation uses previous-frame slots to query current-frame points, encouraging consistent assignments across time. This reduces the risk that slot feature differences reflect reassignment rather than scene dynamics.

B. Temporal Slot Propagation

  • Independent per-frame binding could assign a slot to unrelated content across frames, mixing assignment changes into the feature differences used to learn dynamics.The temporal update is the binder used in the pretraining path.Source: B. Temporal Slot Propagation · p. 3
  • The binder initializes the first frame from K learned queries, then uses previous-frame slots to bind each subsequent frame’s points.Points first compete across slots; assignments are renormalized over points for weighted-mean aggregation, followed by residual aggregation and refinement.Source: B. Temporal Slot Propagation · pp. 3–4
  • Binding is iterated three times per frame, with two propagated steps during pretraining and across the two-frame observation window at inference.At inference, the binder restarts from the learned queries at each policy call rather than persisting across a rollout.Source: B. Temporal Slot Propagation · p. 4

C. Slot-Level Latent Actions

DynaSlots extracts one latent action from each temporally corresponding slot’s feature change, then predicts future slot features with a cross-slot diffusion model. The forward model uses slot-specific noise targets and cross-slot context.

C. Slot-Level Latent Actions

  • One latent action per slot is extracted from temporally corresponding slot features using an inverse dynamics model.Feature differences focus the action on what changed; stable slot content yields small differences, and the same model supports reverse consistency.Source: C. Slot-Level Latent Actions · p. 4

D. Cross-Slot Diffusion Forward Dynamics

  • The diffusion path predicts future slot features from current slots and their per-slot actions, while self-attention models cross-slot dependencies.A three-layer Transformer applies self-attention to slot summaries; conditioning states and actions come from the online branch, while noised targets come from corresponding EMA slots.Source: D. Cross-Slot Diffusion Forward Dynamics · p. 4
  • Each slot has an independent noise level and denoising path, but its condition includes cross-slot context.The generator is a diffusion transformer with adaptive layer normalization, and the projection and Transformer are shared across slots and prediction directions.Source: D. Cross-Slot Diffusion Forward Dynamics · p. 4

E. Self-Supervised Interaction Head

A lightweight head predicts pairwise proximity-and-motion relations between slot centroids. It is trained against symmetric kinematic pseudo-labels derived from the slots, without annotation.

E. Self-Supervised Interaction Head

  • A lightweight head predicts each slot pair’s probability of a proximity-and-motion relation between their soft centroids.The prediction is trained with binary cross-entropy against a symmetric kinematic pseudo-label computed from the slots.Source: E. Self-Supervised Interaction Head · p. 4
  • The pseudo-label is positive when the centroids are nearby at either frame and at least one moves beyond the displacement threshold.The proximity threshold is 0.2 times the scene bounding-box diagonal, and the motion threshold is 0.02.Source: E. Self-Supervised Interaction Head · p. 4
  • The symmetric label provides a slot-level proxy for contact and co-motion, but does not establish physical contact or identify the motion source.The interaction loss backpropagates into shared slot and latent-action representations, and the head is trained jointly with forward dynamics.Source: E. Self-Supervised Interaction Head · p. 4

F. Training Objective

The training objective combines forward and reverse denoising losses with slot-level regularization, interaction supervision, and an entropy term. The setup uses task-specific demonstrations and trains the downstream policy on pretrained slot representations.

F. Training Objective

  • Four loss groups combine forward and reverse noise-prediction losses with slot-level VIC regularization, interaction loss, and a slot-entropy term.The VICReg-style terms prevent collapse, decorrelate slots, and align online slot contents with corresponding EMA targets; entropy sharpens per-point competition weights.Source: F. Training Objective · pp. 4–5
  • Low-activation slots receive less emphasis but retain a training signal, including for the variance term.Default weights are 1.0 for forward, reverse, invariance, and variance losses; 0.04 for covariance; and 0.1 for interaction and entropy.Source: F. Training Objective · p. 4
  • Pretraining runs for 300 epochs at frame stride k=4 on each task’s own demonstrations, and the downstream diffusion policy uses pretrained slot representations and robot state.The encoder and binder are pretrained separately per task rather than on a shared corpus.Source: F. Training Objective · p. 5

IV. Experiments

DynaSlots achieves higher reported success rates than the compared baselines on MetaWorld and Adroit. Ablations show that temporal slot propagation contributes most, while interaction supervision, reverse consistency, and slot count also affect performance.

IV. Experiments

  • Results average four independently trained policy seeds, with MetaWorld averaged across 14 tasks and Adroit across Door and Pen.For each seed, the selected score is from the checkpoint with the highest evaluation success rate.Source: §IV · p. 5

A. Evaluation on Simulation

  • 80.8% on MetaWorld and 88.0% on Adroit are the highest mean success rates, exceeding AFRO by 4.8 and 5.0 percentage points, respectively.DynaSlots also exceeds DP3 by 11.1 pp on MetaWorld and 13.0 pp on Adroit.Source: A. Evaluation on Simulation · p. 5

B. Ablation Study

  • Removing temporal slot propagation costs 12.3 pp on MetaWorld and 12.0 pp on Adroit, the largest ablation drops.Without it, the binder uses independent per-frame slot attention; the passage attributes the dynamics signal to cross-frame correspondence between slots.Source: B. Ablation Study · p. 5
  • On the same MetaWorld and Adroit benchmarks, removing DynaSlots’ interaction head costs 2.6 and 2.5 percentage points, respectively, versus the full model’s 80.8% and 88.0% means.Source: B. Ablation Study · p. 5
  • The slot-assignment figure shows high-response points in color and points assigned to other slots in light gray for a representative K=6 frame.It visualizes slot assignment rather than an ablation result.Source: Figure 3 · p. 5
  • K=6 performs best: K=4 drops 6.3/5.5 pp and K=8 drops 3.3/3.5 pp on MetaWorld/Adroit.The reported interpretation is that too few slots underfit scene separation, whereas more slots fragment the scene.Source: B. Ablation Study · p. 5

C. Slot Assignment Analysis

The slot assignment analysis finds that unsupervised competition yields coherent spatial regions while using only some of the binder’s available slots. Real-world experiments use task-specific demonstrations and varied initial layouts.

C. Slot Assignment Analysis

  • Active slots claim coherent spatial regions without segmentation labels, while other slots retain small residual assignment mass.The assignment maps are soft, and K=6 is capacity rather than a per-frame target.Source: C. Slot Assignment Analysis · p. 5

D. Real-World Experiments

  • Real-world evaluation uses 100 expert trajectories per task and tests each task across five initial layouts with ten trials per layout.The four tasks include pick-and-place, destination selection, and placement into a container.Source: D. Real-World Experiments · p. 5

1) Real-World Task Performance:

DynaSlots leads the reported real-world task suite on both target-task and RH20T pretraining, and retains stronger performance under scene clutter. The reported clutter evaluation focuses on transferring a ball from a plate into a bowl.

1) Real-World Task Performance:

  • 79.0% suite average on target-task data and 73.0% after RH20T pretraining, with DynaSlots best on all four real-world tasks in both settings.Target-task gains over AFRO are 10.0, 6.0, 10.0, and 8.0 pp across tasks (a)–(d); the RH20T average is 10.0 pp above AFRO.Source: 1) Real-World Task Performance · p. 6, Table III · p. 7

2) Cluttered-Scene Evaluation:

3) Unseen-Object Evaluation:

On task (c), DynaSlots retains the highest success rate and shows the smallest decline when scene objects are unseen. The conclusion reports consistent gains over scene-level baselines across simulation and real-world manipulation benchmarks.

3) Unseen-Object Evaluation:

  • 72.0% success with unseen objects, 18.0 pp above AFRO; DynaSlots drops 6.0 pp versus AFRO’s 14.0 pp and DynaMo-3D’s 12.0 pp.The comparison uses matched original-scene scores; each condition has five layouts and ten trials per layout.Source: Table V · p. 7, 3) Unseen-Object Evaluation · p. 7
  • Across the three real-world evaluations, DynaSlots is the best-performing method in every reported condition.Its success rate remains highest after replacing scene content with unseen objects, with the smallest reduction among methods.Source: 3) Unseen-Object Evaluation · p. 7

V. Conclusion

  • Across simulation and real-world manipulation benchmarks, DynaSlots consistently outperforms scene-level baselines.The conclusion identifies evaluating more embodiments and extending compositional-generalization studies beyond the current placement setting as future work.Source: §V · p. 7

Related papers

Loading 2610.32330v1…