Source-linked AI summary
SE3-Nets: Learning Rigid Body Motion using Deep Neural Networks
Arunkumar Byravan, Dieter Fox
TL;DR
SE3-NETS address the challenge of learning action-conditioned rigid-body scene dynamics from raw depth data without explicit segmentation labels. They predict object masks and SE(3) transforms rather than pointwise flows, achieving more consistent simulated predictions and working on real Baxter robot data.
Problem
Vanilla pointwise-motion networks do not explicitly represent objects and their motion, while learning from raw depth observations and actions requires object-level dynamics.
Method
SE3-NETS decompose scenes into k objects using dense masks and SE(3) transforms, then blend the transformed point cloud through a differentiable transform layer.
Results
SE3-NETS outperform standard flow baselines on three simulated datasets, reduce prediction error relative to zero motion, and produce consistent motion on simulated and real robot tasks.
Takeaways & Limitations
Object-centric rigid-transform structure yields sharper, less noisy, and more interpretable motion predictions than naive flow-based modeling.
Abstract
from arXiv · showhide
We introduce SE3-Nets, which are deep neural networks designed to model and learn rigid body motion from raw point cloud data. Based only on sequences of depth images along with action vectors and point wise data associations, SE3-Nets learn to segment effected object parts and predict their motion resulting from the applied force. Rather than learning point wise flow vectors, SE3-Nets predict SE3 transformations for different parts of the scene. Using simulated depth data of a table top scene and a robot manipulator, we show that the structure underlying SE3-Nets enables them to generate a far more consistent prediction of object motion than traditional flow based networks. Additional experiments with a depth camera observing a Baxter robot pushing objects on a table show that SE3-Nets also work well on real data.
I. INTRODUCTION
SE3-NETS learn rigid-body scene dynamics from raw depth observations and actions, addressing the inability of vanilla flow networks to represent objects explicitly. The architecture jointly segments salient objects and predicts their SE(3) motions without explicit segmentation supervision.
- Motivation: Robot control requires predicting how applied forces change environments, including object motion under pushes.The motivation concerns reasoning about object boundaries, mass, friction, and forces when arranging objects on a table.
- Learning setting: The paper learns physical intuition from raw 3D point clouds, continuous action vectors, and point-wise associations between consecutive observations.No higher-level object segmentation information is provided during training.
- Problem: Vanilla networks can predict per-point 3D motion but do not explicitly represent objects and their motion, limiting usefulness for control and higher-level reasoning.This limitation motivates an object-centric representation.
- Approach: SE3-NETS jointly learn salient-object segmentation and action-conditioned motion using k dense masks and k SE(3) transforms combined by a differentiable transform layer.The design separates object motion from object location in the environment.
- Reported scope: SE3-NETS learn from control, raw depth data, and point-wise associations without explicit segmentation labels, while experiments report advantages over standard deep-learning baselines and application to real robot data.The introduction states this result without giving a numerical metric.
II. RELATED WORK
The paper relates SE3-NETS to learned robotics dynamics, physics prediction, 3D motion estimation, attention, disentangled representations, and spatial transformers. Its distinctive combination is action-conditioned dense point-cloud prediction with object masks and SE(3) motion.
- Robotics: Prior robotics work includes physics-based optimal control, low-dimensional learned dynamics, tracking-based rigid-body models, and predictive models of raw depth images.These approaches differ in their representations and assumptions about available tracking or state information.
- Physics prediction: Related physics-prediction studies model stacked-block stability, ball motion, or object dynamics in images, whereas SE3-NETS explicitly predicts dense point clouds.The paper also distinguishes its SE(3)-and-mask motion encoding from closely related RGB-based action-conditioned models.
- 3D motion: Most related 3D-motion work predicts rotation between images, while SE3-NETS operate on 3D data, incorporate continuous actions, and predict rigid motion with object masks.Other rigid-transform models cited for depth registration do not model object motion or action effects.
- Attention and disentanglement: SE3-NETS use a differentiable dense pointwise attender to focus on moving scene parts and disentangle object motion from location.The paper connects this design to attention, disentangled representations, and spatial-transformer networks while restricting transformations to SE(3).
- SE3-NETS: Given a 3D point cloud X and continuous action u, the model represents scene dynamics as k object masks and rigid transforms producing transformed point cloud Y.The number k is a prespecified limit on distinct moving objects or parts, including background.
A. Encoder
The network encodes point-cloud and action inputs, then decodes motion classes into masks and rigid transforms. Its mask representation permits differentiable motion blending, while sharpening and skip connections support object-shaped predictions.
- Encoder: A convolutional encoder processes the point cloud, a fully connected network encodes the control vector, and late fusion concatenates both representations.The fused encoding is passed to the decoder for subsequent predictions.
- Decoder: The decoder first groups points that move together into motion classes and then predicts SE(3) parameters for each group.This decomposes motion prediction into grouping and rigid-transform estimation.
- Motion masks: The mask decoder produces k dense masks at input resolution using deconvolution and skip-add connections for sharper object contours.The stated purpose of skip-add connections is improving shape reconstruction and overall performance.
- Motion masks: Each point receives a per-point probability distribution over k motion classes, allowing smooth interpolation between multiple motions.The relaxation makes the otherwise discrete labeling formulation differentiable.
- Rigid transforms: Each predicted rigid transform consists of a rotation R and translation t, with rotations represented by a three-parameter axis-angle vector.The decoder predicts k transforms, one for each motion class including background.
C. Transform layer
The transform layer blends pointwise applications of predicted SE(3) transforms using object-mask weights, while weight sharpening encourages rigid, object-level assignments. At test time, the network segments the input cloud and transforms points to produce the output cloud.
- Transform layer: The transform layer produces each output point by blending the input point transformed by each of k predicted SE(3) transforms.The mask weights form a convex combination of transformed input points.
- Transform layer: Because mask relaxation occurs in 3D point space, the effective transform for an individual point is generally not itself an SE(3) transform.This blending permits combinations of rigid and non-rigid motion and avoids potential singularities from blending in SE(3) space.
- Enforcing Rigidity: Without explicit regularization, the flexible framework can overfit rigid-motion data and produce blurry predictions.The method addresses this issue by encouraging mask weights toward binary assignments.
- Enforcing Rigidity: Weight sharpening smoothly pushes mask probabilities toward binary decisions rather than imposing a nondifferentiable hard assignment.Gaussian noise and a growing exponent separate the decisions during training, yielding nearly binary distributions by its end.
- Enforcing Rigidity: At test time, the network segments the input cloud into k objects, predicts their motion, and applies the corresponding rigid transform to generate the output cloud.This is the final inference procedure described for the transform layer.
IV. EVALUATION
The evaluation tests SE3-NETS on simulated motion-prediction tasks, robustness to noise and hyperparameters, and real-world data collected with a Baxter robot.
- Evaluation: SE3-NETS are evaluated on multiple simulated motion-prediction tasks, robustness to noise and hyperparameter choices, and real-world Baxter robot data.The evaluation presents simulated results first, followed by robustness tests and real-world experiments.
A. Simulated data collection
The simulated datasets use fixed-camera point clouds of rigid bodies responding to applied forces, with varied object configurations and a Baxter arm-control setting. Each example predicts the point cloud 0.15 seconds into the future under a held-fixed control.
- Simulation setup: All simulated tasks input a 240×320 3D point cloud and an n-dimensional continuous control vector, then predict the resulting cloud 0.15 seconds later.The control is assumed fixed over this prediction interval.
- Single Box: The Single Box dataset contains 9000 scenes in which a randomly forced ball collides with a box at a random table position.Scenes run for one second, and frames where the box falls off the table are discarded.
- Multiple Boxes: The Multiple Boxes dataset varies object size, mass, and count, using scenes with 1–3 objects and retaining examples with one box-ball collision.Examples involving multiple collisions are discarded because the resulting motion is difficult to model.
- Multiple Boxes: The Multiple Boxes controls match the Single Box controls and have dimensionality n = 10.
- Baxter: The Baxter dataset contains approximately 11,000 scenes and 220,000 examples of a randomly controlled right arm moving 1–4 joints.Each scene lasts one second, uses a constant velocity, and resets the arm pose every 20 scenes.
- Baxter: Baxter controls are 14-dimensional commanded joint-velocity vectors with 1–4 nonzero entries.
- Household Objects: The Household Objects dataset uses 11 irregular LineMOD objects, with 1–4 randomly placed objects per scene and approximately 10,000 scenes.A ball collides with a randomly selected object; the dataset includes substantial toppling and fast rotation.
B. Training
Training uses scheduled weight sharpening, compares SE3-NETS with flow and zero-motion baselines, and evaluates flow error under clean and noisy inputs. The reported results emphasize sharper, more consistent predictions and graceful degradation under noise.
- Training: Training initializes the transform layer to identity, then gradually increases the sharpening noise σ and exponent γ from an initially zero penalty.This schedule is used with Torch, ADAM, and batch normalization.
- Baselines: The comparison includes standard and large SE3-NETS, a no-penalty variant, standard and large flow networks, and a zero-motion baseline.The large SE3-NET has approximately six times as many parameters as the original, while the large flow network has approximately eight times as many.
- Metric: The evaluation metric is average per-point scene-flow MSE in centimeters over points with nonzero ground-truth flow.It captures errors in both mask and SE(3) predictions and is also used to train the flow networks.
- Prediction results: Figure 2 compares input clouds, ground truth, and network predictions across three simulated datasets after projecting predicted 3D clouds into 2D.Flow-network clouds are formed by adding predicted flow to the input; black regions denote newly visible areas that no network fills.
- Quantitative results: Table I reports average per-point flow MSE across tasks and networks, with the large SE3-NET achieving the best flow error despite not being directly trained to predict flow.The no-motion baseline indicates motion magnitudes above 20 cm for box datasets and below 1 cm for Baxter.
- Sequential prediction: Multi-step feedback predictions remain consistent across time, whereas the large flow network degrades as noise cascades over successive predictions.The comparison feeds predictions back as inputs for four consecutive times and resets ground truth at each frame.
- Household Objects: On household objects, SE3-NETS correctly segment objects and produce sharp, consistent topping and sliding predictions, while the flow baseline smears objects.The associated results include complex shapes, holes, toppling, and large motions.
- Robustness: With noise added to depth or data associations, SE3-NETS degrade gracefully compared with the large errors of the flow baseline.Table II reports average per-point flow MSE for increasing noise in both input modalities.
C. Results on simulated data
On simulated datasets, SE3-NETS outperform flow baselines while producing sharper, more consistent rigid-motion predictions across objects, complex shapes, and multi-step forecasts.
- SE3-NETS significantly outperform small and large flow networks across the first three simulated datasets.Five-fold cross-validation found the improvements over large flow baselines statistically significant.
- Rigid-object segmentation and SE(3) transforms yield sharper, less noisy predictions than unconstrained flow networks.Flow predictions smear across the canvas, whereas points belonging to an object move together in an interpretable manner.
- The networks segment independently moving balls and boxes without explicit segmentation supervision, while training examples with independent ball motion help separate them.When the ball always contacts the box during training, the network often masks both objects together.
- SE3-NETS model complex-shaped objects, including objects with holes, with sharper predictions and can handle toppling or large motions.Their predictions were significantly sharper than those of the large flow network on the household objects dataset.
- Multi-step SE3-NET predictions remain consistent over time, whereas large-flow predictions degrade as noise cascades through recurrent forecasts.The comparison used five future steps on a Baxter sequence with the control vector fixed.
D. Robustness
Robustness experiments show that SE3-NETS tolerate extra predicted transforms and uncertain point associations, retaining an advantage over flow baselines under noisy supervision.
- Using more transforms than the true object count tests sensitivity to the k hyperparameter.The experiments used k = 8 for Baxter and k = 6 for the other datasets.
- Figure 5 compares direct RGB-rendered box masks with arg-max-colored Baxter masks across k predicted mask channels.Distinct colors indicate near-binary SE3-NET masks, while mixed colors in the No Penalty masks indicate non-binary assignments.
- SE3-NET predictions remain robust when training associations include spurious matches within increasingly large spatial and depth windows.The tested windows were 9 × 9 with ±10 cm depth differences and 15 × 15 with ±20 cm differences.
- Under uncertain associations, SE3-NETS strongly outperform the flow baseline, with errors almost half as large.The authors attribute this robustness to structural constraints that average over association noise.
E. Results on real data
On limited real-world Baxter data, SE3-NETS jointly predict object and arm motion and maintain consistent forecasts for more than one second, despite increasing errors over time.
- The real-data experiment uses Baxter poking a randomly placed tabletop object among a Cheez-It box, mustard bottle, and Pringles can.The robot applies pokes in random directions while keeping the end-effector level and at constant orientation.
- For real data, the small SE3-NET predicts 0.27 seconds into the future from commanded joint angles and velocities.The control vector contains n = 14 values.
- Although supplied the ground-truth robot-arm mask, the network still segments the object and jointly predicts arm and object motion.This modification was introduced because the real-world dataset was limited and training needed to be accelerated.
- On held-out pokes, cascading predictions for more than one second preserves consistent object and arm motion, although errors increase over time.The flow network performs very poorly in the comparison, but it does not receive the ground-truth arm label.
- The real-world study is presented as a proof of concept that SE3-NETS can learn scene dynamics from limited real-world data.The authors plan to collect more data and train larger networks for dynamics such as toppling and falling.
V. DISCUSSION
SE3-NETS model environment changes as rigid transforms applied to 3D points, selectively segmenting moving objects and predicting their SE(3) motions. Across simulated and real robot tasks, they generalize across scenes and produce motion predictions consistent with observed rigid motion, while broader generalization remains a future goal.
- SE3-NETS predict environment changes using a series of rigid transforms applied to 3D points, focusing on moving scene parts.The model segments scenes into objects and predicts SE(3) motion for each distinct object.
- Across four simulated and one real robot task, SE3-NETS perform strongly with multiple rigid bodies in motion.
- SE3-NETS generalize across different scenes and produce motion predictions consistent with observed rigid motion compared with traditional flow networks.
- Larger real-data collections are expected to enable generalization across many types of objects and scenes.
- On real Baxter-robot data, predictions remain sharper than flow baselines, although learning data associations remains a key area for improvement.The current setup provides data associations during training.