PaperlayerView exact revision on arXiv

Source-linked AI summary

MoSE3: Learning World-Space SE(3) at Every Pixel

Jiahuan Cheng, Zhiyi Li, Tian Xia, Ruojin Cai, Yilun Du, Qianqian Wang

arXiv:2610.03716v1cs.CV

TL;DR

Point tracks capture pixel translation but not the rotation or rigid grouping needed for a more structured account of scene motion. MoSE3 predicts per-pixel SE(3) from monocular RGB video through tracks and rigidity embeddings, with Art-Kubric supplying articulated-scene labels. It achieves state-of-the-art SE(3) estimation and 3D point tracking, with strong real-world generalization.

  • Problem

    Existing approaches leave feed-forward per-pixel world-space SE(3) prediction from monocular RGB video largely out of reach amid geometry, regression, and annotation challenges.

    Source: §1 · p. 2

  • Method

    MoSE3 predicts dense 3D point tracks and rigidity embeddings, recovers SE(3) through differentiable fitting, and introduces Art-Kubric for articulated-scene supervision.

    Source: §1 · p. 2

  • Results

    MoSE3 achieves state-of-the-art SE(3) estimation at pixel, part, and object levels on rigid and articulated benchmarks and state-of-the-art average 3D point tracking across three datasets, with strong real-world generalization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MoSE3 provides a structured per-pixel motion representation that exposes rotation and rigid grouping alongside translation.

    Source: §6 · p. 10

    MoSE3 depends on π3 camera poses and pointmaps, and its accuracy degrades when its point-track or rigidity-embedding intermediates degrade, notably under fast motion; deformable scenes are evaluated only qualitatively.

    Source: §6 · p. 10

Abstract

from arXiv · show

Dense 3D point tracking has been a prominent paradigm for modeling motion in dynamic scenes, but a point track is just a 3-DoF translation curve per pixel: it captures where pixels go, not the rotation of the underlying part, nor which pixels move together as one body. We propose MoSE3, the first feed-forward model that predicts dense SE(3) motion from monocular RGB video, producing full 6-DoF rigid transforms at every pixel in world space. Per-pixel SE(3) motion offers a richer view of how a scene moves: rotation, translation, and grouping all at once. Directly predicting SE(3) is challenging: rotations lie on a curved manifold that is ill-suited to Euclidean regression, and annotations for SE(3) are particularly difficult to acquire. To address these challenges, MoSE3 predicts per-pixel SE(3) through two jointly learned intermediates, 3D point tracks and rigidity embeddings, and recovers SE(3) by differentiably fitting transforms within each soft rigid cluster, enabling end-to-end prediction and supervision. To close the data gap, we introduce Art-Kubric, a large-scale synthetic dataset with dense SE(3) and rigidity labels for articulated objects with rich physical interactions. MoSE3 achieves state-of-the-art SE(3) estimation at pixel, part, and object levels on both rigid and articulated benchmarks, and state-of-the-art average 3D point tracking accuracy across three datasets, while showing strong generalization to real-world videos despite being trained solely on synthetic motion data.

1 Introduction

Existing motion representations track pixel translations but omit rotation and rigid grouping, limiting their structure for tasks such as physical reasoning. MoSE3 predicts per-pixel world-space SE(3) from monocular RGB video, and the paper introduces Art-Kubric to address the shortage of articulated-scene supervision.

1 Introduction

  • Per-pixel world-frame SE(3) represents a full 6-DoF transform, so shared transforms indicate moving parts and deformable objects can be represented as locally rigid pieces.The paper contrasts this structured representation with pixel translation curves that omit rotation and rigid grouping.Source: §1 · p. 2
  • The paper identifies difficult direct SE(3) regression and scarce long-range SE(3) annotations, alongside the challenge of recovering geometry and camera motion in dynamic scenes.It positions prior recovery approaches as limited to settings such as known CAD models or depth sensors, parametric body models, or slow per-video optimization.Source: §1 · p. 2
  • MoSE3 predicts full per-pixel SE(3) motion from monocular RGB video by combining 3D point tracks and rigidity embeddings, then fitting transforms differentiably within soft rigid clusters.Art-Kubric provides dense SE(3) and rigidity labels for articulated scenes with multi-body physical interactions.Source: §1 · p. 2
  • State-of-the-art SE(3) estimation spans pixel, part, and object levels on rigid and articulated benchmarks, alongside state-of-the-art average 3D point tracking across three datasets.The experiments also report strong generalization to real-world videos.Source: §1 · p. 2

2 Related Work

Prior motion-estimation work ranges from image-plane flow and tracking to world-space 3D tracking and SE(3) estimation. The paper distinguishes MoSE3 by predicting dense world-frame SE(3) motion feed-forward from monocular RGB video.

Optical Flow and 2D Point Tracking

Scene Flow and 3D Point Tracking

  • World-space 3D tracking predicts long-range point trajectories, but its primary output remains point translation rather than rotation and grouping [58] [77] [41] [61].The passage describes recent feed-forward methods as leveraging geometric foundation models for long-range tracks on continuous videos.Source: §2 · p. 3

SE(3) Motion Estimation

  • Prior SE(3) approaches include object-level pose estimation, category-specific deformable models, per-video optimization, and pairwise scene motion; MoSE3 instead predicts per-pixel world-frame SE(3) feed-forward from monocular RGB video.The related work contrasts MoSE3’s closed-form alignment of predicted 3D tracks with iterative reprojection-based refinement in RAFT-3D.Source: §2 · p. 3

3 Method

MoSE3 recovers per-pixel world-frame SE(3) from 3D point tracks and rigidity embeddings using differentiable weighted Horn alignment. Joint losses supervise the intermediates and recovered motion, while a trainable tracking branch draws on a frozen geometry backbone.

3.1 SE(3) Prediction Formulation

  • MoSE3 reduces SE(3) prediction to dense 3D point tracks and rigidity embeddings, then uses a differentiable weighted Horn fit over soft rigid neighbors to recover each pixel’s transform.SE(3) supervision therefore flows back into both intermediate predictions.Source: §3.1 · p. 4

3.2 Joint Supervision of Tracks, Rigidity Embeddings, and SE(3)

  • The model predicts camera-coordinate tracks, visibility logits, and rigidity embeddings, and jointly supervises tracking, visibility, rigidity affinities, and recovered SE(3) motion.The geometry backbone also supplies pointmaps and camera-to-world poses used to obtain world-frame tracks.Source: §3.2 · p. 4

3.2.1 Tracking Losses

  • Track positions are parameterized by normalized image coordinates and signed depth, with losses for image coordinates, depth, query-frame surface gradients, and temporal displacement.Visibility logits receive binary cross-entropy supervision against ground-truth visibility flags on valid annotated tracks.Source: §3.2.1 · p. 5

3.2.2 Rigidity Embedding Loss

  • Rigidity embeddings use soft-affinity objectives from SE(3) annotations when available and from track geometry on eligible clips without those labels.Track affinity uses the spread of pairwise distance over time and requires at least two jointly valid frames; the supervision can include clips without SE(3) labels.Source: §3.2.2 · p. 5

3.2.3 SE(3) Losses

  • When SE(3) labels are available, the recovered transforms are directly supervised with a loss comparing rotation and the transformed weighted support centroid.The loss uses a Huber penalty and balances its rotation and translation terms.Source: §3.2.3 · p. 6

3.2.4 Full Loss

  • The full objective is a weighted sum of tracking, visibility, embedding, and SE(3) losses, with unavailable supervision terms zeroed.Training applies the losses jointly in a single stage, across three phases that differ only in learning-rate schedule.Source: §3.2.4 · p. 6

3.3 Geometry-Grounded Tracking Architecture

  • A trainable tracking branch attends to tokens from a frozen geometry branch, preserving pretrained geometric priors while learning motion from tracking data.The rigidity-embedding head is query-frame invariant, unlike the point-tracking head, because rigid-body identity is a scene property.Source: §3.3 · p. 6

4 The Art-Kubric Dataset

Art-Kubric combines articulated-object motion, rigid-body interactions, and dense motion annotations in a configurable synthetic dataset. Its release provides large-scale tracks and per-link SE(3) labels, including for occluded or small links.

Dataset capabilities

  • Art-Kubric combines articulated assets, dense long-range tracks, per-link SE(3) ground truth, shared-rigid-motion partitions, physics-driven interactions, and configurable rendering.Its assets span 46 PartNet-Mobility, 226 Articraft, and 7 Infinigen-Articulated categories.Source: §4 · p. 6

Dataset release

  • 5,000 scenes contain 13–30 rigid and articulated objects each, with about two-thirds moving and 163M tracks carrying rigid-group labels, occlusion flags, and link transforms.The release also includes depth, surface normals, three-granularity segmentation, and full camera parameters.Source: §4 · p. 7
  • The pipeline supports unlimited re-rendered clips with the same supervision, and its Kubric MOVi-F-compatible resolution and file layout work with existing tracking loaders unchanged.Tab. 7 contrasts its supervision channels with prior corpora.Source: §4 · p. 7

Dataset motivation

  • Unlike prior corpora described, Art-Kubric combines articulated multi-link motion, physics-driven inter-body contact, and exact per-link SE(3) labels, including for occluded or too-small links.Its rigid-motion partition separates links within one object and merges bodies at rest with the floor.Source: §4 · p. 7

5 Experiments

The experiments evaluate MoSE3 on dense and object/part-level SE(3) estimation and world-coordinate 3D tracking. MoSE3 leads on both rigid and articulated SE(3) benchmarks and has the best average tracking accuracy at both reported horizons.

5.1 SE(3) Evaluation

  • MoSE3 achieves the best per-pixel and object/part-level SE(3) estimation across all metrics on rigid HO3D and articulated iTACO.It also outperforms every baseline under the same k-NN pipeline; learned rigidity embeddings further improve per-pixel estimates and cluster IoU.Source: Table 1 · p. 8, §5.1 · p. 8
  • SE(3) evaluation measures per-pixel errors and clusters pixels into rigid bodies for object/part-level comparisons.Baselines use k-NN grouping, while MoSE3 can cluster its learned rigidity embeddings; ProxyPose instead predicts one pose per query and is reported only at object/part level.Source: §5.1 · p. 8

5.2 3D Tracking Evaluation

  • MoSE3 has the best average world-coordinate 3D tracking accuracy at L-16 and L-50, ranking second only on ADT L-16.The evaluation covers PointOdyssey, ADT, and PStudio.Source: Table 2 · p. 9, §5.2 · p. 9

5.3 Ablation Study

  • Directly regressing SE(3) degrades accuracy and generalization relative to decomposing prediction into point tracks and rigidity embeddings, then recovering transforms in closed form.All ablations, including the full model, use identical reduced-scale settings because of compute constraints.Source: §5.3 · p. 9
  • Removing Art-Kubric degrades SE(3) accuracy on articulated iTACO, while HO3D SE(3) accuracy is unchanged and cluster IoU falls.The passage attributes this difference to Art-Kubric’s multi-object articulated supervision, which existing rigid-only Kubric scenes do not provide.Source: §5.3 · p. 10
  • Jointly supervising tracks with rigidity and SE(3) losses improves their rigid coherence and yields more accurate SE(3) than either track-only variant.A controlled comparison holds the full model’s embedding fixed, isolating the tracks entering the fit as the source of the remaining gap.Source: §5.3 · p. 10
  • Removing either the rigidity or SE(3) loss degrades SE(3) accuracy, with the SE(3) loss having the larger impact.Removing rigidity loss more strongly degrades the embedding and cluster IoU; SE(3) supervision shapes it for local fitting but not global body separation.Source: §5.3 · p. 10

6 Conclusion

MoSE3 predicts per-pixel SE(3) from monocular RGB video using tracks and rigidity embeddings, and achieves state-of-the-art SE(3) estimation in rigid and articulated scenes. Its recovered motion also depends on camera geometry and intermediate quality, while deformable-scene evaluation remains qualitative.

6 Conclusion

  • MoSE3 achieves state-of-the-art per-pixel and object-level SE(3) estimation in rigid and articulated scenes, and sets a new state of the art for 3D point tracking.The conclusion also reports a large-scale synthetic dataset for multi-object articulated scenes.Source: §6 · p. 10
  • The model predicts per-pixel SE(3) by combining dense 3D point tracks and rigidity embeddings, then analytically recovering transforms with a differentiable closed-form Horn fit.Art-Kubric supplies synthetic SE(3) supervision for articulated scenes.Source: §6 · p. 10
  • The structured representation exposes rotation and rigid grouping alongside translation, with downstream relevance to physical reasoning, manipulation, and articulated-object understanding.Source: §6 · p. 10
  • MoSE3’s recovered SE(3) is vulnerable to errors in π3 camera geometry and to degraded tracks or embeddings, especially under fast motion.Deformable scenes are evaluated only qualitatively; a quantitative benchmark is left for future work.Source: §6 · p. 10

A.1 Training Setup

The training setup jointly supervises motion tracks, visibility, rigidity, and per-pixel SE(3), using weighted Horn fitting and multiple motion losses. It also specifies coordinate and scale conventions and the datasets supplying tracking and SE(3) labels.

A.1 Training Setup

  • MoSE3 is jointly supervised for tracks, visibility, rigidity embeddings, and per-pixel SE(3), and trains for 200 epochs in three learning-rate phases.Initialization uses a pretrained π3 checkpoint with its encoder and geometry decoder frozen; optimization uses AdamW, and loss weights cover tracking, visibility, embeddings, and SE(3).Source: §A.1 · p. 19

A.2 Per-Anchor SE(3) Recovery via Weighted Horn

  • Per-anchor SE(3) is recovered by a differentiable weighted Horn fit using soft rigidity weights over 4,096 query-frame support points.The quaternion solution is chosen for more stable gradients, and the fit propagates gradients into both tracking and rigidity heads.Source: §A.2 · pp. 19–20

A.3 Coordinate and Scale Conventions

  • Tracks use centered image coordinates and decoded depth; focal lengths are fitted per video from π3 bearings, with no ground-truth intrinsics needed at inference.Predicted and ground-truth 3D quantities are normalized and scale-aligned for supervision.Source: §A.3 · p. 20

A.4 Tracking Loss Details

  • Tracking supervision combines UV/depth tracks and visibility with query-frame surface gradients and fixed-query-camera temporal displacements; surface gradients require dense reconstruction labels.Valid off-screen and behind-camera targets are retained, while near-zero-depth projection outliers are filtered.Source: §A.4 · pp. 21–22

A.5 Dataset

  • The six-source training mix includes three datasets with rigid-body partitions and SE(3) labels, which alone supervise the SE(3) and SE(3)-affinity losses.Track, visibility, and tracking-affinity losses use all six sources.Source: §A.5 · p. 22

B Architecture Details

The tracking branch is initialized alongside a frozen geometry branch, while the point-tracking head uses alternating attention blocks to predict target-frame tracks and visibility.

B.1 Tracking Branch

  • The tracking branch parallels frozen π3 geometry layers, with tracking queries attending to both tracking-side and geometry-side keys and values.The shared and frozen layers are 0–9; layers 10–35 have trainable tracking counterparts.Source: §B.1 · p. 22

B.2 Point-Tracking Head

  • The point-tracking head alternates frame-wise and global self-attention, then predicts per-pixel normalized image coordinates, depth, and visibility for each target frame.Its track and visibility outputs are absolute target-frame values, not offsets from the query pixel.Source: §B.2 · p. 22

C.1 Ablation Setup and Full Results

The ablation compares the full model with alternatives spanning architecture, training data, SE(3) recovery strategy, and loss design. Variants share a training schedule, but the supplied passages do not provide the table’s metric values.

C.1 Ablation Setup and Full Results

  • All variants use an identical schedule with all losses enabled, so reported metric differences reflect the changed variable.Each variant trains for 80 epochs on four H200 GPUs; the track-only rigidity-clustering variant reuses embeddings from the full-loss model.Source: §C.1 · pp. 22–23
  • The ablation tests direct SE(3) regression, removing Art-Kubric, track-only posthoc recovery alternatives, and removal of individual loss terms against the full model.The table caption identifies the full model as the bottom row.Source: Table 4 · p. 23
  • The ablation reports ADD in centimeters, AUCr at a 30° threshold, and AUCADD at 10 cm for HO3D and 20 cm for iTACO.Source: Table 4 · p. 23

C.2 SE(3) Evaluation Detail

The evaluation derives per-pixel and object-level SE(3) from point tracks using weighted Procrustes fits, with support and weighting tailored to each method. It also specifies clip alignment, metric aggregation, and ProxyPose’s separate evaluation constraints.

C.2 SE(3) Evaluation Detail

  • Predicted tracks are globally scale-and-translation aligned once per clip using visible labeled pixels, while excluded pixels remain available to methods as neighbors or clustering negatives.The same alignment is applied to all predicted points before metrics are computed.Source: §C.2 · p. 23
  • MoSE3 weights all pixels by rigidity-embedding similarity for per-pixel fits, while baselines use uniformly weighted nearest neighbors and select K per clip by a composite score.The fit uses temperature τ=0.01 for MoSE3; the baseline K sweep is {4, 8, 16, 32, 64, 128, 256}.Source: §C.2 · p. 23
  • Per-pixel results aggregate mean RRE, mean ADD, AUCr up to 30°, and AUCADD at dataset-scaled thresholds of 10 cm or 20 cm.ADD uses ground-truth surface samples at frame 0 transformed by predicted and ground-truth poses.Source: §C.2 · p. 23
  • The supplementary point-tracking table reports L-16 and L-50 accuracy across PointOdyssey, ADT, and PStudio in camera and world coordinates.It follows Track4World’s released evaluation code, including a per-sequence scale-and-shift fit using ground-truth-occluded points before TAPVid-3D median scaling.Source: Table 5 · p. 24
  • Object-level evaluation matches HDBSCAN clusters to ground-truth masks, then refits one SE(3) per frame using the assigned tracks.MoSE3 clusters rigidity embeddings and weights tracks by similarity to the cluster centroid; baselines cluster concatenated SE(3) predictions at two reference frames and use uniform weights.Source: §C.2 · p. 24
  • ProxyPose is evaluated only at object/part level under its own protocol, and ADD is omitted because diverged clips produce extreme translation errors.The tracker diverges on 5 of 54 HO3D clips and 7 of 55 iTACO clips; AUCADD and RRE remain comparable.Source: §C.2 · p. 24

C.3 Additional Results

On nine real-world YCBInEOAT clips, MoSE3 is reported best on every metric at both evaluation granularities. Additional in-the-wild visualizations report more stable, coherent motion axes than ProxyPose across rigid and deformable scenes.

C.3 Additional Results

  • Supplementary 3D tracking results cover camera and world coordinates on PointOdyssey, ADT, and PStudio using Track4World’s released evaluation code.Before TAPVid-3D median scaling, the code fits per-sequence scale and shift using points occluded in ground truth.Source: §C.3 · p. 24
  • MoSE3 achieves the best result on every metric at both granularities on nine YCBInEOAT robot-manipulation clips.The table evaluates rigid objects and reports ADD, AUCr, and AUCADD.Source: Table 6 · p. 25, §C.3 · p. 25
  • ProxyPose is reported only at object/part level under its own single-query protocol, with ground-truth intrinsics, and has no predicted mask for IoU.Its ADD is omitted because a few diverged clips dominate the mean.Source: §C.3 · p. 25

C.4 Qualitative Visualizations

  • MoSE3 produces more stable and coherent SE(3) axes than ProxyPose across in-the-wild rigid and deformable scenes.The visualizations draw axes at sampled pixels and include examples from several real-world video sources.Source: §C.4 · p. 25

D Art-Kubric: Additional Details

Art-Kubric combines varied synthetic scene production with dense motion and rigid-group annotations, and its release supports re-rendering under custom settings. The paper also identifies licensing constraints and potential misuse through fine-grained human motion analysis.

D Art-Kubric: Additional Details

  • The production setting generates 5,000 scenes at 512×512 and 120 frames at 60 fps, with 32,768 tracks per scene.It uses a pool of articulated URDFs, rigid objects, and HDRI environment maps.Source: D Art-Kubric: Additional Details · p. 25
  • Per-scene randomization spans 18–90° vertical field of view, 4–7 m camera radii, varied lighting, and object placements that permit contact.A randomized camera-distance exponent reduces the field-of-view/object-size dependence described for a fixed exponent.Source: D Art-Kubric: Additional Details · p. 25
  • Rigid-part segmentation merges links with identical motion within 10^-3 tolerance, making the grouping target shared transforms rather than object instances.The scene statistics include about 100 kinematic links collapsing into about 85 rigid-part segments, with roughly 163M long-range tracks across the release.Source: D Art-Kubric: Additional Details · p. 26
  • The generation pipeline can re-render clips under custom lighting, scenes, and asset distributions, with a dataset-level control for realized asset-source shares.The pipeline solves that share control from measured per-source placement rates.Source: D Art-Kubric: Additional Details · p. 26
  • The released data is CC BY-NC 4.0 because scenes include PartNet-Mobility assets whose terms restrict use to non-commercial research and education.The generation pipeline does not redistribute upstream assets, so users re-rendering scenes obtain each source asset under its own license.Source: D Art-Kubric: Additional Details · pp. 26–27
  • Among the datasets compared, Art-Kubric is the only one providing all seven listed properties, spanning annotations and scene regimes.The comparison concerns which annotation channels and scene regimes datasets expose; rigid-grouping denotes pixels partitioned by shared transform rather than instance.Source: D Art-Kubric: Additional Details · p. 27, Table 7 · p. 27

E Broader Impacts

  • The paper notes that fine-grained human motion analysis could support surveillance, while the model outputs geometric motion without human-identity supervision.The authors say they foresee no additional harms beyond those of monocular geometry and tracking models built on π3-class foundations.Source: E Broader Impacts · p. 28

Related papers

Loading 2610.03716v1…