Source-linked AI summary
$μ_0$: A Scalable 3D Interaction-Trace World Model
Seungjae Lee, Yoonkyo Jung, Jusuk Lee, Jonghun Shin, Amir Hossein Shahidzadeh, Yao-Chih Lee, H. Jin Kim, Jia-Bin Huang, Furong Huang
TL;DR
Robot learning lacks scalable, embodiment-compatible action supervision despite abundant video data. μ0 addresses this gap by learning smooth 3D interaction traces from heterogeneous videos, then reusing the frozen model with action experts; its policies outperform π0 in simulation and real-world tasks while using video-only pretraining.
Problem
Action-labeled robot data is scarce, expensive, hardware-specific, and incompatible across embodiments, while pixel-space and direct-action models have important limitations for manipulation.
Method
TraceExtract converts heterogeneous videos into event-captioned, globally aligned 3D traces, which pretrain μ0 to forecast smooth semantic keypoint trajectories for later reuse by embodiment-specific action experts.
Results
Across trace forecasting, simulation, and real-world manipulation, μ0 outperforms trace and tokenized-VLM baselines; its action-expert policy reaches 30.25% average success in simulation and 91.7% in real-world tasks.
Takeaways & Limitations
Frozen 3D interaction traces provide a compact, embodiment-agnostic motion prior for scalable downstream robot control after video-only pretraining.
Takeaways & Limitations
μ0 inherits perception-stack errors, omits explicit forces, tactile feedback, and contact modes, and is evaluated mainly on tabletop manipulation with limited embodiments and task families.
Abstract
from arXiv · showhide
World models that capture how actions induce physical change enable scalable robot learning without reliance on embodiment-specific action labels. Pixel-space video models provide broad visual priors but expend model capacity on dense appearance reconstruction, while direct action models require embodiment-specific labels that hinder scalability. We present $μ_0$, a scalable world model based on 3D traces. Rather than predicting dense pixels or directly modeling actions, $μ_0$ forecasts smooth 3D trajectories for salient interaction points such as objects, tools, hands, and contact regions, yielding a compact, embodiment-agnostic motion interface. To enable training from diverse video sources, our TraceExtract system automatically extracts 3D supervision by selecting keypoints, constructing globally aligned traces, and associating motion segments with hierarchical language captions. This TraceExtract supervision pretrains $μ_0$ by combining a pretrained vision-language backbone with a modular trace expert, which represents each query via B-spline control points and predicts future traces. Experiments show that $μ_0$ outperforms baselines in both 2D and 3D trace prediction, including trace prediction models and tokenized VLM methods. Because $μ_0$ is frozen and reusable, it can be paired with action experts for downstream robot embodiments. Despite action-free pretraining, the resulting trace-conditioned policies achieve performance competitive with VLA models pretrained with action supervision, such as $π_0$. These results establish 3D traces as a scalable and transferable representation for cross-embodiment manipulation.
1. Introduction
The paper proposes 3D interaction traces as a compact, embodiment-agnostic alternative to pixel prediction and direct action modeling. TraceExtract scales video-only supervision, while μ0 forecasts reusable motion priors that support downstream action experts.
- Motivation: 3D traces of semantic interaction points compactly describe task-relevant motion across robot embodiments.They focus on object parts, tools, hands, and contact regions rather than dense pixels or embodiment-specific actions.
- Motivation: Existing motion-centric systems can miss small task-critical regions, confuse object and camera motion, and align language only at the episode level.These limitations motivate semantic selection, globally aligned 3D tracking, and event-level language.
- Contributions: TraceExtract converts heterogeneous human and robot videos into event-captioned 3D supervision using semantic keypoints, global 3D lifting, and hierarchical language.The pipeline scales trace curation by roughly 8× over prior 3D trace datasets.
- Contributions: μ0 combines a pretrained VLM with a permutation-equivariant Trace Expert that forecasts smooth semantic keypoint traces using B-spline targets and semantic flow matching.The model is designed as a query-conditioned 3D trace-space world model.
- Contributions: After video-only pretraining, frozen μ0 can provide reusable motion features to action experts for target robot embodiments.The paper evaluates this approach against action-labeled VLA baselines in simulation and real-world manipulation.
2. TraceExtract: A Scalable Cross-Embodiment Data Pipeline
TraceExtract turns uncurated manipulation videos into compact, globally consistent, event-captioned 3D trace supervision. It selects task-relevant entities, preserves their motion across camera and memory constraints, and aligns captions to motion events.
- Pipeline overview: TraceExtract targets interaction-centric keypoints on objects, tools, hands, and contact regions because their 3D motion captures task-relevant change.Useful traces require semantic selection, consistent 3D tracking, and event-level language.
- Pipeline overview: The pipeline converts heterogeneous videos into {observation, trace, language} triplets for training μ0.It remedies fixed-grid, short-clip trace curation with task-relevant keypoints, globally consistent identities, and motion-aligned language.
- Semantic Keypoint Sampling: DINOv2 entity clusters guide per-entity keypoint allocation, spatially diverse sampling, and filtering of static or background-dominated tracks.This addresses area bias that can undersample small objects, tool tips, and contact patches.
- Global–local reconstruction: Global–local reconstruction establishes a shared coordinate frame with sparse anchors, aligns dense local chunks, and tracks keypoints across long videos.The design addresses egocentric camera motion, changing scene contents, and full-video memory limits.
- Motion-centric chunking: TraceExtract segments demonstrations at acceleration valleys between action anchors to create short motion-centric captioning events.Hierarchical VLM captioning uses start, midpoint, and end frames, then merges adjacent captions into finer and coarser descriptions.
- Trace Supervision Interface: The resulting supervision pairs observations and event or task captions with past and future reference-camera 3D traces for query keypoints.These tuples train μ0 to predict future motion for the interaction-centric query set.
3. µ0: Query-Conditioned Trace World Model
μ0 is a query-conditioned 3D dynamics model that combines VLM context, exchangeable spline queries, and semantic flow matching to forecast smooth future traces. Its frozen trace features can feed action experts, while evaluation emphasizes prediction quality and efficient inference.
- Overview: μ0 predicts 3D motion for interaction-centric keypoints from observations, language, and optional keypoint history instead of generating pixels.Its design addresses semantic–metric fusion, unordered query sets, and multimodal future dynamics.
- Overview: The model uses VLM scene-language context, a permutation-equivariant trace expert, and semantic flow matching for flexible future generation.Together these components produce compact metric motion tokens from semantic keypoints.
- Spline queries: Each keypoint is represented by exchangeable per-query tokens and cubic B-spline controls grounded in local visual features.B-spline targets provide compactness, smoothness, and easier denoising.
- Semantic flow matching: Conditional flow training denoises noisy control points using VLM context, query conditions, and flow-time modulation.Validity and rigidity losses handle trajectory termination and preserve local geometry within DINO clusters.
- Trace-conditioned action expert: The action interface freezes μ0 and trains an action expert that consumes trace-denoising features alongside robot observations, proprioception, and language.The policy can read features from a single partial-denoising step before predicting continuous action chunks.
4. Experiment
Experiments evaluate μ0 for trace forecasting and downstream action generation in simulation and real-world manipulation. Across these settings, μ0 improves trace and policy performance, with scaling and trace features supporting transfer to robot control.
- Trace Prediction Quality: μ0 achieves the best Top-5 ADE, FDE, and DTW across all 2D horizons and every reported 3D metric.Metrics are computed on moving points; evaluation includes Average Displacement Error, Final Displacement Error, and Dynamic Time Warping.
- Trace Prediction Quality: 0.29s prediction latency is 2.9× faster than Track2Act’s 0.85s, the next-fastest reported 2D baseline.
- Simulated Action Generation: 30.25% average success rate lets μ0 + action expert outperform π0 by 5.0 points in eight RoboCasa365 tasks.The comparison uses video-only pretraining for μ0, while π0 is an action-labeled VLA baseline.
- Real-World Action Generation: 91.7% average success rate makes μ0 + action expert the top method across three real-world UR3 tasks.The tasks include Pick into Sink, Pour Almonds, and Unfold Towel, with Pick & Place and Pour averaged over multiple objects.
- Scaling Analysis: Trace prediction improves with larger models and more pretraining data, reaching the best top5-DTW with the 2.59B model.
- Scaling Analysis: The performance gap versus the w/o Trace variant widens as action-head size decreases, indicating motion structure from trace-space pretraining.The result suggests limited policy capacity cannot recover the motion structure supplied by traces.
5. Related Work
Related work develops embodiment-agnostic world models and motion-guided manipulation through pixels, actions, flows, tracks, and intermediate representations. μ0 differs by targeting semantic, globally aligned 3D traces without fixed-grid inference-time depth requirements.
- World Models and Visual Motion Priors: Pixel-space and world-action models provide visual or joint action-frame predictions but expend capacity on dense appearance rather than geometry and contact.
- Trace-Guided Manipulation: Trace-guided manipulation methods include VLM waypoint tracking, video generation with track extraction, and direct track-predicting policies.
- Trace-Guided Manipulation: TraceGen uses fixed-grid traces over short clips, requires inference-time depth, and relies on hand-designed trace replay rather than a reusable query-conditioned 3D world model.
6. Conclusion
The paper introduces μ0 as a query-conditioned 3D trace-space world model for cross-embodiment manipulation. Its experiments support 3D interaction traces as a compact representation, while its scope remains limited by perception errors and tabletop-focused evaluation.
- Conclusion: μ0 predicts smooth future 3D motion for semantically selected interaction keypoints instead of pixels or embodiment-specific actions.
- Conclusion: TraceExtract converts heterogeneous videos into event-captioned 3D trace tuples through semantic keypoint selection, globally aligned tracking, and motion-centric captioning.
- Conclusion: After video-only pretraining, frozen μ0 can be reused by action experts as an embodiment-agnostic motion prior for downstream robot control.
- Limitations and Future Work: The action-expert evaluation focuses on tabletop manipulation with limited embodiments and task families, leaving mobile manipulators, dexterous hands, and longer-horizon tasks for future work.
A. Dataset Construction
TraceExtract constructs scalable 3D supervision by selecting task-relevant keypoints, reconstructing a shared global frame, tracking identities across chunks, and captioning motion events hierarchically.
- Semantic Keypoint Selection: DINOv2 patch descriptors are clustered into entity groups, whose identities propagate temporally through feature similarity and spatial overlap.
- Semantic Keypoint Selection: Keypoint budgets are allocated by visible entity coverage with minimum allocation for small regions, reducing background bias and missed tool tips or contact patches.
- Motion Selection: Keypoints are marked moving when maximum visible-trace displacement exceeds τm = 40 pixels, making motion selection robust to tracker jitter.
- Global–Local Reconstruction: A hybrid VGGT scheme combines sparse global anchors with dense local passes to reconstruct long videos without fitting the full sequence in memory.
- Global–Local Reconstruction: Each local chunk is aligned directly to shared global anchors using an SE(3) transform, preventing alignment errors from compounding across chunks.
- Global–Local Reconstruction: Progressive tracking propagates active groups using prior global 3D positions, preserving identities despite camera motion and chunk boundaries.
- Event-Centric Captioning: Smoothed acceleration maxima define action anchors, while constrained chunk boundaries and hierarchical captions pair motion segments with object states and state changes.
B. Architecture and Training Details
μ0 combines a pretrained vision-language prefix with a modular trace expert that predicts smooth, query-conditioned 3D futures as B-spline control points. Its training adds localized semantic cues, conditional flow matching, validity and rigidity objectives, and partial-denoising features for action adaptation.
- Architecture: The conditioning backbone uses a frozen SmolVLM2-2.2B prefix, while a 20-layer Trace Expert cross-attends to its key-value cache and maintains a motion-specific stream.The Trace Expert uses half the VLM hidden width and alternates cross-attention with self-attention every two layers.
- Trace targets: Each query combines past history, a current 3D anchor, and a future target, with anchor subtraction and per-axis 95th-percentile scaling producing normalized motion targets.The model uses h=8 history steps and H=32 future steps.
- Tokenization: Per-query tokens encode clean history and noisy future controls with segment embeddings, Fourier coordinates, and localized DINO features for part-level semantics.Bilinear sampling extracts DINO features at each keypoint’s current-frame pixel coordinate before MLP fusion.
- Training objectives: The Trace Expert uses conditional flow matching from Gaussian noise to clean control points, with flow-time adaLN-Zero conditioning and a masked velocity loss over valid keypoints.The network predicts the constant target velocity ϵ−P⋆, while a validity head handles trace truncation.
- Training objectives: Auxiliary validity and rigidity objectives terminate traces after occlusion or track loss and preserve local geometry among keypoints sharing DINO clusters.Inference integrates the velocity field with four Euler steps and decodes the absolute trace through the B-spline basis.
- Action adaptation: The action expert uses a single partial-denoising step as a motion descriptor, fuses it into VLM features through gated cross-attention, and predicts continuous action chunks.It also conditions on gripper-camera images, proprioception, and language; the gate is initialized weakly and can strengthen during training.
C.1. Training Strategy
The evaluation measures multi-sample trace accuracy in normalized image-depth space across several horizons. Metrics capture average, final, temporally aligned, and worst-case path deviations, alongside latency and parameter count.
- Evaluation setup: Each evaluation slot predicts 16 keypoint traces over T∈{8,16,32} future steps in normalized (u,v,z) image-depth space.UV coordinates come from resized 256×256 frames, while z is measured in metric meters.
- Trace metrics: minADE and minFDE report the minimum across S samples of mean displacement over all timesteps and displacement at the final timestep.Both use Euclidean pointwise cost and average over valid future steps.
- Trace metrics: minDTW evaluates path shape after monotonic temporal alignment, reducing sensitivity to small timing misalignments.The metric takes the minimum over predicted samples.
- Trace metrics: The discrete Fréchet distance captures the maximum pointwise displacement under optimal monotonic alignment, making it sensitive to large excursions and endpoint errors.It is reported alongside model parameters because downstream consumers care about worst-case path deviations.
D.2. Additional results on Trace Prediction
Additional evaluations compare μ0 with trace predictors and general-purpose VLMs on 2D/3D forecasting, then examine simulation and real-robot policy settings. The reported comparisons emphasize trace coherence, parameter efficiency, and randomized-scene evaluation.
- Trace prediction: Fréchet-distance evaluation compares Top-1 FD and Top-5 FD across T∈{8,16,32} against 2D and 3D baselines using shared image-text inputs except depth-conditioned methods.Table 3 also reports model parameters for the trace-prediction comparison.
- Trace prediction: μ0 maintains a favorable parameter-efficiency trade-off while counting every component active during trace inference, including frozen backbones and denoising networks.Training-only teacher networks are excluded from the reported inference-time parameter count.
- Qualitative results: Across diverse manipulation tasks, μ0 produces coherent, task-relevant traces, while baselines more often produce sparse, noisy, overly dense, or spatially misaligned traces.Figure 7 includes ground truth, μ0, general-purpose VLMs, and trace-prediction baselines.
- Simulation evaluation: RoboCasa365 evaluates eight household manipulation tasks with shared demonstrations, RGB observations, language instructions, and proprioceptive states across randomized layouts and object instances.The study uses 100 demonstrations per task and 50 evaluation rollouts per task.
- Real-robot evaluation: In real-robot experiments, a fixed-base UR3 with a two-finger gripper uses third-person and wrist RGB cameras plus a 7D proprioceptive state.The same task-specific hyperparameters are used across real-world methods, while μ0’s VLM and trace expert remain frozen.
E.1. Ablation Studies
The ablations test μ0’s trajectory parameterization, localized semantic features, rigidity regularization, and robustness to missing modalities. These studies isolate architectural, optimization, and input-modality choices affecting trace prediction.
- Architectural and optimization design: The ablations remove B-spline parameterization, DINOv2 keypoint features, or the rigidity loss to measure their effects on trace prediction and intra-part consistency.The direct-regression variant predicts H=32 raw anchor-relative future steps instead of D=10 control points.
- Architectural and optimization design: Rigidity-loss variants change λrig or set λrig=0 to test the auxiliary objective’s role in preserving physical consistency within object parts.The rigidity objective regularizes keypoints belonging to the same part cluster.
- Input modality robustness: Input-modality ablations omit metric depth or historical traces, requiring prediction from a static RGB observation alone.These variants evaluate robustness to missing or degraded input information.
E.2. Scaling Analysis
Scaling experiments show that µ0’s trace prediction improves with both model capacity and training data, while frozen trace features benefit downstream action learning across action-head sizes.
- Data scaling: Data scaling improves top5-DTW from 0.134/0.200/0.235 to 0.127/0.187/0.223 as the pretraining set grows from 5% to 100%.The gains are most consistent at longer horizons, where diverse videos support temporally extended motion prediction.
- Model scaling: The monotonic model-scaling trend indicates that trace prediction remains capacity-limited at the current scale.
- Action-head scaling: Trace features raise success from 10.675% to 25.625% with a 200M action head and from 28.25% to 30.25% with a 400M head.The larger improvement with the smaller head suggests structured trace information is especially useful when action-head capacity is limited.