PaperlayerView exact revision on arXiv

Source-linked AI summary

SMART: Zero-Shot Sim-to-Real Articulated Object Manipulation via Large-Scale Synthetic Pretraining

Jicong Ao, Shuhan Jiang, Yuling Zhong, Yanwen Liu, Yuhan Gao, Jiangyuan Zhao, Yang Zhang, Shiqiang Zhu, Chenjia Bai, Xuelong Li

arXiv:2610.07652v1cs.ROcs.AI

TL;DR

High-quality real-world demonstrations for articulated-object manipulation are difficult to collect at scale, and existing synthetic data and general-purpose pipelines have limited articulated-object coverage and lack explicit part-level and articulation-aware designs. SMART combines an articulation-aware simulator with agentic task generation and distributed synthesis to build SMART-Data. Pretraining on the data improves downstream VLA performance and supports zero-shot sim-to-real transfer with positive scaling trends.

  • Problem

    Real-world articulated-manipulation demonstrations are difficult to collect at scale, while synthetic datasets cover limited articulated-object categories and general-purpose pipelines lack explicit part-level semantics and articulation constraints.

    Source: §1 · pp. 1–2

  • Method

    SMART uses an articulation-aware simulation platform, agentic task generation, and distributed synthesis to construct SMART-Data for VLA pretraining.

    Source: §1 · pp. 2–3

  • Results

    Pretraining on SMART-Data improves downstream VLA performance and enables zero-shot sim-to-real transfer, with positive performance-scaling trends.

    Source: §1 · p. 3, §8 · p. 18

  • Takeaways & Limitations

    Synthetic demonstrations provide a scalable supervision source for contact-rich articulated-object manipulation and support transfer across robot platforms, tasks, and policy architectures.

    Source: §1 · p. 3

    Synthesis throughput decreases for long-horizon tasks because planning and execution success rates are lower.

    Source: §7 · p. 18

Abstract

from arXiv · show

The ability to interact with articulated objects is essential for embodied intelligent systems, but collecting large-scale real-world demonstrations for these interactions remains challenging due to the precise contact and constraint-following motions involved. Although simulation provides a promising alternative, existing synthetic data efforts cover limited articulated-object categories, while general-purpose synthesis pipelines lack explicit designs for part-level semantics and articulation constraints, hindering agentic task generation and scalable synthesis of high-quality articulated-manipulation demonstrations. To bridge this gap, we introduce SMART, a scalable system leveraging large-scale Synthesized Manipulation demonstrations for ARTiculated-object manipulation. At its core, we develop SMART-Sim, a simulation platform with articulation-aware design that enables effective task generation and efficient demonstration collection. Building on SMART-Sim, we apply agentic task generation and design a scalable distributed synthesis system, using them to synthesize SMART-Data, comprising over 1M demonstrations across 44 atomic task types, 5 robot setups, and 2,507 articulated objects. The vision-language-action (VLA) model pretrained on SMART-Data shows competitive performance on simulation benchmarks and achieves zero-shot sim-to-real transfer and scalable performance in real-world articulated-object manipulation tasks. This highlights the potential of synthetic demonstrations in providing effective and scalable supervision for improving VLA model performance in contact-rich articulated-object manipulation.

1 Introduction

Articulated-object manipulation requires precise contact with functional parts while respecting joint constraints, yet high-quality real-world demonstrations remain difficult to collect at scale. SMART combines an articulation-aware simulator, agentic task generation, and distributed synthesis to produce data for VLA pretraining and evaluate transfer.

1 Introduction

  • Real-world articulated-manipulation demonstrations are costly and hard to scale, while transferring human-collected demonstrations can introduce embodiment mismatches and physical infeasibilities.These challenges are especially pronounced when demonstrations must maintain precise contact while satisfying articulation constraints.Source: §1 · p. 2
  • Existing synthetic datasets cover limited articulated-object categories, and general-purpose synthesis pipelines lack explicit part-level semantics and articulation constraints.These gaps leave large-scale synthetic data for VLA learning and sim-to-real transfer across diverse articulated-object tasks underexplored.Source: §1 · p. 2
  • SMART-Sim provides articulation-aware designs for asset annotation, motion generation, skill design, and domain randomization; agentic task generation and distributed synthesis produce SMART-Data.The data covers diverse articulated objects and manipulation skills.Source: §1 · pp. 2–3
  • SMART-Data pretraining improves downstream VLA performance, while synthesized demonstrations support zero-shot sim-to-real transfer and positive performance-scaling trends.The dataset contains over 1M trajectories across 5 robot setups and 23 object categories.Source: §1 · p. 3

2 Related Work

Prior approaches use object models, affordance prediction, and simulation pipelines to support manipulation, while real-world demonstration collection and synthetic datasets each have coverage or scalability constraints. The paper situates SMART-Sim within this landscape of simulation and data-synthesis work.

2 Related Work

  • Articulated-object methods use pretrained encoders, annotated simulation assets, explicit object models, or affordance prediction, but often depend on simplified models and accurate geometry and articulation perception.These dependencies limit scalability across diverse objects and real-world settings.Source: §2 · p. 3
  • Teleoperation, handheld interfaces, and egocentric demonstrations provide real-world VLA data, but collection cost, tracking precision, pose estimation, retargeting, and robot alignment remain challenges.Small action errors are particularly problematic for articulated tasks requiring sustained contact and joint-constraint compliance.Source: §2 · p. 3
  • Simulation frameworks and trajectory-generation methods support broad scenes and scalable demonstrations, while large synthetic datasets have supported pretraining and real-world transfer.Large-scale synthetic VLA data for diverse, contact-intensive articulated-object manipulation remains underexplored.Source: §2 · p. 4

3 Simulation Platform

SMART-Sim combines articulation-aware asset support, motion generation, manipulation skills, and domain randomization to synthesize demonstrations. Its pipeline uses part and joint annotations to guide feasible interaction motions and varies spatial, physical, and visual conditions.

3 Simulation Platform

  • The platform supports diverse robot embodiments and assets, flexible annotation and motion generation, efficient demonstration collection, and comprehensive domain randomization.Its articulation-aware designs account for object part structure, semantic roles, and physical constraints.Source: §3 · p. 4
  • SMART-Sim has the largest articulated-object asset base among compared frameworks and uniquely combines AI-generated tasks, multi-environment parallelism, and distributed data synthesis.Table 1 compares frameworks; some report object totals without separating rigid and articulated objects.Source: Table 1 · p. 5, §3 · p. 5

3.1 Simulation Assets

  • The asset library includes 2,507 articulated objects across 88 functional categories, while action-frame annotations provide pose targets for motion generation.Articulated-part annotations record joint type, valid motion range, and manipulation semantics; feasible action frames are sampled around an anchor frame.Source: §3.1 · p. 5, §3.2 · pp. 5–6

3.2 Manipulation Motion Generation

  • For contact-rich interactions, trajectory optimization uses joint types, positions, and valid motion ranges as explicit constraints to maintain contact and follow articulation constraints.SMART-Sim also groups joint types and manipulation semantics into 23 reusable skills that agents can select and compose for task generation.Source: §3.2 · pp. 6–7

3.3 Task Domain Randomization

  • Domain randomization varies object and robot configurations, part-wise friction and density, joint stiffness and damping, component materials and textures, and lighting.The randomization follows articulated objects’ part structure, functional semantics, and physical constraints.Source: §3.3 · p. 7

4 Data Synthesis

SMART’s synthesis pipeline turns natural-language task descriptions into executable articulated-object tasks, then collects and renders successful demonstrations through distributed stages. SMART-Data comprises over 1M episodes across five robot setups, with broad task, skill, duration, and object coverage.

4.1 Agentic Task Generation

  • The agent translates task descriptions into structured simulator configurations, grounds assets, and composes existing annotated manipulation skills into parameterized programs.Its unified format specifies environment, task, and simulation settings, linking high-level intent to executable configurations.Source: §4.1 · pp. 8–9

4.2 Data Collection

  • A distributed scheduler separates trajectory collection, visual rendering, and dataset conversion, with successful trajectories rendered and converted in parallel stages.The released data were generated with 96 workers; one server with eight RTX 4090 GPUs generated approximately 144 hours of data per day.Source: §4.2 · pp. 9–10

4.3 Dataset Statistics

  • SMART-Data contains over 1M episodes across five robot setups, totaling approximately 500M frames and over 4,600 hours of manipulation data.It includes 23 manipulation skills, 44 atomic task types, and 1,122 distinct scenes.Source: §4.3 · p. 10
  • The trajectories comprise 69.4% atomic, 20.9% composite, and 9.7% long-horizon tasks, spanning durations from hundreds to millions of frames and 23 functional object categories.Those categories come from 88 available in SMART-Sim’s asset library and include varied revolute, prismatic, and compound mechanisms.Source: §4.3 · pp. 10–11

5 Simulation Experiments

The experiments compare SMART-Data pretraining with no first-stage pretraining while holding architecture and downstream training fixed. SMART-VLA performs strongly on LIBERO and RoboCasa365, with gains also reported for early learning and composite or long-horizon tasks.

5.1 Experiment Setup

  • The controlled comparison keeps the VLA architecture and downstream training recipe fixed, making pretraining data the primary difference between variants.SMART-Data-only first-stage pretraining predicts tokenized robot action sequences; second-stage training uses each benchmark’s downstream dataset.Source: §5.1 · pp. 11–12

5.2 Key Findings

  • SMART-VLA scores 96.8% on LIBERO and 28.8% on RoboCasa365, versus 94.7% and 20.0% for SMART-VLA(w/o S1), respectively.On RoboCasa365, it outperforms every compared baseline; on LIBERO, it surpasses π0 and Qwen3-VL-π and essentially matches π0.5.Source: Table 3 · p. 12, Table 4 · p. 12, §5.2 · p. 12
  • At 5k LIBERO post-training steps, SMART-VLA reaches 29.9% average success versus 15.1% for SMART-VLA(w/o S1), and finishes at 96.8% versus 94.7% at 30k.The curve is non-monotonic after 15k steps, falling to 82.1% at 20k before recovering to 96.8% at 30k.Source: §5.2 · pp. 12–13, Figure 7 · p. 13
  • SMART-VLA reaches 94.0% on LIBERO-Long and exceeds the strongest non-SMART baselines on RoboCasa365 Composite Seen and Composite Unseen splits.Its gains over SMART-VLA(w/o S1) are 6.0 points on LIBERO-Long, 13.7 points on Composite Seen, and 3.1 points on Composite Unseen.Source: §5.2 · p. 13

6 Sim-to-Real Experiments

Real-world experiments test SMART-Data pretraining and synthetic post-training across three robot platforms and twelve articulated-object tasks. SMART-VLA transfers zero-shot, stays more robust to an unseen robot height, and benefits from increased synthetic-data budgets.

6.1 Experiment Setup

  • Evaluation spans twelve tasks (eight types plus four cross-platform variants), three dual-arm platforms, and fifteen trials per policy; it measures success rate and action-chunk quality.The platforms are RealMan RM75, R1Pro, and AC1; action-chunk quality uses NRMSJ and NMAV.Source: §6.1 · pp. 13–15, Figure 8 · p. 14, Figure 9 · p. 15

6.2 Sim-to-Real Experiment Results

  • SMART-VLA averages 76.7% SR across twelve tasks, 26.7 points above SMART-VLA(w/o S1) (50.0%), and exceeds π0.5 and SMART-VLA(Real) by 17.3 and 7.3 points.It matches or outperforms π0.5 on ten of twelve tasks; the results cover three embodiments.Source: §6.2 · pp. 15–16, Figure 10 · p. 16
  • SMART-VLA records the lowest NRMSJ (0.0581) and NMAV (0.0673); collision-free planning and constrained in-contact motion are proposed as structured priors for stable action generation.Lower NRMSJ and NMAV indicate smoother and more consistent trajectories.Source: §6.2 · p. 16, Table 5 · p. 16
  • At an AC1 deployment height 10 cm above synthesis height, SMART-VLA maintains at least 60% SR on all four tasks, with a 7.5-point average drop versus 19.2 for π0.5 and 45.0 for SMART-VLA(w/o S1).Figure 11 compares success rates at seen and unseen heights.Source: §6.2 · p. 17, Figure 11 · p. 17

6.3 Performance Scaling Experiment

  • Across five tasks, success rises with more demonstrations; SMART-VLA reaches at least 80.0% on all five at 500 episodes, while drawer pulling reaches 40.0% at 200 episodes.Loosely constrained tasks largely saturate by 50 episodes, whereas toaster-lever pressing, drawer pulling, and microwave knob rotation keep improving at larger scales.Source: §6.3 · pp. 17–18, Figure 12 · p. 17
  • Pretraining improves data efficiency: at 50 episodes SMART-VLA reaches 73.3% on R1Pro bucket-handle lifting and 60.0% on toaster-lever pressing, near SMART-VLA(w/o S1) at 500 episodes.SMART-VLA also succeeds on microwave knob rotation with one episode, while SMART-VLA(w/o S1) records no success through ten episodes.Source: Figure 12 · p. 17, §6.3 · p. 18

7 Limitations

The paper identifies synthesis throughput, agentic task generation, and articulated-asset preparation as limitations.

7 Limitations

  • Long-horizon tasks reduce synthesis throughput because planning and execution succeed less often.The proposed remedy is to synthesize shorter subtasks independently and reconnect them through consistent boundary states.Source: §7 · p. 18
  • Agentic task generation is susceptible to LLM hallucinations and depends on the underlying LLM/VLM capabilities.The authors suggest stronger verification and human-in-the-loop checks to reduce such errors.Source: §7 · p. 18
  • Articulated assets require substantial post-processing for geometry separation and articulation annotation.The proposed future direction is jointly generating articulation attributes and manipulation annotations to reduce manual effort.Source: §7 · p. 18

8 Conclusion

SMART combines simulation, agentic task generation, and distributed synthesis to build articulated-manipulation training data. Experiments report improved downstream VLA performance and data efficiency, zero-shot sim-to-real transfer, and positive scaling with task-specific synthetic demonstrations.

8 Conclusion

  • SMART-Sim supports automatic motion generation, diverse robot embodiments, and extensive domain randomization for synthesizing articulated-object manipulation demonstrations.Agentic task generation and distributed synthesis build SMART-Data across robot setups, objects, and manipulation skills.Source: §8 · p. 18
  • Experiments show SMART-Data pretraining improves downstream VLA performance and data efficiency and enables zero-shot sim-to-real transfer.The reported evaluations include simulation benchmarks and real-world tasks.Source: §8 · p. 18
  • Performance scales positively with the number of task-specific synthetic demonstrations used for post-training.The authors present synthetic data as a potential complement to real-world demonstrations for articulated-object manipulation.Source: §8 · p. 18

Contributions

The contribution statements assign work across simulation architecture, data synthesis and annotation, experiments, and project supervision.

Contributions

  • Jicong Ao led the project, designed the simulation architecture, optimized cluster-scale synthesis, and designed tasks and conducted experiments for sim-to-real evaluation.Ao conducted sim-to-real experiments on R1Pro and RM75.Source: Contributions · pp. 18–19
  • Shuhan Jiang developed the simulation framework and distributed synthesis pipeline, and conducted simulation-benchmark and AC1 sim-to-real experiments.Yuling Zhong contributed framework features, synthesized RM75 and R1Pro data, and led RM75 sim-to-real experiments.Source: Contributions · p. 19
  • Yanwen Liu developed domain-randomization modules and conducted initial synthesis trials and preliminary post-training experiments.Liu also resolved technical issues supporting reliable operation.Source: Contributions · p. 19
  • Yuhan Gao provided background-scene assets and conducted articulated-object annotation and task-configuration generation for synthesis.Jiangyuan Zhao and Yang Zhang provided PRTS infrastructure, conducted pretraining, and analyzed post-training.Source: Contributions · p. 19
  • Chenjia Bai advised and supervised the project.Source: Contributions · p. 19

A.1 Task File Illustration

This section documents SMART task configuration and asset grasp-pose annotation, along with a constraint-aware solver for converting grounded manipulation skills into constrained motions. It also points to the sampled parameter ranges used for SMART-Sim domain randomization.

A.1 Task File Illustration

  • The toolbox-opening task file specifies an AC1 robot using its left arm to reach the upper-lid handle and flip the lid upward by revolute rotation.The example labels the task type as revolute_door_open_horizontally and includes a detailed execution description.Source: §A.1 · p. 24

A.2 Asset Annotation

  • The appendix illustrates grasping-pose annotation approaches for both rigid and articulated objects.The subsection identifies this as the annotation processing introduced in Sec. 3.2.Source: §A.2 · p. 26, Figure 13 · p. 27

A.3 Constraint-Aware Motion Solver

  • SMART-Sim turns grounded manipulation skills into constrained motion problems, enforcing end-effector pose constraints at target configurations and joint constraints along the path.It derives a target end-effector pose by applying a transformation for the specified joint displacement or rotation to the annotated action frame.Source: §A.3 · p. 27
  • The solver favors well-conditioned arm configurations, generates skill-constrained trajectories balancing smoothness, efficiency, and path satisfaction, and rejects trajectories violating collision, joint-limit, contact, or configuration checks.Path constraints derive from the action frame and articulation model; the same solver interface supports translation, rotation, and composite skills.Source: §A.3 · p. 28

A.4 Randomization Details

B Dataset Illustration

SMART-Data is illustrated through samples spanning diverse scenes, objects, and manipulation tasks. Task and skill definitions can be reused across hardware configurations, with robot embodiments detailed separately.

B Dataset Illustration

C.1 Robot Platform Details

  • The same task and skill definitions can be reused across hardware configurations through the setup assembly interface.Source: §C.1 · p. 29
  • Franka’s single-arm and dual-arm configurations count as one robot embodiment in the main text.Source: Table 7 · p. 30

C.2 Task Details

The real-world evaluation covers eight articulated-object task types, with task-specific success criteria and randomized initial conditions. The protocol tests policies on real robots, while action chunks are evaluated using success rate and motion-quality metrics.

C.2 Task Details

  • Eight real-world task types cover bucket lifting, drawer pulling, dispenser and toaster pressing, door sliding and closing, knob rotation, and toolbox-lid flipping.Each task has a specified initial state and success criterion; four task types are repeated across platforms.Source: §C.2 · p. 30
  • Object and robot initial configurations vary within small bounded ranges around nominal resets, and lighting is randomized at each environment reset.Source: §C.2 · p. 30

C.3 Training Details

  • Pretraining and post-training hyperparameters are listed in Table 8, with data packing used for pretraining batches.Source: §C.3 · p. 30, Table 8 · p. 31

C.4 Evaluation Protocol Details

  • For AC1, action-quality metrics use 12 arm degrees of freedom, exclude the gripper, and are computed on quantile-normalized actions.Action chunks contain T=50 steps and D=12 action dimensions; the evaluation includes NRMSJ and NMAV.Source: §C.4 · p. 31
  • Policies use all 500 task-specific synthetic demonstrations for sim-to-real evaluation and recursively sampled subsets of 500, 200, 100, 50, 10, and 1 for scaling tests.Each policy is evaluated directly on the real robot, with success measured over 15 independent trials per policy and task.Source: §C.4 · p. 31
Loading 2610.07652v1…