Source-linked AI summary
EmbodiedSmith: Scaling Embodied Data through Recursive Self-Improvement Flywheel in Simulation
Yikai Qin, Yifei Deng, Mingjian Liang, Wenxuan Song, Zepeng Lin, Zhiyi Jiang, Jiajun Fu, Qiao Sun, Huashuo Lei, Xicheng Gong, Jiayi Chen, Han Zhao, Shuanghao Bai, Pengxiang Ding, Pengwei Wang, Haoang Li
TL;DR
Simulation offers scalable, diverse data and reproducible evaluation, but current generation methods have limitations in scene and skill diversity, physical realism, execution success, and complex embodiments. EmbodiedSmith unifies asset, scene, and task generation through recursive scene–task refinement; this improves task-generation success, and downstream policy experiments find that greater data diversity improves generalization.
Problem
Simulation supports scalable data generation and reproducible evaluation, but existing pipelines have limited scene, asset, and skill diversity, physical realism, execution success, and support for complex embodiments.
Source: §1 · pp. 1–2Method
EmbodiedSmith unifies asset, scene, and task generation with autonomous and language-driven customization, using a recursive loop in which scenes anticipate tasks and task generation guides targeted scene edits.
Source: §1 · p. 2Results
The scene–task loop improves task-generation success, including for long-horizon tasks, and downstream policy experiments find that greater data diversity improves generalization.
Source: §1 · pp. 2–3Takeaways & Limitations
Jointly refining simulation environments and executable task plans offers a promising path toward scalable training data and evaluation infrastructure for generalist robot models.
Source: §5 · p. 13
Abstract
from arXiv · showhide
Scaling robotic foundation models requires diverse training data and reliable evaluation environments. Simulation offers a scalable solution, yet existing generation pipelines remain constrained by predefined assets and skills, a disconnect between scene generation and task generation, and limited support for complex embodiments and physics. We introduce EmbodiedSmith, a framework for scalable embodied data generation through recursive self-improvement (RSI). EmbodiedSmith unifies asset, scene, and task generation in a pipeline that supports autonomous creation and language-driven customization. Its core is an agentic refinement loop: scene generation anticipates downstream task requirements, while task generation guides targeted scene edits, allowing scenes and tasks to iteratively improve one another. This joint refinement improves task generation success, including for long-horizon tasks. The framework further supports mobile manipulators, humanoids, and dexterous hands, as well as interactions involving deformable objects and fluids, broadening the range of behaviors and physical phenomena represented in generated data. Together, these capabilities provide a flexible simulation engine for both robot pretraining and evaluation. Extensive experiments validate the quality, diversity, and generation efficiency of the resulting data, while downstream policy experiments demonstrate that increased data diversity improves generalization.
1 Introduction
EmbodiedSmith addresses simulation-generation limits with a unified, customizable pipeline for diverse assets, scenes, tasks, and embodiments. Its scene–task refinement improves generation success, and downstream policy experiments find that greater data diversity improves generalization.
1 Introduction
- Existing simulation pipelines have limited scene, asset, and skill diversity, physical realism, task-execution success, and support for complex embodiments; simulation offers scalable, diverse data and reproducible evaluation.The cited limitations include decoupled scene and task generation, with execution success becoming more challenging for complex or long-horizon tasks.Source: §1 · pp. 1–2
- EmbodiedSmith integrates object, layout, scene, and skill generation into a pipeline supporting autonomous creation and language-driven customization.The framework is introduced as a scalable embodied-data-generation approach through recursive self-improvement.Source: §1 · p. 2
- The framework supports assets and tasks involving deformable objects and fluids, addressing gaps in physical realism and data diversity.Source: §1 · p. 2
- Scene generation anticipates task requirements and task generation makes targeted scene edits, enabling mutual refinement that improves success even for long-horizon tasks.Figure 1 depicts scene generation coupled with task generation and execution, using feedback to refine scenes and replan.Source: §1 · p. 2, Figure 1 · p. 2
- Task generation extends to mobile manipulators, humanoids, and dexterous hands, enabling data involving human-like dexterity.Source: §1 · p. 2
- Greater data diversity improves downstream policy generalization, while experiments validate the quality, diversity, and generation efficiency of EmbodiedSmith's environments and demonstrations.The framework also serves as pretraining data and an evaluator for studying how task, environment, and embodiment diversity affect pretrained models.Source: §1 · p. 3
2 Related Work
Related work spans data-driven, procedural, foundation-model-guided, and agentic scene generation, alongside physically grounded asset creation and robot-demonstration synthesis. The cited gap is reusable generation of feasibility-checked whole-body demonstrations for newly instantiated scene–task combinations.
2 Related Work
- Prior scene-generation approaches range from learned layouts and procedural grammars to language-guided and agentic refinement; Figure 2 depicts a scene pipeline from specifications and configurations to simulation-ready 3D scenes.Source: §2 · p. 3, Figure 2 · p. 4
- Sim-ready asset work extends from articulated structure and kinematics to physical properties, deformable assets, and automated simulator validation.Source: §2 · p. 3
- Humanoid benchmarks and demonstration-generation methods cover locomotion, manipulation, language-conditioned tasks, or adapted source demonstrations, while reusable skill-based generation for new scene–task pairs remains less examined.Source: §2 · p. 3
3 Method
EmbodiedSmith links task-conditioned scene construction, executable task planning, and embodiment-specific execution in one simulation pipeline. Feedback from planning and execution supports iterative scene edits, task revisions, and validation.
3 Method
- The pipeline generates validated robot demonstrations by constructing task-conditioned scenes, grounding tasks into executable skill plans, and adapting execution for humanoid and dexterous embodiments.Scene, task, and embodiment stages are connected in the RSI pipeline.Source: §3 · p. 4
3.1 Environment Construction
- The asset pipeline combines canonicalized, validated open-source assets with generated assets, while physical-asset construction and simulation checks target intended behavior and downstream interaction.The evaluated deformable configuration uses uniform material properties and documents incomplete post-release recovery as a limitation.Source: §3.1 · pp. 4–5
- Scene generation parses task and embodiment context into a typed relational graph, then solves a geometry- and reachability-validated initial scene that is not guaranteed executable.The graph encodes functional zones, fixtures, orientations, support surfaces, and robot approach directions; downstream planning and execution test feasibility.Source: §3.1 · p. 5
- Scene Edit applies schema-constrained additions, replacements, rescaling, or repositioning to adapt a scene to new tasks while preserving unrelated elements and specifying checks for revalidation.Edits use task, embodiment, and generation or execution feedback before replanning.Source: §3.1 · p. 5
3.2 Robot Trajectory Generation
- The Task Plan Agent grounds task references in scene entities and atomic skills, orders them in a Skill DAG, and links requirements to checkable success conditions.The Harness schedules workflow and independent reviews, while simengine performs simulation-based validation and records execution.Source: §3.2 · pp. 5–6
- The Harness builds and compiles candidate configurations, executes them with recorded evidence, and uses programmatic checks and independent review to guide repair and revalidation.A revised draft must pass build checks and validation again before final delivery.Source: §3.2 · p. 6, Figure 3 · p. 7
3.3 Embodiments for Human-level Dexterity
- Embodiment-specific adapters support humanoid execution while preserving reusable task skills, and dexterous-hand execution uses contact-checked poses and explicit finger commands.The dexterous pipeline records per-link contact telemetry and applies phase-wise force gates.Source: §3.3 · pp. 6, 8
4 Experiments
The experiments assess environment generation, robotic demonstration generation, and policy training with diverse demonstrations. Results report strong scene metrics, more reliable task generation, and improved generalization from mixed data.
4.1 Environment Generation
- The environment benchmark compares five baselines on 20 indoor specifications using perceptual and geometry-based metrics.Each method receives three independent generations per scene.Source: §4.1 · p. 8
- CF reaches 7.13 ± 1.48 versus 5.57 ± 1.68 for the strongest baseline, while the method also leads on SL, PP, VQ, COL, OOB, and OPN.NAV is 0.9984 ± 0.0130, comparable to the top baseline's 0.9988 ± 0.0052; SceneSmith has the highest TF score.Source: §4.1 · p. 8
- Generated scenes are denser yet structured, with clear functional regions and preserved circulation space.The comparison describes sparse scenes, unused areas or weak layouts, and locally inconsistent arrangements among baselines.Source: §4.1 · p. 8, Figure 4 · p. 9
4.2 Robotic Demonstration Generation
- Demonstration generation is evaluated across 16 fixed-base and mobile tasks, using repeated agent sessions and simulation seeds.Configurations are evaluated with seeds not used during generation or refinement.Source: §4.2 · p. 9
- In Table 3, EmbodiedSmith outperforms each baseline in ASR on its supported tasks; results cover per-task demonstration generation in fixed-base and mobile manipulation, with mobile performance varying by task.Source: §4.2 · p. 9, Table 3 · p. 11
- Removing execution feedback reduces ASR to almost 0% across task categories, despite lower token consumption.Removing placement validation particularly affects long-horizon fixed-base tasks; mobile-task drops are smaller.Source: §4.2 · p. 10, Table 4 · p. 11
4.3 Generated Data for Policy Training
- Policy experiments use study-room and kitchen tasks with lengths of three to six atomic skills, and compare clean and mixed datasets.The simple setting reuses clean-data layouts; the random setting varies scenes using random seeds.Source: §4.3 · pp. 11–12
- Models trained on mixed data achieve higher average success than clean-trained models, with especially pronounced gains in randomized scenes.Clean-trained models perform well on fixed layouts but drop sharply under randomization, while mixed training combines fixed-layout and randomized demonstrations.Source: §4.3 · pp. 11–12, Table 5 · p. 12
5 Conclusion
The conclusion presents EmbodiedSmith as an RSI framework for scalable embodied data generation. It integrates asset, scene, and task generation, with experiments supporting the quality and efficiency of its generated environments and demonstrations.
5 Conclusion
- EmbodiedSmith integrates asset, scene, and task generation to support diverse tasks, complex embodiments, and physical interactions.Its coupled environment-task loop enables mutual refinement and improves overall generation success rates.Source: §5 · p. 13
- Experiments demonstrate the quality, diversity, and generation efficiency of the resulting environments and demonstrations.The conclusion describes jointly refining environments and executable task plans as a promising path toward scalable training data and evaluation infrastructure.Source: §5 · p. 13
A Abstract
- Appendices B–E provide further details on scene generation, demonstration generation, policy-training data procedures, and robot evaluations.The appendix material includes metrics, task definitions, robot configurations, randomization procedures, and training details.Source: A Abstract · p. 17
B Details of Environment Generation
The scene-generation pipeline uses a self-constructed, simulation-ready asset library to support scene composition. It selects assets according to scene semantics and downstream physical requirements.
B.1 Used Assets Library
- Each baseline retains its native asset source and generation pipeline, with identical input specifications and matched random seeds when supported.Source: §B.1 · p. 17
- A self-constructed asset library serves as the asset index for scene composition.It contains simulation-ready assets with standardized geometric representations and semantic metadata.Source: §B.1 · p. 17
- Standardized geometry and semantic metadata let the pipeline retrieve objects using scene semantics and downstream physical requirements.Source: §B.1 · p. 17
- Given an input specification, the pipeline progressively selects suitable assets, assigns functional roles and spatial relationships, and instantiates them.Source: §B.1 · p. 17
B.2 Evaluation Metrics
The evaluation uses perceptual and geometry-based metrics to assess generated environments. Perceptual scores are assigned from rendered views, while geometry metrics quantify collisions, traversability, boundaries, and openings.
B.2 Evaluation Metrics
- Environment evaluation uses five perceptual metrics and four geometry-based metrics.Source: §B.2 · p. 17
- GPT-5.5 scores rendered scenes from 1 to 10 for content/function, spatial layout, physical plausibility, text fidelity, and visual quality.These assess room content, organization, apparent physical soundness, specification consistency, and rendering quality.Source: §B.2 · p. 17
- Geometry metrics count mesh collisions and out-of-bound objects, measure navigable floor area, and quantify unobstructed door and window sides.Navigability is the fraction of valid floor area in the largest continuously traversable region under robot-width constraints.Source: §B.2 · p. 17
C.1 Task Definitions and Success Criteria
The appendix defines a 16-task evaluation, task-specific completion criteria, supported robot skills, and the task-level tools exposed by the harness.
C.1 Task Definitions and Success Criteria
- The evaluation covers 16 fixed-base and mobile manipulation tasks grouped as Easy and Long horizon, with task-specific completion conditions.Multi-stage tasks require both intermediate events and final states; recall tasks may also require specified observations, occlusions, or intermediate manipulations.Source: §C.1 · pp. 17–18
C.2 Robot Skills
- Parameterized atomic skill nodes specify execution resources, goal parameters, and, when applicable, a target entity.The simulation execution layer’s supported skills are summarized in Table S1.Source: §C.2 · p. 18, Table S1 · p. 18
C.3 Agent Tools
- The harness exposes five task-level tools, with their interfaces summarized in Table 1.Source: §C.3 · p. 18
C.4 Agent Configuration and Evaluation Protocol
The evaluation protocol tests independently generated task plans under bounded feedback-driven refinement, then evaluates frozen candidates across held-out seeds.
C.4 Agent Configuration and Evaluation Protocol
- Each task and scene receives 10 independent agent sessions, with up to 30 refinement rounds for plans and initial placements in the fixed-scene evaluation.Task-success criteria, scene assets, and object layout remain unchanged during this refinement.Source: §C.4 · p. 18
- Plans and placements are frozen before evaluation on 10 shared held-out seeds excluded from generation and refinement.The scene is reset for each trial, and frozen candidates run without further agent-driven refinement or scene edits.Source: §C.4 · pp. 18–19
- Task-enabled randomization can vary initial object and robot poses and, when configured, lighting, producing different trajectories for the same frozen plan.Source: §C.4 · p. 19
C.5 Evaluation Metrics and Ablation Settings
The section defines success-rate and token metrics, then specifies two ablations that isolate feedback and placement validation.
C.5 Evaluation Metrics and Ablation Settings
- ASR averages candidate success rates within each task and then across tasks; Top-5 ASR averages the five highest candidate success rates per task and then across tasks.Candidate success rate is the fraction of successful trials across ten held-out seeds.Source: §C.5 · p. 19
- Token is mean cumulative VLM output tokens per session, and sessions without executable plans count as zero success and remain in token accounting.Source: §C.5 · p. 19
- The ablations disable execution-feedback plan revision or placement validation, while retaining original task-success criteria.Without placement validation, initial placement uses three rendered views and object world coordinates; feedback-driven revision and normal navigation remain enabled.Source: §C.5 · p. 19
C.6 Robot Configurations and Demonstration Recording
The appendix specifies shared robot and control settings, demonstration capture and filtering, and the varied scene prompts used for evaluation.
C.6 Robot Configurations and Demonstration Recording
- Each task uses its specified robot configuration; fixed-base tasks hold the base pose fixed, while mobile tasks allow navigation to a suitable working location.Arm trajectories use CuRobo with scene collision geometry, and scene-generation methods share robot configurations and low-level control interfaces within each task.Source: §C.6 · p. 19
- Demonstrations are recorded in Isaac Sim 6.0.1 at 30Hz physics and rendering, with aligned observations, robot and object states, actions, and execution-stage annotations.Only successful episodes passing trajectory-integrity checks enter the demonstration dataset; failed episodes are retained for diagnosis.Source: §C.6 · p. 19
- The scene-prompt collection spans room and workspace layouts, including study, living-room, bedroom, and kitchen settings with task-relevant objects and navigation space.The prompts used in demonstration-generation evaluation are listed in Table S2, with shared prompts listed once.Source: Table S2 · p. 19, §C.6 · pp. 20–21
- Other tabletop prompts define cup-pairing and cube-counting arrangements, including covered objects and matching spare cubes for the latter.Source: §C.6 · p. 21
- Workspace prompts specify layouts for block-order, button-sequence, mobile cup-cube, and mobile block-memory tasks.Source: §C.6 · pp. 21–22
D Task Randomization for Policy Training and Evaluation
Task randomization controls asset composition, object layouts, and observation and initial conditions through a nested group structure. Execution feedback filters unsuccessful trajectories and escalates resampling from episode conditions to layouts and asset compositions.
D Task Randomization for Policy Training and Evaluation
- Randomization spans asset composition, task-object and distractor layouts, and observation and initial conditions, with activation and perturbation ranges configured for each task.Layout sampling respects clearance, support-region, and reserved manipulation-space constraints; observation changes include lighting, textures, camera poses, and robot-base pose.Source: D Task Randomization for Policy… · p. 22
- Big groups fix asset composition, small groups vary initial layouts, and episodes resample enabled observation conditions and robot-base perturbations.This hierarchy separately controls asset-composition coverage, layout coverage, and episodes per layout, with combinations sampled automatically.Source: D Task Randomization for Policy… · p. 22
- Collection counts only verified, successfully saved trajectories, and failures trigger resampling from episode conditions to layouts and then asset composition.Candidates that fail do not count toward group targets; sampling continues until the collection target is met.Source: D Task Randomization for Policy… · p. 22
- The task set covers ten fixed-base and six mobile-manipulation tasks, grouped into easy and long-horizon tasks.Figure S1 illustrates block-stacking collection with distractor, lighting, and texture randomization enabled.Source: D Task Randomization for Policy… · p. 22, Table S3 · p. 23
E Robot Evaluation Experiment Details
The evaluation appendix specifies ten policy tasks across study-room and kitchen scenes, alongside shared generation constraints and representative execution stages. It also points to collection statistics and model-training hyperparameters.
E Robot Evaluation Experiment Details
- Shared requirements keep each task tied to the scene’s final assets and IDs, the fixed Franka FR3 control channel, and the specified directional reference frame.Tasks must use registered skill interfaces and asset annotations, preserve physical interactions, and be executed and verified before batch collection.Source: E Robot Evaluation Experiment Details · pp. 23–24
- Ten evaluation tasks comprise five study-room and five kitchen tasks, with prompts specifying initial states, action sequences, success conditions, and policy instructions.Representative rollouts show execution stages in temporal order; tabletop distractors are disabled in the illustrations.Source: E Robot Evaluation Experiment Details · p. 23, Table S4 · p. 25, Figure S2 · p. 28
- Kitchen prompts cover sequential block-and-cup placement, bottle righting, and nesting one bowl into another with physical support and stable release.The bowl task requires actual stacking rather than relying on nearby centers or increased height.Source: E Robot Evaluation Experiment Details · p. 26
- Other kitchen prompts specify closing a hinged microwave door to its valid closed state and wiping a tabletop strip with sponge contact and path coverage.The wiping task also requires returning and releasing the sponge at its storage location.Source: E Robot Evaluation Experiment Details · p. 26
- Table S5 summarizes trajectory-collection task statistics, while Table S6 reports training hyperparameters, including 10k warmup steps for π0.5 and 5k for FastWAM.FastWAM retains a 100k-step cosine schedule.Source: Table S5 · p. 27, Table S6 · p. 27