Source-linked AI summary
4DSynth: Controllable Procedural World Synthesis for Dynamic Embodied Simulation
Zehao Qi, Haochen Luo, Jia-Wang Bian, Zeyu Ma, Shuyang Sun
TL;DR
4DSynth addresses the lack of a unified, editable representation for visually diverse, physically interactive, time-changing embodied environments. It converts language, masks, or photographs into geometry-grounded 4D scenes and uses them to generate 4DSynth-Nav. On 333 tasks, both evaluated vision-language models failed the majority of tasks, while the procedural representation made failures reproducible and difficulty axes tunable.
Problem
Existing procedural simulators, language-guided systems, and dynamic generators do not provide one editable representation that keeps geometry, motion, camera state, and simulation state consistent.
Method
4DSynth unifies four scene-realization routes in a shared Stage and WorldState, then plans geometry-grounded actor and camera trajectories and exports physics-ready environments and tasks.
Results
Both vision-language models failed the majority of 333 4DSynth-Nav tasks; Gemini 3.1 Pro achieved 33.3% SR versus 13.2% for Qwen3-VL-30B.
Takeaways & Limitations
The shared procedural representation supports editable, regenerable environments and makes benchmark failures reproducible while allowing task difficulty axes to be varied independently.
Abstract
from arXiv · showhide
Embodied agents need environments that are visually diverse, physically interactive, and changing over time. Procedural simulators can generate large interactive scene collections, and recent 4D generators produce compelling visual dynamics. Combining these properties in one environment, however, still demands extensive manual effort, and the result is rarely editable or controllable enough to reuse at scale. We present 4DSynth, a controllable procedural system that turns a natural-language description, a blueprint mask, or a single photograph into an editable 4D environment with explicit geometry, animated actors, collision-free trajectories, and physics-ready simulation state. Multiple scene routes share one geometry-grounded representation, so the same pipeline handles animation, camera planning, rendering, and task generation. To validate the full pipeline, we construct 4DSynth-Nav, an interactive navigation benchmark generated entirely from 4DSynth's procedural scenes. Two vision-language models evaluated across three difficulty tiers both fail the majority of tasks and stall after early subtasks. The same procedural controllability that produces these environments also makes each failure reproducible and each difficulty axis independently tunable. This paper presents both a controllable generation pipeline and the scalable benchmark it enables, offering a practical foundation for developing and evaluating embodied agents.
I. INTRODUCTION
4DSynth addresses the difficulty of combining visually varied, physically interactive, and time-changing environments in one editable representation. It unifies multiple scene routes with geometry-grounded animation, camera planning, simulation, and task generation, and introduces 4DSynth-Nav to evaluate embodied agents.
- Motivation: Existing procedural, language-guided, and dynamic simulators use separate representations and authoring pipelines, making consistent editable 4D scenes difficult to produce.The required scene must keep geometry, actor motion, camera motion, and simulation state aligned.
- Contributions: 4DSynth provides a unified procedural authoring architecture in which four scene-realization routes feed one shared Stage and provenance-preserving WorldState.This replaces route-specific downstream pipelines with shared representations.
- Contributions: Its geometry-grounded 4D synthesis measures animated assets, plans collision-free actor and camera trajectories, and derives physics-ready OpenUSD output from procedural scene semantics.Explicit geometry preserves object identity, support relations, collision geometry, and procedural parameters through rendering and simulation.
- Benchmark: 4DSynth-Nav contains 333 automatically validated navigation and pick-and-place tasks with animated obstacles, generated from procedural scenes.The benchmark is designed to expose where vision-language agents fail under controlled task conditions.
- Positioning: The system differs from existing simulation platforms and procedural or 4D-generation methods by constructing editable worlds, rather than only providing fixed scenes or visually dynamic representations.Its scope includes geometry, moving actors, camera motion, simulation state, and downstream tasks.
III. METHOD
The method separates semantic control, scene realization, and geometry-grounded 4D synthesis. A schema-validated specification feeds multiple realization routes that converge on a shared Stage, after which animation, observation synthesis, baking, and export use the same scene representation.
- Architecture: A constrained language-model interface converts requested scenes, actors, actions, spatial goals, and camera intent into a schema-validated 4D specification.This separates semantic control from later scene realization and synthesis stages.
- Architecture: Four realization routes—native indoor, native outdoor, layout-conditioned, and single-image real-to-sim—converge on a common geometry-grounded backend.The routes accept different user conditions while sharing downstream processing.
- Shared representation: All routes end in a Stage recording finished geometry, walkability, and contents, while OpenUSD is reserved for the exported simulator scene.Actor synthesis, observation synthesis, and baking operate on this internal Stage.
- Outputs: Baking combines the scene, actor tracks, and camera track into an editable 4D environment used for rendered video, physics, and downstream outputs.The shared representation keeps these products consistent with the synthesized geometry and motion.
- Outdoor route: Native outdoor scenes use a coarse scene to establish provisional motion and viewing support before detailed terrain and population are realized around the relevant region.The final Stage is extracted only after this coarse-to-final materialization process.
A. Controllable Scene Synthesis
4DSynth supports native, prebuilt, text- or mask-conditioned, and single-image scene sources while preserving a shared editable procedural representation. Scene realization validates geometry, supports reproducibility, and uses camera-conditioned finalization for native outdoor environments.
- Scene sources: The 4D backbone maps different scene sources to the same Stage, allowing native and prebuilt scenes to share downstream processing.Prebuilt scenes retain their geometry while the system re-extracts and validates the Stage before running actor, navigation, camera, and rendering stages.
- Native generation: Native generation creates requested indoor rooms or homes and outdoor scenes from eight terrain biomes, while recording the random seed for regeneration.Continuous parameters remain with the generator rather than being fixed by the category request.
- Layout-conditioned synthesis: Text and masks provide complementary control: text specifies scene content while masks specify spatial layout and object count.Both are normalized into a Canonical Layout before structure construction, placement, completion, and validation.
- Real-to-sim compilation: Single-image compilation combines metric depth, segmentation, object-centric reconstruction, and procedural grounding to produce an editable scene supported by image evidence.The route stores metric boxes, footprints, support relations, facing directions, boundaries, and openings in a PerceivedScene.
- Controllability: Compiled scenes remain parametric because appearance alignment selects among procedural materials, while bounded schema-validated choices and immutable artifacts support auditing and regeneration.Geometry is settled conservatively using observed footprints, support relations, and mesh-level collision constraints.
- Outdoor finalization: Native outdoor finalization uses provisional actor routes and camera views to anchor detailed terrain, population, and ground cover, then repairs both against final geometry.Final motion may detour around newly realized terrain or vegetation but must remain collision-free within an actor-sized connected component.
B. 4D Animation on a Unified Stage
4DSynth distills evaluated scenes into a unified Stage that supports actor-scale walkability, collision-aware motion planning, multi-agent coordination, and baked animation. The same geometry and terrain representation underlies grounding, collision tests, camera validation, and task generation.
- Stage extraction: Stage extraction creates a metric support grid, walkability set, object inventory, and collision and visibility geometry for both indoor floors and outdoor terrain.Cells enter the walkable set only when support is stable and sufficiently level, sufficiently dry, and free of body-occupying solid geometry.
- Stage extraction: For actor i with measured body radius r_i, planning uses F_i = Erode(W, r_i), and scenes without actor-scale physical support are rejected.The Stage also provides terrain height and evaluated geometry for grounding, collision tests, camera validation, and task generation.
- Actor assets: Imported character assets are measured for stature, collision radius, visual envelope, sole offset, and natural ground speed, and these values drive clearance, erosion, separation, grounding, playback, and framing.Clips with negligible root displacement are treated as in-place actions.
- Trajectory planning: Actors plan routes inside the physically walkable region intersected with F_i, using A∗ between reachable anchors or long corridors within connected free-space components.Simplification and smoothing are accepted only when every segment remains inside the feasible region, and paths use measured clip speed.
- Multi-agent coordination: Multiple actors are coordinated through reactive separation while preserving feasible regions and endpoints, after which tracks are baked with aligned headings, bounded turn rates, repeated cycles, and grounded feet.Stationary actors become obstacles, and narrow corridors enforce single-file order.
C. Geometry-Valid Observation Synthesis
4DSynth separates camera intent from geometric feasibility, then validates visibility and collision constraints against the synthesized Stage. Its layout and image compilation routes preserve explicit scene structure for downstream observation synthesis.
- Camera intent: Camera intent uses nine composable primitives interpreted in target-relative spherical coordinates, while framing distance follows subject extent and field of view.The framing rule is not tuned per scene.
- Geometric feasibility: Two planning modes keep the camera inside the Stage’s valid domain and outside measured actor bodies.Exact planning projects invalid poses to feasible ones and rechecks framing; fast planning evaluates 11 deterministic whole-trajectory transforms.
- Scene compilation: Text or masks are normalized into a Canonical Layout IR that records rooms, openings, object footprints, categories, and support chains.The shared backend then constructs structure, places assets, and completes compatible content.
- Scene compilation: Single-image compilation extracts metric and semantic evidence, realizes sized native assets, freezes geometry, and aligns materials to the source view.The output remains an editable procedural scene rather than a purely visual reconstruction.
- Visibility: Environment occlusion is a soft cost, so planning prefers clear views without detouring around every foreground object.Fast planning requires fully framed, unoccluded target views in at least 80% of frames; exact planning defaults to 90% after first reveal.
D. Physics-Ready Export to Simulation
4DSynth exports one finished procedural scene to Isaac Sim and derives physical properties from the same generative representation that produces appearance. Animated humans are represented as verified kinematic solid obstacles.
- OpenUSD export: The exporter writes the finished 4D scene once to an OpenUSD stage and validates content through mesh and prim counts.Transferred light parameters are repaired, with additional sun lighting and smoothed terrain normals for outdoor scenes.
- Physical properties: Procedural material identity maps to density and friction, while object-level physical values are area-weighted aggregates over material regions.This ties physical properties to the generative material representation.
- Physical properties: Every mesh receives a static collider, and movable objects additionally receive convex-hull rigid-body collision with density-derived mass.No separate physics labeling is required because annotations are read from procedural parameters.
- Animated actors: Animated humans enter Isaac Sim as measured-radius and measured-height capsules following generated tracks as kinematic solid obstacles.They push and block but are not pushed; headless verification checks loading, settling, and planned displacement playback.
IV. EXPERIMENTS
The experiments qualitatively assess scene-route range and editability, then show layout compilation, photograph-based real-to-sim reconstruction, and geometry-consistent 4D environments. The comparisons illustrate system behavior rather than provide a metric ranking.
- Qualitative scene synthesis: Qualitative comparisons evaluate the range and editability of synthesized procedural scene routes, not a metric ranking.All shown examples are rendered directly from synthesized procedural scenes.
- Layout-conditioned synthesis: Blueprint-mask compilation preserves recognizable room topology, openings, and furniture arrangement while procedural factories supply geometry, materials, and lighting.Individual assets and room surfaces remain editable afterward.
- Real-to-sim compilation: Single-image compilation preserves more room envelope, object inventory, and support structure as separately editable procedural instances than the compared VIGA reconstructions.Its appearance alignment also matches wall, floor, and furniture materials to the source view.
- 4D environments: Synthesized 4D scenes show collision-free actor trajectories, validated in-place actions, and camera framing maintained against final geometry.The physics-enabled scenes underlying the navigation benchmark come from the same pipeline without per-scene mesh authoring.
B. Benchmark Construction
4DSynth-Nav is generated from physics-enabled procedural indoor scenes through automated probing, constrained target selection, and validation. Its task construction enforces reachability, visibility, clearance, and object-selection invariants.
- Benchmark composition: 4DSynth-Nav uses 122 physics-enabled indoor scenes containing procedural household objects and animated characters as dynamic obstacles.Character motion includes solo runs, two runners, run–dance, and run–jump trajectories.
- Automated task generation: Scene probing extracts object paths, bounding boxes, floor heights, support relations, and per-object reachability using Isaac Sim and PhysX flood fill.The flood fill uses a 0.25 m grid, two sweep heights, and a 0.40 m-radius agent.
- Benchmark composition: Table I presents the resulting 4DSynth-Nav benchmark composition.The supplied table passage identifies the benchmark-composition table but provides no row values.
- Automated task generation: Target selection rejects sealed-cabinet pickups, prefers unique semantic classes, separates pickup supports from phase targets, and caps floor pickups at 30%.Targets must also satisfy validated reachability radii of 1.0 m for pickups and 1.5 m for destinations.
- Task validation: Validation checks room-boundary geometry, eight-directional sphere sweeps at two heights, initial-view conditions, forward clearance, and flood-fill reachability.The supplied conditions distinguish L2 targets inside the initial frustum from L1/L3 targets outside it.
C. Dataset Statistics
4DSynth-Nav organizes automatically validated navigation and pick-and-place tasks into three tiers that vary subtask count and initial target visibility, using visual feedback and standardized episode metrics to evaluate agents.
- Task tiers: The benchmark has three tiers, L1–L3, controlled by number of subtasks and whether the initial target is visible.The one-phase, initially visible combination is omitted because it exercises neither difficulty factor.
- Task tiers: L1 requires exploration to find one target outside the initial camera frustum, whereas L2 uses two phases with the first target initially visible.L3 combines L2’s two-phase structure with L1’s initially hidden target.
- Agent protocol: Agents receive up to three recent first-person images, a natural-language instruction, and action feedback before emitting plans of at most five discrete actions.Episodes terminate after action or VLM-call limits, or when the task ends.
- Scene generation: The benchmark’s procedural scenes include blueprint-conditioned multi-room layouts, single-image reconstructions, and synthesized 4D environments with editable geometry, actors, and camera trajectories.These routes preserve authored layouts or appearance while producing downstream simulation-ready scenes.
- Metrics: Evaluation reports Success Rate, Subtask Progress, Goal Distance, Collisions, and Steps to capture completion, partial progress, geometric error, contacts, and efficiency.SR and SP are percentages, GD is measured in meters, and collisions and steps are per-episode means.
E. Results
The benchmark distinguishes model capability while showing that both evaluated vision-language models remain far from solving dynamic embodied navigation, especially on harder multi-stage tasks.
- Overall results: Gemini 3.1 Pro reaches 33.3% SR and 40.7% SP, compared with Qwen3-VL-30B’s 13.2% SR and 22.4% SP.Gemini also reduces final goal distance from 4.27 m to 2.68 m.
- Tier results: On L1, Gemini reaches 57.5% SR versus Qwen’s 16.8%, while using fewer steps and ending closer to the goal.The reported values are 79.5 versus 120.2 steps and 2.41 m versus 4.74 m final distance.
- Tier results: On L3, SR rises from 4.5% for Qwen to 18.2% for Gemini, but both models still fail most tasks.The paper treats the comparison diagnostically because task types and model training differ.
- Failure pattern: Both models make partial progress and then stall, with subtask progress exceeding full-task success on the two-phase tiers.This gap is reported for both L2 and L3.
F. Trajectory Example
Trajectory examples show successful completion of a two-goal task and failure after a blind-start first phase, while the shared procedural pipeline makes such behaviors reproducible in editable scenes.
- Successful trajectory: Qwen3-VL-30B completes the L2 two-goal example in 101 steps after reaching Goal 1 within 40 steps and ending 1.3 m from Goal 2.The agent turns toward the second goal after entering Goal 1’s acceptance region.
- Failed trajectory: In the L3 failure, the agent reaches Goal 1 but loops away from Goal 2 and ends 8.0 m from it after 133 steps.The episode records SP = 0.5 despite the phase-1 success.
- Interpretation: The paired episodes demonstrate multi-stage capability alongside errors in judging task completion status.The failure persists even after the harness advances the objective prompt to the second phase.
- Benchmark connection: 4DSynth ties geometry, actor and camera trajectories, rendering, physics, and task generation to shared representations, enabling reproducible trajectory-based evaluation.4DSynth-Nav contains 333 tasks with static and dynamic obstacles, and failure analyses include looping trajectories and partial progress.