Source-linked AI summary
Text2Sim: Agentic Physics-Based Simulation Generation with Distilled Expertise
Xiaoyu Xiong, Tsun-Hsuan Wang, Yi-Ling Qiao, Tao Du, Minchen Li
TL;DR
Authoring dynamic physical scenes is labor-intensive because geometry, behavior, simulation, and rendering must be coordinated. Text2Sim turns text requests into editable simulations using specialized agents and retrieved Debug Cards. It achieves the highest aggregate physical, visual, and human-preference scores among four baselines.
Problem
Authoring dynamic physical scenes is labor-intensive because geometry, physical behavior, simulation, and rendering require coordinated design and debugging.
Source: §1 · p. 1Method
Text2Sim turns text-only requests into editable Genesis simulations through a Planner, specialized Writers, an independent Critic, and role-retrieved Debug Cards.
Source: §1 · p. 2Results
Text2Sim achieves the highest aggregate physical, visual, and human-preference scores among four baselines; physical 95.32 and visual 70.23 exceed baseline means by 13.52 and 11.50 points.
Source: §1 · p. 2, §7 · p. 15Takeaways & Limitations
Explicit programs and recorded dynamics provide a basis for VLM training datasets and simulator benchmarks, while multimodal inputs let sketches and existing assets guide scene creation.
Source: §7 · p. 15Its physical scope is bounded by the simulator’s supported material models, solvers, and contact mechanisms, while repeated agent calls, simulation, and rendering impose costs that rise rapidly with scene scale.
Source: §7 · p. 15
Abstract
from arXiv · showhide
Creating diverse physical simulations remains labor-intensive because assets, layout, physical parameters, motion, control, and rendering must be designed and debugged jointly. We present Text2Sim, a simulation-specialized agentic pipeline that converts a text-only request into an executable, editable dynamic case. Built on Genesis, Text2Sim uses a hierarchical agentic structure that combines a Planner with specialized Writers, asset-generation tools, and an independent Critic. Compact skills (Debug Cards) distilled from graphics demonstrations provide role-specific physical guidance for execution-based repair. We evaluate physical quality, visual quality, and human preference on 42 held-out prompts spanning rigid, articulated, deformable, and cloth phenomena, with a paper-level split between experience construction and evaluation. We design automatic physical and visual scorers to evaluate the quality of the results, and Text2Sim achieves higher scores than all four state-of-the-art baselines on both metrics. In blinded user studies with these baselines, significantly more participants prefer Text2Sim than prefer the baselines, which is consistent with the results from our automatic scorers. The pipeline also supports a broad range of downstream applications; we select dataset construction and extension to multimodal input as two representative examples. We will release the code, the Debug Card library, and a dataset of generated cases, each pairing the text prompt and rendered video with the executable program, assets, physical parameters, controls, and recorded states.
1 Introduction
Dynamic physical-scene authoring is labor-intensive because geometry, physical behavior, simulation, and rendering must be coordinated. Text2Sim combines specialized agents with retrieved simulation experience to generate and repair editable simulations from text.
Pipeline
- A Planner coordinates specialized Writers, while an independent Critic uses execution evidence to route targeted revisions to the responsible Writer.Text2Sim converts text-only requests into executable, editable simulations without requiring a user-provided asset library.Source: §1 · p. 2
Debug Cards
- Debug Cards distilled from 99 graphics demonstrations provide role-specific guidance for connecting observed failures to physical checks and repairs.The Planner retrieves cards based on the event, execution evidence, and receiving agent’s role.Source: §1 · p. 2
Evaluation
- 95.32 physical and 70.23 visual scores exceed the four-baseline means by 13.52 and 11.50 points, respectively.The evaluation also reports higher aggregate human preference for Text2Sim than for each baseline.Source: §1 · p. 2
- The evaluation uses 42 held-out prompts across rigid, deformable, and cloth cases, with a paper-level split between Debug Card construction and evaluation.The prompts come from published graphics demonstrations and cases designed by human simulation experts.Source: §1 · p. 2
Applications
- Generated cases pair prompts and rendered video with executable programs, assets, physical parameters, controls, and recorded states for dataset construction and multimodal extensions.Demonstrations include sketches specifying soft-letter shapes and order, and a supplied cloth mesh reused for curtains.Source: §1 · p. 2
2 Related Work
Prior work generates assets, layouts, task-specific environments, or physical videos, while Text2Sim focuses on authoring complete, editable dynamic scenes. It combines agentic construction and repair with reusable physical experience.
Physics-based simulation
- Simulation benchmarks and neural-simulator training both require assembled, checked scenes; Text2Sim generates editable programs and physical records to support these needs.Scene preparation entails specifying geometry, materials, initial and boundary conditions, and controls.Source: §2 · pp. 2–3
Video generation
- Unlike end-to-end video generation, Text2Sim authors executable physical scenes that can be inspected, modified, and rerun alongside their parameters and controls.The passage identifies robotics and engineering environments with strict physical constraints as compatible applications.Source: §2 · p. 3
Physical scene generation
- Prior scene-generation systems emphasize geometry, initial layouts, or task-specific representations, whereas Text2Sim jointly authors assets, physical parameters, and temporal behavior.The comparison includes simulation-ready layouts and specialized robotics, game, brick-assembly, and origami systems.Source: §2 · p. 3
Agentic systems
- Software execution alone cannot verify physical stability or the intended visual event; Text2Sim adds quantitative and qualitative execution records and reusable Debug Cards to agentic generation.The related work contrasts its approach with systems using execution feedback, local program search, and multimodal feedback in physics-engine loops.Source: §2 · pp. 3–4
3 Design Goals and Simulation Foundation
Text2Sim targets authors who can describe a physical event but would otherwise assemble and debug its assets, simulation code, and presentation manually. Its design goals combine editable output, broad physical coverage, coordinated agents, and reusable experience.
Executable and editable output
- The output exposes geometry, material parameters, motion, control, and rendering in an inspectable, revisable simulation program.Preserving the program and execution evidence supports reuse in dataset applications.Source: §3 · p. 4
Coverage across general solid scenes
- One authoring interface targets articulated rigid bodies, volumetric deformables, and cloth, including supported contacts across these families.The design-goal examples include rigid impact, soft-block compression and shear, and cloth–rigid contact.Source: §3 · p. 4
Coordinated agent cooperation
- A Planner coordinates specialized Writers and an independent Critic so bounded tasks remain consistent and execution feedback can guide targeted revisions.The goal passage links the complete coordinated design to the main baseline comparisons.Source: §3 · p. 4
Reusable physical experience
- Reusable physical skills distilled from SIGGRAPH demonstrations are encoded as Debug Cards and retrieved to guide physical setup and repair.The ablation study is cited as supporting the value of incorporating this experience.Source: §3 · p. 4
Simulation foundation
- Genesis supplies articulated-rigid, volumetric-deformable, and cloth simulation, two-way IPC contact coupling, and path-traced rendering, while Meshy1 generates meshes.Text2Sim addresses the authoring decisions that configure these capabilities: geometry, physical parameters, controls, and presentation.Source: §3 · p. 4
Writer responsibilities
- Fig. 3 assigns scene settings, body geometry and properties, actions and logging, and camera and lighting to four distinct Writers.The before-and-after impact frames share a crop and label simulation times in seconds.Source: Figure 3 · p. 4
4 Method
Text2Sim turns text requests into executable simulations through coordinated agents that build, inspect, and revise the case. The Planner routes tasks and execution feedback to specialized agents, with Debug Cards providing targeted simulation guidance.
4.1 Problem and Overview
- Text2Sim maps a text request to an executable simulation program, assets, and execution evidence through a generation-and-repair loop.The Planner coordinates the agents, and an independent Critic reviews the resulting evidence.Source: §4.1 · p. 4
4.2 Multi-Agent Structure
- A Planner coordinates specialized Writers and an independent Critic, linking execution evidence to the agent responsible for each revision.Explicit asset and control interfaces let local revisions preserve the rest of the scene.Source: §4.2 · pp. 4–5
4.2.1 Planner.
- The Planner decomposes each request, dispatches bounded tasks, mediates external resources, and decides whether checks pass or the attempt budget is exhausted.It forwards execution code, numerical records, and visualizations to the Critic, then uses the diagnosis to dispatch targeted revisions.Source: §4.2.1 · p. 5
4.2.2 Writers.
- Four Writers separately author the stage, movable bodies, temporal actions, and rendering, with fixed interfaces and local repair ownership.A deterministic integrator assembles their modules and checks required interfaces.Source: §4.2.2 · p. 5
4.2.3 Critic.
- The Critic jointly reviews execution logs, numerical and event records, sampled frames, and video, then identifies failures and recommends a responsible module and revision.This separates generation from judgment and avoids accepting a case solely because its code runs.Source: §4.2.3 · pp. 5–6
4.2.4 Assets.
- An independent Asset branch generates and validates required geometry or articulated mechanisms before exposing them to Writers through a shared manifest.Mesh checks include topology, orientation, scale, and simulator compatibility; articulated assets are structurally checked and previewed.Source: §4.2.4 · p. 6
4.3 Debug Cards
- Debug Cards distill reusable simulation diagnoses and repairs from graphics demonstrations, matching symptoms with checks and repair directions under stated use conditions.Cases are generated from demonstrations, evaluated against execution traces and rendered results, then iteratively regenerated as cards are refined or consolidated.Source: §4.3 · pp. 6–7
- The Planner selects role- and task-relevant cards using the plan and execution evidence, then attaches them to downstream requests rather than letting agents search independently.This keeps guidance focused and its use auditable.Source: §4.3 · p. 7
5 Evaluation and Results
Across 42 held-out tasks, Text2Sim leads the baselines in physical quality, visual quality, and user preference. The evaluation also examines task-level behavior, the role of Debug Cards, and comparison with GPT-6-Astra.
5.1 Experimental Setup
- The benchmark uses 42 held-out prompts—10 rigid, 18 deformable, and 14 cloth—with source-paper separation between Debug Card construction and evaluation.Four baselines receive the same prompts and Genesis execution interface, with up to 30 simulation attempts per case.Source: §5.1 · p. 8
5.2 Evaluation Metrics
- Physical scoring audits code, runtime records, and rendered evidence, while visual scoring uses only the prompt and output video; blinded A/B judgments add human preference.The physical rubric covers Scene, Body, Action, and Render; 60 participants assessed 42 tasks in the user study.Source: §5.2 · p. 8
5.3 Main Results
- 95.32 physical (vs Claude-Opus-5 88.60) and 70.23 visual (vs GPT-5.6-Sol 59.89): Text2Sim leads the strongest baselines on both metrics.The physical margin is 6.72 points and the visual margin is 10.34 points.Source: §5.3 · pp. 9, 11
- In the wheel-crawler and dolphin cases, Text2Sim completes the requested chain motion and dolphin passage, while baseline examples include missed releases, partial transits, and shape-key animation.Text2Sim scores 95.74 physically on the wheel-crawler and 98.13 on the dolphin task.Source: Figure 5 · p. 9, §5.3 · p. 10, Figure 6 · p. 10
- 96.16 on the cloth task, Text2Sim transports objects with the displaced cloth; GPT-5.6-Sol produces no completed video, and Code2Worlds leaves tableware behind and shows penetration.The visual comparison also notes that Text2Sim keeps the cloth, tabletop, and transported objects distinguishable through the pull.Source: §5.3 · pp. 10–11, Figure 7 · p. 11
- 61.41 overall preference puts Text2Sim 7.24 points above Claude-Opus-5 (54.17), the strongest baseline, with GPT-5.6-Sol fixed at 50.Text2Sim also leads the other aggregate user-study dimensions: task faithfulness, physical plausibility, and visual quality.Source: §5.3 · p. 12
5.4 Ablation Study
- With Debug Cards, Text2Sim rises from 91.34 to 95.32 physically and from 63.50 to 70.23 visually—the largest gains among the four tested pipelines.The jointed-net example describes repairs to topology, initialization clearance, strand slack, and hinge resistance; peak center sag increases from 0.427 to 0.661 m.Source: Figure 8 · p. 12, §5.4 · p. 13
5.5 Comparison with Newly Released GPT-6-Astra
- Text2Sim leads GPT-6-Astra by 5.86 physical and 9.59 visual points on the same 42 prompts, although GPT-6-Astra outperforms GPT-5.6-Sol.In the rigid tumbler example, Text2Sim shows more coherent bulk motion and more realistic glass and lighting.Source: Figure 9 · p. 12, §5.5 · pp. 13–14, Figure 10 · p. 13
6 Applications
Text2Sim’s executable scenes expose aligned prompts, videos, programs, assets, parameters, controls, and states for dataset and benchmark construction. The same authoring workflow also accepts sketches and supplied assets as additional inputs.
6.1 Dataset Construction
- Text2Sim can batch-generate aligned description–video–dynamical representation triples, pairing each request and rendered video with its program, assets, physical parameters, controls, and recorded states.After checks for prompt fulfillment and simulation validity, these outputs can be curated as training and evaluation data linking visible motion to physical structure.Source: §6.1 · p. 14
- The planned release covers 141 generated cases, including the 42 evaluation cases, with executable modules, assets, controls, recorded states, and Critic scores.The passage says these artifacts cover application needs, so training sets, question sets, and benchmarks require selection and annotation rather than new simulation authoring.Source: §6.1 · p. 14
- Recorded states can support annotations of trajectories and event order, questions about observed or subsequent states, and execution-record checks of those annotations.Editable programs also permit controlled variants by changing material or actuation, or by changing rendering while retaining a fixed trajectory.Source: §6.1 · p. 14
- Text2Sim offers batch generation and repair of editable scene programs as a way to expand candidate simulator tests, reducing repeated expert scene-design and debugging work.A benchmark case would document geometry, materials, conditions, controls, duration, observables, and a compatible validated reference.Source: §6.1 · pp. 14–15
6.2 Extension to Multimodal Input
- The shared construction, execution, critique, and repair workflow accepts an image or digital asset alongside text to specify object shape, identity, or appearance.The examples use a handwritten sketch for letter shapes and order, and a supplied textured cloth asset.Source: §6.2 · p. 15
- The sketch-conditioned letters retain the input glyph forms as they bend, collide, scatter, and settle, while the supplied curtain asset retains its ACM SIGGRAPH graphic through folding and motion.For the curtains, the agent supplies the surrounding window, support structure, and actuation.Source: §6.2 · p. 15, Figure 12 · p. 15
7 Conclusions
The conclusion presents Text2Sim as a pipeline for turning text into editable simulations, with coordinated agents and Debug Cards supporting construction and repair. It reports leading aggregate scores and identifies simulator scope and scaling costs as practical limitations.
7 Conclusions
- Text2Sim unifies asset preparation, physical setup, control, and rendering in executable, editable simulations, coordinated by a Planner, specialized Writers, an independent Critic, and retrieved Debug Cards.The conclusion reports the highest aggregate physical, visual, and human-preference scores against four state-of-the-art baselines.Source: §7 · p. 15
- Explicit programs and recorded dynamics provide a basis for VLM training datasets and simulator benchmarks, while multimodal input lets sketches and existing assets guide scene creation.The conclusion frames these applications as supporting creation, adaptation, and reuse of physical scenes.Source: §7 · p. 15
- The system’s physical scope is bounded by the underlying simulator’s supported material models, solvers, and contact mechanisms, while repeated agent, simulation, and rendering costs rise rapidly as scenes scale.These are the two practical limitations identified for the current system.Source: §7 · p. 15