PhySGA: A Physics-Space Grounded Agentic Framework for Generalizable and Interactive 3D Composition

1 Introduction

• Interactive composition provides physically grounded environments for embodied models and a testbed for evaluating foundation-model physical intelligence.Unlike static visual question answering, it requires semantic understanding to be translated into executable actions validated by simulation feedback.Source: §1 · p. 2
• Static language and diffusion priors struggle with precise hanging, gravity-driven settling, and hidden containment because these require alignment, physical simulation, or interactive kinematic actions.The section argues that complex target relations must be realized while adhering to physical laws.Source: §1 · p. 2
• The Assembly Agent plans globally and makes metric-grounded coarse adjustments, while the Critic Agent diagnoses simulated contact failures and makes pixel-grounded fine corrections.Action Routers execute physical commands, and the Self-Evolving Skill Library reuses historical interaction experience.Source: §1 · p. 2
• PhySGA formulates composition as interactive assembly in physical space, preserving physical validity while constructing complex target relations.This contrasts with one-shot prediction and visual-prior approaches.Source: §1 · p. 2
• PhySGA significantly outperforms state-of-the-art methods across rigid, articulated, and deformable tasks in Isaac Sim.The authors present it as a scalable solution for complex simulator task construction.Source: §1 · p. 2

2.1 Simulation Scene Generation

• Language-guided scene-generation pipelines primarily predict static coordinates and use simulation as binary verification, whereas PhySGA executes composition step by step in the simulator.Earlier approaches also use procedural rules or learned distributions over static layouts.Source: §2.1 · p. 3

2.2 3D Object Composition

• Visual alignment of static terminal poses does not guarantee physical plausibility and fails for gravity-driven settling and visually occluded containment.PhySGA instead casts composition as closed-loop physical construction validated by physical laws.Source: §2.2 · p. 3

2.3 Evaluating Physical Intelligence

• Executable construction in a simulator is presented as a more rigorous testbed for physical intelligence than passive visual questioning because it validates semantic reasoning through dynamic interaction.PhySGA adopts continuous simulation feedback rather than static predictions.Source: §2.3 · p. 3

3 Method

• PhySGA is introduced as a physics-space grounded agentic framework for interactive 3D composition.The overview is illustrated in Figure 2.Source: §3 · p. 3
• The method section presents the task formulation and overall architecture before introducing the collaborative high-level agents.These topics are assigned to Sections 3.1 and 3.2, respectively.Source: §3 · p. 3
• The method section separately details modular action routing and self-evolving skills.These topics are assigned to Sections 3.3 and 3.4, respectively.Source: §3 · p. 3

3.1.1 Interactive Composition Representation

• Composition begins from a simulator state and reaches the target through iterative transitions that include geometric changes, velocity assignments, and physical simulation.Each adjustment or simulation step is applied through an environment transition function until the final target state is reached.Source: §3.1.1 · pp. 3–4

3.1.2 Feedforward-Feedback Agentic Architecture

• The Assembly Agent plans N steps and makes coarse metric adjustments, while the Critic Agent conditionally refines each step using simulated physical evidence until requirements are satisfied.The validated state returns to the feedforward process, separating global step-wise planning from pixel-grounded correction.Source: §3.1.2 · p. 4, Figure 2 · p. 4

3.1.3 Unified Asset Pipeline

• The unified asset pipeline standardizes semantics, metric geometry, canonical orientation, and physical properties across generated and retrieved rigid, articulated, and deformable assets.It also provides CAD-style orthographic multi-view renderings with metric annotations for spatial reasoning.Source: §3.1.3 · p. 4

3.2.1 Shared State Machine Design

• Both agents cycle through observation, decision, execution, and validation, adding each action and its environmental consequence to memory for the next reasoning round.They continue or loop back according to whether requirements are satisfied.Source: §3.2.1 · p. 5

3.2.2 Assembly Agent

• The Assembly Agent plans a global trajectory of semantic target states and checks, then uses metric-grounded adjustments from orthogonal views for coarse coordination and structural alignment.Each planned step describes a target state and an observable check condition rather than prescribing low-level commands.Source: §3.2.2 · p. 5

3.2.3 Critic Agent

• After simulation, the Critic Agent diagnoses failures and correction aspects, refines the state, and can continue from the new state or roll back to try another correction path.For fine alignment, it maps representative object pixels to target positions across multiple views; the Action Router triangulates those correspondences into simulator commands.Source: §3.2.3 · pp. 5–6

3.3 Action Router

• Three modular routers execute distinct physical commands: geometric transformations, articulated joint movements, and world-space velocities for deformable objects.Separating high-level semantic reasoning from low-level physical details lets the agents focus on planning and spatial alignment.Source: §3.3 · p. 6

3.4 Self-Evolving Skill Library

• Before a new instruction, the library retrieves compatible skills; afterward, it uses interaction memory and the final physical outcome to add or update reusable knowledge.Distilled insights include Assembly planning bottlenecks and the Critic’s root-cause error analyses.Source: §3.4 · p. 6
• The ⊕ operator appends a retrieved skill to the prompt of either agent.Source: §3.4 · p. 6

4.1 Experimental Setup

• The benchmark contains 40 tasks—20 rigid, 10 articulated, and 10 deformable—each tested with three prompt variants for 120 evaluation scenes.Tasks cover physical contact relations, kinematic-joint actuation, and soft-body settling.Source: §4.1 · p. 6
• The study compares PhySGA with SAGE, SceneSmith, and Interact3D using object-level baselines and evaluates all final compositions in Isaac Sim under a unified protocol.The baseline comparison uses GPT-5.5 as the base VLM and gives each method the same 120 language prompts.Source: §4.1 · p. 6
• Evaluation uses physical metrics PR and CR and visual metrics AFS, RFS, and COS, with RFS highlighted as the primary indicator of instructed-relation success.Visual scores are averaged across the 120 scenes; PR and CR measure invalid collisions and successful target contacts, respectively.Source: §4.1 · pp. 6–7

4.2 Comparison with Baselines

• PhySGA consistently produces physically coherent compositions across object categories, including millimeter-level slot insertion and hanging, multi-step articulated interactions, and deformable settling.The qualitative comparison reports floating assets for SAGE and severe visual misalignments after simulation for Interact3D.Source: Figure 3 · p. 7, §4.2 · p. 7
• Table 1 presents the quantitative comparison with existing methods, with best and second-best results marked in bold and underlined.Source: Table 1 · p. 7

4.3 Comparison with Vanilla VLM-based Coding Agents

• The vanilla-agent comparison uses Qwen3.8-Max, Gemini-3.1-Pro-Preview, DeepSeek-V4-Flash-Vision-Exp, and GPT-5.5 with identical tasks and asset instances, focusing on physical and relational metrics.Asset Fidelity Score is omitted because configurations share the same asset instances.Source: §4.3 · p. 8
• Table 2 compares vanilla VLM-based coding agents with PhySGA using the same base VLMs, marking the best and second-best results.Source: Table 2 · p. 8

4.4 Ablation Study

• Removing any single module lowers overall composition quality; the complete PhySGA has the highest CR, RFS, and COS and the lowest PR.The ablations cover global planning, metric adjustment, error attribution, pixel alignment, action routers, and self-evolving skills.Source: Table 3 · p. 8, §4.4 · p. 8

• PhySGA pairs Assembly Agent geometric and kinematic planning with Critic Agent diagnosis of simulated contact dynamics and pixel-grounded fine adjustments.Source: §5 · p. 8
• The framework targets interactive composition of rigid, articulated, and deformable objects, translating agent decisions into simulator commands and reusing skills distilled from prior runs.Source: §5 · p. 8
• Isaac Sim experiments show improved relation fidelity and overall composition quality over existing methods and vanilla VLM-based coding agents.Source: §5 · p. 9