ScIntBench: Physical Interaction Plausibility in 3D Scene Generation

1. Introduction

• Seven evaluation dimensions in existing protocols focus mainly on semantic quality and initialization-time physical plausibility, leaving limited evidence about active manipulation and plausible responses.Static checks include collisions, boundary violations, support correctness, and passive stability under gravity.Source: §1 · p. 2
• A scene that is valid at rest may lack action prerequisites or behave implausibly when manipulated, motivating separate questions about evaluation coverage and interaction-readiness barriers.The paper distinguishes interaction availability from execution reliability.Source: §1 · p. 2
• ScIntBench unifies semantic fidelity, static physical validity, and interaction-time physical reliability, using multi-view evaluation and standardized physics-based interaction tests.The interaction tests assess action executability, post-interaction integrity, and dynamic plausibility.Source: §1 · p. 2
• Strong semantic and static performance did not ensure interaction readiness; failures included missing articulations, inconsistent physical metadata, and post-interaction physical violations.The benchmark evaluates 12 systems across 10 room types.Source: §1 · p. 2
• The benchmark contributes four interaction-based metrics and a systematic evaluation of 12 state-of-the-art systems across ten room types.The evaluation highlights challenges between static scene quality and interaction-time reliability.Source: §1 · p. 2

2. Related Work

• Scene-generation paradigms have progressed from data-driven models and semantic or vision-language planning to agentic workflows and general-purpose coding agents.This diversification motivates unified evaluation across systems with different generation mechanisms and capabilities.Source: §2 · p. 2
• Scene-refinement approaches include physics critics, articulated assets, physical-property estimation, and simulated task feedback, yet often emphasize initialization-time validity or downstream success.The cited examples include SAGE [28], SceneSmith [17], and EmbodiedSmith [20].Source: §2 · p. 3
• Prior evaluation protocols assess semantic fidelity, geometric consistency, and static physical plausibility, while newer benchmarks also consider instruction satisfaction and scene-state preservation.SceneTeract evaluates navigability, reachability, and interaction clearance; SceneSmith and EmbodiedSmith include simulated robotic manipulation focused primarily on downstream task success.Source: §2 · p. 3
• The paper identifies the missing piece as a unified protocol for evaluating interaction availability and physical reliability during and after manipulation across generation paradigms.Table 1 compares representative protocols by semantic-view setting, static physics, active-interaction physics, and generator categories.Source: §2 · p. 3, Table 1 · p. 3

3. ScIntBench

• ScIntBench combines semantic fidelity, static physical validity, and interaction-time physical reliability to assess simulation readiness.Its metric dimensions and performance aspects are summarized in Fig. 1.Source: §3 · p. 3

3.1. Benchmark Construction

• Benchmark instances extend SceneEval prompts across 10 room types with shared interaction specifications defining target objects, actions, required capabilities, success conditions, and physical constraints.An LLM proposes candidate interactions from room descriptions; specifications are established independently of generated scenes.Source: §3.1 · p. 3

3.2. Active Interaction Evaluation

• For each prescribed task, ScIntBench checks the target and native physical capabilities before executing supported actions in a physics simulator.The task set is shared across evaluated methods, enabling consistent tests of interaction availability and physical reliability.Source: §3.2 · p. 4
• Four interaction metrics capture goal success (ISR), graded capability and progress (ICP), post-manipulation violations (PIV), and perceptual dynamic plausibility (DP).ISR uses predefined action-specific criteria; PIV checks scene objects after settling, while DP rates observable behavior during interaction.Source: §3.2 · p. 4

3.3. Semantic and Static Physical Evaluation

• Semantic evaluation uses one top-down and four corner views; static physical evaluation checks collisions, support, out-of-bounds placements, and passive stability under gravity.The five views assess realism, functionality, layout correctness, and scene completeness; static checks establish a reference for interaction-time assessment.Source: §3.3 · p. 4

3.4. Standardized Evaluation Protocol

• The standardized protocol preserves generated geometry, poses, and native properties, without automatically supplementing missing physical properties or articulation mechanisms.It uses Blender for rendering and Isaac Sim for simulation-based assessments, with consistent task procedures and success criteria.Source: §3.4 · pp. 4–5

4.1. Experiment Setup

• The evaluation compares 12 systems across 30 prompts, 10 room types, and 203 shared tasks, using rendering, physics simulation, and semantic, static, and active metrics.The shared task set yields 2,436 scene–interaction evaluation instances.Source: §4.1 · p. 5

4.2. Overall Performance on ScIntBench

• Among five methods with executable interactions, ICP rankings match Func rankings, while ISR follows a similar ordering except for SAGE and Holodeck.The authors suggest that functionally appropriate scene content may be particularly important for enabling physical interaction.Source: §4.2 · p. 5
• Claude Code has the highest ISR but relatively high PIV; SAGE has the lowest PIV despite fewer task completions, and SceneSmith has the highest DP.These metrics capture complementary aspects of interaction readiness.Source: §4.2 · p. 5
• Claude Code leads Real, Func, Lay, ISR, and ICP; Codex ranks second in Func, Lay, ISR, and ICP, but both score lower Comp than SAGE and SceneSmith.Codex and Claude Code require about 28 minutes per scene, substantially less than SAGE and SceneSmith.Source: §4.2 · pp. 5–6
• SAGE pairs COL 0.41% and STAB 98.33% with ISR 11.54%, while 7 of 12 systems cannot complete any prescribed interaction.The comparison shows that strong static physical scores do not guarantee interaction readiness.Source: §4.2 · p. 5, Table 2 · p. 6

4.3. Active Metrics Validation

• Reference scenes completed all 10 matched BEHAVIOR-1K and iGibson tasks, while each evaluated generation method achieved at most five confirmed successes.The matched set contains seven tasks in BEHAVIOR-1K and three in iGibson.Source: §4.3 · p. 7
• A 100-case study compared MLLM-based ICP and DP scores with mean human ratings using Spearman’s ρ and Kendall’s τ.Three annotators independently rated the sampled interaction cases.Source: §4.3 · p. 7

4.4. Interaction Failure Analysis

• Of prescribed scene–interaction pairs, 1,534 could not be executed because targets or native physical prerequisites were missing; SceneSmith and Codex reached 0.47° and 0.11°, respectively, in refrigerator-opening examples, below the required 60°.A chair-pushing interaction displaced two non-target books from a table onto the floor.Source: §4.4 · p. 8

• ScIntBench evaluates semantic quality, static physical plausibility, and active interaction behavior in text-conditioned 3D indoor scenes.Source: §5 · p. 8
• Across 12 scene generation methods, strong semantic and static performance does not necessarily translate into interaction readiness.Source: §5 · p. 8
• Many generated scenes lack capabilities required for physical manipulation, while substantial limitations in interaction success and physical reliability remain.Source: §5 · p. 8
• ScIntBench provides a standardized basis for assessing simulation readiness and informing future advances in interactive 3D scene generation.Source: §5 · p. 8

• The supplementary material details the implementation of evaluated baselines and construction of ScIntBench.Source: Supplementary Material · p. 11
• It provides definitions, scoring criteria, and aggregation procedures for all evaluation metrics, alongside simulation configurations and interaction execution protocols.Source: Supplementary Material · p. 11
• Additional benchmark results and validation of the proposed active metrics are also included.Source: Supplementary Material · p. 11

A. Baseline Implementation Details

• Table 5 summarizes evaluated methods’ training strategies and backbone models, distinguishing task-specific trained generators from methods relying primarily on pretrained foundation models.Source: A. Baseline Implementation Details · p. 11
• SAGE and SceneSmith use two NVIDIA L40S GPUs, while the remaining evaluated methods primarily use one NVIDIA RTX 4090 GPU.Source: A. Baseline Implementation Details · p. 11

B.1. Scene Prompts and Room Types

• Thirty indoor scene descriptions from SceneEval [25] cover 10 room types, producing 360 generated scenes.Source: §B.1 · p. 11
• Figure 4 overviews the 10 room types included in ScIntBench.Source: Figure 4 · p. 11

B.2. Interaction Task Construction

• A two-stage procedure identifies applicable action categories and explicitly mentioned entities, then judges each action–entity pair for semantic applicability.Source: §B.2 · pp. 11–12
• The action library covers opening hinged objects, pushing rigid objects, and probing meaningful accessible horizontal support surfaces.Source: §B.2 · pp. 11–12
• Table 5 distinguishes task-specific scene generation models trained on 3D-FRONT [5] from approaches primarily relying on pretrained foundation models.Source: Table 5 · p. 12
• Task construction inventories explicit entities without inventing contents or implicit components, and evaluates every selected action–entity pair, including inapplicable pairs.Source: §B.2 · p. 12

B.3. Benchmark Example

• The bedroom example pairs a room prompt with a summary of interaction tasks, target objects, action families, and main success criteria.Source: §B.3 · p. 12

C. Metrics Details

• Metric evaluations cover semantic quality, static physical plausibility, and active interaction behavior without adding missing physical capabilities to generated scenes.Source: C. Metrics Details · p. 12

C.1. Semantic Quality Evaluation

• The MLLM-based judge scores realism, functionality, layout, and completeness on a 0–10 scale.These criteria respectively assess visual plausibility, suitability for intended activities, spatial organization, and whether the scene sufficiently fulfills the description.Source: §C.1 · pp. 12–13
• Five standardized Blender views—one top-down and four corner views—are assessed individually alongside the scene description and structured geometric layout.Each image is rendered at 1024 × 1024 resolution.Source: §C.1 · p. 12
• The scoring prompt directs the judge to lower ratings for critical issues such as missing key objects, layout errors, or unrealistic elements.Its guidance includes examples such as missing essential furniture and obstructed or crowded layouts.Source: §C.1 · p. 13

C.2. Static Physical Evaluation

• Static physical plausibility covers collision, out-of-bounds status, support correctness, and passive stability.Collision detection applies a 5-mm contact-filtering procedure; out-of-bounds classification requires fewer than 99% of sampled surface points to intersect the floor; stability uses 120 simulation steps and thresholds of 0.2 m displacement or 8° orientation change.Source: §C.2 · pp. 13–14

C.3. Interaction Success Rate (ISR)

• Interaction Success Rate is the proportion of prescribed tasks meeting predefined numerical success conditions, with missing targets or native prerequisites counted as failures.The same shared task set is used to compare generation methods.Source: §C.3 · p. 14

C.4. Interaction Capability Progress (ICP)

• Interaction Capability Progress uses a six-level ordinal rubric to distinguish target presence, physical interactability, initiated or substantial progress, and task success.The rubric assigns scores of 0, 2, 4, 6, 8, or 10; a shared assessment rubric is used across methods and action categories.Source: §C.4 · p. 14

C.5. Post-Interaction Physical Violation (PIV)

• Post-Interaction Physical Violation measures scene objects with newly appeared or worsened interpenetration, out-of-bounds status, unintended displacement, or lost support after execution.Intended task motion, such as opening a door or translating the target, is not itself a violation, but the moved object remains subject to physical checks.Source: §C.5 · p. 14

C.6. Dynamic Plausibility (DP)

• Dynamic Plausibility scores observed physical behavior from interaction and settling-period video evidence on a fixed 0–10 rubric.The evaluation considers motion and causality, contact and collision, gravity and support, friction and sliding, and rigidity and articulation; task failure alone does not imply low plausibility.Source: §C.6 · pp. 14–15

C.7. Metric Aggregation and Evaluation Coverage

• Across 30 scene descriptions and 12 methods, 2,436 prescribed method–interaction pairs include 902 validly executed interactions and 1,534 unsupported interactions.Post-interaction violation and dynamic-plausibility averages are conditional on validly executed interactions and are accompanied by coverage.Source: §C.7 · p. 15

D.1. Simulation Environment

• Interaction and physical simulation use Isaac Sim 5.1, preserving generated geometry, object identities, spatial arrangements, and native physical properties.The evaluated interaction families are hinged opening, rigid pushing, and surface probing; robot policies are validated on positive-control scenes before application to generated scenes.Source: §D.1 · p. 15, §D.2 · p. 15, §D.3 · p. 15

D.3. Execution Protocol

• Each interaction starts from the original scene state, followed by actuator withdrawal and five seconds of settling; tasks lacking required targets or native capability are marked unsupported.Interaction videos are recorded at 12 FPS for subsequent scoring.Source: §D.3 · p. 15

E. Additional Results

• Figure 6 distinguishes successful, executed-but-unsuccessful, and unsupported interaction outcomes across twelve scene-generation methods.Figure 7 adds qualitative comparisons for living-room and kitchen prompts.Source: E. Additional Results · p. 15, Figure 6 · p. 16

F.1. Validation on Reference Scenes

• Table 7 compares selected interactive reference scenes with scenes generated by five methods using matched tasks and fixed task-specific protocols.The reference sets contain seven planned tasks in BEHAVIOR-1K and three each in ReplicaCAD and iGibson, with results aggregated within each set.Source: §F.1 · p. 16, Table 7 · p. 16

F.2. Human Evaluation

• Participants inspect target and overview interaction videos alongside the intended action, success criteria, and physical measurements before rating ICP and DP.ICP uses a six-level rubric and DP a 0–10 scale; human annotations are used to assess correlation with ScIntBench metric scores.Source: §F.2 · p. 16, Figure 8 · p. 18