PaperlayerView exact revision on arXiv

Source-linked AI summary

InfiniBench: Infinite Benchmarking for Visual Spatial Reasoning with Customizable Scene Complexity

Haoming Wang, Qiyao Xue, Wei Gao

arXiv:2511.18200v2cs.CV

TL;DR

VLM spatial reasoning is difficult to evaluate across complex scenes because existing benchmarks lack scalable, fine-grained control over scene properties and failure conditions. InfiniBench generates customizable 3D scenes and informative videos through agentic constraint refinement, cluster-based layout optimization, and task-aware camera trajectories, outperforming prior generation methods in complex settings and supporting diagnostic spatial-reasoning benchmarks.

  • Problem

    Existing benchmarks lack the diversity, scalability, and fine-grained customizability needed to isolate VLM spatial-reasoning failures under distinct scene-complexity conditions.

    Source: §1 · p. 2

  • Method

    InfiniBench translates natural-language scene descriptions into physically plausible 3D scenes and VLM videos using iterative constraint refinement, cluster-based layout optimization, and task-aware camera trajectories.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 4

  • Results

    InfiniBench preserves prompt fidelity and physical plausibility while generating complex, high-density scenes where other methods fail, and supports benchmarks for measurement, perspective-taking, and spatiotemporal tracking.

    Source: §1 · p. 3, §4.2 · p. 7

  • Takeaways & Limitations

    Parameterized benchmarks enable fine-grained analysis of VLM spatial-reasoning failures and can support spatially grounded training or prompt tuning.

    Source: §1 · p. 3, §6 · p. 9

    A single constraint-generation pass can produce logically conflicting or physically impossible requirements, so InfiniBench uses iterative refinement.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Modern vision-language models (VLMs) are expected to have abilities of spatial reasoning with diverse scene complexities, but evaluating such abilities is difficult due to the lack of benchmarks that are not only diverse and scalable but also fully customizable. Existing benchmarks offer limited customizability over the scene complexity and are incapable of isolating and analyzing specific VLM failure modes under distinct spatial conditions. To address this gap, instead of individually presenting benchmarks for different scene complexities, in this paper we present InfiniBench, a fully automated, customizable and user-friendly benchmark generator that can synthesize a theoretically infinite variety of 3D scenes with parameterized control on scene complexity. InfiniBench uniquely translates scene descriptions in natural language into photo-realistic videos with complex and physically plausible 3D layouts. This is achieved through three key innovations: 1) a LLM-based agentic framework that iteratively refines procedural scene constraints from scene descriptions; 2) a flexible cluster-based layout optimizer that generates dense and cluttered scenes previously intractable for procedural methods; and 3) a task-aware camera trajectory optimization method that renders scenes into videos with full object coverage as VLM input. Experiments demonstrate that InfiniBench outperforms state-of-the-art procedural and LLM-based 3D generation methods in prompt fidelity and physical plausibility, especially in high-complexity scenarios. We further showcased the usefulness of InfiniBench, by generating benchmarks for representative spatial reasoning tasks including measurement, perspective-taking and spatiotemporal tracking.

1. Introduction

InfiniBench addresses the difficulty of evaluating VLM spatial reasoning in complex scenes by generating customizable, physically plausible 3D benchmarks from natural-language descriptions. Its agentic constraints, cluster-based optimization, and task-aware camera trajectories support both scalable scene generation and diagnostic analysis of VLM failures.

Motivation

  • Current VLMs struggle to bridge 2D visual perception and 3D spatial reasoning, particularly when compositional, relational, and observational complexity increase.These dimensions respectively concern scene content, object relationships, and viewpoint-dependent visibility.Source: §1 · p. 2

Contribution

  • InfiniBench generates theoretically unlimited 3D scenes from user-specified scene-complexity constraints, translating natural-language descriptions into physically plausible layouts.Its design combines procedural generation’s scalability and control with natural language’s expressive power.Source: §1 · p. 2

Method

  • InfiniBench combines an LLM agent that refines scene constraints, a cluster-based layout optimizer, and camera trajectory optimization for informative VLM videos.The optimizer supports dense scenes, while the camera method seeks clear coverage of task-relevant objects and relationships.Source: §1 · pp. 2–3

Results

  • Its evaluations show that InfiniBench preserves prompt fidelity and physical plausibility while generating complex, high-density scenes where procedural and LLM-based methods fail.The comparison spans scenes with different complexity levels and both procedural and LLM-based generation methods.Source: §1 · p. 3

Applications

  • Customizable benchmarks enable fine-grained analysis of VLM failures in measurement, perspective-taking, and spatiotemporal tracking tasks.The paper also identifies customizable benchmarking as a route toward spatially grounded training or prompt tuning.Source: §1 · p. 3

2. Related Work and Motivation

Prior benchmarks and scene-generation methods provide insufficient control, realism, physical coherence, or scalability for analyzing spatial reasoning under distinct complexity conditions. InfiniBench responds with parameterized complexity variation, constraint-based layout optimization, and task-aware camera trajectories.

2.1. Variability of Scene Complexity

  • InfiniBench separates compositional, relational, and observational complexity through user-specified scene parameters rather than coarse categories such as task type or room count.This isolates distractors, object arrangements, and occlusion as distinct sources of reasoning difficulty.Source: §2.1 · p. 3

2.2. Layout Generation in 3D Scene Generation

  • LLM-based layout generation becomes physically implausible as object counts and constraint intricacy grow, producing incorrect orientations, boundary violations, and overlaps.The limitation stems from flawed spatial logic when LLMs directly generate exact layouts.Source: §2.2 · p. 3
  • InfiniBench separates high-level LLM planning from low-level execution by converting natural-language descriptions into constraints that a dedicated optimizer satisfies physically.This framework improves scalability while remaining user-friendly.Source: §2.2 · p. 3

2.3. Camera Trajectory Generation

  • Poor camera viewpoints can hide critical objects or spatial relationships, making otherwise valid spatial reasoning questions unanswerable.Existing alternatives also struggle with expensive objective design, object coverage, or scene-specific retraining.Source: §2.3 · pp. 3–4
  • InfiniBench redefines frontier-based exploration around object clusters and samples viewpoints to manage occlusion while covering task-relevant objects.The approach adapts a lightweight robotic-navigation idea to camera trajectory generation.Source: §2.3 · p. 4

3. Method

InfiniBench converts natural-language scene descriptions into procedural constraints, optimizes dense physically plausible layouts, and renders task-relevant video views through a three-stage pipeline. Its agentic refinement, cluster-based layout optimization, and task-aware camera planning address failures that arise as scene complexity increases.

3. Method

  • InfiniBench uses an LLM-based agentic framework, cluster-based layout engine, and task-aware camera trajectory algorithm to turn prompts into informative videos of physically plausible 3D scenes.The pipeline iteratively refines procedural constraints, optimizes layouts, and renders VLM-consumable video frames.Source: §3 · p. 4

3.1. Agentic Generation of Scene Constraints

  • Natural-language prompting reduces the technical burden of manually scripting complex procedural constraints, with API knowledge and examples making high-level requests machine-readable.The framework translates requests such as a cluttered office desk into explicit object-count and spatial-relation constraints.Source: §3.1 · p. 4
  • Iterative refinement feeds optimizer failures, bird’s-eye collision maps, and unmet-constraint summaries back to the LLM so it can revise logically conflicting or physically impossible scene constraints.This loop addresses failures such as fitting three monitors onto a desk with insufficient surface area.Source: §3.1 · p. 4

3.2. Layout Optimization for Complex Scenes

  • The layout strategy replaces hierarchical placement because fixing large objects first can leave no space for smaller related objects as scene complexity grows.The method instead repositions dynamic groups such as tables and surrounding chairs without breaking their internal relationships.Source: §3.2 · p. 4
  • Cluster-based optimization treats related objects as movable units, expanding the action space while preserving internal relationships and enabling valid layouts for previously intractable high-density scenes.Clusters are identified from the semantic graph, moved with cluster-level operations, and collision-checked using collective bounding boxes.Source: §3.2 · pp. 4–5

3.3. Camera Trajectory Optimization

  • Camera optimization seeks the shortest collision-free path that gives every task-relevant object a clear, full, unobstructed view, using target selection, viewpoint sampling, and iterative path planning.Candidate viewpoints are scored for validity, object coverage, and occlusion, then connected with Dijkstra’s algorithm before rendering video frames.Source: §3.3 · p. 5

4. Evaluation of InfiniBench

InfiniBench is evaluated against procedural and LLM-based scene generators across object-count and occupancy complexity using fidelity, semantic alignment, realism, and physical-plausibility measures. It maintains both the prompt adherence of LLM-based methods and the physical coherence of procedural methods, with the strongest results in high-complexity settings.

4.1. Experimental Setup

  • Evaluation measures prompt fidelity, text-image alignment through CLIP, GPT-5-rated layout realism, and physical plausibility using out-of-boundary and collision counts.Experiments vary total object count and scene occupancy ratio to test two dimensions of scene complexity.Source: §4 · pp. 5–6, §4.1 · p. 6

4.2. Main Results

  • LLM-based generators preserve required scene complexity but lose physical plausibility, while procedural frameworks preserve plausibility but fail to match quantitative complexity requirements.At high object counts, LayoutGPT’s collision count reaches 13.5, versus 2.4 at low counts; Luminous and Infinigen Fidelity scores fall to 0.42 and 0.64.Source: §4.2 · pp. 6–7
  • InfiniBench reaches top performance across all metrics in high-complexity settings, achieving the highest CLIP and Realism scores while maintaining high Fidelity and near-perfect physical plausibility.It avoids the tradeoff between matching requested complexity and preventing out-of-boundary or collided objects.Source: §4.2 · pp. 6–7, Table 2 · p. 6, Table 1 · p. 6
  • Qualitatively, baseline scenes often contain missing, intersecting, or haphazardly placed objects, whereas InfiniBench produces coherent, cluttered scenes that reflect detailed requests.These comparisons are presented in Figure 9 alongside the quantitative results.Source: §4.2 · p. 7, Figure 9 · p. 7
  • Constraint refinement alone raises Fidelity from 0.64 to 0.71, cluster optimization alone raises it to 0.68 with marginally higher Realism, and combining both yields a dramatic synergistic improvement.The ablation compares the base Infinigen optimizer with each component and the full system.Source: §4.2 · p. 7, Table 3 · p. 7
  • Constraint refinement typically converges within five iterations, after which the reusable constraints can support many generated scenes and improve layout-generation compute efficiency.The iteration-count effect is evaluated in Table 4.Source: §4.2 · p. 7, Table 4 · p. 7

5. Using InfiniBench to Diagnose VLMs’ Failures in Spatial Reasoning

InfiniBench generates customizable benchmarks that vary compositional, relational, and observational scene complexity to diagnose VLM spatial-reasoning failures. Across these dimensions, performance declines with visual clutter and irrelevant objects, while higher viewpoints improve perspective-taking and spatiotemporal reasoning but not measurement.

5.1. VLM’s Spatial Reasoning with Varying Compositional Scene Complexity

  • VLM reasoning performance declines markedly as the number of scene objects increases, indicating difficulty managing greater visual clutter.The benchmarks vary object counts from 5 to 50.Source: §5.1 · p. 8, Figure 10 · p. 8, Figure 11 · p. 8
  • Gemini-2.5-Pro and Cambrian-S-7B achieve higher accuracy than other models as compositional scene complexity grows.The authors hypothesize that degradation reflects repetitive counting and overestimated counts relative to ground truth.Source: §5.1 · p. 8

5.2. VLM’s Spatial Reasoning with Varying Relational Scene Complexity

  • Increasing irrelevant objects slightly reduces performance, with reasoning paths indicating that incorrect object referencing likely causes the degradation.The relational-complexity benchmarks vary irrelevant-object counts.Source: §5.2 · p. 8, Table 5 · p. 8, Figure 12 · p. 8

5.3. VLM’s Spatial Reasoning with Varying Observational Scene Complexity

  • Performance on Perspective-taking and Spatiotemporal tasks improves substantially as the camera viewpoint moves closer to a bird’s-eye view.The compared views use camera heights of 2.5 m and 1 m; Table 6 reports the observational-complexity results.Source: §5.3 · p. 9, Table 6 · p. 9, Figure 13 · p. 9
  • Measurement tasks show negligible differences across the compared camera perspectives, unlike Perspective-taking and Spatiotemporal tasks.The authors associate higher-viewpoint gains with clearer and more comprehensive visibility of the scene layout.Source: §5.3 · p. 9, Table 6 · p. 9

6. Conclusion

InfiniBench is introduced as a fully automated benchmark generator for complex, physically plausible 3D scenes with customizable scene complexity. The supplied examples illustrate how scene constraints encode furniture relationships and clutter distribution in dining rooms.

6. Conclusion

  • InfiniBench creates complex and physically plausible 3D scenes while allowing systematic customization of scene complexity.The conclusion frames this customization as supporting fine-grained diagnostic analysis of VLM spatial-reasoning failures.Source: §6 · p. 9

A.1. Example 1: Dining Rooms

  • The dining-room constraint example encodes relationships among rooms, tables, chairs, dishware, utensils, storage, and display items.The example includes chairs around tables and clutter distribution for dishware and utensils.Source: §A.1 · pp. 12–13
  • Dining-room constraints include chair-placement relationships and distance, alignment, asymmetry, and stable-surface terms for layout optimization.These terms operationalize spatial organization and clutter placement in the procedural constraint language.Source: §A.1 · p. 12

A.2. Example 2: Bathrooms

The bathroom example specifies logical and geometric constraints for essential fixtures, accessibility, and spatial organization. It combines required hardware counts with distance-based and accessibility-related optimization terms.

A.2. Example 2: Bathrooms

  • Bathroom constraints enforce one sink and one toilet, constrain bathtub and hardware counts, and place bathroom elements within the room layout.The example specifically states that essential hardware and spatial accessibility are enforced.Source: §A.2 · p. 13
  • The bathroom layout minimizes distances involving bathtubs, sinks, hardware, mirrors, and room-floor relationships while adding accessibility costs.These terms encode practical fixture placement and access requirements.Source: §A.2 · p. 13
  • Listing 2 provides the in-context bathroom constraint example supplied to the LLM agent.The listing demonstrates the syntax and APIs used for this procedural scene description.Source: §A.2 · p. 13

B. CoT Reasoning for Refinement of Scene Constraints

InfiniBench refines scene constraints through an iterative feedback loop between layout optimization, structured error reporting, and LLM revision. Error reports combine unsatisfied-constraint summaries with bird’s-eye-view segmentation metadata.

B. CoT Reasoning for Refinement of Scene Constraints

  • The layout optimizer executes the current constraints, reports unsatisfied constraints and visual conflicts, and enables the LLM to revise them when requirements remain unmet.The LLM uses the original scene description and structured error report to produce revised constraints through Chain-of-Thought reasoning.Source: B. CoT Reasoning for Refinement of… · p. 13

C. Details of Cluster-Based Optimization

InfiniBench replaces rigid hierarchical optimization with cluster-based scene layout optimization, preserving object relationships while enabling dense, physically plausible room configurations.

C. Details of Cluster-Based Optimization

  • Cluster-based optimization addresses rigid-hierarchy limitations by grouping objects around stable parent relationships and updating cluster information from the global scene state.Cluster roots are typically large furniture, while child objects maintain a “stable against” relationship with their parent.Source: C. Details of Cluster-Based Optimization · p. 13, §C.1 · p. 14

C.2. Augmented Action Space

  • The augmented action space resamples, translates, and rotates entire clusters while preserving the relative positions and orientations of their child objects.These operations complement standard individual-object actions such as translation, rotation, deletion, and relation changes.Source: §C.2 · p. 14

C.3. Collision Checking

  • Collision checks immediately revert invalid operations and use collective cluster bounds or aggregate meshes to preserve physical plausibility.Cluster-level validation ensures grouped objects do not intersect other scene elements.Source: §C.3 · p. 14

D. Gallery of Generated 3D Scenes

  • Generated examples vary compositional and relational complexity across living rooms, bedrooms, and dining rooms, while camera trajectories demonstrate different observational complexities.The gallery includes multiple viewpoints and optimized trajectories, with Infinigen room-shape generation used for more realistic renderings.Source: D. Gallery of Generated 3D Scenes · p. 14

E. Details of the Camera Trajectory Optimization Algorithm

InfiniBench optimizes a task-aware camera path that samples valid viewpoints, checks visibility and occlusion, and connects selected views into a navigable trajectory covering all task-relevant objects.

E. Details of the Camera Trajectory Optimization Algorithm

  • The camera trajectory is optimized in four degrees of freedom: horizontal position plus yaw and pitch, while height and roll remain fixed for stable human-like views.Fixing vertical height approximates eye level, while zero roll maintains view stability.Source: E. Details of the Camera Trajectory… · p. 14
  • Each iteration targets the closest unvisited object, randomly samples nearby oriented viewpoints, and evaluates them against accessibility, field-of-view, and occlusion constraints.A candidate is valid only when it lies in the navigable floor-plan area, fully contains the target in view, and satisfies the occlusion threshold.Source: E. Details of the Camera Trajectory… · pp. 15–16
  • The best valid viewpoint becomes the next keyframe, after which Dijkstra’s algorithm plans a shortest collision-free path from the current camera position.The process repeats as objects are marked visited and removed from the remaining target list.Source: E. Details of the Camera Trajectory… · p. 17
  • The method supports observational-complexity examples including partial occlusion and a 2-meter bird’s-eye camera height.These settings are illustrated through generated camera trajectories and scene frames at selected waypoints.Source: Figure 21 · p. 18, Figure 22 · p. 18
  • The optimizer produces a complete smooth trajectory by iteratively selecting viewpoints until every scene object is captured at least once.Selected viewpoints are connected by navigable paths to provide a comprehensive visual record.Source: E. Details of the Camera Trajectory… · p. 18

F. Details of InfiniBench Implementation

InfiniBench uses Blender Cycles for photorealistic rendering and reduces rendering cost through lower sampling with denoising while retaining visually similar frames.

F. Details of InfiniBench Implementation

  • Blender Cycles path tracing provides photorealistic frame rendering, but the typically recommended 8192 samples require approximately 5 minutes per frame.The implementation therefore treats sample count as a key efficiency parameter.Source: F. Details of InfiniBench Implementation · p. 18
  • The implementation details include an algorithm that extracts object meshes, selects targets, samples viewpoints, checks collisions and visibility, computes occlusion, and plans paths.The workflow repeats until all remaining targets are processed and returns the accumulated camera path.Source: F. Details of InfiniBench Implementation · p. 19
  • With denoising enabled, rendered frames appear nearly identical to originals rendered with 8192 samples, with only some fine textures slightly blurred.The listed configuration specifies 128 samples, adaptive threshold 0.01, denoising enabled, and exposure 1.Source: F. Details of InfiniBench Implementation · p. 19

G. Additional Qualitative Comparisons

InfiniBench is qualitatively compared with baseline scene generators under matched assets, room geometry, and rendering settings to assess generated scene layouts.

G. Additional Qualitative Comparisons

  • The comparison uses the same 3D asset library and square room plan across methods, with Blender rendering standardized for baselines built in different engines.For baselines lacking direct natural-language input, constraints are iteratively refined until their programs become executable.Source: G. Additional Qualitative Comparisons · p. 19
  • Figure 25 presents additional qualitative examples comparing scenes generated by InfiniBench against baseline approaches.The comparison focuses on scene-generation outputs under controlled rendering conditions.Source: G. Additional Qualitative Comparisons · p. 19, Figure 25 · p. 20

Related papers

Loading 2511.18200v2…