PaperlayerView exact revision on arXiv

Source-linked AI summary

LychSim: A Controllable and Interactive Simulation Framework for Vision Research

Wufei Ma, Chloe Wang, Siyi Chen, Jiawei Peng, Patrick Li, Alan Yuille

arXiv:2605.12449v1cs.CV

TL;DR

Vision research still needs simulation for controlled OOD evaluation and closed-loop optimization, but existing platforms demand substantial graphics expertise. LychSim addresses this with an accessible, procedural, MCP-enabled framework that provides rich 2D and 3D ground truths and supports synthetic-data generation, adversarial examination, and interactive scene design.

  • Problem

    Simulation remains important for controlled OOD evaluation and closed-loop vision optimization, while professional engines impose substantial technical barriers on researchers.

  • Method

    LychSim combines a streamlined Python API, procedural high-fidelity data generation with rich 2D and 3D ground truths, and native MCP integration for interactive scene manipulation.

  • Results

    Across three case studies, LychSim supports challenging OOD synthesis, adversarial examination of vision models, and interactive scene planning and editing by agentic models.

  • Takeaways & Limitations

    LychSim provides an accessible, controllable simulation environment for computer vision research with diverse synthetic data, semantically aligned 3D annotations, and closed-loop interaction.

  • Takeaways & Limitations

    Interactive scene generation still exhibits physically implausible layouts and object collisions, attributed largely to current models’ limited spatial reasoning.

Abstract

from arXiv · show

While self-supervised pretraining has reduced vision systems' reliance on synthetic data, simulation remains an indispensable tool for closed-loop optimization and rigorous out-of-distribution (OOD) evaluation. However, modern simulation platforms often present steep technical barriers, requiring extensive expertise in computer graphics and game development. In this work, we present LychSim, a highly controllable and interactive simulation framework built upon Unreal Engine 5 to bridge this gap. LychSim is built around three key designs: (1) a streamlined Python API that abstracts away underlying engine complexities; (2) a procedural data pipeline capable of generating diverse, high-fidelity environments with varying out-of-distribution (OOD) visual challenges, paired with rich 2D and 3D ground truths; and (3) a native integration of the Model Context Protocol (MCP) that transforms the simulator into a dynamic, closed-loop playground for reasoning agentic LLMs. We further annotate scene-level procedural rules and object-level pose alignments to enable semantically aligned 3D ground truths and automated scene modification. We demonstrate LychSim's capability across multiple downstream applications, including serving as a synthetic data engine, powering reinforcement learning-based adversarial examiners, and facilitating interactive, language-driven scene layout generation. To benefit the broader vision community, LychSim will be made publicly available, including full source code and various data annotations.

1. Introduction

Although self-supervised and weakly supervised pretraining reduces reliance on synthetic data, simulation remains important for analyzing vision systems and creating controlled scenarios with aligned 2D and 3D ground truths. LychSim addresses this need with an easy-to-use, controllable, interactive framework featuring a streamlined Python API and a procedural data pipeline.

  • Motivation: Simulation remains critically important because it supports analysis of complex vision systems and enables comprehensive, aligned 2D and 3D ground truths in diverse controlled scenarios.The introduction identifies simulation as serving two key objectives, including rigorous analysis and controlled evaluation.
  • LychSim: LychSim is a controllable and interactive simulation framework designed for computer vision research.The framework is built around ease of use, procedural data generation, and controllable interaction.
  • Procedural data pipeline: LychSim includes a built-in procedural data pipeline that generates diverse environments with rich 2D and 3D ground truths.The procedural pipeline is presented as a core system design alongside the Python interface and interactive simulation.
  • Ease of use: Its streamlined Python API abstracts UE5 and C++ complexities, allowing researchers to script and manipulate high-fidelity 3D scenes without prior computer graphics expertise.This design emphasizes accessibility for researchers who do not have extensive computer graphics or Unreal Engine development expertise.
  • Availability: The authors intend to release LychSim publicly, including its complete C++ and Python components.The introduction frames the release as a contribution intended to benefit the broader computer vision community.

2. Simulation System Design

LychSim combines UE5’s high-quality asset ecosystem with annotated procedural structure to automate diverse scene generation and targeted OOD sampling. Its rendering pipeline provides comprehensive 2D and 3D ground truths, including annotations for partially occluded and object-part regions.

  • Asset foundation: UE5 provides LychSim with high-quality artist-created 3D assets while avoiding rendering artifacts and material inconsistencies from cross-platform asset transfer.Raw assets remain unstructured and difficult to manipulate automatically, motivating unified scene representations.
  • Procedural structure: LychSim annotates object categories, canonical scales, pose alignments, and scene-level spatial rules to support semantically aligned poses, placement, and scene manipulation.Rules include navigable floors, roads, pedestrian walks, and dynamic trajectories that guide structural generation.
  • Procedural structure: A hybrid pipeline combines diverse UE5 Fab environments and object libraries with annotated rules to generate indoor and outdoor scenes spanning architectural styles, geographies, and lighting conditions.The approach combines high-quality artist-created environments with rich object meshes and materials.
  • OOD data generation: Annotated procedural rules enable targeted synthesis of challenging OOD data with uncommon viewpoints, severe occlusions, high-density layouts, and semantically cluttered same-category groups.The simulator uses scene control to produce data with varying visual complexity.
  • Ground-truth annotations: LychSim supplies standard and novel 2D and 3D ground truths, including depth, segmentation, surface normals, point maps, bounding boxes, visibility, and part-level outputs.UE5 render targets directly produce object-part IDs and per-pixel 3D vertex positions, while scene modeling extends analysis beyond visible regions.

3. Python and Agent Integration

LychSim lowers the technical barrier to Unreal Engine simulation through a streamlined Python API that supports consistent scene manipulation and rich ground-truth generation. Its MCP server exposes these capabilities as standardized tools, enabling agentic LLMs to interact with and reason over dynamic 3D scenes in a closed loop.

  • Python Integration: LychSim’s streamlined Python integration abstracts Unreal Engine’s technical complexities, reducing the need for computer graphics and C++ expertise.The API is designed to address the time and effort required to master professional simulation engines.
  • Python Integration: A unified interface handles StaticMesh, SkeletalMesh, and Blueprint assets through consistent high-level spawning and interaction procedures.This removes discrepancies between asset classes that otherwise require distinct engine workflows.
  • Python Integration: Simple Python commands support programmatic asset spawning, coordinate adjustment, removal, and retrieval of synchronized RGB, depth, segmentation, and point-map ground truths.The API also provides object-level annotations for reconstructing the 3D scene.
  • Agent Integration: LychSim’s MCP integration bridges reasoning agentic LLMs and the 3D simulator by enabling closed-loop interaction with the environment.The standardized interface moves agents beyond static data processing toward interaction with the 3D world.
  • Agent Integration: A dedicated MCP server exposes the Python API as tools for scene navigation, structured state queries, real-time visual capture, and programmatic object manipulation.These tools allow AI agents to inspect and modify simulated scenes through standardized operations.

4. Case Studies

LychSim’s procedural simulation pipeline supports high-fidelity synthetic data with comprehensive 2D and 3D ground truths for diagnosing spatial VLM weaknesses and scaling VLM post-training. Its case studies also demonstrate RL-based adversarial evaluation and interactive, language-driven 3D scene planning, while exposing failures from limited spatial reasoning.

  • For evaluation and analysis: LychSim generates high-fidelity synthetic data with comprehensive 2D and 3D ground truths for evaluating spatial VLMs and supporting scalable post-training.The framework is positioned as both a diagnostic benchmark and a synthetic data engine for improving 2D and 3D spatial understanding.
  • For evaluation and analysis: An RL-based adversarial examiner trains a Gaussian policy to explore 3D camera viewpoints around target objects and minimize SAM prediction IoU.The examiner systematically searches the parameter space to reveal weaknesses caused by limited dataset coverage, with failure examples shown for Segment Anything.
  • Interactive scene planning: Through an Unreal Engine 5 plugin and MCP server, agentic LLMs can plan complete scenes from specifications, inspect multiple viewpoints, and correct physically implausible layouts.The interactive pipeline uses scene specification and skill files together with available MCP tools.
  • Interactive scene planning: Agentic models can edit generated scenes through multi-turn natural-language requests, extending LychSim beyond feed-forward 3D layout generation.The demonstrated systems include Claude Opus 4.6 and Gemma 4.
  • Interactive scene planning: The interactive pipeline still produces physically implausible layouts and object collisions, largely because current state-of-the-art models have limited spatial reasoning capabilities.These failure patterns are identified as limitations despite the promise of interactive 3D scene design.

5. Related Works

This section situates LychSim among synthetic data engines, simulation platforms for analyzing vision systems, and automated 3D scene-layout methods. LychSim extends UnrealCV with richer ground truths, procedural generation, agentic LLM integration, and support for both rule-based and LLM-driven planning.

  • Synthetic data engine: Simulation provides perfectly aligned ground truths and closed-loop environments for computer vision training and optimization.Existing synthetic data frameworks span Blender-based, Unreal-based, and physics-focused platforms.
  • Synthetic data engine: LychSim builds on UnrealCV by adding more 2D and 3D ground truths, procedural scene generation, and agentic LLM integration.Physics-focused alternatives include NVIDIA Isaac Sim, PyBullet, and MuJoCo, which emphasize rigid-body dynamics, contact physics, and high-frequency control loops.
  • Analyzing vision systems in simulation: Simulation addresses real-world dataset limitations by providing perfect annotations and systematic control over visual factors such as complexity and occlusion.These capabilities help isolate underlying failure modes in vision systems.
  • Automated scene layout generation: LychSim unifies rule-based procedural generation with native MCP support for closed-loop, multi-turn, LLM-driven spatial planning.Related approaches separately use programmatic constraints or foundation models to synthesize plausible 3D layouts from natural-language prompts.

6. Conclusions

LychSim is presented as a highly controllable, interactive simulation framework that bridges complex 3D graphics engines and computer vision research. It combines a streamlined Python API, procedural data generation, and native MCP integration to provide an accessible closed-loop environment.

  • Contributions: LychSim bridges complex 3D graphics engines and the evolving needs of computer vision research.The framework is designed to make simulation more suitable for vision research.
  • Contributions: The framework combines a streamlined Python API, a robust procedural data pipeline, and native Model Context Protocol integration.These three components form LychSim’s core system design.
  • Contributions: Together, these components provide an accessible, powerful, and closed-loop simulation environment.LychSim moves beyond standard annotations by explicitly modeling underlying 3D structures.

Appendix … A.3. Procedural Rules and Object Annotations

LychSim provides hierarchical 2D and 3D ground truths, an MCP bridge for interactive agentic simulation, and annotation tools for procedural rules and consistent object placement. These components support systematic evaluation, closed-loop planning, and diverse physically plausible scene generation.

  • A.1. Comprehensive 2D and 3D Ground Truths: LychSim extracts scene-, view-, and object-level ground truths spanning global conditions, pixel-aligned geometry, and fine-grained object structure.The annotations include visual factors from environmental conditions to occluded or part-level geometries.
  • A.1. Comprehensive 2D and 3D Ground Truths: Scene-level annotations record directional and ambient lighting, quantitative fog visibility, and rain parameters for evaluation under weather and illumination shifts.These macroscopic variables capture global environmental variation.
  • A.1. Comprehensive 2D and 3D Ground Truths: View-level annotations include depth maps, instance masks, surface normals, and dense per-pixel 3D vertex coordinates for 3D learning.The dense point maps are rendered directly and aligned to image pixels.
  • A.1. Comprehensive 2D and 3D Ground Truths: Object-level annotations provide 2D and 3D boxes, semantically aligned 3D poses, occlusion and truncation ratios, and part segmentations.Occlusion and truncation are quantified through 3D geometric projections, while part segmentation supports general mesh parts and semantic hierarchies.
  • A.2. Model Context Protocol (MCP): MCP is built on the LychSim Python API, enabling agentic LLMs to query spatial states, place objects, and receive real-time visual feedback.Parallel rendering of multiple camera viewpoints reduces communication overhead during iterative agentic planning.
  • A.3. Procedural Rules and Object Annotations: The interactive annotation tool lets users define spatial relationships and semantic regions by selecting anchor objects in the Unreal Editor, then export annotations with one command.The tool is designed to create procedural rules directly inside the editor for downstream data generation.
  • A.3. Procedural Rules and Object Annotations: LychSim defines four procedural primitives: road and street areas, person-navigable areas, vehicle trajectories, and pedestrian trajectories.These spatial priors support diverse, physically plausible layouts across domains.
  • A.3. Procedural Rules and Object Annotations: Each 3D object is annotated with a semantic category, canonical scale, bottom-center sampling offset, and canonical 3D pose alignment for consistent procedural placement.The offset provides ground-level alignment across disparate mesh sources, while pose alignment standardizes object orientation.

B. Code and Data Releases

LychSim will be publicly released with complete source code, simulation assets, procedural annotations, and onboarding materials. The project specifies open licenses for code and annotations, distinct terms for underlying meshes, and ongoing maintenance with community contributions.

  • Code and Data Access: The release will include complete C++ and Python source code, the MCP server, 3D assets, procedural rules, and object-level pose alignments.Documentation, API references, and quick-start tutorials will support onboarding.
  • License: LychSim source code will use the MIT license, while newly curated annotations will use CC BY 4.0.Underlying 3D meshes remain governed by standard Fab Asset Marketplace terms.
  • Project Maintenance and Community: The development team will maintain the repository by addressing bugs, supporting future engine compatibility, and reviewing community pull requests.The team also welcomes feedback and collaborative contributions to expand LychSim’s functionality and environmental diversity.

C. Supplementary Documents · 27 Floor Z: -20 · 28 X range: 420 -- 869 (depth ~449 cm)

The supplementary materials document LychSim’s MCP-enabled scene-planning workflow, including state inspection, object editing, sizing, camera control, procedural placement guidance, and verification. They also specify coordinate conventions and example loft-office scene assets and geometry.

  • C. Supplementary Documents: The MCP spawn-object schema requires a unique object name, an Unreal Engine asset path, world-space location, rotation, scale, and collision behavior.The schema reports success or explicit errors and supports collision policies including always spawn, skip if colliding, and adjust if possible.
  • C. Supplementary Documents: LychSim’s scene-planning skill combines state queries, object spawning/editing, mesh sizing, and camera controls through MCP tools.Supported tools include list_objects, get_object_location, add_object, update_object, delete_object, get_mesh_extent, and camera pose functions.
  • C. Supplementary Documents: The workflow reads a scene specification, snapshots persistent scene state, plans functional zones, places anchors first, stacks smaller items, and verifies results from multiple angles.It emphasizes preserving existing scene props, using collision adjustment, checking screenshots critically, and iterating on overlaps or incorrect orientations.
  • 27 Floor Z: -20: The supplementary documents define world coordinates as centimeters in a left-handed, Z-up system, with yaw=0 facing +X and typical LoftOffice floor Z = -20.Most static meshes spawn from their base, but the documentation advises verification because this is not universal.
  • C. Supplementary Documents: Scene placement maps specification options to tool arguments for collision handling, duplicate skipping, selective cleanup, rotation locking, and final camera restoration.The specification advises using adjust_if_possible, checking existing object IDs, deleting only previously spawned objects, and restoring the canonical final camera pose.

29 Y range: -410 -- 180 (width ~590 cm)

The layout specifies a furnished workspace with a desk setup, seating, plants, lighting, and table decor. Furniture should form functional groupings with clear walkways, while plants occupy peripheral areas and stacked objects use mesh extents for correct placement.

  • Layout Requirements: The scene includes one table with a monitor and stacked books, three mixed office and regular chairs, a soft reading-nook chair, two floor-sized plants, a floor lamp, and a vase.The plants are scaled to approximately 5.0 for floor-sized placement.
  • Layout Requirements: Furniture should be arranged in functional desk and reading groupings rather than scattered randomly, with chairs facing their associated table or focal point.The intended organization separates a desk area from a reading corner.
  • Layout Requirements: Walkways should remain clear with approximately 80 cm minimum spacing between furniture groups, while plants should sit near walls or corners rather than in the room center.These constraints support circulation and peripheral plant placement.
  • Layout Requirements: Stacked objects such as the monitor, books, and vase should use mesh extents to compute correct Z offsets.The stacking rule applies to objects placed on the table.
Loading 2605.12449v1…