PaperlayerView exact revision on arXiv

Source-linked AI summary

SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds

Jiawei Ren, Yan Zhuang, Xiaokang Ye, Lingjun Mao, Xuhong He, Jianzhi Shen, Mrinaal Dogra, Yiming Liang, Ruixuan Zhang, Tianai Yue, Yiqing Yang, Eric Liu, Ryan Wu, Kevin Benavente, Rajiv Mandya Nagaraju, Muhammad Faayez, Xiyan Zhang, Dhruv Vivek Sharma, Xianrui Zhong, Ziqiao Ma, Tianmin Shu, Zhiting Hu, Lianhui Qin

arXiv:2512.01078v2cs.AI

TL;DR

Agents that succeed in structured digital tasks still face complex, dynamic physical and social worlds. SimWorld combines realistic, open-ended simulation with a multimodal language-agent interface and customizable reasoning scenarios. Delivery-task evaluations reveal distinct strategies and a trade-off between high profits and consistent behavior.

  • Problem

    LLM/VLM agents have advanced in structured domains, but physical and social worlds are complex, noisy, dynamic, and unpredictable.

    Source: §1 · p. 3

  • Method

    SimWorld combines physically and socially grounded, language-steerable world simulation with multimodal observations, open-ended language actions, and customizable physical and social reasoning scenarios.

    Source: §1 · p. 3

  • Results

    Delivery-task evaluations expose a trade-off: DeepSeek-V3 and Claude-3.5-Sonnet earn high profits but vary more, while Gemini-2.5-Flash is steadier; GPT-4o-mini fails across metrics.

    Source: §1 · p. 4, §3.2 · p. 17

  • Takeaways & Limitations

    The delivery case study surfaces distinct model strategies and emergent behaviors in long-horizon multi-agent cooperation and competition.

    Source: §1 · pp. 3–4

Abstract

from arXiv · show

While LLM/VLM-powered AI agents have advanced rapidly in math, coding, and computer use, their applications in complex physical and social environments remain challenging. Building agents that can survive and thrive in the real world (for example, by autonomously earning income or running a business) requires massive-scale interaction, reasoning, training, and evaluation across diverse embodied scenarios. However, existing world simulators for such development fall short: they often rely on limited hand-crafted environments, simulate simplified game-like physics and social rules, and lack native support for LLM/VLM agents. We introduce SimWorld, a new simulator built on Unreal Engine 5, designed for developing and evaluating LLM/VLM agents in rich, real-world-like settings. SimWorld offers three core capabilities: (1) realistic, open-ended world simulation, including accurate physical and social dynamics and language-driven procedural environment generation; (2) a rich interface for LLM/VLM agents, with multimodal world inputs and open-vocabulary actions at varying levels of abstraction; and (3) diverse and extensible physical and social reasoning scenarios that are easily customizable by users. We demonstrate SimWorld by deploying frontier LLM agents (e.g., GPT-4o, Gemini-2.5-Flash, Claude-3.5, and DeepSeek-Prover-V2) on long-horizon multi-agent delivery tasks involving strategic cooperation and competition. The results reveal distinct reasoning patterns and limitations across models. We open-source SimWorld and hope it becomes a foundational platform for advancing real-world agent intelligence across disciplines: https://simworld.org.

1. Introduction

SimWorld addresses the difficulty of developing agents for complex physical and social worlds by combining realistic, open-ended simulation, an LLM/VLM interface, and extensible reasoning scenarios. Its delivery-task demonstration reveals varied model strategies and limitations, while the platform is intended for research across disciplines.

1. Introduction

  • Existing simulators are limited by game-like dynamics, narrow or static tasks, scripted small communities, or lack of natural-language interfaces.These limitations characterize game platforms, domain-specific simulators, and social sandboxes, respectively.Source: §1 · p. 3
  • SimWorld combines realistic physical and social dynamics with open-ended procedural worlds that users can modify through natural-language prompts.Its worlds include physical forces and social behaviors alongside controllable scenes and on-the-fly editing.Source: §1 · p. 3
  • Its Gym-like interface pairs multimodal observations with open-vocabulary language actions, decomposes high-level commands into low-level actions, and returns feedback for continued reasoning.This closed loop supports long-horizon reasoning at an appropriate level of abstraction.Source: §1 · p. 3
  • SimWorld supports extended physical and social reasoning scenarios, including a multi-agent delivery economy where agents bid, invest, share orders, and navigate dynamic environments.The task is designed to demonstrate collaboration and competition over long horizons.Source: §1 · p. 3
  • Claude-3.5-Sonnet and DeepSeek-V3 earn the highest profits but can behave erratically, while Gemini-2.5-Flash and DeepSeek-Prover-V2 favor more stable, conservative strategies.Conscientious agents focus on task completion, whereas open agents explore but frequently lose money.Source: §1 · p. 4
  • The paper presents SimWorld as infrastructure for studying complex systems and emergent behaviors across robotics, business, public health, social science, education, and other disciplines.The authors open-source the platform and support large-scale agent–environment and agent–agent interactions.Source: §1 · p. 4

2. The SimWorld Simulator

SimWorld separates its Unreal Engine backend from Python environment and agent layers, linking them through UnrealCV+ to support environment generation, agent interaction, and varied observations and actions.

2. The SimWorld Simulator

  • SimWorld uses a three-tier architecture that separates the Unreal Engine backend from Python-based Environment and Agent layers.The Environment layer provides environment generation and a Gym-like interface; the Agent layer supports diverse observations, open-ended actions, and reasoning and planning components.Source: §2 · p. 4
  • The Environment layer connects to the Unreal Engine backend through UnrealCV+ and provides infinite environment generation and a standard Gym-like agent interface.Source: §2 · p. 4
  • The Agent layer supports diverse input observations, open-ended output actions, and different reasoning and planning components.Source: §2 · p. 4

2.1. Unreal Engine Backend

The Unreal Engine backend supplies SimWorld’s rendering and physics foundation, with curated and generated scenes, diverse assets, and physically grounded entities and interactions.

2.1. Unreal Engine Backend

  • The backend provides high-fidelity rendering and physics through scenes, an asset library, and physics simulation.Source: §2.1 · pp. 4–5

2.1.1. Diverse Scenes

  • SimWorld combines over 100 handcrafted scenes with procedurally generated urban environments, supporting both varied settings and customizable, controlled experimentation.The handcrafted scenes span visual and structural styles; procedural generation varies parameters such as city size, road density, and layout style.Source: §2.1.1 · p. 6
  • Handcrafted and procedurally generated scenes together provide environments for controlled experiments and open-ended agent research.Source: §2.1.1 · p. 6

2.1.2. Rich Assets and Physics Realism

  • The asset library combines static and dynamic assets with lighting, weather, animations, and interactions to support physics-driven simulations.Source: §2.1.2 · p. 6
  • Human, vehicle, and robot embodiments share a physics framework, while modeling embodiment-specific movement and control.The framework includes shared physical attributes such as mass, inertia, and contact forces.Source: §2.1.2 · p. 7
  • SimWorld models continuous physical dynamics such as gravity and inertia, with physically informed animations intended to keep motion coherent with environmental forces.The paper contrasts this with Minecraft’s discrete, block-based mechanics.Source: §2.1.2 · p. 7

2.2. Environment

The Environment layer abstracts Unreal Engine worlds into agent-facing tools for city generation, scene editing, navigation, traffic, and Gym-like interaction. These components support customizable environments and agent access through structured interfaces.

2.2. Environment

  • The Environment layer abstracts Unreal Engine worlds and integrates city generation, scene editing, traffic, a Gym-like interface, and waypoints for navigation.Source: §2.2 · p. 8

2.2.1. Procedural City Generation

  • SimWorld’s procedural generator creates customizable urban environments with road networks, buildings, traffic, and street elements, enabling effectively infinite scenarios.Its pipeline proceeds through road, building, and street-element generation.Source: §2.2.1 · p. 8

2.2.2. LLM-based Scene Editing

  • A retrieval-augmented LLM scene agent grounds natural-language edits in the scene graph, places matching library assets, and can generate a new asset when needed.It uses spatial anchors and contextual landmarks to identify the intended location.Source: §2.2.2 · p. 9

2.2.3. Waypoint System

  • A hierarchical navigation graph combines coarse waypoints for roads and intersections with fine-grained points for walkable paths, supporting flexible navigation.The waypoint system also supports the traffic system and action planner.Source: §2.2.3 · p. 9

2.2.4. Traffic System

  • The traffic system coordinates vehicle spawning and routing, intersection control, and pedestrian movement to simulate urban traffic phenomena.It is built on waypoints and is designed to generalize across procedurally generated city layouts.Source: §2.2.4 · p. 10

2.2.5. Gym-like Interface for Agent-Environment Interaction

  • A standard Gym-like interface supports agent interaction and integration with reinforcement-learning pipelines, while asynchronous and synchronous modes serve different simulation needs.Asynchronous execution lets agents advance independently; synchronous execution waits for every agent’s action to maintain temporal alignment.Source: §2.2.5 · p. 11

2.3. Agent

SimWorld provides a unified LLM/VLM agent interface spanning perception, reasoning, planning, and execution. Agents can use multimodal observations and open-ended actions, which the action planner translates into low-level execution.

2.3. Agent

  • The interface accommodates low-level control and high-level reasoning through agent, observation, action, and planner components.The planner bridges agent reasoning and executable actions.Source: §2.3 · p. 12, §2.3.4 · p. 13

2.3.1. Agent Framework

  • The agent framework structures a common perception, reasoning, planning, and execution loop across humanoid, vehicle, and robot embodiments.Researchers can also plug in components such as world models, memory systems, and mental-state modules.Source: §2.3.1 · p. 12

2.3.2. Observation Space

  • Agents can access visual inputs and structured semantic information, including scene graphs and GPS-like localization.Visual observations include color images, depth maps, and semantic segmentation masks.Source: §2.3.2 · p. 12

2.3.3. Action Space

  • The action space combines high-level natural-language commands with low-level primitive controls and interactions.Supported primitives vary by embodiment and include vehicle controls, robot movement, and human interactions with objects, vehicles, and other people.Source: §2.3.3 · pp. 12–13, Table 2 · p. 13

2.3.4. Action Planner

  • The action planner parses high-level plans into primitive actions and executes them step by step according to the current environment state.Rule-based and visual-based executors support different ways of carrying out those actions.Source: §2.3.4 · p. 13

1. Assets Preparation

The assets-preparation materials reference marketplace downloads and model-generated assets. UnrealCV+ connects the Unreal Engine backend and environment layer through bidirectional TCP communication.

1. Assets Preparation

  • The assets-preparation materials reference marketplace downloads and assets generated by models.A workflow figure is identified for constructing a task suite.Source: §1 · p. 14, Figure 8 · p. 14

2.4. UnrealCV+ Communication Module

  • The communication module connects the Unreal Engine backend with the environment layer over a bidirectional TCP connection.UnrealCV+ is implemented in Python and C++ for data exchange and fine-grained simulation control.Source: §2.4 · p. 15
  • A customized command set supports scene control, actor manipulation, and data queries between the environment layer and Unreal Engine.Examples include spawning actors, querying pedestrian positions, and executing robot actions.Source: §2.4 · p. 15

3. Case Study: Delivery Task

In the delivery task, DeepSeek-V3 and Claude-3.5-Sonnet achieved the highest mean profits, while Gemini-2.5-Flash was more consistent; model behavior also varied with competition, resources, and persona. The task evaluates agents in a city-scale setting with economic, energy, and order-sharing dynamics.

3. Case Study: Delivery Task

  • The delivery task places agents in a city-scale environment with energy management, purchases, and order sharing, evaluating decision-making and social reasoning.Agents aim to grow and thrive; the task process includes asset preparation, city generation, agent spawning, simulation, and evaluation.Source: §3.1 · p. 15, §3 · p. 15, Figure 10 · p. 17

3.1. Task Formulation

  • Agents choose among bidding, pickup, delivery, sharing, and scooter-purchase actions, with language models selecting high-level actions and the planner executing low-level ones.The study evaluates total profit, order success, energy efficiency, sharing, and investment.Source: §3.1 · pp. 15–16, Table 3 · p. 18

3.2. Main Results

  • DeepSeek-V3 (69.475 ± 16.772) and Claude-3.5-Sonnet (69.068 ± 20.685) had the highest mean profits, while Gemini-2.5-Flash’s mean profit was 42.423 ± 3.103 and GPT-4o-mini scored zero across metrics.Claude-3.5-Sonnet also led in mean successful orders (2.733 ± 1.102) and energy efficiency (0.5411 ± 0.1981); Table 4 reports averages and standard deviations over three simulations.Source: §3.2 · pp. 16–17, Table 4 · p. 18

3.3. Ablation Study

  • In model competition, flexible bid distributions were associated with higher win rates, while lower bids helped DeepSeek-Prover-V2 and Qwen3-32B win head-to-head contests.Higher bids reduced success for GPT-4o and LLaMA-3.2-11b; inactive QwQ-32B and GPT-4o-mini acquired fewer orders.Source: §3.3 · pp. 18–19, Figure 11 · p. 19
  • With more initial capital, agents relied less on aggressive bidding, prioritized pickup more, and were more willing to purchase scooters.The study associates sufficient capital with investment and shared delivery when agents also have enough orders taken.Source: §3.3 · p. 19
  • As order availability increased, agents performed fewer pickup and delivery actions and more often waited; shared deliveries also became more common.In low-resource settings, agents were more motivated to pursue delivery tasks.Source: §3.3 · p. 19, Figure 12 · p. 20
  • Conscientious agents bid less and completed more tasks, while agreeable agents were less inactive and had higher bid win rates.Higher openness was associated with reduced delivery engagement, possibly because exploration of competitive or unconventional bidding diverted attention from task execution.Source: Figure 13 · p. 20, §3.3 · p. 20

4. Related Works

Prior simulators cover varied social and embodied settings, but existing embodied platforms are often limited by scene type, handcrafted environments, or weaker support for dynamic multi-agent interaction. SimWorld is presented as a procedurally generated, LLM/VLM-compatible platform for realistic multi-agent environments.

4. Related Works

  • Embodied simulators often focus on indoor, driving, or natural settings, and many rely on limited handcrafted scenes.Some platforms use rule-based procedural generation, while dynamic multi-agent interaction across diverse environments remains a stated gap.Source: §4 · p. 21
  • Language-driven social simulators model large-scale social phenomena, while Minecraft-based platforms explore collective behavior and AI civilizations.Examples include OASIS, Casevo, MineLand, Project sid, and VirtualCommunity.Source: §4 · p. 21
  • SimWorld addresses the stated gap with a scalable, procedurally generated, LLM/VLM-compatible platform for multi-agent collaboration, competition, and benchmarking.The paper contrasts this engine-based approach with neural world models that generate interactive video predictions.Source: §4 · p. 21

Related papers

Loading 2512.01078v2…