PaperlayerView exact revision on arXiv

Source-linked AI summary

Virtual Community: An Open World for Humans, Robots, and Society

Qinhong Zhou, Hongxin Zhang, Xiangye Lin, Zheyuan Zhang, Yutian Chen, Wenjun Liu, Zunzhe Zhang, Sunli Chen, Lixing Fang, Qiushi Lyu, Xinyu Sun, Jincheng Yang, Zeyuan Wang, Bao Chi Dang, Zhehuan Chen, Daksha Ladia, Quang Vinh Dang, Jiageng Liu, Chuang Gan

arXiv:2508.14893v3cs.CVcs.CLcs.RO

TL;DR

Existing platforms provide limited support for large, heterogeneous human–robot communities in scalable open worlds. Virtual Community combines unified simulation with geospatially grounded generation and introduces open-world planning and robot-cooperation challenges; baseline results show that planning remains difficult and no method prevails across all three tasks.

  • Problem

    Existing platforms rarely support large, heterogeneous human–robot communities in scalable open worlds, limiting study of complex multi-agent behavior.

    Source: §1 · p. 2

  • Method

    Virtual Community combines Genesis-based simulation of human-like agents and robots with generative, geospatially grounded open-world construction.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    No single planning method prevails across the three tasks: the Heuristic Planner performs strongly in delivery, while the LLM Agent leads in search but performs poorly on the other two tasks.

    Source: Baselines · p. 8

  • Takeaways & Limitations

    The Community Planning and Community Robot challenges provide evaluated baseline tasks for open-world planning and multi-robot cooperation.

    Source: §6 · pp. 10–11

    Outdoor scenes lack sufficient detail to accurately reflect real-world physical and visual properties.

    Source: §6 · p. 11

Abstract

from arXiv · show

The rapid progress in AI and Robotics may lead to a profound societal transformation, as humans and robots begin to coexist within shared communities, introducing both opportunities and challenges. To explore this future, we present Virtual Community-an open-world platform for humans, robots, and society-built on a universal physics engine and grounded in real-world 3D scenes. With Virtual Community, we aim to enable the study of embodied social intelligence at scale. To support these, Virtual Community features: 1) An open-source multi-agent physics simulator that supports robots, humans, and their interactions within a society; 2) A large-scale, real-world aligned community generation pipeline, including vast outdoor space, diverse indoor scenes, and a community of grounded agents with rich characters and appearances. Leveraging Virtual Community, we propose two novel challenges. The Community Planning Challenge evaluates multi-agent reasoning and planning ability in open-world settings, such as cooperating to help agents with daily activities and efficiently connecting other agents. The Community Robot Challenge requires multiple heterogeneous robots to collaborate in solving complex open-world tasks. We evaluate various baselines on these tasks and demonstrate the challenges in both high-level open-world task planning and low-level cooperation controls. We hope that Virtual Community will unlock further study of human-robot coexistence within open-world environments.

1 INTRODUCTION

Virtual Community targets the limited support for large, heterogeneous human–robot communities in scalable open worlds. It combines unified simulation and real-world-aligned world generation, and introduces challenges for open-world planning and robot cooperation.

1 INTRODUCTION

  • Existing platforms rarely support large, heterogeneous human–robot communities exploring scalable, nonlinear indoor–outdoor worlds, limiting study of their complex interactions.The paper also identifies a need for physically realistic simulation and diverse, populated environments; existing approaches trade interactive richness against scene realism and input requirements.Source: §1 · p. 2
  • The platform unifies human-like avatars and diverse robots in a Genesis-based framework with physical and social interactions and distinct observation and action spaces.Source: §1 · p. 2
  • Its generation pipeline combines real-world geospatial data and generative models to create scalable outdoor and indoor scenes, annotations, and scene-grounded agent communities.Source: §1 · p. 2
  • Community Planning covers daily assistance and social connection, while Community Robot tests robot cooperation in indoor and dynamic open-world tasks.Source: §1 · p. 2
  • Virtual Community enables unified simulations of human and robot communities in generated open worlds, exceeding existing solutions in scope and capability.The authors identify studying embodied general intelligence alongside complex, interconnected human communities as a motivating direction.Source: §1 · p. 2

2 RELATED WORKS

Related work spans embodied simulation, social intelligence, and generative models, but the paper identifies limitations in open-world scale and integration. Virtual Community combines generative scene and community construction with its simulation platform.

Embodied AI Simulation

  • Many embodied simulators focus on indoor household tasks; existing outdoor platforms lack diverse, scalable environments for larger populations and complex interactions.Source: §2 · p. 3

Embodied Social Intelligence

  • Embodied social intelligence studies often use small populations, constrained household settings, or 2D worlds, while open-world planning and learning face adaptability and scalability challenges.The passage notes MARL's growing state and action spaces and resource demands, and that planning models may rely on assumptions that limit generalization.Source: §2 · p. 3

Foundation and Generative models for Embodied AI

  • Prior foundation-model work addresses agents, scenes, or task generation; Virtual Community integrates generative models into a simulator to build expansive scenes and agent communities.Source: §2 · p. 3

3 GENERATING OPEN WORLDS FOR SIMULATION

Virtual Community builds simulation-ready open worlds from geospatial data, populates them with scene-grounded communities, and simulates human avatars and diverse robots in one framework.

3.1 SCALABLE 3D SCENE CREATION

  • Geospatial scenes can contain noisy geometry and lack ground-level visual detail; the pipeline simplifies and reconstructs meshes, refines textures, and adds interactive objects.It also uses geospatial annotations for places, buildings, and transit, and combines retrieval and generation for multi-room indoor scenes.Source: §3.1 · pp. 3–5
  • The pipeline generates scenes at arbitrary locations and produced 35 annotated scenes from cities worldwide.Figure 2 depicts scene and agent generation from real-world geospatial data, followed by simulation; Figure 3 describes clean geometry and realistic textures as supporting physics simulation and real-world style fidelity.Source: §3.1 · p. 4, Figure 2 · p. 4, Figure 3 · p. 5

3.2 AGENT COMMUNITY GENERATION

  • An LLM uses scene details and agent appearances to generate character profiles and social relationships organized into groups with shared places and activities.A validator checks that referenced places exist and prompts the LLM to correct mismatches.Source: §3.2 · p. 5
  • Human-like avatars use diverse skins and high-fidelity meshes generated from synthetic face images.Source: §3.2 · pp. 5–6

3.3 UNIFIED SIMULATION FOR HUMAN AND ROBOT COMMUNITY

  • The unified framework simulates human agents and robots, implementing avatar simulation while largely inheriting robot simulation from Genesis.Source: §3.3 · p. 6
  • Avatar motions use SMPL-X poses and Mixamo clips, with collision detection and kinematic attachments for object and vehicle interactions.Generated agent schedules include activity times, descriptions, locations, and commute time between locations.Source: §3.3 · p. 6
  • The simulator supports drones, quadrupeds, humanoids, wheeled robots, and mobile manipulators, sharing a simulation loop with avatars but using distinct control frequencies.Robot controllers expose selected action spaces, and decomposed building meshes support faster background collision detection.Source: §3.3 · p. 6

4 OPEN WORLD MULTI-AGENT PLANNING

The Community Planning Challenge evaluates cooperative daily assistance and competitive social influence in open-world communities. Its baseline results expose challenges in task planning, navigation, and persuasion.

4 OPEN WORLD MULTI-AGENT PLANNING

  • The Community Planning Challenge evaluates multi-agent planning in outdoor and indoor environments through cooperative assistant tasks and competitive community influence.Assistant-task metrics are success rate, time consumed, and—on carry—human following rate; results average 24 scenes.Source: §4 · p. 6, Table 1 · p. 7, §4.1 · p. 7

4.1 COMMUNITY ASSISTANT TASKS

  • Assistant tasks cover carrying items while following people, delivering objects, and searching for targets, with agents evaluated in single- and two-assistant settings.Carry tasks involve multiple objects and waypoints, requiring adaptive scheduling, routing, and task-dependency management.Source: §4.1 · pp. 6–7

Baselines

  • No single baseline prevails across all three assistant tasks: heuristics perform strongly on delivery, while the LLM agent leads search but struggles on carry and delivery.The authors attribute common failures to underestimating navigation and search costs; the LLM agent also struggles to track task progress from action history.Source: Table 1 · p. 7, Baselines · p. 8
  • Removing explicit distance modeling degrades both LLM and MCTS planners, especially MCTS; the LLM planner can exceed its baseline in the two-assistant setting.The passage suggests cooperation helps offset the missing distance modeling in that setting.Source: Baselines · p. 8, Table 2 · p. 8

4.2 COMMUNITY INFLUENCE TASK

  • In the influence task, two main agents navigate to community members and use dialogue to persuade them, with friendship rankings recorded at each day’s end.Each community has thirteen other members, and the main agents have access to members’ daily schedules.Source: §4.2 · pp. 8–9
  • The o1-backed agent achieves higher average friendship rankings and conversion rates than gpt-4o in most communities, while Table 3 finds dialogue generation contributes more than target selection.The ablation reports that stronger backbones improve both components; Figure 4 compares agents across five communities.Source: §4.2 · p. 9, Figure 4 · p. 9, Table 3 · p. 9

5 OPEN WORLD MULTI-ROBOT COOPERATION

The Community Robot Challenge tests heterogeneous robots cooperating on open-world delivery and human-assistance tasks. Results highlight the difficulty of manipulation and following people in dynamic environments.

5 OPEN WORLD MULTI-ROBOT COOPERATION

  • The challenge builds on carry and delivery tasks, asking robots to transport objects or pick up and carry a human avatar’s personal items while following them.These scenarios add physics-level collaboration in open-world environments.Source: §5 · pp. 9–10
  • The evaluated robots are a mobile manipulator and a four-wheel carrier, with observations including RGB–D images, segmentations, base pose, and task information.Their action spaces comprise 11 degrees of freedom for the manipulator and four for the carrier.Source: §5 · p. 10
  • The heuristic baseline uses A* navigation and inverse-kinematics grasp planning with RRT-Connect; the implemented VLA baseline achieved near-zero performance.The passage does not provide the VLA result’s numerical value.Source: §5 · p. 10
  • All baselines score higher on delivery than carry, performance drops significantly without an oracle grasp, and the heuristic planner outperforms RL on the two robot tasks.The authors attribute the heuristic advantage to explicit configuration-space path planning, whereas RL must discover effective control sequences under sparse rewards; Table 4 averages results over 21 scenes.Source: §5 · p. 10, Table 4 · p. 10
  • Separating RL into reach and place policies slightly improves performance on both tasks.Table 5 reports the decomposed-policy ablation.Source: §5 · p. 10, Table 5 · p. 10

6 CONCLUSION

Virtual Community combines open-world scene and community generation with physically realistic simulation of avatars and robots. It introduces the Community Planning and Community Robot challenges, while noting that outdoor scenes do not yet accurately reflect real-world physical and visual properties.

6 CONCLUSION

  • The platform supports scalable open-world scene and community generation, physically realistic simulation of avatars and robots, and two evaluated challenges: Community Planning and Community Robot.The conclusion identifies insufficient outdoor-scene detail as a limitation for accurately representing real-world physical and visual properties.Source: §6 · pp. 10–11

A 3D SCENE GENERATION DETAILS

  • The scene pipeline takes latitude, longitude, and radius to produce textured meshes and simulator-loadable object data, using aligned tiles, filtered terrain heights, mesh simplification, texture transfer, and enhancement.The pipeline uses OpenStreetMap annotations for terrain construction and diffusion models for texture inpainting and super-resolution.Source: A 3D SCENE GENERATION DETAILS · p. 19, §A.1 · p. 19, §A.2 · p. 20

A.3 STREET VIEW REPROJECTION DETAILS

  • Street-view reprojection refines camera poses, inpaints dynamic objects, projects corrected images onto meshes, and aligns color and exposure across images.Camera pose correction uses LoFTR keypoint matches and the 5-point method; the pipeline also applies view-frustum clipping and greedy camera-mesh matching.Source: §A.3 · pp. 20–21

A.4 INDOOR SCENE GENERATION PIPELINE

  • Indoor generation retrieves matching layouts or populates empty rooms with detected and positioned large objects, then uses language models to place small objects; scenes also receive place, building, and transit annotations.Virtual Community currently provides 33 amenity types and two transit amenities: bus stops and bicycle stations.Source: §A.4 · p. 21, §A.5 · p. 21

B.1 AVATAR MODELS AND MOTIONS

  • Avatars use skeletal motion mapped to SMPL-X, while scene-grounded profiles and social networks inform schedules with explicit activity times, locations, and commute durations.Avatar meshes have 71 joints and are decimated to reduce vertex count by 90%; initialization rejects positions not connected to the map boundary.Source: §B.1 · p. 22, §B.2 · p. 23

C.2 TRAFFIC CONTROL

  • Traffic simulation builds road networks from OpenStreetMap data, converts them to OpenDRIVE, and coordinates vehicles and pedestrians using path planning, collision avoidance, and simplified traffic rules.Rules include blocking junction entry until clear, choosing less congested routes, pedestrian collision avoidance, and lane changes under congestion.Source: §C.1 · p. 23, §C.2 · pp. 23–24

D ROBOT SIMULATION

  • The robot setup includes mobile manipulators, quadrupeds, humanoids, and carriers, with a nested simulation loop balancing slower avatar control and faster robot control.Each one-second outer loop contains 100 robot inner steps, each executing a physics frame, avatar-animation frame, and robot observation-control step.Source: D ROBOT SIMULATION · p. 24, §D.1 · p. 24, §D.2 · pp. 24–25

E.1 SCENE BENCHMARKING

The generated scenes span diverse real-world locations and are benchmarked for visual fidelity and geometric complexity. The reported comparison finds improved visual effects and reduced geometric complexity relative to the original Google 3D Tiles data.

E.1 SCENE BENCHMARKING

  • The authors generated 35 scenes across 17 countries in North America, Europe, and Oceania, each measuring 800m × 800m.Figures show generated scenes and close-up views of their geometry and textures.Source: §E.1 · p. 25, Figure 11 · p. 26, Figure 12 · p. 26
  • Generated scenes significantly improve visual effects and reduce geometric complexity relative to original Google 3D Tiles data.Visual fidelity is measured with FID and KID against Google Street View images; geometric complexity uses average mesh faces per scene, excluding roof faces.Source: §E.1 · p. 25, Table 8 · p. 26

E.2 SIMULATION BENCHMARKING

This section benchmarks simulation speed and memory, describes avatar observations and task generation, and reports planning-challenge findings alongside planner and reward implementations.

E.2 SIMULATION BENCHMARKING

  • Speed benchmarking records physics frames per second under RGB observations at 1 Hz and depth rendering at 100 Hz, varying building and avatar counts on one NVIDIA A100 GPU.The supplied passages describe the benchmark settings but do not state its speed results.Source: §E.2 · p. 27, Figure 13 · p. 27
  • From 1 to 50 agents, peak RAM rises from 17.45 GB to 38.46 GB, average RAM from 15.11 GB to 36.01 GB, and peak GPU memory from 4.38 GB to 14.26 GB.The authors report approximately linear memory scaling, averaging about 0.43 GB peak RAM, 0.43 GB average RAM, and 0.20 GB peak GPU memory per additional agent.Source: §E.2 · p. 27

F.1 AVATAR OBSERVATION SPACE

  • Avatar observations include RGB and depth images, pose and position information, nearby-agent messages, and other states such as cash, accessible locations, and action status.The observation space also includes segmentation maps and camera extrinsics.Source: §F.1 · p. 28

F.2 TASK GENERATION

  • Procedural task generation selects valid places and objects, positions assets, and creates evaluation metadata that the simulator checks after task completion.Outdoor objects are placed at unoccluded points; indoor objects go on selected semantic surfaces.Source: §F.2 · p. 28

F.3 ADVANCED SETTING OF THE COMMUNITY PLANNING CHALLENGE

  • In the advanced planning setting, baseline performance drops as the task region grows; HP+LLM outperforms other two-assistant combinations, while LLM+LLM fails to generate an optimal plan.Most baselines struggle to finish within the step limit, with time consumption approaching that maximum.Source: §F.3 · p. 29, Table 9 · p. 29

F.4 IMPLEMENTATION OF THE HEURISTIC PLANNER

  • Heuristic planners use expert-defined finite-state automata, with separate logic for each of the three tasks.The delivery-task automaton is illustrated in Figure 15.Source: §F.4 · pp. 29–30, Figure 15 · p. 29

F.5 REWARD DESIGN FOR COMMUNITY ROBOT CHALLENGE

  • The mobile-manipulator reward sums object-distance and optional goal-distance penalties with grasp and placement bonuses.The placement bonus is triggered when the cube remains within 0.1 m of the goal for 100 consecutive steps.Source: §F.5 · p. 30

F.6 VLA BASELINE IMPLEMENTATION

The VLA baseline is fine-tuned on Virtual Community pick-and-place trajectories for the mobile manipulator, but achieves near-zero performance on the challenge tasks. The authors identify task, visual-domain, and training-data gaps as possible reasons.

F.6 VLA BASELINE IMPLEMENTATION

  • The baseline uses a pretrained π0 backbone fine-tuned by behavior cloning on approximately 3,000 successful pick-and-place trajectories collected in Virtual Community.Initial robot poses were randomly sampled, and training used a learning rate of 5e-5, batch size 32, and up to 5e4 steps with validation-loss early stopping.Source: §F.6 · pp. 30–31
  • At each step, the model receives multimodal observations including manipulator joint positions and velocities, and outputs joint increments executed at 20 Hz.The passages specify joint states and control frequency; they do not enumerate the full multimodal observation set.Source: §F.6 · p. 31
  • Despite successful fine-tuning, the VLA baseline achieves near-zero performance on Community Robot Challenge tasks.The authors cite the gap from tabletop to ground-level picking, outdoor visual-domain differences, and insufficient data and training iterations as reasons.Source: §F.6 · p. 31, Table 11 · p. 31

G SINGLE AGENT TASK

Community Commute tests whether an embodied agent can plan daily routes across an open-world transit system. The section specifies the task interfaces, evaluation measures, and planning baselines.

G SINGLE AGENT TASK

  • Community Commute asks an agent to minimize time and energy while traveling between 4–8 scheduled places, using buses or rental bikes as well as walking.Routes average 2.5 km; buses can be boarded only at stops when they arrive, and bikes must be returned to a station.Source: G SINGLE AGENT TASK · p. 31
  • The evaluation covers 100 episodes across 10 daily schedules and 10 scenes, measuring travel time, price, walking distance, and late rate.Each episode assesses a full day of commutes; agents complete 5.5 commutes per episode on average.Source: G SINGLE AGENT TASK · p. 31, §G.2 · p. 32

G.1 OBSERVATION SPACES

  • Agents receive RGB, depth, pose, camera, nearby-agent messages, and state observations, while commute actions include walking, turning, and boarding or leaving buses and bikes.Bike riding replaces walking and turning actions until the agent dismounts.Source: §G.1 · p. 32, §G.2 · p. 32

G.3 BASELINES

  • The baselines include direct walking, LLM-generated commute plans, MCTS planning, and PPO- or A2C-based reinforcement-learning planners.The RL planners select actions directly from observations rather than using the hierarchical decision-making framework used by MCTS and LLM planners.Source: §G.3 · pp. 32–33
  • MCTS uses UCB1 selection and simulates walking, bus, and bike choices with a reward balancing target progress and travel cost.The reward trade-off parameter α is set to 1; nodes can branch to walking or available bus stops and bike stations.Source: §G.3 · p. 33

G.4 RESULTS

In Community Commute, the rule-based walker leads on time, cost, and late rate, while consuming nearly twice the walking distance of GPT-4o. The supplementary sections also describe the interface, computational requirements, physics features, and simulation scope.

G.4 RESULTS

  • The rule-based walker performs best on travel time, cost, and late rate, but walks nearly twice the distance of the GPT-4o LLM agent.MCTS and LLM planners struggle to use transit effectively; RL agents have longer travel times and higher late rates than other low-level navigation planners.Source: Table 12 · p. 33, §G.4 · pp. 33–34

H GRAPHICAL USER INTERFACE AND VISUALIZATION TOOL

I COMPUTATIONAL RESOURCES

  • Scene-generation experiments typically require about four hours and 50 GB of GPU memory, excluding parallelizable inpainting and upscaling.A simulation episode requires at least 12 GB of system RAM and 6 GB of GPU memory; 15 GB RAM and 8 GB GPU memory are recommended for smoother runs.Source: I COMPUTATIONAL RESOURCES · p. 34

J.1 PHYSICS SIMULATION AND REAL-WORLD FIDELITY

  • Built on Genesis, Virtual Community supports physics-based interactions including deformable objects, robot collision meshes, and avatar collision detection.Figure 18 depicts a demo fountain and robot collision meshes.Source: §J.1 · p. 34, Figure 18 · p. 35

J.2 ETHICS DISCUSSION

  • The characters and profiles are LLM-generated rather than real people, and the Community Influence task is conducted entirely in simulation without human subjects.Scenes are produced through an online API pipeline for physics-engine use, without offline storage or redistribution of the cited map and place data.Source: §J.2 · p. 36

J.3 LIMITATIONS AND FUTURE WORK

The authors identify scene visual quality and supported agent population as current limitations, then outline future platform expansions. The prompt appendix gives templates for schedule and commute planning, community influence, and assistant-robot tasks.

J.3 LIMITATIONS AND FUTURE WORK

  • Generated-scene visual quality remains limited by insufficiently dense and comprehensive 2D data, while simulation typically supports 15–25 agents per scene.Future work includes richer scene details and objects, larger agent populations, dynamic social events and weather, and improved rendering and texture fidelity.Source: §J.3 · p. 36

K PROMPTS FOR LLM PLANNERS

  • The LLM prompt templates cover day-schedule generation and commute planning using character, place, schedule, and transit information.The commute template asks the agent to choose among bus, bike, and walking options and produce a stepwise plan.Source: K PROMPTS FOR LLM PLANNERS · pp. 36–37
  • Community Influence prompts ask agents to choose potential friends and generate dialogue, while assistant-robot prompts cover task actions and coordination messages.The two-assistant setting includes conversation history in the planning prompt and a separate communication prompt.Source: K PROMPTS FOR LLM PLANNERS · pp. 37–40
Loading 2508.14893v3…