Source-linked AI summary
Minimal Local Simulation Foundations for LLM- and VLM-Driven Agents in 2D and 3D Environments
Ryuki Hyodo
TL;DR
Contemporary LLM- and VLM-driven simulation platforms can be difficult to study, modify, or run on ordinary computers. This paper presents two minimal, locally operated foundations that expose the perception–reasoning–action loop for education, rapid prototyping, and domain-specific extensions.
Problem
Many contemporary simulation platforms obscure the basic perception–reasoning–communication–action–observation loop through platform complexity, remote services, specialized simulators, and substantial compute requirements.
Method
The paper releases SD-AgentFoundry-2D for locally hosted LLM-driven multi-agent simulation and SD-AgentFoundry-3D for locally hosted VLM-driven embodied simulation, using compact, replaceable mechanisms and desktop operating-system setup paths.
Results
The two foundations provide complementary environments for studying social interaction and embodied decision making through configurable scenarios, visualizations, and auditable logs.
Takeaways & Limitations
The codebases offer accessible, extensible starting points for students, researchers, and practitioners to develop scenarios, agent designs, measurements, interfaces, and safeguards.
Takeaways & Limitations
Representative runs are qualitative usage examples rather than controlled evaluations, and the 3D VLM loop maps text through a deterministic interpreter rather than learning embodiment-specific actions.
Abstract
from arXiv · showhide
Large language models (LLMs) and vision-language models (VLMs) are expanding the range of behaviors that can be represented in agent-based simulations, but many contemporary platforms are difficult to study, modify, or run on ordinary computers. We present two intentionally minimal simulation foundations for education and rapid prototyping. SD-AgentFoundry-2D provides a two-dimensional multi-agent environment in which locally hosted LLM agents move, communicate, respond to place occupancy, and encounter spatially localized fire events. SD-AgentFoundry-3D provides a three-dimensional digital-twin environment in which a locally hosted VLM receives first-person images and produces natural-language movement instructions. Both codebases are designed to run locally on macOS, Windows, and Linux and are deliberately left open to modification rather than developed as finished applications. Together, they offer accessible starting points for learning about generative social simulation and for building domain-specific extensions.
1 Introduction
Foundation-model agents broaden agent-based simulation beyond fixed behavioral rules, but accessible, modifiable local platforms remain needed for education and exploratory study. SD-AgentFoundry addresses this need with two compact foundations exposing the perception–reasoning–communication–action loop.
- Foundation models let simulators ask agents to interpret situations, use memory, communicate, and choose actions instead of following only fixed rules.
- Prior systems span populated social-computing prototypes, interactive two-dimensional towns, component-based modeling, large-scale social experiments, and urban-dynamics simulation.
- Simulated behavior can reproduce some human-study findings while exhibiting systematic distortions, so individual-level prediction requires human grounding and evaluation against human responses.
- Textual and two-dimensional environments efficiently represent communication and social organization but abstract away much spatial and perceptual structure relevant to physical action.
- SD-AgentFoundry presents compact local foundations for LLM-driven multi-agent social simulation and VLM-driven embodied simulation, rather than another feature-complete simulator.
- The codebases target students, researchers, and practitioners by exposing readable, replaceable mechanisms for developing scenarios, agents, measurements, and interfaces.
2 System Architecture
SD-AgentFoundry combines a discrete 2D LLM multi-agent simulator with a USD-based 3D VLM embodied simulator. Both use modular local components, bounded or interpreted interfaces, configurable environments, and auditable outputs.
- SD-AgentFoundry-2D: SD-AgentFoundry-2D uses a bounded integer grid with configurable places, capacities, personas, and stay-or-cardinal-movement actions.
- SD-AgentFoundry-2D: Communication depends on proximity and shared area, messages use the pre-movement neighborhood, and movements execute after all agents decide.
- SD-AgentFoundry-2D: The 2D simulator supplies numerical occupancy and localized fire information while leaving avoidance, warning, coordination, or inaction to the LLM.
- SD-AgentFoundry-2D: Local Ollama calls separately determine communication and movement-related decisions, while JSONL logs, PNG frames, YAML configuration, and replaceable interfaces support inspection.
- SD-AgentFoundry-3D: SD-AgentFoundry-3D loads USD scenes such as the Kibo digital twin and uses first-person rendering, VLM inference, natural-language interpretation, and bounded actions.
- SD-AgentFoundry-3D: The 3D system records raw responses, parsed commands, poses, boundary effects, timing, input-image paths, and effective configuration for reproducibility and auditing.
3 Example Runs & Beyond
The representative 2D and 3D runs demonstrate how the foundations connect agent decisions, communication or visual perception, movement, logging, and visualization. They are illustrative workflow examples rather than controlled evaluations, while the configurable environments support later behavioral studies.
- 2D representative run: The 2D example uses 20 LLM agents, two accessible places, natural-language communication, and two fires to illustrate responses to shared hazards.Agents use local information and received messages to decide movement and communication.
- 2D representative run: A single illustrative 2D snapshot shows fires, perception radii, agent locations, and pairs eligible for local communication.The example demonstrates integrated workflow components rather than causal behavioral patterns or validated human response.
- Research extensions: The 2D foundation supports controlled comparisons of LLM behavior by varying scenarios or holding conditions fixed while changing the model.Possible measurements include movement preferences, hazard sensitivity, warning propagation, communication frequency, and collective place choice.
- 3D representative run: A 3D example loads the Kibo USD scene and directs one VLM-controlled avatar to find and approach the JAXA logo using first-person visual input.The agent receives an RGB image, task prompt, and recent action history at each decision.
- 3D representative run: The 3D workflow connects scene loading, first-person rendering, VLM inference, action interpretation, movement, logging, and visualization in one illustrative run.Its snapshot is not a performance benchmark or evidence of reliable task completion.
- Research extensions: The 3D foundation supports studies of visual grounding, object search, wayfinding, viewpoint dependence, and language or scene-layout sensitivity across configurable digital twins.Researchers can vary scenes, tasks, poses, camera settings, movement limits, history length, model backends, and response interpreters.
- Scope boundary: The VLM-driven 3D loop maps natural-language outputs to bounded actions and is suitable for interpretable high-level navigation, not learned low-level control.Trained VLA policies instead connect visual and linguistic inputs directly to embodiment-specific actions and visuomotor skills.
4 Summary
LLM- and VLM-driven simulation can support exploration of social interaction and embodied decision making, but the two foundations target different abstraction levels. Their intentionally basic, local, and extensible implementations prioritize educational accessibility and modification over completeness.
- Overall contribution: LLM- and VLM-driven agents support exploration of social interaction and embodied decision making beyond fixed behavioral rules.The paper frames applications including hypothesis generation, planning, emergency communication, and service prototyping within the simulations' supported scope.
- Complementary foundations: SD-AgentFoundry-2D emphasizes communication and collective behavior in a transparent 2D world, whereas SD-AgentFoundry-3D emphasizes visual grounding and physical action in a 3D digital twin.The foundations therefore expose complementary levels of abstraction.
- Design goals: Both systems are basic, locally operated, and designed for macOS, Windows, and Linux, with educational accessibility and extensibility prioritized over completeness.Readable implementations, configurable scenarios, visualizations, and auditable logs are intended to support user modifications.
A Prompts Used in the Simulations
The appendix specifies the prompts supplied to the local models, including dynamic state, communication, action, memory, fire, and movement information. It also provides representative 2D and 3D prompt assemblies with explicit response conventions and history handling.
- A Prompts Used in the Simulations: Prompt templates use runtime-inserted values and omit conditional blocks when their information is unavailable.Angle-bracketed expressions denote runtime values, while conditional sections are omitted when corresponding information is unavailable.
- A.1.1 Communication prompt template: The 2D communication prompt asks an agent to choose an optional message for nearby agents using observations, experiences, thoughts, and current social context.It includes place occupancy, perceived fires, nearby agents, retained memories, and received messages, while communication is limited to nearby agents.
- A.1.1 Communication prompt template: The communication response is JSON-shaped with a message field and a brief reasoning field.Messages are capped at 200 words and may be omitted when the agent does not want to send one.
- A.1.2 Action-decision prompt template: The 2D action-decision prompt combines position, place geometry, occupancy, nearby agents, memories, messages, and perceived fire information before requesting an action.Available actions are stay or move one grid cell up, down, left, or right within the field boundaries.
- A.1.2 Action-decision prompt template: The action response specifies move or stay, an optional direction, memory for the next step, and reasoning in a JSON-shaped format.Direction is required only for movement, and the prompt supplies the selected communication message when one was chosen.
- A.2.1 Dynamic text-prompt template: The 3D VLM prompt combines a task, movement conventions, current pose, bounded recent history, and a conditional nudge after three idle decisions.The VLM receives this text together with the current first-person RGB image and is asked for one next movement.
- A.2.2 Example text prompt: The 3D example asks an agent inside the ISS Kibo module to find and approach a JAXA logo using a first-person image and a final single-movement sentence.Movement examples include moving forward, turning, looking up, or staying, with distances in metres and angles in degrees.