Source-linked AI summary
AgentSims: An Open-Source Sandbox for Large Language Model Evaluation
Jiaju Lin, Haoran Zhao, Aochi Zhang, Yiting Wu, Huqiuyue Ping, Qin Chen
TL;DR
LLM evaluation faces constrained abilities, vulnerable benchmarks, and unobjective metrics. AgentSims provides an interactive, program-based sandbox for building task evaluations; the paper presents task-based evaluation as a one-for-all response, while noting limits in simulation fidelity and fine-grained assessment.
Problem
Existing LLM evaluation is constrained, vulnerable to benchmark gaming, and often uses unobjective metrics, while old benchmarks become outdated.
Source: §1 · pp. 1–2Method
AgentSims provides an interactive, visual, program-based infrastructure where researchers build LLM evaluation tasks in a simulated town with buildings and residents.
Source: §1 · p. 2Results
Task-based evaluation is proposed as a one-for-all response, testing overall ability through diverse tasks and using task pass rates instead of black-box ratings.
Source: §1 · p. 2Takeaways & Limitations
AgentSims supports task design across disciplines and lets researchers combine planning, memory, and learning systems to study their effects.
Source: §1 · p. 2Simulation fidelity depends on LLM accuracy and building and equipment diversity; task pass rates miss fine-grained abilities such as math reasoning and do not explain success or failure.
Source: Limitations · p. 6
Abstract
from arXiv · showhide
With ChatGPT-like large language models (LLM) prevailing in the community, how to evaluate the ability of LLMs is an open question. Existing evaluation methods suffer from following shortcomings: (1) constrained evaluation abilities, (2) vulnerable benchmarks, (3) unobjective metrics. We suggest that task-based evaluation, where LLM agents complete tasks in a simulated environment, is a one-for-all solution to solve above problems. We present AgentSims, an easy-to-use infrastructure for researchers from all disciplines to test the specific capacities they are interested in. Researchers can build their evaluation tasks by adding agents and buildings on an interactive GUI or deploy and test new support mechanisms, i.e. memory, planning and tool-use systems, by a few lines of codes. Our demo is available at https://agentsims.com .
1 Introduction
The introduction argues that existing LLM benchmarks constrain evaluation to limited formats, remain vulnerable, and rely on metrics with objectivity problems. It proposes task-based evaluation and AgentSims as an accessible platform for designing and studying such tasks.
1 Introduction
- Task-based evaluation uses agents’ success at predefined goals in simulated environments to address limited ability coverage, benchmark vulnerability, and subjective or biased metrics.The authors argue that varied task settings and emergent behaviors are less likely to be represented in training data, while task pass rates avoid black-box ratings.Source: §1 · pp. 1–2
- AgentSims is an interactive, visualized, and program-based infrastructure for creating LLM evaluation tasks in an artificial town.Its stated aim is to streamline task design for researchers with varied backgrounds and programming proficiency.Source: §1 · p. 2
- LLM researchers can combine planning, memory, and learning systems to study the effects of different system designs.The introduction also notes that agent performance varies with support systems and calls for a standard implementation to support reproducibility.Source: §1 · p. 2
- Researchers from fields such as behavioral economics and social psychology can use the interactive UI to design maps and create agents.The paper presents this interface as a way to lower the entry threshold for those researchers.Source: §1 · p. 2
2 Related Work
The related work reviews LLM benchmarks spanning general abilities, exams, knowledge, dialogue, and safety, and multi-agent systems applied to cooperation and simulation. It identifies a gap in user-friendly, evaluation-oriented infrastructure for constructing benchmarks.
2.1 Benchmarks for Large Language Models
- Benchmarks reviewed cover general abilities, standardized exams, pretraining knowledge, multiturn dialogue, and safety, but the authors say existing datasets do not fully demonstrate LLM abilities.The cited examples include a dialogue benchmark limited to two turns for simplicity.Source: §2.1 · p. 2
2.2 Multi Agent Cooperation
- Prior multi-agent work applies cooperation to healthcare suggestions, role-playing, software development, and alignment, while sandbox agents can produce individual and emergent social behaviors.The passage describes these as applications of multi-agent cooperation and simulated societies.Source: §2.2 · p. 2
- The authors distinguish AgentSims from task-oriented multi-agent systems by its focus on easy benchmark construction and a user-friendly interface.They identify the absence of a standard paradigm for agent support systems as an additional gap.Source: §2.2 · pp. 2–3
3 Key Components
AgentSims combines LLM-driven agents and an interactive built environment. Its agents use planning, memory, and tool-use systems, while buildings and equipment provide the environment for agent interactions.
3 Key Components
- AgentSims combines LLM-supported generative agents with buildings and equipment that make up the sandbox environment.Source: §3 · p. 3
3.1 Generative Agents
- Agents use planning to decompose goals into subtasks, memory to retain and retrieve experiences, and tool use to learn equipment-operation skills from feedback.Source: §3.1 · pp. 3–4
3.2 Buildings and Equipment
- Buildings contain interactive equipment whose support functions return operation outcomes based on predefined rules or model-generated text.Agents can refine operations from feedback.Source: §3.2 · p. 4
4 Interaction scenarios
AgentSims offers a graphical User Mode for designing and participating in experiments, alongside a Developer Mode for extending agent systems and the simulated environment.
4 Interaction scenarios
- AgentSims separates an interactive User Mode for researchers with limited interest in support systems from a flexible Developer Mode for building support mechanisms.Source: §4 · p. 4
4.1 User Mode
- User Mode lets researchers create agents and buildings through a graphical interface, select agent goals, biographies, memory and planning systems, and intervene by interacting with agents.Buildings use preconfigured equipment, which can be modified in Developer Mode.Source: §4.1 · pp. 4–5
4.2 Developer Mode
- Developer Mode lets users add functions and personalize agents by adjusting abstracted LLM-backbone and support-system functions.Source: §4.2 · p. 5
- Developers configure buildings and equipment through JSON files; equipment can include support functions, and a separate configuration file specifies commodity purchases or salaries.Buildings are defined by type and contained equipment, represented by facility IDs in a two-dimensional array.Source: §4.2 · p. 5
5 Implementation
AgentSims uses a Python-based backend with a browser-accessible Unity WebGL frontend.
5 Implementation
- AgentSims runs on Python 3.91 and uses the provided requirements file installed through PyPI.Source: §5 · p. 6
5.1 Backend
- The backend uses Tornado, websockets for API calls and push notifications, and mysql-connector-python to access a MySQL database.Source: §5.1 · p. 6
5.2 Frontend
- The web client is built with Unity and embedded as WebGL, making it accessible through a browser after proxying with nginx.Source: §5.2 · p. 6
6 Example Application Tasks
AgentSims supports scenario-based evaluation of LLM social and long-term planning abilities, as well as social data generation and preliminary social-science experiments.
6.1 Subject LLM as participants
- Researchers can test an LLM’s social abilities by placing it among baseline agents, including GPT-4-driven colleagues in a hostile scenario.The subject agent’s emotional understanding and social adaptation can then be assessed.Source: §6.1 · p. 6
6.2 Subject LLM as mayor
- Researchers can assess long-term planning by making the subject LLM mayor or company president and analyzing task success under different difficulties.The agent may need to recruit residents, issue or revise policies, and establish buildings to address task requirements.Source: §6.2 · p. 6
6.3 Applications besides Evaluation
- AgentSims can generate socially situated data and support controllable preliminary social-science experiments, including hypothesis feasibility checks.The authors note that some social-judgment and participation data require more than a single prompt.Source: §6.3 · p. 6
7 Conclusion
AgentSims is a visualized, program-based infrastructure for constructing LLM evaluation sandboxes. It aims to help researchers build evaluation tasks and provides open code and documentation tutorials.
7 Conclusion
- AgentSims is a visualized, program-based infrastructure for constructing LLM test sandboxes.Source: §7 · p. 6
- The infrastructure aims to help researchers effectively build LLM evaluation tasks.Source: §7 · p. 6
- The project intends to make its code openly available and continuously update documentation with comprehensive tutorials.Source: §7 · p. 6
Limitations
AgentSims’ simulation is constrained by LLM accuracy and the diversity of its buildings and equipment, and cannot fully represent real-world cases. Task-based evaluation has limited fine-grained diagnostic reach, including for math reasoning and explaining task outcomes.
Limitations
- AgentSims’ simulation is limited by LLM accuracy and the diversity of its buildings and equipment.Source: Limitations · p. 6
- The sandbox can never fully reflect real-world cases.Source: Limitations · p. 6
- Task-based evaluation can hardly reflect fine-grained abilities such as math reasoning, and task pass rates do not explain why models succeed or fail.Source: Limitations · p. 6