MineLand: Simulating Large-Scale Multi-Agent Interactions with Limited Multimodal Senses and Physical Needs

1 Introduction

• MineLand combines large-scale scalability, limited multimodal senses, and physical needs to foster more ecologically valid multi-agent interactions.Limited access to visual, auditory, and environmental information prompts communication; physical needs include food and resource management.Source: §1 · p. 2
• The benchmark includes 4,499 programmatic tasks, 1,536 creative tasks, and 18 hybrid tasks combining features of both categories.The programmatic and creative task counts are each twice those of MineDojo.Source: §1 · p. 2
• MineLand allows customization of agent numbers and supports cooperative and competitive modes.Source: §1 · p. 3
• MineLand supports 64 agents, x8 times than before; reported findings also include social dynamics, challenging benchmark tasks, and more effective teamwork with reduced workload per agent.The authors also report that Alex’s multitasking mechanism is beneficial.Source: §1 · p. 3
• The paper presents MineLand, its benchmark, and Alex as three contributions intended to advance multi-agent simulation and embodied AI research.The introduction situates the work against simulators that struggle with scale and assume perfect information and limitless capabilities.Source: §1 · pp. 1, 3

2.1 Architecture

• Existing Minecraft simulators run a game client per player, while MineLand simplifies each client to a thread and supports 64 or more agents on a consumer desktop PC.The architecture comprises the Bot, Environment, and Bridge modules.Source: §2.1 · pp. 3–4

2.2 Observation Space

• Agents receive tactile, auditory, and first-person visual information, with perception constrained by distance, environmental obstructions, and direction.Source: §2.2 · p. 4

2.3 State Space

• The state space blends task-oriented activity with daily-life rhythms, including physical needs such as oxygen and hunger.Examples include cooking to meet food needs alongside jobs such as lumberjack or farmer.Source: §2.3 · p. 4

2.4 Action Space

• MineLand supports low-level actions and code-based high-level actions for complex tasks, alongside expanded task categories and cooperative or competitive multi-agent modes.High-level actions can generate sequences of low-level actions that continue, skip steps, or be interrupted by special events.Source: §2.4 · pp. 4–5, Figure 3 · p. 5

2.5 Communication

• Communication uses auditory information, visually perceived body language, and text, with channels constrained by distance rather than centralized broadcast.New messages can interrupt executing high-level action code, allowing agents to switch actions before prior code finishes.Source: §2.5 · p. 5

3 MineLand Benchmark Suite and Dataset

• The benchmark doubles MineDojo’s programmatic and creative task quantities, adds construction and stage-performance hybrid tasks, and supports flexible player counts and cooperative or competitive modes.Competitive mode can compare AI-agent capabilities and support adversarial-learning algorithms.Source: §3 · p. 6

3.1 Programmatic Task

• MineLand provides 4,499 programmatic tasks, with customizable parameters including agent count and cooperative or competitive mode.Source: §3.1 · p. 6

3.2 Creative Task

• The dataset contains 1,536 creative tasks and is compatible with MineDojo tasks.Source: §3.2 · p. 6

3.3 Hybrid Task

• Hybrid tasks use multiple references rather than unique ground truth and include construction and stage performance, evaluated with VLM and human ratings or LCS-based and human scores.Construction ratings range from 1 to 5; stage-performance scores assess behavioral completeness, coherence, and naturalness.Source: §3.3 · pp. 6–7

• Alex replaces the conventional LLM with a VLM, improves memory, planning, and acting components, and adds multitasking alongside predefined personality traits.Source: §4 · p. 7
• Interrupts control attention across tasks, while Alex’s Memory Library saves and restores internal states during rapid switches between communication and goal-driven work.This mechanism supports simultaneous coordination and scheduling across multiple tasks.Source: §4 · p. 7

5.1 Experiments of Simulators Performance

• With vision disabled, MineLand ran 4× as many concurrent agents; at eight agents, its CPU and memory usage were 35.6% and 38.0% of Malmo’s.Source: Table 2 · p. 8, §5.1 · p. 8

5.2 Experiments of Social Dynamics

• Competition reduced code iterations but produced more errors, while high extraversion and agreeableness yielded cooperation in >8/10 trials versus 0/10 without assigned personality.Source: Table 3 · p. 8, §5.2 · p. 8

5.3 Experiments of Construction Tasks

• Alex’s construction abilities remain limited by complex planning and high-precision manipulation: it cannot plan auxiliary structures or perform fine-grained actions with its available APIs.Source: §5.3 · p. 8

5.4 Experiments of Stage Performance Tasks

• In stage performances, single-agent tasks typically scored highly across all three metrics, while multi-agent keypoint scores tended to exceed appropriateness and collaboration scripts drew lower human scores due to unnecessary dialogue.Source: §5.4 · pp. 8–9, Table 5 · p. 9

5.5 Experiments of Multi-Agent Cooperation

• Cooperation reduced per-agent code iteration cost by 20% versus independent work excluding chat cost, though coordination required extra iterations.Source: §5.5 · p. 9

5.6 Experiments of Multitasking

• Across ten runs, Alex with multitasking handled concurrent events and prioritized higher-priority events, whereas Alexw/o mt failed to respond timely and was killed by a zombie.Source: §5.6 · p. 9, Table 6 · p. 9

6 Conclusion

• MineLand supports 64 or more agents with limited multimodal senses and physical needs, prompting communication and collaboration in more ecologically valid interactions.Agents have needs including sleep, food, and resource management.Source: §6 · p. 10, §A.1 · p. 14

B Broader Impact

• MineLand’s simulator, benchmark, and agent framework support research on more authentic interactions and may inform AI applications such as human-computer interaction, robotics, and game design.The section also identifies studying human social dynamics as a potential use of agent interactions in the simulator.Source: B Broader Impact · p. 15

C.4 Supporting up to 100 Agents

• A simultaneous combat experiment with 100 agents demonstrates MineLand’s ability to manage many agents acting concurrently.The experiment is shown in Figure 5.Source: §C.4 · p. 17, Figure 5 · p. 17

D Details of MineLand Benchmark Suite

• The MineLand Benchmark Suite spans programmatic, creative, and hybrid tasks, with programmatic tasks covering survival, harvest, tech-tree, and combat objectives.Task definitions are formalized so developers can add new tasks.Source: D Details of MineLand Benchmark Suite · p. 17

E Details of Alex

• Alex balances individual daily-life goals with community-based tasks such as resource gathering, crafting, and construction.Its VLM-based design addresses agents’ need to handle daily routines alongside assigned tasks.Source: E Details of Alex · p. 18

E.2 Hierarchical Planning Component

• Alex uses hierarchical planning to connect long-term community goals and individual aspirations with short-term tasks, interleaving daily routines with task-oriented actions.This allows agents to pursue broader objectives in the context of daily life.Source: §E.2 · p. 20

E.3 Action Module

• Alex’s action module converts plans into executable code and checks execution, enabling self-correction when actions fail or deviate from the plan.MineLand agents can generate code through the Mineflayer API, avoiding the extra plan-to-code mapping step that can accumulate errors.Source: §C.3 · p. 17, E Details of Alex · p. 19, §E.3 · p. 20

E.4 Multitasking Mechanism

• Alex prioritizes queued events by urgency, deferring lower-priority interruptions or switching context to address events that need attention.The framework also supports handling two tasks concurrently in specific scenarios.Source: §E.4 · pp. 20–21

F Details of Hyper-Parameters

• MineLand defaults to survival mode, peaceful difficulty (normal in combat tasks), view-distance 6, simulation-distance 3, and cooperative task mode.These settings apply to the MineLand Minecraft server by default.Source: F Details of Hyper-Parameters · p. 21

G Experiments of Minecraft Simulator Performance Comparison

• The performance comparison covers different simulator conditions, including MineLand’s visual and headless modes.The table caption identifies MineLand* as an optimized version with significantly reduced maximum CPU usage.Source: G Experiments of Minecraft Simulator… · p. 21, Table 9 · p. 22
• MineLand’s agent capacity exceeds the tested limits of the comparison platforms, including Malmo’s result of eight agents.The passage notes that original MineDojo and MineRL support one agent, MarLÖ defaults to two, and Malmo originally supported one.Source: G Experiments of Minecraft Simulator… · p. 21
• 32 agents with display, 128 headless; at eight agents, MineLand uses 35.6% of Malmo’s CPU and 38.0% of its memory.The comparison used a desktop with an Intel i5-12400F CPU and 64GB memory.Source: G Experiments of Minecraft Simulator… · p. 21, Table 9 · p. 22

H Experiments of Multimodal Observation

• The comparison measured average completion time over five attempts and recorded the agents’ travel paths.The objective was to explore from a mountain summit to find the ocean.Source: H Experiments of Multimodal Observation · p. 22, Figure 8 · p. 23
• Nearly twice the success rate and about half the task completion time were reported for MineLand versus MineLandw/o vision.Completion time excludes unfinished tasks.Source: Table 10 · p. 22, H Experiments of Multimodal Observation · p. 22
• Visual information helps agents locate the ocean and choose directions, while agents without vision choose movement directions randomly.The task began at a mountain summit and imposed a 100-second limit, excluding decision-making time.Source: H Experiments of Multimodal Observation · pp. 22–23

I Experiments of Reinforcement Learning Agent

• MineLand’s RL functionality is an ancillary capability; the authors’ core focus remains LLM/VLM agents.The RL example uses Stable Baselines3 with low-level actions.Source: I Experiments of Reinforcement… · p. 23
• For the “combat 1 zombie” task, the DQN with MLP policy’s reward curve rises, indicating learning strategies for higher rewards, then stabilizes later, suggesting convergence to a stable policy.The curve covers 3000-5000 total time steps. The reward is the amount of blood volume lost by the zombie.Source: Figure 9 · p. 23, I Experiments of Reinforcement… · p. 23

J Experiments of Limited Senses

• In sheep shearing, an agent with limited vision cannot shear a sheep outside its field of view, but it can communicate with other agents to locate one.The limited-vision setting uses a view distance of 6 and a 250ms visual refresh interval; long view distance is used as the unlimited-vision condition.Source: J Experiments of Limited Senses · p. 24, Figure 10 · p. 24

K Experiments of Physical Needs

• Alex with physical needs survives one day in its shelter, while the same agent without physical needs fails to eat and is killed by a zombie.The test starts with health 1, satiety 0, and two breads in inventory.Source: K Experiments of Physical Needs · p. 24
• With health at 1 and food at 0, Alex plans to consume two breads, collect 20 dirt, and build a simple shelter.After eating, the plan prioritizes shelter to avoid threats and allow health regeneration.Source: K Experiments of Physical Needs · p. 24
• In repeated building-task experiments, Alex with physical needs lives longer and tends to eat before building shelter, unlike the version without needs, which explores and gathers resources.The authors cite this contrast as evidence that physical needs matter for simulating real life.Source: K Experiments of Physical Needs · p. 25

L Experiments of Single Agent

• Alex obtained diamonds in 2 of 6 attempts while progressing through a sequence of prerequisite items.With limited senses, Alex could locate a target only when it could visually discover or reason that the target was nearby; Voyager received target locations directly from the system.Source: L Experiments of Single Agent · p. 25

M Experiments of Construction Tasks

• Despite its limited construction abilities, Alex showed promise in selecting appropriate materials for Monument and Stone Stele construction.Figure 12 illustrates Alex completing the construction task.Source: M Experiments of Construction Tasks · p. 25, Figure 12 · p. 26

N Experiments of Stage Performance Tasks

• Alex completed stage-performance examples for “Make friends,” “Exchange items,” and “Cook food.”The supplied tables illustrate the three tasks but do not report performance scores.Source: Table 11 · p. 26, Table 12 · p. 27, Table 13 · p. 27

O.1 Experiments of Simulating Conformity Phenomena

• Conformist agents followed the majority, even when it chose incorrectly, while independent agents chose correctly regardless of others’ opinions.The conformist agent was an exception when it was the only agent doing the task.Source: §O.1 · p. 27, Table 14 · p. 28

O.2 Experiments of Simulating Personality Traits

• In a 48-agent treasure hunt, all high-trait agents planned to cooperate, while only a small portion of low-trait agents did.The task instructions allowed either interacting or minimizing interactions; the reported pattern was described as consistent with the five-factor model of personality.Source: §O.2 · p. 28, Table 15 · p. 28

P Experiments of Comparing Different VLMs and LLMs

• Table 16 compares Alex using different VLMs or LLMs for its action component on two white-wool harvesting tasks.Fractions in the table count successful completions among three attempts; 0/3 denotes failure within 15 code iterations or exceeding the task area.Source: P Experiments of Comparing Different… · p. 28, Table 16 · p. 28

Q Evaluation Metrics For Construction Tasks

• Construction evaluations combine VLM-based and human scoring on a 1–5 scale, with higher scores indicating closer resemblance to the blueprint.The scoring criteria are shown in Table 17.Source: Q Evaluation Metrics For Construction… · p. 29, Table 17 · p. 29

R Evaluation Metrics For Stage Performance Tasks

• Stage-performance tasks use human scores from 1 to 5, with higher scores indicating greater script alignment and naturalness.The scoring criteria are shown in Table 18.Source: R Evaluation Metrics For Stage… · p. 29

S Limitations

• Visual misidentification can obstruct progress and cause task failure, even with a strong VLM such as GPT-4.In one example, Alex mistook wood for a crafting table, placed it, and consequently could not craft items.Source: S Limitations · p. 29
• Construction is constrained by limited long-term planning and high-precision manipulation; Alex lacks the low-level APIs needed for fine-grained actions.The authors suggest richer low-level APIs and actions as a possible way to improve construction abilities.Source: S Limitations · pp. 30–31

T Prompts

• Figures 16 and 17 show portions of the prompts for the short-term and long-term plan generators.Source: Figure 16 · p. 32, Figure 17 · p. 33