Baba Is AI: Break the Rules to Beat the Benchmark

1. Introduction

• The benchmark probes whether agents can distinguish relevant stimuli from distractor objects and rules, and combine previously seen rules in novel ways.These abilities are presented as components of human adaptation to novel environments.Source: §1 · p. 1
• Baba Is You motivates the benchmark because players can rearrange rules to change how the environment and agent behave.The benchmark studies learning and composing rules, as well as making or breaking them.Source: §1 · p. 1
• In the example game, agents must first deactivate [wall is stop], then create [door is win], and finally move onto the door.Rules are made by aligning movable text blocks; an active rule changes object properties.Source: §1 · pp. 1–2, Figure 2 · p. 1
• Baba Is AI is a simplified Baba Is You implementation based on Gymnasium Minigrid, evaluated with multimodal models on visual game inputs.The paper names GPT-4o, Gemini-1.5-Pro, and Gemini-1.5-Flash as the evaluated models.Source: §1 · p. 2
• The benchmark tests systematic compositionality in rule-based generalization by allowing agents to manipulate the rules that ground written commands in the environment.The authors describe this as a broader notion of generalization than current benchmarks.Source: §1 · p. 2

2. Method

• The game instructions define an active rule as three horizontally aligned blocks in the form “object is property.”Active rules modify object properties.Source: §2 · p. 2
• LLMs receive a static image of the initial environment and produce plans to break an active rule, activate a rule, or move to a specific object.The prompts constrain these actions to objects and rule blocks present in the environment.Source: §2 · p. 2
• After general game instructions, LLMs study 10 image-and-plan examples, derive plans from them, and describe an algorithm to apply to unseen test environments.The unseen environments are selected to assess different types of generalization.Source: §2 · p. 2
• Mean accuracy for all three models is lower in distractor environments that add a central wall, although [wall is stop] is inactive and does not affect movement.The wall and its rule blocks add distractors to the environments.Source: Figure 4 · p. 3
• Accuracy is exact match with the winning plan, measured on five samples per test environment across five random seeds.Each seed uses different in-context and test examples.Source: §2 · p. 3

3. Results

• GPT-4o is perfect on the first four distractor environments, while all models decline on the final task with both object and rule distractors; Gemini-1.5-Flash outperforms Gemini-1.5-Pro.The five environments increase distractors, culminating in an irrelevant active rule and a distractor object.Source: §3 · p. 3
• Adding inactive wall-rule blocks lowers mean accuracy for all three models relative to Figure 3, although the wall does not affect movement.Figure 4 adds a wall and its rule blocks as distractors to the Figure 3 environments.Source: §3 · p. 3
• Superficially similar environments with identical objects still require distinct winning solutions, adding another rule-manipulation challenge.The benchmark probes composition by testing a novel sequence of previously learned strategies.Source: §3 · pp. 3–4, Figure 6 · p. 4
• All three LLMs achieve low accuracy on the novel break{rule} then make{rule} then goto{object} composition.Accuracy remains low when the training and test strategies are alternated.Source: §3 · pp. 3–4, Figure 5 · p. 4

• Baba Is AI examines compositional generalization when agents can modify environmental rules, a capability static-rule benchmarks overlook.Source: §4 · p. 4
• Accuracy is low on challenging environments, where superficially similar layouts can require breaking wall is stop or switching control from Baba to the key.Source: §4 · p. 4
• LLMs make grounding mistakes by referring to nonexistent objects and path-planning mistakes by falsely judging clear paths blocked.Source: §4 · p. 5

• The prompt begins with general instructions describing the game and the task for the LLMs.Source: Figure 7 · p. 7
• Models receive 10 example grid images with winning plans and generate reasoning steps to derive each plan from its image.Source: Figure 7 · p. 7
• After the examples, models describe an algorithm and apply it to new environments that systematically differ from the in-context examples.Source: Figure 7 · p. 7

• GPT-4o makes a grounding mistake by referring to a blue ball that is absent from the environment.Source: Figure 8 · p. 8
• GPT-4o makes a path-planning mistake by claiming the key blocks the route to the door when the path is clear.Source: Figure 8 · p. 8