PaperlayerView exact revision on arXiv

Source-linked AI summary

Baba is LLM: Reasoning in a Game with Dynamic Rules

Fien van Wetten, Aske Plaat, Max van Duijn

arXiv:2506.19095v1cs.AI

TL;DR

This paper asks whether LLMs can reason in Baba is You, a puzzle game where players manipulate text to change rules. It evaluates prompting and finetuning, finding that stronger models do better but still struggle with dynamic rule changes.

  • Problem

    The paper asks whether LLMs can reason in Baba is You, where players manipulate text to change rules and must understand the effects on game state.

    Source: §1 · pp. 1–2

  • Method

    The study tests six LLMs with three prompt types and finetunes two open models using textual and structural game data.

    Source: §1 · p. 2, Evaluation of Reasoning · p. 9

  • Results

    GPT-4o and Gemini Flash 1.5 outperform smaller models on identifying objects, interpreting mechanics, and generating partial solution paths, but still struggle to fully understand the game.

    Source: §5 · p. 15

  • Takeaways & Limitations

    Baba is You offers a challenging testbed for reasoning about dynamic rules and reflection by reasoning LLMs.

    Source: §5 · p. 16

    Evaluation covered 14 relatively simple levels, and finetuning used a small dataset of 15 level solutions and 298 game-mechanics questions.

    Source: Limitations & Further work · p. 16

Abstract

from arXiv · show

Large language models (LLMs) are known to perform well on language tasks, but struggle with reasoning tasks. This paper explores the ability of LLMs to play the 2D puzzle game Baba is You, in which players manipulate rules by rearranging text blocks that define object properties. Given that this rule-manipulation relies on language abilities and reasoning, it is a compelling challenge for LLMs. Six LLMs are evaluated using different prompt types, including (1) simple, (2) rule-extended and (3) action-extended prompts. In addition, two models (Mistral, OLMo) are finetuned using textual and structural data from the game. Results show that while larger models (particularly GPT-4o) perform better in reasoning and puzzle solving, smaller unadapted models struggle to recognize game mechanics or apply rule changes. Finetuning improves the ability to analyze the game levels, but does not significantly improve solution formulation. We conclude that even for state-of-the-art and finetuned LLMs, reasoning about dynamic rule changes is difficult (specifically, understanding the use-mention distinction). The results provide insights into the applicability of LLMs to complex problem-solving tasks and highlight the suitability of games with dynamically changing rules for testing reasoning and reflection by LLMs.

1 Introduction

Baba is You tests whether language models can reason about rules that players create by rearranging text blocks. The study evaluates prompting and finetuning as approaches to this challenge.

1 Introduction

  • Players manipulate text blocks to change rules, requiring models to understand how those changes affect game state and apply that understanding dynamically.The game’s mechanics contrast pieces that mention objects with aligned pieces that form rules in use [Wilson, 2017, Saka, 1998].Source: §1 · pp. 1–2
  • The study evaluates six LLMs across three prompt types and finetunes Mistral 7B and OLMo 7B using game data.The models include GPT-4o, Gemini-Flash 1.5, OLMo 2 13B and 7B, Mistral 7B, and Mixtral 8x7B.Source: §1 · p. 2
  • Prompt-based learning achieves weak results when dynamic rule changes are necessary, even for LLMs with enhanced reasoning capacities.Finetuning two open LLMs improves performance somewhat, but reasoning about dynamic rule changes remains challenging.Source: §1 · p. 2
  • The game is presented as a challenging testbed for reasoning about dynamic rules and reflection by current reasoning LLMs.An earlier study found that state-of-the-art LLMs struggled with rule-manipulation generalization [Cloos et al., 2024].Source: §1 · p. 2

2 Related work

Related work describes prompting and finetuning approaches for improving LLM reasoning, alongside prior efforts to use AI agents and language models in games and puzzles.

2 Related work

  • Prompt-based learning uses structured inference-time prompts, while chain-of-thought methods elicit intermediate reasoning steps before an answer.Zero-shot chain-of-thought adds a reasoning cue, and Plan-and-Solve prompting asks the model to plan before executing the solution [Wang et al., 2023a].Source: §2 · pp. 2–3
  • Finetuning adjusts pretrained model parameters for specific tasks, often using smaller domain-specific datasets and labeled examples.Instruction tuning trains on instruction-output pairs; PEFT methods such as LoRA modify only a subset of parameters to reduce memory and compute costs [Hu et al., 2021].Source: §2 · p. 3
  • Prior game-agent work spans chess, reinforcement learning, and LLM-based puzzle solving, including text-based alternatives to traditional search methods.ChessGPT introduced a game-and-language dataset, while GPT-2 was trained on solved examples for puzzles such as mazes, Sudoku, and the Rubik’s Cube [Feng et al., 2024; Noever and Burdick, 2021].Source: §2 · p. 3

3 Method

The study represents Baba is You as a grid where aligned text blocks create rules that determine object behavior, and players can change those rules by moving text. It evaluates six LLMs with different prompt formats and analyzes their reasoning on game levels; two models are also finetuned on combined game-related data.

3 Method

  • Aligned object, verb, and property text creates active rules, while rearranging text can change or break rules during play.A level is complete when the controlled object reaches the win-condition object, or when one object satisfies both conditions.Source: §3 · pp. 3–4
  • The simplified game represents levels as character strings and allows players to move, create rules, and break rules by pushing text blocks.Objects can also be pushed when a rule assigns them the PUSH property.Source: §3 · p. 4

3.1 Experimental Setup

  • Six LLMs were evaluated using three prompt formats, which progressively add active rules and possible actions to a simple prompt.Prompts also include the ASCII level and a Plan-and-Solve sentence to elicit chain-of-thought reasoning.Source: §3.1 · pp. 4, 6

Evaluation of Reasoning

  • Reasoning was manually analyzed across level interpretation, problem formulation, solution formulation, and action formulation, while Mistral 7B and OLMo 7B were finetuned on three combined datasets.The datasets cover general reasoning questions, Baba is You mechanics questions, and levels paired with expected solutions; the latter dataset is handwritten, and training used LoRA.Source: Evaluation of Reasoning · pp. 6, 8, 9
  • Models and prompt formats were tested on 14 levels, with correctness assessed across five runs and accepted when a solution appeared in at least three.The levels target rule creation, transformation, immutability, and logical reasoning; most are Keke AI competition demos, except level 14.Source: Evaluation of Reasoning · pp. 6, 8

4 Results

Across prompts, GPT-4o and Gemini Flash 1.5 generally outperform smaller models, but all models struggle with aspects of rule-based reasoning and action formulation. Finetuning improves some level-analysis steps without clearly improving puzzle solving.

Simple prompt (1)

  • With the simple prompt, most OLMo and Mistral outputs were incorrect because the models struggled to identify active rules and recognize objects and text.GPT-4o also struggled with vertically placed rules.Source: Simple prompt (1) · p. 9

Rule-extended (2) and Action-extended (3)

4.2 Finetuning

  • Finetuning improved Mistral’s classification of objects and text blocks and OLMo’s problem-statement formulation, but neither showed clear improvement in puzzle solving.The finetuned models made more correct steps, yet still did not fully solve the levels.Source: §4.2 · pp. 13, 15, Figure 9 · p. 14, Figure 10 · p. 14

5 Discussion & Conclusion

The models show partial ability on Baba is You but continue to struggle with spatial interpretation and dynamic rule manipulation. The authors identify representation, reasoning, and reflection as distinct challenges, and note that the evaluation and finetuning data are limited.

5 Discussion & Conclusion

  • Even GPT-4o and Gemini Flash 1.5 struggled to fully understand the game, while finetuning improved analysis but not full solution generation.Models also struggled with two-dimensional grid interpretation and understanding how rule creation or breaking is performed.Source: §5 · pp. 15–16
  • Representation remains difficult because the models were not pretrained on Baba is You and struggled to interpret the board spatially.The authors suggest further finetuning and pretraining may be necessary for improvement.Source: §5 · p. 16
  • Finetuning enhanced reasoning, but the authors distinguish this challenge from accurately representing the game state.The discussion frames reasoning as manipulating state representations according to rules.Source: §5 · p. 16
  • The models showed weak use–mention reasoning about dynamic rules; accurate reflection may require explicit external algorithms or methods such as analogical prompting.The paper used Plan-and-Solve prompting, a zero-shot chain-of-thought method.Source: §5 · p. 16, Limitations & Further work · p. 16

Limitations & Further work

  • The evaluation covered only 14 relatively simple levels, and the finetuning dataset contained 15 solution examples and 298 game-mechanics questions.Manual error analysis may introduce subjectivity; the authors suggest automated evaluation for more consistent, scalable assessment.Source: Limitations & Further work · pp. 16–17
Loading 2506.19095v1…