PaperlayerView exact revision on arXiv

Source-linked AI summary

LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models

Jiayi Gui, Yiming Liu, Jiale Cheng, Xiaotao Gu, Xiao Liu, Hongning Wang, Yuxiao Dong, Jie Tang, Minlie Huang

arXiv:2408.15778v4cs.AIcs.CL

TL;DR

Benchmarks have not adequately tested LLM reasoning while following instructions and rules. LOGICGAME evaluates rule-based execution and planning through games, and experiments find substantial shortcomings, especially on complex tasks.

  • Problem

    Reasoning with instruction-following capabilities remains underexplored, despite extensive alignment focused on following human instructions.

    Source: §2 · p. 4

  • Method

    LOGICGAME uses rule-based games spanning execution and planning, with varied difficulty and tasks adapted from real-world scenarios.

    Source: §1 · p. 3, §3.1.1 · p. 4

  • Results

    About 20% overall accuracy and less than 10% on level 3 tasks, even for the best-performing LLMs.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that more effort is needed to improve models’ ability to handle complex reasoning scenarios.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated notable capabilities across various tasks, showcasing complex problem-solving abilities. Understanding and executing complex rules, along with multi-step planning, are fundamental to logical reasoning and critical for practical LLM agents and decision-making systems. However, evaluating LLMs as effective rule-based executors and planners remains underexplored. In this paper, we introduce LogicGame, a novel benchmark designed to evaluate the comprehensive rule understanding, execution, and planning capabilities of LLMs. Unlike traditional benchmarks, LogicGame provides diverse games that contain a series of rules with an initial state, requiring models to comprehend and apply predefined regulations to solve problems. We create simulated scenarios in which models execute or plan operations to achieve specific outcomes. These game scenarios are specifically designed to distinguish logical reasoning from mere knowledge by relying exclusively on predefined rules. This separation allows for a pure assessment of rule-based reasoning capabilities. The evaluation considers not only final outcomes but also intermediate steps, providing a comprehensive assessment of model performance. Moreover, these intermediate steps are deterministic and can be automatically verified. LogicGame defines game scenarios with varying difficulty levels, from simple rule applications to complex reasoning chains, in order to offer a precise evaluation of model performance on rule understanding and multi-step execution. Utilizing LogicGame, we test various LLMs and identify notable shortcomings in their rule-based logical reasoning abilities.

1 Introduction

Benchmarks have not adequately assessed LLMs’ ability to reason while following rules and instructions. LOGICGAME evaluates rule-based execution and planning, and experiments reveal substantial shortcomings in these abilities.

1 Introduction

  • LOGICGAME covers execution tasks such as string manipulation and arithmetic, alongside planning games involving math and pure logic puzzles.The benchmark includes problems with varying difficulty levels.Source: §1 · p. 3
  • Problems rely solely on the given rules, while evaluation checks intermediate steps as well as final answers.Intermediate steps are deterministic and can be automatically verified, helping distinguish rule-following from guessing.Source: §1 · p. 3
  • Even the best-performing LLMs achieve about 20% overall accuracy and less than 10% on level 3 tasks.The paper also reports that few-shot demonstrations may help execution while damaging planning performance.Source: §1 · p. 3

2 Related Work

Prior reasoning benchmarks address several distinct areas, while LOGICGAME frames rule-based reasoning as the integration of logical reasoning with instruction-following.

2 Related Work

  • Multi-step reasoning is presented as important for complex decision-making and planning, including tasks undertaken by LLM agents.The passage also notes chain-of-thought prompting and specialized training as techniques used to elicit reasoning.Source: §2 · p. 3
  • Earlier benchmarks cover inductive, deductive, and abductive logic, mathematical reasoning, commonsense reasoning, and theory of mind.Mathematical benchmarks range from grade-school problems to Olympiad challenges and theory proving.Source: §2 · p. 4
  • LOGICGAME presents rule-based reasoning as an integration of logical reasoning and instruction-following, compared with other logical reasoning benchmarks in Table 1.The table’s comparison includes exemplars, process verifiability, and determinism.Source: §2 · p. 4, Table 1 · p. 4

3 LOGICGAME

LOGICGAME is built from rule-based scenarios spanning execution and planning, then standardizes, tiers, and evaluates responses across Chinese and English versions.

3.1 Data Construction

  • Dataset construction combines real-world-inspired problem design, standardized output constraints, difficulty levels with exemplars, and bilingual versions.The construction process is organized into four phases: problem design, output constraints, difficulty and exemplars, and Chinese-English translation.Source: §3.1 · p. 4

3.1.1 Problem Collection and Design

  • The benchmark uses real-world-inspired games across execution and planning, with deterministic rule application for execution and multi-step search for any valid planning path.Problems are crowdsourced and designed to limit reliance on natural-language inference; execution steps are verifiable, while planning may use intermediate checkpoints or state variables.Source: §3.1.1 · pp. 4–5

3.1.2 Output Constraint Design

  • Responses use JSON with an answer list, and multi-step or more complex problems also require a process list.Level 0 single-step problems are assessed only on the final answer; Levels 1–3 and certain Level 0 problems are assessed on both answer and process.Source: §3.1.2 · pp. 5–6

3.1.3 Difficulty Levels and Exemplars

  • Four difficulty levels increase rule complexity and the number of reasoning steps, while two exemplars per question test learning from demonstrations.Each exemplar includes a question, correct answer, step-by-step solution, and explanations.Source: §3.1.3 · p. 6

3.1.4 Building Bilingual Benchmark

  • The benchmark provides Chinese and English versions to address potential bias against models trained primarily on English data.Source: §3.1.4 · p. 6

3.2 Evaluation Protocol

  • Evaluation compares responses with reference answers and processes, using answer accuracy, character-level process accuracy, and their logical-AND composite, AP-Acc.Figure 3 compares 14 models on AP-Acc for the Chinese and English versions.Source: §3.2 · p. 6, Figure 3 · p. 7

4 Experiments

Experiments across 14 LLMs find uneven performance across model complexity and task type, while few-shot context has task- and difficulty-dependent effects.

4.1 Experimental Setup

  • Fourteen LLMs were evaluated, with temperature set to 0 and a 2048-token maximum.Other inference parameters remained at their default values.Source: §4.1 · p. 7

4.2 Main Results and Analysis

  • Some models decline gradually from Level 0 to Level 3, while others drop sharply after Level 1.The authors suggest this may indicate greater consistency across task complexities for some models.Source: §4.2 · p. 8
  • Execution and Planning strengths vary by model: o1-preview favors Execution, while llama-3-70b-chat favors Planning.Within the GPT family, gpt-4-turbo-0409 does better in Planning, while gpt-4o does better in Execution.Source: §4.2 · p. 8
  • JSON-format failures occur occasionally, but low overall error rates minimally affect rankings among similarly performing models.Source: §4.2 · p. 8

4.3 Few-shot Results

  • The Reversi case-study example visualizes initial, reference, and model board states, with errors highlighted in red.The stated rules use an even-sized grid, alternating turns, and piece flips when pieces are trapped in a line; Figure 6 omits JSON constraints.Source: Figure 6 · p. 9, §4.3 · p. 9
  • Execution accuracy generally improves with added shot context, especially for stronger models; its benefits are greatest at Level 0 and diminish as difficulty rises.The shot-setting comparisons are shown in Figure 4 and Figure 5.Source: Figure 4 · p. 8, Figure 5 · p. 8, §4.3 · p. 9
  • Few-shot effects on Planning are heterogeneous: performance often declines with added context, and 1-shot effects shrink at higher difficulty while 2-shot effects remain patternless.Figure 4 compares shot settings for Execution and Planning; Figure 5 compares them across difficulty levels.Source: Figure 4 · p. 8, Figure 5 · p. 8, §4.3 · pp. 9–10

5 Discussion

The Reversi case study finds that models can follow the format and interpret the initial board yet still fail to produce the correct outcome.

  • All models failed to give the correct Reversi answer, although all except llama-3-8b-chat followed the format and interpreted the initial setup correctly.Source: §5 · p. 10
  • Observed errors included misplaced pieces, incorrect captures, and excessive board alterations.Examples point to failures in applying piece-flipping rules, understanding initial conditions, and preserving board structure.Source: §5 · p. 10
  • The examples distinguish rule-application errors from failures to preserve the board state.Source: §5 · p. 10

6 Conclusion

The conclusion presents LOGICGAME as a benchmark for rule-based reasoning and reports persistent deficiencies on complex reasoning tasks.

  • LOGICGAME evaluates LLMs’ rule understanding, execution, and planning across multiple difficulty levels.Source: §6 · p. 10
  • Its evaluation checks both final outcomes and reasoning processes to assess whether models follow given rules rather than merely guess answers.Source: §6 · p. 10
  • Experiments indicate significant deficiencies in current large models on rule-based reasoning tasks; the paper calls for more effort on complex reasoning abilities.Source: §6 · p. 10

A JSON prompt

The evaluation prompt specifies JSON output and illustrates the requested answer and process fields.

A JSON prompt

  • The prompt requests a JSON object with an answer list and a process list, each containing strings.It presents a String Synthesis example as input.Source: A JSON prompt · p. 13

B Data overview of different categories

The data overview classifies categories and reports their sample sizes in Table 2.

B Data overview of different categories

C Additional evaluation metrics and error analysis

The section defines error and correctness metrics for evaluating JSON-constrained model outputs, then reports error patterns across English and Chinese versions.

C Additional evaluation metrics and error analysis

D Few shot results of en version

Few-shot effects in the English version differ between Execution and Planning, and are generally weaker as difficulty increases for Execution.

D Few shot results of en version

E Problems all models fail

Figure 10 highlights five categories with the poorest average AP-Acc scores across models. Reversi is especially difficult, while performance varies slightly across models on Constrained Linear Arrangement.

  • Many models score close to zero on Reversi, with o1-mini and o1-preview as exceptions.Source: E Problems all models fail · p. 15
  • Constrained Linear Arrangement shows slight performance variation across models in the planning scenario.Source: E Problems all models fail · p. 15
  • Figures 8 and 9 show few-shot P-Acc differences across execution and planning categories and difficulty levels, respectively.The differences compare 1-shot and 2-shot settings against 0-shot.Source: Figure 8 · p. 16, Figure 9 · p. 16
  • Figure 10 identifies Reversi, Synthesis and Decomposition, Minesweeper, Constrained Linear Arrangement, and Logic Puzzle as the five poorest-performing categories.The figure reports average AP-Acc scores for these categories.Source: E Problems all models fail · p. 15, Figure 10 · p. 16
Loading 2408.15778v4…