PaperlayerView exact revision on arXiv

Source-linked AI summary

Witness: Discovery, Deciphering, and Epiphany in Interactive Puzzle Environments

Guanghan Ning, Ping Liu, Linyi Li, Huangjie Zheng, Arjun Neervannan, Huu Nguyen, Michael Sklar, Deniz Zorlu, Nicolai Ouporov

arXiv:2609.32208v1cs.AI

TL;DR

Interactive rule-discovery puzzles offer a controlled setting for studying how agents infer hidden rules, but low success alone does not reveal whether interpretation, discovery, or planning failed. WITNESS provides grid-based hidden-rule games for controlled evaluation and RL training. RL improves performance on held-out rules and external discovery benchmarks, while the results indicate that rule acquisition is a major difficulty for Opus-5.

  • Problem

    Interactive rule-discovery puzzles study how agents gather evidence about hidden rules and act toward a goal, but low success alone cannot distinguish failures in interpretation, rule discovery, or planning.

    Source: §1 · p. 1

  • Method

    WITNESS combines ASCII-observed grid puzzles with hidden rules, generated training and evaluation games, and validation splits for held-out compositions and primitives.

    Source: §1 · p. 2

  • Results

    RL improves performance on held-out rules and four external discovery benchmarks, with a mean gain of 4.1 percentage points across the external benchmarks.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The results indicate that rule acquisition is a major difficulty for Opus-5, while Qwen3.8-27B remains substantially limited even when ground-truth rules are provided.

    Source: §1 · p. 2

    Held-out status refers only to the RL training split; classic rule primitives may have appeared in model pretraining.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Automated science needs agents that can work out the rules of an unfamiliar environment by interacting with it. Interactive rule-discovery puzzles offer a controlled setting for studying this ability: an agent infers hidden rules through experimentation and uses what it has inferred to reach a stated goal. We ask what limits current language models on these puzzles and whether reinforcement learning (RL) improves performance on rules held out from training. To study both, we introduce WITNESS, a 2D grid-based puzzle environment with ground-truth ASCII observations and controlled access to rules. An agentic pipeline generates games for WitnessGym, the RL training suite, and WitnessBench, comprising public validation and private test games. The validation set separately tests new compositions of trained rule primitives and primitives absent from training. Under a shared harness, the best of 18 frontier proprietary and open-weight models solves only 24\% of private test level slots, with scores sensitive to the observation interface and agent configuration. Providing ground-truth rules raises Opus-5's validation RHAE-L5 (relative human action efficiency over the first five levels) from 59.9 to 97.8, whereas a 27B open-weight model gains only 2.1 points and remains limited even with the rules provided. RL on WitnessGym raises the 27B model's private test RHAE-L5 from 2.1 to 5.4 and yields a mean gain of 4.1 points on four external discovery benchmarks. Together, these results point to rule acquisition as a major difficulty for frontier models like Opus-5 while smaller models further struggle on rule-based execution, and indicate that RL on hidden-rule puzzles transfers to broader rules and real-world tasks beyond training. Benchmark is available at: https://witnessbench.ai

1 INTRODUCTION

Interactive rule-discovery puzzles provide a controlled way to study how agents infer hidden rules and act on them, but low success alone cannot identify where they fail. WITNESS enables controlled evaluation and training; results show both rule-acquisition difficulty and gains from RL on held-out rules and external benchmarks.

1 INTRODUCTION

  • Interactive rule-discovery puzzles let agents gather evidence about hidden rules and use it to pursue a goal, but low success does not identify whether interpretation, discovery, or planning failed.Existing benchmarks leave relevant controls incomplete: some retain visual interpretation, omit spatial structure, or provide fixed known rules.Source: §1 · p. 1
  • WITNESS provides ASCII-observed grid puzzles with hidden rules, and its training and evaluation games distinguish held-out compositions from held-out primitives.WitnessGym is the RL training suite; WitnessBench includes public validation and private test games.Source: §1 · pp. 2–3
  • Under a shared harness, the best of 18 models solves only 24% of private test level slots, while rule access benefits Opus-5 more than Qwen3.8-27B.Opus-5 rises from 59.9 to 97.8 RHAE-L5 with ground-truth rules; Qwen3.8-27B rises from 8.0 to 10.1.Source: §1 · p. 2
  • RL-trained Qwen3.8-27B improves on private test games, held-out rule splits, and four external discovery benchmarks.The private-test score rises from 2.1 to 5.4 RHAE-L5, with a mean gain of 4.1 percentage points across the external benchmarks.Source: §1 · pp. 2–3
  • The reported results also include no improvement for Qwen3.5-9B under the same RL setup and no additional gains from hypothesis-keyed credit assignment over episode-advantage broadcasting.Together, the contributions cover the environment, a shared-harness evaluation, and RL transfer evidence.Source: §1 · p. 3

2 BENCHMARK

WITNESS combines a grid-puzzle engine, training and evaluation suites, a game-generation pipeline, a shared harness, and an action-efficiency metric. Its benchmark separates training games from held-out validation and private test rules.

2.1 GAME ENGINE AND MECHANICS

  • The engine presents an M × N grid for drawing a start-to-goal path; the agent must infer directional action mappings while satisfying hidden rules.Invalid moves are refused, rule-violating submitted paths are rejected with violations highlighted, and some games require eliciting hints.Source: §2.1 · p. 3

2.2 GAME SUITE: WITNESSBENCH AND WITNESSGYM

  • Validation tests new compositions of training primitives and primitives absent from training, while 17 private test games use rules not used in training.WitnessBench has 8 public validation games; WitnessGym has 22 public training games that share the engine and harness but none of the benchmark rules.Source: §2.2 · p. 4

2.3 GAME GENERATION AND VALIDATION

  • WitnessForge generates rule-based games and curricula, then validates candidate rules and levels through programmatic checks, model play, independent audits, and replayed solutions.Across released games, 5 of 42 candidate rules were rejected and 97% of candidate levels were filtered, leaving 760 levels.Source: §2.3 · p. 4

2.4 EVALUATION HARNESS

  • A shared harness gives models ASCII observations, recent-attempt history, persistent within-game rule memory, and six actions, with no code execution or external tools.Runs end when all levels are solved or a level exceeds its 300-action budget or the game reaches 1000 LLM calls.Source: §2.4 · pp. 4–5

2.5 EVALUATION METRIC AND PROTOCOL

  • RHAE scores solved levels by action efficiency relative to a human reference, caps the score at 1.15, and assigns zero to unsolved or unreached levels.The reference is the mean action count across successful plays by team members without the rule specification, including failed attempts and resets.Source: §2.5 · p. 5
  • RHAE-L5 emphasizes the first five levels, while RHAE-L20 measures the full twenty-level curriculum; the game score cannot use efficiency to offset an unsolved level.Game scores are weighted by level index and capped by the weighted completion fraction.Source: §2.5 · p. 5
  • Dataset scores average game scores across each split per seed, with means and standard deviations reported across five test seeds and three validation seeds.This yields 85 test runs per model.Source: §2.5 · p. 5

3 MODEL EVALUATION: WHAT LIMITS CURRENT MODELS

Under the shared harness, models solve only a minority of private-test level slots, and results vary with both evaluation window and harness configuration. Providing ground-truth rules sharply helps Opus-5, while substantial difficulty remains for Qwen3.8-27B.

3.1 OVERALL PERFORMANCE

  • 36.4 ± 3.7 RHAE-L5 was Fable-5’s highest mean, but it solved 414 of 1,700 level slots and completed 2 of 85 runs; 12 of 18 models completed none.The highest RHAE-L20 belonged to GPT-6-Astra-Pro at 12.8 ± 1.1, illustrating that rankings depend on the evaluation window.Source: §3.1 · pp. 5–6, Table 1 · p. 6

3.2 OBSERVATION AND HARNESS COMPARISONS

  • Replacing ASCII with a raw 64×64 grid changed RHAE-L5 by widely varying amounts across models, while removing observation, memory, and interaction support together reduced scores further.The mean-score ordering of Opus-4.8 and Kimi-K3 reversed between ASCII and raw-grid settings; the minimal harness result reflects the combined removal of multiple components.Source: Table 2 · p. 6, §3.2 · p. 7
  • Prime-Agent scored higher for all four models, but typed-move-only rescoring was much lower, indicating that most credited levels involved programmatic search or solution generation.The rescored values describe those runs and are not estimates of performance without code, since code-based solutions may affect later levels.Source: §3.2 · p. 7

3.3 EFFECT OF PROVIDING GROUND-TRUTH RULES

  • 34.1 points lifted Opus-5 to 97.8 RHAE-L5 with every evaluated level solved in all 40 runs under the rule card; Qwen3.8-27B gained 1.0 point to 10.1.The rule-card probe used eight validation games, five seeds, thinking off, and the first five levels; its scores are not comparable to Table 1.Source: §3.3 · p. 7, Table 3 · p. 7

4 RL TRAINING: DOES IT TRANSFER BEYOND TRAINING RULES

RL training on hidden-rule games improved Qwen3.8-27B’s performance on held-out rules and external discovery benchmarks, while results varied by model and credit-assignment scheme. The paper reports transfer beyond the training games, with the clearest external gains on cipher decryption and experiment design.

Training setup

  • Training used GRPO with outcome-based rewards on 22 games, selecting checkpoints on eight validation games before evaluating on 17 private-test games.Episodes ended after all levels were solved, 300 LLM calls, or an unsolved level’s 300-action budget; training and reported Section 4 evaluations used thinking off.Source: §4 · p. 8

Learning and transfer within the suite

  • 2.1 to 5.4 RHAE-L5 was Qwen3.8-27B’s private-test change after RL, with solved level slots rising from 67 to 122 of 1,700.The gains were on rules absent from training, and validation RHAE-L5 rose for both held-out compositions and held-out primitives; Qwen3.5-9B did not improve under the same settings.Source: §4 · p. 8

Credit assignment

  • Credit-C upweighted earlier attempts whose rule hypotheses survived a successful episode, but survival could misidentify correctness, so the remaining experiments used broadcast credit.Within the compared schemes, credit-A was below credit-B, which was below credit-C; credit-C was on par with broadcast.Source: §4 · pp. 8–9

Transfer to external benchmarks

  • 4.1 points (95% CI 2.1 to 6.2) was the mean gain across four external benchmarks at iteration 20, with clearest gains on CipherBank and WILT.Per-benchmark intervals excluded zero for CipherBank (+4.8) and WILT (+2.3), had a lower bound at zero for PhysGym (+3.4), and included zero for Failing-to-Falsify (+6.1).Source: §4 · p. 9

5 RELATED WORK

Related benchmarks differ in what agents observe, whether rules are provided, and which units are held out for evaluation. The paper positions WITNESS among interactive discovery benchmarks and game-based RL work, with external evaluations drawn from scientific discovery and hypothesis testing.

Interactive rule discovery and puzzle benchmarks

  • WITNESS combines an entity-level 2D board, hidden rules, and validation splits defined by their relationship to training rules, enabling controlled probes and transfer evaluation.ARC-AGI-3 combines visual interpretation with rule discovery, while DiG-Bench and MazeBench avoid the visual gap but lack spatial structure.Source: §5 · p. 9

Learning and generalization through games

  • Prior game-based RL studies report transfer to other games or reasoning tasks, while this work separately evaluates held-out rule compositions and primitives.The cited examples include transfer from Sokoban and Tetris to Blocksworld and WebShop, and gains of up to 10 points on eight reasoning benchmarks from self-play on zero-sum games.Source: §5 · p. 9

Hypothesis testing and scientific discovery

  • Four external evaluations probe rule inference across domains: physical-law discovery, queried-example induction, falsification-seeking, and hidden-encoding inference.These correspond to PhysGym, WILT, Failing-to-Falsify, and CipherBank, respectively.Source: §5 · p. 10

Agent interfaces and harnesses

  • A shared harness controls one source of variation in model comparisons because perception, memory, code execution, and interaction policies can change what evaluations measure.The related work notes that scaffold differences affect ARC-AGI-3 scores and that tool-enabled systems are not directly comparable to fixed action-only interfaces.Source: §5 · p. 10

6 CONCLUSION

The conclusion presents WITNESS as an environment for studying interactive rule discovery and reports evidence that RL gains extend to held-out rules and external benchmarks.

6 CONCLUSION

  • The highest-scoring model under the shared harness solved only 24% of private-test level slots.The conclusion identifies this as evidence of the difficulty of the benchmark task.Source: §6 · p. 10
  • RL raised Qwen3.8-27B’s private-test RHAE-L5 from 2.1 to 5.4 and improved performance on held-out compositions and primitives.The conclusion describes these gains as evidence of transfer beyond the training rules.Source: §6 · p. 10
  • The trained model gained an average of 4.1 percentage points across four external benchmarks.The conclusion presents the result as transfer beyond the training games.Source: §6 · p. 10

A APPENDIX: ABLATIONS, EXPERIMENT DETAILS, QUALITATIVE STUDIES, FULL PROMPTS, AND PIPELINE PSEUDO-CODE

The appendix documents transfer results and RL training dynamics, defines rescoring for tool-assisted rollouts, and describes the agent’s prompts and evaluation and discovery protocols. It also details how games are generated, qualified, tested, and packaged, including prompt constraints for evidence-based rule discovery and bounded planning.

A.3 DETAILED RL SETTINGS AND TRAINING DYNAMICS

  • RL samples eight episodes for each of 22 training games per iteration, and Credit-C additionally credits first-added rules that appear in later verified reflections.Training runs for 30 iterations; episodes end upon completing the game, reaching 300 LLM calls, or exhausting a 300-action budget for an unsolved level.Source: §A.3 · pp. 15, 19
  • With identical training settings, the 27B model learns to beat more levels while the 9B model shows no useful learning signal; their validation performance on known-rule composites and unseen rules converges.Training on primitive-rule games also generalizes to test games with unseen rules.Source: Figure 6 · p. 16
  • As training progresses, the 27B model completes more levels with fewer calls, reflects more to plan actions, and preserves fewer earlier rule hypotheses than the 9B model.The 9B model continues to produce verbose tokens and preserve most low-level hypotheses that do not contribute to solving.Source: Figure 7 · p. 16

A.4 OOD GENERALIZATION

  • At iteration 20, broadcast RL gains 4.1 mean points (95% CI 2.1 to 6.2) across four external benchmarks; credit-C gains 2.7 (0.6 to 4.9).Broadcast gains are clearest on CipherBank (+4.8) and WILT (+2.3); per-benchmark evidence is uneven, and iteration-10 mean gains have intervals including zero.Source: Table 7 · p. 20, §A.4 · p. 20

A.5 MIS-BEHAVIORS IN OSS HARNESS

  • The OSS harness uses raw 64 × 64 image frames and Python/HTTP tools, unlike the canonical human-readable board harness, which allows no code execution or programmatic search.The evaluation covers 17 private test games; model-attended scoring excludes unattended completions, repeated failed submissions meeting the preregistered brute-force threshold, and background action execution.Source: §A.5 · p. 20
  • Verbal-only scoring additionally excludes wins selected by programmatic search, enumeration, empirical rule-fitting, or solver reuse, and credits only sequences explicitly typed by the model that match submitted moves.Under preregistered Rule B, the first excluded level and every later level in that rollout receive zero credit.Source: §A.5 · p. 21
  • Observed probing included attempts to locate game rules or query undocumented referee endpoints, but none retrieved privileged information; rollout scores reflect termination after errors or resource limits.An attempted probe alone does not exclude a level; exclusions follow the stated scoring criteria.Source: §A.5 · p. 21

A.6 PROMPTS

  • The shared prompt gives only basic mechanics and broad object, action, quantity, and geometry priors, then instructs agents to observe, hypothesize, predict, act, verify, and explore systematically.It defines zero-indexed (row, col) coordinates with (0,0) at the top left and recommends testing actions contrastively and avoiding cycles.Source: §A.6 · pp. 21–22
  • The prompt advises preferring simple hypotheses, testing one rule at a time, separating mechanics from goals, learning from failures, and looking for patterns across levels.It limits each reflection to two new rules, requires at least two supporting observations, and directs deletion of rules whose counter-evidence matches or exceeds support.Source: §A.6 · p. 22
  • Prompted plans are capped at 16 actions, require known-rule justifications, and must be abandoned after three consecutive no-op actions.The prompt also directs agents to stop exploring when more than 95% of budget is consumed and switch strategy after ten consecutive actions without state change.Source: §A.6 · p. 22
  • Each reflection presents the board, established action semantics, recent observations, and current rule memory, then requests analysis, memory operations, and optionally a plan.Harness notices, disputed-rule warnings, and recent-attempt history appear conditionally; reset preserves learned rules but does not refund actions, while confirm submits the solution.Source: §A.6 · pp. 23–24

A.7 EVALUATION DETAILS

  • Table 8 reports the exact model identifiers and sampling settings for the Table 1 runs: 17 unseen games, five seeds, up to 20 levels, 300 actions per level, and 1,000 LLM calls per rollout.All models used temperature 1.0, no top-p, top-k, or seed parameter, and a cached system prompt; reasoning settings varied by model.Source: Table 8 · p. 24

A.8 PIPELINE

  • The pipeline independently implements and cross-examines two versions of each rule, retries with counterexamples up to four rounds, then merges a certified module.Rules are also screened for necessity, visibility, non-triviality, and novelty.Source: §A.8 · p. 25
  • The pipeline assembles levels in an Introduce → Accumulate → Reinforce sequence, audits the schedule, and packages games after blind human judging, adversarial testing, and deterministic replay.Level generation begins with a large candidate pool; sequencing measures which incorrect theories levels can eliminate and embeds visual feedback and exploration constraints.Source: §A.8 · p. 25
  • Candidate levels must have optimal solution depth of at least five, require rule mechanics, and yield zero greedy-baseline wins and at most 0.004 random-play wins.The pipeline also requires structural validation and deduplicates accepted levels by canonical hash.Source: §A.8 · p. 25

A.1 ABLATION: GIVEN GROUND-TRUTH RULES

The ground-truth-rule ablation tests how models perform when game rules are supplied at evaluation time. Opus-5 solves all validation games and levels efficiently, while RL-trained models can gain efficiency yet overthink the rule cards.

A.1 ABLATION: GIVEN GROUND-TRUTH RULES

  • Opus-5 solves all games and levels efficiently with ground-truth rules, while RL-trained models can improve RHAE by over 31% but show more plan-less calls.For RL-trained models, plan-less calls rise from 2% to 57%, especially on more complex levels.Source: §A.1 · p. 13
  • The rule-card ablation uses eight held-out validation games, five seeds, and an 8192-token output cap, and its diagnostic scores are excluded from other results.The cards state each game's true win condition and mechanics, including action mapping, in the evaluation system prompt.Source: Table 5 · p. 14
  • The table caption reports little improvement for the base model from rule cards, but significant improvements for RL policies, identifying rule discovery as Opus-5's main bottleneck.The caption characterizes the hint-assisted scores as diagnostic rather than general benchmark results.Source: Table 5 · p. 14
  • Rarely, human players choose a more efficient path than Opus-5; in those cases, Opus-5 selects a non-optimal path in 3/5 and 1/5 seeds, respectively.In most other cases, especially early levels requiring rule discovery, Opus-5 with supplied rules is more efficient and can exceed an un-capped RHAE score of 100.Source: Figure 5 · p. 14

A.2 ABLATION: LEVEL DIVERSITY VS RULE DIVERSITY

Gains on trained layouts carry over to unseen layouts with the same rules, supporting the conclusion that level diversity is unnecessary and rule diversity can be prioritized.

A.2 ABLATION: LEVEL DIVERSITY VS RULE DIVERSITY

  • Because gains carry over to unseen layouts, the authors conclude that level diversity is unnecessary and training can focus on rule diversity.Training used fixed levels; alternate level sets varied boards while keeping the rules identical.Source: §A.2 · p. 15, Table 6 · p. 15
  • Gains on trained layouts carry over to reserved layouts with the same rules but different boards, and the trained–unseen gap does not grow with training.The comparison tracks levels completed, action efficiency on completed levels, and official game score.Source: Table 6 · p. 15
  • The comparison evaluates trained and reserved level sets using levels completed, action efficiency on completed levels, and official game score.The reserved layouts were not used in training.Source: Table 6 · p. 15

Related papers

Loading 2609.32208v1…