PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games

Shengyuan Ding, Xilin Wei, Xinyu Fang, Haodong Duan, Dahua Lin, Jiaqi Wang, Yuhang Zang

arXiv:2606.19338v1cs.CV

TL;DR

Multimodal policies must recall observations that are no longer visible and use them in closed-loop, non-Markov interaction, a capability existing benchmarks incompletely isolate. RNG-Bench evaluates this ability in two controlled games and finds substantial residual errors attributable to forgetting earlier observations, while fine-tuning improves benchmark performance and transfers without degrading general multimodal capability.

  • Problem

    Existing benchmarks incompletely isolate remember-to-act behavior, where past observations must inform actions and errors alter subsequent observations.

  • Method

    RNG-Bench evaluates hidden-state reconstruction in closed-loop Matching Pairs and 3D Maze games across controlled difficulty axes, using duels and a Memory Gap diagnostic.

  • Results

    Across leading MLLMs, performance leaves substantial headroom, with Gemini-3.1-Pro winning all 16 Matching Pairs duels and fine-tuning Qwen3.5-9B improving RNG-Bench performance and transfer.

  • Takeaways & Limitations

    RNG-Bench localizes residual failures primarily to belief-state maintenance and provides a testbed for interactive multimodal models.

  • Takeaways & Limitations

    The study covers two environments, uses a diagnostic Memory Gap under an oracle interface, and demonstrates fine-tuning feasibility on one base-model family.

Abstract

from arXiv · show

Deploying multimodal foundation models as closed-loop policies increasingly requires conditioning actions on observations that are no longer visible. However, existing benchmarks either expose the full state, conflate hidden-state reconstruction with other agent skills, or test recall only after an episode has ended. We introduce RNG-Bench (Reconstructive Non-Markov Games), a benchmark suite designed to isolate a base model's ability to reconstruct past observations and act on them during multi-step interaction. RNG-Bench includes two complementary games: Matching Pairs, where card identities briefly revealed at specific locations must later be recalled, and 3D Maze, where egocentric views must be integrated into a spatial map. Both games are evaluated under a unified harness with three controlled difficulty axes: grid size, visual pattern, and observation modality. The benchmark further introduces a head-to-head duel protocol to control for instance-level variance and a Memory Gap metric that disentangles forgetting from poor action selection. The hardest configurations require contexts of roughly 128K tokens and 350 image inputs per episode, and remain far from saturated by frontier MLLMs. Memory Gap analysis shows that most residual errors stem from forgetting earlier observations rather than from suboptimal decision making. Finally, fine-tuning Qwen3.5-9B on optimal-policy rollouts and filtered model demonstrations improves performance on RNG-Bench and transfers to existing benchmarks without degrading general multimodal capability.

1. Introduction

RNG-Bench isolates remember-to-act in closed-loop non-Markov games, where agents must reconstruct earlier observations because the current view is insufficient for optimal action. It evaluates two complementary games with controlled difficulty axes and a Memory Gap diagnostic, revealing substantial headroom for current multimodal models.

  • Motivation: In non-Markov interaction, earlier observations determine the correct action, and recall errors can causally alter later observations and compound across an episode.The current observation is not a sufficient statistic, so models must infer hidden state from interaction history before acting.
  • Benchmark gap: RNG-Bench targets remember-to-act rather than post-hoc remember-to-answer, so each recall feeds back into the next observation.This distinguishes the benchmark from trajectory question answering, where recall errors do not affect subsequent inputs.
  • Benchmark design: RNG-Bench combines Matching Pairs and 3D Maze in a closed-loop harness, varying grid or map size, visual pattern, and observation modality.Matching Pairs tests recalling briefly revealed symbols by location, while 3D Maze tests reassembling egocentric views into a spatial map.
  • Diagnostics: The Memory Gap separates forgetting from decision making by comparing performance with reconstructed state against action selection given the correct state.The diagnostic attributes failures to forgetting earlier observations rather than conflating memory with decision quality.
  • Results: 62.3% of pairs were matched by GPT-5.4 on image Matching Pairs at 10×10, versus 25.3% for Qwen3.5-397B; Gemini-3.1-Pro won 16 of 16 duels.On 3D Maze at 13×13, Gemini-3.1-Pro achieved 50.0% SR and 49.7% GS; the optimal policy used 3.24 moves per matched pair versus 8.01 for the strongest model.

2. Related Work

Existing game benchmarks assess broad interactive reasoning and multimodal action, but their varied failure sources make memory-specific errors difficult to isolate. Long-context and retrieval benchmarks evaluate information use across extended inputs, including position sensitivity and realistic reasoning, but address different evaluation settings.

  • Game Benchmarks: Game benchmarks test reasoning, planning, and multimodal action, but errors may reflect perception, rule understanding, exploration, planning, or action formatting rather than memory alone.AgentBench, GameBench, BALROG, and GameWorld exemplify broad interactive evaluation suites.
  • Long-Context and Retrieval Benchmarks: Long-context benchmarks measure how models use information distributed across extended inputs, while retrieval and reading benchmarks test information retrieval and comprehension.Representative suites probe position sensitivity, literal-match shortcuts, and realistic long-context reasoning.

3. Benchmark Design

RNG-Bench evaluates in-context state tracking for action in two diagnostic POMDP games, requiring models to reconstruct hidden state from episode history and immediately use it to act. Its design varies observation and task conditions systematically, adds matched-board duels, and separates memory failures from action-selection errors with the Memory Gap.

  • Unified POMDP formulation: Models act as history-based policies that select actions from raw in-context episode histories without an external belief module by default.At each step, the history contains prior observations and actions together with the current observation.
  • Complementary environments: Matching Pairs tests static identity-location memory, while 3D Maze tests incremental spatial-map construction from egocentric views.Matching Pairs hides previously revealed card identity-location bindings; 3D Maze hides maze topology, visited cells, position, and facing direction.
  • Controlled difficulty axes: The benchmark varies grid or maze size, visual pattern, and observation modality, while also manipulating feedback, prompting, minimaps, and response budgets.Both environments support text, 2D image, and 3D rendering; maze configurations range from 5×5 to 15×15, and each is evaluated over five seeds.
  • Duel protocol: Matching Pairs duels place two models on the same board and swap turn order across two games to control board randomness and first-mover bias.Models can incorporate cards revealed by their opponent, and the winner removes more pairs.
  • Diagnostics: The Memory Gap compares normal and oracle-hidden-state scores to distinguish belief-state tracking failures from action-selection failures.The compared score may be SR, Score%, or Efficiency, depending on the setting.

4. Experiments

Experiments show that RNG-Bench exposes distinct hidden-state demands: GPT-5.4 leads Matching Pairs, while Gemini-3.1-Pro leads 3D Maze and the duel evaluation. Performance degrades with larger hidden states, and external memory, visual modality, and action-history feedback substantially affect results.

  • Main results: GPT-5.4 leads Matching Pairs at 62.3%, while Gemini-3.1-Pro leads 3D Maze with 50.0% success and 49.7% game score.Matching Pairs uses a 10×10 board with 50 pairs; 3D Maze uses 13×13 mazes without a minimap.
  • Main results: The task ranking reverses because Matching Pairs stresses identity retention and pairwise retrieval, whereas 3D Maze requires spatial belief updating and route planning.GPT-5.4 reaches 20.0% SR and 30.5% GS in 3D Maze despite leading Matching Pairs.
  • Duel setting: Gemini-3.1-Pro wins all 16 duel games and tops Elo, while GPT-5.4 and Qwen3.5-397B achieve 50% and 47% win rates.The duel tests tracking both the model’s own flips and cards revealed by its opponent, and its ranking partially differs from single-player evaluation.
  • Hidden-state scale: Performance drops sharply as hidden-state size grows in both environments, indicating belief-state maintenance is a bottleneck rather than rule comprehension.Matching Pairs scales hidden state through board size, while 3D Maze scales it through maze size, path complexity, and longer action histories.
  • External memory: External memory roughly doubles Matching Pairs performance but recovers a smaller share on 3D Maze, where spatial navigation couples hidden-state tracking with action planning.The interventions provide a known-card memory map for Matching Pairs and a visited-state minimap for 3D Maze, quantified with MemGap.
  • Observation and action history: Text-only observations dominate image-based modalities, while removing action-feedback text collapses image-noise Matching Pairs to near chance.The modality ablation varies symbolic text, ASCII-style or local-patch images, noise-pattern cards, and 3D first-person views; the history ablation retains only board images.

5. Training with Non-Markov Trajectories

Fine-tuning Qwen3.5-9B on optimal-policy trajectories transfers to larger unseen boards and mazes, while adding model rollouts improves recovery and performance. The same training also improves targeted external memory and spatial benchmarks with minimal change on general multimodal tasks.

  • Setup: Qwen3.5-9B is fine-tuned on action tokens from smaller Matching Pairs boards and 3D mazes, with strictly larger, disjoint evaluation instances.Observation-token losses are masked; training covers boards from 2×4 to 8×8 and mazes from 5×5 to 9×9.
  • Data construction: 32K oracle-generated trajectories provide the optimal-only baseline, while rmix32k combines 26K optimal trajectories with 6K model rollouts at the same total budget.The matched budget isolates the contribution of the rollout component over optimal-only supervision.
  • Held-out scale generalization: 14.6 and 5.0 are the opt32k scores for Matching Pairs and 3D Maze on held-out larger sizes, up from 0.0 and 1.5.Optimal-only supervision transfers to unseen board and maze sizes.
  • Held-out scale generalization: 29.5 is rmix32k’s Score%, while rollout augmentation halves response cost per matched pair and produces the only non-zero maze SR.The oracle-only pool is described as mistake-free and potentially lacking recovery behavior.
  • External-benchmark transfer: +3.4 is the group-mean gain across four targeted external memory and spatial benchmarks, with EMeMBench showing the largest delta at +5.2.The general multimodal group shifts by +0.5, indicating limited change outside the targeted capabilities.

6. Conclusion

RNG-Bench isolates in-context belief-state tracking in controllable non-Markov games and shows that leading MLLMs fail as latent state grows, especially with image observations. Its analyses attribute errors to belief-state maintenance and establish a testbed for interactive MLLMs.

  • Conclusion: RNG-Bench isolates in-context belief-state tracking from rule understanding and perception in controllable non-Markov games.The benchmark provides a testbed for evaluating interactive MLLMs.
  • Conclusion: Performance collapses as the latent state grows, with image observations creating a larger bottleneck than history length.This pattern is observed across leading MLLMs.
  • Conclusion: Removing the action trace alone reduces Matching Pairs performance to near-chance, implicating belief-state maintenance in the failures.The analysis separates this failure from rule understanding and perception.

Limitations

RNG-Bench’s scope is limited to two controllable-hidden-state environments and does not yet cover broader game genres, model families, or visual styles. Its perceptual tracking setup and Memory Gap metric provide practical evaluation tools, but not a standalone causal decomposition.

  • Scope: RNG-Bench evaluates only Matching Pairs and 3D Maze, leaving broader coverage of game genres, model families, and visual styles to future work.The environments were selected for their controllable hidden state.
  • Evaluation setup: Image-based hidden-state tracking is measured through the model’s perceptual interface rather than direct access to hidden state.This makes the evaluation dependent on the perceptual interface used in the image settings.
  • Evaluation setup: The Memory Gap metric is intended as a practical diagnostic under the oracle interface, not as a standalone causal decomposition.Its interpretation is therefore limited to the evaluation setting in which the oracle interface is available.

Ethics Statement · A. More Related Work

The paper uses only synthetic visual observations without human or sensitive data, and situates RNG-Bench within research on memory, multimodal understanding, reasoning, and MLLM evaluation.

  • Ethics Statement: All visual observations are synthetically generated, with no human-subject data, personal or sensitive information, real people, or real-world copyrighted content.
  • A. More Related Work: Memory benchmarks assess retaining, updating, and organizing information across conversations, sessions, continual feedback, or multi-turn accumulation.
  • A. More Related Work: LoCoMo and LongMemEval use question answering over dialogue histories, while MemoryBench, MemBench, and MemoryAgentBench study continual or multi-turn memory.
  • A. More Related Work: RNG-Bench requires grounding visual observations across multiple turns, building on vision-language pre-training and visual instruction tuning.
  • A. More Related Work: Recent reasoning research spans chain-of-thought prompting, latent or non-autoregressive reasoning, step-level supervision, diffusion language models, and visual reasoning.
  • A. More Related Work: MLLM evaluation and alignment research includes broad benchmarks, open-source toolkits, multimodal instruction-following tests, and human-preference alignment benchmarks.

B. More Analysis

Additional analyses show that performance deteriorates sharply with scale and visual ambiguity, while external memory and richer history help unevenly across models. Errors reflect distinct memory, mapping, and control bottlenecks, with text observations often outperforming images in Matching Pairs.

  • Scale and memory: Matching Pairs Score% falls from 90.6% at 4×4 to 0.7% at 12×12, while response cost per matched pair rises from 4.59 to 720.3D Maze Game Score peaks at 7×7 (66.7%) before declining from 9×9 onward, alongside declining Explore%.
  • External memory: Memory maps roughly double Matching Pairs Score% for Qwen3.5-397B (38.3 →78.7) and Kimi-K2.5 (43.3 →80.3), whereas minimaps yield smaller 3D Maze gains.On 3D Maze, Qwen3.5-397B improves from 23.8 to 40.2 and Kimi-K2.5 from 24.6 to 35.6, indicating bottlenecks beyond memory access.
  • Observation modality: Text players defeat image players in 100% of Kimi-K2.5 and Qwen3.5-397B duels, with gaps of +22.7 to +35.7 pairs; Seed-2.0 is the exception.For Seed-2.0, the image player wins on 8×8 (−3.3), while the text player narrowly wins on 8×10 (+1.8).
  • Visual patterns: Matching Pairs performance is highly sensitive to identifier distinctiveness: Qwen3.5-397B drops from 75.8% (ASCII) to 20.0% (Poker), and Kimi-K2.5 from 72.5% to 30.1%.Wall color tags do not consistently improve 3D Maze, unlike the strong visual-pattern effect in Matching Pairs.
  • Spatial mapping: Ask-output prompting raises Seed-2.0-Lite completion from 1/5 to 4/5, but does not improve completion for Kimi-K2.5 or Qwen3.5-397B.Successful Kimi and Qwen runs are near-optimal when they occur, while Seed-2.0-Pro maintains 3/5 success under the prompting condition.
  • History dependence: Seed-2.0-Pro achieves 3/5 success with full history but 2/5 with windows of 3, 5, or 10 turns, showing that history effects are model-dependent.Kimi-K2.5 drops from 2/5 under full history to 1/5 at windows of 5 and 10, while Seed-2.0-Lite remains unstable at 1–2/5.

C. Visualization Cases

Trajectory visualizations distinguish spatial drift, local oscillation, and reasoning–action mismatch as recurring failure modes in 3D Maze navigation. Successful runs maintain coordinate- or landmark-grounded spatial reasoning, while failures persist despite minimaps or perceptual aids when position tracking breaks down.

  • 3D Maze failure modes: The visualizations identify spatial drift, local oscillation, and reasoning–action mismatch as three common failure modes.Successful and failed runs on the same maze expose where reasoning diverges from effective navigation.
  • Minimap trajectories: 53 steps: Seed-2.0 reaches the minimap maze goal with Eff 0.47 by grounding each move in explicit coordinates and headings.Its trajectory advances steadily toward the goal, unlike the failed comparison run.
  • Minimap trajectories: 236 steps: Kimi-K2.5 exhausts the same minimap maze budget without reaching the goal, oscillating after losing spatial closure.The failure includes repeated wall collisions and shows that an external minimap does not by itself ensure consistent position tracking.
  • No-minimap trajectories: 28 steps: Seed-2.0 reaches the no-minimap 7×7 maze near-optimally, whereas Kimi-K2.5 revisits cells and oscillates despite recognizing dead ends.Seed-2.0 maintains coordinate-level awareness, while Kimi-K2.5 lacks the spatial map needed to act on its realization.
  • Visual disambiguation: 53 steps: Color-tagged walls help Seed-2.0 navigate the 7×7 maze, while uniform walls lead the same model to lose orientation and time out.Color tags provide perceptual anchors that stabilize position tracking but do not eliminate failures.

D. Matching Pairs Case Studies · E. Potential Risks

The case studies show that model performance depends on maintaining and reusing hidden card-location bindings, while benchmark results should be interpreted only as controlled evidence about hidden-state tracking. The benchmark is intended for research evaluation and diagnosis, not direct safety-critical deployment decisions.

  • D. Matching Pairs Case Studies: 31 of 50 pairs: GPT-5.4 finishes ahead of Gemini-3.1-Pro’s 16 of 50 on the same 10×10 noise board.The single-player trajectory compares both models under identical board conditions.
  • D. Matching Pairs Case Studies: Gemini-3.1-Pro exhibits plateau phases in which continued card flips produce no additional matches.These plateaus indicate difficulty maintaining and reusing card-location bindings after the cards disappear from the current observation.
  • D. Matching Pairs Case Studies: 15 pairs: Gemini-3.1-Pro beats GPT-5.4’s 9 when Gemini-3.1-Pro moves first on the same 8×10 texture board.The duel trajectory evaluates both models under a fixed board and player order.
  • D. Matching Pairs Case Studies: 18 pairs: Gemini-3.1-Pro still beats GPT-5.4’s 6 when GPT-5.4 moves first.The result suggests the duel outcome is not mainly driven by first-move advantage.
  • D. Matching Pairs Case Studies: Gemini-3.1-Pro gains more matches late in the game and appears to use earlier observations more effectively.This pattern contrasts with GPT-5.4’s stronger single-player result on the 10×10 noise board.
  • E. Potential Risks: Benchmark scores should not be over-interpreted as complete measures of model intelligence or real-world reliability.The authors report task settings, metrics, and limitations, framing RNG-Bench as controlled evidence about hidden-state tracking rather than a deployment decision tool.

F. SFT Training Details

Qwen3.5-9B fine-tuning uses a shared full-model recipe with frozen vision components, while opt32k and rmix32k differ only in training-data composition. The configurations combine environment rollouts with optimal-policy data under controlled sequence, optimization, and evaluation settings.

  • Finetuning scope: Full fine-tuning updates the language model while freezing the vision tower and multimodal projector, using the qwen3_5_nothink chat template.The same fine-tuning recipe applies across configurations; only training-data composition changes.
  • Optimization: 1e−5 learning rate, cosine scheduling, 0.1 warmup ratio, 1 epoch, and effective batch size 128 define the AdamW optimization setup.The effective batch size uses per-device batch size 1, gradient accumulation 16, and 8 GPUs.
  • Sequence and visual budget: 28,160-token and 29,000-token cutoff lengths are used for opt32k and rmix32k, respectively, with 65,536-pixel per-image limits and patch cropping enabled.Each configuration uses one training seed, while downstream evaluation aggregates over 5 environment seeds.
  • Data composition: opt32k combines 16k Matching Pairs and 16k 3D Maze optimal-policy rollouts, while rmix32k replaces each block with a 16k rollout-plus-optimal mixture.Both datasets contain approximately 32k records and are concatenated using the concat mix strategy.
Loading 2606.19338v1…