PaperlayerView exact revision on arXiv

Source-linked AI summary

MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents

Vernon Toh, Navonil Majumder, Zhengyuan Liu, Nancy F. Chen, Soujanya Poria

arXiv:2608.31022v1cs.AIcs.CV

TL;DR

Existing benchmarks do not cleanly isolate how agents construct and interpret perceptual states under partial observability. MNIST-PRO addresses this by converting MNIST into sequential glimpse-based search with lookback constraints and comparing memory representations. The evaluation reveals a clear gap between full-observability recognition and agentic perception, with bottlenecks in state construction and interpretation, exploration, and belief updating.

  • Problem

    Existing benchmarks struggle to isolate perceptual-state construction and interpretation because they either provide visual inputs upfront or add physical and control complexities.

  • Method

    MNIST-PRO formulates MNIST recognition as a partially observable, sequential glimpse-based benchmark with lookback constraints and evaluates multiple memory representations and sensing conditions.

  • Results

    Experiments reveal a clear performance gap between full-observability visual recognition and agentic perception, with performance depending substantially on evidence representation and agents often committing before sufficient exploration.

  • Takeaways & Limitations

    Accurate agentic perception requires more than acquiring visual evidence: agents must represent, integrate, interpret, and decide when the perceptual state is reliable enough to act.

  • Takeaways & Limitations

    In the native multi-turn setup, the variables are not independently controlled, making it difficult to determine which perceptual-processing stage caused a failure.

Abstract

from arXiv · show

AI agents in partially observable environments need to coordinate active sensing with working memory to maintain an evolving perceptual state. However, existing benchmarks struggle to isolate this perceptual-state construction and interpretation capability because they introduce physical and control complexities. We address this with MNIST-PRO, a benchmark that isolates agentic perception by converting MNIST digit recognition into a sequential, glimpse-based search task with lookback constraints. We evaluate ten multimodal models across four memory representations, including raw visual history, textual states, structured metric grid maps, and a consolidated visual canvas. While models excel under full observability, partial observability exposes a clear performance gap. We identify three distinct bottlenecks. First, perceptual-state construction and interpretation present a challenge, as agents struggle to integrate fragmented glimpses. Second, agents often stop exploring before they see the full sequence. Third, models often fail to revise early, incorrect beliefs even when faced with subsequent contradictory evidence. These results show that simply acquiring visual evidence is not enough. Agents must also be able to build and update a reliable perceptual state.

1 Introduction

MNIST-PRO isolates agentic perception by turning MNIST recognition into sequential, partially observable exploration with memory and lookback constraints. Experiments show that agents struggle to construct, represent, interpret, and update perceptual states, especially as task demands increase.

  • Benchmark and motivation: Existing benchmarks either provide visual inputs upfront or introduce control complexities that make perceptual-state construction difficult to isolate.MNIST-PRO is designed to distinguish failures in evidence acquisition, state consolidation, and interpretation.
  • Benchmark and motivation: MNIST-PRO converts fully observed MNIST recognition into a controlled, partially observable benchmark requiring agents to acquire, integrate, retain, and interpret glimpses over time.The benchmark uses localized glimpses instead of the full image and removes physical distractions through a 2D canvas.
  • Findings: Multimodal agents with strong fully observable MNIST recognition often struggle to integrate fragmented evidence into a coherent perceptual state and make correct decisions from it.The results expose a substantial gap between visual recognition and agentic perception under partial observability.
  • Findings: Raw visual history, textual states, metric grid maps, and consolidated visual canvases produce markedly different results, with no representation consistently dominating across agents.Constructing a consolidated canvas only after exploration may outperform continuous consolidation, which the authors hypothesize can expose confusing incomplete states.
  • Findings: Increasing from one digit to an ordered, partially observable two-digit sequence makes perceptual-state construction substantially harder and challenges state maintenance and interpretation.Some agents collect substantial evidence from both digits but still fail to produce correct final predictions.
  • Findings: Agents often commit before exhausting their sensing budget, while using more sensing alone does not guarantee success.Effective behavior requires deciding both where to gather evidence and when the perceptual state is reliable enough to support action.

2 MNIST-PRO

MNIST-PRO isolates agentic perception by requiring agents to navigate partially observable MNIST canvases, integrate sequential glimpses, maintain perceptual state, and predict under sensing constraints.

  • Benchmark design: MNIST-PRO masks the image and reveals only a fixed-size glimpse, requiring sequential exploration to observe the full canvas.The environment isolates spatial tracking and memory without additional visual noise or rendering complexity.
  • POMDP formulation: The benchmark models agentic perception as a POMDP with hidden canvases, glimpse coordinates, masked observations, movement actions, and terminating predictions.Movement shifts the glimpse window, while prediction actions end the episode.
  • Agentic loop: Agents update a perceptual state from past observations and actions because the current observation reveals only part of the underlying state.This perceptual state serves as a running estimate of the environment’s underlying state.
  • Agentic loop: The agentic loop separates evidence acquisition, perceptual-state construction, state interpretation, and final prediction before committing to an action.The perceptual state is updated as z_t = F_θ(z_t−1, o_t, p_t), while prediction factors into interpretation and task decoding.
  • Task structure: Two task levels progressively test spatial integration, long-horizon search, sequential retention, and order tracking across single and concatenated digits.Level 1 uses one centered digit; Level 2 requires locating, recognizing, and preserving the order of multiple digits.
  • Scaling difficulty: Reducing the glimpse size or increasing the stepwise spatial demand increases the memory required to retain information over time.The benchmark varies observation-window dimensions and step size while keeping the underlying dataset unchanged.
  • Memory representations: MNIST-PRO compares internal, externalized, and procedural memory, including visual history, textual states, metric maps, consolidated canvases, and persistent skills.The harness evaluates whether agents discover strategies, reuse procedures, and revise beliefs using feedback.

3 Experiments and Results

The experiments compare recognition from full images with sequential-glimpse performance, then analyze interaction trajectories, alternative state representations, autonomous procedures, and visual-history conditions.

  • Experimental program: The evaluation begins by comparing full-image recognition with performance from sequential glimpses.This comparison separates recognition under full observability from agentic perception under partial observability.
  • Experimental program: Trajectory replay with alternative state representations is used to identify where errors arise during the agentic process.The experiments also vary visual history and glimpse size.

3.1 Experimental Setup

The setup evaluates ten multimodal models on partially observable single- and multi-digit tasks using controlled episodes, step budgets, and three complementary metrics.

  • Data and models: The dataset preprocessing inverts, upscales, and binarizes MNIST images before using them as Level 1 inputs and part of Level 2 inputs.The supplied passage describes 28 × 28 images upscaled to 224 × 224 pixels and thresholded at intensity 200.
  • Data and models: Ten proprietary and open-source multimodal models are evaluated using their default API configurations.The model set includes eight proprietary models and two open-source models.
  • Episode settings: The evaluation uses 100 test episodes for each task under partially observable active vision.Level 1 episodes are balanced across digit classes, while Level 2 contains two independently sampled digits.
  • Episode settings: Each episode starts with a random 64 × 64 glimpse, moves in 32-pixel steps, and ends at prediction or the step budget.The supplied setup specifies separate step budgets for the two levels elsewhere in the section.
  • Metrics: The evaluation reports control accuracy, partial-observability classification accuracy, and average step count.Control accuracy provides an upper bound, while average steps indicate exploration efficiency.

3.2 Native Multi-turn Setup

The native multi-turn setup gives agents an initial prompt and glimpse, then requires them to explore through image-only turns and respond with actions. Results show substantial differences between exploration effort and prediction accuracy, while the setup does not independently control the sources of failure.

  • Native Multi-turn Setup: Agents receive the task prompt and first glimpse, then subsequent turns provide only an image while agents continue acting.The setup disables tool use to systematically ablate tools and study exploration from the initial instruction.
  • Results: Gemini-3.7-Flash achieves the highest reported accuracy, reaching 75.0% on Level 1 and 47.0% on Level 2.Gemini-3.6-Flash follows with 64.0% and 28.0%, while Gemini-3.1-Pro-Preview reaches 53.0% and 18.0%.
  • Results: Claude-5-Sonnet explores most extensively but achieves only 23.0% Level 1 accuracy and 0.0% Level 2 accuracy.Its average step counts are 15.96 on Level 1 and 43.24 on Level 2.
  • Results: Table 2 reports accuracy and average steps separately for Level 1, Level 2, and their mean across configurations.Bold marks the partially observable configuration with the highest average accuracy for each model.
  • Interpretation of Results: The setup does not independently control evidence acquisition, perceptual-state construction, interpretation, and commitment decisions, limiting attribution of failures.Interaction-trace analysis identifies misinterpretation of accumulated evidence as the primary cause and motivates targeted conditions.

3.3 Results of Different Internal State Construction Mechanisms

Internal-state representations affect exploration and accuracy differently across task levels and models. Partial observability is especially difficult for multi-digit sequences, where performance remains low and errors increasingly reflect search and sequence-tracking demands.

  • Recognition Versus State Construction: Gemini-3.1-Pro-Preview drops from 99.0% control accuracy to 38.0%, while Claude-5-Sonnet drops from 94.0% to 24.0% under partial observability.The drops persist even when models can look back at all previous glimpses, showing that full visual history does not close the gap.
  • Memory Representations: Coordinate tracking affects models differently: Gemini-3.1-Pro-Preview falls from 56.0% with textual memory to 53.0% with a grid map, while Gemini-3.7-Flash rises from 54.0% to 57.0%.The added path-integration work can degrade performance for some models but provide a stable geometric state for others.
  • Memory Representations: For Level 1, seven of eight proprietary models perform best with raw visual history, whereas Level 2 often favors textual states or metric grid maps.Examples include Gemini-3.6-Flash at 53.0% versus 47.0% in Level 1, and Gemini-3.7-Flash at 28.0% with a grid map in Level 2.
  • Failure Analysis: The most frequent Level 1 substitutions are 3→2, 0→2, and 4→9.These substitutions characterize identity errors in single-digit recognition.
  • Failure Analysis: Figure 3 decomposes recognition, positional and exact-sequence accuracy, outcome distributions, and visual-coverage effects to analyze scaling failures.Its Level 2 framing highlights search and sequence-construction failures beyond single-digit recognition.
  • Memory Representations: Results across textual state and metric-grid representations show no consistent pattern across models.Different models achieve different comparative outcomes between the two state representations.
  • Multi-Digit Sequences: No model exceeds 30.0% accuracy on Level 2 under partial observability, regardless of memory representation.Examples include 5.0% for Gemini-3.1-Pro-Preview with image-only input and 28.0% for Gemini-3.7-Flash with a metric grid map.
  • Exploration Steps: Restricting visual lookback lengthens exploration: Gemini-3.1-Pro-Preview rises from 4.52 steps with unbounded history to 7.18 textual-state steps and 7.15 metric-grid steps.The comparison shows that persistent written memory can require more exploration when prior visual frames are unavailable.

3.4 Diagnostic Analyses

Diagnostic analyses show that performance depends on how agents explore, construct perceptual states, interpret them, and use autonomous tools—not merely on recognizing digits.

  • Task Scaling: 39.0% single-digit accuracy across configurations drops substantially when agents must preserve the order of two digits.The drop occurs despite near-perfect control accuracy under full observability.
  • Failure Modes: 24.0% of Level 2 outputs are invalid, while exact sequences account for only 10.2% of results.One-position-correct outcomes account for 31.5%.
  • Failure Modes: 323 Level 2 predictions contain only one digit, including 238 episodes that never explore the other digit.A further 266 episodes expose less than 25% of the second digit.
  • Failure Modes: Gemini-3.7-Flash retained an incorrect digit belief in 17 of 39 examined cases despite representing states containing two digits.This contrasts with its rare early stopping under Textual State and Metric Grid Map conditions.
  • Coverage and Success: Exact-sequence accuracy rises from 1.0% at ≤25% minimum coverage to only 26.0% at the highest coverage range.Looking at both digits is necessary, but greater coverage alone does not ensure correct interpretation.
  • External State Construction: Offline canvas construction improves every agent, while online canvas construction can underperform because it commits earlier.For GPT-5.6-Terra on Level 1, online accuracy is 31.0% after 6.91 steps versus 56.0% for offline Textual State + Canvas after 24.09 steps.
  • Harness Systems: Harness autonomy improves Gemini-3.7-Flash from 68.0% to 88.0% on Level 1 and from 38.0% to 63.0% on Level 2.The gains coincide with approximately twice as many sensing steps on Level 2.
  • Harness Systems: Persistent memory and correctness feedback do not exceed Gemini-3.7-Flash’s no-memory Agentic MCP condition.Average accuracy is 75.5% without persistent memory, compared with 73.5% with it and 74.5% with correctness feedback.

3.5 Stress Tests

Stress tests show that longer visual lookback and larger glimpse windows improve efficiency and task success, especially when agents face longer multi-digit horizons.

  • Glimpse History: H=1 provides zero visual lookback, intermediate histories retain recent frames, and H=∞ permits full lookback.The ablation varies visual glimpse history from 1 to ∞.
  • Glimpse History: Increasing textual-state history from H=1 to H=∞ reduces Level 2 Gemini-3.6-Flash steps from 39.29 to 15.89.The reduction is nearly 60% and illustrates the larger benefit on longer horizons.
  • Glimpse History: Gemini-3.7-Flash Level 1 textual-state success rises from 54.0% at H=1 to 64.0% at H=∞.Metric Grid Map accuracy also rises from 57.0% to 64.0%.
  • Glimpse Window Size: Gemini-3.7-Flash Level 2 Image Only accuracy increases from 3% to 74% as the glimpse window grows from 32 × 32 to 128 × 128.Larger windows reveal more evidence per observation and reduce exploration steps.

4 TL;DR: 1) Seen ≠Using; 2) Same Evidence + Better Representation → Better Use

MNIST-PRO separates seeing evidence from using it effectively: accuracy depends on evidence quality, state representation, and interpretation, while premature stopping remains a distinct failure mode.

  • Seen ≠ Using: High stroke coverage does not guarantee high accuracy, showing that acquired evidence can remain poorly composed or interpreted.Claude-5-Opus, Claude-5-Fable, and GPT-5.6-Sol (xhigh) exhibit this pattern.
  • Representation: Offline visual canvases substantially improve Claude-5-Opus and Claude-5-Fable, and also improve GPT-5.6-Sol (xhigh).The assisted representation changes how already acquired evidence is organized without necessarily changing acquisition.
  • Interpretation: Replacing GPT-5.6-Sol (xhigh) with Gemini-3.7-Flash raises Level 2 Image Only accuracy from 59% to 74% on the same trajectories.The smaller Textual State increase from 26% to 33% reflects information lost during acquisition.
  • Evidence Quality: Gemini-3.7-Flash reaches 74% on GPT-5.6-Sol (xhigh) Image Only canvases but only 41% on canvases from its own trajectories.Aggregate coverage alone therefore does not characterize evidence usefulness.
  • Three Bottlenecks: Predictor-swap experiments separate evidence acquisition, perceptual-state organization, and interpretation as distinct factors.The comparison reverses across Image Only and Textual State trajectories.
  • Premature Decisions: GLM-4.6V, GPT-5.6-Sol, and Qwen-3.8-27B often commit with limited coverage despite enough Level 1 budget for a systematic sweep.This behavior is characterized as overoptimistic stopping.

5 Related Work

Related benchmarks cover passive vision, active visual search, and working memory under partial observability, but differ in whether they isolate perceptual-state construction.

  • Vision-Language Benchmarks: Most vision-language benchmarks present complete visual inputs once and evaluate passive recognition or reasoning.Examples include MMBench, MMMU, MM-Vet, SEED-Bench, BLINK, and MMT-Bench.
  • Active Perception and Visual Search: Active-perception benchmarks restrict the initial field of view and evaluate how models shift their gaze to resolve visual ambiguities.ActiView and ActiveVision are cited examples.
  • Working Memory under Partial Observability: Working-memory evaluations commonly use embodied navigation, spatial-temporal mapping, or GUI interaction under partial observability.Examples include ALFRED, SpatialBench, VSI-Bench, and OSWorld.

6 Conclusion

MNIST-PRO shows that partial-observability perception requires more than acquiring visual evidence: agents must actively explore and represent and interpret visual history. Performance and behavior vary by memory representation, with visual canvases helping some models while others fail to interpret them or explore informative regions.

  • MNIST-PRO transforms MNIST recognition into sequential, glimpse-based search with lookback constraints to evaluate exploration and perceptual-state construction.
  • The benchmark reveals a clear performance gap between full-observability visual recognition and agentic perception.
  • Simply acquiring visual evidence does not guarantee accurate decision-making; success depends on how agents represent and interpret visual history.
  • Programmatically consolidating glimpses into a visual canvas improves accuracy for some models, but other models fail to interpret canvases or explore informative regions.
  • Capable agents require both active sensory control and structured state representation rather than relying solely on expanding raw context windows.
Loading 2608.31022v1…