MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents

1 Introduction

• MNIST-PRO converts MNIST into a controlled partially observable task where agents explore through localized glimpses, enabling evaluation of perceptual-state construction while limiting physical distractions.Near-perfect recognition under full observability provides a high ceiling for basic visual perception, while the 2D canvas supports tracking image coverage and exploration paths.Source: §1 · pp. 2–3
• Strong MNIST recognition does not ensure effective agentic perception: agents struggle to integrate fragmented glimpses and may misinterpret even successfully constructed perceptual states.The benchmark reveals a gap between fully observable visual recognition and partially observable perception.Source: §1 · pp. 3–4
• Raw visual history, textual states, metric grid maps, and visual canvases produce different results, with no representation consistently best; consolidation timing also matters.Continuous visual consolidation after each exploration step can be less effective than exploring first and consolidating at the end; the authors hypothesize that exposing incomplete states may confuse agents.Source: §1 · pp. 3–4
• Moving from one digit to an ordered, partially observable two-digit sequence makes perceptual-state construction harder, and agents can gather evidence from both digits yet predict incorrectly.The longer horizon challenges not only evidence acquisition but also maintenance and interpretation of the constructed state.Source: §1 · p. 3
• Agents often commit before exhausting their sensing budget, but using more of that budget alone does not ensure success.They must choose where to gather further evidence and when the perceptual state is reliable enough for a decision.Source: §1 · pp. 3–4
• MNIST-PRO diagnoses whether failures stem from insufficient evidence acquisition, ineffective consolidation into a perceptual state, or incorrect interpretation of that state.Its evaluation framework varies visual lookback, sensing capacity, task horizon, and memory representation.Source: §1 · p. 4

2 MNIST-PRO

• The benchmark uses a fixed-size glimpse window that masks the image, making performance depend on exploration, combining successive glimpses, and remembering prior observations.This setup isolates spatial tracking and memory without additional visual noise or rendering complexity.Source: §2 · p. 4

2.1 POMDP Formulation

• In the POMDP, agents see only a masked region at each step, move or predict, and must update information from past observations and actions to estimate the hidden state.Episodes end at a prediction or the step budget, which is set to allow canvas coverage with the specified stride.Source: §2.1 · p. 5

2.2 The Required Agentic Loop

• The agentic loop separates perceptual-state construction from interpretation: agents update state from observations and locations, then read it out for prediction and action selection.Figure 2 depicts evidence acquisition, state construction, and state interpretation as distinct workflow components.Source: §2.2 · p. 6, Figure 2 · p. 6

2.3 Benchmark Task Levels and Complexity

• The two task levels progress from integrating spatially disjoint features in one digit to searching concatenated digits and retaining their sequence order.Window dimensions and step size control difficulty; smaller windows lengthen coverage trajectories and increase memory demands without changing the dataset.Source: §2.3 · pp. 6–7

2.4 A Taxonomy of Memory for Multimodal Agents

• The memory taxonomy covers internal and externalized state construction plus procedural memory, with the main benchmark using image-only, textual-state, and metric-grid configurations.The visual memory canvas, in online and offline modes, probes how spatial information derived from partial inputs can be combined.Source: §2.4 · p. 7

2.4.1 Internal State Construction

• Internal-state configurations range from visual history without a written summary to textual or coordinate-grounded spatial memory for tracking glimpses over time.Textual State and Metric Grid Map use zero visual lookback; the latter records coordinates and features in a structured map retained through conversation history.Source: §2.4.1 · pp. 7–8

2.4.2 Externalized State Construction

• Externalized visual memory offers an online canvas that aligns and accumulates glimpses, and an offline canvas that tests final interpretation separately from sequential planning.Online mode supplies both the consolidated canvas and latest glimpse, keeping visual input constant while the environment handles alignment and fusion.Source: §2.4.2 · pp. 8–9

2.4.3 Persistent Memory

• The persistent-memory harness tests whether tool-using agents discover reusable strategies, retain procedures across episodes, and revise them using correctness feedback.Agents may autonomously assemble glimpses into a canvas or choose how many observations to use for prediction.Source: §2.4.3 · p. 9

• The experiments first compare full-image recognition with performance from sequential glimpses.Source: §3 · p. 9
• Trajectory analysis and replay with alternative state representations examine where errors arise.Source: §3 · p. 9
• Harness experiments test whether models discover and reuse their own procedures.Source: §3 · p. 9
• The experiments also vary visual history and glimpse size.Source: §3 · p. 9

Experimental Setup

• Images are inverted, upscaled to 224 × 224, and binarized at intensity 200; Level 2 concatenates two digits into a 224 × 448 image.These processed binary images are the inputs to the single-digit and multi-digit tasks, respectively.Source: §3.1 · p. 9
• The study evaluates ten proprietary and open-source models with default API settings, plus GPT-5.6-Sol at xhigh reasoning effort in selected experiments.The open-source models are Qwen-3.8-27B and GLM-4.6V (Team et al., 2025).Source: §3.1 · p. 9
• Agents inspect 100 episodes per task through 64 × 64 glimpses, moving in 32-pixel steps before predicting or exhausting 36-step or 78-step budgets.Level 1 episodes are balanced across ten digit classes; Level 2 samples two digit labels independently and uniformly from 0–9.Source: §3.1 · pp. 9–10
• Evaluation reports fully observable control accuracy, partial-observability classification accuracy, and average steps to prediction as a measure of exploration efficiency.Control accuracy on the unmasked image provides an upper bound.Source: §3.1 · p. 10

3.2 Native Multi-turn Setup

• Without tool use, agents receive a task prompt and first glimpse, then only an image each turn; the full trace of images, thoughts, and actions remains available.This setup is designed to examine exploration from the initial instruction while systematically ablating tools.Source: §3.2 · p. 10
• Gemini-3.7-Flash has the top native multi-turn accuracy (75.0% Level 1; 47.0% Level 2), ahead of Gemini-3.6-Flash (64.0%; 28.0%) and Gemini-3.1-Pro-Preview (53.0%; 18.0%).Source: §3.2 · p. 10
• Claude-5-Sonnet explores most extensively but reaches only 23.0% accuracy on Level 1 and 0.0% on Level 2.It averages 15.96 steps on Level 1 and 43.24 on Level 2; Claude-5-Opus, Claude-5-Fable, and GPT-5.6-Terra also use more steps for moderate or poor accuracy.Source: §3.2 · p. 10
• GPT-5.6-Sol, Qwen-3.8-27B, and GLM-4.6V often stop after few steps before gathering enough information, yielding 0.0% Level 2 accuracy and 37.0%, 29.0%, and 13.0% on Level 1, respectively.Their average step counts are 5.74, 4.62, and 2.76 on Level 1, and 13.99, 9.23, and 5.25 on Level 2, respectively.Source: §3.2 · p. 10
• Trace analysis finds failures are primarily due to misinterpreting accumulated evidence, though this setup cannot independently isolate acquisition, state construction, interpretation, or commitment.The authors use targeted experimental conditions next to isolate these components.Source: §3.2 · p. 11

The Gap Between Recognition and Perceptual State Construction

• Under control, models achieved 83.0% to 99.0% accuracy, but partial observability sharply reduced performance even with unlimited lookback.Gemini-3.1-Pro-Preview fell from 99.0% control accuracy to 38.0%, and Claude-5-Sonnet from 94.0% to 24.0%.Source: §3.3 · p. 11

Memory Representations Help and Hurt Differently

• With no visual lookback, grid maps hurt Gemini-3.1-Pro-Preview (56.0% textual vs. 53.0% grid) but helped Gemini-3.7-Flash (54.0% vs. 57.0%).Coordinate tracking can add path-integration burden for some models while providing a more stable geometric state for others.Source: §3.3 · p. 11

Textual State Does Not Show a Consistent Pattern

• Seven of eight proprietary models performed best with raw visual history on Level 1, whereas Level 2 favored textual or grid memory inconsistently across models.On Level 1, Gemini-3.6-Flash scored 53.0% with visual history versus 47.0% with textual state; Level 2 comparisons varied by model and representation.Source: §3.3 · pp. 11–12

Challenges of Multi-Digit Sequences

• 30.0% was the ceiling no model exceeded on Level 2 under partial observability, and most scored below 20.0% with image-only history.Gemini-3.1-Pro-Preview reached 20.0% with textual state and Gemini-3.7-Flash 28.0% with a metric grid map.Source: Figure 3 · p. 12, §3.3 · p. 12

Influence of Visual History on Exploration Steps

• 4.52 steps was Gemini-3.1-Pro-Preview’s Level 1 average with unbounded visual history, compared with 7.18 textual-state steps and 7.15 grid-map steps.Restricting visual lookback generally lengthened exploration before prediction.Source: §3.3 · p. 12

3.4.1 Internal Perceptual-State Failure Modes

• Scaling from single digits to sequences sharply reduces recognition accuracy, and greater visual coverage alone does not resolve the failures.Level 2 positional accuracy is around 23.5%–29.0% and exact-sequence accuracy around 9.5%–11.5%, versus roughly 39.0% Level 1 accuracy; exact-sequence accuracy reaches only 26.0% even when agents see more than 75% of both digits. Audits identify premature stopping, mismatches between stated and actual exploration, and failure to revise incorrect beliefs; Gemini-3.7-Flash rarely predicts after seeing only one digit, but still carries incorrect identities forward.Source: §3.4.1 · p. 13

3.4.2 External Perceptual-State Construction Helps

• Offline canvases improve accuracy across agents, with Textual State + Canvas consistently outperforming Image Only + Canvas.For Claude-5-Opus, Textual State + Canvas raises accuracy from 13.0% to 67.0% on Level 2, versus 36.0% for Image Only + Canvas; for Claude-5-Fable, the corresponding figures are 18.0%, 69.0%, and 42.0%. Because the trajectories are fixed, the gains localize much of the error downstream of evidence acquisition; online canvas construction can perform worse when it prompts earlier commitment.Source: §3.4.2 · pp. 13–14, Table 3 · p. 14, L1 L2 Avg L1 L2 Avg L1 L2 Avg L1 L2 Avg · p. 16

3.4.3 Autonomous Harnesses Help but Persistent Memory Doesn’t

• Gemini-3.7-Flash’s harness raises accuracy over Memory Canvas from 68.0% to 88.0% on Level 1 and from 38.0% to 63.0% on Level 2, while using more steps.It uses 15.67 versus 5.56 steps on Level 1 and 41.00 versus 20.40 on Level 2; harness behavior includes constructing and inspecting canvases, though system-prompt steering may also contribute. In a limited matched evaluation, Claude-5-Opus reaches 80.8% accuracy (21/26) with the harness versus 46.2% (12/26) under native multi-turn interaction; GPT-5.6-Terra shows no meaningful harness accuracy improvement.Source: Table 4 · p. 15, L1 L2 Avg L1 L2 Avg L1 L2 Avg L1 L2 Avg · pp. 15–17
• Persistent memory does not materially improve accuracy for GPT-5.6-Terra or Gemini-3.7-Flash, despite retaining or standardizing useful canvas-building procedures.GPT-5.6-Terra executes its stored procedure in only approximately 2% of episodes, and Level 1 accuracy is 31.0% without persistent memory, 32.0% with it, and 28.0% with correctness feedback. Gemini-3.7-Flash averages 75.5% without memory, 73.5% with memory, and 74.5% with memory and feedback; its exploration effort increases especially on Level 2 without corresponding accuracy gains.Source: L1 L2 Avg L1 L2 Avg L1 L2 Avg L1 L2 Avg · pp. 15–16

3.4.4 Exploration and Spatial Integration

• Structured metric grid maps reduce redundant exploration and preserve tracking better than unstructured textual states as tasks grow more complex.Gemini-3.7-Flash’s Level 2 revisit rate falls from 45.1% with textual state to 13.6% with a grid map, which retains 61.7% canvas coverage; full visual history can also terminate exploration prematurely, as Gemini-3.1-Pro-Preview reaches 57.7% coverage across 3.5 moves versus 71.7% across 6.2 moves with a grid map.Source: §3.4.4 · p. 18
• Exploration efficiency varies across model groups: open-source models show severe looping or limited coverage, while proprietary models trade off coverage and path efficiency.GLM-4.6V has revisit rates of 61.8% on Level 1 and 62.3% on Level 2 with textual state, and covers only 46.7% and 33.1% with a grid map. Claude-5-Fable achieves high stroke coverage, whereas GPT-5.6-Terra makes 61.4 moves with a 46.1% revisit rate under the Level 2 grid-map configuration.Source: §3.4.4 · p. 18

3.4.5 Digit Topology

• Recognition success declines as digit topology grows more complex: simple strokes are easiest, curves harder, and closed loops hardest.Under zero-lookback, metric grid maps improve curve accuracy for Gemini-3.7-Flash from 36.7% to 56.7%, but reduce its closed-loop accuracy from 57.5% with textual state to 45.0%.Source: §3.4.5 · p. 19

3.5.1 Visual-History Horizon

• On Level 2 Textual State, Gemini-3.7-Flash accuracy rose from 6.0% (H=1) to 23.0% (H=∞), while Metric Grid Map accuracy rose from 28.0% to 35.0%.On Level 1, Textual State accuracy rose from 54.0% to 64.0% for Gemini-3.7-Flash and from 47.0% to 58.0% for Gemini-3.6-Flash; Metric Grid Map accuracy rose from 57.0% to 64.0% for Gemini-3.7-Flash. H=1 provides zero visual lookback; H=∞ allows full lookback.Source: §3.5.1 · pp. 19–20, Table 7 · p. 20
• For Level 2 Textual State, Gemini-3.6-Flash average steps fell from 39.29 (H=1) to 15.89 (H=∞), a nearly 60% reduction.Longer history reduces exploration steps across models and tasks; Gemini-3.7-Flash’s Level 2 average steps fell from 34.84 to 15.90 under the same conditions.Source: §3.5.1 · pp. 19–20, Table 7 · p. 20

3.5.2 Glimpse Size

• On Level 2 Image Only multi-digit recognition, Gemini-3.7-Flash accuracy rose from 3% to 74% as the glimpse window increased from 32 × 32 to 128 × 128.Larger windows consistently improve performance and reduce average steps by revealing more visual evidence per observation.Source: §3.5.2 · p. 20

Same Evidence + Better Representation → Better Use

• On Level 1, using Gemini-3.7-Flash instead of GPT-5.6-Sol (xhigh) to predict from the same canvas raises accuracy from 66.5% to 84.3%, a gain of 17.8 percentage points.Because the trajectory and canvas stay fixed, the improvement isolates a difference in interpretation of the same constructed perceptual state.Source: §4 · p. 21
• On Level 2 Image Only trajectories, swapping in Gemini-3.7-Flash raises accuracy from 59% to 74%, but on Textual State trajectories it rises only from 26% to 33%.Gemini-3.7-Flash reaches 86.8% when both digits are present in the canvas, but 0% when one is absent; improved interpretation cannot recover unobserved information.Source: §4 · p. 21
• On Level 2 Image Only canvases, Gemini-3.7-Flash scores 41% on its own trajectories and 74% on GPT-5.6-Sol (xhigh) trajectories, but the pattern reverses for Textual State canvases, at 51% versus 33%.These predictor swaps show that aggregate coverage does not fully capture evidence usefulness: the more informative state depends on the trajectory and representation.Source: §4 · pp. 21–22

Seen ≠Using

• Models can acquire substantial visual evidence without using it effectively; integrating glimpses into a visual canvas improves some models’ performance, but more coverage does not guarantee higher accuracy.GPT-5.6-Sol (xhigh) trajectories observe strokes from both digits in 38% of cases, compared with 76% for Gemini-3.7-Flash, illustrating that aggregate coverage may miss whether the most informative regions were seen.Source: §4 · pp. 20–21, Figure 4 · p. 22

Overoptimistic stopping

• GLM-4.6V, GPT-5.6-Sol, and Qwen-3.8-27B often commit with low coverage while leaving sensing budget unused, a pattern characterized as overoptimistic stopping.The authors hypothesize that familiar MNIST shape priors can prompt early hypotheses; agents sometimes err despite enough remaining Level 1 budget for a systematic sweep, and exhaustive sensing still cannot guarantee correct state construction or interpretation.Source: §4 · pp. 22–23
• Unlike MNIST-PRO’s familiar domain, where priors may encourage premature stopping, agents in ScrambleToolBench often resort to exhaustive search in an unfamiliar environment (Toh et al., 2026).The authors relate both patterns to task priors and note that recent agentic-search work identifies premature stopping and evidence sufficiency as challenges (Kausik et al., 2026; Choubey et al., 2026).Source: §4 · p. 22

Vision-Language Benchmarks

• Most vision-language benchmarks present complete static visual input once, whereas MNIST-PRO measures sequential information gain through active glimpse acquisition and integration under a strict budget.Examples include MMBench, MMMU, MM-Vet, SEED-Bench, BLINK, and MMT-Bench.Source: §5 · p. 23

Active Perception and Visual Search

• Active perception benchmarks such as ActiView (Wang et al., 2025), ActiveVision (Zhang et al., 2026a), and V∗ (Wu and Xie, 2023) allow expanding visual history, unlike MNIST-PRO’s H=1 constraint.MNIST-PRO also systematically tests online and offline consolidation of past glimpses onto a merged visual canvas.Source: §5 · p. 23

Working Memory under Partial Observability

• Navigation, mapping, and GUI benchmarks evaluate memory in settings where physical noise, rendering complexity, and tool-use confound state estimation; MNIST-PRO controls these factors and structures memory representations.Examples include ALFRED (Shridhar et al., 2020), SpatialBench (Xu et al., 2026), VSI-Bench (Yang et al., 2025), and OSWorld (Xie et al., 2024).Source: §5 · p. 23

6 Conclusion

• A clear performance gap separates full-observability visual recognition from agentic perception, showing that visual evidence alone does not guarantee accurate decisions.Success depends on how agents represent and interpret their visual history.Source: §6 · p. 23
• MNIST-PRO evaluates how multimodal agents explore and represent partially observed visual environments using sequential, glimpse-based MNIST search with lookback constraints.The benchmark analyzes active exploration and state construction as distinct capabilities.Source: §6 · p. 23
• Consolidating glimpses into a visual canvas improves accuracy for some models, while others fail to interpret the canvas or explore informative regions.The findings point to a need for both active sensory control and structured state representation, rather than relying solely on larger raw context windows.Source: §6 · p. 23