PaperlayerView exact revision on arXiv

Source-linked AI summary

VISTA: A Visual Harness for Reasoning in an Interactive World

Qiushi Han, Keya Hu, Linlu Qiu, Cathy Wu, Kaiming He

arXiv:2610.02200v1cs.AIcs.CV

TL;DR

Long-horizon visual interaction is challenging when image representations lose detail and earlier observations leave context. VISTA preserves visual history for model-directed retrieval, achieving strong results on ARC-AGI-3 and outperforming minimal-harness baselines across diverse visual benchmarks.

  • Problem

    In unfamiliar interactive environments, visual details needed later may be lost in one-time image encodings or displaced by context limits, while robust AI interaction remains uncertain.

    Source: §1 · pp. 1–2

  • Method

    VISTA gives multimodal models visual observations, lossless memory of returned frames, and tools to retrieve and inspect past visual evidence during interaction.

    Source: §1 · p. 2

  • Results

    VISTA substantially outperforms same-model minimal-harness baselines across diverse visual benchmarks; Claude Opus 5.0 achieves 100.00 RHAE on ARC-AGI-3 and uses 57.4% fewer actions than first-time humans.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VISTA’s simple design and strong performance across diverse benchmarks suggest its potential as a general-purpose harness for tasks beyond games.

    Source: §7 · p. 10

    Because the evaluated models postdate the public benchmarks, the games may have appeared in training data; private ARC-AGI-3 games would provide a stronger generalization test.

    Source: §6 · p. 10

Abstract

from arXiv · show

We show that multimodal models possess strong reasoning abilities and that an appropriate harness can unlock their potential to solve tasks across diverse interactive environments. We introduce VISTA, a visual harness that gives a general-purpose multimodal model long-horizon vision. VISTA allows the model to directly perceive the environment through visual observations and maintains a lossless visual memory that preserves past observations in their original form. The model can actively retrieve these observations and reorganize its visual input as it reasons. On ARC-AGI-3, VISTA improves Claude Opus 5.0's Relative Human Action Efficiency score from 40.68 to a perfect 100.00, with the model completing all 25 public games using 57.4% fewer actions than first-time human participants. VISTA's simple design also allows it to extend naturally to diverse visual environments with minimal adaptation. Across three additional benchmarks covering a diverse range of visual games and puzzles, it substantially outperforms baselines using the same underlying model with minimal harnesses. Our results highlight VISTA's potential as a general-purpose visual harness for advancing multimodal agents in complex visual environments.

1 Introduction

The introduction asks whether AI can interact robustly with unfamiliar dynamic environments and identifies limits in one-shot visual encoding and long-horizon context. VISTA preserves visual history for model-directed inspection, achieving strong ARC-AGI-3 results and extending across other visual benchmarks.

1 Introduction

  • Existing VLMs encode images once, while long-horizon interactions may remove earlier observations or replace them with textual summaries as context fills.The introduction frames robust interaction in unfamiliar environments as an open question and studies it using ARC-AGI-3 interactive games.Source: §1 · pp. 1–2
  • VISTA stores every returned frame, including intermediate animations, in original form and lets the model retrieve frames and inspect regions as reasoning needs change.The model interleaves observation, natural-language reasoning, inspection, and action.Source: §1 · p. 2
  • 100.00 RHAE: Claude Opus 5.0 with VISTA completes all 25 public ARC-AGI-3 games using 57.4% fewer actions than first-time humans.The comparison is with official baselines scoring 40.68 for Opus 5.0 high and 13.33 for GPT-5.6 Sol max.Source: §1 · p. 2
  • Across GameWorld, AI GameStore, and BabyVision, VISTA substantially outperforms same-model baselines with minimal harnesses and beats human baselines on two benchmarks.The three evaluations cover browser games and visual tracking problems, among other visual reasoning tasks.Source: §1 · p. 2

2 Related Work

Related work spans interactive reasoning through symbolic programs, computer-use agents, and game-playing systems. VISTA instead works directly from visual observations and preserves a lossless interaction history for active retrieval.

ARC-AGI benchmarks

  • ARC-AGI-3 approaches often use program synthesis to abstract textual or symbolic observations into executable programs, but explicit abstractions may become harder as environments grow visually complex.VISTA instead operates on visual observations and reasons in natural language.Source: §2 · pp. 2–3

Computer-use agents

  • Computer-use agents range from text-based systems using interface structures to vision-based agents acting on screenshots; VISTA follows the vision-based paradigm.Its focus is a visual-centric harness for observing and interacting with environments.Source: §2 · p. 3

Game-playing agents

  • Game agents include trained per-game policies, model-based planners, and foundation-model systems, while VISTA preserves the complete visual history and supports revisiting arbitrary frames and regions.Prior agents typically keep bounded visual context or selectively store and compress past experiences.Source: §2 · p. 3

3 VISTA: A Visual Harness

VISTA addresses the limits of one-time visual encoding by preserving original observations and letting the model revisit evidence as it reasons. Its components support model-directed inspection, while a mechanical harness executes model-requested interactions.

3 VISTA: A Visual Harness

  • VISTA counters one-time encoding limits by preserving original observations in lossless memory and letting the model retrieve and inspect them during interaction.The model can reorganize visual input without relying on relevant details being captured in the initial encoding and retained in context.Source: §3 · p. 3

3.1 Visual Harness Components

  • Visual observations preserve object appearance and spatial relations; lossless memory stores every returned frame, and visual inspection enables selecting frames and regions across space and time.Frames remain available throughout interaction, including intermediate animation frames, for comparing states and replaying action outcomes.Source: §3.1 · pp. 3–4, Figure 3 · p. 4

3.2 Pipeline

  • The harness mechanically executes model-requested tool calls; within each game turn, the agent observes, reasons, may inspect past frames or read pixels, then acts.Notes for game understanding and the current state are optional; when context approaches its limit, the agent can hand off to a fresh context while notes, visual memory, and action history remain available.Source: §3.2 · p. 4

4 VISTA on ARC-AGI-3

VISTA is evaluated on ARC-AGI-3 with two multimodal backends, and its strongest results pair high action efficiency with fewer actions than human players. Ablations examine the harness components and input choices, while qualitative examples show how agents revisit visual history and inspect details.

4.1 Setup

  • ARC-AGI-3’s 25 public games require agents to infer rules and goals across levels; RHAE compares action efficiency with first-time humans and weights later levels more.Claude Opus 5.0 and GPT-5.6 Sol are evaluated independently.Source: §4.1 · p. 5

4.2 Results

  • 100.00 RHAE on 25 public games, versus 40.68 for the same model under the official implementation, used 7,302 actions—57.4% fewer than humans’ 17,135.Table 1 gives the system-level comparison; the reported Claude Opus 5.0 result is described as perfect performance without program synthesis.Source: Table 1 · p. 5, §4.2 · p. 5

4.3 Ablations

  • Text grids preserved comparable performance but slightly increased actions; they used 71.9M tokens per game versus 30.7M for image observations.Figure 6 compares RHAE, total actions, and average token use for the two observation representations.Source: Figure 6 · p. 6, §4.3 · p. 7
  • Adding lossless visual memory and model-directed inspection raised RHAE to 94.10, and exact pixel readout completed VISTA at 99.00.The preceding configurations add rendered images, higher action limits, continuous conversation, and game-model notes.Source: §4.3 · p. 6, Figure 5 · p. 6
  • 100K context reduced RHAE by only 0.4 while roughly halving token use; 4× images used 75% fewer image tokens than 8× at comparable performance.The 200K context setting achieved the highest RHAE, while 16× image scale provided no additional benefit.Source: §4.3 · p. 7, Figure 7 · p. 7
  • VISTA scored 66.93 RHAE with GLM-5.3 Flash 320B and 84.12 under 3D renderings, which required 20,640 actions.The 3D result still outperformed official textual baselines by a large margin.Source: §4.3 · pp. 7–8, Figure 8 · p. 7, Figure 9 · p. 8

4.4 Qualitative Analysis

  • Agents use visual memory inspection both to reconstruct maps across turns and to zoom into local details to infer block orientation.Figure 10 illustrates retrieval of frames across 22 turns and enlargement of small blocks by about 13×.Source: Figure 10 · p. 8, §4.4 · p. 8

5 General Visual Environments

VISTA is tested on interactive browser games and static visual reasoning tasks using the same models as the official implementations. It outperforms minimal-harness baselines across all three benchmarks and exceeds fresh human performance on two.

5.2 Results

  • VISTA exceeds reported fresh-human performance on GameWorld and AI GameStore.GameWorld reports success rate and progress; AI GameStore reports the geometric mean of human-normalized scores.Source: §5.1 · p. 9
  • VISTA consistently outperforms the corresponding minimal-harness baselines across GameWorld, AI GameStore, and BabyVision.The benchmarks include interactive games and static visual reasoning tasks; the comparisons use the same model as the official implementation.Source: §5.2 · p. 9, §5 · p. 9
  • On BabyVision’s 39 visual-tracking questions, VISTA improves accuracy from 41.0% with the official implementation to 63.2%.The static-image tasks let the agent selectively inspect regions within a single image.Source: §5.1 · p. 9, §5.2 · p. 10

6 Limitations

The authors identify unresolved training-data exposure and model capability as relevant qualifications on VISTA’s results.

6 Limitations

  • Models’ existing capabilities remain a key factor in the reported success.The authors describe VISTA as a way to elicit those capabilities, not as a replacement for them.Source: §6 · p. 10
  • Because the models were released after the public benchmarks, the authors cannot rule out that benchmark games appeared in training data.Source: §6 · p. 10
  • Evaluation on novel games, such as ARC-AGI-3’s private set, would provide a stronger test of generalization.Source: §6 · p. 10

7 Conclusion

VISTA combines raw visual observations, lossless visual memory, and retrieval of past experience to support interaction in visual environments. It completes all 25 public ARC-AGI-3 games with a perfect score, and the authors propose evaluating it on more realistic tasks.

7 Conclusion

  • VISTA completes all 25 public ARC-AGI-3 games with a perfect score and extends to diverse visual games and puzzles with minimal adaptation.The authors suggest evaluating it on more complex and realistic tasks, including embodied tasks in environments closer to the physical world.Source: §7 · p. 10
  • The harness preserves raw observations in lossless visual memory and lets the model revisit past visual experience at its original level of detail.The model can retrieve and reorganize visual input as it reasons.Source: §7 · p. 10
  • The authors suggest VISTA’s performance across diverse benchmarks indicates potential as a general-purpose harness for tasks beyond games.They identify more complex, realistic tasks, such as embodied tasks, as future evaluation targets.Source: §7 · p. 10

A.1 Baselines: Official Implementations

The appendix describes baseline setup across ARC-AGI-3, GameWorld, AI GameStore, and BabyVision. The protocols differ in their observation inputs, interaction limits, and evaluation procedures.

Baseline protocols

  • When available, the appendix uses authors’ reported benchmark results; otherwise, it reruns official implementations with reported agent configurations.Models are accessed through subscription-based CLI interfaces rather than direct API calls.Source: §A.1 · p. 15

ARC-AGI-3

  • ARC-AGI-3 baselines use the organizers’ published results for Claude Opus 5.0 and GPT-5.6 Sol, while GLM-5.3 Flash 320B results are obtained using the original codebase.The minimal interface supplies textual grids and available actions, retains up to seven intermediate frames, and discards earliest turns beyond a 175K-token context window; per-game limits are 2,000 actions for Claude and GPT-5.6 Sol and 3,000 for GLM.Source: §A.1 · p. 15

GameWorld

  • For GameWorld, the baseline reproduces the official GPT-5.6 Sol computer-use agent, using screenshots and keyboard or mouse actions, with each request including the preceding two steps and a 100-action task limit.The reproduction follows the official interaction and evaluation protocols.Source: §A.1 · p. 15

AI GameStore

  • For AI GameStore, the baseline reproduces the official evaluation loop with one decision per game second, five frames sampled at 0.2-second intervals, and 120 decisions per run.Requests include current frames and the preceding decision’s scratchpad, action, and result; the game pauses during model inference.Source: §A.1 · p. 15

BabyVision

  • For BabyVision, the official one-shot baseline gives the model an image and question in one request, without further visual inspection or interaction.The benchmark’s reference code extracts the final answer.Source: §A.1 · p. 15

A.2 Ablation Details

The ablations incrementally modify the official ARC-AGI-3 implementation, with configurations (i)–(vi) directly comparable to one another. Image observations and higher action and time limits improve reported RHAE scores, while later steps add conversation continuity, notes, and VISTA’s visual tools.

Ablation setup

  • The ablations incrementally modify the official implementation; configurations (i)–(vi) share the same model access interface and are directly comparable, while the official result is an external reference.The experiments use the Codex CLI subscription interface rather than the official implementation’s API access.Source: §A.2 · p. 15

(i): Image observations

  • 47.32 RHAE follows replacing textual grid observations with rendered images, which require approximately 308 image tokens versus roughly 4K text tokens for an equivalent 64 × 64 grid.The authors hypothesize that improved token efficiency contributes to the gain; seven frames use roughly 2.2K image tokens versus roughly 28K text tokens.Source: §A.2 · p. 15

(i)→(ii): Increased action and time limits

  • 51.66 RHAE is reached after increasing the action limit to 2,000 total actions and the runtime limit to 48 hours per game; further increases bring no substantial additional benefit.Many runs under the official limits terminated unfinished, and the change improved the score from 47.32.Source: §A.2 · p. 15

(ii)→(iii): Continuous conversation

  • Continuous conversation allows multiple play calls in one conversation and uses native context compaction to keep a compressed representation of earlier interactions available.The official implementation produces at most one action per request and truncates the oldest interactions when the context limit is exceeded.Source: §A.2 · p. 15

(iii)→(iv): Add notes

  • Two notes provide persistent game knowledge and current-level state or plans, with a handoff summary written before the agent resumes in a fresh context.GUIDE.md stores reusable information across levels and sessions; WORKING.md stores temporary state and plans for the current level.Source: §A.2 · p. 16

(iv)→(vi): Add VISTA’s visual harness

  • VISTA’s visual harness adds lossless visual memory, model-directed inspection, and exact pixel readout, with the agent receiving only the final frame after each action and choosing when to revisit memory.These additions complete the harness.Source: §A.2 · p. 16

A.3 Evaluation Metrics

The appendix defines benchmark-specific scoring and describes the tools and prompts used to guide agents through interactive and static visual tasks.

A.3 Evaluation Metrics

  • RHAE compares action efficiency with first-time human baselines, gives greater weight to later levels, and assigns zero credit to unfinished levels.Game scores combine weighted level scores with a cap based on the weighted fraction of levels completed; total actions are also reported.Source: §A.3 · p. 16
  • GameWorld reports task success rate and progress, with progress crediting partial completion across the 170 tasks.The benchmark’s official evaluator derives both metrics from serialized game-state fields.Source: §A.3 · p. 16
  • AI GameStore scores normalize each game’s raw score against its published human median and aggregate normalized scores across ten games using a geometric mean.Source: §A.3 · p. 16

A.4 Agent Tools

  • A host-side controller manages tool calls, archived frames, and agent notes, while enforcing sequential game actions and resuming after context compaction.Only play changes the environment; note-writing tools update stored notes, and the other tools are read-only.Source: §A.4 · p. 17

A.5 Agent Prompts

  • Prompts ask agents to maintain revisable accounts of the environment, predict and review action outcomes, and preserve durable knowledge in GUIDE.md and WORKING.md.For static BabyVision tasks, prompts omit persistent notes and direct agents to inspect enlarged regions or read pixel values when needed.Source: §A.5 · p. 18

A.6 Human References

The appendix describes how human reference performances were obtained and reports run-to-run variation and the scope of per-game ARC-AGI-3 results.

A.6 Human References

  • ARC-AGI-3’s RHAE human baseline uses the upper-median action count from first-time players who completed each level, with ten first-time participants testing each game.The broader study involved 486 public participants; games were included only when at least two participants independently completed all levels.Source: §A.6 · p. 19

B.1 Run-to-run Variation

  • 99.04 ± 0.17 is GPT-5.6 Sol’s mean ARC-AGI-3 RHAE across three runs; corresponding reported means are 63.33 ± 2.37% success rate and 79.30 ± 2.14% progress on GameWorld, and 63.23 ± 3.00% on BabyVision.Source: §B.1 · p. 19

B.2 Per-game Results on ARC-AGI-3

  • Table 2 presents per-game results for Claude Opus 5.0 and GPT-5.6 Sol alongside the first-time human action reference.Source: §B.2 · p. 19

B.3 Interaction trajectories

The trajectories illustrate VISTA’s interaction across ARC-AGI-3 and other visual tasks, including retrieving earlier observations and inspecting image regions.

B.3 Interaction trajectories

  • In BP35 level 1, the player navigates under upward gravity by moving and clicking green blocks to clear obstacles and reach the pink goal.The trajectory shows one frame per action, with rings marking clicks.Source: Figure 12 · p. 21
  • BP35 level 3 adds a toggleable orange block that can open a path or prevent the player from rising into purple spikes.Source: Figure 13 · p. 22
  • In Minecraft Clone, the agent inspects the hotbar and ore vein to distinguish iron from stone, then completes the mining task in 21 actions.Source: §B.3 · p. 23
  • In Pac-Man, the agent revisits the initial maze to recover corridors obscured as dots disappear, and solves the 1,500-point task in 42 actions.Source: §B.3 · p. 23
  • The additional examples show VISTA magnifying regions or reading pixels in Connect the Lines and Maze to inspect visual details and trace paths.Figure 14 also shows these inspection patterns alongside the GameWorld trajectories.Source: Figure 14 · p. 23

Related papers

Loading 2610.02200v1…