Source-linked AI summary
VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents
Zirui Wang, Junyi Zhang, Jiaxin Ge, Long Lian, Letian Fu, Lisa Dunlap, Ken Goldberg, XuDong Wang, Ion Stoica, David M. Chan, Sewon Min, Joseph E. Gonzalez
TL;DR
Multi-step visual decision-making remains poorly understood in VLMs, especially how they integrate perception, memory, and action over long horizons. VisGym addresses this gap with 17 customizable environments and finds that frontier models achieve only 46.61% success in easy settings and 26.00% in hard settings.
Problem
VLM behavior remains poorly understood when visual observations must be integrated into multi-step decision-making.
Method
VisGym provides 17 customizable long-horizon environments spanning symbolic puzzles, image understanding, navigation, and manipulation for systematic VLM evaluation and training.
Results
46.61% and 26.00% success were achieved in easy and hard settings, respectively, while analyses identified cross-domain failures in context use and visual grounding.
Takeaways & Limitations
VisGym offers a unified framework for diagnosing and improving VLMs in visually interactive decision-making.
Abstract
from arXiv · showhide
Modern Vision-Language Models (VLMs) remain poorly characterized in multi-step visual interactions, particularly in how they integrate perception, memory, and action over long horizons. We introduce VisGym, a gymnasium of 17 environments for evaluating and training VLMs. The suite spans symbolic puzzles, real-image understanding, navigation, and manipulation, and provides flexible controls over difficulty, input representation, planning horizon, and feedback. We also provide multi-step solvers that generate structured demonstrations, enabling supervised finetuning. Our evaluations show that all frontier models struggle in interactive settings, achieving low success rates in both the easy (46.6%) and hard (26.0%) configurations. Our experiments reveal notable limitations: models struggle to effectively leverage long context, performing worse with an unbounded history than with truncated windows. Furthermore, we find that several text-based symbolic tasks become substantially harder once rendered visually. However, explicit goal observations, textual feedback, and exploratory demonstrations in partially observable or unknown-dynamics settings for supervised finetuning yield consistent gains, highlighting concrete failure modes and pathways for improving multi-step visual decision-making. Code, data, and models can be found at: https://visgym.github.io/.
1. Introduction
VisGym is introduced as a diverse, customizable suite of 17 long-horizon environments for systematically diagnosing visual interactive decision-making in VLMs. Across 12 state-of-the-art models, results expose low success rates and cross-domain failures involving context, perception, feedback, goals, and demonstrations.
- VisGym: VisGym provides 17 customizable environments spanning symbolic puzzles, real-image understanding, navigation, and manipulation, with controlled representation, difficulty, history, horizon, and feedback.The suite includes distinct observability and dynamics and oracle multi-step solvers for supervised finetuning.
- Results: 46.61% and 26.00% success were achieved by the strongest models across 12 models in the easy and hard settings, respectively.These results demonstrate substantial difficulty in multi-step visual decision-making even for state-of-the-art systems.
- Failure modes: Models struggle to use long-term context, exhibit degraded performance with unbounded histories, and find visually rendered symbolic tasks harder than their text-based counterparts.The analyses identify a reversed-U relationship between context length and performance, alongside low-level perceptual grounding limitations.
- Failure modes: Explicit textual feedback consistently improves performance, whereas explicit goal observations can help substantially but may backfire when models misidentify visually presented goals.Limited visual perception can make explicit goals worse than providing no goal at all.
- Failure modes: Under partial observability or unknown dynamics, standard demonstrations are insufficient, while information-revealing demonstrations significantly improve supervised finetuning outcomes.Such demonstrations expose hidden states or clarify task dynamics.
2. VisGym
VisGym provides 17 customizable visually interactive environments built on Gymnasium for evaluating and training multimodal agents. Its interface combines function-based actions, natural-language instructions, textual feedback, and stochastic multi-step solvers that generate demonstrations for supervised fine-tuning.
- Environment Suite: VisGym contains 17 visually interactive environments with initialization parameters controlling task configuration and difficulty.The suite extends the Gymnasium framework, which also underlies MuJoCo and Atari.
- Function-Conditioned Action Space: Actions are represented as parameterized function calls, allowing models to use function-calling capabilities and compose strategies across domains.This replaces the discrete or continuous action vectors used in standard Gymnasium environments.
- Function Instructions: Each task provides natural-language descriptions of its available functions and argument constraints before zero-shot rollouts begin.These instructions define the task-specific function set and parameter spaces.
- Environment Feedback: Textual feedback describes each action’s effect alongside visual transitions, helping models with weaker visual perception ground their actions.Feedback examples include “invalid format,” “out of bounds,” and “executed.”
- Solver: Heuristic multi-step solvers support multiple strategies and optional stochasticity to generate diverse demonstration trajectories for supervised fine-tuning.The unified step function handles parsing, validation, execution, and feedback, while the modular interface supports new tasks and varied action spaces.
3. Evaluating Frontier Models with VisGym
VisGym exposes substantial weaknesses in frontier VLMs during multi-step visual interaction: even the strongest model reaches only 46.61% success on Easy and 26.00% on Hard. The evaluation further reveals specialization across models, limited long-horizon handling, and recurring failures in action control, state management, termination, and visual reasoning.
- 3.1. Evaluation Setup: The evaluation covers 12 proprietary, open-weight, and GUI/game-specialized VLMs using a consistent prompting and inference pipeline.Models are evaluated on 70 episodes per task and setting, with easy and hard interaction limits of up to 20 or 30 steps.
- 3.2. Result and Analysis: 46.61% on VisGym (Easy) and 26.00% on VisGym (Hard) are achieved by the best-performing frontier model, Gemini-3-Pro, demonstrating the suite’s difficulty.Success rates are averaged across VisGym tasks and reported as the best frontier-model performance.
- 3.2. Result and Analysis: GPT-5 handles long-context interactions best, Gemini 2.5 Pro excels at low-level spatial perception, and Qwen3-VL-235B-Instruct excels at object localization.These specializations appear in matchstick rotation and hard-setting performance, spatially demanding puzzles, and Referring Dot-Pointing, respectively.
- 3.2. Result and Analysis: Most models peak at only 3–5 successful steps before sharply declining, indicating limited ability to sustain long-context multi-step visual interactions.The pattern is reported for Gemini 2.5 Pro, Claude Sonnet 4, and Llama-4-Maverick.
- 3.2. Result and Analysis: Recurring failures include restricted action looping, state mismanagement, premature termination, and failure to use visual or spatial information.Models repeat fixed actions, ignore feedback or prior errors, stop before reaching goals, and overlook visual cues such as alignment or targets leaving the frame.
4. Diagnosing Frontier Models with VisGym
VisGym diagnoses how interaction-history length, observation modality, feedback, and goal visibility alter frontier-model performance. The results expose task-dependent context effects, visual-grounding bottlenecks, reliance on textual feedback, and perception errors that can undermine explicit goals.
- 4.1. Turns to Keep in Conversation History: Limited history helps most, with performance improving through roughly four previous turns before unbounded history causes declines from stale or irrelevant observations.History effects are task- and model-dependent: Gemini 2.5 Pro scales well in Maze2D but reverse-scales in Sliding Block, while GPT-5 scales well in Matchstick Rotation.
- 4.2. Representing Observation in Text: GPT-5 often achieves 3–4× higher success with ASCII observations than images, while open-weight models struggle in both modalities and Matchstick Equation reverses this trend.Gemini 2.5 Pro shows mixed modality effects, and distorted figlet-style ASCII likely explains Matchstick Equation’s visual advantage.
- 4.3. Removal of Text-based Feedback: Removing text-based environment feedback consistently lowers all models’ average performance, showing that they struggle to infer action validity from visual transitions alone.The feedback channel includes execution information such as formatting errors and constraint violations, including walls or occupied cells.
- 4.4. Providing Final Goal at Beginning: Providing the final goal observation substantially improves performance across tasks, but perception errors can invert this benefit when models falsely judge misaligned images as identical.Gemini 2.5 Pro made identical-image errors in 80% and 57% of Zoom-In Puzzle and Matchstick Equation pairs, versus 18%, 2%, and 0% for Colorization, Jigsaw, and Patch Reassembly.
5. Training with VisGym
Solver-generated demonstrations enable strong supervised fine-tuning on most VisGym tasks, while generalization, module contributions, and demonstration curation reveal distinct requirements for multi-step visual decision-making. Stronger base models generalize better to unseen difficulty, and information-revealing trajectories substantially improve learning in unknown-dynamics and partially observable settings.
- 5.1. Supervised Fine-Tuning Experiments: Fine-tuned models achieve state-of-the-art performance on most tasks, validating VisGym’s learnability and the effectiveness of structured solver-generated demonstrations.Demonstrations are filtered to completed trajectories without initial-state overlap with the test split.
- 5.2. Stronger Base Model Generalizes Better: Qwen3-VL nearly doubles Qwen2.5-VL’s average success rate on unseen hard variants despite comparable easy-setting performance after identical mixed-task fine-tuning.Easy-setting examples include 0.59 vs. 0.64, while hard variants test difficulty generalization because only easy demonstrations are used for training.
- 5.3. Vision and LLM Both Matter: Fine-tuning both vision and LLM components benefits most tasks, but LLM gains are larger, especially under partial observability or unknown dynamics.Vision fine-tuning is particularly important for the Zoom-In Puzzle, yet fine-grained visual encoding alone is often insufficient for multi-step decisions.
- 5.4. Importance of Information-Revealing Behaviors for SFT Curation: Information-revealing demonstrations raise Matchstick Rotation success from 32.9% to 70.0% by exposing action effects before the final aligning move.The strategy uses two unit-scale exploratory steps instead of three stochastic moves toward the target.
- 5.4. Importance of Information-Revealing Behaviors for SFT Curation: In Mental Rotation, fully exposing each axis’s 3D geometry improves both metrics, while subsequent training on baseline demonstrations deteriorates performance.This controls for trajectory quantity and length, indicating that gains arise from state-disambiguating structure rather than longer demonstrations.
6. Related Work
Prior benchmarks progressed from fully observable visual control to multimodal, multi-step reasoning and embodied interaction. VisGym unifies these directions in a 17-task gym with solvability guarantees and targeted diagnostics for analyzing VLM and VLA failures.
- Control and robotics benchmarks: Modern robotics suites extend early visual-control benchmarks toward partially observable, multi-task, and long-horizon manipulation and navigation across diverse embodiments.Early platforms such as Atari, OpenAI Gym, and DeepMind Lab primarily targeted low-level motor control in fully observable settings.
- VLM benchmarks: VLM benchmarks evolved from visual question answering to text-mediated action, multi-step planning, and tool use across embodied, graphical, and visual-reasoning tasks.Examples include VLM-Gym, VisualAgentBench, EmbodiedBench, and WebArena.
- VisGym: VisGym combines reasoning and control in an RL-style gym spanning 17 multimodal tasks across visual puzzles, spatial reasoning, manipulation, and grounding.Each environment includes an oracle solution supporting solvability and synthetic trajectory generation for post-training.
- VisGym: VisGym adds controllable difficulty and diagnostics for history utilization, representation, feedback specificity, and perception–action causality to identify why models fail.These designs support systematic analysis of VLMs and VLAs across domains and interaction levels.
7. Conclusion … E. Configuration of Environments
VisGym is a unified suite of 17 visually interactive environments for training and evaluating multimodal agents, supported by structured solvers, episode specifications, multiple observation formats, and configurable interfaces and difficulty. The supplementary sections detail solver behaviors, episode progressions, text renderings, interaction semantics, and task configurations.
- 7. Conclusion: VisGym provides 17 visually interactive environments that challenge and train vision–language models in multi-step visual decision-making.The suite is presented as a unified playground bridging perception and reasoning toward more capable, adaptive visual intelligence.
- A. Solver Design: The multi-step solvers use task-specific strategies, including shortest-path search, greedy control, backtracking, permutation actions, and reversible padding to generate demonstrations.Examples include BFS or DFS for matchstick equations, graph search for mazes, greedy per-axis control for Fetch tasks, and reversible detours when longer trajectories are requested.
- B. Environment Episode Progression: The appendix organizes detailed episode progressions for all 17 environments and provides a page-indexed summary of their descriptions.The listed environments span colorization, counting, puzzles, mazes, mental rotation, robotic manipulation, patch reassembly, pointing, sliding blocks, video unshuffle, and zoom-in tasks.
- C. ASCII-based Observation Visualization: Four environments provide example episode variants rendered in text, with instructions adapted to preserve task meaning in ASCII-based observations.The examples cover Sliding Block, Maze 2D, Patch Reassembly, and Matchstick Equation; Patch Reassembly changes how anchor cells are identified in text.
- D. VisGym Interface: VisGym’s generic step function parses action names and payloads, rejects invalid formats or actions with feedback, and returns observations, rewards, termination, truncation, and outcome feedback.Non-terminal transitions receive zero reward, while termination computes the final reward from the environment state.
- E. Configuration of Environments: Environment configurations expose tunable task parameters that define easy and hard settings, with difficulty increasing observation complexity in some tasks and reward precision requirements in others.Table 3 summarizes the tunable parameters, easy and hard configurations, and source datasets for all tasks.
F. Analyzing Model Failures
VisGym’s trace analysis uses GPT-4.1-based behavior clustering and annotation to identify four recurring failure modes across tasks. Action looping dominates failures, while task difficulty, visual cues, model weakness, and anthropomorphic behavior shape their prevalence and expression.
- GPT-4.1-based StringSight extracts trace behaviors, clusters them into higher-level patterns, and annotates each trace for the four failure modes.The four labels are restricted action space/action looping, state mismanagement, early termination, and failure to use visual or spatial information.
- The four recurring failures are action repetition, failure to update state from feedback, premature stopping before the goal, and ignoring visual or spatial cues.Examples include repeating invalid moves, continuing into a wall after collision feedback, terminating before maximum steps, and ignoring changed or aligned objects.
- Action looping occurs in more than 60% of traces, making it the most common failure, followed by early termination, state mismanagement, and visual-information neglect.Figure 14 summarizes failure frequencies; traces may exhibit multiple behaviors.
- Matchstick Equation and Sliding Block show particularly high action repetition and state mismanagement, while Maze reaches up to 70% visual-information neglect and high action repetition.The task patterns are attributed to task difficulty, frequent invalid moves, and the availability of clear visual progress signals.
- UI TARS 1.5 7B exhibits 87% action looping and 35% state mismanagement, supporting the pattern that uncertainty prompts repeated moves despite feedback.The model-specific comparison appears in Figure 15(a).
- Early termination includes explicit giving-up statements and occurs much more often on hard Matchstick Equation tasks, especially among Gemini and Gemma models.The paper attributes the task pattern to limited comprehension and suspects the model pattern reflects greater chattiness and anthropomorphism.
F.1. Failure changes per ablation
Failure patterns vary with information settings: more history reduces immediate action repetition but worsens state management, while feedback, initial state information, and textual representations alter distinct error behaviors. Visual inputs are generally processed less effectively than text, except for Matchstick Equation tasks.
- Different amounts of chat history: More history reduces immediate action looping but eventually worsens state management, consistent with poorer performance under full history.With no history, models repeat default actions; with extensive context, they struggle to manage earlier state and revert to default behavior.
- Feedback vs. no feedback: Without feedback, models terminate less often and show less giving up, action looping, and state mismanagement despite lower overall performance.The reduced giving up largely reflects fewer opportunities to respond to invalid-move feedback, suggesting additional failure modes outside the taxonomy.
- Ground truth state given at the beginning: Providing ground-truth state initially reduces guessing and early giving up, yielding lower action looping and early termination.
- Image vs. text representation: Except for Matchstick Equation, models process visual information more effectively as text than as images, with text sharply reducing action looping.The text representation appears to provide clearer guidance for selecting actions.
F.2. Failure Trajectories Visualization
StringSight visualizes failure trajectories by pairing prompts, images, raw outputs, parsed actions, and classified reasons with supporting evidence. The examples expose action looping, state mismanagement, premature termination, and failure to use visual or spatial information.
- Visualization: StringSight visualizes each trajectory with the prompt, image, raw model output, parsed action, and labeled reason and evidence for its failure classification.“Reason” explains the assigned failure category, while “Evidence” identifies trajectory evidence supporting that classification.
- Restricted action space and action looping: GPT-5 repeatedly swaps the same Jigsaw positions, producing action looping without making progress.The repeated action is (“swap”, (0, 0), (0, 1)).
- State mismanagement: Claude Sonnet 4 repeats a blocked Maze 2D move after reaching the identical state, disregarding earlier environment feedback.The model repeats (“move”, 2) despite previously receiving “Cannot move into a wall.”
- Early termination: Gemma 3 27B Instruct terminates Matchstick Equation at step 13 despite being allowed 30 total steps.This trajectory illustrates early termination before the available action horizon is exhausted.
- Failure to use visual or spatial information: Gemini 2.5 Pro continues rotating Mental Rotation 3D (Cube) in the wrong direction after visual information shows it is moving farther from the target.The model ignores visual information during the final three steps.
G. Additional Performance Analysis · Sample trajectory for “Action Looping" (GPT-5)
Additional analyses show that frontier models perform best on simple symbolic tasks and struggle with 3D spatial, long-horizon, and memory-intensive tasks. Performance also degrades with difficulty, while a GPT-5 trajectory illustrates action looping through repeated identical commands.
- G. Additional Performance Analysis: Tasks requiring memory, long-horizon planning, or strong 3D spatial understanding remain the most difficult for current models.
- G. Additional Performance Analysis: Over 20% average accuracy occurs on Referring Dot-Pointing and Counting, whereas Mental Rotation 3D, Patch Reassembly, and Mental Rotation 3D (Objaverse) remain near 1–2%.Sliding Block, Maze 3D, Fetch Pick-Place, and Video Unshuffle also remain below 5% on average.
- G. Additional Performance Analysis: All models tend to reach the 20-step Easy or 30-step Hard limit, with a U-shaped trajectory pattern in which runs terminate early or continue to the maximum.Figure 18 distinguishes correct trajectories in green from incorrect trajectories in red.
- G. Additional Performance Analysis: Counting and Jigsaw show the largest Easy-to-Hard drops, with Jigsaw performance approaching zero when scaling from 2x2 to 3x3.Patch Reassembly, Sliding Block, and Video Unshuffle have smaller absolute gaps because Easy performance is already approximately zero.
- G. Additional Performance Analysis: Task-specific rankings reveal strengths such as Qwen2.5 VL 72B Instruct on Counting (rank 2) and Gemma 3 27B Instruct on Maze 3D (rank 3), alongside model-specific weaknesses.GPT-5 performs poorly on Video Unshuffle and Maze 3D, while Gemini 2.5 Pro performs poorly on Jigsaw (Hard) and Patch Reassembly.
- G. Additional Performance Analysis: Patch Reassembly, Sliding Block, and Video Unshuffle can be scaled by increasing patches, blocks, or frames as models improve on easier settings.The analysis expects larger Easy-to-Hard gaps once models reliably solve the Easy configurations.
Sample trajectory for “State Mismanagement" (Claude Sonnet 4)
In this 9x9 maze trajectory, Claude Sonnet 4 repeatedly revisits directions already shown to be blocked, exhausting its step budget without reaching the red target. The example illustrates state mismanagement through failure to retain impossible moves and seek new routes.
- After receiving repeated “Cannot move into a wall” feedback, the model continues retrying blocked directions instead of seeking new routes.Blocked left and up moves are explicitly identified as previously impossible, yet the trajectory revisits them multiple times.
- The trajectory shows the model recognizing individual wall collisions but failing to preserve that state across subsequent actions.It correctly acknowledges that a wall blocks movement left, then later attempts left again from positions where the trajectory records additional wall feedback.
- Claude Sonnet 4 exhausts all available steps without reaching the red target square.The model reports having one step remaining before ultimately stating that all available steps were used unsuccessfully.
Sample trajectory for “Early Termination" (Gemma3 27B)
The trajectory shows Gemma3 27B repeatedly producing illegal matchstick moves, failing to use undo, and ultimately giving up after twelve illegal moves. Its reasoning increasingly attributes failure to visual misjudgment or broken validation before submitting a final nonsense move.
- Task specification: The task requires one-match edits that produce valid symbols and a mathematically correct final equation, with exactly one formatted action per step.Available actions are move, undo, and stop, and the model is allowed 30 total steps.
- Reasoning failure: The model repeatedly misjudges visually valid transformations, proposing targets such as 1 + 1 = 2 and 0 + 0 = 0 after illegal moves.Its own explanations repeatedly identify difficulty visualizing valid moves.
- Interaction failures: An attempted undo fails because no move history exists, and later attempts to move matches from 0 to 4 and at random are also illegal.The environment reports “Nothing to undo” and illegal moves for both targeted and arbitrary actions.
- Failure outcome: After twelve illegal moves, Gemma3 27B explicitly gives up and submits a nonsense move as final.The model states, “I give up,” and suspects the segment definitions or move validation are broken.
Sample trajectory for “Visual/Spatial Perception" (Gemini 2.5 Pro) … Zoom-In Puzzle
This section presents sample interactive trajectories across visual perception, navigation, spatial reasoning, manipulation, symbolic puzzles, referring, and temporal or zoom-based visual ordering. The examples illustrate sequential decision-making through visual diagnosis, goal-directed actions, and environment-specific success conditions.
- Sample trajectory for “Visual/Spatial Perception" (Gemini 2.5 Pro): The Gemini 2.5 Pro trajectory repeatedly miscorrects the 3D object’s yaw, pitch, and roll while trying to match the target orientation.The model identifies an upside-down S/Z discrepancy, restores the Z configuration with a 180-degree roll, then continues making yaw and pitch corrections; later, the object becomes heavily distorted.
- Colorization: Colorization requires adjusting hue and saturation on a color wheel until the selected color matches the target region.The agent can rotate the wheel, change saturation, and stop when the color selection closely matches the target.
- Counting: Counting requires marking horse instances with coordinate-based dots before recording a final integer count.The final guess succeeds only when it matches the true number of objects.
- Jigsaw: The Jigsaw task asks the agent to reconstruct a scrambled 3x3 image by swapping pieces or reordering all nine positions.Piece coordinates are zero-based, with index = row * 3 + col.
- Matchstick Equation: The Matchstick Equation task requires moving one match per action so the resulting symbols form a mathematically correct equation.Each move removes a segment from one symbol and adds it to another while requiring valid glyphs at both positions; undo is available.
- Matchstick Rotation: Matchstick Rotation asks the agent to move and rotate a blue stick to align with a red target despite an unknown unit-move scale.The scale may be tiny or huge, making the action magnitude unspecified.
- Maze 2D: Maze 2D requires navigating an 11x11 grid of gray walls and white paths from a blue circle to a red target square.The environment provides action-execution feedback during the navigation sequence.
- Maze 2D / Maze 3D: The sample trajectories cover 2D and 3D maze navigation, with the 3D task requiring movement and viewpoint turns to reach a red-sphere target.Moving into a wall leaves the agent in place, while turning changes only its facing direction; the episode allows up to 100 steps.
- Mental Rotation 2D / Mental Rotation 3D (Cube): Mental-rotation tasks require undoing unknown image rotations: 2D images must be aligned to the original within ±5.0°, while 3D objects must be returned to their target orientation using Euler-angle rotations.
- MuJoCo Fetch (Pick-and-Place) / MuJoCo Fetch (Reach): In MuJoCo Fetch, the agent moves a robot end-effector, controls a sticky gripper, and must pick up a grey cube and place it at a red target position.The supplied Fetch Reach passages contain only intermediate successful-action feedback and no distinct task objective.
- Patch Reassembly: Patch Reassembly asks the agent to place six irregular pieces on an 8×8 board so the grid is completely filled without gaps or overlaps.The action space supports placing and removing patches before submitting the arrangement.
- Referring Dot-Pointing: The referring task requires marking the normalized image coordinates of a plant with a brown pot before submitting the point.Coordinates map to pixels using x = round(x_norm*(500-1)) and y = round(y_norm*(375-1)).
- Sliding Block: Sliding Block presents target and current configurations of a 5×4 Klotski puzzle, requiring directional block moves until every block reaches its target position.Blocks are identified by numbers 1–10 and can move up, right, down, or left.
- Video Unshuffle: Video Unshuffle asks the agent to reorder five shuffled frames chronologically while tracking the described action of pushing a chair with a hand.The agent may swap frames or provide the complete ordering in one action.
- Zoom-In Puzzle: Zoom-In Puzzle asks the agent to arrange an original image and five zoomed views from least to most zoomed.The ordering can be achieved through pairwise swaps or a complete reorder action.