PaperlayerView exact revision on arXiv

Source-linked AI summary

VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents

Zirui Wang, Junyi Zhang, Jiaxin Ge, Long Lian, Letian Fu, Lisa Dunlap, Ken Goldberg, XuDong Wang, Ion Stoica, David M. Chan, Sewon Min, Joseph E. Gonzalez

arXiv:2601.16973v1cs.CV

TL;DR

VisGym addresses the limited understanding of how VLMs integrate perception, memory, and action during long-horizon visual interactions. It introduces 17 customizable environments and solver-generated demonstrations for controlled evaluation and training, finding that frontier models achieve only 46.61% success in Easy and 26.00% in Hard settings. The analyses identify failures in long-context use, visual grounding, and state inference, while textual feedback, explicit goals, and information-revealing demonstrations can improve outcomes.

  • Problem

    VLM behavior remains poorly understood when visual observations must be integrated into multi-step decision-making over long horizons.

  • Method

    VisGym provides 17 customizable long-horizon environments spanning multiple domains, with controlled settings and oracle multi-step solvers for supervised finetuning.

  • Results

    Frontier models achieve only 46.61% success in Easy and 26.00% in Hard settings, with recurring failures in long-context use, visual grounding, and state inference.

  • Takeaways & Limitations

    Textual feedback, explicit goal observations, and information-revealing demonstrations provide supported pathways for improving multi-step visual decision-making.

Abstract

from arXiv · show

Modern Vision-Language Models (VLMs) remain poorly characterized in multi-step visual interactions, particularly in how they integrate perception, memory, and action over long horizons. We introduce VisGym, a gymnasium of 17 environments for evaluating and training VLMs. The suite spans symbolic puzzles, real-image understanding, navigation, and manipulation, and provides flexible controls over difficulty, input representation, planning horizon, and feedback. We also provide multi-step solvers that generate structured demonstrations, enabling supervised finetuning. Our evaluations show that all frontier models struggle in interactive settings, achieving low success rates in both the easy (46.6%) and hard (26.0%) configurations. Our experiments reveal notable limitations: models struggle to effectively leverage long context, performing worse with an unbounded history than with truncated windows. Furthermore, we find that several text-based symbolic tasks become substantially harder once rendered visually. However, explicit goal observations, textual feedback, and exploratory demonstrations in partially observable or unknown-dynamics settings for supervised finetuning yield consistent gains, highlighting concrete failure modes and pathways for improving multi-step visual decision-making. Code, data, and models can be found at: https://visgym.github.io/.

1. Introduction

VisGym is introduced to study multi-step visual decision-making across diverse environments and expose where current VLMs break down. Evaluations reveal low success rates and recurring failures involving long-context use, visual grounding, and information-revealing feedback.

  • Motivation and contribution: VisGym provides 17 customizable environments spanning symbolic puzzles, real-image understanding, navigation, and manipulation.The suite varies observability and dynamics while supporting controlled analysis of interactive decision-making.
  • Evaluation findings: 46.61% and 26.00% are the strongest models’ success rates in the easy and hard settings, respectively.The evaluation covers 12 state-of-the-art models.
  • Evaluation findings: Unbounded long-term context can reduce performance, producing a reversed-U relationship as context grows.This indicates difficulty leveraging extended interaction histories effectively.
  • Evaluation findings: Symbolic task variants are substantially easier than visually rendered counterparts, highlighting limitations in low-level perceptual grounding.Models also struggle to infer states and outcomes from purely visual transitions.
  • Evaluation findings: Explicit textual feedback improves performance, while explicit goal observations can backfire when models misidentify them.Information-revealing demonstrations significantly improve supervised finetuning under partial observability or unknown dynamics.
  • Implications: VisGym serves as a unified framework for diagnosing and improving VLMs in visually interactive decision-making.The framework is intended to support understanding and training of multimodal agents.

2. VisGym

VisGym combines a modular environment interface with function-based actions, natural-language instructions, visual transitions, textual feedback, and solver-generated demonstrations. These design choices support customizable task configuration and multimodal supervision for VLM agents.

  • Environment design: Each VisGym environment exposes initialization parameters controlling task configuration and difficulty.The suite is built on the Gymnasium framework and supports varied environment definitions.
  • Environment design: Actions are represented as parameterized function calls rather than standard discrete or continuous action vectors.This lets models compose strategies across domains using function-calling capabilities.
  • Environment design: Natural-language function descriptions and argument constraints are included in the initial prompt for zero-shot rollouts.Each task defines its own function set and parameter spaces.
  • Environment design: Textual feedback describes the effects of actions alongside visual transitions.Examples include invalid format, out of bounds, and executed.
  • Solver and supervision: Heuristic multi-step solvers generate diverse demonstration trajectories for supervised fine-tuning.Multiple solving strategies and optional stochasticity increase trajectory diversity.
  • Solver and supervision: A unified step function parses, validates, executes, and provides feedback for task-specific actions and instructions.The modular structure supports adding tasks, varying action spaces, and generating visual and textual supervision.

3. Evaluating Frontier Models with VisGym

VisGym evaluation shows that frontier VLMs perform poorly on long-horizon interactive tasks and exhibit recurring failures in action selection, state management, and visual reasoning.

  • Evaluation and Long-Horizon Behavior: Models benefit from multi-turn evaluation but remain limited in handling long-context visual interactions.Successful trajectories for most models peak around 3–5 steps before dropping sharply at longer horizons.
  • Overall Performance: 46.61% and 26.00% are the best frontier-model success rates in VisGym’s easy and hard settings, respectively.Gemini-3-Pro is the best-performing frontier model, yet VisGym remains challenging.
  • Common Failure Patterns: Models repeatedly restrict their action space by looping fixed operations or repeating fixed-magnitude actions instead of adapting actions to task state.Examples include repeated swaps, directional movements, or rotations with unchanged magnitudes.
  • Common Failure Patterns: Models mismanage state by failing to update internal representations, revisit explored areas, or repeat illegal actions after receiving feedback.These failures include continuing into walls and repeating invalid moves.
  • Common Failure Patterns: Early termination and failure to use visual or spatial information are additional recurring errors across tasks.Models may stop before reaching the goal or ignore cues such as target displacement and successful alignment.

4. Diagnosing Frontier Models with VisGym

Controlled diagnoses show that interaction design strongly affects frontier-model performance: limited history helps, visual representations can hinder reasoning, and explicit feedback or goals expose perception bottlenecks.

  • Turns to Keep in Conversation History: Limited history improves performance up to roughly four previous turns, whereas full unbounded history causes a subsequent decline.Irrelevant or stale observations can outweigh the useful information provided by longer histories.
  • Representing Observation in Text: GPT-5 often achieves 3–4× higher success with ASCII representations than with visual observations, indicating a visual-grounding bottleneck on several symbolic tasks.Gemini 2.5 Pro shows mixed effects, while open-weight models struggle in both modalities.
  • Environment Feedback: Removing textual environment feedback consistently reduces average performance because models struggle to infer action validity from visual transitions.Feedback describes execution effects and constraint violations such as invalid formats or collisions.
  • Explicit Goal Observations: Providing the final goal observation can substantially improve performance, but limited visual perception can cause models to misidentify the goal and perform worse.Gemini 2.5 Pro incorrectly judged initial and goal images identical in 80% and 57% of tested cases for two tasks.

5. Training with VisGym

VisGym supports solver-generated supervised finetuning, which improves task performance and reveals that generalization, module contributions, and demonstration structure vary across environments.

  • Supervised Finetuning: Solver-generated demonstrations enable finetuned models to achieve state-of-the-art performance on most tasks.The results validate both the learnability of VisGym environments and the effectiveness of structured multi-step demonstrations.
  • Difficulty Generalization: Qwen3-VL nearly doubles Qwen2.5-VL’s average success rate on harder variants despite identical mixed-task finetuning setups.Both models achieve comparable performance on easy variants, including 0.59 vs. 0.64.
  • Module Specificity: Most tasks benefit from finetuning both the vision encoder and LLM, with the LLM generally contributing the larger gain.The LLM advantage is especially pronounced under partial observability or unknown dynamics, while Zoom-In Puzzle primarily benefits from vision finetuning.
  • Data Curation: Information-revealing demonstrations raise Matchstick Rotation success from 32.9% to 70.0% by exposing action-magnitude effects before the final alignment move.The demonstrations use two unit-scale exploratory steps followed by the aligning move.
  • Data Curation: Continued training on baseline demonstrations deteriorates performance after information-revealing finetuning, indicating that demonstration structure matters beyond trajectory quantity or length.The informative demonstrations expose hidden states or dynamics needed for accurate state representations.

6. Related Work

VisGym builds on benchmarks for visual control and VLM reasoning by unifying multimodal reasoning and control in a customizable gym with targeted diagnostics.

  • Visual Control Benchmarks: Earlier visual-control benchmarks established evaluation platforms for vision-based decision-making but focused primarily on low-level motor control.Later work expanded toward manipulation, navigation, partial observability, and multitask settings.
  • VLM Benchmarks: VLM benchmarks progressed from visual question answering toward text-mediated actions and studies of perception, reasoning, and interaction.This progression broadened evaluation beyond static visual understanding.
  • VisGym: VisGym combines 17 multimodal tasks with oracle solutions, controllable difficulty, and diagnostics for history use, representation, feedback, and perception–action causality.These features support both synthetic trajectory generation and analysis of why models fail.
  • VisGym: VisGym extends the RL-style gym paradigm across visual puzzles, spatial reasoning, manipulation, and grounding under varied interaction conditions.The suite is designed to support systematic analysis across domains and levels of interactivity.

7. Conclusion

VisGym provides a unified suite of 17 visually interactive environments with solvers that generate structured multi-step demonstrations for training multimodal agents.

  • VisGym environments: 17 environments span visual puzzles, navigation, manipulation, and image-based tasks with configurable episode progressions.The suite includes tasks such as Video Unshuffle, Colorization, Counting, Jigsaw, Matchstick Equation and Rotation, mazes, mental rotation, and MuJoCo interaction.
  • Trajectory construction: Several solvers pad optimal or corrective trajectories with reversible or exploratory actions to meet requested episode lengths.Padding appears in mental rotation, patch reassembly, sliding block, colorization, jigsaw, and maze solvers.
  • Multi-step solvers: Oracle solvers generate trajectories using search, greedy control, state machines, interpolation, or structured action sequences across environments.Examples include BFS for Sliding Block, graph search for mazes, per-axis control for MuJoCo tasks, and staged exploration for Matchstick Rotation.
  • Solver variants: The solver library supports alternative strategies, including permutation-based and swap-based reordering for Video Unshuffle and Jigsaw, and BFS, DFS, and SOS for Matchstick Equation.These variants expose different action sequences and levels of exploration while targeting the same task solution.

F. Analyzing Model Failures

The failure analysis pipeline uses trace annotation and clustering to identify recurring model behaviors, with action looping, early termination, state mismanagement, and visual-information failures prominent across tasks.

  • Failure taxonomy: StringSight uses a VLM annotator to extract trace behaviors and cluster them into higher-level failure patterns.The pipeline records a behavior description, reasoning, and trace evidence, and allows multiple behaviors per trace.
  • Common failure modes: Four recurring failures are restricted action spaces and action looping, state mismanagement, early termination, and failure to use visual or spatial information.Examples include repeating fixed actions, ignoring prior feedback, stopping before reaching the goal, and continuing after visual state changes.
  • State mismanagement: State mismanagement includes ignoring textual or environmental feedback, revisiting explored areas, and repeating illegal actions after prior errors.The taxonomy distinguishes this from merely repeating an action without progress.
  • Failure frequency: Action looping occurs in more than 60% of traces and is followed by early termination, state mismanagement, and failure to use visual or spatial information.Matchstick Equation and Sliding Block show particularly high frequencies of some failure behaviors.
  • Early termination: Early termination includes giving up before the step limit, occurring more often on hard tasks such as Matchstick Equation.Some models explicitly state that the task is unsolvable before exhausting the available steps.

F.1. Failure changes per ablation

Ablation analyses show that history, feedback, initial state information, and representation alter distinct failure patterns, sometimes reducing annotated failures without improving overall performance.

  • Chat history: More chat history reduces immediate action repetition but does not eliminate state mismanagement.The authors associate full-history reductions in looping with less immediate repetition, while models may later revert to default behavior after struggling with earlier state.
  • Feedback: Removing feedback reduces termination, action looping, and state mismanagement, despite decreasing overall performance.The reduction in termination is largely attributed to fewer instances of models giving up after invalid-action feedback.
  • Initial state: Providing ground-truth state at task start lowers action looping and early termination by reducing guessing and giving up.This ablation changes the initial information available before interaction begins.
  • Representation: For tasks other than Matchstick Equation, text representations are processed more effectively than image representations, with less action looping.The text-based format is described as providing clearer guidance for action selection.

F.2. Failure Trajectories Visualization

Trajectory visualizations pair prompts, observations, model outputs, parsed actions, and labeled reasons and evidence to illustrate the four failure categories with concrete episodes.

  • Visualization format: The visualizations display each trajectory’s prompt, image, raw output, parsed action, and StringSight’s reason and evidence labels.This layout connects the model’s emitted text and executed action to the annotated failure category.
  • Action looping: GPT-5 loops on Jigsaw by repeatedly applying the same swap action without making progress.The repeated action is swap(0, 0), (0, 1).
  • State mismanagement: Claude Sonnet 4 repeats a wall-colliding move in Maze 2D after receiving feedback that the move is invalid.The same move is taken again from the identical state later in the trajectory.
  • Early termination: Gemma 3 27B Instruct gives up at step 13 of a 30-step Matchstick Equation episode.The trajectory ends before the allowed interaction budget is exhausted.
  • Visual grounding: Gemini 2.5 Pro continues rotating away from the target in Mental Rotation 3D after visual evidence indicates the wrong direction.The example illustrates failure to use the changing visual state to correct the action sequence.

G. Additional Performance Analysis

VisGym evaluations show broad variation in task difficulty, substantial easy-to-hard degradation, and recurrent action-looping behavior across models. Performance often deteriorates when models continue interacting without incorporating feedback or changing strategy.

  • Difficulty of Each Task: Over 20% average accuracy occurs on Referring Dot-Pointing and Counting, while Mental Rotation 3D (Cube), Patch Reassembly, and Mental Rotation 3D (Objaverse) remain near 1%.These averages sort the easiest and hardest tasks across models.
  • Number of Steps: Models commonly reach the 20-step Easy or 30-step Hard limit, while successful trajectories tend to terminate early or continue until the limit.The step-count distribution is U-shaped across trajectories.
  • Easy to Hard Performance Drop: The largest Easy-to-Hard drops occur on Counting and Jigsaw, with Jigsaw falling near zero when scaling from 2x2 to 3x3.Counting also declines sharply as the number of objects increases.
  • Scalability: Task difficulty can be scaled by increasing patches, blocks, or frames, enabling harder variants as models improve.The paper specifically identifies Patch Reassembly, Sliding Block, and Video Unshuffle as naturally scalable.
  • Model Rankings: Model rankings reveal task-specific strengths and weaknesses rather than a single uniformly dominant model.The ranking procedure assigns tied models the worse numerical rank.
  • Action Looping: In a Jigsaw trajectory, the model repeatedly issues the same swap command between identical coordinates without making progress.The repeated action appears across six consecutive steps.

Sample trajectory for “State Mismanagement" (Claude Sonnet 4)

The trajectory illustrates state mismanagement in a partially observed maze: after receiving wall feedback, the model revisits blocked directions and ultimately exhausts its step budget without reaching the target.

  • State Mismanagement: The model repeatedly revisits directions previously identified as blocked instead of retaining those failed-state constraints.The trajectory explicitly attributes this behavior to failure to remember impossible moves.
  • Task Setting: In the 9x9 maze, the agent must move through white paths from a blue starting circle toward a red target square.The task provides visual maze structure and action feedback during interaction.
  • Trajectory: After an initial failed left move, the model acknowledges the wall but continues exploring without establishing a reliable route.It alternates successful moves with further failed attempts and repeated reconsideration.
  • Feedback Use: The model receives multiple “Cannot move into a wall” responses yet retries blocked directions during later steps.Repeated wall feedback appears at steps corresponding to left, down, and other attempted directions.
  • Outcome: The trajectory ends after the available steps are exhausted without reaching the red target square.The model reports that no successful path was found.

Sample trajectory for “Early Termination" (Gemma3 27B)

The trajectory shows early termination after repeated illegal matchstick moves, with the model abandoning the task and submitting despite lacking a mathematically validated solution.

  • Early Termination: After twelve illegal moves, the model gives up and suspects that segment definitions or move validation are broken.The trajectory records repeated failed attempts and eventual submission of a nonsense move.
  • Task Setting: The task requires fixing a broken equation by moving exactly one match per action while preserving valid symbols.A successful submission must produce a mathematically correct equation.
  • Trajectory: The model repeatedly proposes alternative equations after receiving illegal-move feedback but does not identify a valid move.Its guesses include transformations such as 1 + 1 = 2 and 0 + 0 = 0.
  • Feedback: When the model attempts to undo an action, the environment reports that there is nothing to undo.This occurs after the model has already accumulated several illegal moves.
  • Outcome: The model eventually switches to random or arbitrary moves before submitting failure.It explicitly states that it is at a complete loss and tries moves without a clear strategy.

Sample trajectory for “Visual/Spatial Perception" (Gemini 2.5 Pro)

The trajectory demonstrates visual and spatial perception failures during 3D rotation: the model recognizes some orientation changes but repeatedly overcorrects and misreads the object’s geometry.

  • Task Setting: The task requires matching a rotated right-hand object to a left-hand target using Euler-angle actions within a 10.0-degree tolerance.The available action specifies yaw, pitch, and roll components.
  • Visual Misinterpretation: The model repeatedly reasons from incorrect visual interpretations, including confusing Z and S configurations and misdiagnosing the remaining axis error.Its later state description says the object is distorted and resembles “<” rather than the target “Z.”
  • Shape and Orientation: The model initially converts the object into the correct Z configuration but continues to misalign its yaw and perspective relative to the target.Later analysis notes that the 180-degree roll fixed the fundamental shape while diagonal orientation remained.
  • Perspective: The model correctly identifies that missing side faces indicate a pitch error and applies a pitch correction that reveals them.This is one of the trajectory’s successful uses of visual feedback.
Loading 2601.16973v1…