V-Gym: Enhancing Agentic Visual Reasoning via Skill-Data Co-Evolution

• Execution-derived skills are hypotheses, and targeted varied practice is needed to test applicability and refine procedures using new evidence.Successful rollouts may not reveal which visual cues or tool states mattered; failures may conflate flawed procedures with missing evidence or execution noise. Multimodal variation must preserve consistency among visual content, questions, and answers, along with evidence needed for the intended reasoning and tool use.Source: §1 · pp. 1–2
• Execution trajectories jointly reveal reusable procedures for skill refinement and task challenges and permissible variations for targeted data generation.V-Gym uses trajectories as the shared foundation for both evolution processes.Source: §1 · p. 2
• V-Gym groups trajectories to propose skill patches that revise procedural guidance and applicability conditions, retaining updates only when validation performance improves.Patches targeting the same skill object are consolidated into a candidate update.Source: §1 · p. 2
• V-Gym balances attributed skill-update gains with exploration to select generation seeds, then uses their trajectories to guide targeted practice-data generation.Generated instances that pass model self-checks expand the data bank for subsequent practice and further skill refinement.Source: §1 · p. 2
• Experiments show consistent gains across models and benchmarks, skill transfer across domains and models, and better skill refinement with evolved data.Source: §1 · p. 3

• Skill-distillation research spans reflections, procedural memories, executable skill libraries, trajectory-based revision, and validation of candidate updates.Source: §2 · p. 3
• Data-construction work includes multimodal synthesis and adaptive generation informed by rollouts, success estimates, failure diagnoses, and verifier feedback.Source: §2 · p. 3
• Prior co-evolution links varied agent components; V-Gym studies skill–data co-evolution with a fixed multimodal solver, using shared trajectories and skill-update gains to guide practice construction and selection.Source: §2 · p. 3

3.1 PROBLEM SETTING

• V-Gym frames skill evolution and data evolution as coupled goals for a fixed-parameter multimodal agent, using practice trajectories and evaluation feedback as shared evidence.Candidate data-bank expansions are considered within a fixed generation budget.Source: §3.1 · pp. 3–4
• At each iteration, the agent practices on selected data with a task-dependent skill, records a tool-assisted trajectory, and receives answer-evaluation feedback.The trajectories inform skill-update rewards, which are attributed to practice instances to guide data evolution.Source: §3.1 · p. 4

3.2 SKILL EVOLUTION

• Trajectory analysis groups practice episodes, compares successful and failed trajectories, and proposes patches to procedures or applicability conditions.Patches may revise existing skills or create new skills with procedures and metadata.Source: §3.2 · p. 4
• Patches targeting the same skill object are merged, then independently validated against the same batch-start bank; only updates with positive validation rewards are accepted.Contributing instances are retained for later data-utility attribution.Source: §3.2 · pp. 4–5

3.3 DATA EVOLUTION

• Data evolution attributes accepted skill-update rewards to contributing practice instances and ranks eligible seeds using utility and exploration.Instances with complete feedback receive a utility observation; complete instances supporting no accepted update receive zero utility, while incomplete feedback yields no observation.Source: §3.3 · p. 5
• Selected seeds’ trajectories specify task challenges to preserve and aspects to vary, guiding generation of diverse practice instances at comparable difficulty.Generated candidates undergo checks for answer and evidence reliability, image-question consistency, challenge preservation, and absence of answer leakage or shortcuts; candidates failing after bounded repair are rejected.Source: §3.3 · p. 6
• Accepted generated instances join the shared data bank, and the next epoch selects practice instances from real and generated data using the same score.New instances start with zero utility and practice-count statistics; existing statistics are discounted between epochs.Source: §3.3 · p. 6

4.1 EXPERIMENTAL SETUP

• The evaluation uses three benchmarks, three LVLM backbones, and comparisons with direct reasoning, tool use, and skill-evolution baselines under a shared rollout budget.V-Gym uses two practice epochs by default and reports Avg.@3 and Pass@3 from three test rollouts per instance.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

• V-Gym leads GPT-5.5 baselines on all three benchmarks, improving Avg.@3 by 4.9, 3.7, and 2.0 points, with corresponding Pass@3 gains of 2.8, 3.0, and 1.3 points.The gains are reported for TIRBench, MMSearch-Plus, and MMBrowseComp, respectively.Source: §4.2 · p. 7

4.3 OUT-OF-DISTRIBUTION TRANSFER

• Without further skill updates, domain transfer raises Avg.@3 from 39.6 to 46.2 on VisualToolBench and from 31.3 to 35.4 on AgentVista; GPT-5.4-mini gains 6.7 and 5.6 Avg.@3 points on TIRBench and MMSearch-Plus.The caption describes direct source-to-target application of skills; both domain-transfer targets also gain Pass@3, and both model-transfer targets improve on both metrics.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 ABLATION STUDY

• On TIRBench, adding skill evolution changes Avg.@3/Pass@3 from 63.1/73.8 to 68.5/79.5, and co-evolving data raises them to 71.1/81.5.Removing the hierarchical bank or validation gate reduces performance, with the largest Avg.@3 drop when the gate is removed.Source: §4.4 · p. 8, Table 3 · p. 8
• Random data selection lowers performance, while generation without trajectory-derived guidance recovers little of the full method’s gain over skill evolution alone.The full method performs best, and removing any component lowers both reported metrics.Source: §4.4 · p. 8

4.5 FURTHER ANALYSIS

• Model self-checks accept 70.4–88.8% of generated samples; manual reviews confirm validity, challenge preservation, and comparable difficulty for 96%, 93%, and 90% of samples across the three benchmarks.Mean pairwise agreement across reviewers is 93.56%.Source: §4.5 · p. 8, Figure 3 · p. 8
• Across four epochs, co-evolution continues improving, finishing 3.6 Avg.@3 points above generic synthesis and 5.4 above fixed-real practice, while fixed-real practice plateaus and declines.Both generation strategies use the same budget, and epoch 1 is shared warm-up.Source: §4.5 · p. 8, Figure 4 · p. 9
• After shared warm-up, co-evolution retains the highest accepted-update rate, ending at 25.6% versus 15.4% for generic synthesis and 8.2% for fixed-real practice.The case study contrasts a rejected, instance-specific fixed-data revision with a co-evolved rule that generalizes across grid boundaries and passes validation.Source: §4.5 · p. 9, Figure 4 · p. 9

5 LIMITATIONS

• The current process is limited for highly complex real-world multimodal tasks and long videos, while repeated rollouts, validation, and generation constrain training and validation set sizes.The authors identify larger, more diverse practice datasets and broader validation coverage as potential directions for improvement.Source: §5 · p. 9

6 CONCLUSION

• V-Gym uses execution trajectories and validation feedback to refine reusable skills and specify targeted, diverse practice data.The framework is described as operating with fixed model parameters.Source: §6 · p. 9
• Evaluations report improvements over compared baselines across diverse visual reasoning benchmarks, with generalization to unseen domains and transfer to weaker models.The authors conclude that learning from experience and adapting subsequent practice are complementary components of agent self-improvement.Source: §6 · p. 9
• Generative AI tools were used for language polishing, literature retrieval, and constructing synthetic multimodal practice data, which underwent self-checks and sampled human review.The authors state that they reviewed the AI-assisted text and references and take responsibility for the final content and artifacts.Source: AI USE STATEMENT · p. 10

A.1 DETAILED CO-EVOLUTION ALGORITHM

• Each epoch selects practice instances, gathers feedback-bearing trajectories in batches, groups them to propose and merge skill patches, and applies only updates with positive validation reward.The final skill bank is frozen for inference.Source: §A.1 · p. 15
• After batch practice, the method updates instance utilities and, before the final epoch, ranks seeds, generates and self-checks candidates, admits accepted data, and discounts accumulated utilities.Generation stops when the budget is met or seeds are exhausted; each seed allows up to five failed attempts.Source: §A.1 · p. 15

A.2 SKILL EVOLUTION DETAILS

• Trajectory groups propose patches to existing or new skills; patches targeting the same object are merged, then tested against the same batch-start bank on a retrieved validation panel using matched fresh rollouts.Updates with positive mean score difference are merged, and the router is refreshed from task-specific applicability metadata.Source: §A.2 · pp. 15–16

A.3 DATA EVOLUTION DETAILS

• Complete instances require trajectories, reflection, and relevant validation outcomes for utility observations and seed eligibility; those supporting no accepted update receive zero utility.Incomplete feedback yields neither an observation nor an increment to the instance’s observation count.Source: §A.3 · p. 16
• Accepted-update rewards are divided equally among distinct contributing instances, then aggregated into one utility observation per complete practice instance for seed ranking.The resulting utility is a proxy for future skill improvement.Source: §A.3 · p. 16
• Generation plans preserve a seed task’s key challenges and comparable difficulty while specifying format requirements and aspects that may vary.Plans do not expose private seed images, raw trajectories, or exact answers; generators build or acquire content, derive answers, and record supporting evidence.Source: §A.3 · p. 16
• Self-checks assess answer and evidence reliability, image-question consistency, challenge preservation, and leakage or shortcuts, followed by duplicate, label-conflict, and validation-overlap checks.Failed candidates receive bounded repair and renewed checks; changed answers require renewed evidence verification.Source: §A.3 · p. 16
• After each nonfinal epoch, generation targets at most ⌊ρM⌋ admitted instances, stopping per seed after acceptance or five failed attempts.Each successful seed contributes one instance to the shared bank.Source: §A.3 · p. 16
• New instances enter with zero utility statistics, but admission does not guarantee next-epoch selection: SELECT reranks the expanded bank and fixes the selected set for that epoch.Statistics are discounted before reranking, and real and generated instances use the same selection rule.Source: §A.3 · p. 16

B.1 TOOL DETAILS

• Table 4 summarizes tool capabilities and main inputs, while callable names and argument schemas come from enabled interfaces.Tool availability does not mean every tool is used on every instance.Source: §B.1 · p. 16, Table 4 · p. 17

B.2 DATASET DETAILS

• TIRBench, MMSearch-Plus, and MMBrowseComp are main evaluation datasets; VisualToolBench and AgentVista test transfer without target-domain practice or generation.Training data support practice, validation compares candidate skill-bank updates, and held-out test data evaluate the frozen bank.Source: §B.2 · p. 17
• Table 6 marks tool availability by dataset, with transfer targets lacking target-domain generation.On MMSearch-Plus and MMBrowseComp, generated images must originate from public webpages; Code Interpreter may process acquired pixels but cannot draw replacement scenes.Source: §B.2 · p. 17, Table 6 · p. 18

B.3 PARAMETER SETTINGS

• Methods receive the same total training rollout budget, Btrain = EMN, while retaining their own sampling and update mechanisms.For methods using held-out validation, including V-Gym and SkillOPT, each candidate update is evaluated on at most 15 validation instances.Source: §B.3 · p. 18
• V-Gym uses solver temperature 0.6 for practice, comparative validation, and test.Table 7 summarizes its parameter settings.Source: §B.3 · p. 18

B.4 HUMAN REVIEW OF EVOLVED DATA

• Three institutional volunteers independently reviewed 100 randomly sampled model-accepted instances from each of three datasets, totaling 300.They assessed validity, preservation of core task challenges, and comparable difficulty.Source: §B.4 · p. 18
• Reviewers classified four error types: unreliable answers or evidence, image-question mismatch, Core-Challenge Drift, and answer leakage or shortcuts.The categories are also illustrated in Figure 5.Source: §B.4 · p. 18
• Final labels use majority vote, with discussion to reach consensus when all three reviewers disagree; mean pairwise agreement before resolution was 93.56%.Accepted means model acceptance during self-checking, whereas Valid means confirmation by manual review.Source: §B.4 · p. 18

B.5 EXAMPLES OF VALID GENERATED DATA

• Trajectory-derived generation plans separate difficulty-defining challenges from aspects that can vary, preserving comparable difficulty while introducing meaningful variation.Figure 6 provides examples of valid generated data.Source: §B.5 · p. 18
• Variation broadens practice around identified bottlenecks while retaining core challenges, creating opportunities to consolidate strategies and test their applicability beyond original instances.Source: §B.5 · p. 18

C.1 TEST-TIME VARIABILITY

• V-Gym has the lowest mean per-benchmark standard deviation of single-run accuracy among compared methods for each backbone.The analysis uses three independent test runs with the model, evaluation settings, and final skill bank fixed, so it measures repeat-test variability rather than variation across skill-evolution runs.Source: §C.1 · p. 19

C.2 DATA-EVOLUTION HYPERPARAMETERS

• At ρ = 0.6, Avg.@3/Pass@3 gains are 2.6/2.0 points over no data evolution, while Pass@3 rises 4.1 points from α = 0.2 to α = 1.Larger budgets yield similar performance, and Avg.@3 is highest at full seed coverage. Figures 5–6 contrast generation failure modes with valid variants preserving challenge and comparable difficulty; Table 8 reports mean per-benchmark sample SD for Table 1 accuracy.Source: Figure 5 · p. 20, Figure 6 · p. 20, Table 8 · p. 20

D CASE STUDIES

• The case study presents examples of the global skill, router, and specific skill used at test time, with the Skill Bank learned by GPT-5.5 on TIRBench.Source: D CASE STUDIES · p. 21

Global Skill: global_skill.md

• The global skill standardizes visual-evidence checks, input fidelity, staged validation, contradiction resolution, and final-answer auditing across task families.Source: Global Skill: global_skill.md · p. 21

Specific Skill: skill_0016.md

• The jigsaw skill extracts consistently identified panels, measures directional seam compatibility, and minimizes global seam costs while enforcing grid dimensions and one use per panel.Source: Specific Skill Router: router.md… · p. 22, Specific Skill: skill_0016.md · p. 22

Specific Skill Router: router.md (sample)

• Router entries specify when skills apply and when they do not, covering complete panel layouts, scaled straight-line distances, and validation of supplied maze commands.The examples also distinguish required from optional Code Interpreter use.Source: Specific Skill Router: router.md… · p. 22

Specific Skill: skill_0016.md (continued)

• It rescores competing layouts under alternative metrics or crops, validates candidates against structural evidence, and audits that each identifier appears exactly once.The verified layout is serialized in the requested order and syntax.Source: Specific Skill: skill_0016.md · p. 22, Specific Skill: skill_0016.md… · p. 23

E PROMPTS

• The appendix provides six compact prompt templates aligned with the method, using supplied-content placeholders and multimodal image inputs.Tool names, inputs, availability, and sampling settings follow the referenced tables.Source: E PROMPTS · p. 23