Self-Evolving Visual Agent via Margin Reward

1 INTRODUCTION

• Existing tool-use learning commonly depends on curated question–answer pairs or expert demonstrations, which are expensive to collect at scale.Valuable tasks are especially needed for visual agents that rely on tool invocation.Source: §1 · p. 1
• Useful training questions should challenge unaided reasoning yet remain solvable with tools, and the suitable difficulty region shifts as the solver improves.A productive curriculum should track the changing relationship between question difficulty and tool utility.Source: §1 · pp. 1–2
• Marvo pairs a challenger that generates question–answer tasks from unlabeled images with a solver that learns through reinforcement learning using tools.The solver’s performance feeds back into the training curriculum, reducing dependence on a fixed manually curated task set.Source: §1 · p. 2
• The margin reward measures the solver’s performance difference with versus without tools, encouraging questions where tool access improves answering.Questions easy in both settings or unsuccessful in both receive little reward.Source: §1 · p. 2
• By measuring relative benefit from tool access, the reward favors questions where tools make a practical difference over questions selected only for difficulty or tool-assisted success.The objective is evaluated with respect to the current solver, so it can guide question generation as capabilities evolve.Source: §1 · p. 2
• 7.1%: Marvo improves Qwen2.5-VL-7B average accuracy across 11,691 questions spanning agentic reasoning, visual search, mathematical reasoning, and real-world reasoning.The paper presents the performance gap between tool-free and tool-augmented answering as a useful signal for generating training questions.Source: §1 · p. 2

2.1 PRELIMINARY

• R-Zero’s uncertainty reward peaks at intermediate solver difficulty but does not indicate whether tools help or improve tool invocation.It uses a solver-voted pseudo-label and reaches its maximum when the most likely answer has probability 1/2.Source: §2.1 · p. 3

2.2 INCENTIVIZING WITH MARGIN REWARD

• Marvo rewards the challenger by the solver’s accuracy difference with and without tools, favoring questions solvable with tools but not unaided.This margin targets questions difficult enough to warrant tool use while still within the solver’s tool-assisted capabilities.Source: §2.2 · p. 3

2.3 EQUILIBRIUM ANALYSIS

• The analysis describes gap-closed and gap-saturated equilibria: tool-use capability is internalized in the former, while an irreducible tool-access gap remains in the latter.At the saturated state, the solver’s gradient vanishes and learning stalls; partially solved states are transient, where the uncertainty gate is active.Source: §2.3 · p. 4

2.4 TRAINING PARADIGM

• The challenger investigates unlabeled images with tools and proposes question–answer pairs; alternating GRPO updates the challenger and tool-enabled solver.Questions with solver pass rates of 0 or 1 are discarded; only the tool-enabled solver is trained to avoid potentially conflicting updates from mixing harnesses.Source: §2.4 · p. 4

3.1 SETTING

• Across the evaluated benchmarks, TIR-Bench, V*, HRBench-4K, MathVerse, MathVision, WeMath, and RealWorldQA measure agentic reasoning, visual search, math, and real-world understanding.The study uses PyVision as its interactive, multi-turn tool harness, with Qwen2.5-VL-7B-Instruct and Qwen3-VL-4B base models trained from SFT checkpoints on a pool of 42,124 images.Source: §3.1 · p. 5

3.2 MAIN RESULTS

• Marvo consistently outperforms R-Zero at matched iterations across both model families, with gains on tool-dependent benchmarks including HR4K, V*, and TIR-Bench.The comparison covers two model families; Table 1 reports results across seven benchmarks.Source: §3.2 · p. 6, Table 1 · p. 6
• Marvo has fewer negative per-round deltas than R-Zero, with 5 of 42 versus 9 of 42, and a smaller largest single-round regression, 0.7 versus 5.5 points.R-Zero’s regressions concentrate on tool-dependent benchmarks, while Marvo’s near-monotone improvement indicates a more stable co-evolution curriculum.Source: §3.2 · pp. 6–7
• On tool-dependent tasks, Marvo improves TIR-Bench by +1.5 for the 4B model and +5.7 for the 7B model, while 7B cumulative gains reach +5.8 on V* and +6.8 on HR4K.After three rounds, the 4B R-Zero model is slightly below its SFT start on TIR-Bench.Source: §3.2 · p. 7

3.3 TRAINING DYNAMICS

• Across three evolution rounds, solver reward rises within each round, while the generated problems remain within the uncertainty gate’s training-signal sweet spot.At round starts, solver reward resets as the updated challenger proposes questions the current solver cannot yet answer well; without tools, probe pass rates drop.Source: §3.3 · p. 7, Figure 4 · p. 7

• Marvo is evaluated at three PyVision-RL post-training stages: the Instruct model, the SFT cold-start checkpoint, and the final RL model.Figure 5 compares the reported PyVision-RL training progress with Marvo runs launched from different checkpoints.Source: §4 · p. 7, Figure 5 · p. 8
• Introducing Marvo to the post-trained model yields little improvement; the authors conjecture that entropy falling from 0.75 to about 0.4 leaves little exploration headroom.They suggest Marvo’s influence fades once policy entropy has collapsed.Source: §4 · pp. 7–8
• Average tool turns per evaluation question rise across tasks, and early-stage Marvo incentivizes tool invocation without human-curated question sets or demonstrations.The authors report that tool invocation ability does not degrade under self-evolution.Source: §4 · p. 7, Figure 6 · p. 8
• The challenger’s benchmark accuracy is consistently below the trained solver’s and essentially flat across rounds, while its results oscillate on TIR-Bench and improve only marginally on V*.The table reports challenger performance.Source: §4 · p. 8, Table 3 · p. 9
• 84.29 to 91.10 on V* with the zoom-in tool as Marvo improves across rounds, while R-Zero gains early momentum but oscillates after Round 1.The comparison suggests Marvo’s growth persists when the action space changes from Python execution to image zoom-in.Source: §4 · p. 8, Table 2 · p. 9
• With H2 capped at three conversation turns, the solver improves round by round but finishes below the corresponding Tab. 1 results, which the authors attribute to a sparse extended-margin signal.The extended margin favors questions requiring more than three turns, making it near zero on most batches.Source: §4 · p. 9, Table 4 · p. 9

• Label-free RL removes the need for gold answers but still presupposes an existing corpus of unlabeled questions, while human-curated tasks for RL with verifiable rewards are costly to scale.Source: §5 · p. 9
• Self-evolution methods aim to eliminate human-written tasks and labels, but self-challenging approaches rely on external verifiers and are confined to domains with explicit correctness oracles.Source: §5 · p. 9
• Marvo defines a margin-reward metric for visual-agent self-evolution, extending beyond text-only reasoning and supporting co-evolution of solvers, challengers, and tasks.Source: §5 · p. 10

6 CONCLUSION

• Marvo uses solver performance differences with and without tools as a challenger reward to create an adaptive curriculum for learning tool invocation without human-annotated data.Source: §6 · p. 10

A LLM AND AI USAGE STATEMENT

• Generative AI tools assisted with language clarity, utility scripts, debugging, and code restructuring; authors retained responsibility and reviewed the outputs.Source: A LLM AND AI USAGE STATEMENT · p. 13

B.1 VISUAL AGENTS

• Zoom-in agents search high-resolution images through crops, while code-execution agents compute visual evidence with Python tools, enabling precise quantitative answers but risking buggy or unproductive traces.Source: §B.1 · p. 13

B.2 REINFORCEMENT LEARNING WITH GRPO

• GRPO samples groups of responses and normalizes rewards against the group mean instead of using a learned value function, then optimizes a clipped surrogate over response tokens.The stated clipping bounds are εlo = 0.2 and εhi = 0.28, with no KL penalty against a reference model.Source: §B.2 · p. 13

B.3 ASYNCHRONOUS TRAINING

• Asynchronous Solver training generates the next batch while the current batch's gradient step runs, making each consumed batch one policy step stale.For learner step θt, the batch comes from πθt−1.Source: §B.3 · p. 13
• The importance ratio compares the current learner policy with the one-step-lagged policy that generated the batch.The asynchronous variant sets πθold = πθt−1 and uses their output-probability ratio.Source: §B.3 · p. 13
• The ratio corrects exactly for one-step staleness and clipping bounds its deviation, while group-relative advantages remain unaffected.The advantages depend only on rewards within a batch sampled from the same stale policy.Source: §B.3 · p. 14

C NASH EQUILIBRIUM ANALYSIS

• At equilibrium, the generated problems fall into gap-closed or gap-saturated cases, with the latter requiring the solver to have saturated accuracy under H1.The characterization assumes the local regularity condition, which excludes intermediate-accuracy plateaus caused by the absence of a feasible accuracy-improving update.Source: C NASH EQUILIBRIUM ANALYSIS · p. 15

2. Gap-saturated equilibrium. If m∗> 0, then

• In the gap-closed case, H1 and H2 are indistinguishable on equilibrium problems, indicating transfer of H1 orchestration behavior through shared solver weights.For a positive margin, the solver is saturated under H1 but remains inaccurate under H2, leaving a harness-specific capability gap.Source: C NASH EQUILIBRIUM ANALYSIS · p. 15, §2 · pp. 15–16

D.1 BASE MODELS

• Marvo is evaluated with Qwen2.5-VL-7B and Qwen3-VL-4B, which differ in architecture, training pipeline, and capacity; both improve after training.The authors describe the challenger–solver co-evolution, accuracy-margin reward, and GRPO updates as model-agnostic, requiring only tool invocation and verifiable answers.Source: §D.1 · p. 16

D.2 SFT DETAILS

• For the PyVision harness, SFT checkpoints provide a lightweight cold start because the base model struggles to operate the code interpreter directly.Qwen2.5-VL-7B uses a provided checkpoint, while Qwen3-VL-4B is fine-tuned on the same SFT data.Source: §D.2 · p. 16

D.3 RL HYPERPARAMETERS

• The supplementary lists the training hyperparameters for the accuracy-margin run and provides a skeleton code base.Figure 8 identifies the SFT hyperparameters and training loss; Table 5 is labeled as the training-hyperparameters table.Source: Figure 8 · p. 16, §D.3 · p. 17, Table 5 · p. 17

TIRBench

• TIRBench tests tool orchestration across 13 tasks and 1,215 samples designed to require multi-step image manipulation rather than static observation.The paper treats it as the primary in-domain benchmark for the method.Source: §D.4 · p. 17

HRBench

• HRBench-4K tests fine-grained perception and spatial reasoning when decisive evidence occupies a small region of an ultra-high-resolution image.The passage notes that zooming should help in this setting, while naive resizing can destroy answer-relevant information.Source: §D.4 · pp. 17–18

V∗Bench

• V∗Bench evaluates visual search in crowded high-resolution images, requiring localization of small task-relevant regions for attribute recognition or spatial relationships.It contains 191 samples: 115 attribute-recognition and 76 spatial-relationship samples.Source: §D.4 · p. 18

RealWorldQA

• RealWorldQA uses over 700 everyday photographs paired with short, verifiable questions to assess practical spatial understanding and grounding in naturalistic scenes.Many images are captured from vehicles and other everyday settings.Source: §D.4 · p. 18

Visual mathematics benchmarks

• MathVision, MathVerse, and WeMath assess visual mathematics through competition problems, diagram interpretation, and fine-grained reasoning-step diagnosis.MathVision covers 3,040 problems across 16 disciplines and five difficulty levels; MathVerse provides 15K instances across six text-versus-vision variants; WeMath organizes problems into 67 knowledge concepts.Source: §D.4 · p. 18

D.5 TRAINING DATA

• The RL training images come from PyVision-RL; after oversized images are filtered out, the pool contains 42,124 images across sources and resolution scales.These unlabeled images are the only inputs used throughout the Marvo experiments.Source: §D.5 · p. 18, Figure 9 · p. 18

E CHALLENGER PROMPT

• The challenger prompt requires code-based inspection of at least one cropped region before producing an image-answerable multiple-choice question with four options and an unambiguous gold answer.The output format specifies the question, options A–D, and a single answer letter.Source: E CHALLENGER PROMPT · p. 19

F.1 PER-ROUND DATA DISTRIBUTION

• RealWorldQA has the smallest performance gains among the benchmarks, and its isolated image cluster is not covered by the training images.The figure compares image embeddings for 100 images from each benchmark with 100 images underlying generated questions in each training round, using a shared two-dimensional projection.Source: Figure 10 · p. 20, §F.1 · p. 21

F.2 ITERATION SCALING

• MathVision-Mini drops from 27.3 at round 5 to 23.4 at round 7, while V* falls from 78.5 in round 6 to 72.3 in round 7.Most benchmarks peak by round 5 or 6; TIR-Bench declines from 21.7 to 21.0, while WeMath stays in the 62–64 range without meaningfully exceeding its round-5 score.Source: Table 6 · p. 21, §F.2 · p. 21
• Proposed-question judge accuracy falls from 68.5% in round 1 to 43.6% in round 8, mirroring benchmark degradation after rounds 5–6.The authors attribute the instability to declining training-data quality and suggest it drives the scaling bound rather than optimization limits.Source: §F.2 · p. 21, Table 7 · p. 21

G.1 CHALLENGER EXAMPLE TRACE

• A full trajectory shows a challenger using the PyVision harness to propose a visual question–answer pair from an unlabeled image.Figure 11 presents an example trajectory.Source: §G.1 · p. 22, Figure 11 · p. 23

G.2 EXAMPLE PROPOSED QUESTIONS

• On the same images, proposed questions increase in complexity as Marvo proceeds.Table 8 presents examples of the proposed questions.Source: §G.2 · p. 22, Table 8 · p. 22