PaperlayerView exact revision on arXiv

Source-linked AI summary

VeriEvol: Scaling Multimodal Mathematical Reasoning via Verifiable Evol-Instruct

Haoling Li, Kai Zheng, Jie Wu, Can Xu, Qingfeng Sun, Han Hu, Yujiu Yang

arXiv:2606.23543v1cs.AIcs.CLcs.CVcs.LG

TL;DR

Visual mathematical reasoning needs harder image-grounded prompts and reliable answer labels as reinforcement-learning data scales. VeriEvol decouples route-specific prompt evolution from hypothesis–test answer falsification, improving the five-benchmark mean by +3.88 over an un-evolved RL baseline with the backbone, SFT initialization, and GRPO recipe held fixed.

  • Problem

    Existing multimodal pipelines often produce low-difficulty or weakly visual prompts, while unverified labels can become repeated reward signal during reinforcement-learning scaling.

  • Method

    VeriEvol combines route-specific type-aware prompt evolution with HTV-Agent’s multi-stage, refutation-seeking answer verification before policy updates.

  • Results

    +3.88: the RL-stage design improves the five-benchmark mean over an un-evolved RL baseline with backbone, SFT initialization, and GRPO recipe held fixed.

  • Takeaways & Limitations

    Verifier channels are complementary rather than redundant, supporting reliability scaling through targeted refutation channels without redesigning the optimizer.

  • Takeaways & Limitations

    VeriEvol depends on seed-pool coverage, and HTV-Agent can accept wrong answers when solvers, verifiers, and tools share a blind spot.

Abstract

from arXiv · show

Scaling reinforcement learning for visual mathematical reasoning requires more than generating harder questions: as data volume grows, the reward labels themselves must remain reliable. Yet existing data pipelines scale supervision while trusting the labeller, and policy-side methods assume the underlying answers are already correct. We instead treat scaling as a verifiable data-construction problem and decouple two axes before any policy update: prompt difficulty, expanded by route-specific evolution operators, and answer reliability, enforced by offline hypothesis-test falsification. We instantiate this as VeriEvol, an iterative framework with two extensible components: a type-aware evolution module that rewrites low-difficulty image-question seeds into harder, image-grounded prompts; and HTV-Agent, a verifier that accepts an answer only after multi-source counter-evidence has failed to refute it. The resulting verified data scales in volume, extends by adding evolution routes or verifier channels, and plugs directly into existing GRPO-style RL recipes. On a five-benchmark visual-math suite, scaling evolved SFT data from 10K to 250K samples raises the mean accuracy from 35.42 to 54.73; then, with backbone, SFT initialization, and GRPO recipe held fixed, VeriEvol adds a cumulative +3.88 over an un-evolved RL baseline, of which +1.82 comes from evolved prompts and +2.06 from the HTV-Agent verifier. We release the prompts, data, models, code, and the full verifier trace of every sample, so that downstream work can scale and audit the pipeline rather than only inspect its outputs.

1 Introduction

The introduction frames visual-mathematical RL as a data-quality and reward-reliability bottleneck, then proposes scaling prompt difficulty and answer verification as properties of the data. VeriEvol operationalizes this approach through multimodal evol-instruct and verified-data scaling for SFT and RL.

  • Visual reasoning requires diagram grounding, chart reading, symbol alignment, and multi-step mathematical inference, making data and reward quality critical training bottlenecks.
  • Scaling multimodal training requires harder, genuinely image-grounded prompts because larger synthetic corpora can otherwise emphasize recognition, local extraction, length, or text priors.
  • Prior data pipelines leave answers unverified, while policy-side methods assume correct answers; VeriEvol instead makes reliability a property of the training data.
  • VeriEvol transforms low-difficulty image-question seeds into harder, image-grounded prompts for SFT and RL, while scaling verified data across training budgets.SFT data growth from 10K to 250K raises the mean by +19.31; verified-data growth from 10K to 130K adds +4.60 at RL.
  • +3.88 is contributed by VeriEvol’s design over an un-evolved RL baseline at a fixed 130K budget.

2 Related Work

Prior work spans multimodal-math benchmarks, SFT data construction, prompt evolution, and RL reward reliability. VeriEvol is distinguished by verifying answers during data construction before policy updates, targeting visual-reasoning failures that step-level text verifiers cannot diagnose.

  • Benchmarks and training data: Canonical benchmarks anchor multimodal-reasoning evaluation, complemented by datasets probing olympiad difficulty, perturbations, knowledge structure, multi-image composition, and multilingual coverage.The named anchors are MathVista, MathVerse, MMMU, and MATH-Vision; additional benchmarks include OlympiadBench, DynaMath, We-Math, multi-visual math, and M3Kang.
  • Benchmarks and training data: SFT data pipelines use cross-modal chain-of-thought, web mining, and code-aided rendering, but do not provide answer verification at RL-grade reliability.Honey-Data-15M and Lin et al.’s multimodal reasoning corpus serve as disjoint SFT and RL seed pools in the experimental setting.
  • Prompt evolution for reasoning: Prompt-evolution methods rewrite seed instructions into more complex variants, with multimodal extensions including knowledge-tree curricula and evolution-style SFT combined with GSPO-based RL.Examples include Evol-Instruct, MMEvol, MathBook-7B, and OpenMMReasoner’s 874K evolution-style SFT samples.
  • RL training: reward design, policy optimization, and answer verification: Prior RL work improves reward reliability through reward design, supervision shaping, and multimodal judging, alongside text-domain RL systems.The passage groups prior efforts into families and identifies reward design and supervision shape as one family’s directions.
  • RL training: reward design, policy optimization, and answer verification: VeriEvol verifies answers before policy updates, rather than assuming answers are correct or correctable inside the policy loop, and targets visual grounding failures and text-only shortcuts.Its data-construction-stage verification addresses failure modes that step-level text verifiers cannot diagnose.

3 VeriEvol: Evolving Prompts, Falsifying Answers

VeriEvol scales visual-math data construction by independently increasing prompt difficulty and answer reliability, using route-specific evolution and auditable hypothesis-test verification. Its outputs are traceable training samples whose prompts remain image-grounded and whose answers pass deterministic, multi-channel acceptance checks.

  • 3.1 Problem Formulation: A training sample combines an image, harder image-grounded prompt, verified answer, and traceability metadata covering routing, evidence, tools, decisions, and reward checks.VeriEvol returns separate SFT and RL data products built from these verified samples.
  • 3.1 Problem Formulation: VeriEvol separates prompt difficulty from answer reliability so each axis can scale, ablate, and audit independently as the data pool expands.Difficulty changes act on questions, while verification acts on candidate answers; their RL-stage contributions are reported as near-additive.
  • 3.2 Type-Aware Prompt Evolution: Routing assigns each image-question seed a tag tuple covering problem family, answer type, recommended tools, and a compact visual-evidence sketch.This prevents one uniform evolution template from over-specifying some image types or under-specifying others.
  • 3.2 Type-Aware Prompt Evolution: Route-specific operators rewrite seeds into harder prompts while preserving necessary visual evidence and aligning reasoning skills and answer formats with routed tags.Each route samples operators from its own constrained operator set.
  • 3.2 Type-Aware Prompt Evolution: The question gate independently checks image grounding, visual consistency, image necessity, difficulty lift, and deduplication before admitting generated prompts.The gate is designed to avoid rewarding the generator’s own stylistic preferences.
  • 3.3 HTV-Agent: Hypothesis–Test Answer Verification: HTV-Agent verifies accepted prompts through independent hypotheses, counter-evidence search, conflict resolution, and a deterministic acceptance gate.Its verification loop can trigger prompt rewriting when a generated question fails the question gate.
  • 3.3 HTV-Agent: Hypothesis–Test Answer Verification: Three solver branches use temperatures 0.2, 0.6, and 0.15, with the tiebreaker invoked only when the first two disagree.Each branch returns a candidate answer, reasoning trace, claimed evidence, and normalized confidence.
  • 3.3 HTV-Agent: Hypothesis–Test Answer Verification: The conjunctive answer gate requires valid format, visual support, verifier approval, executable assertions, and solver-cluster agreement; any failing channel rejects the sample.Answers are normalized before checking equivalent representations such as percentages, option labels, symbolic forms, and unit-bearing numbers.

4 Experiments

Experiments isolate the contributions of prompt evolution and answer verification under controlled SFT/RL protocols, showing gains from both components and continued scaling with verified data. External comparisons and diagnostic analyses further characterize where VeriEvol leads, how evolved prompts improve training dynamics, and how verification improves reliability.

  • Experimental protocol: Three RL pipelines share VeriEvol-SFT-Init and the GRPO recipe, isolating RL-stage effects from un-evolved prompts, evolved prompts, and verified answers.RL-Origin, RL-Evol, and RL-Evol+Verifier differ only in RL data construction.
  • Ablations: +2.93 mean accuracy: VeriEvol-SFT raises the five-benchmark mean from 51.80 to 54.73 versus Seed-only SFT, peaking at +3.98 on DynaMath.The comparison changes only whether the SFT corpus is evolved, isolating the SFT-stage contribution of evolution.
  • Ablations: +3.88 mean accuracy: full VeriEvol improves over RL-Origin, with +1.82 from evolution and a further +2.06 from HTV-Agent verification.Evolution raises the average from 55.24 to 57.06, while verification improves every benchmark, peaking at +3.59 on DynaMath and +3.05 on We-Math.
  • Training dynamics: ≈0.43 terminal reward and ≈0.31 policy entropy: evolved prompts outperform original prompts at ≈0.35 reward and ≈0.22 entropy during GRPO training.Harder, image-grounded prompts preserve multiple reasoning routes and exploration as reward rises.
  • Scaling: 54.52→59.12 mean accuracy: increasing verified RL data from 10K to 130K improves the five-benchmark mean by +4.60 under fixed training conditions.The SFT initialization, GRPO recipe, verifier model, and one-epoch budget remain fixed.
  • Verifier analysis: +4.51pp mean: full HTV-Agent improves over the Raw single-call baseline across six multimodal-math benchmarks and never degrades any benchmark.Multi-solver self-consistency contributes most on visually rich tasks, including +6.00pp on MathVision from voting alone.

5 Limitations, Broader Impact, and Responsible Release

VeriEvol’s coverage and verification gates remain imperfect, narrowing the RL-ready subset and leaving risks from bias, reward shortcuts, licensing, and privacy. The release addresses these risks through transparency measures, license tracking, deduplication, filtering, and metadata substitution when raw-image redistribution is unclear.

  • Limitations: VeriEvol cannot fully offset under-represented domains or diagram styles, and imperfect gates may allow textual cues or wrong answers through shared blind spots.The RL-ready subset is necessarily narrower than the SFT-ready subset.
  • Positive impact, risks, and safeguards: The framework may reduce visual-math data-construction costs, but synthetic pipelines can amplify source bias, generator or judge preferences, and spurious reward shortcuts.It can also make construction more transparent for smaller teams.
  • Positive impact, risks, and safeguards: The release tracks per-source licenses, deduplicates and filters content, and substitutes metadata plus source pointers when raw-image redistribution rights are unclear.Raw images with restrictive licenses or privacy concerns pose legal and ethical risks.

6 Conclusion

VeriEvol frames visual-mathematical RL data construction as jointly scaling prompt difficulty and answer reliability before policy updates. Its verified data improves fixed-recipe RL, supports composable extensions, and is released with audit artifacts.

  • Framework: VeriEvol decouples route-specific prompt evolution from hypothesis-test answer falsification before policy updates, preserving image dependence and label reliability as data scales.The framework is designed to plug into existing policy optimizers.
  • Results: +3.88 mean accuracy over an un-evolved RL baseline results from the RL-stage design with backbone, SFT initialization, and GRPO recipe held fixed.The gain reaches +5.92 on MathVision_MINI, while verified RL data continues scaling from 10K to 130K samples (+4.60).
  • Composability and release: VeriEvol-RL is composable with future GRPO-style optimizers, length penalties, and rollout reweighting, and can extend to other image-grounded tasks with deterministic or programmatic checks.The authors will release code, data, models, and verifier traces for downstream auditing and reuse.

A Training Compute and Hyperparameters

VeriEvol uses a fixed, reproducible training configuration across SFT and GRPO-based RL. SFT initializes every RL run, while RL variants share the same batching, reward, prompt, and answer-extraction setup.

  • SFT stage: SFT fine-tunes Qwen2.5-VL-7B-Instruct on 250K VeriEvol-SFT samples for 10 epochs with global batch size 512 and learning rate 5 × 10^-5.The resulting VeriEvol-SFT-Init checkpoint initializes every reported RL run.
  • RL stage: RL variants use GRPO with global batch size 256, group size 16, dynamic batching, and binary verifier-based rewards plus a small format-error penalty β.The same prompt format and reference-answer extraction routine apply across RL variants and data-scale ablations.
  • Compute and reproducibility: All runs use homogeneous H20 80 GB GPUs, with each run occupying one 8-GPU node at tensor-parallelism degree 8.The SFT run additionally uses sequence-packed data parallelism across two nodes.

B HTV-Agent Component Ablation: Full Details

This appendix ablates HTV-Agent’s self-consistency, programmatic execution, and verifier-filter components as an inference-time judge, while documenting latency and data-validity caveats. The results indicate complementary evidence channels, with self-consistency especially influential on MathVision, but cross-configuration comparisons remain approximate.

  • Ablation setup: HTV-Agent is evaluated with three ablations: single-solver, self-consistency without python_exec, and the full three-solver pipeline with python_exec and verifier filtering.All configurations use Gemini-3.5-Flash, 300 samples per benchmark, and shared high-effort inference settings; the raw single-call baseline is also compared.
  • Component decomposition: +6.00pp is recovered by self-consistency voting alone on MathVision, without code execution, making it the dominant component there.The decomposition targets OlympiadBench and MathVision, where all four configurations were run.
  • Latency: Inference-time HTV-Agent is substantially more expensive than a single LLM call because it runs three solver passes plus verification.At data construction, this cost is paid once per training sample and reused across downstream RL runs consuming VeriEvol-RL.
  • Qualitative findings: The full agent never degrades the listed-suite mean, while multi-solver voting and programmatic execution provide complementary evidence channels.A verifier filter that can abstain to the raw answer is described as a safety net, though the appendix notes validity caveats.
  • Data validity: Cross-seed comparisons are approximate, and some self-consistency runs ended before completing all 300 samples.The appendix attributes these caveats to multiple seeds and API channels used during the ablation campaign.

C Evolution Operator Design Space and the 12-Topic Instantiation

The appendix defines VeriEvol’s evolution-operator design space across visual route, reasoning skill, answer type, and verification contract. Its taxonomy pairs image-grounded reasoning operators with explicit rejection criteria to support both SFT-usable and RL-usable data.

  • Design axes: The operator taxonomy organizes VeriEvol along four axes: visual route, reasoning skill to amplify, answer type, and verification contract.The design documents constraints for producing data usable in both supervised fine-tuning and reinforcement learning.
  • Visual routes: Visual-pattern routes support pattern completion, transformations, symmetry detection, combinatorial counting, rule induction, and missing-cell reasoning.These routes are triggered by repeated shapes, cells, transformations, colors, symmetry, or missing entries and require observable rule support.
  • Visual routes: Multi-panel and temporal routes apply delta, ordering, causal-transition, comparison, aggregation, counterfactual, and viewpoint-consistency operators.Their contracts require panel identity and ordering and reject changes that are not visibly supported or require external video context.
  • Visual routes: Graph, set, probability, 3D-geometry, timeline, and uncertainty routes pair specialized reasoning operators with visibility, determinacy, or tolerance-based rejection rules.Examples include reachability, inclusion-exclusion, surface/volume computation, duration aggregation, and robust ranking under uncertainty.

C.1 Reasoning-Skill Operators · C.2 Answer-Type and Reward Contracts

C.1 defines route-independent operators that make visual-mathematical prompts harder while preserving checkable grounding and calibrated difficulty. C.2 conditions evolution and reward contracts on answer type, specifying normalization, evidence, and executable verification requirements.

  • C.1 Reasoning-Skill Operators: Unit and scale conversion requires mapping across axis scales, map scales, table units, percentages, or dimensional units, with normalized values passing tolerance.The unit mapping must be stated or visible.
  • C.1 Reasoning-Skill Operators: Rate and change reasoning asks for slopes, percent changes, temporal deltas, growth, decay, or comparisons supported by at least two grounded measurements.The operator targets explicit visual measurements rather than unsupported change claims.
  • C.1 Reasoning-Skill Operators: Interpolation asks for approximate values inside visible intervals or plotted ranges, specifying tolerance and rejecting unbounded extrapolation.This keeps approximation tied to visible support.
  • C.1 Reasoning-Skill Operators: Symbolic-numeric translation converts visual relations into equations, expressions, inequalities, or substitutions whose variables bind to evidence and simplify to the final answer.Other operators include evidence localization, counterfactual changes, distractors with violated constraints, and calibrated rollout pass rates.
  • C.2 Answer-Type and Reward Contracts: Set, list, or ranking prompts request complete condition-satisfying objects, ordered ranks, top-k results, or grouped categories with order and alias normalization.Verification checks completeness and absence of unsupported items.
  • C.2 Answer-Type and Reward Contracts: Boolean or verdict prompts test whether visual-mathematical statements are true, false, satisfiable, increasing, connected, or consistent, requiring supporting constraints or counterexamples.Subjective yes/no prompts are rejected.
  • C.2 Answer-Type and Reward Contracts: Region, span, cell, or bounding evidence contracts require answer support through crops, text spans, table cells, plotted points, graph nodes, or diagram components.Metadata may include coordinates, cell IDs, OCR spans, graph IDs, or crop references.
  • C.2 Answer-Type and Reward Contracts: Program assertion answers are represented as executable assertions over extracted quantities, with reward determined by assertion pass/fail and format-error penalties.Free-form rationales remain suitable for SFT when no deterministic final-answer checker exists.

C.3 From Taxonomy to the 12-Topic Implementation · C.4 HTV-Agent Decider Prompt (verbatim)

The implementation discretizes the taxonomy into 12 pre-bundled topic-specific evolution templates, while HTV-Agent uses a constrained, tool-free decider to resolve solver–verifier conflicts. Its deterministic post-processing rejects unresolved samples instead of forcing them into training datasets.

  • C.3 From Taxonomy to the 12-Topic Implementation: 12 topic tags discretize the visual-route, reasoning-skill, answer-type, and verifier-contract design space into pre-bundled evolution prompt templates.Each tag bundles a recommended reasoning operator, answer-type operator, and matching verifier contract.
  • C.4 HTV-Agent Decider Prompt (verbatim): HTV-Agent realizes conflict-aware adjudication as a constrained LLM call without tools or code execution.The user prompt supplies the question, optional context, choices, solver hypothesis, and verifier report in that order.
  • C.4 HTV-Agent Decider Prompt (verbatim): The decider must summarize the verification, assess its logical quality, and produce the final answer in an exact tagged format.It also outputs confidence from 0–100 and can request reconsideration when neither answer is reliable.
  • C.4 HTV-Agent Decider Prompt (verbatim): When high-quality verification explicitly rejects the solver’s answer, the decider trusts verification; otherwise, it trusts a sound initial answer or preserves agreement.Disagreement with low confidence triggers <require_rethink>true</require_rethink>.
  • C.4 HTV-Agent Decider Prompt (verbatim): The decider receives structured fields for the question, optional context, multiple-choice options, solver answer, and verification report.Context and options are omitted when unavailable or irrelevant.
  • C.4 HTV-Agent Decider Prompt (verbatim): Responses are parsed for final_answer, confidence, and require_rethink tags; unresolved samples are deterministically rejected rather than assigned to VeriEvol-SFT or VeriEvol-RL.This prevents uncertain cases from being forced into either training set.
  • C.4 HTV-Agent Decider Prompt (verbatim): Sampling temperature 0 makes the decider deterministic for identical solver and verifier inputs.The same inputs therefore yield the same adjudication behavior.

D Visual-Dependence Case Studies · E Prompt-Evolution Case Studies

The case studies show that visual dependence and type-aware prompt evolution improve multimodal mathematical reasoning: evolved systems correct image-reading errors, while evolved prompts require multi-step, image-grounded reasoning instead of simple lookup.

  • D Visual-Dependence Case Studies: RL-Evol + Verifier corrects two RL-Origin failures by grounding geometry and chart reasoning in visual evidence.The geometry case extracts an angle measurement from the figure, while the chart case identifies a column using its position relative to a labelled axis.
  • D Visual-Dependence Case Studies: In the geometry example, RL-Origin asserts an unsupported angle, whereas RL-Evol + Verifier extracts the correct measurement directly from the figure.This is one of two side-by-side reasoning-trace case studies referenced in Section 4.4.
  • D Visual-Dependence Case Studies: In the chart example, RL-Origin reads the wrong column label, whereas RL-Evol + Verifier identifies the column by its position relative to a labelled axis.Both examples are drawn from the 12 paired flips identified in the cited passage.
  • E Prompt-Evolution Case Studies: Figure 5 presents three seed–evolved pairs from science-diagram seeds to illustrate the effect of type-aware prompt evolution.The examples make the prompt transformation concrete through before-and-after comparisons.
  • E Prompt-Evolution Case Studies: Before evolution, each prompt is a single-step recognition or lookup query often answerable from text priors or a single label.These seed prompts generally require limited visual reasoning.
  • E Prompt-Evolution Case Studies: After route-specific evolution, prompts require multi-step reasoning grounded in spatial relations, labels, and quantities read from the image.The evolved prompts shift the task from retrieval toward image-grounded reasoning.

F Iterative Construction Algorithm

VeriEvol constructs its SFT and RL corpora iteratively by routing each seed, generating and filtering evolved questions, verifying answers with HTV-Agent, and applying rollout-based eligibility checks for RL inclusion.

  • Iterative construction: HTV-Agent supplies a verified answer, accepted reasoning trace, evidence, verifier, and metadata before the answer-level gate accepts or rejects the candidate.Candidates failing the question-level or answer-level gates are discarded or regenerated.
  • Iterative construction: Accepted candidates are materialized into VeriEvol-SFT, while VeriEvol-RL receives only samples with deterministic or high-confidence checking and rollout pass rates satisfying ρmin < ρj < ρmax.The rollout pass rate is estimated with N student samples.
  • Iterative construction: Each seed is routed using image, OCR, layout, and question features before route-specific operators generate candidate questions.The routing tags are z = (t, y, u, c), and candidates are generated from the evolution operator library O.
  • Prompt evolution: Evolved prompts convert simple recognition seeds into image-grounded, multi-step reasoning tasks while retaining short, deterministically checkable answers.Examples include a Moon-phase duration answer of 14.75 and a microscope-control count of 5.
Loading 2606.23543v1…