PaperlayerView exact revision on arXiv

Source-linked AI summary

TRON: Targeted Rule-Verifiable Online Environments for Visual Reasoning RL

Tianze Yang, Yucheng Shi, Ruitong Sun, Jingyuan Huang, Ninghao Liu, Jin Sun

arXiv:2606.01599v1cs.AI

TL;DR

Visual reasoning RL lacks scalable, verifiable, and controllable training signals beyond fixed curated datasets. TRON generates and exactly verifies fresh visual reasoning rollouts online, and its RL post-training consistently improves three model families across ten external multimodal reasoning benchmarks.

  • Problem

    Visual reasoning RL lacks scalable training signals because each instance requires a visual scene, unambiguous question, calibrated difficulty, and reliable verifier.

  • Method

    TRON provides 520 generator-verifier environments that produce fresh image-question rollouts with exact rewards across five ability buckets and controllable difficulty.

  • Results

    Across ten external multimodal reasoning benchmarks, TRON RL consistently improves all three model families; Qwen3-VL-4B’s average score rises from 52.61 to 55.23.

  • Takeaways & Limitations

    The results support online environments as a practical substrate for visual reasoning training.

  • Takeaways & Limitations

    Synthetic environments may diverge in visual style and language from real benchmark data, and the audit cannot guarantee distributional alignment with every external benchmark.

Abstract

from arXiv · show

Reinforcement learning (RL) for visual reasoning needs scalable, verifiable, and controllable training signals. Existing visual RL post-training trains on static curated datasets, with fixed image-question-answer samples bounded by their collection budget. In this work, we introduce TRON (Targeted, Rule-verifiable Online eNvironments), an online environment substrate: a training rollout is generated on demand by a controllable generator-verifier program that samples a fresh latent visual state, renders an image, asks a question, and exactly verifies the answer. A single run can therefore draw an unbounded stream of fresh instances at the difficulty level required by the current curriculum. The current TRON suite contains 520 environments organized into five ability buckets (spatial, mathematical, diagram, pattern/logic, and counting); the same substrate supports both a single full model trained on all buckets and per-bucket ability-specialist models, with no additional data collection. We also introduce a substrate analysis covering generation reliability, instance and level diversity, cross-environment near-duplicates, and base-model pass rate by difficulty level. RL post-training with METHOD consistently improves performance on ten external multimodal reasoning benchmarks across Qwen3-VL-4B, Qwen2.5-VL-7B, and MiMo-VL-7B-SFT.

1. Introduction

TRON addresses the limits of static visual reasoning datasets by providing a procedural online substrate that generates fresh, difficulty-controlled instances with exact, rule-verifiable rewards. Its 520 environments target five ability buckets and support both broad and specialist RL training without additional data collection.

  • Motivation and approach: TRON replaces finite image-question-answer datasets with procedural environments that generate fresh visual reasoning instances and exact rewards during training.This avoids dataset exhaustion and allows training data to be controlled by the model’s targeted reasoning needs.
  • TRON substrate: Each environment samples a latent visual state, renders an image, asks a question, and verifies the answer with a deterministic rule rather than human labels or an LLM judge.The answer is solved before image rendering, enabling exact matching and noise-free rewards.
  • TRON substrate: 520 generator-verifier programs span five ability buckets: spatial, mathematical, diagram, pattern, and counting.The environments target operations including chart aggregation, cube rotation, occluded counting, visual analogy, and graph search.
  • Controllability and diversity: TRON provides diversity across reasoning mechanisms, visual instances, and progressively harder levels through distinct environments, varied layouts and distractors, and difficulty ladders.The same program can re-enter at different levels, while generation remains controllable as the model improves.
  • Substrate audit: The substrate audit checks rendering and verifier correctness, instance and level diversity, cross-environment near-duplicates, and base-model pass rates by difficulty.This audit addresses the risk that superficial variation, template collapse, or verifier errors produce unhelpful training signals.
  • Training scope: TRON supports training one full model across all buckets or separate per-bucket ability specialists without additional data collection.The paper also uses the shared substrate to study ability transfer.

2. Related Work

Prior work established RLVR and visual RL post-training as approaches for improving reasoning in language and vision-language models. These methods use rule-based rewards but train on static curated data.

  • RLVR and visual RL post-training: RLVR has become a central recipe for improving language-model reasoning, with GRPO and DAPO refining the optimization.The passage identifies RLVR and these optimization variants as prior work.
  • RLVR and visual RL post-training: Visual RL post-training extends RLVR to mathematical, spatial, grounding, counting, and multimodal reasoning tasks.The cited work applies RLVR-style training across several vision-language reasoning capabilities.
  • RLVR and visual RL post-training: These methods share rule-based rewards with TRON but train on static curated data.The passage explicitly contrasts the prior methods’ static curated training data with the present work.

3. TRON Environments

TRON defines online visual-reasoning environments as generator–verifier programs that produce fresh, exactly checkable instances across controllable difficulty levels. Its 520 environments span five reasoning abilities, with diversity arising within environments, across levels, and across mechanism classes.

  • Environment definition: Each TRON environment pairs a task-state space and ten difficulty levels with a deterministic generator that renders an image, question, and ground-truth answer.The level set is L = {0, 1, . . . , 9}.
  • Environment definition: TRON removes static-dataset limits: training size depends on compute, successor models cannot absorb a fixed item set, and difficulty remains curriculum-controlled per sample.These properties distinguish TRON from static image–question–answer triples used in prior visual RL datasets.
  • Suite organization: 520 environments cover five ability axes: spatial, math, diagram, pattern/logic, and counting.Each environment is a Python program targeting one reasoning mechanism.
  • Difficulty and diversity: Difficulty changes the mechanism within an environment, scaling deduction steps in angle-chase and chart complexity through series, time points, and stacking.Angle-chase ranges from a one-step triangle sum at ℓ=0 to a four-step composite chain over parallels at ℓ=9.
  • Difficulty and diversity: Instance diversity comes from randomized latent states and question stems, level-dependent mechanism shifts, and changes in mechanism class across environments.Randomized latent-state attributes include values, layout, palette, and label names.

4. RL Post-Training

TRON enables RL post-training from an unbounded stream of freshly generated, exactly verifiable instances rather than a fixed corpus. Its rollout-coupled curriculum and configurable sampler support both full-model and ability-specialist training, using deterministic verifier rewards without an LLM judge.

  • Data generation: Each sampling step selects an environment, level, and fresh seed to generate a new image-question-answer instance online, so training steps never repeat an instance.Training data comes from the TRON substrate rather than offline epochs or a fixed corpus.
  • Data generation: Curriculum promotion advances environments from level ℓ to ℓ+1 when recent verifier accuracy crosses a threshold, while a sliding window retains lower-level skills.The curriculum tracks recent verifier accuracy over a target number of graded trajectories.
  • Data generation: The same 520-environment substrate trains either one model across all buckets or per-bucket ability specialists through a sampler switch, without extra data.The configuration supports both full-model and specialist training modes.
  • Training recipe: The DAPO-style objective uses prompt-grouped advantages, n = 8 responses per prompt, deterministic verifier scores, and rewards based on answer and wrapper-format correctness.Verifier-side numeric or symbolic normalization is applied when appropriate, and no LLM judge computes the reward.

5. Experiments

Experiments validate TRON’s environment quality, difficulty axis, and diversity, then show consistent gains across three VLMs and external reasoning benchmarks. Ability-specialist results further indicate transfer follows underlying capabilities across visual formats, not visual-format alignment alone.

  • 5.1 Environment audit: The 520-environment audit covers 8,320 probes across four levels and four seeds, achieving 99.1% success while examining quality, difficulty, and diversity.The audit targets malformed probes, ineffective difficulty scaling, within-environment repetition, and cross-environment near-duplicates.
  • 5.1 Environment audit: Qwen3-VL-4B-Instruct’s mean pass rate falls from 72.8% at ℓ=0 to 41.3% at ℓ=9, validating a substantive difficulty axis.Intermediate levels score 59.9% at ℓ=3 and 48.0% at ℓ=6.
  • 5.2–5.3 Main results: TRON training raises average scores from 52.61 to 55.23 for Qwen3-VL-4B, 40.85 to 43.35 for Qwen2.5-VL-7B, and 63.37 to 66.50 for MiMo-VL-7B-SFT.Improvements span external benchmarks covering mathematics, spatial reasoning, charts, scientific figures, visual puzzles, and logical reasoning.
  • 5.2–5.3 Main results: MM-HELIX improves for every backbone, while SpatialEval gains substantially for Qwen2.5-VL-7B and MiMo-VL-7B-SFT, with transfer spanning multiple reasoning families.The results are presented as evidence that diverse environments transfer practiced operations rather than memorized benchmark templates.
  • 5.4 Ability specialists: Specialists transfer across visual formats when underlying capabilities match, including Math gains of +20.0 on MM-HELIX maze and +11.6 on WeMath position.Spatial also transfers to WeMath angles/length (+12.6) and route map (+7.1), while Count transfers to MathVerse volume (+7.8).

6. Discussion

TRON’s live, parametric environments preserve fresh training instances and an advancing curriculum while providing exact rewards. Converting rollouts into a static snapshot would lose these benefits because it is bounded, exhaustible, and unable to advance its curriculum after collection.

  • Live environments: Live parametric environments provide exact rewards for every sampled instance while supplying fresh latent states and renderings.Freshness reduces memorization pressure during training.
  • Live environments: Their per-environment difficulty ladders let the curriculum advance on demand.Difficulty progression remains controllable during training rather than being fixed at collection time.
  • Static rollouts: A static parquet snapshot is bounded, exhausts once seen, and cannot advance its curriculum after collection.Pre-generating rollouts therefore forfeits both fresh instances and an advancing curriculum.

7. Conclusion

TRON provides an online substrate with 520 generator-verifier programs across five ability buckets, producing fresh visual reasoning rollouts with exact rewards and local difficulty control. An accompanying audit and RL post-training results support its practicality for visual reasoning training.

  • TRON substrate: TRON comprises 520 generator-verifier programs organized into five ability buckets for visual reasoning RL.The substrate produces fresh image-question rollouts with exact rule-based rewards and a local difficulty ladder.
  • Substrate audit: The substrate includes an audit measuring quality, diversity, and difficulty before training.This audit evaluates the generated environments before RL post-training.
  • Training results: RL post-training with TRON-DAPO consistently improves performance across ten external multimodal reasoning benchmarks and three multimodal models.The models are Qwen3-VL-4B, Qwen2.5-VL-7B, and MiMo-VL-7B-SFT.

8. Limitations

TRON’s synthetic environments may not align with real-world benchmark distributions, while several analyses depend on author-chosen difficulty and diversity settings. Its ability buckets also overlap, so they represent coarse categories rather than strictly isolated capabilities.

  • Distributional alignment: Synthetic visual styles and language may diverge from real benchmark data, especially in photographic or dense scientific perception domains.The audit detects internal quality failures but cannot guarantee distributional alignment with every external benchmark.
  • Difficulty control: Difficulty levels are author-chosen generator parameters, and individual environments need not vary monotonically or by uniform step sizes.Aggregate base-model pass rate nevertheless falls monotonically across levels.
  • Diversity analysis: Diversity grades depend on hand-chosen normalizations, component weights, duplicate thresholds, convex-combination weights, and A–D cutoffs rather than learned criteria.The settings target an informative histogram for the current suite, so alternative definitions of sufficient diversity would shift grade boundaries.
  • Ability buckets: The five ability buckets overlap because many environments exercise multiple mechanisms, making the labels a coarse partition rather than a clean factorization.For example, chart tasks may require multi-step reasoning, while graph-algorithm tasks may require counting.

9. Ethical Considerations · A. Fine-Grained Environment Coverage · B. Qualitative Environment Examples

TRON uses procedurally generated, rule-verifiable environments without newly collected data or personally identifiable information, while acknowledging benign intended uses and potential misuse. Its coverage spans representative generator–verifier programs with multiple seeds and difficulty levels, illustrated by qualitative examples pairing rendered instances, prompts, and verified answers.

  • 9. Ethical Considerations: All TRON training environments are generated procedurally, with no newly collected or scraped data and no personally identifiable information.Evaluation uses public multimodal reasoning benchmarks and open-source vision-language models under their respective licenses.
  • 9. Ethical Considerations: TRON targets visual reasoning improvements for benign applications including educational tools, assistive vision, scientific figure understanding, and structured-visual analysis.The passage also notes that improved reasoning could be repurposed for surveillance or other sensitive monitoring tasks.
  • 9. Ethical Considerations: Large language models assisted with language polishing and coding, while the authors retained responsibility for ideas, methodology, experiments, analyses, and claims.Assistance included writing and debugging procedural environment generators and analysis code.
  • A. Fine-Grained Environment Coverage: Table 6 provides representative, non-exhaustive coverage of the suite, where each environment is a generator–verifier program with multiple seeds and difficulty levels.The table expands the high-level suite composition reported in Table 1.
  • A. Fine-Grained Environment Coverage: The fine-grained environment listing is representative rather than exhaustive, reflecting the suite’s organization around reusable generator–verifier programs.This framing distinguishes listed examples from the complete environment inventory.
  • B. Qualitative Environment Examples: Figures 3–7 sample the 520 training environments across ability buckets, generator families, and Levels 0, 5, and 9.Rows show two generator families, while columns show the three levels.
  • B. Qualitative Environment Examples: Each qualitative panel pairs a rendered visual instance with its task prompt and verified answer, exposing both sides of the environment contract.Only repeated answer-format boilerplate is omitted for readability.

C. Full-Model Training Details

The three full TRON runs use a shared DAPO-style recipe, online sampler, and curriculum mechanism, with backbone-specific settings adapted to a four-GPU H100 80 GB node. Curriculum levels advance when recent verifier-scored rollouts reach the specified accuracy threshold.

  • Training setup: Three full TRON runs share the DAPO-style recipe, online environment sampler, and curriculum-promotion mechanism.Backbone-specific batch sizes and vLLM memory settings are adjusted for a four-GPU H100 80 GB node.
  • Curriculum promotion: n = 8 rollouts per prompt are scored 0/1 by the verifier at each training step.The curriculum manager uses the four most recent rollout groups for each environment’s current level.
  • Curriculum promotion: 4 × 8 = 32 scored trajectories form the sliding curriculum buffer for each environment level.The buffer is reset after promotion.
  • Curriculum promotion: ≥0.80 mean buffer accuracy advances an environment from level ℓ to level ℓ + 1.This promotion rule is identical across all three full-model runs.

D. Ability-Specialist Training Details

Ability-specialist runs isolate one TRON ability bucket while retaining the full model’s training stack and rule-based rewards. All five specialists improve monotonically on bucket validation, with Count capped at step 100 to limit overfitting and the others running to step 200.

  • Training setup: Each specialist samples one ability bucket, uses the DAPO/GRPO stack and rule-based reward, and starts from Qwen3-VL-4B-Instruct.Separate checkpoints, validation generations, and curriculum state are maintained for each ability.
  • Training setup: 100 steps caps Count because its 30 environments would otherwise receive disproportionately many updates compared with the other buckets’ 104–144 environments.The cap keeps per-environment update counts roughly comparable and reduces overfitting risk.
  • Training setup: 3200 generated prompts per epoch use batch sizes of 32, eight rollouts per prompt, four GPUs with tensor parallelism 4, and a 5 × 10^-6 actor learning rate.Rollouts use temperature 1.0 and maximum prompt and response lengths of 8192; clipping uses [0.2, 0.28] and actor KL loss coefficient 0.005.
  • Training dynamics: All five specialists improve monotonically on bucket validation and advance curriculum difficulty, with Count stopping at step 100 while the others run through step 200.Figure 8 reports validation-accuracy gain over step 0 and mean curriculum difficulty across audited levels.

E. Quality Gate Implementation · F. Diversity Audit Details · G. Example Environment Implementation

The appendices specify model-free quality gates, formal diversity-audit signals and thresholds, and a concrete clock-angle environment implementation. Together, these details define how TRON filters broken rollouts, measures diversity and duplication, and constructs level-controlled visual reasoning tasks.

  • E. Quality Gate Implementation: Quality checking uses four gates: generation success, image validity, question/answer completeness, and canonical-answer verification.Only probes whose generation call completes successfully enter the remaining three checks; image validity requires size ≥64 px, grayscale std ≥2.0, and foreground ratio [0.001, 0.98].
  • E. Quality Gate Implementation: The gates are deliberately syntactic and model-free, catching broken rendering, missing fields, and verifier failures before RL training.The verifier check detects arbitrary-string acceptance and rejection of the canonical correct answer.
  • F. Diversity Audit Details: Seed diversity combines image pHash distance, normalized question-template variety, and answer-bucket entropy with weights (0.45, 0.25, 0.30).Signals near 0 indicate seed collapse on the corresponding visual, wording, or answer axis.
  • F. Diversity Audit Details: Level diversity measures cross-level pHash distance, template-set Jaccard distance, and foreground-ratio shifts with weights (0.55, 0.30, 0.15).If all three signals approach 0, the difficulty axis changes only hidden metadata.
  • F. Diversity Audit Details: Cross-environment duplication requires simultaneous visual, pixel, and prompt similarity using thresholds (20, 8, 0.50).The audit compares mean pHash Hamming distance, thumbnail-pixel mean absolute error, and maximum normalized-template token-Jaccard similarity.
  • F. Diversity Audit Details: Overall diversity uses weights (0.55, 0.35, 0.10) for seed, level, and cross-environment terms, with A/B/C/D thresholds (0.65, 0.50, 0.35).These constants support reporting histograms and are not learned or used by the RL trainer; raw per-signal values are also retained for inspection.
  • G. Example Environment Implementation: The example Math-bucket clock environment increases question difficulty through a level ladder, renders sampled clock states with matplotlib, and returns question, answer, and image.It supports clock-reading, hand-angle, future-angle, and overlap-count questions; numerical verification uses absolute tolerance 0.001.
Loading 2606.01599v1…