Source-linked AI summary
Decouple to Generalize: Context-First Self-Evolving Learning for Data-Scarce Vision-Language Reasoning
Tingyu Li, Zheng Sun, Jingxuan Wei, Siyuan Li, Conghui He, Lijun Wu, Cheng Tan
TL;DR
DoGe addresses costly multimodal data needs and limitations in synthetic and self-rewarded reasoning by separating context learning from task solving and evolving its training curriculum. Across seven benchmarks, it stably outperforms the baseline, with reported average gains for both model sizes.
Problem
RL post-training is difficult to scale in specialized VLM domains because high-quality multimodal data is costly to annotate and existing synthetic-data and self-rewarding methods have distribution or alignment limitations.
Source: §1 · p. 1Method
DoGe trains a Thinker to analyze question-masked context, scores its analyses through a Solver, then applies learning to original tasks and evolves data through a seed-problem curriculum.
Source: §3.1 · p. 4, §3.2 · p. 5, §3.3 · p. 6Results
Across seven benchmarks, DoGe stably outperforms the baseline, with average performance increases of 5.7% for 3B models and 2.3% for 7B models.
Source: §4.2 · p. 9, §4.3 · p. 10, §5 · p. 11Takeaways & Limitations
The authors present DoGe as a pathway toward self-evolving LVLMs, reporting improved generalization and greater reinforcement-learning exploration.
Source: §1 · p. 2, §5 · p. 11
Abstract
from arXiv · showhide
Recent vision-language models (VLMs) achieve remarkable reasoning through reinforcement learning (RL), which provides a feasible solution for realizing continuous self-evolving large vision-language models (LVLMs) in the era of experience. However, RL for VLMs requires abundant high-quality multimodal data, especially challenging in specialized domains like chemistry, earth sciences, and multimodal mathematics. Existing strategies such as synthetic data and self-rewarding mechanisms suffer from limited distributions and alignment difficulties, ultimately causing reward hacking: models exploit high-reward patterns, collapsing policy entropy and destabilizing training. We propose DoGe (Decouple to Generalize), a dual-decoupling framework that guides models to first learn from context rather than problem solving by refocusing on the problem context scenarios overlooked by synthetic data methods. By decoupling learning process into dual components (Thinker and Solver), we reasonably quantify the reward signals of this process and propose a two-stage RL post-training approach from freely exploring context to practically solving tasks. Second, to increase the diversity of training data, DoGe constructs an evolving curriculum learning pipeline: an expanded native domain knowledge corpus and an iteratively evolving seed problems pool. Experiments show that our method consistently outperforms the baseline across various benchmarks, providing a scalable pathway for realizing self-evolving LVLMs.
1 Introduction
RL post-training is difficult to scale in specialized VLM domains where high-quality multimodal data is costly and existing synthetic-data and self-rewarding methods have limitations. DoGe separates context learning from task solving in a two-stage RL loop.
1 Introduction
- Costly manual annotation of domain-specific multimodal knowledge and reasoning data makes RL post-training difficult to scale in specialized domains.The paper names chemistry, earth science, and multimodal mathematics as examples.Source: §1 · p. 1
- Synthetic questions can be repetitive, while self-rewarding objectives may not align well with specialized-domain reasoning.The paper says these approaches struggle to sustainably improve specialized-domain performance.Source: §1 · p. 1
- DoGe trains a learnable Thinker to analyze context before a frozen Solver applies that analysis, using Solver pass rate as reward before GRPO annealing.This forms a two-stage learning-application loop.Source: Figure 1 · p. 2, §1 · p. 2
- Question-answer supervision overlooks contextual information and can incentivize reward shortcuts, causing entropy collapse and reduced exploration.The paper links these effects to limits on generalizable performance in specialized domains.Source: §1 · p. 2
2 Related Work
Related work covers multimodal post-training methods, including RL fine-tuning and reward design, alongside approaches to self-evolution through parameter updates, in-context learning, and domain-specific frameworks.
2.1 Vision Language Model Post-Training
- Vision-language post-training spans instruction tuning, modular adaptation, RL fine-tuning, and reward design, with RLVR using objective correctness signals for alignment and verifiability.Examples include synthetic instruction data, multimodal adapters, and self-rewarding RL.Source: §2.1 · p. 3
2.2 Self-Evolving Vision Language Model
- Self-evolution work includes parameter-update methods, training-free in-context learning, and domain-specific applications such as image-quality assessment and medical imaging.The surveyed approaches span software agents, self-reflection, unified learning, and external textual knowledge libraries.Source: §2.2 · p. 3
3 Method
DoGe separates contextual learning from task application in a two-stage RL framework and builds an evolving curriculum from multimodal domain data and seed problems.
3 Method
- DoGe frames VLM self-evolution as two connected modules, Learning and Application, built on multimodal GRPO.The method section formalizes task objectives before describing the approach and data synthesis.Source: §3 · p. 3
3.1 Preliminaries
- Each reasoning sample contains multimodal context, a question, and a ground-truth answer; DoGe uses a question-masked Thinker for context analysis and a Solver for the original task.The Solver conditions on the context, question, and Thinker analysis; explicit question and answer cues are removed from the Thinker input.Source: §3.1 · pp. 3–4
3.2 Decoupling Cognitive Process: 2-Stage RL
- In Learning from Context, the Thinker explores question-masked scenarios while a frozen Solver tests each analysis on the original problem and supplies an accuracy-based reward with a format bonus.The format reward is binary and checks for the specified thinking and final-answer format.Source: §3.2 · p. 5
- In Learning from Application, the Thinker is optimized on original problems using answer correctness and a format bonus, and its updated parameters initialize the next-round Solver.Both stages use GRPO, and the update enables the system to self-bootstrap across rounds.Source: §3.2 · pp. 5–6
3.3 Iterative Curriculum Data Synthesis
- The curriculum pipeline gathers raw multimodal data, enriches sparse samples, synthesizes reasoning question-answer pairs, and updates a seed pool with occasionally solvable problems.The pool can be initialized with synthesized or high-quality seed data; the paper gives 0.1–0.3 pass rate as an example selection range.Source: §3.3 · p. 6
4 Experiment
DoGe is evaluated across three specialized data domains and seven benchmarks using an iterative data-synthesis pipeline and two-stage RL training. It outperforms the baseline across benchmarks, while ablations report higher entropy and more stable multi-round learning.
4.1.1 Data Preparation
- Raw multimodal data from mathematics, chemistry, and Earth science is enriched into a knowledge pool, while an evolving seed pool supplies problems filtered by model accuracy across iterations.Information-poor samples receive analysis reports; the seed pool is initialized differently for chemistry than for Earth science and multimodal mathematics.Source: §4.1.1 · pp. 6–7
4.1.2 Benchmarks
- Seven benchmarks assess scarce-domain reasoning, general visual reasoning, and hallucination, spanning Earth science, chemistry, mathematics, interdisciplinary tasks, visual reasoning, and hallucination.The benchmark set includes MSEarthMCQ, ChemBench, MathVista, MathVision, MMMU, MMStar, and HallusionBench.Source: §4.1.2 · pp. 7–8
4.1.3 Training Details
- Training uses Qwen2.5VL-3B and 7B models, GRPO in both DoGe stages, and distinct sampling and training-step settings across the stages.Experiments use 8 A100 GPUs; the 3B stage settings include 100 Thinker steps and 150 stage-two steps, while both 7B stages use 150 steps.Source: §4.1.3 · p. 8
4.2 Main Results
- 3B and 7B models improved by 5.7% and 2.3% on average across seven benchmarks, respectively; DoGe also improved HallBench by 2.0%.The authors report gains across all benchmark tests, including specialized-domain and general-domain reasoning.Source: §4.2 · p. 9
4.3 Ablation study on Our method
- DoGe maintains higher policy entropy and exploration during training than the baseline, with Figure 4 comparing average entropy across training.The authors attribute this to training Thinker on context before stage-two GRPO annealing.Source: §4.3 · p. 9, Figure 4 · p. 10
- Across iterations, DoGe stably outperforms naive GRPO, especially on reasoning-intensive tasks, while the baseline fluctuates and can encounter reward hacking with low-quality data.The authors also report that DoGe stabilizes multi-round self-evolving learning.Source: Table 2 · p. 9, §4.3 · p. 10
4.4 Analysis on Data Synthesis
- Synthetic and manually annotated math data share similar question contexts, but the synthetic data covers a wider range of question-surface designs.Figure 5 visualizes embedded question text with t-SNE; the authors connect limited image and question diversity in manual data to repeated actions and reduced entropy.Source: §4.4 · p. 10, Figure 5 · p. 10
5 Conclusion
DoGe separates contextual learning from applying that learning to solve problems, using GRPO to train the two-stage process. The paper concludes that this approach improves generalization and exploration while reducing sensitivity to data quality and stabilizing training.
5 Conclusion
- DoGe decouples cognitive processes into two RL stages, first learning from context and then applying that knowledge to reasoning.The conclusion describes Thinker analysis feeding Solver problem solving, with quantitative rewards used to update the model.Source: §5 · p. 11
A RLVR for VLMs
- GRPO samples multiple responses per task and uses rule-based rewards with group-relative advantages instead of PPO’s value model.Its clipped objective and KL divergence mechanism are used to stabilize updates and limit deviation from the old or reference policy.Source: A RLVR for VLMs · p. 15
- The GRPO description uses ϵ for numerical stability, while DoGe follows DAPO in decoupling εl and εh and uses β to control KL regularization.The paper says decoupled ε values help stabilize entropy and enhance exploration.Source: A RLVR for VLMs · p. 15
B.1 Training Hyperparameter
The appendix reports the compute setup and points to separate parameter tables for the two model sizes.
B.1 Training Hyperparameter
- Experiments use 8 A100 GPUs in total.The GRPO baseline shares DoGe stage-two hyperparameters.Source: §B.1 · p. 15
- The GRPO baseline shares DoGe stage-two hyperparameters, while Solver is not learnable and uses only vLLM sampling parameters.Source: §B.1 · p. 15
- Tables 3 and 4 give the detailed experimental parameters for the 3B and 7B base models, respectively.Source: §B.1 · p. 15
B.2 Prompt Template for GRPO Training
The templates separate context analysis from answering and provide prompts for two routes to synthesizing problems. The appendix also labels hyperparameter tables for the 3B- and 7B-series experiments.
B.2 Prompt Template for GRPO Training
- The appendix labels one hyperparameter table for the 3B series and another for the 7B series.Source: Table 3 · p. 16, Table 4 · p. 16
- The stage-one prompt asks the model to analyze context with the question removed and produce a report to guide solving the original problem.The template supplies that context through {information}.Source: §B.2 · p. 16
- The stage-one output is an internal monologue enclosed in think tags, with the Deep-Thinking Report ending at the closing tag.Source: §B.2 · p. 16
- The stage-two template includes the question and Thinker analysis, requests a boxed answer, and specifies a Python-list format for multiple-choice responses.The placeholders identify the question as {Question} and the analysis as {deepthought}.Source: §B.2 · pp. 15, 17
- A second stage-two template includes the question and answer-format instructions but omits the expert-analysis block.Source: §B.2 · p. 17
B.3 Prompt Template for Problem Synthesis
- Problem synthesis has two paths: sample the Multimodal Knowledge Pool or sample the Seed Problem Pool to create variants.The passages name these Synthesis 1 and Synthesis 2; the {rules} component is optional and varies with the generation objective.Source: §B.3 · p. 17
- The knowledge-pool prompt asks a question designer to create problems from supplied information and return questions and answers in JSON.Source: §B.3 · p. 18
- The seed-pool prompt asks for challenging variants of reference multimodal problems and returns the generated problems and answers in JSON.Source: §B.3 · p. 18
C.1 Collected Raw Data Examples
The appendix shows raw examples from multimodal mathematics, chemistry, and earth science, followed by synthetic problems whose questions are removed to retain context. The examples include chemical descriptors, an Earth-observation image description, and masked-question cases.
C.1 Collected Raw Data Examples
- The collected raw data examples span multimodal mathematics, chemistry, and earth science.Source: §C.1 · pp. 18–19
- The chemistry example gives a molecule’s SMILES, IUPAC name, formula, molecular weight, and other descriptors.Source: §C.1 · p. 19
- The earth-science example describes an aerosol image from NASA’s EPIC camera aboard the NOAA DSCOVR spacecraft.Source: §C.1 · p. 20
C.2 Synthetic and Masked Problems
- Synthetic questions are generated with Gemini2.5-flash-lite, then masked so only contextual content remains.Source: §C.2 · p. 20
- The masked examples retain context for a cloud-classification question, a molecule question, and a geometry question.The cloud example retains the context phrase “cloud classification confidence meaning”; the geometry example retains the diameter, center, and angle information.Source: §C.2 · pp. 21–22
D Case Study
The case study compares DoGe, the base model, and a baseline using responses to a triangle-number problem. The passages show different solution approaches, including one that selects an arbitrary value for an unknown.
D Case Study
- The case study compares output cases from DoGe, the base model, and the baseline.Source: D Case Study · p. 22
- A separate response uses the visible values 1, 6, 7, and 3 to set each triangle-side sum to 14 and solve for two unknowns.The supplied excerpts introduce inconsistent unknown labels across these response examples.Source: D Case Study · pp. 23–24
- One response treats the three side sums as equal, obtains x = 2 and y = 0, and then states that their sum is needed.The excerpt does not give a numeric value for that final sum.Source: D Case Study · pp. 24–25
- Another response derives the inked-number sum as y + z = 2x + 6, then assumes x = 0 because it says any consistent value may be chosen.It subsequently reports y + z = 6.Source: D Case Study · p. 25