SPARK: Synergistic Policy And Reward Co-Evolving Framework

1 INTRODUCTION

• RLVR addresses objective tasks with explicit verifiers, leaving open-ended helpfulness and safety unaddressed; RLHF can handle subjective tasks.RLHF distills human or synthetic preferences into a learned reward model to guide policy optimization.Source: §1 · p. 1
• SPARK recycles RLVR rollouts and correctness data to train the model itself as a generative reward model using pointwise, pairwise, and reflection objectives.The objectives teach the model to assess correctness, compare responses, and fix incorrect responses.Source: §1 · p. 2
• SPARK co-evolves reward accuracy and policy quality, and supports test-time self-reflection without a separate reward-model training loop.The paper describes continually sampled, on-policy reward data and a feedback loop in which better reward accuracy yields better policy gradients and higher-quality rollouts.Source: §1 · pp. 2–3
• SPARK-VL-7B gains 9.7% on 7 reasoning benchmarks, 12.1% on 2 reward benchmarks, and 1.5% on 8 general benchmarks over baselines.The framework also reports improvements with SPARK-VL-32B and SPARK-7B.Source: §1 · pp. 2–3

Reinforcement Learning with Verifiable Reward

• RLVR, including GRPO-based approaches, has been applied to reasoning-intensive tasks and extended to perception and agent domains.This work adopts a GRPO-based algorithm to jointly enhance policy reasoning and reward judgment in one model.Source: §2 · p. 3

Reinforcement Learning from Human Feedback

• RLHF typically trains an independent reward model from human or teacher-generated preference data, then uses it to guide policy optimization.The cited passage identifies policy and reward development in isolation as a limitation of existing paradigms.Source: §2 · p. 3
• SPARK instead treats policy and reward as complementary capabilities that evolve jointly, reinforcing each other.The paper presents this as a way to break the isolation between policy and reward models.Source: §2 · p. 3

3.1 SPARK TRAINING WITH VERIFIABLE REWARD

• For each VQA sample—or prompt alone for language-only LLMs—SPARK generates answer groups, scores final answers with verifiable rewards, and computes normalized advantages for policy updates.The advantage uses the group mean reward and standard deviation, with a small constant for numerical stability.Source: §3.1 · pp. 3–4
• SPARK maximizes expected verifiable reward while regularizing the policy toward a reference policy, with λ controlling the KL-divergence.The authors characterize this as efficient and stable when outputs are objectively verifiable.Source: §3.1 · p. 4

3.2 SPARK ON-POLICY DATA GENERATION

• On-policy reward values create pointwise correctness judgments, pairwise answer comparisons, and reflection samples that feed back into the unified model.Incorrect answers are fed back for reflection and refinement; pointwise and pairwise supervision can also use reasoning traces instead of final answers.Source: §3.2 · pp. 4–5

3.3 TEST TIME SCALING WITH SELF-REFLECTION

• At test time, the model generates an answer, judges its reasoning and prediction, and accepts or iteratively revises the solution until it judges an answer correct.The illustrated mathematical case covers reasoning, judgment, and reflection; the figure omits some content.Source: §3.3 · p. 5, Figure 3 · p. 6

4.2 RESULTS ON MATHEMATICAL AND REWARD BENCHMARKS

• 9.7% on mathematical benchmarks and 12.1% on reward benchmarks yield a 10.6% overall gain for SPARK-VL-7B over Qwen2.5-VL-7B.SPARK-7B gains 5.4% on mathematical benchmarks, 12.0% on reward benchmarks, and 7.1% overall; SPARK-VL-32B gains 7.8% and 3.0% on mathematical and reward benchmarks, respectively.Source: §4.2 · pp. 7–8, Table 2 · p. 7, Table 3 · p. 7
• Policy-Only training favors mathematical reasoning, Reward-Only better enhances judgment, and combining both sources surpasses either single-source variant.SPARK adds co-evolution and reflection-driven data generation, which the authors say further improves the integration of reasoning and judgment.Source: §4.2 · p. 7
• RewardBench2 (+3.5%) and VL-RewardBench (+15.0%) improve despite training reward data being mathematics-specific, indicating judgment generalization beyond mathematics.The two benchmarks span diverse domains.Source: §4.2 · p. 7

4.3 RESULTS ON GENERAL BENCHMARKS

• SPARK-VL-7B improves by 2.2% on MMBench, 3.2% on MMStar, and 1.8% on MMVet, averaging 1.5% across eight general benchmarks.These results are reported as gains beyond mathematics and judgment tasks.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 JUDGMENT ACCURACY ANALYSIS

• SPARK consistently outperforms the Qwen2.5-VL-7B baseline in self-judgment recall, precision, and F1, with pronounced gains on MMK12, MathVerse, and WeMath.The evaluation has the model reason over seven mathematical datasets and judge its own prediction.Source: §4.4 · pp. 8–9, Figure 4 · p. 8

4.5 ABLATION STUDIES

• Reflection-based test-time scaling reduces Qwen2.5-VL-7B performance across benchmarks, gives only marginal gains with GRPO+Policy, and yields substantially larger gains with SPARK.The reported drop is attributed primarily to weak judgment and reflection skills causing frequent misjudgments, especially with more reasoning rounds.Source: §4.5 · p. 9, Table 5 · p. 9
• Combining answer-based and CoT-based reward data scores 57.1, above answer-only at 55.3 and CoT-only at 56.2.The authors interpret the combined sources as complementary training signals.Source: §4.5 · p. 9, Table 6 · p. 9
• SPARK avoids a separate reward-model training stage and repeated reward-model inference, using on-policy verifiable rewards to reduce data and computational costs.The paper reports that RM-based RL requires large-scale human or teacher annotations and doubles GPU usage during optimization.Source: §4.5 · pp. 9–10, Table 7 · p. 9

• SPARK recycles RLVR rollouts into judgment and reflection objectives, allowing one model to act as both policy and generative reward model.The framework is described as efficient, on-policy, and stable.Source: §5 · p. 10
• The co-evolving feedback loop links improved reward accuracy with stronger reasoning, while stronger reasoning further refines reward judgment.The conclusion connects this process to self-reflection and stability.Source: §5 · p. 10
• The paper reports substantial improvements on mathematical, reward, and general benchmarks and presents SPARK as a scalable, generalizable RL solution.Source: §5 · p. 10

• Section A.1 summarizes the models, datasets, and benchmarks used in SPARK experiments.Source: OUTLINE · p. 15
• Section A.2 provides evaluation prompt templates and on-policy prompts for reward and reflection data generation.Source: OUTLINE · p. 15
• Sections A.3 and A.4 cover related work on self-reward and self-reflection, then illustrate SPARK reasoning cases on mathematical and reward benchmarks.Source: OUTLINE · p. 15

• SPARK-VL-7B, SPARK-VL-32B, and SPARK-7B use Qwen2.5-VL-7B, Qwen2.5-VL-32B, and Qwen2.5-7B backbones, respectively.The variants cover multimodal and text-only settings at different scales.Source: §A.1.1 · p. 15, Table 8 · p. 15
• The multimodal comparisons include VL-Rethinker, MM-Eureka, OpenVLThinker, Vision-R1, and R1-OneVision; language comparisons include Qwen2.5-Math-Instruct, Simple-RL-Zero, Eurus-2-PRIME, and Open-Reasoner-Zero.Source: §A.1.1 · p. 15
• Larger multimodal baselines VL-Rethinker-32B, MM-Eureka-32B, and Vision-R1-32B serve as scale references, with model sources and parameter scales tabulated in the supplementary material.Source: §A.1.1 · pp. 15–16, Table 8 · p. 15

A.1.2 BENCHMARKS

• The evaluation spans mathematical reasoning, reward-related judgment, and general multimodal understanding benchmarks.The setup is intended to cover specialized domains as well as broader tasks.Source: §A.1.2 · p. 16
• Seven multimodal and six text-only datasets assess numerical, symbolic, logical, and competition-style reasoning.The multimodal set includes MathVista and MathVision; the text-only set includes AIME24, GSM8k, and MMLU STEM.Source: §A.1.2 · p. 16
• RewardBench2 and VL-RewardBench assess correctness and quality judgments, with mathematical subsets reported separately.Separate subset results are used to analyze the interplay between reward judgment and mathematical reasoning.Source: §A.1.2 · p. 16
• Eight general multimodal benchmarks range from knowledge-intensive question answering to real-world perception challenges.The suite includes MMMU, MMVet, MMBench, MMStar, POPE, ScienceQA, SeedBench, and RealWorldQA.Source: §A.1.2 · p. 16
• A complete statistical summary of all benchmarks is provided in Table 9.Source: §A.1.2 · p. 16

A.1.3 TRAINING DATA PREPARATION

• On-policy reward and reflection data are generated during policy RFT using verifiable reward signals rather than teacher-generated rewards or manually annotated traces.This approach is described as greatly reducing cost.Source: §A.1.3 · p. 16
• SPARK-VL training uses image-question-answer triples, while SPARK-7B uses Simple-RL-Zero-25k, without manually annotated reward data or reflection traces.SPARK-VL-7B uses 19k randomly sampled ViRL-39k instances, and SPARK-VL-32B uses 24k difficulty-filtered instances.Source: §A.1.3 · p. 17

A.2 PROMPTS

• Mathematical evaluation prompts separate reasoning from the final answer, while reward benchmarks use their official prompts.The math prompt asks for the final value in \box{} rather than imposing rigid reasoning tags; VL-RewardBench follows its original prompt format.Source: §A.2 · p. 17
• During GRPO, reward-filtered chains of thought or final answers are reformatted into samples that train judgment and reflection.The reward signals also compute policy-update advantages; Figure 7 shows four prompt templates for generating reward and reflection data.Source: §A.2 · p. 18, Figure 7 · p. 18

A.3 RELATED WORKS

• Unlike prior self-reward and reflection approaches relying on external annotation or separate reward models, SPARK unifies policy and reward capabilities in one GRPO-optimized model.The framework also combines reasoning, judgment, and self-reflection without preference annotation or external reward modeling.Source: §A.3 · p. 18

A.4 CASE STUDY

• The case studies illustrate reasoning with self-judgment and report that mathematical-task judgment transfers to broader visual domains.The transfer examples include VL-RewardBench cases.Source: §A.4 · p. 18

A.5 THE EVOLVING LANDSCAPE OF LARGE VISION LANGUAGE MODELS

• The background describes multimodal models expanding into document parsing, video, and multi-image understanding, alongside a hybrid alignment recipe.That recipe uses DPO for broad preference alignment or verifiable-feedback RL to sharpen reasoning.Source: §A.5 · p. 19