PaperlayerView exact revision on arXiv

Source-linked AI summary

From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement

Qinsi Wang, Jing Shi, Huazheng Wang, Kun Wan, Yiran Wu, Bo Liu, Qingyun Wu, Hai Helen Li, Yiran Chen, Handong Zhao, Wentian Zhao

arXiv:2607.23802v2cs.AI

TL;DR

RLVR remains difficult to apply to open-ended tasks because their quality lacks deterministic verification. This paper proposes RLSVR, which transforms such tasks into proxy environments with self-verifiable rewards, and shows that SpyRL consistently outperforms existing self-improvement methods across summarization, creative writing, and mathematical reasoning.

  • Problem

    RLVR is effective in deterministic domains but open-ended tasks lack verifiers and often require biased, costly, or capability-limited external evaluators.

  • Method

    RLSVR transforms open-ended tasks into proxy environments whose internal rules and interaction outcomes generate verifiable rewards, instantiated by information-asymmetric self-play in SpyRL.

  • Results

    SpyRL consistently outperforms existing self-improvement methods across summarization, creative writing, and mathematical reasoning, including 75.4% and 77.3% win rates on Qwen3-8B.

  • Takeaways & Limitations

    Task and environment transformation can extend scalable RLVR-based self-improvement to open-ended tasks beyond inherently verifiable domains.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has driven recent progress in reasoning-oriented large language models (LLMs) by enabling large-scale optimization. However, its applicability remains largely limited to domains such as mathematics and coding, where correctness can be deterministically verifiable. Open-ended tasks instead often rely on human preferences, reward models, or LLM-based judges, introducing evaluation bias, judge capability bottlenecks, and additional inference costs. Drawing on the principle of self-supervised learning, which constructs pretext tasks to derive supervision from the data itself, we propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a task-transformation-based training paradigm for extending RLVR to open-ended tasks. RLSVR transforms open-ended tasks into verifiable proxy environments whose internal rules and interaction outcomes automatically generate reward signals. We instantiate RLSVR with SpyRL, a Self-PlaY Reinforcement Learning method inspired by social deduction game Who Is the Spy?. Agents receive asymmetric information, complete the same target task, and vote to identify a designated spy. Because the spy identity is predetermined, voting outcomes provide fully verifiable rewards, while successful identification remains closely related to output quality. Experiments on text summarization, creative writing, and mathematical reasoning show that SpyRL outperforms existing self-improvement methods on non-verifiable tasks and yields consistent gains on verifiable reasoning tasks. These results demonstrate that task transformation can extend scalable RLVR-based self-improvement beyond inherently verifiable domains. Models and code have been released at https://github.com/wangqinsi1/RLSVR/tree/SpyRL.

1 Introduction

RLSVR extends RLVR to open-ended tasks by transforming them into proxy environments with automatically verifiable rewards, inspired by self-supervised pretext tasks. Its SpyRL instantiation uses information-asymmetric self-play, achieving gains across summarization, creative writing, and mathematical reasoning.

  • Motivation: RLVR scales reasoning-model training in deterministic domains such as mathematics and coding but remains brittle for open-ended tasks requiring subjective judgment.Prior approaches use learned preference signals or model-based feedback to relax strict verifiability.
  • RLSVR: RLSVR transforms an open-ended task into a proxy environment whose rewards arise automatically from the transformed task and are therefore self-verifiable.The paradigm adapts self-supervised learning’s task-transformation principle to RLVR.
  • SpyRL: SpyRL implements RLSVR through information-asymmetric self-play inspired by Who Is the Spy?, with most agents receiving complete inputs and one spy receiving degraded information.Agents perform the same target task in a multi-agent environment, enabling identity inference to generate rule-based rewards.
  • Results: SpyRL significantly outperforms existing self-improvement methods on summarization and creative writing while producing consistent gains on mathematical reasoning.The method is evaluated on text summarization, creative writing, and mathematical reasoning, with suspicion votes correlating with output quality.
  • Results: 75.4% and 77.3% win rates on Qwen3-8B summarize SpyRL’s performance on summarization and creative writing, respectively.These results exceed existing self-play methods, which yield only marginal gains.
  • Results: 8.97% and 6.16% improvements on Qwen3-4B and Qwen3-8B, respectively, demonstrate SpyRL’s gains in mathematical reasoning across seven benchmarks.The paper presents these results as evidence that task and environment transformations can incorporate non-verifiable tasks into scalable RLVR training.

2 RLSVR: Reinforcement Learning with Self-Verifiable Rewards

RLSVR addresses RLVR’s lack of deterministic verification for open-ended tasks by transforming them into proxy environments with automatically generated, self-verifiable rewards. The transformation preserves the original task capabilities while making interaction outcomes exactly checkable against environment-sampled ground truth.

  • Motivation: RLVR scales through deterministic verification, but open-ended objectives lack verifiers and approximate evaluators reintroduce evaluation limitations.The latent quality function for summarization and creative writing cannot be directly verified, motivating task transformation.
  • Task transformation: RLSVR transforms the original task into a proxy environment whose latent variable is sampled and recorded as episode ground truth.The latent variable may encode perturbations, withheld information, or output-generation conditions, while remaining hidden from the policy.
  • Task transformation: The policy performs the original target task on observations constructed from the input and hidden latent variable, preserving the capabilities exercised during training.The environment can construct one or more observations from (x, z), and the policy generates outputs conditioned on each observation.
  • Task transformation: A verifiable interaction asks the policy to infer the hidden variable from task outputs, with correctness designed to depend on output quality.Examples include identifying an output produced under a hidden condition or recovering withheld information.
  • Self-verifiable rewards: RLSVR computes rewards by checking interaction outcomes against the sampled latent variable, requiring no human annotation, learned reward model, or external judge.Because the environment creates the ground truth, reward computation is deterministic and rule-based.

3 SpyRL: Instantiating RLSVR via Information-Asymmetric Self-Play

SpyRL instantiates RLSVR as a closed-loop, information-asymmetric self-play game that converts open-ended output quality into verifiable spy-identity detection. Alternating performing and detection stages couple task performance with rule-based rewards without external verification.

  • Information-Asymmetric Self-Play: SpyRL alternates Performing and Detection stages, using asymmetric information so n−1 civilians receive full inputs while one spy receives corrupted information.The spy is assigned through a predetermined identity and task-relevant information degradation.
  • Verifiable Identity Detection: Because the environment explicitly assigns the spy identity, detection correctness provides a directly computable reward signal without an external verifier.Detectors infer the spy from the set of public outputs and vote for a player identity.
  • Coupled Optimization: Detection outcomes determine performing rewards, while performer output quality determines detection difficulty, creating a mutually dependent closed-loop optimization system.This coupling uses whether an output reveals the underlying information deficiency as a computable surrogate for output quality.
  • Collective Detection: Multi-player voting and collective decision-making reduce reliance on isolated proposer–solver evaluations and mitigate bias amplification from single verifiers.Group-relative advantages further let detectors learn from relative performance within the same epoch without an additional critic network.
  • Capability-Oriented Task Design: SpyRL aligns performing capabilities with mathematical reasoning, creative writing, and text summarization, while incentivizing players to produce stronger task outputs to avoid being identified as the spy.Examples include clearer mathematical derivations and more natural, innovative creative writing.

4 Experiments

Experiments evaluate SpyRL across summarization, creative writing, and mathematical or verifiable reasoning, finding strong performance on both open-ended and verifiable tasks. Additional evaluations show alignment with human preferences, positive cross-task transfer, and the importance of alternating optimization, adversarial group size, and role-specific reward calibration.

  • Experimental Setup: SpyRL evaluates three domains using asymmetric information: civilians receive complete inputs, while the spy receives a continuously masked span.The spy must infer the missing information before responding, creating the transformed game environment.
  • Non-verifiable Tasks: SpyRL achieves the best performance on summarization and creative writing, attaining the highest ROUGE-L on every benchmark for both backbones.On GovReport, SpyRL improves over Absolute Zero from 33.2 to 36.7 with Qwen3-4B and from 32.5 to 34.1 with Qwen3-8B.
  • Verifiable Tasks: SpyRL achieves the best results across five mathematical benchmarks for both Qwen3-4B and Qwen3-8B and also leads on MMLU-Pro and GPQA-D.The gains are particularly pronounced on AIME24 and AIME25, while the advantage extends beyond pure mathematical problem solving.
  • Human Evaluation: Human evaluation on 400 creative-writing prompts uses ten Ph.D. students ranking anonymized responses across five dimensions.The study reports SpyRL pairwise win rates when its response is preferred.
  • Rubric-as-Reward Baselines: SpyRL outperforms Qwen3.5-27B-RaR across all dimensions, with overall win rates of 59.3% on WritingPrompts and 56.2% on WritingBench.The comparison uses rubric-as-reward methods with Qwen3.5-27B and GPT-4o as rubric executors under the same GRPO framework.
  • Ablations: Ablations show that alternating performing and detection optimization, the spy mechanism, larger groups, and role-advantage estimation are important for sustained gains.Performance rises from 5.5 to 9.3 when scaling players from 3 to 5, while removing role-specific baselines lowers seven-benchmark performance.

5 Conclusion · A Related Work

The paper presents RLSVR as a task-transformation paradigm extending verifiable-reward reinforcement learning to open-ended tasks, and situates SpyRL within prior work on debate, self-play, and non-verifiable-domain reinforcement learning.

  • 5 Conclusion: RLSVR extends RLVR to open-ended tasks by transforming them into proxy environments with rewards verifiable by construction.It applies the task-transformation principle of self-supervised learning and uses environment-assigned latent variables instead of external judges or reward models.
  • 5 Conclusion: The paper instantiates RLSVR with SpyRL, an information-asymmetric self-play game.
  • Multi-Agent Debate and Interaction: Prior multi-agent debate research has improved LLM factuality, reasoning, evaluation quality, divergent thinking, and truthfulness.The cited work includes Irving et al. (2018), Du et al. (2024), Chan et al. (2023), Liang et al. (2024), and Khan et al. (2024).
  • Multi-Agent Debate and Interaction: Social deduction games have also been used to train LLMs with multi-agent reinforcement learning.The passage cites Sarkar et al. (2025).
  • Self-Play for LLMs: Self-play has driven major advances in AlphaGo, AlphaZero, and OpenAI Five, while asymmetric self-play can create automatic curricula.The passage cites Silver et al. (2016; 2017; 2018), Berner et al. (2019), and Sukhbaatar et al. (2017).
  • Self-Play for LLMs: In LLMs, self-play has been adapted for self-improvement and reasoning.The passage cites Chen et al. (2024) and Yuan et al. (2024).
  • Self-Play for LLMs: Proposer-solver frameworks include Absolute Zero, R-Zero, and Tool-R0.The passage cites Zhao et al. (2025), Huang et al. (2025), and Acikgoz et al. (2026).
  • RL Beyond Verifiable Domains: RL beyond verifiable domains uses preference signals, LLM judges, rubric-based rewards, self-rewarding approaches, and process reward models to approximate feedback.The passage contrasts these methods with deterministic verification and cites RLHF, LLM-as-a-Judge, rubric-based rewards, self-rewarding, and process reward model research.

B Extended Related Work … C.2 Alternating Optimization Strategy

The paper situates SpyRL as a self-play framework that converts asymmetric multi-agent interactions into verifiable rewards for open-ended tasks, then stabilizes training through role-calibrated advantages and adaptive stage alternation.

  • B Extended Related Work: SpyRL converts information-asymmetric adversarial interactions into rule-based, verifiable RL rewards applicable to arbitrary open-domain tasks.Unlike approaches that mainly improve outputs at inference time or rely on game-specific outcomes, SpyRL uses interaction outcomes as training rewards.
  • B Extended Related Work: SpyRL extends self-play to non-verifiable domains by making game success depend directly on target-task proficiency, without answer-level verification.The approach addresses the limited applicability of RLVR and game-specific self-play signals to open-ended objectives.
  • B Extended Related Work: Preference-based RL methods replace deterministic verification with learned or model-generated signals, requiring costly preference data or external reward models whose quality limits learning.The cited alternatives include RLHF, Constitutional AI, RLAIF, DPO, LLM-as-a-Judge, and rubric-based rewards.
  • C Implementation Details: The state-transition implementation uses thresholds for Detection and Performing transitions together with a minimum dwell time.These parameter settings are reported in Table 11 and support the alternating optimization mechanism.
  • C.1 Role-Advantage Estimation: Role-Advantage Estimation maintains separate exponential moving-average baselines for spy and civilian rewards to remove role-induced bias from policy gradients.The spy observes degraded input g(x), whereas civilians observe full input x, producing different expected reward distributions even under identical policies.
  • C.1 Role-Advantage Estimation: RAE makes gradients measure performance relative to each role’s typical outcome, preventing systematic role preference and stabilizing training.The role-calibrated advantages replace raw performing-stage rewards in the policy gradient.
  • C.2 Alternating Optimization Strategy: SpyRL alternates between Detection and Performing stages using mini-batch identification accuracy and uncertainty rates to respond to saturation signals.Easy spy identification triggers Performing training, whereas frequent detection failure or abstention triggers Detection training.
  • C.2 Alternating Optimization Strategy: Hysteresis thresholds, minimum per-phase dwell time, and exclusive gradient routing prevent chattering while dynamically switching the active module.The alternating scheme is intended to avoid strategic stagnation, role collapse, and divergence through verifiable game-mechanic supervision.

C.3 Training Hyperparameters

The GRPO training setup uses Qwen3-4B-Instruct-2507 with eight rollouts per prompt, a 1024-sample effective batch, and 100 optimization iterations. Interactive generation uses vLLM across eight GPUs with bounded context lengths and a five-player, one-round environment, while memory is managed with checkpointing and FSDP.

  • Optimization and Algorithm Settings: Training uses GRPO in verl with Qwen/Qwen3-4B-Instruct-2507 as both actor and reference base model on a single node with 8 GPUs.The implementation uses the same base model for actor and reference policies.
  • Optimization and Algorithm Settings: The rollout samples n = 8 responses per prompt; 128 prompts per batch therefore produce an effective batch size of 1024 samples.The actor learning rate is 1 × 10−6 for 100 training iterations.
  • Optimization and Algorithm Settings: The PPO mini-batch size is 128, the micro-batch size is 2 per GPU, the KL penalty coefficient is 0.001, and the entropy coefficient is 0.These settings limit deviation from the reference model while disabling entropy regularization.
  • Generation and Rollout: Interactive rollouts use vLLM with tensor model parallelism set to 8 and GPU memory utilization restricted to 0.45.Prompts are limited to 12,288 tokens, responses to 4,096 tokens, and total model length to 16,384 tokens; the environment has 5 players and 1 round.
  • Memory Management and System Configurations: Gradient checkpointing is enabled for the actor, while FSDP offloads reference-model parameters to the CPU and keeps actor parameters and optimizer states on the GPU.This configuration is used to manage GPU memory while maintaining training throughput.

C.4 Baselines & Metrics · C.5 Prompt Design and Configurations

The evaluation compares SpyRL with proposer-solver and pure self-play baselines using automated metrics and position-bias-mitigated GPT-4o A/B testing. Its two-stage prompts encode asymmetric roles, structured story-generation constraints, and evidence-based spy detection with uncertainty and deterministic extraction.

  • C.4 Baselines & Metrics: SpyRL is benchmarked against R-Zero’s proposer-solver reinforcement-learning loop and Absolute Zero’s generator-verifier self-play framework.R-Zero optimizes a proposer against solver feedback without external human preference data, while Absolute Zero bootstraps strategies through iterative self-refinement.
  • C.4 Baselines & Metrics: Automated metrics assess format adherence, exact-match spy identification, word-count constraints, and repetition penalties.These rule-based measures target structural compliance and detection accuracy across game rounds.
  • C.4 Baselines & Metrics: GPT-4o performs pairwise A/B tests against the base model and baselines to evaluate creativity, strategic depth, and narrative coherence.This judge-based protocol supplements automated metrics for open-ended story-writing quality.
  • C.4 Baselines & Metrics: Each GPT-4o comparison is queried twice with the candidate order swapped to mitigate position bias.Model A is awarded a win only when preferred in both permutations or when it wins one permutation and ties the other; opposing outcomes become ties.
  • C.5.1 Performing Stage Prompt Design: The Performing Stage prompt dynamically supplies role information and instructions, making one template support civilians and the spy under asymmetric information.It asks agents to reason before answering, infer the hidden prompt as the spy, or plan narrative structure as civilians.
  • C.5.1 Performing Stage Prompt Design: The Performing Stage enforces a 300-500-word continuous story with a turning point, payoff, coherence, and a required “Answer:” output marker.These constraints target high-quality, parseable story generation.
  • C.5.2 Detection Stage Prompt Design: The Detection Stage evaluates stories across five anomaly dimensions, permits boxed N/A uncertainty, limits output to <= 2000 tokens, and requires boxed player-ID extraction.The rubric covers off-theme mismatch, shallow meaning, weak narrative craft, low creativity, and consistency with other stories.

D Additional Experiments

Additional experiments provide broader validation of SpyRL by examining its effectiveness, training stability, task transferability, evaluation robustness, and human alignment. A/B tests compare trained models with their own untrained backbones on summarization and writing benchmarks using GPT-4o win rates.

  • Experimental scope: Additional experiments assess SpyRL from multiple perspectives, including effectiveness, training stability, task transferability, evaluation robustness, and human alignment.These experiments complement the main results and provide a more comprehensive validation of the framework.
  • Evaluation against untrained models: Summarization A/B tests report GPT-4o win rates for trained models against their own untrained backbones, with 50% indicating parity.The table averages results over five benchmarks and includes untrained-model rows as calibration references.
  • Evaluation against untrained models: Writing A/B tests likewise compare trained models with their own untrained backbones using GPT-4o win rates, where higher values are better.A 50% win rate denotes parity with the untrained model.

D.1 A/B Evaluation against the Untrained Base Model

The complementary A/B evaluation compares each method with its own untrained backbone, showing that SpyRL’s gains are not an artifact of the evaluation anchor. Untrained backbones remain near chance against themselves, while baseline self-evolution yields limited improvements on open-ended tasks.

  • Evaluation setup: Tables 13 and 14 compare every method against its own untrained backbone under the same evaluation setup for summarization and creative writing.This complements the main-text comparisons, where each row directly contrasts SpyRL with one competing method.
  • Untrained baselines: Untrained backbones achieve average win rates of 51.7% and 51.8% on summarization and overall creative-writing win rates between 51.0% and 51.8%.The swapped-order aggregation removes most position bias, so deviations from 50% reflect quality differences.
  • Self-evolution baselines: On summarization, R-Zero averages 51.9% and 51.5%, while Absolute Zero reaches 61.5% and 59.0% across the two Qwen3 backbones.The passage describes R-Zero’s results as statistically indistinguishable from no training at all.
  • Self-evolution baselines: On creative writing, Absolute Zero reaches overall win rates from 54.0% to 58.1%, while R-Zero is preferred less often than untrained Qwen3-4B at 48.8% and 46.5%.The reported Absolute Zero values are 54.0% and 55.2% with Qwen3-4B, and 56.4% and 58.1% with Qwen3-8B.
  • Reward mechanism: R-Zero and Absolute Zero depend on verifiable solver feedback, whereas SpyRL uses the environment-assigned spy identity, which remains defined for open-ended generation.Thus, the baseline proposer–solver loop lacks reliable optimization feedback on open-ended tasks, while SpyRL retains a verifiable reward.

D.2 Evaluation on More Challenging Reasoning Benchmarks · D.3 Joint versus Alternating Two-Stage Optimization

On larger, harder reasoning benchmarks, SpyRL achieves the strongest and most consistent gains, while alternating optimization substantially outperforms joint two-stage training. The results support task transformation and staged updates as effective strategies for self-improvement.

  • D.2 Evaluation on More Challenging Reasoning Benchmarks: SpyRL is evaluated against Qwen3-4B, R-Zero, and Absolute Zero on three larger, more demanding benchmarks: AMC, Olympiad-Bench, and SuperGPQA.The Qwen3-4B checkpoints reuse the mathematical-reasoning setting without further fine-tuning.
  • D.2 Evaluation on More Challenging Reasoning Benchmarks: 55.0 on AMC, 42.7 on Olympiad-Bench, and 30.1 on SuperGPQA make SpyRL the best-performing method on all three benchmarks.The untrained backbone scores 47.5, 34.8, and 25.4 on the same benchmarks, respectively.
  • D.2 Evaluation on More Challenging Reasoning Benchmarks: SpyRL gains 6.7 points on average across the three benchmarks, versus 1.6 for R-Zero and 3.6 for Absolute Zero.SpyRL is the only method that improves on every benchmark; R-Zero falls below the untrained backbone on AMC.
  • D.2 Evaluation on More Challenging Reasoning Benchmarks: Information-asymmetric self-play converts incomplete derivations, overlooked conditions, and unsupported assumptions into verifiable training signals by exposing them during spy detection.This mechanism explains why SpyRL remains effective on more difficult reasoning problems.
  • D.3 Joint versus Alternating Two-Stage Optimization: Jointly updating the performer and detector creates a tightly coupled, highly non-stationary optimization process because detector votes reward the performer while performer outputs train the detector.The problem is particularly pronounced during early training.
  • D.3 Joint versus Alternating Two-Stage Optimization: Alternating optimization updates only one stage per epoch, allowing detector strengthening to provide more reliable voting rewards while the performer is optimized.The other stage remains fixed during each epoch.
  • D.3 Joint versus Alternating Two-Stage Optimization: Joint training lowers GSM8K accuracy from 84.5 to 76.8, Math500 accuracy from 68.2 to 53.1, and Minerva accuracy from 42.3 to 33.1.Its average score across five benchmarks drops from 42.4 to 35.3.
  • D.3 Joint versus Alternating Two-Stage Optimization: Alternating training raises AIME 2025 accuracy from 6.7 to 20.0 and achieves an average score of 50.8 across five benchmarks.This corresponds to improvements of 8.4 points over the base model and 15.5 points over joint training; GSM8K rises from 84.5 to 93.4 and Math500 from 68.2 to 79.5.

D.4 Evaluation with an Alternative LLM Judge

Using Gemini-3.5-Flash as an alternative judge, SpyRL is evaluated under the same anonymized, swapped-order A/B protocol and consistently outperforms the original Qwen3 models on summarization and creative writing. The gains remain stable across evaluators, suggesting they are unlikely to reflect evaluator-specific preferences.

  • Evaluation protocol: SpyRL’s alternative-judge evaluation repeats anonymized A/B comparisons with swapped response orders for summarization and creative writing.Gemini-3.5-Flash compares SpyRL-trained outputs with corresponding original Qwen3 outputs for each test instance.
  • Summarization: 79.8% versus 52.3% average win rates for Qwen3-4B and 76.1% versus 52.0% for Qwen3-8B show consistent summarization gains across five benchmarks.The improvements span GovReport, Multi-News, QMSum, VCSum, and SAMSum.
  • Creative writing: SpyRL outperforms the original models across novelty, emotion, coherence, consistency, and overall quality on WritingPrompts and WritingBench.The comparison covers five quality dimensions across both creative-writing benchmarks.
  • Creative writing: 81.8% and 81.4% overall win rates for Qwen3-4B, and 77.8% and 77.0% for Qwen3-8B, are reported on WritingPrompts and WritingBench, respectively.These results are obtained with Gemini-3.5-Flash as the alternative judge.
  • Robustness: The improvements remain stable under a different LLM judge and are unlikely to arise from evaluator-specific preferences.The alternative-judge results are consistent with the GPT-4o-based main evaluation.

D.5 Agreement between GPT-4o and Human Evaluation

GPT-4o shows strong agreement with human judgments when evaluating SpyRL’s creative-writing outputs, achieving high precision and recall across all five dimensions. Overall quality reaches 91.0% precision and 93.8% recall, while consistency has the lowest recall at 79.4%.

  • Evaluation setup: GPT-4o’s agreement with human annotations is evaluated as binary classification of pairwise SpyRL preferences, using human judgments as ground-truth labels.Precision measures GPT-4o wins also preferred by humans, while recall measures human-preferred SpyRL wins identified by GPT-4o.
  • Agreement results: Precision ranges from 85.7% to 91.0%, and recall ranges from 79.4% to 93.8% across the five evaluation dimensions.GPT-4o exhibits strong agreement with human judgments across all dimensions.
  • Agreement results: Overall quality achieves 91.0% precision and 93.8% recall, while consistency has the lowest recall at 79.4%.GPT-4o also performs strongly on novelty and emotion.
Loading 2607.23802v2…