PaperlayerView exact revision on arXiv

Source-linked AI summary

DE-Venus: A Data-Efficient RLVR Framework for Large Language Models

Shenzhi Yang, Guangcheng Zhu, Kai Tang, Zhengqing Zang, Xing Zheng, Haobo Wang, Yingfan Ma, Bowen Song, Bo Han, Bo An, Lei Feng, Weiqiang Wang, Junbo Zhao, Gang Chen

arXiv:2609.03324v1cs.LG

TL;DR

RLVR scaling faces expensive rollouts, costly verified labels, and unreliable supervision. DE-Venus unifies selection, weak supervision, and refinement as an evolving supervision lifecycle over verl, and reports preserved or improved quality with sharply reduced labels or data, alongside faster convergence in selected business settings.

  • Problem

    RLVR requires costly rollouts and reliable targets, while unlabeled data cannot directly provide verifiable rewards and supervision may be noisy or harmful.

  • Method

    DE-Venus treats supervision as evolving state and localizes active selection, weak-supervision construction, and training-time refinement while preserving verl’s distributed execution contracts.

  • Results

    DE-Venus configurations exceed fully supervised references with 10% of labels, while PivotTrace exceeds its full-data reference using 57.9% of training examples and 29.3% of annotations.

  • Takeaways & Limitations

    DE-Venus reduces annotation and training demands while retaining a common scalable RLVR pipeline across heterogeneous data-efficiency methods.

  • Takeaways & Limitations

    A systematic comparison across RLVR algorithms beyond the default GRPO backbone is beyond the scope of the report.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) improves large language model reasoning, but its practical scaling is constrained by expensive on-policy rollouts and the cost of obtaining reliable targets at scale. Existing methods address sample selection, incomplete supervision, or noisy labels separately, often entangling supervision logic with distributed training and hindering controlled comparison and reuse. We present DE-Venus, a unified framework for data-efficient RLVR that treats supervision as evolving state across data preparation and policy optimization. It organizes this lifecycle into three modules: Active Data Selection allocates training and annotation budgets; Weak Supervision Construction derives learning signals from unlabeled examples; and Training-Time Supervision Refinement filters or corrects unreliable supervision. DE-Venus supports seven representative methods and a data-selection pipeline by expressing method-specific decisions as offline dataset transitions or online transformations of targets, rewards, batches, and advantages while preserving verl's distributed execution contracts. Across public benchmarks and three business scenarios, separate configurations preserve or improve model quality with only 10% of labels or as little as 13% of relevant data; selected business configurations also reduce observed convergence steps by 63%--75%. DE-Venus thus reduces annotation and training costs without sacrificing scalable RL execution.

1 Introduction

RLVR scaling is limited by costly rollouts, expensive verified labels, and unreliable supervision. DE-Venus unifies data selection, weak supervision, and supervision refinement while preserving scalable execution and reducing resource requirements.

  • On-policy RLVR repeatedly spends generation and optimization resources, including on questions unlikely to provide informative reward variation.
  • Verified reference answers are costly, specialized domains require scarce expertise, and existing or constructed supervision can be unreliable.
  • DE-Venus treats supervision as evolving state and separates selection, construction, and revision from distributed RL execution.
  • The framework organizes interventions into Active Data Selection, Weak Supervision Construction, and Training-Time Supervision Refinement.
  • DE-Venus provides reproducible implementations for seven representative methods and an active data-selection pipeline across public benchmarks and three business scenarios.
  • Selected configurations surpass fully supervised references using only 10% of labels or 57.9% of training data with 29.3% annotations, while business settings reduce observed convergence steps by 63%–75%.

2 Design and Implementation

DE-Venus models supervision as changing state and coordinates interventions across data preparation and policy optimization without replacing verl’s distributed runtime.

  • DE-Venus represents supervision state through changing content, confidence, and persistence from data preparation to policy optimization.
  • Active Data Selection chooses examples for retention, weak supervision, or annotation before optimization.
  • Weak Supervision Construction derives pseudo targets or target-free rewards for examples without trusted answers.
  • Training-Time Supervision Refinement reassesses supervision using reward dynamics, representation geometry, and rollout evidence.
  • The supervision control plane owns dispatch, local transformations, provenance, and persistent transitions, while verl retains rollout, optimization, validation, and checkpoint services.

2.1 Design Principles

DE-Venus isolates supervision semantics from distributed execution, preserves native interfaces, makes supervision lifetimes explicit, and controls infrastructure-level variation.

  • DE-Venus delegates model placement, rollout generation, distributed computation, optimization, and checkpointing to verl while localizing supervision decisions.
  • Online interventions emit targets, token-level rewards, aligned DataProto selections, or advantages understood by downstream verl components.
  • Step-local decisions affect in-memory targets, rewards, rollout groups, or advantages, whereas persistent decisions update versioned datasets across training rounds.
  • Hydra configuration and shared workflows retain common models, loaders, rollout engines, GRPO implementation, resources, logging, validation, and checkpoint paths when permitted.

2.2 Architecture Overview

The architecture places DE-Venus supervision orchestration and method logic between experiment configuration and verl’s distributed RL runtime. Module ordering indicates when supervision may change, not a mandatory execution sequence.

  • The experiment interface defines reproducible runs through Hydra configuration, launch workflows, and a training Parquet dataset.
  • Supervision orchestration assembles runtime context, dispatches online methods, and manages persistent transitions.
  • The weak-supervision algorithm band implements the three intervention modules and shares voting, verification, reward, uncertainty, and provenance services.
  • The distributed RL runtime remains owned by verl and supplies dataset protocols, rollout engines, GRPO updates, distributed workers, and runtime services.
  • The modules share a Parquet schema and use DataProto and worker contracts when entering online optimization, but their ordering is not compulsory.

2.3 Unified Invocation and the verl Boundary

DE-Venus adopts verl’s configuration and execution lifecycle while routing supervision-specific decisions through a unified dispatch boundary. Offline selection and between-round relabeling produce standard datasets, while online methods reuse the distributed runtime without forking its backend.

  • DE-Venus inherits verl’s configuration and invocation model rather than introducing a parallel control stack.Hydra adds supervision-setting and method selectors while model, rollout, optimization, resource, validation, and checkpoint options remain in verl’s configuration space.
  • TaskRunner assembles the shared Ray execution context and dispatches the selected online trainer through WeakTrainerInterface.The standard init_workers() and fit() lifecycle is preserved.
  • Offline selection runs before optimization, while persistent promotion and relabeling run between rounds and emit Parquet datasets consumed through RLHFDataset.These components are outside the online dispatch boundary and preserve provenance artifacts.
  • Minimal invasion preserves worker APIs and downstream objects even when methods require paired rollouts, probes, filtering, or specialized reward ordering.DE-Venus does not reconstruct the cluster or fork the distributed execution backend.

2.4 Three Supervision Modules

DE-Venus groups supervision interventions by when decisions are made, what evidence they use, and which boundary objects they modify. The three modules share one execution substrate while supporting offline selection, online weak-signal construction, and refinement of supervision during or across training.

  • Three Supervision Modules: The three modules share one execution substrate but differ in decision time, evidence source, and boundary object.The grouping follows the intervention point in the supervision lifecycle rather than only semi-supervised, unsupervised, or noisy-label assumptions.
  • Active Data Selection: Active Data Selection uses an offline generate–score–route pipeline to select, retain, annotate, or route candidate examples.Scoring can use empirical accuracy, consistency, entropy, self-certainty, attention pivots, and hidden-state dynamics; selected records are materialized in the common Parquet schema.
  • Weak Supervision Construction: Weak Supervision Construction derives pseudo targets or target-free signals from grouped rollouts through method-local online trainers.Pseudo-target methods write inferred answers before verifier scoring, while target-free methods convert confidence or uncertainty into reward signals.
  • Weak Supervision Construction: Weak-supervision outputs terminate at native verl contracts, after which advantage estimation, actor updates, validation, and checkpointing use the shared runtime.The contract is either a substituted verifier target or a response-shaped reward tensor for GRPO.
  • Training-Time Supervision Refinement: Training-Time Supervision Refinement filters or transforms rollout supervision online and can promote examples or correct labels using persisted probe evidence.TraPO uses temporal reward and pass-rate behavior, GeoMin uses representation distributions, and OLR proposes corrections under configured reliability rules.

2.5 Unified Data Model

DE-Venus connects cross-round dataset evolution with within-step RL optimization through persistent sample identity and transient rollout groups. Dataset metadata and runtime tensors remain aligned as selection, target changes, and groupwise operations occur.

  • Unified Data Model: DE-Venus uses a two-level data model separating persistent sample supervision state from runtime trajectories generated during one optimization step.The persistent level spans dataset rounds, while the runtime level is regenerated for sampled rollouts.
  • Persistent Level: Each round’s dataset records prompts, supervision targets, label states, and task metadata under persistent sample identity.Selection can change dataset membership, while promotion or correction can change targets without changing retained-sample identity.
  • Runtime Level: RLHFDataset copies persistent keys into DataProto, whose tensor and non-tensor partitions carry aligned optimization quantities and supervision metadata.Controller-side transformations preserve correspondence between semantic metadata and trajectories.
  • Runtime Level: At optimization step t, sampled records receive fresh group identifiers and repeated prompts before generation.The resulting grouping supports sibling-trajectory operations independently of persistent sample identity.
  • Identity and Grouping: The shared uid enables majority voting and GRPO normalization after batch balancing or reordering, whereas later-step joins use persistent index keys.Probe histories and successive Parquet datasets join through extra_info.index rather than uid.

2.6 Contract-Preserving Dataflows

DE-Venus expresses online supervision methods as optional transformations around verl’s rollout-to-update path and expresses persistent decisions as reproducible dataset transitions. Contract-preserving placement ensures downstream verl stages can execute unchanged.

  • Online Dataflow: Online methods compose optional transformations for target construction, batch admission, reward construction, and advantage transformation around the rollout-to-update path.Each transformation is placed after required evidence becomes available and before the first downstream consumer.
  • Online Dataflow: Unused transformations act as the identity, allowing method-specific interventions without changing the surrounding execution path.The intervention surfaces correspond to native verl-consumed supervision objects.
  • Contract Preservation: Sequence-level scores are placed at the final valid response token to produce verl-compatible response-shaped reward tensors.Aligned DataProto selections preserve row mapping across tensor and non-tensor partitions.
  • Persistent Dataflow: Persistent decisions use generated evidence and method-specific rules to produce accepted selections, promotions, or target revisions.Active Data Selection builds the initial dataset, TraPO promotes reliable unlabeled samples, and OLR revises unreliable targets; each transition is reproducible from persisted inputs and manifests.

2.7 Diagnostics, State, and Extensibility

DE-Venus makes supervision decisions observable, assigns state by lifetime, and provides separate extension paths for online and offline methods.

  • Diagnostics: Supervision diagnostics track agreement, pass rates, rewards, confidence, admitted groups, and reward or advantage statistics alongside native verl metrics.Persistent records retain response or probe data and transition manifests for sample-level auditing.
  • State: Step-local supervision state stays in DataProto, while resumable and cross-round state is persisted according to later behavioral dependencies.TraPO persists decision histories and buffers with normal training state; cross-round artifacts use Parquet datasets, evidence records, and manifests.
  • Extensibility: Online methods specialize trainer intervention surfaces and register through WeakTrainerInterface and the common Hydra entry point.They reuse the execution context assembled by TaskRunner.
  • Extensibility: Offline methods implement the evidence–decision–materialization flow and emit RLHFDataset-compatible Parquet data with required evidence or provenance.This keeps method-specific data preparation separate from the distributed execution substrate.

3 Data-Efficient RLVR: From Data Curation to Reliable Supervision

Data-efficient RLVR addresses costly and imperfect supervision across the full lifecycle: selecting data, constructing signals for unlabeled examples, and refining supervision during optimization.

  • Motivation: Data-efficient RLVR targets costly verified labels, wasted rollout and annotation budgets, and unreliable supervision that can produce harmful policy updates.The framework therefore covers selection, weak-signal construction, and training-time refinement as one lifecycle.
  • Unified interface: Standard verified rewards compare rollout answers with trusted targets, while weak rewards use pseudo targets or target-free intrinsic signals and advantages are reliability-weighted.Refinement can filter with w_i = 0, softly reweight with 0 < w_i < 1, or correct a pseudo target.
  • Lifecycle overview: Figure 2 places selection before training, weak supervision construction after selection, and refinement during policy optimization, with noisy-label correction as a separate branch.The pipeline distinguishes preparation-time routing from training-time correction.
  • Active Data Selection: Candidate selection allocates annotation and optimization resources by routing examples to external labeling, weak supervision, training, or exclusion.Selection can use verified difficulty, response or representation uncertainty, and calibrated thresholds.
  • Active Data Selection: Probe-calibrated triage uses a small strategically distributed annotation set to relate uncertainty scores to empirical correctness and set data-routing thresholds.PivotTrace exemplifies this process by calibrating pivot-count rankings with sliding-window statistics.
  • Weak Supervision Construction: Weak supervision converts unlabeled examples into response-level learning signals through self-consistency, cross-view agreement, or confidence-based rewards.TTRL uses majority pseudo targets, Co-Rewarding uses semantically equivalent views, and Intuitor or EM-RL use confidence or negative entropy.
  • Training-Time Supervision Refinement: Training-time refinement reassesses supervision as policy behavior changes, retaining, promoting, reweighting, or replacing signals based on optimization evidence.TraPO compares pseudo-label pass-rate trajectories with verified examples, while GeoMin uses representation structure from labeled rollouts.

4 Experiments

The experiments evaluate DE-Venus across incomplete, corrupted, and selectively allocated supervision on public benchmarks and business scenarios. Configurations preserve or improve quality while reducing labels, training data, or observed training cycles.

  • Evaluation scope: The evaluation covers incomplete supervision, corrupted supervision, and joint allocation of annotation and training budgets across public benchmarks and three business scenarios.Compared methods share the DE-Venus training and evaluation pipeline, with variation focused on supervision or selection strategy.
  • Incomplete supervision: 42.7% ID and 67.0% OOD average accuracy: TTRL achieves the strongest unsupervised result, showing useful learning signals can be constructed without labels.The 10%-label regime is also evaluated for weakly supervised RLVR.
  • Incomplete supervision: GeoMin reaches 47.9% ID and 69.5% OOD accuracy with 10% of labels, exceeding full supervision’s 46.7% and 69.0% references.These correspond to gains of 1.2 ID points and 0.5 OOD points; TraPO reaches 43.8% ID and 67.2% OOD.
  • Corrupted supervision: OLR improves both ID and OOD averages at every tested noise ratio under inactive and active noisy-label regimes.Largest reported gains are 8.8 ID and 5.4 OOD points under inactive noise, and 6.4 ID and 8.1 OOD points under active noise.
  • Corrupted supervision: At ρ = 0.9 under active noise, both configurations deteriorate sharply and OLR provides only limited recovery, indicating refinement mitigates rather than eliminates extreme corruption.Across the remaining settings, the reported deltas are uniformly positive without changing the underlying GRPO execution path.
  • Data and business efficiency: 57.9% of training examples and 29.3% of annotations: PivotTrace exceeds the fully supervised full-data reference by 0.3 ID points and 0.6 OOD points.In business scenarios, reliability-aware unlabeled-data use improves a normalized metric by up to 14 points, filtering removes 28% of medical-empathy training examples within 1.9 points of full-data training, and compact safety subsets preserve principal safety metrics.

5 Conclusion

DE-Venus provides a unified supervision lifecycle for data-efficient RLVR while preserving a shared distributed RL substrate. Experiments across multiple settings show that its configurations can improve efficiency without sacrificing competitive model quality.

  • DE-Venus organizes data-efficient RLVR around selecting, constructing, and refining supervision through a lightweight layer over verl.Method-specific logic is localized to explicit data, reward, filtering, relabeling, and advantage-transformation interfaces.
  • The framework enables heterogeneous RLVR methods to be implemented, evaluated, and extended without repeatedly rebuilding rollout and optimization infrastructure.
  • Experiments cover weakly supervised learning, noisy-label learning, data selection, and representative business scenarios.
  • The results support treating RLVR data efficiency as an end-to-end supervision problem rather than solely as a dataset-size or reward-function choice.
Loading 2609.03324v1…