PaperlayerView exact revision on arXiv

Source-linked AI summary

Apriel-1.5-OpenReasoner: RL Post-Training for General-Purpose and Efficient Reasoning

Rafael Pardinas, Ehsan Kamalloo, David Vazquez, Alexandre Drouin

arXiv:2604.02007v2cs.LG

TL;DR

Multi-domain RLVR recipes are often insufficiently disclosed, while heterogeneous rollouts complicate joint optimization and long reasoning traces increase inference cost. Apriel-1.5-OpenReasoner addresses these issues with reproducible post-training, adaptive domain sampling, and a difficulty-aware length penalty, improving benchmark performance over Apriel-Base with shorter traces and length generalization beyond training.

  • Problem

    Multi-domain RLVR lacks fully disclosed recipes and faces domain imbalance from heterogeneous rollouts, while long reasoning traces increase inference cost and latency.

  • Method

    The paper trains Apriel-1.5-OpenReasoner on Apriel-Base across five public domains using adaptive sampling and a difficulty-aware length penalty without additional training overhead.

  • Results

    Apriel-1.5-OpenReasoner improves over Apriel-Base across AIME 2025, GPQA, MMLU-Pro, and LiveCodeBench while producing 30–50% shorter reasoning traces and generalizing from a 16K-token training budget to 32K-token inference.

  • Takeaways & Limitations

    The model matches strong similarly sized open-weight models at lower token cost, supporting a higher observed accuracy-versus-token-budget Pareto frontier.

  • Takeaways & Limitations

    The study uses Apriel-Base with the vision encoder omitted, so multimodal reasoning is outside the work’s scope.

Abstract

from arXiv · show

Building general-purpose reasoning models using reinforcement learning with verifiable rewards (RLVR) across diverse domains has been widely adopted by frontier open-weight models. However, their training recipes and domain mixtures are often not disclosed. Joint optimization across domains poses significant challenges: domains vary widely in rollout length, problem difficulty and sample efficiency. Further, models with long chain-of-thought traces increase inference cost and latency, making efficiency critical for practical deployment. We present Apriel-1.5-OpenReasoner, trained with a fully reproducible multi-domain RL post-training recipe on Apriel-Base, a 15B-parameter open-weight LLM, across five domains using public datasets: mathematics, code generation, instruction following, logical puzzles and function calling. We introduce an adaptive domain sampling mechanism that preserves target domain ratios despite heterogeneous rollout dynamics, and a difficulty-aware extension of the standard length penalty that, with no additional training overhead, encourages longer reasoning for difficult problems and shorter traces for easy ones. Trained with a strict 16K-token output budget, Apriel-1.5-OpenReasoner generalizes to 32K tokens at inference and improves over Apriel-Base on AIME 2025, GPQA, MMLU-Pro, and LiveCodeBench while producing 30-50% shorter reasoning traces. It matches strong open-weight models of similar size at lower token cost, thereby pushing the Pareto frontier of accuracy versus token budget.

1 Introduction

Apriel-1.5-OpenReasoner addresses reproducibility and optimization challenges in multi-domain RLVR with adaptive sampling and difficulty-aware length control. The resulting model improves benchmark performance while using shorter reasoning traces and generalizing beyond its training output budget.

  • Open-weight models often omit training recipes and domain mixtures, limiting reproducibility and understanding of multi-domain reasoning performance.
  • Asynchronous multi-domain RLVR can distort intended domain ratios because domains differ in rollout length, difficulty, verification latency, and sample efficiency.
  • Apriel-1.5-OpenReasoner uses a reproducible RL post-training recipe on Apriel-Base across five public training domains.
  • 30–50% shorter outputs accompany improvements over Apriel-Base on AIME 2025, GPQA, MMLU-Pro, and LiveCodeBench, despite a 16K-token training budget and 32K-token inference generalization.
  • Adaptive domain sampling maintains target mixture ratios during asynchronous multi-domain RL training.
  • Difficulty-Aware Length Penalty allows difficult problems to receive longer reasoning while encouraging concise traces on easier problems.

2 Background

RLVR trains language models with programmatically verifiable rewards, while policy-optimization and distributed-training methods address stability, variance, throughput, and data freshness. The paper uses sequence-level optimization and PipelineRL for multi-domain post-training.

  • 2.1 Reinforcement Learning with Verifiable Rewards: RLVR frames autoregressive language modeling as sequential decision-making guided by rewards that verify output correctness.
  • 2.1 Reinforcement Learning with Verifiable Rewards: The RLVR objective maximizes expected reward for responses generated by the policy given a prompt.
  • 2.2 Policy Optimization: PPO and GRPO operate at token level despite sequence-level rewards, creating a mismatch associated with high-variance gradients and training instability.
  • 2.2 Policy Optimization: GSPO replaces token-level importance ratios with sequence-level ratios, aligning optimization with sequence-level rewards and reducing reweighting variance.
  • 2.3 On-Policy RL for LLMs: PipelineRL uses in-flight weight updates so rollout and training proceed concurrently, keeping data near on-policy while maintaining GPU utilization.

3 Related Work

Related work covers two broad choices for general-purpose reasoning post-training—staged versus joint multi-domain RL—and several approaches to controlling reasoning length. These approaches trade optimization simplicity, transfer, forgetting management, and length adaptation in different ways.

  • RL post-training of LLMs: Staged domain-wise optimization simplifies training but can limit cross-domain transfer and requires careful management of forgetting across stages.
  • RL post-training of LLMs: Joint multi-domain optimization trains across domains simultaneously, motivating methods that address heterogeneous optimization dynamics.
  • Reasoning Length Control: Reasoning-length control methods include test-time budget forcing, budget-conditioned training, training-time penalties, explicit budget tokens, routing, and adaptive policy objectives.

4 Multi-Domain RL

The recipe addresses heterogeneous multi-domain rollouts with adaptive sampling and controls reasoning length using problem difficulty. It combines domain-ratio correction with a difficulty-aware penalty that preserves longer reasoning for hard problems while discouraging overlong traces on easier ones.

  • Adaptive Multi-Domain Sampling: Asynchronous domains differ in rollout and reward-computation time, causing faster domains to produce more completed trajectories and drift from target mixtures.Coding rollouts may require sandbox execution, whereas math or logic answers can often be checked more lightly.
  • Adaptive Multi-Domain Sampling: Adaptive sampling corrects domain-mixture drift by adjusting sampling weights from completed-rollout counts before selecting the next domain.Underrepresented domains are up-weighted, overrepresented domains are down-weighted, and corrections are clipped; static target weights are used initially until 50 completions.
  • Difficulty-Aware Length Penalty: Difficulty-Aware Length Penalty modulates the standard length penalty by problem difficulty, allowing difficult problems to reason longer while encouraging concision on easier ones.Difficulty is estimated from solve rates across rollouts for a prompt.
  • Multi-Domain RL: Five training environments support joint multi-domain RL with public datasets and verifiable reward functions.The supplied passages identify the environments collectively but do not enumerate their dataset names here.
  • Difficulty-Aware Length Penalty: For correct overlong rollouts, the penalty is scaled by solve rate, while incorrect rollouts retain a fixed penalty and unfinished maximum-length rollouts receive the full penalty.The method changes the upstream reward and does not alter the policy loss; experiments use γ = 1.0 and λf = 1.0.

5 Experiments

Experiments evaluate Apriel-1.5-OpenReasoner against comparable models and ablations across four benchmarks, emphasizing accuracy, output-token efficiency, domain-mixture effects, and reasoning-trace structure.

  • Experimental Setup: Apriel-1.5-OpenReasoner is evaluated across four held-out benchmarks using comparable-scale baselines and a shared 32K output-token cap.The benchmarks are AIME 2025, GPQA, MMLU-Pro, and LiveCodeBench.
  • Accuracy–Efficiency Trade-off: Apriel-1.5-OpenReasoner achieves higher accuracy with fewer output tokens than comparable-scale baselines across all four benchmarks.On AIME-25 it reaches 78.3% accuracy with 41% fewer tokens than Nemotron-Cascade; on LiveCodeBench it matches Nemotron-Cascade using 7.4K versus 16.0K tokens.
  • Accuracy–Efficiency Trade-off: Relative to Apriel-Base, output length falls by approximately 45% on GPQA, 46% on MMLU-Pro, and 32% on AIME-25.The larger reductions occur on straightforward questions, while longer traces remain more warranted for AIME-25.
  • Length-Penalty Ablation: DAP improves accuracy over the standard length penalty with small-to-moderate increases in output length across benchmarks.Accuracy improves by 6.6% on AIME-25 with 2% more tokens and by 3.1% on LiveCodeBench with 6% more tokens.
  • Domain Mixture Ablation: The proposed domain mixture achieves the highest accuracy on all four benchmarks, while math-and-code-only training notably lowers LiveCodeBench accuracy to 67.2% versus 70.8%.Uniform weighting underperforms the proposed mixture across accuracy measures.
  • Reasoning-Trace Analysis: Apriel-1.5-OpenReasoner uses a comparable number of reasoning steps but approximately 35% fewer tokens per step than Apriel-Base on correct AIME 2025 responses.The analysis attributes efficiency gains to tighter expression and reduced overhead rather than shallower reasoning.

6 Conclusion

The paper presents Apriel-1.5-OpenReasoner as a reproducible 15B multi-domain RL model combining adaptive domain sampling with difficulty-aware length control. It improves reasoning quality and inference efficiency, including beyond its 16K-token training budget.

  • Conclusion: Apriel-1.5-OpenReasoner is a 15B open-weight model trained with a reproducible multi-domain RL post-training recipe on Apriel-Base.The authors release a complete, openly available training pipeline alongside the model.
  • Conclusion: Adaptive domain sampling preserves target domain ratios under asynchronous rollout dynamics, while the difficulty-aware length penalty allocates more budget to harder problems without extra training or policy-loss changes.These mechanisms address domain imbalance and reasoning-length control within the same post-training recipe.
  • Conclusion: The model improves over Apriel-Base on challenging benchmarks while producing substantially shorter outputs and remaining effective under a 32K output-token budget after 16K-token training.The conclusion frames the result as joint improvement in reasoning quality and inference efficiency at practical model scale.

Reproducibility Statement

The reproducibility statement describes open documentation, public data, and planned release of the model artifacts and code needed to reproduce the RL post-training procedure.

  • Reproducibility: Hyperparameters, search spaces, and training details are documented in the main text and appendices, with key design decisions justified in their respective sections.The statement identifies the base-model choice, policy objective, adaptive sampling, and length penalty as documented design decisions.
  • Reproducibility: The training pipeline is based on publicly available PipelineRL, and all datasets used for training and evaluation are publicly accessible.Training sources and evaluation benchmarks are described in the paper and appendices.
  • Reproducibility: The authors plan to release model weights, training configurations, and all code necessary to reproduce the RL post-training procedure.This commitment follows the transparency principles associated with the LLM360 initiative.

A Training Domains and Verifiers

The training recipe spans five domains, each pairing a dataset with a verifiable reward function. Together, they cover mathematics, code generation, instruction following, logical puzzles, and function calling.

  • Five environments cover mathematics, code generation, instruction following, logical puzzles, and function calling.
  • Mathematics: Approximately 129K math problems span high-school through competition-level difficulty.
  • Code Generation: Code generation uses sandbox execution and assigns reward 1 only when all test cases pass.
  • Instruction-Following: Instruction following rewards the proportion of verifiable output constraints satisfied, from 0 to 1.
  • Logical Puzzles: Logical puzzles comprise 29 types and approximately 12K tasks spanning symbolic, deductive, constraint, and combinatorial reasoning.
  • Function Calling: Function-calling correctness requires an exact function-name match and arguments within predefined valid sets.

B Training Details

The training details identify the launch configuration, hyperparameter search space, and domain-rate ordering used to configure Apriel-1.5-OpenReasoner.

  • Hyperparameters: The Scope column distinguishes rollout generation and reward computation, optimization and weight updates, or parameters affecting both.
  • Hyperparameters: Table 4 provides the reference launch configuration for Apriel-1.5-OpenReasoner using GSPO and DAP.
  • Hyperparameter Search Space: Table 5 lists the exact candidate sets used in the reported hyperparameter sweeps.
  • Hyperparameter Search Space: Domain-rate vectors use the ordering M/C/L/I/F: math, coding, logic, ifeval, and function calling.
  • Training Progress: Figure 5 reports mean reward progress during training separately for each domain.

D Details of Reasoning Trace Analysis

The reasoning-trace analysis segments model outputs into steps, classifies each step among eight cognitive categories, and compares distributions between Apriel-Base and Apriel-1.5-OpenReasoner.

  • Step Categories: Eight categories include problem setup, forward reasoning, verification, backtracking, subgoal setting, backward chaining, final answer, and other.
  • Classification: The analysis segments traces and classifies every step into exactly one of the eight categories using GPT-5.4.
  • Results: 21% to 14%: non-productive steps decrease from Apriel-Base to Apriel-1.5-OpenReasoner.
  • Results: 11% to 17%: the combined share of verification, backtracking, subgoal setting, and backward chaining increases between the two models.
  • Results: Figure 6 compares step-type distributions and reports fewer verification and non-productive steps for Apriel-1.5-OpenReasoner on AIME 2025.

D.1 Example of Reasoning Trace Analysis

The trace-analysis procedure preserves each model output exactly while segmenting it into contiguous steps and assigning one cognitive-behavior label per step.

  • Extracting Steps from Reasoning Traces: The segmenter may create as many steps as needed and returns sequential step objects containing verbatim source text and brief content.
  • Extracting Steps from Reasoning Traces: Every trace character belongs to exactly one contiguous, non-overlapping source-text segment, with no edits or omissions.
  • Extracting Steps from Reasoning Traces: Segmentation continues through repeated answers, later checking, abrupt endings, and any available prefix under response limits.
  • Classifying Reasoning Steps: The classifier assigns exactly one dominant cognitive behavior type to each extracted step.
  • Classifying Reasoning Steps: Problem setup translates or reformulates the task before solving, whereas forward reasoning performs direct computation or deduction.
  • Classifying Reasoning Steps: Verification checks an established result, while backtracking explicitly acknowledges failure and pivots to an alternative approach.
  • Classifying Reasoning Steps: Subgoal setting announces an intermediate target, and backward chaining reasons from the desired outcome toward required inputs.
  • Classifying Reasoning Steps: Final answer states the result; other covers repeated results, redundant re-derivations, and filler without new information.
Loading 2604.02007v2…