PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?

Chuxuan Hu, Yuxuan Zhu, Antony Kellermann, Caleb Biddulph, Suppakit Waiwitlikhit, Jason Benn, Daniel Kang

arXiv:2506.19733v3cs.CL

TL;DR

The paper examines whether reinforcement post-training gains generalize beyond the domains used for training, a question prior evaluations have addressed mainly within training domains. It combines observational and single-domain interventional studies of RPT, finding that transfer is inconsistent and depends on similarity in reasoning patterns.

  • Problem

    Prior evaluations largely test RPT within training domains, leaving its generalizability to unseen domains understudied.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The paper combines observational comparisons of open-weight RPT models with base models and interventional RPT post-training on single domains, evaluating across domains.

    Source: §1 · pp. 1–2

  • Results

    RPT gains transfer between structured domains such as math and code, but structured-to-unstructured transfer is limited, while unstructured-domain gains can transfer to structured domains.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    RPT is most effective when target tasks share reasoning patterns with post-training data, and its benefits do not generalize broadly to unseen domains.

    Source: §1 · p. 2

    The study treats math and code as structured reasoning and knowledge-intensive tasks as unstructured reasoning, a distinction whose rationale is quantified in Appendix B.

    Source: §2 · p. 3

Abstract

from arXiv · show

Reinforcement post training (RPT) has recently shown promise in improving the reasoning abilities of large language models (LLMs). However, it remains unclear how well these improvements generalize to new domains, as prior work evaluates RPT models on data from the same domains used for post-training. To understand the generalizability of RPT, we conduct two studies with specific focus on Reinforcement Learning with Verifiable Rewards (RLVR). (1) Observational: we compare a wide range of open-weight RPT models against their corresponding base models across multiple domains, including both seen and unseen domains in their fine-tuning data. (2) Interventional: we fine-tune LLMs with RPT on single domains and evaluate their performance across multiple domains. Both studies converge on the same conclusion that, although RPT brings substantial gains on tasks similar to the fine-tuning data, the gains generalize inconsistently and can vanish on domains with different reasoning patterns.

1 INTRODUCTION

The introduction asks whether RPT gains generalize beyond post-training domains and presents observational and interventional studies to investigate that question. It reports that transfer depends on reasoning-pattern similarity, with limited generalization to domains requiring different patterns.

1 INTRODUCTION

  • The study combines an observational comparison of RPT models with their base models and an interventional study that post-trains on individual domains.The framework focuses on Reinforcement Learning with Verifiable Rewards (RLVR).Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The observational study evaluates 18 open-weight RPT models with publicly disclosed post-training data across seen and unseen domains, including legal, financial, and medical benchmarks.Prior work evaluated RPT models within their post-training domains.Source: §1 · p. 1
  • Mixed-domain post-training in existing models makes it difficult to isolate how RPT generalizes at a finer granularity.The interventional study addresses this by post-training on math, coding, or knowledge-intensive reasoning data separately.Source: §1 · p. 2
  • RPT gains transfer between structured domains such as math and code, but structured-domain gains fail to generalize to unstructured domains; unstructured-domain gains can transfer to structured domains.The reported pattern is asymmetric across reasoning structures.Source: §1 · p. 2
  • RPT is most effective when the target task shares reasoning patterns with the post-training data, while benefits remain limited across different patterns and unseen domains.The introduction frames this as a boundary on RPT's generalizability.Source: §1 · p. 2

2 BACKGROUND

The background motivates studying RPT beyond training domains and organizes reasoning tasks by domain and reasoning structure. It distinguishes structured math and code tasks from flexible, context-sensitive knowledge-intensive tasks.

2 BACKGROUND

  • Recent RPT models are described as achieving strong results in mathematics, coding, and scientific reasoning benchmarks.The background gives examples including 100% accuracy on AIME 2025, 81% on SWE-bench Verified, and around 93.2% on GPQA Diamond.Source: §2 · p. 2
  • Out-of-domain generalizability of RPT remains understudied, and prior evaluations have largely stayed within training domains.Mixed-domain training makes generalizability to unseen domains difficult to assess, although many reasoning tasks are underrepresented or absent from training corpora.Source: §2 · pp. 2–3
  • The study focuses on code, math, and knowledge-intensive reasoning, with the latter including legal, financial, and medical subdomains.It examines both high-level domains and variation across subdomains.Source: §2 · p. 3
  • Math and code are treated as structured reasoning because they use deterministic logical steps and require precise syntax and formal semantics.Knowledge-intensive tasks are treated as unstructured when they require flexible, context-sensitive reasoning without a fixed sequence of operations or well-defined intermediate representation.Source: §2 · p. 3
  • Knowledge-intensive tasks may require broad world knowledge, interpretive judgment, and handling ambiguity or incomplete information.Examples include interpreting lengthy legal or financial documents and evaluating conflicting evidence.Source: §2 · p. 3

3 STUDY DESIGN

The study examines cross-domain, reasoning-structure, within-domain, and stability questions through observational comparisons of RPT models with their base models and controlled single-domain post-training experiments. It evaluates these studies across math, code, and knowledge-intensive benchmarks using aggregated accuracy changes and statistical tests.

3 STUDY DESIGN

  • The design asks whether RPT transfers across domains and reasoning structures, across subdomains, and consistently across settings, using observational and controlled interventional studies.The observational study compares 18 RPT models with their corresponding base models; the interventional study trains three models from one base model on disjoint single-domain datasets.Source: §3 · p. 3

3.1 EVALUATION SETTINGS

  • Evaluation covers 16 benchmarks across math, code, and knowledge-intensive reasoning, including medical, legal, and financial tasks and varying difficulty levels.Generation uses consistent hyperparameters; AMC 2023 and AIME 2024 are run 16 times, while other benchmarks are run once.Source: §3.1 · p. 4
  • The study reports aggregated accuracy improvement by domain and uses a Cochran–Mantel–Haenszel test to assess the association between RPT and accuracy improvement.Benchmarks are treated as independent strata; odds ratios above 1 indicate improvement, and reported asterisks mark p < 0.05.Source: §3.1 · p. 4

3.2 EXPERIMENTAL SETTINGS

  • The observational sample is filtered from public model listings, manually verified, and finalized at 18 RPT models with publicly disclosed training data and corresponding base models.The selection process starts with 466 models and manually verifies 31 candidate RPT models before final selection.Source: §3.2 · p. 5, Table 1 · p. 5
  • The interventional study trains one model per disjoint math, code, or knowledge-intensive dataset, cleaning training data to avoid overlap with evaluation sets.The math and code datasets each sample 40,000 problems; the knowledge-intensive set contains 40,000 non-math, non-code examples.Source: §3.2 · p. 5
  • The three RPT runs share GRPO settings and training hyperparameters, with additional experiments varying the algorithm, training duration, or base model.Alternatives include DAPO, two epochs with intermediate checkpoints, and Llama-3.2-3B-Instruct as the base model.Source: §3.2 · pp. 5–6

4 FINDINGS

RPT gains are strongest on tasks sharing the training domain or reasoning patterns, while transfer across different reasoning patterns is limited. This pattern persists across training configurations and model sizes.

4.1 RPT GAINS DO NOT GENERALIZE TO ARBITRARY UNSEEN DOMAINS

  • ID pass@1 rises 2.87% while OOD pass@1 falls 3.19% across existing RPT models; single-domain models likewise show no significant OOD gains.The controlled study also finds significant OOD drops for Math-RPT and Code-RPT, but no significant OOD gains for Knowledge-RPT.Source: §4.1 · pp. 6–7, Table 2 · p. 7, Figure 2 · p. 7

4.2 RPT GAINS GENERALIZE ACROSS DOMAINS WITH SIMILAR REASONING PATTERNS

  • Math/code transfer is mutual, with math-trained models gaining 2.18% on math and 4.77% on code; math-to-code transfer is stronger than the reverse.The interventional analysis finds math-to-code transfer notably stronger and more consistent than code-to-math transfer.Source: §4.2 · p. 7, Figure 3 · p. 8, Figure 4 · p. 8
  • Structured-trained models average -0.27% pass@1 on knowledge-intensive tasks, versus gains of 11.08% on math and 5.82% on code.The interventional results also show knowledge-intensive task drops for Math-RPT and Code-RPT.Source: §4.2 · pp. 7–8, Figure 3 · p. 8, Figure 4 · p. 8
  • Knowledge-intensive RPT transfers toward structured tasks, with observational gains of 21.40% on math and 12.16% on code.Intervention finds significant math gains and no noticeable code degradation, despite underperformance on knowledge-intensive tasks.Source: §4.2 · p. 8, Figure 3 · p. 8, Figure 4 · p. 8

4.4 RPT GENERALIZABILITY REMAINS WEAK ACROSS CONFIGURATION VARIANTS

  • Math-domain gain gaps are similar across GRPO versus DAPO and DeepSeek-R1-Distill-Qwen-1.5B versus Llama-3.2-3B-Instruct.The authors interpret the consistency as evidence that weak generalizability is not specific to the compared algorithm or base model.Source: Table 3 · p. 9, §4.4 · p. 9

4.3 INTRA-DOMAIN RPT GAINS DEPEND ON SUBDOMAIN STRUCTURAL SIMILARITY

  • Structured models generalize within their domain, whereas knowledge-intensive RPT can significantly underperform on other knowledge-intensive tasks.For Fino1-8B, pass@1 drops from 3.26% to 1.28% on PubMedQA, from 6.42% to 4.84% on LegalBench, and from 64.18% to 48.39% on TabFact.Source: §4.3 · p. 9

4.4 RPT GENERALIZABILITY REMAINS WEAK ACROSS CONFIGURATION VARIANTS

  • As math RPT training progresses, the in-/out-of-domain gain gap widens and eventually stabilizes.The authors attribute stabilization from a full epoch onward to repeated training on the same finite data collection.Source: §4.4 · p. 9, Figure 5 · p. 10
  • As model size grows, average in-domain gain increases 16.5% more than out-of-domain gain.The authors report that larger models do not deliver better generalizability.Source: §4.4 · p. 10

5 RELATED WORK

Prior RPT work spans structured and unstructured tasks, but commonly evaluates within one domain or task type. This paper examines data generalizability across training and inference.

5 RELATED WORK

  • Prior RPT work covers structured tasks such as math and coding and unstructured tasks such as search use and open-ended question answering, but typically evaluates within one domain or task type.Source: §5 · p. 10
  • This study focuses on RPT data generalizability across training and inference, alongside broader concerns about training alternatives, test-time compute, and lengthy reasoning.Source: §5 · p. 10

6 CONCLUSION

Across observational and interventional studies, RPT yields substantial gains within training domains, but generalization to unseen domains is limited, with little transfer to unstructured domains.

6 CONCLUSION

  • RPT produces substantial improvements within training domains, but generalization to unseen domains is limited.This pattern is consistent across the paper’s observational and interventional studies.Source: §6 · p. 10
  • The paper finds evidence of cross-domain transfer between structured domains such as math and code, but little evidence of transfer to unstructured domains.Source: §6 · p. 10
  • The findings motivate a more nuanced understanding of cross-domain knowledge transfer in LLMs.Source: §6 · p. 10

B REASONING TEMPLATE ANALYSIS

The analysis tags reasoning traces and compares their step-type distributions across and within domains; math and code patterns are more similar to each other than to knowledge-intensive reasoning, which also varies more across datasets.

B REASONING TEMPLATE ANALYSIS

  • For each evaluation benchmark, researchers randomly sampled min(dataset size, 100) tasks (1,470 total), had Claude Sonnet 4.5 solve them, and GPT-4o tag trace steps; Jeffreys divergence measured symmetric differences in step-type distributions within and across domains.Source: B REASONING TEMPLATE ANALYSIS · p. 18
  • EXECUTE_STEP dominates all three domains; PLAN is especially prominent in math and code, while SETUP is comparatively more prominent in knowledge-intensive tasks.Almost no steps are labeled OTHER, suggesting the tagset covers the reasoning process well.Source: B REASONING TEMPLATE ANALYSIS · p. 18, Figure 6 · p. 19
  • The analysis uses normalized domain-level reasoning-step distributions and pairwise dataset comparisons to quantify template differences within and across domains.The distributions are used to quantify the paper’s cross-domain hypothesis.Source: B REASONING TEMPLATE ANALYSIS · p. 18, §B.1 · p. 18, §B.2 · p. 18

B.1 MATH AND CODE DOMAINS SHARE SIMILAR REASONING TEMPLATES (§4.2)

  • Jeffreys divergences are 0.18 for math–code, 0.29 for math–knowledge-intensive, and 0.69 for code–knowledge-intensive, indicating greater template similarity between math and code.Source: §B.1 · p. 18

B.2 KNOWLEDGE-INTENSIVE DOMAIN HAS LARGER INTRA-DOMAIN VARIANCE (§4.3)

  • Average within-domain Jeffreys divergences are 0.15 for math, 0.14 for code, and 0.19 for knowledge-intensive tasks, indicating greater reasoning-template diversity in the latter.Source: §B.2 · p. 18

C RPT GENERALIZABILITY ON MODELS ACROSS DIFFERENT SIZES

  • Across model-size variants, larger models show no clear generalizability improvement, and several show reduced generalizability as size increases.The authors attribute the Fino family’s exception to a base-model change rather than size; Table 4 describes the in-domain versus out-of-domain comparison.Source: C RPT GENERALIZABILITY ON MODELS… · p. 19, Table 4 · p. 19

D.1 REWARD DEFINITIONS

  • Domain-specific binary rewards mark math answers correct by equivalence, code by passing unit tests, and knowledge answers by matching the ground-truth string.Manual checks of 100 training examples found reward correctness rates of 94% for math, 99% for code, and 97% for general knowledge.Source: §D.1 · pp. 19–20
  • Over 625 training steps, 50-step moving-average rewards rose 9.7% for math, 21.7% for code, and 71.0% for knowledge-intensive tasks.Figure 7 presents training reward trajectories for the three domains.Source: Figure 7 · p. 20

D.2 OPTIMIZATION DYNAMICS

Across 625-step single-domain RPT runs, moving-average training rewards increased in all three domains, with the largest increase in knowledge-intensive tasks.

D.2 OPTIMIZATION DYNAMICS

  • During RPT training, 50-step average training rewards increased by 9.7% in math, 21.7% in code, and 71.0% in knowledge-intensive domains, with increases across all domains.Source: §D.2 · p. 20
  • Training rewards increased noticeably in all three domains over the course of RPT training.Source: §D.2 · p. 20
  • The authors interpret the reward increases as evidence that single-domain RPT learned domain-specific reasoning techniques, improving task-specific performance.Source: §D.2 · p. 20

E MIXED DOMAIN TRAINING

A uniformly sampled mixed-domain RPT dataset yields a reported accuracy improvement over individual single-domain RPT models; the supplied appendix tables identify additional observational and interventional result sets.

E MIXED DOMAIN TRAINING

  • The mixed-domain RPT model improves overall accuracy by 2.7 percentage points over individual single-domain RPT models.The authors describe this result as a lower bound because the 13,333 examples per domain were uniformly sampled without optimization; Table 5 reports the comparison with the best single-domain model.Source: E MIXED DOMAIN TRAINING · p. 20

F.1 OBSERVATIONAL STUDY RESULTS

  • Table 6 reports observational study results, with model IDs corresponding to those in Table 1.Source: Table 6 · p. 21

F.2 INTERVENTIONAL STUDY RESULTS

  • Table 7 contains full interventional evaluation results across models and benchmarks.Source: Table 7 · p. 21

G RELATIVE IMPROVEMENT

Relative improvements make the domain-wise performance gaps from RPT clearer and reinforce the conclusion that gains generalize inconsistently. The metric also shows that changing RPT variants does not remove those gaps, while relative performance rises and then stabilizes as training continues.

G RELATIVE IMPROVEMENT

  • Relative improvement complements absolute aggregated improvement with proportional gains that remain well-defined when base-model accuracy is zero.The paper introduces the relative metric as a secondary way to compare improvements.Source: G RELATIVE IMPROVEMENT · p. 22
  • 256.71% relative gap: in-domain accuracy increases more than 3.5× as much as out-of-domain accuracy in the observational study.The relative-improvement contrast is sharper than the 6.07% absolute accuracy improvement reported for the primary metrics.Source: G RELATIVE IMPROVEMENT · p. 22
  • Relative improvements show larger domain separations than accuracy-based results in both observational and interventional settings.The pattern is reported across Figures 8–10 and compared with the corresponding absolute-result figures.Source: G RELATIVE IMPROVEMENT · p. 22, Figure 8 · p. 22, Figure 9 · p. 23
  • 28.38% for the Llama-based configuration versus 8.75% for the Qwen-based configuration in relative accuracy gaps, despite absolute gaps of 4.94% versus 5.23%.The paper reports that adjusting RPT implementation variants does not improve generalizability.Source: G RELATIVE IMPROVEMENT · p. 22, Table 9 · p. 23
  • Relative performance first increases and then stabilizes as RPT epochs grow, making the training-domain overfitting pattern more pronounced.The relative-improvement view shows this growth and convergence more clearly than the two primary metrics.Source: G RELATIVE IMPROVEMENT · p. 23, Figure 11 · p. 23

H CHECKPOINT EVALUATIONS AT 100-STEP INTERVALS

At 100-step intervals during math-domain RPT training, in-domain gains rise while out-of-domain gains fall. Their gap widens as training advances, then the trends stabilize, particularly after the first epoch.

H CHECKPOINT EVALUATIONS AT 100-STEP INTERVALS

Related papers

Loading 2506.19733v3…