PaperlayerView exact revision on arXiv

Source-linked AI summary

Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVR

Yongjin Yang, Jiarui Liu, Yinghui He, Lechen Zhang, Bernhard Schölkopf, Zhijing Jin

arXiv:2606.25178v2cs.AI

TL;DR

Multi-domain RLVR curricula are typically fixed or learnability-driven despite uneven cross-domain transfer. TAC selects domains using both local learnability and gradient-based transferability, achieving the best macro-averaged accuracy across two models and six domains, with gains of up to 2.8 points over baselines.

  • Problem

    Multi-domain RLVR lacks curricula that account for whether updates in one domain benefit the others, despite uneven transfer across reasoning tasks.

  • Method

    TAC is an online bandit curriculum combining per-domain learnability with gradient-geometry estimates of cross-domain transferability from training gradients.

  • Results

    +1.8 pp on Qwen3-1.7B and +1.6 pp on Llama3.2-3B over the strongest baseline, with TAC ranking first on 10/14 benchmarks for both.

  • Takeaways & Limitations

    Cross-domain transferability is a useful curriculum signal for multi-domain reasoning RLVR across balanced and imbalanced training mixtures.

  • Takeaways & Limitations

    TAC is evaluated only in RLVR, leaving transferability signals for non-verifiable or model-judged rewards as future work.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has been extended from single-domain training to multi-domain reasoning suites spanning mathematics, programming, and science. However, the training curriculum (how often each domain is sampled) is typically fixed or hand-tuned, even though reasoning skills transfer unevenly across domains. Existing learnability-based curricula adapt to where the policy is currently improving, but are blind to whether a gradient step on the selected domain benefits the remaining domains. In this paper, we propose Transfer-Aware Curriculum (TAC), a bandit-style online curriculum that prioritizes domains whose updates broadly benefit the rest of the training suite. TAC repurposes signals already produced by RL training: per-domain advantages capture local learnability, and projected gradients, taken from the GRPO step being computed, estimate cross-domain transferability via gradient-geometry alignment, at negligible cost (<1% wall-clock overhead). Across a six-domain reasoning suite, TAC achieves the best macro-averaged accuracy on both Qwen3-1.7B and Llama3.2-3B, outperforming proportional random sampling, a hand-designed schedule, and a learnability-only bandit, and improving over the last of these by up to 2.8 points (10% relative). Ablations show performance degrades sharply when the transferability term is removed, and TAC remains robust on imbalanced training mixtures where learnability-only curricula over-commit to dominant domains. Our findings establish cross-domain transferability as a key signal for curriculum design in multi-domain RLVR.

1 Introduction

Multi-domain RLVR reasoning training transfers unevenly across domains, so curricula based only on local learnability can over-select narrowly useful domains. TAC addresses this by combining learnability with gradient-based cross-domain transferability, achieving stronger macro-averaged accuracy across models and benchmarks at negligible overhead.

  • Motivation: Cross-domain transfer varies sharply by source domain, making pooled multi-domain training insufficient for uniformly improving reasoning tasks.The paper reports markedly different transfer profiles across remaining domains under equal single-domain RL-training budgets.
  • Motivation: Existing curricula prioritize domains with high on-policy advantages or reward variance, but do not assess whether their updates benefit other domains.This limitation can cause curricula to over-commit to locally rich but globally narrow domains.
  • Method: TAC formulates domain selection as a multi-armed bandit combining GRPO-advantage learnability with gradient-based transferability through a single coefficient β.The method jointly prioritizes domains that are currently learnable and broadly beneficial to the remaining training suite.
  • Method: TAC estimates transferability from pairwise cosine similarities of per-domain exponential moving averages of projected gradients already generated during RL training.The signal requires no held-out probes, extra rollouts, or oracle annotations.
  • Results: 1.6–2.8 points: TAC improves macro-averaged accuracy across 14 evaluation benchmarks on Qwen3-1.7B and Llama3.2-3B, up to 10% relative, with <1% wall-clock overhead.It outperforms proportional sampling, a hand-designed math-to-others schedule, and a learnability-only bandit across a six-domain suite.

2 Preliminaries

The paper formulates multi-domain RLVR as online domain selection, motivated by the changing usefulness and cross-domain transfer of optimization updates. TAC uses a bandit curriculum whose feedback combines learnability with gradient-based transferability.

  • Motivation: A fixed domain mixture is generally suboptimal because each domain’s optimization signal and transfer to other domains drift during training.The curriculum therefore adapts the sampling distribution online from training history.
  • Bandit formulation: Each reasoning domain is treated as a bandit arm: the curriculum samples a domain, applies one single-domain GRPO update, observes feedback, and updates its value estimate.The pulled arm is refreshed with an exponential moving average, while TAC can also refresh unsampled arms after recomputing transferability.
  • Arm selection: Arms are sampled from a Boltzmann distribution over UCB-augmented values, with τ = 0.85 keeping exploration influential when domain value gaps are small.This revisits domains whose values dipped early rather than starving them.
  • Feedback design: Prior bandit curricula use learnability feedback, prioritizing domains where the policy improves while remaining blind to whether their updates benefit other domains.TAC retains the bandit template but pairs learnability with a gradient-based measure of cross-domain transferability.

3 Method

TAC selects training domains using both local learnability and cross-domain transferability. It estimates transferability from projected GRPO gradient alignment, normalizes the resulting signal across active domains, and incorporates fresh feedback into bandit updates.

  • Transfer-Aware Curriculum: TAC defines its curriculum score as a weighted combination of GRPO-advantage learnability and gradient-based cross-domain transferability.The two terms respectively measure locally available optimization signal and alignment with updates from other domains.
  • Local Learnability: Mean absolute GRPO advantage serves as an on-policy learnability proxy, becoming large during mixed success-failure rollouts and zero when all rewards match.TAC feeds a running z-score of this signal into the curriculum because its raw scale varies across domains and training stages.
  • Projected-Gradient Transferability: TAC sketches GRPO gradients from the last N transformer layers into a shared low-dimensional space, then maintains per-domain exponential moving averages of unit update directions.Random projections reduce the cost of comparing high-dimensional gradients, while normalization prevents unusually large steps from dominating accumulated states.
  • Projected-Gradient Transferability: Every Kc steps, each initialized domain receives a transferability score from its mean cosine similarity with other active domains, temporally smoothed and min–max normalized.The bounded relative score ranks domains by current alignment and uses a floored EMA scale to prevent spikes when cross-domain spread collapses.
  • Bandit Update: At comparison steps, TAC updates unsampled bandit arms using fresh transferability and cached learnability, allowing transfer signals to reallocate sampling mass without revisiting those domains.Arms that have never been pulled lack cached learnability and remain untouched.
  • Transferability Interpretation: The cosine alignment predicts first-order loss changes on other domains while preserving improvement direction and removing gradient-magnitude differences.Relative normalization rewards transfer compared with other active domains, so TAC favors domains that both carry local learning signal and align with the training suite.

4 Experiments

Across six balanced reasoning domains and two model backbones, TAC achieves the strongest macro-averaged accuracy while adapting domain sampling to cross-domain transferability. Its gains persist across benchmarks, model families, and imbalanced mixtures, whereas learnability-only scheduling can over-focus on domains with high local learnability.

  • Experimental setup: The experiments train on six GURU domains capped at 1,000 queries each and evaluate Qwen3-1.7B-Base and Llama3.2-3B-Instruct.The domains are math, codegen, logic, simulation, table, and stem; the cap prevents math and stem from dominating by two orders of magnitude.
  • Main results: +1.8 pp (+5.6% relative) on Qwen3-1.7B and +1.6 pp (+5.4% relative) on Llama3.2-3B are TAC’s macro-accuracy improvements over the strongest baselines, with first place on 10/14 benchmarks for both.SEC is the strongest baseline on Qwen3-1.7B, while M2O is strongest on Llama3.2-3B.
  • Main results: TAC’s gains span logic, math, table, and stem benchmarks rather than concentrating in one target domain.Examples include Zebra +9.8 over Random on Qwen, MATH +5.0 over Random on Llama, FinQA +7.5 over Random on Llama, and GPQA +1.9 over Random on Qwen.
  • Curriculum dynamics: TAC shifts sampling toward transferable domains: table becomes highest in transferability, while learnability-only scheduling remains concentrated on high-learnability stem.Across the run, table is high on both signals and sampled most, logic has low learnability but high transferability, and math is low on both.
  • Ablation: β ≈0.2 is optimal, while β =1 degrades sharply and β =0 is slightly weaker, confirming that transferability contributes beyond learnability.Performance remains stable within ±0.2 points around the default settings across the tested hyperparameters.
  • Imbalanced mixtures: 32.7 macro-averaged accuracy, +1.9 over Random, and +1.5 over SEC show that TAC remains effective under imbalanced training mixtures, ranking first on 11 of 14 benchmarks.The skewed initialization gives math and stem about 1.9× the initial mass of simulation and table.

5 Related Work

Prior work connects multi-domain LLM reasoning to multi-task learning, gradient-conflict mitigation, and data-mixture optimization. RL curricula adaptively select tasks using difficulty or learning-progress signals, with recent LLM-reasoning methods introducing accuracy-based backward transfer and domain-specific schedules.

  • Multi-Domain Reasoning for LLMs: Multi-domain reasoning builds on shared representations, gradient-conflict mitigation, and data-mixture optimization from multi-task learning and language-model pretraining.The paper situates its RL training of LLM reasoners within these established multi-domain training ideas.
  • Curriculum Learning for RL: Curriculum learning in RL typically estimates task difficulty or learning progress to adaptively select tasks, goals, or environments for improved sample efficiency.Recent work extends these curriculum principles to RL training of LLM reasoners.
  • Curriculum Learning for RL: Omni-Thinker updates its sampling distribution using an accuracy-based backward-transfer signal computed on held-out evaluation probes.This is presented as a recent curriculum method for RL training of LLM reasoners.

6 Conclusion

TAC is a bandit-style curriculum for multi-domain RL that combines local learnability with gradient-based cross-domain transferability. Its scope is curriculum design, while its broader impact is enabling more efficient training across diverse reasoning domains.

  • Contribution: TAC combines local learnability with gradient-geometry estimates of cross-domain transferability that evolve with the policy during training.The estimates are computed entirely from training gradients.
  • Contribution: TAC consistently outperforms proportional sampling, a hand-designed math-to-others curriculum, and a learnability-only bandit across a six-domain reasoning suite.The comparison covers both Qwen3-1.7B and Llama3.2-3B-Instruct.
  • Limitations: TAC targets the sampling distribution over training domains rather than the optimizer, reward model, or policy architecture.Its gains are complementary to, rather than substitutes for, advances in those other areas.
  • Broader Impact: TAC supports simultaneous training across broader reasoning portfolios, including scientific reasoning, table understanding, and structured logic.The paper frames this as progress beyond the mathematics-and-code focus of RL post-training.
  • Broader Impact: By improving sample efficiency without introducing new data sources or deployment capabilities, TAC lowers the compute barrier to training competent reasoning models.This may broaden access to reasoning-model training.

Supplementary Material · A Experiment Prompts · B Implementation Details

The supplementary material specifies experiment prompts across mathematics, code generation, logic and visual reasoning, simulation, table reasoning, and STEM, while noting that experiments use VERL2’s default prompt templates. It also details dataset-specific formatting and answer constraints for each task family.

  • A Experiment Prompts: Experiments use the default prompt templates from the VERL2 repository.
  • A.1 Math: Math prompts insert the question and require the final answer within \boxed{}.
  • A.2 Code Generation: Code-generation prompts cover LeetCode2K, LiveCodeBench, PrimeIntellect, and TACO, with variants for starter code and STDIN-based input/output.The prompts require Python solutions enclosed in delimiters and, for STDIN tasks, reading from stdin and writing to stdout.
  • A.3 Logic and Visual Reasoning: Logic and visual-reasoning prompts include ARC-AGI, BARC, graph logical reasoning, ordering puzzles, and zebra puzzles.They provide task-specific inputs or clues and generally require answers in <answer> tags, including grids or structured dictionaries where specified.
  • A.4 Simulation: Simulation prompts address CodeI/O input prediction and output prediction without writing code.Both variants request JSON-formatted answers in \boxed{} tags and provide a reference code snippet as reasoning guidance.
  • A.5 Table Reasoning: Table-reasoning prompts ask models to answer questions using one or more tables and place final answers within \boxed{}.Multiple answers are separated by | when required.
  • A.6 STEM: OPENSCIENCEREASONING-2 uses the source dataset’s input field as-is, without additional template wrapping.

B.1 Datasets

The training suite aggregates curated data across six reasoning domains, using domain-specific sources and replacing GURU’s STEM source with OpenScienceReasoning-23. Fixed-seed subsampling balances the mixture, while binary verifiable rewards use rule-based or execution-based verification.

  • Training data: The suite aggregates GURU-curated data across six domains without reapplying upstream deduplication, heuristic filtering, or difficulty filtering.The curated subsets were already processed using a weak/strong model pass-rate scheme.
  • Per-domain sources: Math combines OR1, DAPO, and DeepScaler; codegen uses LeetCode, TACO-Verified, PrimeIntellect, and LiveCodeBench.PrimeIntellect and LiveCodeBench subsets come from the pre-filtered DeepCoder release.
  • Per-domain sources: Logic uses ARC-AGI-1/2, BARC, and three GURU-synthesized tasks, while simulation uses PyEdu Code I/O and table uses HiTab and MultiHierTT.The synthesized logic tasks are Zebra Puzzle, Ordering Puzzle, and Graph Puzzle; tables are linearized into markdown.
  • Per-domain sources: STEM replaces WebInstruct-Verified with NVIDIA’s OpenScienceReasoning-23, enabling higher-quality reasoning traces and rule-based answer matching instead of a 1.5B verifier model.The replacement source is formatted as multiple-choice questions with ground-truth answers, eliminating verifier-model noise from STEM rewards.
  • Reward design: All domains use binary verifiable rewards, with rule-based matching for five domains and execution-based verification for generated programs under a 30-second timeout and 10 GB memory limit.Rule-based matching extracts answers from \boxed{} or <answer> tags; execution-based verification runs code against test cases.
  • Subsampling: A fixed seed applies two-stage subsampling because raw data are dominated by math and STEM by two orders of magnitude, ensuring every method trains on identical data.Each source is first sampled at ratio 0.2, then pooled by domain and capped using per-domain budgets.

B.2 Training Details · B.3 Evaluation Details · C More Experiments

The paper specifies TAC’s bandit and GRPO/DAPO training implementation, then evaluates saved checkpoints with unified generation, domain-wise aggregation, and three-seed averaging. Training uses shared configurations across curricula, while only data sampling differs.

  • B.2 Training Details: All curricula use the same DAPO configuration with GRPO-style updates; only the data-sampling component differs between TAC, Random, Math-to-Others, and SEC.DAPO uses asymmetric clipping, dynamic batching, grouped advantages, no KL regularization, and dynamic per-GPU token budgets.
  • B.2 Training Details: TAC samples six domains with centered log-proportional Q-value initialization, five round-robin warmup rounds per epoch, and pool resets when data is exhausted.High-Q domains may be oversampled within an epoch, but no data is permanently withheld across epochs.
  • B.2 Training Details: TAC updates unsampled arms using transferability recomputed at each comparison step and cached normalized learnability from each arm’s most recent pull.Arms without a prior selection lack cached learnability and are skipped until first pulled.
  • B.2 Training Details: Training uses Qwen3-1.7B-Base and Llama3.2-3B-Instruct with FSDP, full parameter and optimizer offloading, synchronous vLLM generation, and validation every 20 update steps.The validation schedule begins before the first training step.
  • B.3 Evaluation Details: Held-out evaluation runs offline on saved checkpoints after converting sharded models to HuggingFace format and generating all benchmark responses in one vLLM session.Generation outputs are cached, enabling scoring to be rerun without regenerating responses.
  • B.3 Evaluation Details: Benchmarks use domain-specific verifiable scoring, including exact match for math, unit tests for code generation, structural parsing for logic and tables, and answer extraction for STEM.Sampling uses temperature 1.0 and top-p 1.0, matching training-time sampling.
  • B.3 Evaluation Details: Overall accuracy is the unweighted mean across six domain means, with every reported number averaged over three training seeds and identical data subsampling across methods.The seeds vary bandit sampling sequences while the data subsampling seed remains fixed.

C.1 More Ablation Study

Additional ablations show that TAC performs best with its default Rademacher projection at dimension r=4096 and with reset-based handling of exhausted domains. These choices outperform alternative projection families, smaller dimensions, and redistribution during training.

  • Projection method: Rademacher projections outperform CountSketch and Gaussian by roughly 0.5–0.7 points in macro-averaged accuracy.CountSketch and Gaussian remain within 0.2 points of each other.
  • Projection dimension: Macro accuracy improves through r=4096, while r=8192 provides no further gain and smaller dimensions lose roughly a point.The sweep covers r ∈{512, 1024, 2048, 4096, 8192}.
  • Data-exhaustion strategy: Reset converges to higher validation accuracy than redistribute throughout training, with the gap widening as more domains exhaust.Reset reshuffles a depleted domain’s data, whereas redistribute re-draws from the remaining domains.

C.2 Complexity Analysis · C.3 A Pairwise View of Transferability

TAC adds negligible training cost while using weak but stable relative pairwise gradient alignment to identify domains whose updates transfer across the mixture. Its transferability signal therefore distinguishes broadly connected domains from isolated ones without materially changing the training budget.

  • C.2 Complexity Analysis: 1.06 s adds 0.9% overhead to a 115 s GRPO step, with projection accounting for the cost while bandit updates are effectively free.Rollout generation and the forward/backward/optimizer pass remain the dominant costs.
  • C.2 Complexity Analysis: 2 ms standard deviation across domains on a 1.06 s mean shows that projection time is stable across sampled minibatch domains.The projection cost depends on parameter count and projection dimension rather than batch composition or response length.
  • C.3 A Pairwise View of Transferability: Table is a transfer hub, showing positive alignment with simulation, stem, and logic and consequently the highest aggregate transferability signal.Its strongest bonds are with simulation and stem.
  • C.2 Complexity Analysis: 7.7 × 10^10 FLOPs at r=1024 and approximately 3.1 × 10^11 at r=4096 make projection arithmetic negligible relative to the optimizer step.The bandit Q-update costs O(M) scalar operations across M=6 domains, while measured wall-clock overhead is dominated by GPU–CPU transfer and Python bookkeeping.
  • C.2 Complexity Analysis: Sub-1% wall-clock overhead and a 10^-5 relative analytic FLOP cost support TAC as a drop-in replacement for fixed-mixture or learnability-only sampling.The measurements confirm stability across domains and batch compositions without modifying the training budget.
  • C.3 A Pairwise View of Transferability: Nearly all pairwise projected-gradient cosine similarities remain within |cos| ≲0.15, indicating weak absolute gradient agreement across domains.Cross-domain normalization is essential because raw cosine magnitudes are small, whereas relative ordering remains robust enough for curriculum use.
  • C.3 A Pairwise View of Transferability: Math and codegen have negative cosine with almost every other domain, with their mutual cosine the most negative pair and their aggregate transferability signals the lowest.Their updates are described as idiosyncratic, so TAC samples them least.

C.4 Scaling Experiments

TAC’s gains generalize across Qwen3 model scales, outperforming Random and SEC on both smaller and larger backbones. Its advantage over SEC remains comparable across scales, while its improvement over Random persists at every tested scale.

  • C.4 Scaling Experiments: 21.6 macro accuracy on Qwen3-0.6B-Base places TAC above 18.1 (Random) and 19.9 (SEC), with first-place results on 9/14 benchmarks.The corresponding gaps are +3.5 versus Random and +1.7 versus SEC.
  • C.4 Scaling Experiments: 45.4 macro accuracy on Qwen3-4B-Base exceeds 43.2 (Random) and 43.8 (SEC), ranking TAC first on 10/14 benchmarks.The corresponding gaps are +2.2 versus Random and +1.6 versus SEC.
  • C.4 Scaling Experiments: +1.6 to +1.8 gains over SEC remain comparable across scales, while TAC improves over Random by +2.1 to +3.5 at every tested scale.The comparison includes the 1.7B results from Table 1, alongside Qwen3-0.6B-Base and Qwen3-4B-Base.

D More Examples

Illustrative rollouts show TAC-trained models solving representative algebra, particle-physics, and constraint-satisfaction problems more reliably than Random-trained models. The examples are qualitative rather than systematic.

  • Example 1: Algebra: TAC correctly identifies all four quartic roots as 6th roots of unity, whereas the Random-trained model concludes they are 12th roots.The Random response makes an argument error when identifying a cube root of unity; TAC avoids this mistake.
  • Example 3: Constraint Satisfaction: 89–100% cell-level accuracy across rollouts: TAC returns the requested JSON schema correctly in all four rollouts, while Random averages ∼36% cell-level accuracy.Random rollouts fail to propagate parity constraints, overwrite the schema, or invent extra rows; TAC produces the correct grid.
Loading 2606.25178v2…