PaperlayerView exact revision on arXiv

Source-linked AI summary

Growing and Transferring Pathways for LLM Reinforcement Learning

Haolin Tian, Yikang Li, Tonghan Wang

arXiv:2610.16489v1cs.LGcs.AI

TL;DR

Resource-intensive reinforcement learning produces updates that may not remain useful beyond the trained model. GrowRL stores RL updates as compact attention and feed-forward pathways that can be transferred to compatible models. Across five backbones, the approach yields competitive mathematical and out-of-distribution performance, with transferred pathways providing immediate gains and rapid adaptation.

  • Problem

    Reinforcement-learning runs can improve reasoning but require considerable resources, and separately stored updates are not necessarily effective on representations from another model.

    Source: §1 · p. 1

  • Method

    GrowRL freezes the pretrained backbone and trains separate attention and feed-forward pathways that can be inserted into structurally compatible models.

    Source: §1 · pp. 1–2

  • Results

    Across five backbones, GrowRL achieves competitive mathematical and out-of-distribution performance, while transferred pathways provide immediate gains and rapid adaptation.

    Source: §4.2 · p. 6, §4.3 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    Explicit pathways offer a way to preserve and reuse the outcome of an RL run across successive stages of model adaptation.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

Reinforcement learning (RL) can substantially improve the reasoning capabilities of large language models, but an effective training run requires considerable resources. Reusing the computation acquired through RL across models would extend the value of that training. We propose GrowRL, an RL method that organizes learned updates into compact computational pathways through structural growth. GrowRL freezes the pretrained backbone and adds an independent causal attention branch and a feed-forward network branch at each Transformer layer. Zero-initialized output projections preserve the pretrained policy at initialization, and RL updates only the added pathways. The learned pathways remain explicit after training, allowing transfer to structurally compatible target models and subsequent adaptation with the target backbone frozen. The architecture also supports sequential transfer of additional pathways. By keeping the learned attention and feed-forward computation separate from the pretrained weights, GrowRL provides an explicit structure for preserving and reusing the outcome of an RL run. Experiments across five backbones show competitive mathematical reasoning and out-of-distribution performance with a small number of trainable parameters. Learned pathways provide immediate gains across all three transfer directions, with performance after only 10 target-side updates already close to that after 100 updates in every case. These results support computational pathways as parameter-efficient and reusable units of reinforcement learning adaptation.

1 INTRODUCTION

GrowRL addresses the challenge of making resource-intensive reinforcement-learning updates useful beyond the model trained. It keeps learned attention and feed-forward pathways separate from the frozen backbone, enabling transfer to compatible models and further adaptation.

1 INTRODUCTION

  • A resource-intensive reinforcement-learning run could benefit other models if its learned computation remains effective on representations produced by a receiving backbone.Reusable updates must improve the original model within a limited trainable budget and retain value after transfer to a compatible model.Source: §1 · p. 1
  • GrowRL freezes the pretrained backbone and trains independent attention and feed-forward branches whose learned updates remain identifiable and insertable into structurally compatible models.Zero-initialized output projections preserve the original model’s behavior at the start of training.Source: §1 · pp. 1–2
  • Across three compatible source–target pairs, transferred pathways improve MATH500 Pass@1 before target-side training and remain useful during further adaptation.On original hosts, the pathways also yield reasoning improvements while training only a small number of added parameters, with performance competitive with full-parameter reinforcement learning across multiple settings.Source: §1 · p. 2
  • GrowRL makes learned computation explicit and modular, allowing pathways to be retained independently, adapted in a receiving model, and extended through sequential transfer.The pathways remain separate from the original backbone parameters.Source: §1 · p. 2

2 RELATED WORK

Prior work explores parameter-efficient reinforcement learning and modular computational structures. GrowRL focuses on preserving RL-acquired computation in distinct pathways that can be transferred, adapted, and composed.

2.1 REINFORCEMENT LEARNING AND PARAMETER-EFFICIENT ADAPTATION

  • RL and parameter-efficient adaptation methods show that updates can be concentrated in a relatively small parameter space, making the organization of trainable parameters an important design dimension.The cited approaches include adapters, LoRA, DoRA, and Tina’s reinforcement-learning updates.Source: §2.1 · p. 2

2.2 MODULAR LEARNING AND COMPUTATIONAL STRUCTURE EXPANSION

  • Adapters, Progressive Networks, LLaMA-Pro, and Tiny-Attention Adapter illustrate modular learning or model expansion for adaptation, composition, continual learning, or contextual computation.These approaches introduce or compose computational structures for different purposes.Source: §2.2 · pp. 2–3
  • GrowRL distinguishes itself by preserving RL-acquired computation in attention and feed-forward pathways that remain separate from the frozen backbone and support transfer, adaptation, and sequential insertion.The related work describes earlier computational structures for task adaptation, module composition, and continual learning.Source: §2.2 · p. 3

3 GROWRL

GrowRL adds attention and feed-forward pathways to a frozen source backbone, then transfers learned pathways to structurally compatible models. Its design also supports sequential insertion and target-side adaptation with the backbone frozen.

3 GROWRL

  • GrowRL learns a pathway on a frozen source backbone and transfers it to a structurally compatible target backbone.Each Transformer layer receives an attention branch and an FFN branch, as illustrated by source training, single transfer, and sequential transfer.Source: §3 · p. 3, Figure 1 · p. 4

3.1 STRUCTURAL GROWTH AND SOURCE-MODEL TRAINING

  • Each added attention and FFN branch shares the corresponding backbone sublayer’s normalized input, and both branches return outputs to the d-dimensional residual stream.The attention branch has its own query, key, value, and output projections; the FFN branch is a narrow SwiGLU transformation, with dh = r = 128 in evaluated models.Source: §3.1 · p. 3
  • Zero-initialized output projections make the added branches output zero initially, while training updates the pathways and keeps the backbone projections, normalizations, embeddings, and language-model head fixed.The branch matrices contain approximately 38.5M trainable parameters for the 1.5B models; source-model training uses Dr. GRPO with eight responses per prompt and binary final-answer correctness rewards.Source: §3.1 · pp. 3–4

3.2 TRANSFER AND SEQUENTIAL TRANSFER

  • Transfer inserts only pathway components into corresponding target layers; compatibility requires matching insertion sites and projection dimensions, while the target keeps its own backbone components.Zero-update evaluation measures immediate transfer, and optional target-side RL adapts the inserted pathway while the target backbone remains frozen.Source: §3.2 · p. 4
  • Sequential transfer inserts a source-trained pathway into a model that already contains a trained pathway, with new normalizations trained alongside the new pathway while earlier parameters remain frozen.The resulting residual-step order is attention–attention–FFN–FFN; further rounds add another attention and FFN residual step.Source: §3.2 · pp. 4–5

4 EXPERIMENTS

Experiments compare GrowRL across five backbones and reasoning tasks, then test pathway transfer, rapid adaptation, sequential transfer, and pathway design choices. The reported results show competitive source-model performance and useful transfer, with outcomes varying by backbone and setting.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers five backbones, five mathematical datasets, six out-of-distribution datasets, and sampled Pass@1 and Pass@8.Methods share training data, rewards, and evaluation protocols.Source: §4.1 · p. 5

4.2 SOURCE-MODEL REINFORCEMENT LEARNING

  • GrowRL is competitive on mathematical reasoning and out-of-distribution tasks across five backbones, approaching Full RL in many settings with few added parameters.Results span general-purpose and mathematics-specialized models at 1.5B, 3B, and 7B scales.Source: Table 1 · p. 6, §4.2 · pp. 6–7, Table 2 · p. 7

4.3 TRANSFER AND RAPID ADAPTATION

  • Across all three transfer directions, GrowRL improves MATH500 without target-side training, outperforms direct LoRA transfer, and gains extend to out-of-distribution tasks.The target backbone remains frozen during immediate transfer.Source: §4.3 · p. 7, Table 3 · p. 8
  • After 10 target-side updates, GrowRL’s MATH500 Pass@1 is already close to its update-100 performance on Qwen2.5-1.5B and Llama-3.2-3B.It remains above LoRA at measured checkpoints; the target backbone is frozen.Source: §4.3 · p. 7, Figure 2 · p. 8
  • Qwen2.5-1.5B rises from 23.03% after one transfer to 42.15% at two; Qwen2.5-7B rises from 39.53% to 53.28% across one to four.Llama-3.2-3B instead peaks at one transfer and declines thereafter; useful transfer count depends on the receiving model.Source: Table 4 · p. 8, §4.3 · p. 8

4.4 FINE-GRAINED ANALYSIS

  • Full-pathway training improves all five mathematical benchmarks on Qwen2.5-Math-1.5B and FineMath-Llama-3B, but results are mixed on Qwen2.5-Math-7B.The comparison is against training only the attention output and FFN down projections.Source: Table 5 · p. 9, §4.4 · p. 9
  • GrowRL’s VAL300 Pass@1 generally increases with transfer strength, and full strength is best among evaluated settings despite local non-monotonic changes.The gain is substantially larger than LoRA’s in the reported comparison.Source: Figure 3 · p. 9, §4.4 · p. 9

5 CONCLUSION

GrowRL learns compact attention and feed-forward pathways on frozen backbones, with competitive mathematical and out-of-distribution performance across five backbones. Transferred pathways provide immediate gains and rapid adaptation, supporting reusable units of RL adaptation.

5 CONCLUSION

  • GrowRL achieves competitive mathematical and out-of-distribution performance across five backbones while training a small number of pathway parameters.The findings show that computation acquired through an expensive RL run need not remain tied to one model.Source: §5 · p. 9
  • Across five backbones, GrowRL achieves competitive mathematical and out-of-distribution performance, while transferred pathways provide immediate gains and rapid adaptation, making computation acquired through expensive RL more explicit, modular, and transferable.Source: §5 · p. 9
  • GrowRL makes learned computation explicit, modular, and transferable as a reusable unit of RL adaptation.Source: §5 · p. 9

A.1 SOURCE-MODEL TRAINING

Source-model training uses Dr. GRPO on MATH problems, with a 500-update budget and frozen backbones. GrowRL trains added attention and feed-forward pathways, while the experiments compare it with full RL and parameter-efficient alternatives.

A.1 SOURCE-MODEL TRAINING

  • Post-transfer adaptation trains only transferred components for 100 target-side updates while keeping the target backbone frozen.Immediate and sequential transfer use no target-side training.Source: §A.1 · p. 13
  • Source-model reinforcement learning uses 5,585 Level 3–5 MATH problems and a standard budget of 500 updates.Each prompt receives eight responses; rewards indicate final-answer correctness, and the setup uses no KL penalty.Source: §A.1 · p. 13
  • The comparison includes Full RL, rank-32 LoRA and DoRA, and a width-192 ReLU Parallel Adapter alongside the FFN.PA is distinct from GrowRL’s sequential-transfer construction.Source: §A.1 · p. 13
  • GrowRL adds one width-128 attention head and one width-128 SwiGLU branch per layer, with zero-initialized output projections and a frozen backbone.The pathway has approximately 38.5M, 77.1M, and 89.9M trainable parameters for 1.5B, 3B, and 7B backbones, respectively.Source: §A.1 · p. 13
  • DoRA results characterize the shared BF16 setup rather than its best performance under a separately optimized precision configuration.The checkpoint audit found that BF16 magnitude parameters changed little or not at all despite updates to their FP32 master copies.Source: §A.1 · p. 13
  • Experiments use NVIDIA A800 GPUs with 80 GB of memory and distribute optimizer states and gradients with ZeRO-2.Forward and backward computation predominantly use BF16, while master parameters and Adam optimizer states use FP32.Source: §A.1 · p. 13

A.2 EVALUATION PROTOCOLS

Evaluation covers mathematical benchmarks and out-of-distribution tasks, reporting sampled Pass@1 and Pass@8. Transfer experiments include immediate transfer, target-side adaptation, and sequential transfer across specified compatible source–target pairs.

A.2 EVALUATION PROTOCOLS

  • Mathematical evaluation covers MATH500, AMC2023, AIME2024, Minerva, and OlympiadBench; out-of-distribution evaluation spans GSM8K, HumanEval, MMLU, ARC-Challenge, and HellaSwag.Source: §A.2 · pp. 13–14
  • Pass@1 is mean response correctness across problems, while Pass@8 is the fraction of problems with at least one correct response among eight samples.Pass@8 does not use majority voting; both metrics are reported as percentages.Source: §A.2 · p. 14
  • Within each backbone comparison, methods share prompts and decoding protocols, with task-specific answer verification or extraction.Mathematical answers use answer verification, HumanEval uses execution-based tests, and multiple-choice tasks use answer-letter extraction.Source: §A.2 · p. 14

A.3 TRANSFER AND POST-TRANSFER ADAPTATION

  • Transfer tests use Qwen2.5-Math-1.5B → Qwen2.5-1.5B, Qwen2.5-Math-7B → Qwen2.5-7B, and FineMath-Llama-3B → Llama-3.2-3B.Immediate transfer has zero target updates; adaptation uses 100 updates with a fresh optimizer and frozen target backbone.Source: §A.3 · p. 14
  • LoRA transfers the complete rank-32 adapter at its original scale, and its Qwen2.5-1.5B trajectory uses an independent evaluation of the same initial adapter weights.That independent sampling can make its update-0 score differ from the immediate-transfer table.Source: §A.3 · p. 14
  • Sequential transfer evaluates zero through four cumulative pathway transfers from fixed source checkpoints, with no target-side optimization between or after transfers.Additional attention and FFN branches follow the preceding computations, and additional normalizations copy the target RMSNorm weights.Source: §A.3 · p. 14

A.4 ADDITIONAL IMMEDIATE-TRANSFER RESULTS

Additional results describe immediate-transfer Pass@8 and target-side MATH500 Pass@1 checkpoints. The accompanying analyses compare trained pathway components with a restricted trainable variant and examine transfer strength without target-side training.

A.4 ADDITIONAL IMMEDIATE-TRANSFER RESULTS

  • Table 6 lists target-side adaptation MATH500 Pass@1 checkpoints by target update.Source: Table 6 · p. 15
  • Table 7 reports immediate-transfer sampled Pass@8 without target-side training, complementing Table 3’s Pass@1 comparison.Each Pass@8 entry uses the same eight responses per problem as its corresponding Pass@1 entry; bold scores mark the best result within each direction, including ties.Source: §A.4 · p. 14, Table 7 · p. 15

A.5 TRAINABLE COMPONENTS AND TRANSFER STRENGTH

  • The component ablation compares training the full pathway with training only attention output and FFN down projections, leaving other branch projections at random initialization.The backbone, including the language-model output head, is frozen in both variants.Source: §A.5 · p. 15
  • The transfer-strength sweep varies α from 0 to 1 on the Qwen2.5-Math-1.5B → Qwen2.5-1.5B transfer without target-side training.It measures sampled Pass@1 on VAL300, a fixed 300-problem MATH test-split set verified to have no overlap with the RL training problems.Source: §A.5 · p. 15

B METHOD AND IMPLEMENTATION DETAILS

GrowRL preserves the pretrained policy at initialization by adding zero-output pathways, then trains their internal projections as output weights change. Its Dr. GRPO objective uses grouped, verified completions and also trains newly inserted pathways after transfer or composition.

B.1 FUNCTION PRESERVATION AND INITIAL GRADIENTS

  • Zero output projections make the added corrections identically zero, preserving the pretrained model’s hidden states and policy in exact arithmetic.The same argument applies when inserting a fresh pathway into an adapted model; transferred learned pathways can immediately change the receiving model’s policy.Source: §B.1 · p. 15
  • At zero output initialization, internal projection gradients vanish while output-projection gradients can be nonzero, enabling internal learning after outputs change.The WO, Wd-only control keeps internal projections at random initialization, though their inputs and activations can change as earlier branches learn.Source: §B.1 · p. 16

B.2 COMPLETE RL OBJECTIVE

  • Dr. GRPO samples groups of completions with binary answer-verification rewards and minimizes a clipped per-prompt loss averaged over the prompt batch.The setup uses G = 8, Tmax = 3000, and ϵ = 0.2, without reward-standard-deviation normalization or a KL penalty; its fixed denominator avoids per-completion length division.Source: §B.2 · p. 16
  • A detached sequence-level importance correction accounts for rollout-engine and training-actor differences, and the same objective updates only newly added pathways after transfer or composition.In GrowRL runs, wi = si1[si ≤3]; identical rollout and actor probabilities give wi = 1, separate from token-level clipping.Source: §B.2 · p. 16

B.3 IMPLEMENTATION

The implementation maps added branches onto compatible backbone layouts while preserving frozen parameters. It also uses mixed-precision computation, with trainable-parameter counts distinguished from potentially larger storage and optimizer allocations.

B.3 IMPLEMENTATION

  • Added attention and FFN branches can be represented by concatenating their channels and output projections with the backbone, with zero-output padding heads preserving grouped-query layouts where required.Each layer has one effective added attention head, and frozen backbone and padding slices are restored after optimizer steps.Source: §B.3 · p. 16
  • Sequential transfer retains explicit residual steps and additional normalization layers.Source: §B.3 · p. 16
  • Forward computation uses BF16 with FP32 master parameters and optimizer states; reported parameter counts cover trainable pathways, while dense storage and optimizer allocation can be larger.Backbone parameters remain unchanged, although hidden states can change as the pathway learns.Source: §B.3 · p. 16

Related papers

Loading 2610.16489v1…