PaperlayerView exact revision on arXiv

Source-linked AI summary

From $f(x)$ and $g(x)$ to $f(g(x))$: LLMs Learn New Skills in RL by Composing Old Ones

Lifan Yuan, Weize Chen, Yuchen Zhang, Ganqu Cui, Hanbin Wang, Ziming You, Ning Ding, Zhiyuan Liu, Maosong Sun, Hao Peng

arXiv:2509.25123v3cs.AIcs.CL

TL;DR

The paper asks whether RL teaches LLMs genuinely new skills or mainly activates existing reasoning patterns. Using a controlled synthetic string-transformation framework that separates atomic from compositional learning, it finds that appropriately incentivized RL acquires compositional skills that generalize to harder and different tasks.

  • Problem

    It remains unclear whether RL teaches LLMs new skills because prior evaluations use coarse metrics and tasks where base models may already perform well.

  • Method

    The paper uses a decontaminated synthetic string-transformation testbed with hidden function definitions, controlled composition depth, and separate atomic-skill and compositional training stages.

  • Results

    RL on compositional problems improves unseen Level-3 performance from near-zero to 30% and Level-4 performance to 15%, with generalization to harder, held-out, and cross-task problems.

  • Takeaways & Limitations

    RL can acquire genuinely new, generalizable compositional skills when training explicitly incentivizes composition on top of pre-existing atomic abilities.

  • Takeaways & Limitations

    The findings rely on synthetic tasks that may not capture the complexity and nuance of real-world reasoning, so extension to natural domains remains open.

Abstract

from arXiv · show

Does RL teach LLMs genuinely new skills, or does it merely activate existing ones? This question lies at the core of ongoing debates about the role of RL in LLM post-training. On one side, strong empirical results can be achieved with RL even without preceding supervised finetuning; on the other, critics argue that RL contributes little beyond reweighting existing reasoning strategies. This work provides concrete evidence that LLMs can acquire genuinely new skills during RL by composing existing ones, mirroring one of the central mechanisms by which humans acquire new cognitive skills. To mitigate data contamination and other confounding factors, and to allow precise control over task complexity, we develop a synthetic framework for our investigation. Specifically, we define a skill as the ability to infer the output of a string transformation function f(x) given x. When an LLM has already learned f and g prior to RL, our experiments reveal that RL enables it to learn unseen compositions of them h(x)=g(f(x)). Further, this compositional ability generalizes to more difficult problems such as compositions of >2 functions unseen during RL training. Surprisingly, our experiments show that compositional skill acquired on a source task transfers to a different target task. This transfer happens even without compositional training on the target, requiring only prior knowledge of the target's atomic skills. Our qualitative analysis shows that RL fundamentally changes the reasoning behaviors of the models. In contrast, next-token training with the same data yields none of these findings. Our systematic experiments provide fresh insights into LLM learning, suggesting the value of first building base models with basic skills, then using RL to incentivize advanced, generalizable skills for complex problems.

1 INTRODUCTION

The paper investigates whether RL teaches LLMs genuinely new skills and finds that RL can acquire compositional abilities when training explicitly incentivizes composition.

  • Research framework: The framework uses a decontaminated string-transformation testbed with meaningless function identifiers, hidden definitions, and controllable composition depth.These controls separate atomic skill acquisition from compositional learning and enable fine-grained difficulty analysis.
  • Results: RL on Level-2 compositions raises unseen Level-3 performance from near-zero to 30% and Level-4 performance to 15%, unlike rejection fine-tuning on the same data.The result supports acquisition of compositional skills through RL rather than simple reuse of existing behaviors.
  • Results: Both RL and compositional incentives are necessary: RFT on Level-2 data and RL on Level-1 atomic problems improve higher-than-Level-2 tasks by less than 1%.The comparison indicates that neither the optimization method nor atomic training alone supplies the required compositional incentive.
  • Generalization: The learned compositional skill generalizes to held-out and harder problems, transfers across domains, and changes reasoning behavior rather than merely reranking familiar solutions.The paper links these findings to improved performance on challenging problems where the base model performs poorly.
  • Interpretation: These findings challenge the view that RLVR merely reweights reasoning patterns already present in base models.The authors argue that coarse pass@k evaluations can hide skill acquisition by mixing tasks of different difficulty.
  • Implications: The results motivate coordinating base-model development with post-training by first establishing basic skills and then incentivizing advanced, generalizable composition.The proposed perspective emphasizes RL’s potential for easy-to-hard generalization and cross-task transfer.

2 BACKGROUND

Prior work leaves open whether RL learns new skills because broad benchmarks and pass@k metrics obscure skill boundaries and difficulty. The paper positions compositional learning as a controlled testbed for resolving this question.

  • Limits of prior evidence: Existing studies often use loose skill definitions or pass@k gaps, showing activation of existing behaviors without directly proving that RL learns no new skills.The paper argues that these observations are insufficient to distinguish capability reuse from acquisition.
  • Limits of prior evidence: Pass@k comparisons can obscure learning because they lack causal skill attribution, fine-grained difficulty and domain analysis, and control over prior model competence.High base-model performance may also reduce the incentive or room for RL to learn new skills.
  • Compositional learning: Compositional reasoning separates atomic knowledge from new combinations, making it a testbed for studying skill acquisition and generalization.The paper connects this structure to cognitive accounts in which humans acquire new skills by composing existing ones.

3 RESEARCH FRAMEWORK

The research framework isolates atomic skill acquisition from compositional learning in a synthetic string-transformation task. It varies nesting depth and compares RL with alternative training under held-out and cross-task evaluation.

  • Task design: The framework controls confounders through meaningless function identifiers, hidden implementations, well-defined skills, and adjustable composition complexity.These design choices support decontaminated evaluation and causal analysis of compositional learning.
  • Task design: The task uses deterministic string functions as atomic skills and nested function applications as compositional skills, with nesting depth defining difficulty.For example, Level 1 applies one function, whereas Level 2 applies two functions.
  • Training protocol: Models first acquire atomic transformations through rejection fine-tuning, then train on compositions with hidden definitions using either online RL or RFT.Removing definitions during fine-tuning and composition training forces reliance on internalized atomic knowledge.
  • Experimental setting: The experiments use Llama-3.1-8B-Instruct as a testbed intended to reduce effects of data contamination beyond the synthetic task design.The model choice follows recent work identifying it as a cleaner environment for studying RL.
  • Evaluation: Evaluation partitions functions into disjoint training and held-out sets and tests generalization across difficulty levels and transfer to Countdown.This protocol examines held-out, easy-to-hard, and cross-task generalization.

4 RL AS A PATHWAY TO GENERALIZABLE SKILL ACQUISITION

RL acquires generalizable compositional skills when training explicitly incentivizes composition, outperforming rejection fine-tuning and extending to harder unseen tasks. The learned compositional ability can transfer across tasks when target-task atomic skills are already available, while changing model failure behavior beyond simple reranking.

  • 4.1 LLMs Acquire New Compositional Skills During RL: 30% on Level 3 and 15% on Level 4 show that RL trained on Level-2 compositions generalizes to unseen harder compositions.RL Level 2 and RL Level 1+2 improve from 5% to around 30% on Level 3 and from 1% to 15% on Level 4.
  • 4.2 RL Is the Key Ingredient to the New Compositional Skills: 64% on Level 2 and 27% on Level 3 show RL substantially outperforming iterative RFT trained on the same compositional data.RFT never surpasses 2.6% on Level 3 and reaches only 15% on held-out Level-2 problems.
  • 4.3 Compositional Skills Learned in RL Are Transferable, But Atomic Skills Are Prerequisites: 35% on Countdown Level 3 demonstrates transfer of compositional RL from string transformations to an unrelated task, provided target atomic skills are present.Multi-Base + RL L1+2 exceeds the Multi-Base baseline by more than 18% at Level 3 and reaches approximately 6% at Level 4.
  • 4.4 RL Expanding Performance Limits Is Not a False Promise: At Level 5, the RL Level 1+2 advantage over the RFT base grows from 4% at pass@1 to approximately 25% at pass@1024, unlike easier levels.The widening gap on hard problems supports new skill acquisition, whereas gaps shrink with more samples on Levels 1-2.
  • 4.5 Behavioral Analysis: RL Transforms Failure Modes: RL Level 2 eliminates “Ignores Composition” errors and correctly solves 28.1% of problems, shifting failures toward atomic execution errors.RFT Base, RFT Level 2, and RL Level 1 show more than 50% “Ignores Composition” failures and more than 35% compositional-structure misunderstandings.

5 CONCLUSION

The paper concludes that RL can teach genuinely new compositional skills when training properly incentivizes composition, while acknowledging that its conclusions concern RL applied to pre-existing atomic competencies.

  • The evaluation framework counts atomic errors only after a model resolves the compositional structure, so models lacking compositional capability do not reveal or receive counted atomic errors.
  • RL can teach genuinely new and powerful skills when the training task properly incentivizes composition.
  • The study shows that pre-existing atomic skills are sufficient for RL to unlock compositional capabilities, but does not claim they are strictly necessary.

LIMITATION

The study relies on synthetic string-transformation tasks to isolate causal learning mechanisms, so its findings may not fully capture real-world reasoning.

  • The main limitation is reliance on synthetic tasks that may not capture the complexity and nuance of real-world reasoning.
  • The paper identifies validation in mathematics, code generation, and scientific reasoning as an important direction for future work.
  • Extending the findings to realistic applications remains an open challenge.

A.1 STAGE 1

The experimental pipeline first builds a shared Stage 1 base model from atomic transformation tasks, then constructs Stage 2 datasets with hidden definitions and compositional problems for RL or RFT comparisons.

  • All experiments except Multi-Base variants begin from the same Stage 1 base model.
  • Stage 1 generates 50k Level-1 problems, filters out problems solved perfectly by the base model, and uses remaining correct responses for supervised fine-tuning.
  • Stage 2 hides function definitions and includes compositional problems, with Level-2 examples formed by randomly composing two functions.
  • RL uses DAPO in a strictly on-policy setup with 16 rollouts per prompt, filtering prompts whose rollouts are all correct or all incorrect.
  • The iterative RFT baseline repeatedly generates rollouts, filters correct responses, and fine-tunes the model on the resulting datasets.

B EVALUATION DETAILS

The evaluation uses synthetic string-transformation tasks, with controlled function compositions and prompts that hide definitions during training and evaluation. The appendix lists the transformation functions and example prompts used across stages.

  • Evaluation setup: Evaluation covers 256 randomly generated problems at each string-transformation level from 1 through 8.Responses are sampled at temperature 1.0 with a maximum length of 8192 tokens.
  • Evaluation setup: The Countdown evaluation contains 128 problems for each level and uses the same evaluation setup as the string-transformation task.
  • Prompt design: Stage 1 prompts may show function definitions for response collection, but actual Stage 1 RFT removes them so models rely on internalized function behavior.The same definitions remain hidden throughout Stage 2 training.
  • String transformations: The function library includes transformations for reordering, prefixing, suffixing, interlacing, rotation, case alternation, shifting, filtering, duplication, recursion, and backchaining.Function names are replaced with meaningless identifiers such as func 1 in the experiments.
  • String transformations: The listed implementations include deterministic shuffling, character duplication, separator insertion, recursive reversal and interlacing, filtering, verification, and backchaining operations.String concatenation is also treated as a function in the task.

E EXAMPLE FOR COUNTDOWN TASK

The appendix provides an example prompt for the Countdown task and notes that the following material contains complete model responses from the case studies.

  • Example prompt: Figure 9 presents example prompts for the Countdown task.
  • Case studies: The appendix includes the complete, unfolded responses from each model for the case studies.

F.1 FULL RESPONSES TO THE LEVEL 2 PROBLEM

The Level 2 responses illustrate how models analyze an interlacing function and the surrounding composition, with stronger responses correctly concatenating both function outputs. Some responses stop early or misread the function relationship.

  • Failure mode: The weaker response misstates the loop behavior and omits the separate func 2 contribution, ending with the interlaced output alone.The authors note that it misses another function or misunderstands the relationship between the functions.
  • Function analysis: The interlacing step alternates characters from vptqj and nar, yielding the sequence v, n, p, a, t, r, q, j.
  • Correct response: The correct Level 2 response returns the interlaced string vnpatrqj followed by xbh, producing vnpatrqjxbh.It correctly identifies that func 7 interlaces vptqj with nar and that func 2 leaves xbh unchanged.

F.2 FULL RESPONSES TO THE LEVEL 3 PROBLEM

The Level 3 responses require tracking a nested composition of vowel removal, character duplication, and alternating case. Correct reasoning produces hHtTkK, while weaker responses apply functions in the wrong order or stop before completing the composition.

  • Failure mode: Another response initially applies transformations to the original input and later acknowledges that it may have used the wrong intermediate string.
  • Failure mode: One response incorrectly reports hToEk after applying alternating case to the vowel-filtered input but fails to account for the duplication function.The authors note that this answer ends too early because another function remains unanalyzed.
  • Correct response: The response correctly tracks the nested pipeline as func 10(func 14(func 2(x))) rather than applying the transformations independently.
  • Correct response: The complete Level 3 composition applies func 2, then func 14, then func 10, producing hHtTkK.Vowel removal yields htk, duplication yields hhttkk, and alternating case yields hHtTkK.
Loading 2509.25123v3…