PaperlayerView exact revision on arXiv

Source-linked AI summary

Incentivizing Self-Learning in Large Language Models via Reinforcement Learning

Lingkai Kong, Yuzhe Gu, Haiteng Zhao, Zijian Wu, Youqing Fang, Zhouqi Hua, Yanan Sun, Kai Chen, Wenwei Zhang

arXiv:2610.39671v1cs.CL

TL;DR

LLMs can access parametric and retrieved knowledge, but solving unfamiliar problems also requires learning principles and integrating them into reasoning. This paper trains models to study textbooks while solving proof problems, using BookNavigator and fine-grained RL rewards. The approach improves performance across mathematical splits and external benchmarks, with higher overall proof accuracy than RL-trained fixed-retrieval baselines.

  • Problem

    For unfamiliar problems, access to relevant information or a preassembled context does not establish how models can learn principles and integrate them into a developing solution from long-form materials.

    Source: §1 · p. 1

  • Method

    The authors train models with RL to alternate mathematical reasoning with autonomous study of structured textbooks through BookNavigator, using exploration incentives and round-level tool credit.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    Self-Learning exceeds the strongest RL-trained fixed-retrieval baseline by 6.4 percentage points overall and achieves the highest accuracy across training-textbook, unseen-textbook, and unseen-subject splits.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support a domain-agnostic self-learning strategy that transfers to unseen textbooks, subjects, and task types, and may strengthen problem-solving on complex, long-context tasks.

    Source: §5 · p. 9

    All RL-trained policies underperform the base model on CL-bench Procedural Task Execution, where tasks emphasize prescribed workflows and detailed operational constraints.

    Source: §3.4 · p. 7

Abstract

from arXiv · show

Large language models (LLMs) draw on parametric knowledge and retrieval-augmented generation (RAG) to solve knowledge-intensive tasks. Yet problem solving in unfamiliar domains calls for tightly coupled knowledge acquisition and reasoning, beyond factual retrieval and aggregation or learning solely from a preassembled context. We study self-learning -- autonomously retrieving, studying, and applying knowledge from long-form reference materials on the fly. To investigate this capability, we formulate a textbook-based mathematical problem-solving task. LLMs must navigate textbook exposition, learn and apply relevant principles, and construct proofs for exercises, with reference solutions withheld. We introduce a BookNavigator interface to support this human-inspired study process and incentivize self-learning through reinforcement learning (RL) with fine-grained rewards. Experiments show that our method improves performance across in-domain and out-of-domain textbooks and subjects, raising Intern-S2-Preview-35B's overall proof accuracy from 51.5% to 79.5% and outperforming RL-trained textbook-free and fixed-retrieval baselines. These gains extend to complex long-context benchmarks involving different task types, with improvements over the base model of 1.7 and 2.3 percentage points on BrowseComp-Plus and CL-bench, respectively. Together, these findings suggest that RL can foster a domain-agnostic self-learning strategy, highlighting the potential of self-learning from long-form corpora to strengthen broader problem-solving capabilities on complex, long-context tasks.

1. Introduction

The paper frames self-learning as autonomously acquiring and applying knowledge from reference materials while solving unfamiliar problems. It presents a textbook-based task and an RL-trained study process, reporting gains across mathematical splits and external benchmarks.

1. Introduction

  • Self-learning requires models to understand unfamiliar principles, judge their applicability, and integrate them into a developing solution—not merely access relevant information.The paper defines it as autonomously learning from reference materials as a task unfolds.Source: §1 · p. 1
  • The study task pairs mathematical exercises with structured textbooks, with solutions withheld so models must connect concepts across sections to construct proofs.The authors use RL to incentivize this multi-round study process.Source: §1 · p. 1
  • The authors position self-learning as a transferable study strategy for acquiring and applying knowledge from unseen textbooks rather than memorizing training-book content.Their findings support self-learning from self-supervised corpora such as textbooks as a promising means of strengthening problem-solving on complex-context tasks.Source: §1 · p. 2
  • The pipeline yields 5,183 proof problems from 158 textbooks and introduces BookNavigator for consulting textbook contents during reasoning.Fine-grained RL combines an annealed exploration bonus with round-level tool credit to encourage textbook use while reducing reinforcement of unproductive calls.Source: §1 · p. 2
  • Self-Learning achieves the highest accuracy across training-textbook, unseen-textbook, and unseen-subject splits, exceeding the strongest RL-trained fixed-retrieval baseline by 6.4 percentage points overall.The trained policy also improves performance on BrowseComp-Plus and CL-bench under their original inference protocols, without BookNavigator.Source: §1 · p. 2

2. Method

The method pairs proof problems with structured textbooks and lets a policy alternate reasoning with textbook study. BookNavigator provides navigation and reading actions, while fine-grained RL rewards encourage exploration and assign credit to individual study rounds.

2.1. Task Formulation

  • At each non-terminal round, the policy reasons from the problem and prior interactions, then chooses a textbook-access action; in the terminal round it outputs a proof without further access.A judge compares the proof with a reference solution excluded from the model-visible textbook, assigning +1 for correct and −1 otherwise.Source: §2.1 · p. 3

2.2. Problem–Textbook Curation

  • The curation pipeline structures advanced mathematics textbooks, separates exercises and solutions from exposition, and excludes solutions and section-dependent exercises from model-visible materials.This leaves reference content such as definitions, theorems, explanatory passages, and proofs of general results for self-directed study.Source: §2.2 · p. 3

2.3. BookNavigator: Interactive Textbook Study

  • BookNavigator is the model’s sole textbook interface, supporting book orientation, keyword-based section search, and reading the complete text of selected sections.Returned observations are appended to the interaction history, allowing later actions to adapt to retrieved information and the developing solution.Source: §2.3 · pp. 3–4

2.4. Reinforcement Learning with Fine-grained Rewards

  • An annealed exploration bonus rewards correct trajectories with at least one successful tool call, maintaining a smaller incentive for textbook access later in training.The schedule provides stronger encouragement for early exploration and retains a positive floor.Source: §2.4 · p. 4
  • A tool judge labels non-terminal rounds useful, neutral, or harmful, and round-level scores redistribute the trajectory advantage across policy-generated tokens.The judge uses subsequent reasoning as retrospective evidence; centered redistribution preserves the token-weighted mean advantage while varying learning signals by round quality.Source: §2.4 · p. 4

3. Experiments

After reinforcement learning, Self-Learning leads the evaluated protocols on overall proof accuracy and on held-out textbooks and subjects. Its gains also extend to non-mathematical benchmarks, while attribution and ablation results distinguish the contributions of textbook access, exploration incentives, and intermediate tool feedback.

3.1. Experimental Setup

  • The evaluation uses 5,183 problems from 158 mathematics textbooks, with held-out-book and held-out-subject splits and fixed-context baselines for comparison.Self-Learning interleaves textbook access with reasoning; the fixed-retrieval baselines provide selected or ordered sections before generation, while Oracle Section supplies the source section.Source: §3.1 · p. 5

3.2. Main Results

  • 79.5% overall proof accuracy places RL-trained Self-Learning 6.4 points above Top-kSections; it also leads on OOD-Book and OOD-Subject by 4.2 and 5.4 points.The OOD accuracies are 85.2% and 86.7%, respectively, showing gains beyond training textbooks and subjects.Source: Table 1 · p. 6, §3.2 · p. 6

3.3. Performance Gain Attribution

  • With RL, No-Book accuracy rises from 55.7% to 71.7% (16.0-point gain) and Paired Textbook accuracy from 51.5% to 79.5% (28.0 points), yet paired access remains 7.8 points better.Source Removal yields 74.6%, above No-Book’s 71.7%; Non-Source Removal retains 79.8%, 5.2 points above Source Removal.Source: §3.3 · pp. 6–7

3.4. Cross-Task Generalization

  • On BrowseComp-Plus and CL-bench, Self-Learning scores 51.9% and 17.1%, improving over Base by 1.7 and 2.3 points, though all RL policies trail Base on Procedural Task Execution.It leads Domain Knowledge Reasoning and Rule System Application by 4.5 and 1.2 points over Base, and ties No-Book in Empirical Discovery and Simulation.Source: §3.4 · p. 7

3.5. RL Ablation

  • Combining tool credit with the annealed exploration bonus yields 85.2% OOD-Book, 86.7% OOD-Subject, and 79.5% overall, while tool credit alone adds 3.3 points on ID-Book.Correctness-only RL raises accuracy but reduces tool-using trajectories from 36.9% to 6.7%; the bonus raises that share to 79.9% and improves OOD-Book and OOD-Subject by 2.0 and 2.9 points over correctness-only RL.Source: Table 4 · p. 8, §3.5 · p. 8

4. Related Work

Prior work studies retrieval-driven reasoning, context learning, adaptation, and fine-grained agentic RL. This work focuses on coordinating textbook knowledge acquisition with evolving proof reasoning within a single problem-solving episode.

Retrieval-Augmented Reasoning

  • This work trains agents on textbook-grounded proof construction, selecting and applying definitions and theorems rather than retrieving included exercise solutions; exercises and solutions are removed from the reference exposition.Source: §4 · pp. 8–9

Meta-Learning and Test-Time Adaptation

  • Unlike meta-learning approaches based on cross-episode adaptation or test-time updates, the policy learns to acquire and apply knowledge within one episode from static references.The approach uses neither repeated-attempt correctness feedback nor test-time weight updates.Source: §4 · p. 9

Fine-grained Agentic RL

  • Fine-grained agentic RL methods add process signals to sparse outcome rewards, while this work uses round-level tool credit alongside a correctness-gated, annealed exploration bonus.A frozen judge scores calls from interaction context and following reasoning without seeing the reference solution or final correctness; credit is centered across sampled trajectories.Source: §4 · p. 9

5. Conclusion

The paper frames self-learning as acquiring, understanding, and applying knowledge to unfamiliar tasks, and studies it through textbook-based mathematical problem solving. Experiments provide evidence of transfer across textbooks, subjects, and task types.

5. Conclusion

  • The paper studies self-learning as autonomously acquiring, understanding, and applying knowledge to solve unfamiliar tasks.Its framework interleaves mathematical reasoning with autonomous textbook study and uses fine-grained agentic RL to incentivize learning.Source: §5 · p. 9
  • Experiments provide evidence for a domain-agnostic learning strategy that transfers to unseen textbooks, subjects, and task types.Source: §5 · p. 9
  • The authors propose long-form corpora as a potential source of self-supervision for strengthening general problem-solving on complex, long-context tasks.Source: §5 · p. 9

A.1. Dataset Statistics

The appendix points to subject-level dataset statistics and reports per-textbook averages for section and token counts.

A.1. Dataset Statistics

  • Table 5 reports subject-wise dataset statistics, including average sections and tokens per textbook across the parsed hierarchy.Source: Table 5 · p. 12, §A.1 · p. 12

A.2. Problem–Textbook Example

The appendix illustrates a proof problem paired with a textbook and documents prompts and an example of textbook navigation. The example shows the solver checking retrieved material before using textbook definitions in its proof.

A.2. Problem–Textbook Example

  • The example asks for a proof that a simple symmetric random walk is recurrent in dimensions 1 and 2 and transient in dimensions at least 3.The problem is paired with Lawler's Introduction to Stochastic Processes, with its source identified as Section 2.2 on recurrence and transience.Source: §A.2 · p. 12

B.1. Shared Solver Prompt

  • The shared solver prompt requires a clearly structured, rigorous solution with reasoning and independently checkable logical steps.Source: §B.1 · p. 13, §B.2 · p. 15

B.2. BookNavigator Prompt

  • BookNavigator supports TOC inspection, keyword search, and full-section reading; its prompt requires TOC-first retrieval and limits repeated, unproductive searches.A solver should check section state and evidence after retrieval, avoid unsupported textbook claims, and stop after two weak or unsuccessful searches.Source: §B.2 · p. 14

B.3. Fixed-Retrieval Baseline Prompt

  • Fixed-retrieval baselines receive the question and selected reference material together, with instructions to judge relevance and solve even when the material is incomplete.Source: §B.3 · p. 15

B.4. Self-Learning Trajectory Example

  • In the trajectory example, the solver rejects a search match that is a different proposition, then reads the section for relevant definitions while noting that the exercise proof is not included.The displayed textbook material supplies definitions and a theorem's rank functions.Source: §02 · p. 16, §03 · p. 16

04 Revise the interpretation and complete the proof Turn 4

The proof completes the interpretation by contracting to Y and comparing rank functions. It uses textbook formulas for restriction and contraction to establish both identities.

04 Revise the interpretation and complete the proof Turn 4

[Further reasoning and output-format deliberation omitted.]

C.1. RL Implementation Details

RL runs start from Intern-S2-Preview-35B and use GRPO with specified rollout, optimization, and policy-weight settings.

C.1. RL Implementation Details

  • Each rollout batch targets 128 problem groups with eight trajectories per problem, followed by two mini-batch optimization steps.Source: §C.1 · p. 17
  • The runs use AdamW with a constant learning rate of 10−6, weight decay 0.1, gradient-norm clipping at 1.0, and no warmup.AdamW uses β1 = 0.9 and β2 = 0.95.Source: §C.1 · p. 17
  • GRPO uses leave-one-out group advantages and token-level importance weighting, with clipped policy weights and masked high-ratio tokens.Weights are clipped to [0.8, 1.28], ratios above 2.0 are masked, and KL regularization is disabled; the estimator also uses group-entropy reweighting and a whitespace-token penalty activated at a 4% fraction.Source: §C.1 · p. 17

C.2. Evaluation Details

Evaluation combines reference-conditioned proof verification, criteria for judging proof rigor, and assessment of textbook-tool use. The appendix also specifies protocols for BrowseComp-Plus and CL-bench.

C.2. Evaluation Details

  • The final-solution judge reviews each proof eight times and accepts it only when all reviews report no error.GLM-5.2 receives the problem, held-out reference solution, and final response without its thinking block or preceding textbook interactions; the response is split into blank-line-delimited steps.Source: §C.2 · p. 17
  • Proof grading prioritizes mathematical validity and problem constraints, using the reference solution as an anchor rather than the only acceptable approach.Alternative methods are accepted when mathematically sound and compliant with problem constraints; under-justified claims and unsupported appeals to external results are treated conservatively.Source: §C.2 · pp. 17–18
  • A frozen copy of the initial policy judges textbook-tool calls using interaction history and next-round reasoning, not final-solution correctness.Calls are labeled useful, neutral, or harmful; the judge has no reference solution or final correctness verdict.Source: §C.2 · pp. 18–19
  • The tool-use rubric treats relevant new information, clarified conditions, corrected misunderstandings, or meaningful navigation leads as useful evidence.Reasonable exploration without demonstrated gain is neutral, while clearly redundant retrieval or irrelevant actions can be harmful; uncertain cases default to neutral.Source: §C.2 · p. 19
  • BrowseComp-Plus uses BM25 retrieval and Qwen3-32B as judge, while CL-bench uses GPT-5.1 with low reasoning effort.The evaluations follow the benchmarks' official protocols.Source: §C.2 · p. 19

Related papers

Loading 2610.39671v1…