PaperlayerView exact revision on arXiv

Source-linked AI summary

StudyBench: Can Self-Evolution Squeeze Textbooks for Olympiad Capability?

Yinghao Chen, Zixi Chen, Bingxiang He, Ziqing Qiao, Huan-ang Gao, Yinuo Xu, Yuxin Zuo, Zeyuan Liu, Yuhao Zhan, Chaojun Xiao

arXiv:2609.00787v1cs.AI

TL;DR

Self-evolution lacks a direct measure of how efficiently it converts training material into transferable problem-solving capability. StudyBench provides a controlled physics benchmark separating textbook absorption from olympiad transfer, and finds that application gains rarely translate to the harder transfer set, leaving a substantial method gap.

  • Problem

    Existing evaluations do not directly isolate how efficiently self-evolution converts training material into transferable capability.

  • Method

    StudyBench pairs 11 physics textbooks with an Application Set of difficult textbook problems and a reachable Transfer Set of olympiad problems, evaluated across three base models.

  • Results

    Application gains rarely transfer: on Qwen3-8B, GEPA raises Application Par@8 from 17.05 to 34.85, while Transfer Par@8 reaches only 7.04 against a 100% guidance ceiling.

  • Takeaways & Limitations

    StudyBench turns self-evolution progress into a measurable target and indicates that the remaining gap is primarily a method problem rather than a data or compute problem.

  • Takeaways & Limitations

    The benchmark is instantiated only in physics, and its capability-gap and guidance-ceiling guarantees apply only to Qwen3-8B under the specified traces.

Abstract

from arXiv · show

Humans need to study only a handful of well-written textbooks to master a discipline and attempt its hardest problems. We argue that an ideal self-evolution method should share the same property, that is autonomously learning from raw training material for transferable problem-solving capability. However, we still lack a direct measurement for it. We introduce StudyBench, a controlled physics benchmark that directly measures how efficiently a self-evolution method converts training material into capability. We organise the test set into an Application Set, consisting of difficult textbook problems and evaluating absorption ability, and a Transfer Set, consisting of olympiad-level problems and evaluating transfer ability. Benchmarking representative self-evolution methods across three base models, we find that improvements on the Application Set rarely translate to the harder Transfer Set. A guidance ablation exposes a Guidance Gap: even the strongest method closes only a small fraction of what the same material unlocks when supplied as in-context guidance. Besides, every method hits a Compute Plateau, saturating well before exhausting its compute budget. The remaining gap is therefore a method problem rather than a data or compute problem. By offering a clean and controlled benchmark, StudyBench turns self-evolution progress from an open-ended pursuit into a measurable target for future research. Our code is released at https://github.com/thunlp/StudyBench.

1 Introduction

StudyBench addresses the lack of a direct measure for how efficiently self-evolution converts training material into transferable problem-solving capability. Its controlled design separates absorption from transfer and finds that application gains rarely become olympiad capability.

  • Motivation: Self-evolution must both absorb knowledge and convert it into transferable problem-solving capability beyond what the training material explicitly states.The paper treats knowledge-to-capability conversion as the harder requirement because real-world problems often lack direct precedents in the training material.
  • Evaluation gap: Existing evaluations do not directly measure this conversion because static exams conflate data, base-model, and algorithm contributions, while lifelong-learning benchmarks target local adaptation.The paper also identifies unreachable test targets as a separate evaluation problem.
  • StudyBench: StudyBench uses 11 physics textbooks plus Application and Transfer Sets to evaluate absorption on difficult textbook exercises and transfer on harder olympiad problems.The Transfer Set retains olympiad problems that Qwen3-8B fails alone but solves with textbook-grounded guidance.
  • StudyBench: The benchmark controls attribution by holding training material, test items, and evaluation protocol fixed across methods within each base model.Its construction also enforces a capability gap and reachability of retained targets.
  • Findings: Application-Set gains remain local to textbook exercises rather than becoming olympiad capability across the benchmarked methods and base models.The paper identifies the remaining gap as a method problem rather than a data or compute problem.
  • Findings: On Qwen3-8B, GEPA raises Application Par@8 from 17.05 to 34.85, but Transfer Par@8 reaches only 7.04.The same passage reports a 100% guidance ceiling and a compute plateau before the budget is exhausted.

2 StudyBench

StudyBench constructs a controlled physics benchmark from textbook material and progressively harder test problems. Its filters ensure retained problems are outside Qwen3-8B’s reliable capability while remaining reachable from the training material.

  • Training material: Each textbook produces nested Corpus, Instructions without Answer, and Instructions with Answer layers for different self-evolution training paradigms.The Corpus contains raw passages; the other layers provide increasingly structured exercise material.
  • Sources: The benchmark draws on 11 textbooks and six international physics and astronomy olympiads, with the two sources forming escalating-difficulty test pools.The textbooks cover the syllabus of all six olympiads.
  • Test sets: Failed textbook problems become the Application Set, while failed olympiad problems undergo an additional textbook-grounded reachability filter to form the Transfer Set.The Application Set measures in-material absorption, whereas the Transfer Set measures capability transfer to problems harder than the textbooks.
  • Filtering: The Capability Filter uses Qwen3-8B pass@8 and retains parents that fail to solve every sub-problem reliably.Fifteen Application parents solved once in eight attempts are additionally admitted to keep easier subjects represented.
  • Filtering: The Naive Reachability Filter decomposes solutions into knowledge points, retrieves textbook fragments, and verifies teacher-generated grounded guidance.Retrying under this guidance certifies that retained olympiad answers are reachable by recombining training material.
  • Evaluation: Parent accuracy Par@k requires one attempt to solve every sub-problem, while Sub@k counts each sub-problem correct if any attempt solves it.Open-weight models use k=8, with three independent sampling seeds; Opus 4.7 uses a single pass@1 run.
  • Evaluation: Multi-part problems retain conversational continuity across sub-problems without exposing gold solutions, and a rule-based verifier handles nine answer types.Failed sub-problems receive a placeholder so later sub-problems remain evaluable.
  • Controls: Capability filtering removes consistently memorized answers from test sets, reducing pretraining leakage as a source of retained performance.The same item set is then evaluated across models and methods.

3 Experiments

The experiments compare self-evolution methods grouped by the training-material layer they consume across three base models. Textbook absorption generally improves, but transfer gains remain small and the pattern persists across model scales.

  • Baselines: Methods consume different training-material layers: Corpus, Instructions with Answer, Instructions without Answer, or no training data.Bonito uses synthetic-data supervised fine-tuning, GEPA and ACE evolve inference-time artefacts, GRPO performs supervised RL, and R-Zero uses self-play.
  • Setup: The benchmark evaluates the same filtered items on Llama-3.2-3B-Instruct, Qwen3-8B, and Opus 4.7 with minimal method-specific modifications.Open-weight methods run on a single 8×NVIDIA-A800-80GB node.
  • Application results: On Qwen3-8B, most methods raise Application Sub@8 by +8.87 to +14.98, with GEPA reaching 34.85 Application Par@8.GEPA leads on both Application metrics in the reported comparison.
  • Transfer results: Transfer Sub@8 improves by at most +2.14, while the best Transfer Par@8 is 7.04 against a 100% Corpus-grounded guidance ceiling.The comparison is reported for GEPA and TTRL at the maximum Transfer Sub@8 gain.
  • Cross-model pattern: Llama-3.2-3B-Instruct barely improves on Application, while supervised GRPO leads Transfer with a +7.86 gain.Opus 4.7 already solves much of the set, and context-evolution methods mostly regress on Application.

4 Analysis

StudyBench’s analysis separates textbook absorption from transfer to harder olympiad problems and tests whether guidance or additional compute closes the gap. Application gains remain substantial, but transfer gains stay small, while longer runs plateau before exhausting compute.

  • The Guidance Gap (RQ1): Application gains of +8.87 to +14.98 ∆Sub@8 rarely translate into Transfer gains above +2.14 ∆Sub@8, with Transfer Par@8 remaining single-digit.The two sets therefore probe complementary abilities: absorption of training material versus transfer to harder problems.
  • The Guidance Gap (RQ1): 100 Par@8 and Sub@8 are reachable for Qwen3-8B under textbook-grounded guidance, while GEPA, ACE, Intuitor, and R-Zero reach 88.89–90.00 guided Par@8.Guided Sub@8 for these methods remains between 97.14 and 98.21, showing that the guidance ceiling is largely preserved after training.
  • The Guidance Gap (RQ1): GEPA’s strongest solo run reaches 7.04 Transfer Par@8, closing only 7% of Qwen3-8B’s parent-level headroom.GEPA also closes 5% of the sub-problem gap; Intuitor and R-Zero close less, while Bonito widens the sub-problem gap.
  • The Guidance Gap (RQ1): The remaining transfer gap is characterized as an internalisation problem rather than a reachability problem, because guidance still unlocks about 90% of Transfer parents.The same qualitative gap persists across Llama-3.2-3B-Instruct and Opus 4.7, whose guidance ceilings differ because the filter was built for Qwen3-8B.
  • The Compute Plateau (RQ2): ACE’s Application Sub@8 rises from approximately 29.4% to approximately 40% by 8.50 GPU hours, then remains in a 38–41% band through 62.54 GPU hours.The curve averages three seeds, with the late band reflecting ±1 standard deviation.
  • The Compute Plateau (RQ2): Across all five profiled loops, late computation produces a noisy plateau or decline rather than a second climb, despite compute spanning 8.12 to 614 GPU hours.The highest plateau is GEPA’s 44.3%, and the analysis concludes that closing more of the gap requires a different loop, not a longer run.

5 Related Work

Related benchmarks examine self-evolution through self-play, memory, interaction, knowledge internalisation, scientific discovery, and engineering tasks. StudyBench complements them by fixing the source corpus and test design while measuring corpus-reachable capability internalisation against a guidance ceiling.

  • Self-Evolution Methods: Self-play and intrinsic-reward methods generate or score practice without a fixed external answer set, whereas GEPA and ACE evolve context without changing model weights.These approaches differ mainly in where their improvement signal comes from.
  • Adaptation and Search: Memory, reflection, test-time search, and evolutionary inference methods accumulate experience or search over trajectories to adapt models after deployment.Examples include Voyager, Reflexion, ExpEL, ReasoningBank, and AlphaEvolve.
  • Existing Benchmarks: Dynamic and lifelong-learning benchmarks measure adaptation over interaction streams but typically leave the training material implicit or submitter-chosen.Other targeted benchmarks study knowledge internalisation, scientific-law discovery, or engineering agents.
  • StudyBench’s Position: StudyBench fixes the source corpus, enforces capability gap and reachability, and provides a guidance ceiling for measuring how much corpus-reachable capability a method internalises.Its comparison unit therefore differs from benchmarks focused on local interaction-stream adaptation or open-ended improvement.

6 Conclusion

StudyBench directly measures how efficiently self-evolution converts fixed training material into transferable problem-solving capability. Its results show strong local application gains but limited transfer, with remaining deficits attributed to methods rather than data or compute.

  • StudyBench measures knowledge-to-capability conversion using a controlled physics benchmark.
  • GEPA lifts Qwen3-8B Application Par@8 from 17.05 to 34.85, yet Transfer Par@8 reaches only 7.04 against a 100% guidance ceiling.
  • Application-Set gains do not become olympiad capability, and the same local-gain pattern holds on Llama-3.2-3B-Instruct and Opus 4.7.
  • Profiled self-evolution loops hit a Compute Plateau before exhausting their compute budget.
  • The remaining gap is therefore a method problem rather than a data or compute problem.

Limitations

StudyBench’s guarantees and conclusions are bounded by its Qwen3-8B-centered filtering, physics-only setting, selected evaluation conditions, and restricted source-material release.

  • Capability-gap and 100% guidance-ceiling guarantees apply only to Qwen3-8B under DeepSeek V4 Pro traces.Opus 4.7 already solves a substantial fraction of both splits.
  • The benchmark is instantiated in physics over 11 textbooks, and its construction principles have not been verified in other disciplines.
  • Fifteen Application parents were admitted despite one Qwen3-8B success in eight attempts, relaxing the Capability Gap to preserve easier subjects.
  • Compute constraints limit the guidance ablation and compute-plateau curves, so neither result is claimed for every method–model pair.
  • The public release omits copyrighted textbook PDFs and raw passages, requiring Corpus-dependent methods to rebuild them from legal copies.

D Alternative-teacher guidance

An independent GLM-5.1 teacher tests whether Transfer-Set reachability depends on DeepSeek V4 Pro’s guidance traces. The resulting scores show that many retained problems remain reachable under a teacher swap.

  • GLM-5.1 guidance unlocks 56 of 90 Transfer parents, achieving 62.22 Par@8.
  • GLM-5.1 guidance also unlocks 242 of 280 Transfer sub-problems, achieving 86.43 Sub@8.
  • DeepSeek V4 Pro retains a 100% score by construction of the Naive Reachability Filter.
  • A majority of Transfer parents are reachable under two independently written guidance traces, so the reachability witness is not an artifact of one teacher’s style.
  • Coverage is broad rather than concentrated: every source and sub-discipline contributes at least one solved parent.

G Verifier consistency

StudyBench uses a rule-based verifier followed by an LLM fallback and measures their agreement on Qwen3-8B Transfer-Set judgments. The rule-based stage is precise but misses many equivalent answers.

  • The two-stage verifier combines a type-aware rule-based judger with DeepSeek-V4-Flash-0731 on residual cases.
  • The stages agree on 4,967 of 6,720 attempt-level judgments, or 73.91%.
  • The rule-based judger has precision 1 and recall 29.65% under the cascade’s reference labels.
  • The rule-based stage is exposed as the RL reward, while the LLM fallback is reserved for leaderboard evaluation.
  • Recovered equivalences concentrate on EQ, IN, NV, TUP, and QL, including algebraic rearrangements, unit synonyms, tolerances, and qualitative aliases.

H Application Set redaction from the training material

The benchmark redacts retained Application Set problems and their solutions from training textbooks to prevent answer leakage. It uses source-specific boundaries and a two-pass audit process because the contributing textbooks organize exercises heterogeneously.

  • Training textbooks have both the problem statement and reference solution redacted for every retained Application Set parent.This prevents releasing textbook material that directly contains every Application Set answer.
  • The redaction runs in two passes: first problem statements, then corresponding solutions and answer keys.Solutions may be stored in separate manuals or files, including chapter-level and aggregate sources.
  • Source-specific layout rules determine block boundaries because the eight contributing textbooks use heterogeneous exercise layouts.

I Convergence curves for the remaining methods

Across the remaining methods, Application-Set performance follows distinct convergence patterns under method-specific GPU-time budgets. GEPA plateaus, Intuitor and R-Zero peak before declining, while Bonito remains near baseline.

  • GEPA climbs to a plateau, whereas Intuitor and R-Zero peak and decline, and Bonito never leaves the baseline band.The figure panels use method-specific GPU-time axes while fixing the accuracy axis at 25–45% for visual comparison.

J Replication detail

The replication uses thin adapters around upstream self-evolution implementations, while preserving each method’s original loop and aligning data loading, evaluation, and training protocols with StudyBench.

  • Replication setup: The replications preserve upstream implementations wherever possible, using thin wrappers and StudyBench-specific data loaders, scorers, adapters, and launch scripts.The stated goal is to align methods under a common evaluation pattern without modifying their core loops.
  • GEPA: GEPA evolves a system prompt through reflective genetic search using a 32-item validation set and a StudyBench-specific adapter.The adapter matches the benchmark’s multi-turn generation protocol and rule-plus-LLM verifier.
  • Bonito: Bonito adds physics-problem extraction and feeds generated instructions back into Qwen3-8B for one self-evolution cycle.The corpus is chunked into 2,048-token passages before instruction extraction.
  • EvoSkill: EvoSkill evolves a frontier of reusable skill files through failure-driven proposals, generators, and evaluators rather than revising one instruction in place.The replication keeps skill-only mutations, a size-3 frontier, and a five-iteration stopping rule without improvement.
  • GRPO methods: GRPO-based replications use benchmark-aligned preprocessing and verification, while Intuitor and TTRL retain label-free rewards based on self-certainty and majority-vote consistency.GRPO is trained with the StudyBench protocol rather than the upstream package’s built-in GSM8K/MATH evaluation.
Loading 2609.00787v1…