PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learning

Jonas Hübotter, Leander Diaz-Bone, Ido Hakimi, Andreas Krause, Moritz Hardt

arXiv:2510.04786v1cs.LGcs.AI

TL;DR

Bounded context limits how long LLMs can learn in-context from experience on target tasks. TTC-RL automatically curates related training tasks and continues training through reinforcement learning. The approach improves reasoning across target tasks and models, raising performance ceilings.

  • Problem

    Because LLM context is bounded and costly to expand, models cannot learn in-context from experience over long timeframes.

    Source: §1 · p. 2

  • Method

    TTC-RL selects target-relevant tasks from a large corpus, then uses reinforcement learning on the agent’s experience attempting those tasks.

    Source: §1 · p. 2

  • Results

    TTC-RL improves reasoning across diverse target tasks and models and significantly raises strong initial models’ performance ceilings.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Targeted test-time training offers a way for models to meta-learn how to reason for a particular task through task-relevant experience.

    Source: §6 · p. 10

    The authors cannot guarantee the cleanliness of the model’s original pretraining data.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Humans are good at learning on the job: We learn how to solve the tasks we face as we go along. Can a model do the same? We propose an agent that assembles a task-specific curriculum, called test-time curriculum (TTC-RL), and applies reinforcement learning to continue training the model for its target task. The test-time curriculum avoids time-consuming human curation of datasets by automatically selecting the most task-relevant data from a large pool of available training data. Our experiments demonstrate that reinforcement learning on a test-time curriculum consistently improves the model on its target tasks, across a variety of evaluations and models. Notably, on challenging math and coding benchmarks, TTC-RL improves the pass@1 of Qwen3-8B by approximately 1.8x on AIME25 and 2.1x on CodeElo. Moreover, we find that TTC-RL significantly raises the performance ceiling compared to the initial model, increasing pass@8 on AIME25 from 40% to 62% and on CodeElo from 28% to 43%. Our findings show the potential of test-time curricula in extending the test-time scaling paradigm to continual training on thousands of task-relevant experiences during test-time.

1 INTRODUCTION

The paper asks how LLMs can keep improving on target tasks when their context is bounded, and proposes self-curated test-time curricula with reinforcement learning. TTC-RL improves target-task reasoning and raises performance ceilings across diverse tasks and models.

1 INTRODUCTION

  • Bounded context limits how long an LLM can learn in-context from experience, motivating continued training for target tasks at test-time.The introduction contrasts this approach with test-time scaling based on extending context for reasoning.Source: §1 · p. 2
  • TTC-RL consistently improves pass@1 across diverse reasoning tasks and models, raises pass@k ceilings, and complements other test-time scaling approaches.The authors also report that a non-thinking model with TTC-RL can perform similarly to the same model thinking for 30k tokens in context.Source: §1 · pp. 2–3
  • TTC-RL automatically selects related training tasks from a large corpus, then uses reinforcement learning on the agent’s task experience to specialize for the target.The approach avoids laborious human curation of datasets.Source: §1 · p. 2
  • The paper introduces latent improvement, a lower bound on reasoning gains from RL, and reports substantial latent reasoning improvements with TTC-RL.The metric addresses the challenge of distinguishing better reasoning from learning an expected output format.Source: §1 · p. 3

2 RELATED WORK

Related work improves LLM performance through test-time compute, general-purpose reinforcement learning, self-play, and test-time training. This paper places targeted test-time RL among these approaches, extending curriculum learning to challenging reasoning tasks.

Test-time scaling and general-purpose RL training

  • Test-time scaling uses methods such as majority voting, reward-model search, and in-context reasoning, while general-purpose RL helps models reason within bounded context.The paper distinguishes its goal of training specialized reasoners for particular target tasks at test-time from general-purpose model improvement.Source: §2 · p. 3

Self-play

  • Prior LLM self-play work generates training curricula, while earlier test-time curricula focused on simple robotic navigation; this work extends curricula to challenging reasoning tasks.The authors frame their approach as enabling continual learning from extensive experience on a single task.Source: §2 · p. 3

Test-time training and test-time RL

  • Test-time training has used supervised or self-supervised losses, and recent unsupervised test-set RL used majority votes as pseudo-labels.The cited work connects test-time RL with unsupervised RL and LLM self-improvement.Source: §2 · p. 3

3 TEST-TIME CURRICULA

The paper defines a test-time curriculum as a targeted subset of training tasks and lets the initial model select relevant tasks for specialization. It describes the SIFT-and-GRPO training pipeline and assembles a diverse corpus of verifiable tasks.

3 TEST-TIME CURRICULA

  • A test-time curriculum is a targeted subset of a large training corpus, and the paper proposes letting the initial model craft that subset rather than relying on human curation.The training tasks have verifiers that indicate whether an attempt was correct.Source: §3 · p. 3

3.1 AUTOMATIC TTCS FOR TARGETED RL

  • SIFT selects informative, target-relevant examples using the model’s latent representations, trading off relevance and diversity; GRPO then trains on the selected curriculum.The authors omit GRPO’s KL penalty and also evaluate selecting tasks within a desired difficulty range.Source: §3.1 · p. 4

3.2 A DIVERSE CORPUS FOR GENERAL-PURPOSE RL POST-TRAINING

  • The verifiable-corpus combines approximately 265k tasks across math, general reasoning, and code, using answer matching, a verifier model, or unit tests as appropriate.The corpus filters tasks with empty answers or fewer than five unit tests, and is deduplicated and decontaminated.Source: §3.2 · pp. 4–5
  • The evaluation benchmarks cover math, coding, and scientific reasoning, with Table 1 marking the best performance per model backbone and percentage-point gains over the initial model.The caption names AIME24, AIME25, MATH500, Codeforces, CodeElo, LCBv6, and GPQA-D.Source: Table 1 · p. 5

4 RESULTS

Across the evaluated models and benchmarks, TTC-RL outperforms general-purpose RL post-training and raises performance ceilings. It also specializes models to target tasks, with transfer among related tasks.

4 RESULTS

  • Evaluations span math, coding, and scientific reasoning, primarily using non-thinking Qwen3 models capped at 8,192 tokens; per-benchmark evaluation supports statistical significance under limited compute.Source: §4 · p. 5

4.1 TTCS ARE COMPLEMENTARY TO EXISTING APPROACHES TO TEST-TIME SCALING

  • TTC-RL improves pass@k across a wide range of k and strengthens majority voting; after further RL, pass@1 exceeds the initial model's maj@64.Source: Figure 4 · p. 6, §4.1 · p. 7

4 RESULTS

  • TTC-RL improves accuracy across models and benchmarks, learns faster, and saturates higher than uniform-task RL; it also outperforms general-purpose RL across curriculum sizes.Figure 3 reports average test accuracy after 250 steps across seven benchmarks.Source: Figure 3 · p. 6, §4 · p. 6

4.1 TTCS ARE COMPLEMENTARY TO EXISTING APPROACHES TO TEST-TIME SCALING

  • With 8k tokens, non-thinking Qwen3-8B + TTC-RL performs close to the 30k-token thinking variant in several domains, particularly coding and GPQA.Source: Figure 4 · p. 6, §4.1 · p. 7

4.2 TTCS EFFECTIVELY SPECIALIZE MODELS

  • Models peak on their target tasks and transfer better among related tasks, especially math; coding-to-math transfer exceeds the reverse, as does code/math-to-GPQA transfer.Source: §4.2 · p. 7, Figure 5 · p. 8
  • Per-question TTC-RL beats benchmark-level TTC on AIME25, but finer specialization is not always beneficial: separate GPQA science curricula perform approximately like a joint TTC.Source: §4.2 · p. 7, Figure 5 · p. 8

5 FURTHER ANALYSIS

The section evaluates whether TTC-RL improves reasoning beyond answer-format learning, relates test-time curricula to self-improvement methods, and discusses corpus and contamination limits.

5.1 ESTIMATING “REAL” IMPROVEMENT

  • Under Assumption 1, TTC-RL produces substantial latent improvement, a lower-bound estimate of gains in reasoning correctness.The estimate relies on strict answer extraction; unintended extracted answers can artificially inflate it and violate the assumption.Source: §5.1 · pp. 8–9, §5.3 · p. 9

5.2 TOWARDS CONTINUAL SELF-IMPROVEMENT AT TEST-TIME

  • Maj-TTRL and TTCs are complementary: TTCs work wherever rewards can be defined, while Maj-TTRL relies on majority votes and structured outputs.Applying Maj-TTRL after TTC-RL outperforms Maj-TTRL alone.Source: §5.2 · p. 9
  • Maj-TTRL gains on Qwen3-4B-Instruct-2507 suggest TTC-RL’s returns are constrained by its fixed training corpus, not necessarily by model capability.The authors identify self-improvement methods and synthetic data as possible ways to extend or combine with TTCs.Source: §5.2 · p. 9

5.3 ON CONTAMINATION AND REWARD HACKING

  • The authors decontaminated training tasks and removed Codeforces and CodeElo input/output examples after finding private test cases leaked into some evaluation prompts.They also manually reviewed model responses and added safeguards against reward hacking.Source: §5.3 · p. 9
  • Pretraining-data cleanliness cannot be guaranteed; gains on LCBv6, with tasks released since February 2025, make prior contamination a less likely explanation.Against an oracle trained directly on test tasks, TTC-RL learns slightly more slowly and levels off at lower accuracy.Source: §5.3 · p. 9

6 DISCUSSION

The discussion frames TTCs as targeted test-time training between general-purpose training and frozen test-time scaling, and outlines extensions beyond fixed curricula and single-task settings.

6 DISCUSSION

  • TTC agents self-curate training tasks and specialize toward a target task through reinforcement learning.The paper reports gains across models and reasoning benchmarks, including substantial gains in latent improvement.Source: §6 · p. 10
  • Targeted test-time training sits between large-scale general-purpose training and frozen test-time scaling.The authors describe TTCs as enabling models to meta-learn how to reason for a particular target task at test-time.Source: §6 · p. 10
  • Proposed directions include self-generated curricula, more sample- and step-efficient RL, scaling-law studies, dynamic target tasks, and standardized test-task specialization.The paper also raises whether scaling TTC-RL or the context window is preferable at different context lengths.Source: §6 · p. 10

CONTRIBUTIONS

The contribution statement credits the project’s conception, leadership, implementation, experiments, infrastructure, and advice across the named authors.

CONTRIBUTIONS

  • JH and IH conceived the project, with MH contributing to the direction of TTC-SFT.Source: CONTRIBUTIONS · p. 10
  • JH led the project and writing with significant help from LDB; JH and LDB contributed equally to implementation and experiments.Source: CONTRIBUTIONS · p. 10
  • IH supported infrastructure, while IH, AK, and MH advised the project.Source: CONTRIBUTIONS · p. 10

A WHY IMITATION LEARNING IS ILL-SUITED FOR TTC’S

The appendix contrasts test-time SFT with on-policy RL, describes SIFT’s uncertainty-based task selection, and connects GRPO’s learning signal to task success rates.

A WHY IMITATION LEARNING IS ILL-SUITED FOR TTC’S

B.1 SIFT

  • SIFT greedily selects corpus tasks to reduce posterior uncertainty about responding to a target prompt, trading relevance against diversity through λ.Large λ favors relevance; small λ favors diversity.Source: §B.1 · p. 18

B.2 GRPO

  • In sparse-reward GRPO, selecting questions with a 50% current success rate is suggested to maximize learning signal.The argument links high absolute advantage to a larger GRPO gradient norm.Source: §B.2 · p. 19

C AUTOBALANCING ACHIEVABILITY WITH TTC’S

The appendix tests curriculum difficulty calibration and several training comparisons and ablations. It reports that balancing task relevance with achievability can help weaker models, while combining TTC-RL with Maj-TTRL gives the strongest math results.

C AUTOBALANCING ACHIEVABILITY WITH TTC’S

  • A-TTC balances target-task relevance with estimated achievability, selecting relevant tasks at an appropriate difficulty.The procedure updates task-difficulty estimates online from batch rewards and selects tasks within an achievability interval.Source: C AUTOBALANCING ACHIEVABILITY WITH TTC’S · p. 20
  • On Qwen3-0.6B, balancing achievability with target-task relevance improves training reward and test scores across AIME24, AIME25, and MATH500.The procedure appears most useful when dataset difficulty is miscalibrated to the model’s capabilities.Source: C AUTOBALANCING ACHIEVABILITY WITH TTC’S · p. 20, Figure 8 · p. 20

D.1 INCREASING CLIP-HIGH IN GRPO IS ESSENTIAL FOR LEARNING

  • Increasing GRPO’s clip-high parameter stabilizes policy entropy and improves test accuracy, while symmetric clipping at ϵhigh = 0.2 yields declining entropy and poor test performance.The authors report that ϵhigh = 0.32 significantly outperformed ϵhigh = 0.28 in preliminary Codeforces experiments.Source: §D.1 · p. 21, Figure 9 · p. 21

D.2 PERFORMANCE VS. STEP

  • Models show substantial variation in test-accuracy development under the same TTC, motivating calibration of curriculum difficulty to each model’s capabilities.The passage attributes variation to models’ different initial capabilities and post-training histories.Source: §D.2 · p. 21

D.3 “RL POST-TRAINING” BASELINE RESTRICTED TO THE TEST ENVIRONMENT

  • Filtering training questions to the code domain is insufficient to match TTC-RL on Codeforces and CodeElo.The comparison uses a random training subset restricted to the target domain, without selecting by instance-level task characteristics.Source: §D.3 · p. 21, Figure 11 · p. 23

D.4 EXTENDED COMPARISON AND COMBINATION OF TTC-RL WITH MAJ-TTRL

  • TTC-RL outperforms Maj-TTRL on most benchmarks for Qwen3-8B and Qwen3-4B-Instruct-2507, with Maj-TTRL ahead only on the latter model.The authors identify the dataset as a performance bottleneck and suggest moving beyond a fixed task corpus.Source: §D.4 · p. 23, Table 4 · p. 23
  • Initializing Maj-TTRL from the final TTC-RL checkpoint yields the strongest results on all math benchmarks.Maj-TTRL uses majority labels as rewards on target tasks and is limited to structured-output domains, unlike TTC-RL’s corpus-based training.Source: §D.4 · p. 23, Figure 12 · p. 24

D.5 ADDITIONAL BENCHMARKS

Additional evaluations show gains on simpler math and coding benchmarks, while further analyses examine training-time improvements and pass@k and latent-improvement results. An entropy cooldown appears to help math slightly but not performance generally.

D.5 ADDITIONAL BENCHMARKS

  • TTC-RL yields substantial gains on simpler math and coding benchmarks, especially for Qwen3-8B-Base; gains are less pronounced for Qwen3-8B.The authors suggest the verifiable corpus may have fewer useful tasks at the needed complexity, or these benchmarks may leave less room for further improvement.Source: §D.5 · p. 24, Table 5 · p. 24

D.6 FURTHER RESULTS AND ABLATIONS

  • Figure 13 shows TTC-RL’s marginal improvement over general-purpose RL post-training remains large throughout training; Tables 6 and 7 provide pass@k and latent-improvement breakdowns.Table 6 reports pass@k for Qwen3-8B before and after TTC-RL on the main benchmarks; Table 7 covers latent improvement normalized for output-format learning.Source: §D.6 · p. 24, Figure 13 · p. 25, Table 6 · p. 25
  • The appendix also compares TTC-RL with oracle training on the test set.Figure 14 presents the ablation on Codeforces and AIME25.Source: §D.6 · p. 24, Figure 14 · p. 25

D.7 UNSUCCESSFUL ATTEMPTS

  • Entropy cooldown with ϵhigh = 0.2 appears to improve math performance slightly, but not performance generally.The experiment continued training after increasing clip-high to stabilize entropy.Source: §D.7 · p. 24, Figure 15 · p. 25

E.1 DATASET

The appendix describes the multi-domain training corpus, data processing, prompts, and RL-training details, then gives qualitative examples of training tasks and responses.

E.1 DATASET

  • The training corpus combines math, coding, and WebInstruct-verified data in a unified task schema.Sources include DAPO-Math-17k, Hendrycks MATH, GSM8K, LiveCodeBench, TACO, PrimeIntellect, Codeforces train, CodeContests, and LeetCode.Source: §E.1 · p. 27
  • The corpus is decontaminated against held-out benchmarks, deduplicated within domains, and filtered for malformed or underspecified tasks.The procedures include similarity-based contamination checks, domain-specific deduplication thresholds, and minimum code-description and test requirements.Source: §E.1 · p. 27

E.2 SYSTEM PROMPTS

  • TTC selection uses task descriptions and environment information, with SIFT embeddings computed using Qwen3-8B.The paper also provides math, coding, and multiple-choice system prompts adapted from evalchemy and says prompts were not tuned for performance.Source: §E.1 · p. 27, §E.2 · pp. 27–28

E.3 DETAILS OF THE RL TRAINING

  • RL hyperparameters remain fixed across models, benchmarks, and baselines, and code training uses only the first 20 test cases per task.Training rewards include a format penalty for failed answer extraction, but not for truncation; WebInstruct-verified tasks also receive a length penalty.Source: §E.3 · pp. 28–29

F QUALITATIVE EXAMPLES

  • The qualitative examples include a CodeElo problem asking for the kth-smallest value among subarrays’ minimum pairwise XOR values.The appendix shows two attempted Python solutions for the problem.Source: F QUALITATIVE EXAMPLES · p. 30, §F.1 · pp. 30–31

F.2 AIME 25, QUESTION 26

This appendix section presents task-specific TTC-RL examples across AIME25 and CodeElo, including answer artifacts, selected training problems, and coding-task prompts.

F.2 AIME 25, QUESTION 26

  • For AIME25 Question 26, the appendix presents Qwen3-8B’s initial answer, its answer after 250 TTC training steps, and the first 10 selected training problems.The passage introduces these as materials for a single question-specific curriculum example.Source: §F.2 · p. 31
  • The illustrated AIME25 task asks about a non-convex 11-gon with perimeter 20 and seeks a value expressed using integers and a square root.The supplied problem text specifies triangle areas of 1 and cosine values of 12/13 for the angles at A1.Source: §F.2 · p. 31

F.3 TTC FOR CODEELO

  • For CodeElo, the appendix lists 10 problems selected by SIFT as relevant to improving benchmark performance.The subsequent excerpts include algorithmic coding problem statements.Source: §F.3 · p. 35
  • The tree task asks for the smallest achievable height after moving a subtree to become a child of the root.The prompt defines height as the maximum vertex depth and depth as the number of edges from the root.Source: §F.3 · p. 37
  • In the interactive box task, choose three distinct indices so one box’s label divides the other two, then give the third box to ROBO.The given box becomes unavailable for later actions.Source: §F.3 · p. 37
  • For an odd n×n binary grid, queries ask whether any right/down path between specified nonadjacent cells spells a palindrome.The top-left cell is known to contain 1 and the bottom-right cell 0; query cells are ordered down and right.Source: §F.3 · pp. 37–38

Related papers

Loading 2510.04786v1…