PaperlayerView exact revision on arXiv

Source-linked AI summary

TTCS: Test-Time Curriculum Synthesis for Self-Evolving

Chengyi Yang, Zhishang Xiang, Yunbo Tang, Zongpei Teng, Chengsong Huang, Fei Long, Yuhan Liu, Jinsong Su

arXiv:2601.22628v1cs.LGcs.AIcs.CL

TL;DR

Difficult test-time reasoning is limited by unreliable pseudo-labels and too few learnable samples. TTCS co-evolves a question synthesizer and reasoning solver through curriculum construction and self-supervised updates, achieving gains across mathematical and general reasoning benchmarks.

  • Problem

    Existing test-time training struggles on difficult reasoning problems because majority-vote pseudo-labels can be unreliable and small test sets lack intermediate learnable samples.

  • Method

    TTCS initializes a capability-aware synthesizer and online solver from the same pretrained model, co-evolving them through iterative GRPO on original and synthetic questions.

  • Results

    TTCS consistently strengthens performance across mathematical and general reasoning benchmarks, including +4.12 average points over TTRL on Qwen2.5-Math-7B.

  • Takeaways & Limitations

    TTCS dynamically constructs tractable curriculum variants that provide valid supervision and bridge difficult test questions for self-evolution.

Abstract

from arXiv · show

Test-Time Training offers a promising way to improve the reasoning ability of large language models (LLMs) by adapting the model using only the test questions. However, existing methods struggle with difficult reasoning problems for two reasons: raw test questions are often too difficult to yield high-quality pseudo-labels, and the limited size of test sets makes continuous online updates prone to instability. To address these limitations, we propose TTCS, a co-evolving test-time training framework. Specifically, TTCS initializes two policies from the same pretrained model: a question synthesizer and a reasoning solver. These policies evolve through iterative optimization: the synthesizer generates progressively challenging question variants conditioned on the test questions, creating a structured curriculum tailored to the solver's current capability, while the solver updates itself using self-consistency rewards computed from multiple sampled responses on both original test and synthetic questions. Crucially, the solver's feedback guides the synthesizer to generate questions aligned with the model's current capability, and the generated question variants in turn stabilize the solver's test-time training. Experiments show that TTCS consistently strengthens the reasoning ability on challenging mathematical benchmarks and transfers to general-domain tasks across different LLM backbones, highlighting a scalable path towards dynamically constructing test-time curricula for self-evolving. Our code and implementation details are available at https://github.com/XMUDeepLIT/TTCS.

1 Introduction

Existing test-time training struggles on difficult reasoning tasks because pseudo-labels become unreliable and raw test questions provide too few learnable intermediate examples. TTCS addresses these issues with a co-evolving synthesizer–solver curriculum.

  • TTRL can reinforce incorrect reasoning when majority voting on difficult questions produces systematically noisy pseudo-labels.AIME24 is cited as an example where incorrect consensus misguides policy updates.
  • Direct optimization on a small set of extremely difficult test questions leaves TTRL without intermediate samples within the model’s capability.The resulting learning process is described as steep and often unclimbable.
  • TTCS constructs diverse, solvable variants that preserve the underlying reasoning structure while matching the solver’s capability frontier.These variants provide valid supervision for bridging the gap to complex test questions.
  • TTCS co-evolves a synthesizer and solver initialized from the same pretrained model through iterative GRPO optimization.The solver evaluates synthetic-question quality implicitly, while training on original and synthetic questions with self-supervised rewards.
  • Experiments report strong performance and generalization across mathematical and general reasoning benchmarks.The conclusion describes substantial mathematical gains and transfer to broader general-domain reasoning tasks.

2 Related Work

The related work positions self-evolving methods and test-time training as routes to improving LLM reasoning without relying on additional human supervision. TTRL adapts models on unlabeled test instances using majority-vote pseudo-labels.

  • Self Evolving for LLMs: Self-evolving research seeks to augment LLM reasoning through self-generated supervision and environmental interactions.
  • Self Evolving for LLMs: Earlier self-evolving approaches include fine-tuning on high-confidence reasoning trajectories, iterative self-play, and role-specialized frameworks.
  • Test-Time Training (TTT): Test-time training dynamically adapts model parameters during inference using self-supervision to address training–testing distribution shifts.
  • Test-Time Training (TTT): TTRL applies reinforcement learning to unlabeled test data by deriving pseudo-labels from multiple sampled responses.The approach allows language models to self-improve during inference.

3 Preliminary

This section introduces GRPO and test-time training as the foundations for TTCS. GRPO normalizes group rewards to form advantages, while TTT adapts a pretrained model using a label-free objective based on majority-vote pseudo-labels.

  • Group Relative Policy Optimization (GRPO): GRPO models language generation as a policy πθ mapping input questions x to responses y and maximizes expected outcome-based reward when ground truth is available.Correctness is represented by a binary reward function R(y, y*).
  • Group Relative Policy Optimization (GRPO): For each question, GRPO samples a group of outputs from the current policy and computes each output’s advantage from group-level reward statistics.
  • Group Relative Policy Optimization (GRPO): The GRPO update uses normalized rewards and a clipped surrogate objective with KL regularization to constrain policy changes.The clip function maintains a trust region, while the KL term limits excessive deviation from the previous policy.
  • Test-Time Training (TTT): TTT adapts a pretrained model’s parameters during inference using only test questions and a label-free objective.It is designed to mitigate distribution shifts between training and testing environments.
  • Test-Time Training (TTT): TTRL derives the pseudo-label y-hat* from the most frequent sampled response and rewards agreement with that consensus.

4 Test-Time Curriculum Synthesis

TTCS trains a synthesizer to generate capability-aligned, diverse variants of test questions while an adaptive solver learns from both original and synthetic data. Their iterative co-evolution targets the solver’s capability frontier, uses self-consistency filtering, and limits collapse through test-question resampling.

  • Framework: TTCS uses a synthesizer policy and solver policy in an iterative GRPO loop, with both initialized from the same pretrained model.The synthesizer generates curriculum variants, while the solver trains on test questions and synthetic questions.
  • Test Questions Guided Synthesis: Each synthetic question preserves the test question’s reasoning structure while varying objects, settings, or constraints to provide focused, task-relevant training signals.The variants differ in surface realization while remaining aligned with the reference questions in distribution.
  • Question Quality Reward: The solver assesses synthetic questions with composite rewards that target its capability frontier, where self-consistency variance is highest between consistently solved and consistently failed problems.Self-consistency scores are computed from sampled responses and majority-vote answers to estimate question difficulty.
  • Similarity Penalty Reward: Similarity penalties discourage copying test questions and redundancy among generated samples, while weighted coefficients control the reference-based and group-level penalties.The final reward combines capability objectives with diversity constraints and includes a format-compliance indicator.
  • Training Data Construction at Test Time: The solver repeatedly samples test questions and augments them with synthetic variants, preventing self-generated data from dominating the training distribution.This resampling strategy addresses the small size of the test set and helps prevent model collapse.
  • Self-Consistency Reward for Solver: Solver updates use majority-vote pseudo-labels, agreement-based rewards, and online filtering that retains samples with consistency scores near 0.5.The filtering condition is |s(x) − 0.5| ≤ δ, and the procedure uses unlabeled test-time data with self-generated supervision.

5 Experiment

TTCS is evaluated across mathematical benchmarks, model backbones, generalization settings, data regimes, and component ablations. It consistently improves performance over test-time training baselines while showing that adaptive synthesis, filtering, diversity, and co-evolution contribute to its effectiveness.

  • Experimental Setting: TTCS evaluates competition-level and fundamental mathematical benchmarks across Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-4B-Base.The evaluation reports mean@32 for AIME24/25 and compares TTCS with pretrained, Self-Consistency, TTRL, and R-Zero baselines.
  • Main Results: +24.19 points: TTCS raises Qwen2.5-Math-1.5B average accuracy from 17.30 to 41.49 and outperforms baselines across most tasks and models.On Qwen2.5-Math-7B, TTCS reaches 52.54 average points, exceeding Self-Consistency by +20.39 points.
  • Main Results: +4.12 points: TTCS exceeds TTRL on Qwen2.5-Math-7B average performance, improving from 48.42 to 52.54, and also leads on Qwen3-4B-Base by +3.62 points.The gains are attributed to synthesized intermediate problems that address the fixed difficulty of raw test questions during test-time training.
  • Main Results: +6.56 points: On AIME24 with Qwen2.5-Math-1.5B, TTCS improves over TTRL from 13.23 to 19.79; on Qwen2.5-Math-7B AIME25, it leads by +5.84 points.The reported results indicate that synthesized curriculum problems provide supervision when raw questions yield unreliable pseudo-labels.
  • Generalization Analysis: TTCS generalizes across domains and unseen datasets, including general-domain benchmarks evaluated during AIME25 test-time training and OOD mathematical benchmarks.Training solely on MATH-500 increases AIME24 accuracy from 7.1 points to 12.9 points.
  • Ablation Study: With 10% of AIME24 data, TTCS reaches 13.33 points versus TTRL’s 9.48, while ablations show losses from static synthesis, unfiltered rollouts, and removed diversity penalties.Accuracy decreases from 62.50 to 55.00 on AMC23 without synthesizer training and from 36.05 to 33.68 on Olympiad without online filtering.

6 Conclusion

TTCS addresses unreliable pseudo-labels and missing intermediate-difficulty samples by constructing tractable variants for test-time self-evolution. Experiments report gains on mathematical benchmarks and general-domain reasoning tasks.

  • TTCS constructs a curriculum of tractable problem variants to address unreliable pseudo-labels and absent intermediate-difficulty learnable samples.The approach uses a capability-aware synthesizer to bridge the learning gap during test-time training.
  • TTCS is a co-evolving test-time training framework for self-evolving.
  • TTCS delivers substantial gains on mathematical benchmarks and generalizes to broader general-domain reasoning tasks.

A.1 Datasets

The evaluation covers competition-level mathematics, standard mathematical proficiency, and general-domain reasoning benchmarks. These datasets span advanced multi-step problems, STEM and Olympiad tasks, and challenging language-understanding settings.

  • Competition-Level Mathematics: Competition-level evaluation uses AMC23 and AIME24&25, whose problems require multi-step reasoning and deep mathematical insight.
  • Standard Mathematical Benchmarks: Standard mathematical evaluation includes MATH-500, Minerva, and OlympiadBench.These datasets assess mathematical problem-solving, STEM proficiency across difficulty levels, and Olympiad-level reasoning.
  • General-Domain Benchmarks: Generalization beyond mathematics is evaluated with BBEH, MMLU-Pro, and SuperGPQA.The benchmarks target difficult language-model tasks, multi-task language understanding and reasoning, and graduate-level questions.

A.2 Evaluation Metrics

The evaluation combines benchmark-specific metrics with pretrained, self-consistency, and test-time-training baselines. Mathematical outputs receive additional answer-matching judgments where format variation matters.

  • Metrics: Mathematical evaluation uses benchmark-tailored metrics and GPT-4o-mini assistance to judge predicted-answer matches under output-format variation.
  • Metrics: Mean@32 averages correctness across 32 stochastic solutions per question for AIME24 and AIME25.Solutions use temperature T = 0.6, providing an estimate of expected pass rate for unstable single runs.
  • Baselines: The pretrained model provides a static zero-shot benchmark for measuring method gains.
  • Baselines: Self-Consistency samples multiple reasoning paths and aggregates final answers through majority voting.
  • Baselines: TTRL performs reinforcement learning directly on unlabeled test instances using repeated rollouts to estimate pseudo-labels by majority voting.
  • Baselines: R-Zero enables data-free Challenger-Solver co-evolution through reinforced self-play without ground-truth annotations.

A.4 Implementation Details

TTCS uses documented training configurations for its synthesizer and solver agents. For challenging mathematics benchmarks, batch and rollout-group sizes are adjusted to support stability and resource-efficient training.

  • For AIME24, AIME25, and AMC23, the batch size and solver rollout group size are both set to 16.These adjustments target training stability and efficient resource utilization.
  • Table 5 summarizes training configurations and hyperparameters for the synthesizer and solver agents.

B Additional Experimental Reults

Additional evaluations show that TTCS transfers beyond its source datasets, improving general-domain reasoning and remaining robust across out-of-distribution mathematical benchmarks.

  • General-domain transfer: TTCS shows a substantial advantage over baselines on general reasoning benchmarks when trained on either AIME24 or AIME25.The evaluations include BBEH, MMLU-Pro, and SuperGPQA.
  • Out-of-distribution robustness: TTCS consistently outperforms TTRL across diverse out-of-distribution transfer scenarios between mathematical benchmarks.Models are optimized on source datasets such as AIME24 or MATH500 and evaluated on distinct mathematical benchmarks.
  • Interpretation: The reported transfer results indicate that TTCS acquires transferable reasoning skills rather than merely overfitting to the source distribution.The appendix presents cross-task mathematical transfer and general-domain evaluations to assess robustness and generalization.

C.1 Theoretical Analysis: Variance-Driven Generative Synthesis

The analysis explains TTCS’s capability-adaptive reward as a variance-seeking signal: self-consistency estimates solver correctness, and the synthesizer targets questions near the solver’s capability frontier. The section also describes similarity penalties, iterative policy updates, and prompt constraints that promote diverse, verifiable synthetic problems.

  • Capability-adaptive reward: The capability-adaptive reward 4s(x′)(1−s(x′))^γ targets synthetic questions where solver uncertainty and outcome variance are highest.The reward peaks at s(x′)=0.5; γ controls how sharply high-uncertainty samples are selected.
  • Gradient signal and uncertainty: When pθ(x′) approaches 0 or 1, Bernoulli reward variance and expected gradient magnitude collapse; near 0.5, both are maximized.This motivates filtering solver-training samples whose self-consistency scores satisfy |s(x)−0.5|≤δ.
  • Self-consistency as a proxy: Self-consistency is used as a practical proxy for correctness because its sample mean concentrates around pθ(x′) as the response-group size G grows.The argument assumes bounded Bernoulli rollouts and no systematic bias.
  • Similarity penalties: The reference-similarity penalty hierarchically rejects direct textual overlap first, then structural redundancy when auxiliary text or Jaccard evidence exceeds its thresholds.The auxiliary condition is Stext>0.45 or Sjacc>0.40, with adaptive text and skeleton thresholds also used.
  • Co-evolution procedure: TTCS alternates synthesizer and solver GRPO updates: generated questions are evaluated for consistency, while mixed original and synthetic data are filtered near the solver’s uncertainty frontier.The algorithm samples auxiliary questions, computes consistency and capability rewards, applies similarity penalties, and updates both policies iteratively.
  • Synthetic-question design: The synthesis prompt requires structurally isomorphic but surface-distinct problems, a changed mathematical object or setting, verifiable complexity, and a single scalar answer.It also bans immediate algebraic equivalents, proofs, subparts, functions, text or Boolean outputs, and non-unique set answers.
Loading 2601.22628v1…