PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Improving LLM Agents at Test-Time

Emre Can Acikgoz, Cheng Qian, Heng Ji, Dilek Hakkani-Tür, Gokhan Tur

arXiv:2510.07841v1cs.LGcs.AIcs.CL

TL;DR

Large-scale inductive fine-tuning is costly and may waste data on redundant information while remaining vulnerable to complex or shifted test scenarios. This paper proposes TT-SI, which detects uncertain test cases, generates targeted examples, and performs temporary test-time adaptation. Across agent benchmarks, TT-SI improves accuracy by 5.48% on average and can outperform standard methods with 68× fewer samples.

  • Problem

    Large post-training datasets are costly, may contain redundant information, and do not guarantee strong generalization to complex or novel agent tasks.

  • Method

    TT-SI identifies uncertain test samples, synthesizes similar examples, and uses them for temporary test-time fine-tuning; TT-D uses a stronger model to generate distilled examples.

  • Results

    +5.48% average absolute accuracy gain is reported across agent benchmarks, while TT-SI surpasses standard learning methods using 68× fewer samples.

  • Takeaways & Limitations

    The results support test-time self-improvement as a promising paradigm for adapting agentic language models during inference.

  • Takeaways & Limitations

    TT-SI depends on a threshold whose best value is sensitive, and it cannot recover knowledge absent from the pretrained model without external information.

Abstract

from arXiv · show

One paradigm of language model (LM) fine-tuning relies on creating large training datasets, under the assumption that high quantity and diversity will enable models to generalize to novel tasks after post-training. In practice, gathering large sets of data is inefficient, and training on them is prohibitively expensive; worse, there is no guarantee that the resulting model will handle complex scenarios or generalize better. Moreover, existing techniques rarely assess whether a training sample provides novel information or is redundant with the knowledge already acquired by the model, resulting in unnecessary costs. In this work, we explore a new test-time self-improvement method to create more effective and generalizable agentic LMs on-the-fly. The proposed algorithm can be summarized in three steps: (i) first it identifies the samples that model struggles with (self-awareness), (ii) then generates similar examples from detected uncertain samples (self-data augmentation), and (iii) uses these newly generated samples at test-time fine-tuning (self-improvement). We study two variants of this approach: Test-Time Self-Improvement (TT-SI), where the same model generates additional training examples from its own uncertain cases and then learns from them, and contrast this approach with Test-Time Distillation (TT-D), where a stronger model generates similar examples for uncertain cases, enabling student to adapt using distilled supervision. Empirical evaluations across different agent benchmarks demonstrate that TT-SI improves the performance with +5.48% absolute accuracy gain on average across all benchmarks and surpasses other standard learning methods, yet using 68x less training samples. Our findings highlight the promise of TT-SI, demonstrating the potential of self-improvement algorithms at test-time as a new paradigm for building more capable agents toward self-evolution.

1 INTRODUCTION

Standard LM agent fine-tuning depends on costly, large-scale datasets and may waste effort on redundant samples. TT-SI instead identifies uncertain test cases, synthesizes targeted examples, and adapts agents during inference, improving benchmark performance with far fewer samples.

  • 1 INTRODUCTION: Standard agent fine-tuning requires large human-curated or LLM-synthesized datasets whose construction is costly and can remain inefficient.The conventional paradigm assumes that dataset quality, diversity, and scale suffice to support broad generalization.
  • 1 INTRODUCTION: TT-SI identifies uncertain test cases, generates similar training examples, and performs temporary test-time updates to adapt agents on-the-fly.The three stages are self-awareness, self-data augmentation, and self-improvement.
  • 1 INTRODUCTION: +5.48% average absolute accuracy gain is achieved across four agent benchmarks for direct inference, with TT-SI using only one synthesized training instance per test case.The evaluated benchmarks are ToolAlpaca, NexusRaven, SealTool, and API-Bank.
  • 1 INTRODUCTION: TT-SI surpasses standard supervised fine-tuning on SealTool while using 68× fewer samples, and its ICL variant provides a training-free alternative when training is infeasible.TT-D further extends gains in complex, context-heavy scenarios such as multi-turn conversations.
  • 1 INTRODUCTION: The framework integrates self-awareness, targeted self-generated data, and iterative self-training as a route toward continual agent adaptation.The proposed direction is inspired by human-like lifelong adaptation.

2 PRELIMINARIES

Inductive fine-tuning assumes large, diverse training data will generalize across distributions, but distribution shift, computation cost, and redundant samples limit this approach. Test-time training addresses these issues by selecting informative test inputs, synthesizing local training signals, and applying temporary updates.

  • 2.1 FUNDAMENTAL ISSUES IN INDUCTIVE FINE-TUNING: Inductive fine-tuning can be undermined by distribution shift, large-data computation costs, and treating redundant samples as equally informative.The training risk may misrepresent test risk when Ptest differs from Ptrain.
  • 2.2 TEST-TIME TRAINING: Test-time training performs small, ephemeral parameter updates during inference, partially collapsing the train–test boundary.Prior approaches include self-supervised adaptation, retrieved-neighbor fine-tuning, and rule-based transformations of in-context examples.
  • 2.3 SELF-IMPROVEMENT IN LLMS AND HOW IT WORKS: Self-improvement shifts outputs toward more certain, high-reward responses without creating knowledge ex nihilo, instead eliciting latent knowledge already represented in the model.The internally induced reward is activated during adaptation rather than explicitly optimized.
  • 2.3 SELF-IMPROVEMENT IN LLMS AND HOW IT WORKS: TT-SI detects uncertain inputs, synthesizes targeted examples, and fine-tunes temporary sample-specific parameters to amplify latent knowledge.Its pipeline uses an Uncertainty Estimator H, Data Synthesis Function G, and Test-Time Fine-tuning T.
  • 2.4 TEST-TIME SELF-IMPROVEMENT FRAMEWORK: For each test input, the framework computes uncertainty, generates K synthetic samples when uncertainty meets the threshold, updates temporary parameters, and restores the original model afterward.The restoration prevents permanent alteration of the base model between test instances.

3 METHOD

The method adapts agents at test time by selecting uncertain inputs, synthesizing targeted examples, and temporarily fine-tuning on those examples. Its three components focus computation on challenging cases and perform localized, iterative adaptation during inference.

  • 3 METHOD: The framework integrates self-awareness, self-augmentation, and self-learning to adapt agents during inference.An uncertainty estimator selects difficult inputs, a synthesis function creates related examples, and test-time fine-tuning updates the model temporarily.
  • 3.1 SELF-AWARE SAMPLE SELECTION AT TEST TIME: Uncertainty estimation identifies challenging inputs without ground-truth labels and filters out high-confidence instances for targeted adaptation.The estimator scores candidate actions, applies a confidence threshold, and concentrates resources on ambiguous samples.
  • 3.1 SELF-AWARE SAMPLE SELECTION AT TEST TIME: The estimator computes negative log-likelihoods, converts them into normalized RSS confidence scores, and uses the gap between the top two scores to measure uncertainty.Samples with uncertainty below a user-defined threshold are selected for subsequent processing.
  • 3.2.1 DATA SYNTHESIS METHOD: For each uncertain input, the synthesis function generates K semantically similar, slightly varied labeled examples that form a temporary query-specific dataset.The uncertain input serves as a seed without its ground-truth label, while the generated examples target the model’s uncertainty region.
  • 3.3 TEST-TIME FINE-TUNING: Test-time fine-tuning locally adapts the model on the synthesized dataset before processing subsequent inputs, repeating detection, synthesis, and adaptation for each uncertain sample.The updates are temporary: after prediction, parameters are restored to their original values for the next iteration.

4 RESULTS

Across four agent benchmarks, TT-SI consistently improves accuracy through uncertainty-guided test-time adaptation, while ablations show that selective targeting and sparse updates are important for efficiency. The method remains effective across inference settings, data scales, model sizes, and uncertainty thresholds.

  • 4.1 MAIN RESULTS: +5.48% average absolute accuracy gain lifts TT-SI over baseline prompting across four agent benchmarks under direct inference.TT-SI also gains +3.85% with majority voting and +3.46% with pass@5.
  • 4.1 MAIN RESULTS: 72.43% accuracy surpasses standard SFT’s 70.20% on SealTool using roughly 68× fewer samples.TT-SI uses 190 uncertain cases, each paired with one synthetic example, instead of the full ∼13k-sample training split.
  • 4.1 MAIN RESULTS: 68.36% accuracy lets training-free TT-SI with ICL exceed the 67.74% standard ICL baseline and improve over the 66.37% base model.Generated demonstrations are inserted into the prompt rather than used for fine-tuning.
  • 4.2 ABLATION STUDIES AND ANALYSIS: TT-SI consistently outperforms SFT across OOD data scales, with larger gains as more uncertain examples are incorporated.The ICL variant also surpasses standard SFT using the same data amounts per scale.
  • 4.2 ABLATION STUDIES AND ANALYSIS: τ = 0.95 balances accuracy and efficiency at 72.43% with 190 updates, whereas selecting all 294 samples reaches 73.47% with substantial overhead.At τ = 0.35, accuracy falls to 68.10%, with FPR=0.09 and TPR=0.42.
  • 4.2 ABLATION STUDIES AND ANALYSIS: +5.76% and +3.02% absolute gains on Qwen2.5-1.5B and Qwen2.5-7B show improvements across model scales, with larger relative gains for the smaller model.On SealTool, the smaller model rises from 66.67 to 72.43, while the larger model rises from 80.95 to 83.97.
  • 4.2 ABLATION STUDIES AND ANALYSIS: 72.43% accuracy from uncertain-sample adaptation exceeds 70.07% from certain-sample adaptation, supporting targeted uncertainty filtering.Applying TT-SI to all samples yields only a marginal +1.04% gain while requiring adaptation to all 294 test samples.

5 DISCUSSIONS

TT-SI adapts agents during inference by filtering uncertain cases, synthesizing targeted examples, and applying lightweight updates. Results support efficient test-time learning while identifying threshold selection and base-model knowledge as important boundaries.

  • 5 DISCUSSIONS: TT-SI measures uncertainty, synthesizes targeted instances, and performs lightweight test-time fine-tuning to improve agents during inference.The pipeline uses Uncertainty Estimator H, Data Synthesis Function G, and Test-Time Fine-tuning T.
  • 5 DISCUSSIONS: TT-SI is presented as a modular algorithm integrating test-time learning with self-awareness, self-augmentation, and self-improvement rather than as one fixed uncertainty metric or generator.The framework allows stronger update rules, uncertainty quantification, and data-generation methods to replace its corresponding components.
  • 5 DISCUSSIONS: TT-SI performance depends on threshold τ selection, whose best setting is sensitive despite consistent gains across tested values.The paper identifies autonomous threshold calibration as an open challenge.
  • 5 DISCUSSIONS: TT-SI cannot recover knowledge absent from the pretrained model, so tasks requiring genuinely new information may need retrieval or search.This limitation defines the method’s existing-knowledge boundary.
  • 5 DISCUSSIONS: Future work extends beyond self-improvement toward self-evolution, including co-adapting the agent and data generator and applying the framework to mathematics or medicine.These directions target broader adaptation and domain-specific uncertainty or knowledge structures.

REPRODUCIBILITY STATEMENT

The paper distinguishes task-specific self-improvement from broader self-evolution and situates TT-SI within targeted, student-like adaptation. Its reproducibility materials emphasize repeated trials and standard-deviation reporting under limited-data conditions.

  • REPRODUCIBILITY STATEMENT: Because single-sample training can have high variance, experiments and baselines are repeated five times with different random seeds and averaged results are reported with standard deviations when appropriate.The protocol is designed to quantify variability under limited-data operation.
  • REPRODUCIBILITY STATEMENT: Self-improving agents autonomously refine performance on specific tasks through iterative self-refinement without external intervention.Their scope is task-specific and does not add external knowledge beyond model parameters.
  • REPRODUCIBILITY STATEMENT: Self-evolving agents instead support broader continuous adaptation across dynamic environments and sequential tasks, including changes to parametric and non-parametric components.This distinction separates bounded task optimization from lifelong generalization.
  • REPRODUCIBILITY STATEMENT: TT-SI mirrors student self-learning by identifying uncertainty, seeking related examples, and using them to address knowledge gaps on similar tasks.The analogy connects self-awareness, self-augmentation, and self-improvement.
  • REPRODUCIBILITY STATEMENT: The broader agent-learning setting commonly relies on task-specific human-curated or LLM-synthesized corpora followed by fine-tuning.These agents must learn tool use and structured-input handling in addition to text generation.

D UNCERTAINTY ESTIMATION RESULTS

The uncertainty experiments compare RSS with perplexity and examine how threshold selection trades off error coverage against unnecessary intervention. RSS separates correct from incorrect predictions more clearly and achieves the strongest reported balance against baselines.

  • D UNCERTAINTY ESTIMATION RESULTS: RSS produces clearer separation between correct and incorrect predictions than PPL, enabling more reliable uncertainty filtering on SealTool.The comparison uses top-1 versus top-2 prediction-score differences, with correct and incorrect cases shown separately.
  • D UNCERTAINTY ESTIMATION RESULTS: Increasing the softmax-difference threshold τ raises both TPR and FPR, trading broader uncertain-case coverage for lower reliability.The estimator flags an input when u(x_i) = p(1) − p(2) < τ.
  • D UNCERTAINTY ESTIMATION RESULTS: At an optimal threshold, H captures nearly 99% of model errors as uncertain while misclassifying 49% of correct answers.The default setting is τ = 0.95 because it provides the reported best trade-off between error coverage and unnecessary intervention.
  • D UNCERTAINTY ESTIMATION RESULTS: Our estimator reaches a 96.10% TPR, 53.46% FPR, and 42.64% J score on SealTool, with the J score more than doubling Perplexity’s 18.89%.It also achieves the best F1 score among the compared Random, Trivial, and PPL baselines.

E DATA GENERATION DETAILS

The data-generation component creates targeted variants of uncertain test instances for online adaptation. It uses the uncertain input as a label-free seed and preserves task meaning while varying surface form.

  • E DATA GENERATION DETAILS: The uncertainty-estimator comparison uses J as the balance measure, with the proposed method outperforming Random, Trivial, and Perplexity baselines.The table summarizes the estimator comparison on SealTool.
  • E DATA GENERATION DETAILS: For each uncertain sample, the data generator produces K new input-output pairs from a hand-crafted prompt and the sample itself as a label-free seed.The agent performs this synthesis during test-time self-augmentation.
  • E DATA GENERATION DETAILS: Generated variants retain the seed’s core semantic meaning and task relevance while introducing controlled surface-level variations.This supports targeted adaptation to query types where the model struggles.

E.1 EXPERIMENTAL RESULTS

TT-SI generates query-specific synthetic examples from uncertain inputs and uses them immediately for inference-time adaptation. The generated examples remain semantically aligned with their uncertain seeds, forming compact local clusters.

  • Data generation: TT-SI creates a temporary query-specific dataset by generating one labeled example from each uncertain input without using its gold label.The generator receives the uncertain sample’s instruction and query, then produces a revised input and answer for immediate adaptation.
  • Synthetic-query analysis: Generated samples form a compact cluster near the uncertain input in the two-dimensional semantic embedding space.The visualization projects samples using UMAP and shows ten self-generated queries around the uncertain seed.
  • Generation prompt: The generation prompt asks for non-duplicate variants that alter names, context, or wording while preserving the required function-calling output format.The prompt supplies the uncertain example as a seed and excludes its labeled output from the generator’s input.

F CHEATING EXPERIMENTS AND TT-SI COMPARISON

The cheating experiment compares baselines trained directly on test data with actual TT-SI and TT-D results. TT-SI approaches cheating TTT performance, suggesting similar generated samples can shift the model toward uncertain-sample distributions without exact ground-truth answers.

  • Experimental setup: The experiment treats baselines trained on the test set as an unrealistic upper bound for ICL, SFT, and TTT.Figure 9 compares the cheating baselines with actual noncheating TT-SI and TT-D results.
  • TT-SI comparison: 72.43% for TT-SI closely approaches 78.89% for cheating TTT on SealTool.Cheating TTT uses explicit test-set training, whereas TT-SI uses highly similar generated samples rather than exact ground-truth answers.
  • Interpretation: Temporary updates increase confidence on samples previously overlooked by the base model’s parameters.The reported interpretation links the improvement to adaptation toward the uncertain-sample distribution.

G IMPLEMENTATION DETAILS OF TT-SI

The implementation uses Qwen2.5-1.5B-Instruct for uncertainty estimation and data synthesis, with LoRA-based test-time fine-tuning. The framework consists of uncertainty estimation, synthetic-data generation, and temporary adaptation components.

  • Framework components: The framework comprises an Uncertainty Estimator H, Data Synthesis Function G, and Test-Time Fine-tuning T.The implementation uses Qwen2.5-1.5B-Instruct and runs on a single NVIDIA A40 GPU.
  • Uncertainty estimation: H estimates candidate-function confidence directly from token logits without temperature scaling.Candidate function names are extracted with regex operations and merged into the test-sample query.
  • Data synthesis: G generates K slight, non-identical variations of each uncertain sample, with K treated as a hyperparameter.The uncertain sample is inserted into the Figure 8 prompt as a seed, and the generated samples include corresponding labels.
  • Test-time fine-tuning: T performs test-time fine-tuning with LoRA through LLaMA-Factory.The reported settings use rank 8, α = 16, five epochs, learning rate 1.0 × 10^-4, and warm-up ratio 0.03.

G.4 EVALUATION

The evaluation covers four established agent benchmarks with different tool-use scenarios and sample counts. Models are assessed on function names, arguments, and argument values or types.

  • Benchmarks: Evaluation uses NexusRaven, SealTool, API-Bank, and ToolAlpaca as established agent benchmarks.NexusRaven emphasizes realistic software-operation tasks, including cybersecurity and enterprise applications across 65 APIs.
  • Evaluation setup: The evaluation uses 318, 294, 361, and 103 test samples for the four benchmarks, respectively.The setup follows prior work with slight metric modifications intended to improve evaluation reliability.
  • Metrics: Performance is evaluated by whether models produce correct function names, arguments, and corresponding values or types.String arguments are highlighted as a challenge because superficial variations can differ from gold-standard outputs.

H ADDITIONAL RUN-TIME OVERHEAD AND RESOURCE USAGE ANALYSIS

TT-SI adds measurable per-sample latency for uncertain cases, while its reported timing analysis excludes model-merging and file-I/O overheads. Despite using far fewer samples than SFT, it achieves faster reported wall-clock training on SealTool.

  • 0.87s, 3.45s, 2.05s, and 0.89s are the reported per-sample costs for uncertainty estimation, synthesis, training updates, and inference, respectively.Together, these steps amount to approximately 7.3 seconds per uncertain sample, while non-uncertain samples require only 0.87 seconds.
  • 3.7× wall-clock speed-up is achieved by TT-SI versus SFT while training on approximately 68× fewer samples.On SealTool, SFT requires 7,966.6 seconds, whereas TT-SI takes approximately 36 minutes for 190 updates and 104 direct inferences.
  • 7.26s is the reported average latency for uncertain SealTool samples, compared with 1.76s for certain samples.At τ = 0.95, TT-SI processes 190 uncertain and 194 certain samples; merge and file-I/O overhead are excluded.
  • Most additional latency comes from model merging, file saving, and vLLM model loading rather than the core H, G, and T algorithmic steps.The authors exclude these I/O-related costs from the reported clock-time analysis and identify more efficient file handling as future work.
Loading 2510.07841v1…