PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic-TTT: Training test-time policy for test-time training

Jiahao Lu, Mohan Kankanhalli

arXiv:2610.12002v1cs.LGcs.AIcs.CL

TL;DR

Test-time training can adapt model parameters from deployment inputs, but its benefits depend on choosing an appropriate method for each query. Agentic-TTT learns a policy to govern adaptation and skill reuse; on its benchmark, it nearly doubles utility over direct inference. The reported gains are bounded by the benchmark’s deliberate enrichment for queries with observed TTT benefits.

  • Problem

    Test-time training can improve models in pre-specified settings, but open-world deployment requires deciding whether and how to adapt because methods differ in suitability and can waste compute or underperform the frozen model.

    Source: §1 · p. 2

  • Method

    Agentic-TTT trains a test-time policy to choose between direct answering, invoking TTT, and reusing accumulated parameter-space skills while accounting for utility and compute.

    Source: §1 · p. 2, §4 · p. 9

  • Results

    Agentic-TTT increases overall utility from 0.256 to 0.510, with positive gains in all eight domains, and nearly doubles utility over direct inference.

    Source: §3.1 · p. 6, §4 · p. 9

  • Takeaways & Limitations

    The results reframe TTT as a learned decision process through which models can decide how to learn from deployment experience.

    Source: §4 · p. 9

    The benchmark deliberately enriches queries with observed TTT benefits, so the reported gains do not establish a consistent advantage across arbitrary deployment distributions.

    Source: E LIMITATIONS AND DISCUSSIONS · p. 25

Abstract

from arXiv · show

Test-time training (TTT) adapts an LLM's parameters using signals derived from test inputs, and can make striking improvements in pre-specified settings such as IMO competitions or designated open problems. By turning deployment experience into parameter updates, TTT provides a direct mechanism for model-level self-improvement. Yet TTT is not universally beneficial: each TTT algorithm works in different settings, and applying an ill-suited method could waste test-time compute or even damage model performance. Therefore, such parameter-level self-improvement requires agency: the model must decide when TTT is warranted, which algorithm to invoke, and whether an existing skill can be reused. To fill this gap, we introduce Agentic-TTT, which learns a test-time policy to govern those decisions. Agentic-TTT turns TTT procedures into callable tools, treats accumulated skills as an evolving deployment environment, and trains its policy using the observed utility gains from its decisions. On our benchmark, Agentic-TTT nearly doubles the utility over the backbone model, learns to trade off utility against compute, and generalizes to domains unseen during training. Together, these results point toward autonomous self-improvement: models that can decide how to learn from their own deployment experience.

1 INTRODUCTION

Agentic-TTT addresses the need for models to decide when and how to adapt during open-world deployment, where an ill-suited TTT method can waste compute or underperform a frozen model. It learns a policy to govern TTT and skill reuse, and reports broad gains in a curated evaluation setting.

1 INTRODUCTION

  • Earlier self-improving agents evolve scaffolds such as prompts, memory, and workflows, while TTT updates model parameters during deployment using signals from test inputs.TTT has shown striking results in pre-specified settings, including IMO competition and open scientific problems.Source: §1 · pp. 1–2
  • Open-world deployment makes fixed TTT choices risky because query distributions are heterogeneous and unpredictable, and an ill-suited procedure can waste compute or underperform the frozen model.The paper identifies agency over whether to adapt, whether to reuse an acquired capability, and which TTT method to choose as the gap it addresses.Source: §1 · p. 2
  • The policy decides whether to answer directly, reuse an existing skill, or invoke a TTT method to acquire a capability, while accumulated skills expand future options.Each TTT invocation produces an isolated LoRA skill over the frozen backbone; Figure 1 depicts building and reusing skills.Source: §1 · p. 2, Figure 1 · p. 2
  • Agentic-TTT nearly doubles Qwen2.5-7B’s overall utility over direct inference, with positive gains across evaluated domains, controllable utility-compute trade-offs, and gains on domains excluded from policy training.The evaluation uses eleven benchmark families across eight domains; improvements also persist with Qwen2.5-14B.Source: §1 · p. 2

2 AGENTIC TEST-TIME TRAINING

Agentic-TTT frames test-time adaptation as sequential decisions over queries and a persistent state of adaptation pools and reusable skills. It trains a policy first to execute valid interactions, then to optimize outcome utility against computation costs.

2.1 PROBLEM FORMULATION

  • For each query, the policy uses persistent adaptation pools and a skill library, selecting pool assignment, TTT adaptation, skill loading, or generation while comparing trajectory utility and compute cost with direct inference.Adaptations become reusable LoRA skills, and the library and pools evolve across the query stream.Source: §2.1 · p. 3

2.2 LEARNING THE AGENTIC-TTT POLICY

  • Because outcome rewards provide little learning signal before valid interactions are sampled, the policy is trained through guidance distillation followed by outcome-driven reinforcement learning.The first stage establishes reliable interaction and a useful prior; the second optimizes measured utility gains while penalizing computation cost.Source: §2.2 · pp. 3–4, §2.2.2 · p. 5

2.2.1 GUIDANCE DISTILLATION POLICY LEARNING

  • Guidance distillation collects hint-guided demonstrations only when empirical TTT evaluation confirms the proposed action’s benefit, while the policy generates its own reasoning during rollouts.After collection, hints are removed from training examples and context tokens are masked from the loss.Source: §2.2.1 · p. 4
  • The supervised objective weights action tokens more heavily than rationale tokens because longer rationales would otherwise dilute the action-learning signal.The paper sets ωz = 1 and ωa = 4 before using the resulting policy to initialize reinforcement learning.Source: §2.2.1 · p. 4

2.2.2 OUTCOME-DRIVEN REINFORCEMENT LEARNING

  • Outcome-driven reinforcement learning optimizes multi-turn outcomes so the policy learns when TTT’s utility gain justifies its cost and when direct generation is preferable.The reward also includes a small, capped penalty for rejected invalid actions, providing a learning signal for cleaner trajectories.Source: §2.2.2 · pp. 4–5
  • Training samples multiple rollouts from each query’s state snapshot, then advances the stream using annotated gold actions; deployment advances the stream autonomously.Resetting and reshuffling the training stream exposes queries to varied cross-query states.Source: §2.2.2 · p. 4
  • The policy update uses mean-centered advantages without group standard-deviation normalization and applies clipped GRPO with a KL regularizer to reasoning and action tokens.The KL term regularizes the policy toward the frozen base distribution.Source: §2.2.2 · p. 5

3 EXPERIMENTS

The experiments test whether Agentic-TTT can learn when and how to use TTT, trade utility against compute, and transfer to unseen domains. Across the reported evaluations, it improves utility, adjusts adaptation behavior with compute penalties, and generalizes across domains and backbones, while some settings still show negative gains.

3.1 MAIN RESULTS: AGENTIC-TTT LEARNS WHEN AND HOW TO APPLY TTT

  • 0.510 overall utility versus 0.256 for Direct Answer, with positive gains in all eight domains; guidance distillation reaches 0.375 before RL improves it further.Agentic-TTT also exceeds the annotation-derived ORACLE on Grid Puzzles and Logic Games.Source: Table 1 · p. 6, §3.1 · p. 6

3.2 TRAINING AGENTIC-TTT TO BALANCE UTILITY AND COMPUTE

  • As λcomp rises from 0 to 0.4, direct-answer rate increases from 11.8% to 39.8% and normalized adaptation compute falls from 1375.5 to 886.9, a 35.5% reduction.Penalties of 0.1 and 0.2 reduce compute by approximately 30% while achieving higher mean utility than λcomp = 0.Source: §3.2 · p. 7, Figure 3 · p. 7
  • Higher compute penalties selectively reduce costly TTRL use from 21.3 to 11.7 invocations while TLM use increases slightly, reallocating compute rather than uniformly suppressing TTT.Method utility gains are measured on training distributions matched to each method and can be substantially smaller or negative on unsuitable inputs.Source: §3.2 · pp. 7–8, Figure 4 · p. 7

3.3 EXTRAPOLATING TTT AGENCY TO UNSEEN DATA SOURCES

  • G1 and G2 achieve broadly comparable gains on held-out and seen domains; G1's largest improvement is on unseen Grid Puzzles, and G3 performs best on jointly unseen Passage and LongBook QA.The regimes exclude domains from guidance distillation and reinforcement learning; Passage and LongBook are withheld from all three.Source: §3.3 · p. 8, Figure 5 · p. 8

3.4 GENERALIZATION ACROSS MODEL FAMILIES AND SCALES

  • Agentic-TTT improves overall utility by 0.254 on Qwen-7B, 0.136 on Llama-8B, and 0.142 on GLM-9B, but gains depend on the backbone.Llama degrades on mathematics and code, while GLM degrades on Logic Games and Long-context QA.Source: §3.4 · pp. 8–9, Table 2 · p. 9
  • Qwen-14B gains 0.189 over Direct Answer, raising overall utility from 0.432 to 0.621, though it has negative gains on Logic Games and Medical QA.The improvement profile differs from Qwen-7B's, indicating different capability gaps across backbone sizes.Source: §3.4 · p. 9, Table 2 · p. 9

3.5.1 ACTION-TOKEN SUPERVISION DURING SFT

  • Increasing action-token weight ωa from 1 to 4 improves final utility by 0.046 and lowers the invalid-action rate from 4.08% to 1.72%.The same subsequent RL recipe is applied to both checkpoints.Source: §3.5.1 · p. 9, Table 3 · p. 9

3.5.2 INVALID ACTION PENALTY IN RL

  • Setting λinv = 0.15 improves utility by 0.043 and reduces budget exhaustion by 7.87 percentage points.Budget exhaustion is the fraction of trajectories that use their entire six-action budget; the penalty helps preserve that finite budget for task completion.Source: §3.5.2 · p. 9, Table 4 · p. 9

4 CONCLUSION

The conclusion reframes test-time training as learned agency over deployment-time adaptation. Agentic-TTT manages TTT procedures and reusable parameter-space skills across queries, nearly doubling utility over direct inference while balancing compute and transferring to unseen domains.

4 CONCLUSION

  • Agentic-TTT reframes TTT from a fixed adaptation procedure into a learned decision process for deployment-time self-improvement.The policy manages heterogeneous TTT procedures and reusable parameter-space skills across a query stream.Source: §4 · p. 9
  • The learned policy nearly doubles utility over direct inference, balances utility against compute, and transfers to domains unseen during training.Source: §4 · p. 9
  • The results suggest that neural models need agency over parameter updates, not only the ability to update parameters, to grow after deployment.Source: §4 · p. 9

APPENDIX

The appendix covers related work, benchmark construction, training and execution details, and limitations and future research directions.

APPENDIX

  • Section A reviews related work, while Section B details benchmark construction.Source: APPENDIX · p. 15
  • Sections C and D describe training implementations and configurations, then the execution harness and prompting configuration.Source: APPENDIX · p. 15
  • Section E discusses the work's limitations and outlines future research directions.Source: APPENDIX · p. 15

A.1 TEST-TIME TRAINING FOR LANGUAGE MODELS

The related work distinguishes deployment-time TTT from methods that adapt before deployment, then positions Agentic-TTT among modular adaptation and self-improving-agent approaches. Its distinctive focus is learned decisions about acquiring and reusing parameter-space skills during deployment.

A.1 TEST-TIME TRAINING FOR LANGUAGE MODELS

  • Deployment-time TTT adapts model weights from test-input signals, but prior LLM methods generally target prespecified tasks rather than heterogeneous open-world workloads.The cited work reports strong results in designated settings; the authors state that no prior LLM TTT work jointly addresses whether to adapt, which method to invoke, and whether an existing adaptation suffices.Source: §A.1 · p. 15
  • Fast-weight TTT for long-context modeling is outside the paper's scope: it designs a model's memory substrate, whereas Agentic-TTT governs capability acquisition and reuse in post-trained LLMs.Source: §A.1 · p. 15

A.2 MODULAR ADAPTERS AND SELF-EXPANDING NETWORKS

  • Modular adapters and self-expanding networks make capabilities reusable, but prior approaches generally rely on pools or creation rules established before deployment or predefined task boundaries.Agentic-TTT instead lets its policy decide whether and how to expand a persistent skill library.Source: §A.2 · p. 15

A.3 SELF-IMPROVING AGENTS

  • Self-improving agents reuse stored skills, and Agentic-TTT extends this paradigm to isolated LoRA adaptations created by heterogeneous TTT methods and managed by a learned LLM policy.Source: §A.3 · p. 16

B.1 DATASET COMPOSITION AND STREAM CONSTRUCTION

The dataset spans eight domains and eleven benchmark families, with sequential workloads and document-based tasks designed to assess adaptation acquisition and reuse. The evaluation separates document ingestion from scored questions and withholds code tests for final verification.

B.1 DATASET COMPOSITION AND STREAM CONSTRUCTION

  • The dataset contains 229 training queries, 45 development queries, and 176 test queries across eight domains and eleven benchmark families.LSAT-AR and MedMCQA include sequential 30-query workloads: 16 queries for adaptation data followed by 14 for assessing skill reuse.Source: §B.1 · p. 16
  • The dataset-composition table reports query counts by split and excludes document-presentation events from those counts.Source: Table 5 · p. 16
  • Code-task evaluation adds 12 withheld test cases per query to public tests and awards utility only when a solution passes the combined suite.Public tests number three per MBPP query and two to four per LiveCodeBench query; the withheld tests come from MBPP+ or the LiveCodeBench hidden suite.Source: §B.1 · p. 16
  • Document-based tasks separate reference ingestion from question answering, testing whether adaptations retain document knowledge for later questions without the source text in answer context.SQuAD has 70 queries across 31 passages and LongBook has 29 queries across 25 books; documents and associated questions remain together within splits.Source: §B.1 · pp. 16–17

B.2 BENCHMARK SCOPE AND IMPLICATIONS

The benchmark deliberately enriches training and test data with queries that benefit from adaptation, so its gains characterize this curated setting rather than expected gains on source distributions. The discussion also identifies limits from outcome verifiability and frontier-model evaluation scope, while describing adaptation frequency as conditional on observed utility and cost.

B.2 BENCHMARK SCOPE AND IMPLICATIONS

  • About 65% of the curated benchmark queries show observed adaptation benefits, versus 7.0% of MATH-500 queries meeting the high-gain criterion, so reported gains should not be read as expected source-distribution gains.The MATH-500 figure uses the PoT reference procedure with Qwen2.5-7B-Instruct.Source: §B.2 · p. 17
  • The prevalence of beneficial adaptation depends on the backbone, data, TTT method, and computation budget.Source: §B.2 · p. 17
  • The policy has no prescribed adaptation frequency: training-distribution outcomes shape whether it learns frequent TTT use or more conservative, positive-net-return adaptation.The training distribution's frequent rewards for adaptation and skill reuse encourage more frequent TTT use.Source: §B.2 · p. 17
  • The benchmark favors explicit reference answers or executable tests, but deployment often lacks ground truth and task checks verify outcomes only within their coverage.Without ground truth or a reliable verifier, individual-query TTT benefits cannot be directly confirmed.Source: §B.2 · p. 17
  • The evaluation targets available backbones rather than frontier models, whose TTT agency requires harder benchmarks near capability boundaries where direct inference leaves room to improve.The authors suggest challenging mathematical, scientific, and engineering workloads, potentially including independently verifiable open research problems.Source: §B.2 · pp. 17–18

C DETAILS OF POLICY TRAINING

Policy training uses a frozen Qwen2.5-7B-Instruct backbone with a trainable policy LoRA, and the paper equips the policy with five TTT methods. To manage expensive and stochastic adaptations, training caches measured utility gains; a 7B-backbone run takes approximately 11 GPU-hours without that caching.

C DETAILS OF POLICY TRAINING

  • Policy training keeps the Qwen2.5-7B-Instruct backbone frozen and optimizes a rank-16 policy LoRA through two-stage guidance distillation and reinforcement learning.Distillation runs for two epochs at learning rate 10−5 and batch size 32; reinforcement learning runs for three epochs at 5 × 10−6.Source: C DETAILS OF POLICY TRAINING · p. 18
  • Agentic-TTT offers five methods: POT uses execution feedback or answer agreement, TTT-FS distills demonstrations, TTRL uses majority-vote pseudo-labels, SEAL distills self-edits, and TLM minimizes language-modeling loss.Source: C DETAILS OF POLICY TRAINING · p. 18
  • Caching estimates each adaptation configuration’s expected utility gain from three independent runs and reuses the mean, reducing overhead and reward variability.Source: C DETAILS OF POLICY TRAINING · p. 18
  • Without TTT caching, training a run with a 7B backbone takes approximately 11 GPU-hours on a single NVIDIA H200 GPU.The setup uses vLLM to accelerate inference wherever supported.Source: C DETAILS OF POLICY TRAINING · p. 18

D.1 POLICY INTERACTION AND EXECUTION

The harness separates policy interaction from answer generation: the policy observes the query, relevant pools and skills, and within-query history, then acts iteratively on feedback. Its action protocol limits each query to six policy turns and one successful adaptation, while final answers use the frozen backbone with the selected skill or no adapter.

D.1 POLICY INTERACTION AND EXECUTION

  • The policy context includes the query, relevance-ranked adaptation pools and skills, and current-query interaction history; nonterminal actions return feedback and an updated state snapshot.The policy continues with its prior reasoning, actions, and observations.Source: §D.1 · p. 18
  • Final-answer decoding disables the policy LoRA and uses only the original query, with either the frozen backbone alone or the selected skill LoRA.Policy reasoning, action history, and environment observations are excluded from the answer-generation context.Source: §D.1 · p. 19
  • Each query allows at most six policy turns and one successful adaptation, and only GENERATE terminates the interaction.Parsing errors, unmet preconditions, and recoverable failures return observations; exhausting the turn budget forces direct frozen-backbone answering without a skill.Source: §D.1 · p. 19, Table 6 · p. 19
  • The action interface includes ASSIGN for archiving a query and supplied material in a pool, ADAPT for training or updating a pool’s skill, and LOAD for activating an existing skill.ASSIGN can create the pool; ADAPT requires an eligible pool and available adaptation allowance.Source: §D.1 · p. 19
  • A requested skill identifier must refer to an existing adapter, while GENERATE submits the final answer using the active skill when requested and available or otherwise the frozen backbone.Generating an answer requires no prior adaptation or loading.Source: §D.1 · p. 19
  • For ADAPT, the policy supplies method-specific payloads through structured actions, and the harness retrieves the specified pool inputs before executing the chosen TTT method.The interface table summarizes required materials, training signals, and hyperparameter settings.Source: §D.1 · p. 19

D.2 POOLS, SKILLS, AND RETRIEVAL

This appendix section describes how persistent pools accumulate query material and skills, retrieve relevant pools, and support reuse or incremental adaptation. It also covers policy prompts and training templates that govern these actions, including when to answer directly, how to distinguish filing queries from training adapters, and how existing skills are represented.

D.2 POOLS, SKILLS, AND RETRIEVAL

  • Pools archive queries and material under query-derived names, retain one persistent adapter updated with new and up to eight replay samples, and retrieve the top 8 pools using semantic and BM25 rankings.A growing pool’s existing adapter remains available for reuse until another adaptation is requested.Source: §D.2 · pp. 19–20

D.3 PROMPT TEMPLATES

  • The policy prompt defines available actions and TTT methods, with one action per turn, a six-turn limit, and generation as the terminal answer action.Assignment files a query for possible future adaptation; adaptation trains or updates an adapter, and generation produces the final answer.Source: §D.3 · pp. 20–21
  • Method templates tailor adaptation inputs: PoT uses execution or agreement feedback, SEAL trains on self-edits, TLM predicts pooled text, and TTRL uses majority-vote pseudo-labels for reinforcement learning.Code feedback reports test results or execution errors, while mathematics uses answer agreement for rewards.Source: §D.3 · pp. 21, 23
  • TTT-FS uses leave-one-out demonstrations: remaining examples provide context, the held-out output is the target, prompt tokens are loss-masked, and text or grid examples are reordered or augmented.Grid demonstrations additionally undergo augmentation.Source: §D.3 · p. 23
  • Guided rollout hints specify the next target action and rationale during collection, but are removed from SFT inputs and absent during evaluation.The first-turn hint is appended to the query, while later hints accompany observations.Source: §D.3 · p. 23
  • A skill is loadable when its trained adapter exists, regardless of its sample count; prompts also caution against inferring adapter changes or denying observed skills without supporting history.The state reminders say not to treat a filed query as creating or updating an adapter, or to claim an adapter is active, newly trained, or updated unless the action history supports it.Source: §D.3 · pp. 24–25
  • Direct generation is targeted when adaptation offers no or only marginal expected gain, the corresponding skill is unbuilt, or the pool is below TTRL’s 16-sample floor.The guidance weighs anticipated benefit against training computation; an under-floor query remains in the pool and is answered with the base model.Source: §D.3 · p. 25
  • Filing a query stores it in a pool but does not perform TTT; when a sequence starts with assignment, the query is filed before a direct base-model answer.The reminders distinguish total samples from new samples: first TTRL adaptation and re-adaptation each require at least 16 samples, with re-adaptation requiring 16 new samples and occurring only as an explicit action.Source: §D.3 · pp. 24–25

E LIMITATIONS AND DISCUSSIONS

The discussion bounds reported gains by benchmark distribution and available adaptation potential, identifies cross-query credit-assignment gaps in training, and notes that adapter reversibility does not eliminate safety risks.

Benchmark selection and distribution dependence

  • Outcome-based benchmark selection limits generality: gains may be smaller or disappear when beneficial adaptation is rare, modest, or unavailable in the deployment distribution.Benefits also depend on TTT-method suitability, backbone improvement headroom, and computation budget.Source: E LIMITATIONS AND DISCUSSIONS · p. 25

Cross-query credit assignment

  • RL optimizes query-level returns rather than complete deployment streams, while gold-generated initial states differ from autonomous states and delay credit for reusable skills.Heuristic-guided SFT primarily supports sample accumulation and skill construction; RL refines decisions using current-query outcomes.Source: E LIMITATIONS AND DISCUSSIONS · pp. 25–26

Safety and reversibility

  • Isolated LoRA skills can be removed without reverting the frozen backbone, but active compromised skills may still produce harmful outputs and downstream consequences.Pseudo-labels may reinforce incorrect predictions, while shared libraries may propagate errors or be poisoned.Source: E LIMITATIONS AND DISCUSSIONS · p. 26

Related papers

Loading 2610.12002v1…