PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Learn-at-Test-Time: Language Agents with Learnable Adaptation Policies

Zhanzhi Lou, Hui Chen, Yibo Li, Qian Wang, Bryan Hooi

arXiv:2604.00830v3cs.LGcs.AI

TL;DR

Language agents often rely on fixed, hand-crafted adaptation rules instead of learning how to improve from experience across episodes. Meta-TTL formulates adaptation-policy discovery as bi-level meta-learning with evolutionary search over tasks, and consistently outperforms the supplied baselines across three benchmarks, including OOD settings.

  • Problem

    Existing test-time learning methods use fixed, hand-crafted adaptation policies, although adaptation determines how agents use experience to improve future behavior.

  • Method

    Meta-TTL uses bi-level optimization: an inner TTL loop evaluates candidate adaptation policies, while an outer reflective evolution loop optimizes them across training tasks.

  • Results

    Meta-TTL consistently outperforms hand-crafted, prompt-optimization, and unoptimized baselines on Jericho, WebArena-Lite, and τ^2-bench across ID and OOD settings.

  • Takeaways & Limitations

    The results suggest that how language agents adapt from experience is itself learnable and that optimized adaptation strategies can transfer beyond the training task distribution.

  • Takeaways & Limitations

    The studied instantiation freezes model weights and mediates behavioral change through system-prompt rewriting.

Abstract

from arXiv · show

Test-Time Learning (TTL) enables language agents to iteratively refine their performance through repeated interactions with the environment at inference time. At the core of TTL is an adaptation policy that updates the actor policy based on experience from previous episodes, thereby improving future behavior. Existing methods rely on fixed, hand-crafted adaptation policies rather than optimizing them for downstream improvement. We argue that optimal adaptation policies should be learned from task environments, not hand-engineered based on human intuition. To achieve this, we introduce Meta-TTL, a framework that formulates the discovery of effective adaptation policies as a bi-level optimization problem. Within this framework, the inner loop executes the standard TTL process, measuring how effectively a candidate adaptation policy helps an agent correct errors across sequential episodes. Guided by the agent's performance, the outer loop employs evolutionary search over a diverse distribution of training tasks to continually optimize the adaptation policy. We evaluate Meta-TTL on Jericho, WebArena-Lite, and $τ^2$-bench across both in-distribution (ID) and out-of-distribution (OOD) settings. Results on all three show that Meta-TTL consistently outperforms single-agent, prompt-optimization, and unoptimized meta-agent baselines, suggesting that the optimized adaptation policy encodes transferable strategies that generalize beyond the training task distribution.

1 INTRODUCTION

Test-Time Learning depends on an adaptation policy that uses accumulated experience to update an agent across episodes, but existing approaches largely rely on fixed or hand-designed rules. META-TTL instead learns this adaptation policy across tasks through bi-level optimization and evolutionary search.

  • Motivation: Test-Time Learning lets agents accumulate experience across repeated interactions, but current LLM agents often treat episodes as independent zero-shot trials.The motivation contrasts iterative error correction with agents that do not adapt effectively without parameter updates or ground-truth supervision.
  • Motivation: The adaptation policy maps accumulated experience to changes in the actor policy across episodes, functioning as a learning algorithm rather than an within-episode behavior policy.Existing methods such as Reflexion rely primarily on the pretrained LLM’s capabilities for this adaptation.
  • Approach: META-TTL treats effective test-time adaptation as learnable and optimizes the cross-episode learning rule from task environments instead of hand-engineering it.The goal is downstream improvement at test time.
  • Approach: META-TTL formulates adaptation-policy discovery as bi-level optimization, using an inner TTL loop to evaluate candidate policies and an outer loop to evolve them across training tasks.The supplied introduction describes the inner loop’s sequential episodes and the outer loop’s task-distribution optimization.
  • Novelty: META-TTL optimizes the adaptation mechanism itself, whereas prior methods focus on within-session actor behavior using fixed verbal feedback or memory accumulation.This distinction identifies what is optimized rather than merely how the actor is prompted during a session.
  • Contributions: The framework is reported to improve average game score by ∼120% on Jericho ID and task success rate by up to ∼15% on WebArena-Lite ID, with gains on τ^2-bench OOD.The reported examples are 50.4 →110.8 on Jericho, 0.55 →0.63 on WebArena-Lite, and 0.33 →0.37 on τ^2-bench.

2 RELATED WORK

Related work studies test-time adaptation through weight updates, external state, experience accumulation, prompt search, and broader agent-configuration evolution. META-TTL differs by learning the adaptation mechanism itself from a distribution of tasks.

  • Test-Time Learning: Test-Time Learning methods either update model weights at deployment or keep parameters fixed while adapting through external state.Weight-frozen approaches include experience-guided attempts and other non-parameter adaptation mechanisms.
  • Test-Time Learning: Weight-frozen TTL accumulates experience through verbal reflection, memory storage, or interaction-based learning of new environment rules.These approaches guide future attempts without necessarily changing model parameters.
  • Adaptation Optimization: Other work searches over actor prompts or evolves the agent configuration, but the adaptation mechanism remains hand-designed and fixed.META-TTL instead learns that mechanism from a distribution of training tasks.
  • Meta-Learning: Meta-learning extracts transferable knowledge from task distributions so learners can adapt efficiently to new tasks, including through context conditioning in LLMs.The related-work discussion frames in-context learning as black-box meta-learning rather than weight-based adaptation.

3 METHODOLOGY

META-TTL frames test-time learning as a bi-level process: an inner loop adapts an actor across episodes, while an outer loop optimizes the adaptation policy across tasks. The learned meta-prompt is validated, retained in a task-wise expert pool, and selected for frozen deployment.

  • 3 METHODOLOGY: META-TTL couples an inner TTL loop, where the actor adapts across episodes, with an outer meta-training loop that improves the meta-prompt.The inner loop rewrites the actor’s system prompt; the outer loop proposes candidates from rollouts and retains task-wise experts.
  • 3.3 REFLECTIVE META-TRAINING: After training, the selected meta-prompt is frozen and deployed on held-out tasks, where it continues rewriting the actor’s system prompt between episodes.Deployment uses the same between-episode update process as training, but the meta-prompt is no longer modified.
  • 3.1 TEST-TIME LEARNING FORMULATION: Each task is modeled as a finite-horizon POMDP, and a TTL session comprises consecutive episodes evaluated by Weighted Area Under the Learning Curve.The environment resets between episodes, so session improvement must come from agent adaptation; later episodes receive larger W-AUC weights.
  • 3.2 LEARNABLE ADAPTATION POLICIES FOR LANGUAGE AGENTS: The adaptation policy operates above the actor policy, mapping trajectory history after each episode to an updated actor policy for the next attempt.Existing methods typically hand-design this mapping, whereas META-TTL seeks to learn it from a distribution of training tasks.
  • 3.2 LEARNABLE ADAPTATION POLICIES FOR LANGUAGE AGENTS: META-TTL instantiates adaptation with a frozen actor whose system prompt is rewritten by a separate meta-agent governed by a learnable meta-prompt.The meta-prompt determines what experience is attended to, how failures are diagnosed, and what guidance is produced.
  • 3.3 REFLECTIVE META-TRAINING: Reflective meta-training proposes candidate meta-prompts, locally validates improvements, globally evaluates survivors, and updates a per-task expert pool.The final deployed prompt is selected as the expert with the highest average validation score, with per-task z-score normalization when reward scales differ.

4 EXPERIMENTS

The experiments evaluate META-TTL across three sequential decision-making benchmarks, testing improvement, generalization, optimization choices, and emergent adaptation strategies. Across matched settings, META-TTL improves W-AUC over baselines and shows advantages over actor optimization, RL-based meta-training, and parameter-based adaptation.

  • Experimental Setup: The experiments ask whether meta-learned adaptation improves test-time learning, how optimization choices affect performance, and which strategies emerge.These correspond to RQ1, RQ2, and RQ3.
  • Experimental Setup: META-TTL is evaluated on Jericho, WebArena-Lite, and τ 2-bench using ID and OOD splits.The benchmarks cover interactive fiction, web navigation, and customer-service tool use.
  • Main Results (RQ1): META-TTL consistently improves W-AUC across all three benchmarks and outperforms baselines in both ID and OOD settings.On Jericho ID games, it reaches 0.41 versus 0.21 for the strongest prompt-optimization baseline.
  • Main Results (RQ1): META-TTL generalizes to OOD tasks, including all three Jericho OOD games and the held-out Telecom domain in τ 2-bench.On Telecom, META-TTL reaches 0.37 W-AUC versus 0.33 for Naive and 0.34 for the strongest prompt-optimization baseline.
  • Comparison of Optimization Choices (RQ2): META-TTL achieves higher W-AUC than actor-prompt optimization on 10 of 11 tasks.The comparison uses GEPA under the same offline rollout budget and deploys the optimized actor prompt frozen at test time.
  • Comparison of Optimization Choices (RQ2): META-TTL matches or exceeds RL-based meta-training while using 272 rather than 1,920 rollouts and producing a readable meta-prompt.Its outer loop is gradient-free, whereas GRPO updates the meta-agent’s weights.
  • Comparison of Optimization Choices (RQ2): META-TTL outperforms parameter-based methods on average at lower cost, completing a TTL session in about 20 minutes.Online SFT and Online GRPO require 1–2 hours of test-time weight updates, while SFT+MAML additionally requires about 6 hours of meta-training.
  • Analysis of Emergent Adaptation Policies (RQ3): Removing fact banks lowers ID performance but raises OOD W-AUC, suggesting task-agnostic strategies contribute to transfer.With GPT-5, OOD rises from 0.28 to 0.34; with GLM-5, it rises from 0.26 to 0.31.

5 CONCLUSION

META-TTL learns adaptation policies through reflective meta-training across tasks and consistently outperforms hand-crafted and unoptimized baselines across three benchmarks, including OOD environments.

  • META-TTL consistently outperforms hand-crafted and unoptimized baselines on Jericho, WebArena-Lite, and τ^2-bench, with gains extending to OOD environments.The framework learns adaptation policies through reflective meta-training across tasks.
  • Fact-bank ablation on Jericho isolates the contribution of learned task-agnostic meta-strategies to OOD gains.The No-Facts condition removes game-specific knowledge while retaining learned task-agnostic strategies.
  • Parameter-updating comparisons use a single Qwen3-8B actor for online baselines, whereas META-TTL uses a separate Qwen3-8B meta-agent running the learned meta-policy.The comparison is reported on Jericho using W-AUC.

A EXPERT SELECTION AND SCORE NORMALIZATION

The appendix addresses expert selection, presents representative optimized meta-prompts, and details the structured, game-aware adaptation strategies evolved by META-TTL.

  • Expert selection and normalization: WebArena-Lite selects candidates by highest raw average success rate because every task provides a directly comparable binary completion signal.The selection criterion is applied across validation tasks.
  • Expert selection and normalization: Jericho applies per-game z-score normalization because raw W-AUC averages can favor candidates overfitted to easier games.Detective is easier to improve on than Temple or Zork 1, making raw averages less generalizable.
  • Expert selection and normalization: For each Jericho game, candidates’ W-AUC scores are normalized using that game’s mean and standard deviation, then averaged for expert selection.The normalized score is zi,g = (si,g − µg)/σg.
  • Expert selection and normalization: P5 wins on raw average W-AUC at 0.371, whereas P11 wins after z-score normalization with raw average W-AUC 0.348.P11’s smaller Zork 1 advantage receives greater normalized weight because that game has lower score variance.
  • Representative optimized meta-prompts: Optimized meta-prompts are represented as game- or benchmark-specific guiding prompts produced from episode trajectories for the next episode.The examples include prompts for Jericho and WebArena-Lite that guide correction of observed failure modes.
  • Evolved adaptation strategies: The evolved Jericho policy requires diagnosis, durable facts, priorities, routes, command scripts, and parser tips, while conditional fact banks activate only for the identified game.These structures separate diagnosis, fact extraction, planning, and scripting while preventing irrelevant game knowledge from being used.

D META-TRAINING OPTIMIZATION TRAJECTORY

The Jericho meta-training trajectory shows progressive improvement through semantically targeted evolutionary proposals, including structured feedback and a fix for cross-game overfitting.

  • 26 iterations over approximately 27 hours improved the seed prompt’s aggregate validation W-AUC from 0.188 through evolutionary proposals.Sixteen of 26 proposals passed local validation, and six achieved a new best aggregate score.
  • Iteration 1 increased W-AUC from 0.188 to 0.318, a +69% improvement, by introducing structured, game-aware feedback and turn-budget awareness.The proposal also added episode-restart semantics and game-specific context.
  • The proposer corrected Detective-specific overfitting by requiring actual game identification and ignoring unrelated stored facts when the log names another game.This evolved into a conditional fact-bank rule and transformed the prompt into a game-agnostic framework.

E CASE STUDIES

The case studies compare optimized and naive meta-prompts with separate GPT-5 meta-agent and Gemini-3-Flash actor configurations on Jericho.

  • The case studies compare the optimized meta-prompt against a naive baseline using a GPT-5 meta-agent and Gemini-3-Flash actor on Jericho.Each case study examines a different aspect of how the learned adaptation policy produces better feedback.

E.1 DETECTIVE (ID GAME): ACTIONABLE VS. GENERIC FEEDBACK

In Detective, actionable feedback uses the first episode’s specific failure to prescribe corrective actions, while generic advice does not. This difference produces sharply divergent improvement across subsequent episodes.

  • Initial conditions: Both conditions begin Episode 0 at comparable scores of ∼114 before meta-agent guidance.The key divergence occurs at Episode 1, after the first feedback.
  • Generic feedback: NAIVE gives generic interactive-fiction advice that ignores Episode 0 observations, causing the actor’s score to drop to 89.Its advice includes broadly applicable exploration instructions rather than a diagnosis of the observed failure.
  • Actionable feedback: OPT diagnoses the missing pistol and incorrect syntax, then prescribes a concrete command sequence for completing the task.The sequence includes retrieving and reading the paper, obtaining the pistol, and shooting the dazed man with it.
  • Performance divergence: 319 yields a 2.7× improvement in one feedback cycle under OPT.This result follows the specific diagnosis and prescribed corrective route.
  • Performance divergence: OPT reaches 340/360 by Episode 4, whereas NAIVE fluctuates between 89–131 without an upward trend.The optimized feedback progressively tightens the route by addressing turn-budget bottlenecks and reordering scoring actions.

E.2 TEMPLE (ID GAME): DIAGNOSING NON-OBVIOUS BLOCKERS

Temple tests whether feedback can identify unconventional actions needed to progress. OPT surfaces the non-obvious blocker, while NAIVE’s generic advice leaves the actor stuck.

  • Task challenge: Temple tests whether the meta-agent can identify unconventional actions, with a maximum score of 35 points.The task therefore distinguishes targeted diagnosis from generic object-examination advice.
  • Generic feedback: NAIVE never exceeds 5/35 because it remains stuck in the study room without attempting CLIMB CHARLES to retrieve the key.Generic advice to examine every object does not surface this required action.
  • Actionable feedback: OPT identifies CLIMB CHARLES, taking the vial, and unlocking the oak door as the missed progression steps.The Episode 1 feedback explicitly assigns the gap to a concrete sequence of actions.
  • Performance improvement: 8–10 points by Episode 2 nearly doubles OPT’s score after the targeted blocker diagnosis.The improvement follows feedback identifying the iron key and subsequent progression actions.

E.3 TRANSFER TO OOD GAMES: META-STRATEGIES GENERALIZE

On the unseen Balances game, OPT transfers its feedback strategy by linking an observed blocker to a specific available tool and action sequence.

  • OOD transfer: OPT recommends LEARN REZROV followed by CAST REZROV ON BOX after connecting the locked box to the spell book’s available tool.The meta-agent had never seen Balances, yet its first feedback matched the blocker to concrete next actions.
  • OOD transfer: NAIVE lists generic spells without connecting any of them to the specific puzzle.The contrast highlights the role of blocker identification and credit assignment in targeted feedback.
Loading 2604.00830v3…