PaperlayerView exact revision on arXiv

Source-linked AI summary

ASCENT: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience

Haodong Lu, Dong Gong

arXiv:2610.05303v1cs.LG

TL;DR

Long-horizon agents encounter related tasks in deployment, but existing approaches do not carry weight updates across a one-pass task stream from one attempt per task. ASCENT self-distills verified trajectories into persistent weights, and improves success and interaction efficiency across tasks as experience accumulates.

  • Problem

    Prior methods do not adapt long-horizon agent weights across a one-pass task stream using only one attempt per task.

    Source: §1 · p. 2

  • Method

    ASCENT uses a frozen initial model conditioned on verified trajectories to distill next-token distributions into persistent LoRA fast weights, optionally removing invalid-action turns.

    Source: §1 · p. 2

  • Results

    ASCENT improves task success and interaction efficiency as experience accumulates, outperforms compared online adaptation baselines, and transfers to held-out scenes across ALFWorld, WebShop, and AppWorld at varied model scales.

    Source: §1 · p. 2, §5 · p. 12

  • Takeaways & Limitations

    Verified experience from a single attempt can be consolidated into agent weights that persist across tasks, without a separate training phase or memory retrieval.

    Source: §1 · p. 2, §5 · p. 12

    ASCENT requires full next-token distributions from its frozen copy and therefore applies only to open-weight models.

    Source: §5 · p. 12

Abstract

from arXiv · show

A large language model (LLM) agent solves long-horizon tasks through many reasoning-action turns, with one verification signal at termination. Deployed agents face streams of related tasks, making their trajectories a natural resource for improvement. In-context adaptation agents store reflections, memories, or skills as text, so reuse depends on retrieving the right experience and on a frozen policy executing it. We study Online Agentic Test-Time Training (OaTTT), which trains the LLM's weights on its own execution trajectories during deployment. The agent executes each task once, in one pass over the stream, and the executed trajectory with its verification result is the only learning signal for weight updates that persist across tasks. Directly imitating or reinforcing the generated tokens of this single attempt destabilizes the policy. We introduce ASCENT (Agentic Self-distillation for Cross-task EvolutioN at Test-time), which instead self-distills verified experience. A stable version of the LLM, its frozen initial copy, receives the verified trajectory as privileged information and predicts next-token distributions along it with this hindsight. Distilling them into persistent LoRA fast weights updates the agent for later tasks, without an external reference solution or stronger teacher. By further removing invalid-action turns, ASCENT distills enhanced privileged experience for more efficient execution. We characterize its population target and the limits of sparse outcome selection. Across ALFWorld, WebShop, and AppWorld at varied model scales, ASCENT improves task success and interaction efficiency as experience accumulates, outperforms online adaptation methods, and transfers to held-out scenes, showing that an agent can consolidate verified experience into its weights without a separate training phase or memory retrieval. Project page: https://artificer-ai-lab.github.io/ASCENT

1 Introduction

Long-horizon agents encounter streams of related tasks, but prior approaches do not adapt their weights across a one-pass stream using one attempt per task. ASCENT self-distills verified trajectories into persistent updates, and improves success and interaction efficiency as experience accumulates.

1 Introduction

  • In-context adaptation stores reflections, memories, or skills as text, leaving reuse dependent on retrieving relevant experience and the frozen policy executing it reliably.These approaches keep model parameters frozen.Source: §1 · p. 1
  • Prior methods do not adapt long-horizon agent weights across a one-pass task stream using one attempt per task.Existing approaches instead use within-episode adaptation, pre-deployment training, multiple attempts, or reset parameters.Source: §1 · p. 2
  • OaTTT uses each task’s single executed trajectory and verification result as the only learning data for persistent weight updates.The task stream is processed in one pass.Source: §1 · p. 2
  • ASCENT distills a frozen initial model’s full next-token distributions, conditioned on verified trajectories, into persistent LoRA fast weights.It can remove invalid-action turns from the teacher’s privileged information.Source: §1 · p. 2
  • ASCENT improves task success and interaction efficiency across ALFWorld, WebShop, and AppWorld at varied model scales, while transferring to held-out scenes.The paper also reports outperforming compared online adaptation baselines.Source: §1 · p. 2

2 Related Work

Prior work adapts agents through retrieved text memories, test-time training, reinforcement learning, and self-distillation. ASCENT differs by carrying verifier-selected updates across tasks from one attempt each, without replay.

2 Related Work

  • In-context methods store reflections, memories, or reusable skills for retrieval, while parametric methods internalize experience in weights [58] [28].Text-based reuse depends on retrieval and the frozen policy executing the retrieved experience over long interactions [58].Source: §2 · pp. 2–3
  • Existing online learning, test-time training, and RL approaches use episodic resets, pre-deployment training, multiple attempts, or learned critics [45] [40] [53] [12] [30] [20] [15].ASCENT persists updates across tasks from one attempt each, selected by environment verification.Source: §2 · p. 3
  • On-policy self-distillation uses student-sampled sequences; ASCENT keeps the initial model fixed and distills at test time after verifier-accepted interactions, without replay [9].Other cited methods condition a teacher on reference solutions, demonstrations, environment feedback, or successful attempts.Source: §2 · p. 3

3 Method

OaTTT updates an agent from each single-attempt trajectory and its sparse verification, with updates carried across tasks. ASCENT replaces direct token imitation or reinforcement with frozen-teacher distillation of verified experience, optionally filtered to remove invalid-action turns.

3.1 Online Agentic Test-Time Training (OaTTT)

  • OaTTT processes a one-pass stream, gives the agent one attempt per task, and uses the trajectory plus sparse episode-level verification for updates that persist across tasks.Outcomes are scored before their trajectory influences the policy; no retries, additional attempts, or replay buffer are used.Source: §3.1 · pp. 3–4

3.2 OaTTT via Direct Imitation or Reinforcement of Experience

  • Across the ALFWorld stream, all five direct-imitation or reinforcement baselines increasingly produce invalid actions and fall below the base model’s success in the stream’s second half.Imitation baselines drift toward near-deterministic first response tokens; policy-gradient methods also drift, with REINFORCE collapsing onto tokens unlikely under the base.Source: Figure 1 · p. 4, §3.2 · p. 5

3.3 ASCENT: OaTTT via Self-Distillation of Verified Experience

  • ASCENT distills hindsight-informed full-vocabulary teacher distributions into LoRA fast weights only after verified success, using a frozen initial model and skipping failed episodes.The teacher receives privileged information from the filtered trajectory, while the student’s updates use its attempt prefixes.Source: §3.3 · p. 5, Figure 2 · p. 5

3.3.1 On-Policy Self-Distillation Objective

  • ASCENT minimizes forward KL from detached frozen-teacher distributions at the same student prefixes used by direct imitation, yielding stable targets across updates.The teacher sees how the verified episode continued to success, supplying full-horizon turn-level guidance without changing its targets as the student updates.Source: §3.3.1 · p. 6

3.3.2 Distilling Verified and Filtered Experience Trajectories

  • The teacher’s hindsight contains later trajectory turns and verified outcome, but does not identify which actions caused success; invalid-action turns can be removed from its privileged input.The filter checks whether an action was executed, not whether it helped; all nonempty responses still provide student prefixes. ASCENT reports the fewest mean turns in Tables 1, 2, 4, and 5, though accepted trajectories may retain unnecessary turns and one attempt cannot reveal alternative-action outcomes.Source: §3.3.2 · pp. 6–7

4 Experiments

Experiments evaluate ASCENT across ALFWorld and WebShop, then analyze transfer, efficiency, co-evolution, and distillation choices. ASCENT improves success over the base and strongest online comparator, with gains extending to held-out scenes and later tasks.

4.1 Experimental Setting

  • ALFWorld evaluation uses a 140-task seen stream and 134 held-out tasks; online methods make one scored attempt per task in a single pass.WebShop is also evaluated; offline baselines use the same fixed-size training split before evaluation.Source: §4.1 · p. 7, Table 2 · p. 9

4.2 Main Experimental Results

  • ASCENT raises exact success by more than 22 points over the base across ALFWorld and WebShop at both scales, exceeds the strongest online comparator, and leads WebShop mean score.It also outperforms every offline in-context adaptation method on ALFWorld.Source: Table 1 · p. 8, §4.2 · p. 9, Table 2 · p. 9
  • ASCENT improves the difficult Heat and Cool task families and transfers to unseen scenes; continued OaTTT substantially exceeds frozen transfer at 4B.At 9B, continued training can preserve the already strong transfer result, and later-stream gains are larger for 4B while the base remains flat.Source: §4.2 · pp. 9–10, Figure 3 · p. 9

4.3 Analysis and Discussion

  • On “put a cool egg in microwave,” ASCENT completes the task in 20 turns, while the base and five in-context variants exhaust the turn budget.The in-context variants retrieve irrelevant or contradictory experience, or fail to follow relevant guidance; ASCENT uses earlier same-family experience without retrieving memory.Source: Figure 4 · p. 10, §4.3 · p. 11
  • ASCENT uses fewer turns than the base on 64% of shared successes at 4B and 67% at 9B, and its turn savings grow later in the stream.Shared successes are tasks solved by both the un-evolved base and ASCENT.Source: Figure 5 · p. 10, §4.3 · p. 11
  • Combining ASCENT with ACE or MemP improves both in-context methods at both scales, but retrieved memory helps over ASCENT alone only at 9B.At 4B ASCENT alone performs best; the authors suggest extra context may distract the weaker model.Source: Figure 5 · p. 10, §4.3 · p. 11

4.4 Ablation Studies

  • Reasoning in privileged information improves success and turn efficiency over action-only targets; shorter filtered information benefits 4B, while full trajectories perform well at 9B.With reasoning included, filtering helps most at 4B; on 9B the filter raises valid-action rate while changing success only slightly.Source: §4.4 · p. 11, Table 3 · p. 11
  • ASCENT improves over the base with reverse KL or Jensen–Shannon divergence at both scales, but reverse KL trails forward KL and JS exceeds it at 9B.JS matches reverse KL at 4B; forward KL is the default and favors coverage of the teacher distribution.Source: Figure 5 · p. 10, §4.4 · pp. 11–12

5 Conclusion

ASCENT distills verified experience into persistent LoRA fast weights, outperforming the base and compared baselines on ALFWorld and WebShop as experience accumulates. Its scope is constrained by full-distribution access and sparse episode-level verification.

5 Conclusion

  • On ALFWorld and WebShop at two model scales, ASCENT outperforms the base and compared baselines, with growing later-stream gains and transfer to held-out scenes.Directly imitating or reinforcing generated tokens destabilizes the policy.Source: §5 · p. 12
  • ASCENT distills verified experience through a frozen copy of the initial model into persistent LoRA fast weights.The trajectory is supplied as privileged information to the teacher.Source: §5 · p. 12
  • Full next-token distributions restrict ASCENT to open-weight models, while sparse episode-level verification weights turns equally and provides no turn-level credit.The teacher’s privileged context contains the student’s response text, which may partially reconstruct generated tokens; isolating that effect remains open.Source: §5 · p. 12

A Additional Method Details

The appendix separates which trajectories verification selects from what forward-KL distillation fits. It characterizes both limits: acceptance does not assign turn-level credit, and verification alone does not identify shorter paths.

A.1 What Verifier Selection and Distillation Identify

  • The population analysis separates verifier selection from the distillation objective and concerns possible trajectories under a fixed collection policy, unlike ASCENT’s single attempt per task.The verification outcome gates updates; reported benchmark success may use a stricter criterion.Source: §A.1 · p. 16

A.1.1 Verifier Selection Reweights Completed Trajectories

  • Verifier selection reweights responses by their relative chance of acceptance but assigns no causal credit to individual turns or outcomes to untried responses.With only one collected response at a history, selection cannot rank alternatives there.Source: §A.1.1 · p. 16

A.1.2 Forward KL Fits Teacher Predictions at Student Prefixes

  • Forward-KL distillation’s unrestricted population target is the mean hindsight-conditioned teacher distribution at student prefixes covered by the training measure.The student determines which prefixes appear and their weights; this identity alone does not establish improved predictions, finite-adapter convergence, or transfer.Source: §A.1.2 · pp. 16–17

A.1.3 Verification Does Not Identify Shorter Paths

  • Verification need not select shorter successful trajectories, because accepted episodes can include valid but unnecessary turns and length shifts depend on covariance with acceptance.With a turn cap, lower all-episode mean turns may reflect more successes even when successful paths are no shorter.Source: §A.1.3 · p. 17

B.1 Evaluation Protocol and Comparison Controls

The evaluation protocol matches task order and key comparison conditions while allowing methods to learn differently. Baselines include offline and online in-context adaptation, online parametric adaptation, and single-attempt imitation or policy-gradient approaches.

B.1 Evaluation Protocol and Comparison Controls

  • Paired methods share task order, verification signals, and the ReAct template for in-context methods, while offline methods use the same fixed-size training split.On WebShop, exact success requires a score of 1, while ASCENT’s adaptation gate accepts scores of at least 0.9.Source: §B.1 · p. 17

B.2 Baseline Details

  • Baselines include a frozen base, offline memory or skill builders, and online methods that update memory or skills after each scored episode.The online context can first affect the next task.Source: §B.2 · p. 17
  • TT-OPSD uses turn-local privileged information without the completed trajectory and keeps invalid-action turns, while online REINFORCE variants update after each episode.Figure 6 shows the teacher prompt prepending privileged information to each shared student prefix; the empty think block reflects disabled thinking mode.Source: §B.2 · pp. 17–18, Figure 6 · p. 18

B.3 Implementation Details

The implementation specifies turn budgets, environment-specific action-validity rules, teacher inputs, and persistent adapter optimization. The held-out and continued-training tables report success, turns, and runtime.

B.3 Implementation Details

  • ALFWorld and WebShop attempts have a 50-turn cap, and unfinished attempts fail; each environment defines action validity using its own execution rules.AppWorld marks turns invalid when code is unparseable or execution raises an error.Source: §B.3 · p. 18
  • The student reads its shared prefix, while the teacher reads the privileged trajectory information prepended to that prefix.Trajectory text conditions the teacher but is not itself a prediction target; the teacher supplies detached full-vocabulary logits from a frozen initial model.Source: §B.3 · pp. 18–19
  • After each accepted episode, ASCENT takes two forward-KL AdamW steps with learning rate 1.5 × 10^-4; adapter and optimizer state persist across tasks.Table 4 reports ALFWorld held-out transfer; Table 5 reports continued OaTTT with pre-update success, mean turns, runtime, and changes from frozen transfer.Source: §B.3 · p. 19, Table 4 · p. 19, Table 5 · p. 20

B.4 Full Benchmark Results

This section directs readers to the full ALFWorld unseen results for frozen transfer and continued OaTTT, and to the full ALFWorld seen and WebShop results, including offline methods.

B.4 Full Benchmark Results

  • Tables 4–5 report full ALFWorld unseen results for frozen transfer and continued OaTTT.Figure 3 summarizes these results in the main paper.Source: §B.4 · p. 20
  • Tables 1 and 2 in the main paper contain the full ALFWorld seen and WebShop results, including offline methods.Source: §B.4 · p. 20
  • The supplied passage identifies the result tables but does not state their numerical outcomes.Source: §B.4 · p. 20

B.5 Additional Evaluation on AppWorld

The AppWorld evaluation tests Qwen3.5-4B on Test-Normal and Test-Challenge using task- and scenario-level completion metrics. ASCENT improves task-level completion over the base on both splits, with its comparative lead over online methods varying by split and metric.

B.5 Additional Evaluation on AppWorld

  • The evaluation uses Qwen3.5-4B on 168 Test-Normal tasks in 56 scenarios and 417 Test-Challenge tasks in 139 scenarios, with one scored attempt and a 30-interaction budget per task.TGC requires passing every state-based task test; SGC requires every task variant in a scenario to succeed.Source: §B.5 · p. 20
  • On Test-Challenge, ASCENT’s task-level gain does not translate into a scenario-level lead.Source: §B.5 · p. 20, Table 6 · p. 21
  • ASCENT improves TGC over the base on both AppWorld splits and leads every listed online comparator on Test-Normal TGC and SGC.Table 6 reports TGC and SGC for Test-Normal and Test-Challenge.Source: §B.5 · p. 20, Table 6 · p. 21

C.1 Teacher Evolution in OaTTT

Teacher evolution changes both how verified trajectories are interpreted and the supervision supplied to later tasks, whereas ASCENT keeps its teacher fixed. The analysis finds that faster evolution weakens results relative to the frozen teacher, while paired shared-success comparisons show an interaction-efficiency gain beyond success/failure composition.

C.1 Teacher Evolution in OaTTT

  • The fastest-evolving teachers lose success along the stream, and no evolving teacher matches the frozen teacher over the whole stream.The slowest EMA (β = 0.999) ends level with the frozen teacher; the figure tracks trailing success and teacher–student adapter distance.Source: §C.1 · p. 21, Figure 7 · p. 22, Table 7 · p. 22
  • An evolving teacher returns student drift as later supervision, and neither episode verification nor invalid-action filtering removes this second feedback path.Verification selects episodes but assigns no credit to individual turns; filtering removes invalid turns but cannot judge the remaining valid turns.Source: §C.1 · pp. 21–22
  • The two teachers closest to the student succeed least; once mean adapter distance exceeds about 1.5, success levels off at 56%–64%, below the frozen teacher’s 65%.Frequent snapshot refreshes hand the teacher the student’s latest weights, and success declines through the stream’s second half.Source: §C.1 · p. 21, Figure 7 · p. 22
  • A self-consistency condition does not establish convergence or task success, because the trained policy determines both selected privileged information and its interpretation.Forward KL measures teacher agreement rather than task success and covers only alternatives retained by the teacher.Source: §C.1 · p. 22
  • In ungated imitation, student entropy on fixed prompts approaches zero as drift grows, while teacher entropy was not logged, leaving transfer of entropy collapse unresolved.The collapsed direct-imitation endpoint also has a next-token gradient about 25× larger than the frozen-teacher checkpoint, but the reported associations do not establish its cause.Source: §C.1 · p. 23
  • A KL anchor limits drift but supplies no episode-conditioned targets, whereas ASCENT’s frozen privileged teacher supplies episode-conditioned soft targets.Online RFT+KL’s success stays well below ASCENT with the frozen privileged teacher.Source: §C.1 · p. 23

C.2 Turn Efficiency under Success Composition

  • Successful-episode turn means compare different solved-task sets, so the analysis separately compares base and ASCENT only on tasks both solve.At 4B, successful episodes average 17.9 turns for the base and 15.8 for ASCENT; at 9B, they average 18.4 and 12.6.Source: §C.2 · p. 23
  • On shared successes, ASCENT has negative mean and median turn differences at both scales and uses fewer turns on most tasks than the base.The paired comparison isolates an interaction-efficiency gain beyond success/failure composition, but not the turns avoided or the cause of each shorter path.Source: §C.2 · p. 23, Table 8 · p. 23

C.3 Retrieval and Execution for In-Context Adaptation

The retrieval-based in-context adaptation analysis finds that top-five retrieval often misses available same-type successes, and retrieval is associated with higher success but does not ensure it. A case study separates scored outcomes from later trace diagnostics and documents the experience available before the focal task.

C.3 Retrieval and Execution for In-Context Adaptation

  • The baseline is evaluated on three 140-task ALFWorld seen streams per scale, and logged and reconstructed retrieval counts agree across all 840 episodes.Source: §C.3 · pp. 23–24
  • Figure 8 distinguishes availability of prior same-type success from whether one appears among the top-k retrieved memories, then splits success by availability and retrieval.Source: §C.3 · pp. 23–24, Figure 8 · p. 24
  • Top-five retrieval often misses an available prior same-type success, and retrieval is associated with higher success, but retrieved episodes can still fail.Same task type is only a proxy for relevance, so a retrieved item may not contain the needed instruction.Source: Figure 8 · p. 24, §C.3 · p. 24
  • Figure 4’s case study is ALFWorld seen-stream episode game idx=135, a task to put a cool egg in microwave; each method uses Qwen3.5-4B and gets one scored attempt with a shared 50-turn budget.Scored outcomes are kept separate from later trace diagnostics shown in the figure.Source: §C.3 · p. 24
  • Before episode 135, the retrieval baseline’s pre-task bank contains 405 memories and returns a contradictory top item, while online MemP has not yet acquired its cool-then-place workflow.ASCENT’s fast weights may reflect verified updates from episodes 98 and 100 before episode 135 is scored.Source: §C.3 · p. 24
Loading 2610.05303v1…