PaperlayerView exact revision on arXiv

Source-linked AI summary

When History Fails to Become Experience: Action Calibration in Language Agents

Jingyu Liu, Zhiwen Wang, Yuxin Jing, Huanyu Zhou, Yong Liu

arXiv:2610.02769v1cs.CLcs.AI

TL;DR

Agents have access to actions and environmental feedback within a task, but it remains unclear how reliably they use that history to revise earlier approaches. The paper tests history interventions, explicit outcome labels, and a learned calibrator that selects experience to record. Outcome labeling improves task success and reduces repetition, and learned calibration improves held-out task completion beyond ordinary history and outcome labeling.

  • Problem

    Interaction history contains task-specific evidence for revising earlier attempts, but agents may not reliably connect past actions with their outcomes when deciding what to do next.

    Source: §1 · p. 1, §5.1 · p. 5

  • Method

    The paper compares history interventions, labels each observation as the outcome of its preceding action, and trains a calibrator to decide when to update and what experience to record.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    Outcome labels improve task success and reduce repetition, while the learned calibrator improves held-out task completion over ordinary history and outcome labeling.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Effective history use depends on using past outcomes to guide later decisions rather than merely retaining interaction history.

    Source: §7 · p. 9

Abstract

from arXiv · show

Language agents should draw on prior attempts and environmental feedback to improve subsequent decisions within the same task. However, providing additional interaction history can sometimes reduce task success, suggesting that agents do not consistently use this information effectively. To investigate this limitation, we examine how agents use history. We find that history improves task completion overall, yet much of this benefit persists even when past actions are shuffled. Disrupting the correspondence between actions and observations causes only a modest decline in task success. We therefore hypothesize that agents do not reliably connect past actions with their outcomes when deciding how to proceed. To test this hypothesis, we explicitly label each returned observation as the outcome of the preceding action. This simple annotation improves task success and reduces next-action repetition without introducing new environmental information. Building on this insight, we introduce a learned calibrator that explicitly reassesses past actions and selectively records experience to guide subsequent decisions, improving task success beyond outcome labeling alone.

1 INTRODUCTION

Interaction history contains task-specific evidence for revising earlier attempts, but more history does not always improve completion. The paper tests whether agents connect actions to outcomes, finding gains from explicit outcome labels and a learned calibrator that selects useful experience.

1 INTRODUCTION

  • Interaction history can contain both unsuccessful action strings that reinforce repetition and feedback indicating those actions failed.The paper frames this tension as a question of how agents use evidence during an ongoing task.Source: §1 · p. 1
  • Figure 1 contrasts ordinary observation history, observations labeled as action outcomes, and a calibrator that selectively retains experience.The figure presents the paper’s progression from history to explicit outcome annotation and selective experience retention.Source: Figure 1 · p. 2, §1 · p. 2
  • Labeling each observation as the preceding action’s outcome improves task success and reduces action repetition without adding environmental information.The annotation draws attention to what a past action achieved, rather than adding a judgment about whether it was appropriate.Source: §1 · p. 2
  • Self-generated experiences retained throughout a task lower success relative to ordinary history and can restate feedback or introduce unsupported judgments.These results motivate learning which experiences to retain.Source: §1 · p. 2
  • The learned calibrator improves held-out task completion over both ordinary history and outcome labeling by learning when to update and what experience to record.Its supervision comes from the effects of experiences on next-action quality across multiple actors.Source: §1 · p. 2
  • Additional reference history can reduce recovery, while much of history’s benefit persists after historical action strings are shuffled.The findings motivate the hypothesis that ordinary history does not reliably elicit action calibration.Source: §1 · pp. 1–2

2 RELATED WORK

Prior work studies both reusable guidance distilled from trajectories and negative effects of conversational or contextual history. This paper focuses on whether agents use feedback already present in an ongoing task to revise earlier approaches.

Summarizing and using experience

  • Prior methods construct reusable guidance from past trajectories, while this work studies feedback already present in an ongoing task’s history.Examples include verbal reflections across trials, procedural skills from trajectory memories (Ding et al., 2026), learned memory operations (Zhang et al., 2026a), and feedback-based experience utility (Zhang et al., 2026b).Source: §2 · pp. 2–3

Negative effects of interaction history

  • Studies of conversational inertia, repeated failed tool calls, and belief inertia do not examine how history’s negative effects interact with its added information or why feedback is underused.The cited work includes Wan et al. (2026), Chen et al. (2026), Gumaan (2026), and Wang et al. (2026).Source: §2 · p. 3

3 MORE HISTORY DOES NOT ENSURE BETTER RECOVERY

The section tests whether extra information from a failed trajectory helps recovery by restoring an early state and supplying the remaining history as a reference. Across the reported aggregates, reference history lowers recovery, and the prompt-length analysis argues that context length alone does not explain the decline.

3 MORE HISTORY DOES NOT ENSURE BETTER RECOVERY

  • The comparison restores the environment at ⌊L/3⌋: Raw receives history through that point, while Reference also receives the recorded ⌊L/3⌋–⌊2L/3⌋ segment.Both continuations use the source actor and have up to 30 new actions; recovery is terminal task success within that budget.Source: §3 · p. 3, Figure 2 · p. 3
  • Reference history lowers recovery in all four environment aggregates and all five model aggregates, including a separate-message comparison from 15.8% to 11.4%.Both conditions resume from the same restored state; the reference segment was recorded in the source trajectory but not executed in the restored environment.Source: Figure 2 · p. 3, §3 · p. 4
  • The evaluation covers five models across ALFWorld, ScienceWorld, WebShop, and AppWorld.Figure 2 pools recovery by environment and by model.Source: §3 · p. 3, Figure 2 · p. 3
  • Reference prompts have a median length of approximately 1.6k tokens, which the authors argue is far below current models’ context capacity and cannot alone explain the decline.The prompts retain historical actions and observations but omit intermediate reasoning; Figure 3 compares prompt lengths at the first request after resumption.Source: §3 · p. 4, Figure 3 · p. 4

4 HOW AGENTS USE THEIR HISTORY

History interventions show that historical observations retain useful task information, while disrupting action–observation pairings has only a modest effect on overall success. The results motivate the hypothesis that ordinary history does not reliably elicit action calibration.

4 HOW AGENTS USE THEIR HISTORY

  • The five conditions compare Raw with Recent-3-Only, Observation-Only, Shuffle-All, and Shuffle-Old; the shuffle conditions keep observations chronological while reassigning historical action strings.Each condition contains 10,000 episodes across five actors and four environments.Source: §4 · p. 4, Table 1 · p. 4
  • Shuffle-All retains most of Raw’s performance, with an overall decrease of only 2.55 percentage points.Shuffle-Old stays close to Raw and improves over Shuffle-All when the three latest interactions remain intact; this gain need not reflect action calibration.Source: Table 2 · p. 5, §4 · p. 5
  • Observation-Only retains much of Raw’s success and outperforms Recent-3-Only, indicating that historical observations preserve useful task information without their accompanying actions.Removing earlier history substantially lowers overall success, so recent context is not generally a substitute for accumulated information.Source: Table 2 · p. 5, §4 · p. 5
  • Because preserving recent interactions may make immediate context more coherent and familiar, Shuffle-Old’s closeness to Raw need not indicate that agents calibrate actions from outcomes.The authors connect the history-intervention findings with poorer recovery from additional reference history.Source: §4 · pp. 4–5

5 TURNING RECORDED FEEDBACK INTO USEFUL EXPERIENCE

Outcome labels make existing feedback easier to connect to past actions, improving task success and reducing repetition. Actor-generated experiences can instead retain redundant or unsupported guidance, lowering task completion.

5.1 MAKING ACTION OUTCOMES EXPLICIT

  • Labeling each observation as the outcome of its preceding action improves task success over Raw across most model–environment combinations and every aggregate.The observations and their order remain unchanged.Source: §5.1 · p. 5, Figure 4 · p. 6
  • Causal Transition reduces next-action repetition for all five models on fixed histories, without adding evidence or action advice.The probe selects histories where feedback does not support repeating the same action.Source: §5.1 · p. 6, Figure 5 · p. 6

5.2 FROM OUTCOME LABELS TO EXPLICIT EXPERIENCE

  • Experience adds a separate actor review call after each nonterminal interaction and retains generated guidance for later decisions.The actor can return NO UPDATE, leaving the history unchanged.Source: §5.2 · pp. 6–7
  • Experience lowers full-rollout success versus both Raw and Causal, despite reducing next-action repetition relative to Raw and Causal.Reduced repetition does not translate into improved task success.Source: Figure 5 · p. 6, §5.2 · p. 7
  • Generated experiences can restate observations or introduce unsupported claims, and persistent guidance can affect later decisions.The paper’s examples include an inventory restatement and a claim unsupported by environment feedback.Source: §5.2 · p. 7, Figure 6 · p. 7

6 LEARNING WHEN AND HOW TO CALIBRATE ACTIONS

The learned calibrator selects and writes experiences using supervision from their effects on actors’ next actions. It improves task success over Raw across model and environment aggregates, gives the best recovery among the tested conditions, and improves success when integrated into an actor.

6.1 CONSTRUCTING LABELS FROM PAIRED NEXT-ACTION COMPARISONS

  • The calibrator uses cross-actor changes in next-action quality to learn when to retain an experience and what to write.A candidate is retained when improvements outnumber degradations or no actor’s next-action quality degrades; unchanged quality across actors also permits retention.Source: §6.1 · pp. 7–8

6.2 TASK COMPLETION EVALUATION

  • The calibrator improves task success over Raw in every model-level and environment-level aggregate.The comparison uses five actors and four environments, with rollout tasks distinct from training tasks.Source: §6.2 · p. 8, Figure 7 · p. 8
  • Content screening alone produces lower overall success than the full pipeline, supporting the use of next-action effects to assess experiences.The full pipeline evaluates candidate experiences through paired next actions across actors.Source: §6.1 · pp. 7–8, §6.2 · p. 8

6.3 FAILURE RECOVERY

  • In same-state recovery, all three interventions exceed both unmodified Reference and the Raw prefix-only baseline, with Calibrator achieving the highest recovery.The interventions reverse the recovery loss from unannotated reference history.Source: §6.3 · p. 9, Figure 8 · p. 9

6.4 INTEGRATING CALIBRATION INTO THE ACTOR

  • The joint actor improves success in all four environments, rising from 47.21% to 52.21% averaged across environments and seeds.It generates calibration and action in one response.Source: §6.4 · p. 9

7 CONCLUSION

The conclusion emphasizes that interaction history supplies evidence for revising attempts, but retaining it alone does not ensure better decisions. Effective history use depends on using past outcomes to guide later decisions.

7 CONCLUSION

  • Interaction history provides evidence for revising earlier attempts, but retaining it does not ensure better decisions.Effective history use depends on using past outcomes to guide subsequent decisions.Source: §7 · p. 9

A SHARED EXPERIMENTAL PROTOCOL

  • Full rollouts allow at most 30 actions and stop at success, terminal failure, or exhaustion of the action budget.Continuations restore the environment to L/3 and receive 30 additional actions.Source: A SHARED EXPERIMENTAL PROTOCOL · p. 12

B.1 NEXT-ACTION REPETITION ON FIXED HISTORIES

  • The repetition probe uses fixed histories ending in two identical actions, screened for feedback that does not support repeating them.The same actor receives each history under all three conditions.Source: §B.1 · p. 12, Table 3 · p. 12
  • Raw, Causal, and Experience are three renderings of the same prefix; Causal labels observations by preceding actions, while Experience inserts generated guidance beside the latest transition.A NO UPDATE or invalid update leaves the Raw input unchanged.Source: §B.1 · p. 12

C COMPLETE DIAGNOSTIC SUCCESS RESULTS

The appendices provide complete diagnostic success results, report that a history-check reminder improves on Raw, describe how generated experiences are retained, and summarize experience-generation rates.

C COMPLETE DIAGNOSTIC SUCCESS RESULTS

D.1 EXPLICIT HISTORY RECHECKING

  • Check improves on Raw and achieves a success rate similar to Causal Transition.Table 5 reports full model–environment results and pools 10,000 tasks per condition.Source: §D.1 · p. 14, Table 5 · p. 14

D.2 RETAINING EXPERIENCE IN LATER PROMPTS

  • Generated experiences are linked to their source transition and shown beside it in subsequent action prompts; terminal transitions are skipped.A separate review call receives the task, history, latest action, and returned observation, and can produce NO UPDATE or an experience.Source: §D.2 · pp. 14–15

D.3 EXPERIENCE GENERATION RATES

  • Table 6 reports experience and NO UPDATE counts and percentages across five actors and four environments.Each cell covers 500 tasks, and the overall total pools responses across 10,000 episodes.Source: §D.3 · p. 15, Table 6 · p. 15

D.4 CANDIDATE EXPERIENCE AND NEXT-ACTION QUALITY

The section evaluates whether generated experiences improve next-action quality and describes experience examples and the shared-state recovery setup. Experience guidance can help or harm, and the recovery conditions distinguish reference history from actual interaction.

D.4 CANDIDATE EXPERIENCE AND NEXT-ACTION QUALITY

  • Next-action comparisons pair each recipient actor’s actions with and without an experience on the same history and judge changed executable actions blindly in reversed A/B order.Identical executable actions are ties without judging; inconsistent judgments are also recorded as ties.Source: §D.4 · p. 15
  • 14.0% of comparisons favor actions generated with an experience, versus 13.0% favoring Raw actions, with both outcomes across all five recipient actors.All candidates were generated by GPT-5.6-Sol; Figure 10 counts completed comparisons including ties, though ties are omitted from its bars.Source: §D.4 · p. 16, Figure 10 · p. 16

D.5 GENERATED EXPERIENCES AND THEIR SUBSEQUENT USE

  • Table 7 contrasts redundant restatements, unsupported claims, and experiences calling for further verification; Figure 6 adds two ALFWorld examples.The examples are drawn from additional Experience rollouts and include GPT-5.6-Sol and Qwen3.5-Plus.Source: §D.5 · p. 16, Table 7 · p. 17

D.6 REFERENCE-HISTORY RECOVERY FROM A SHARED STATE

  • The shared-state study restores 1,000 unsuccessful trajectories at ⌊L/3⌋, gives each continuation 30 new actions, and uses 1,000 continuations per condition.Reference adds the recorded segment from ⌊L/3⌋ to ⌊2L/3⌋ without executing it in the restored environment; intervention conditions share that segment.Source: §D.6 · p. 17
  • Causal + Reference labels each returned observation as the outcome of its preceding action, while Experience retains generated updates for later decisions.Reference records are marked separately from actual interaction, and the Calibrator applies the frozen model and update rule to the reference-augmented continuation.Source: §D.6 · p. 18

E.1 FULL TASK-SUCCESS RESULTS: 500 TASKS PER CELL

The section reports full task-success evaluations, an ablation of paired next-action comparisons, and history interventions presented as separate chat messages. The full calibrator outperforms content screening alone, while the native-chat results preserve the main ordering of history conditions.

E.1 FULL TASK-SUCCESS RESULTS: 500 TASKS PER CELL

  • Table 8 reports task-success rates for Raw, Causal Transition, and the trained calibrator, using 500 tasks per cell.The three conditions contain 10,000 episodes each.Source: §E.1 · p. 18, Table 8 · p. 18

E.2 ABLATING PAIRED NEXT-ACTION COMPARISONS

  • 51.45% pooled success with the Full Calibrator exceeds 49.61% with content screening alone, with gains in every model-level and environment-level aggregate.The ablation uses 500 tasks per model–environment cell; content screening omits paired next-action comparisons.Source: §E.2 · p. 18, Table 9 · p. 19

F.1 FULL-ROLLOUT HISTORY INTERVENTIONS

  • The separate-message evaluation uses seven conditions on the same 100 tasks per model–environment cell, totaling 2,000 episodes per condition across five actors and four environments.Rollouts allow at most 30 actions, with an 8,192-token output limit per actor request.Source: §F.1 · p. 19
  • In separate-message history, Causal improves over Raw while Experience does not; Shuffle-Old stays close to Raw and exceeds Recent-3-Only.The setup alternates historical observations and executed actions across user and assistant messages, then supplies the current observation and available actions.Source: F HISTORY AS SEPARATE USER AND… · p. 19, §F.1 · p. 21

F.2 REFERENCE-HISTORY CONTINUATION

Reference history reduces recovery in the separate-message continuation setting. The section also specifies history rendering, the Experience review interface, and how paired-action judgments and joint calibration outputs are produced.

F.2 REFERENCE-HISTORY CONTINUATION

  • The separate-message recovery comparison restores the environment to ⌊L/3⌋ and gives the source actor up to 30 new actions; Reference adds the recorded middle segment as a separate message.That segment is explicitly marked as not executed in the restored environment, followed by the actual current observation and available actions.Source: §F.2 · p. 21
  • With user–assistant history, Reference lowers pooled recovery from 158/1,000 (15.8%) to 114/1,000 (11.4%); recovery declines in all five model aggregates and all four environment aggregates.Each model–environment cell contains 50 paired sources.Source: §F.2 · p. 21, Table 11 · p. 21

G.1 HISTORY RENDERING

  • Causal Transition labels a returned observation as the preceding action’s outcome, whereas Experience stores an experience beside its source transition for later prompts.Raw preserves the ordinary observation–action sequence; NO UPDATE inserts no text.Source: §G.1 · pp. 21–22

G.2 EXPERIENCE REVIEW

  • The review interface uses the task, raw history, latest completed action, and returned observation to decide whether a new task-relevant correction or durable progress fact warrants one concise experience.Otherwise it returns NO UPDATE; any experience must state a general implication of the observation, not a plan or concrete next action.Source: §G.2 · p. 22

G.3 JUDGING AND JOINT GENERATION

  • Changed candidate actions are judged twice with reversed A/B order, while identical executable actions are ties without a judge call.The joint Qwen3-8B actor emits one calibration block followed by the environment action; later calibration reviews the immediately preceding completed transition.Source: §G.3 · p. 22

Related papers

Loading 2610.02769v1…