Source-linked AI summary
What if LLMs Ate Their Words: Causal History Effects in Multi-Turn Interaction
Jinnan Li, Zheren Fu, Yue Wang, Jinzhe Li, Yuan Wu, Yi Chang
TL;DR
Prior work shows multi-turn degradation and assistant-history effects, but how they vary across tasks, models, turns, and internal states remains unclear. The paper analyzes completed conversations by replaying observed user messages while editing assistant history, and finds task- and model-dependent degradation alongside selective, measurable effects of history interventions.
Problem
It remains unclear how multi-turn degradation and assistant-history effects vary across tasks, models, individual turns, and internal states.
Source: §1 · p. 1Method
The paper retrospectively replays observed user messages while modifying assistant history, including neutralizing prior responses and intervening on individual turns.
Source: §1 · p. 2, §3.3 · p. 4Results
Multi-turn degradation has a task × model structure, and behavior-changing history interventions are associated with measurable downstream state differences that vary across tasks.
Source: §5 · pp. 8–9, §6 · p. 9Takeaways & Limitations
Assistant history can support or interfere with later behavior, so history management should be selective rather than uniform.
Source: §5 · pp. 8–9Replay effects are conditional on the observed user-message history and do not estimate the full adaptive interaction process.
Source: §5.1 · p. 9
Abstract
from arXiv · showhide
Multi-turn interaction creates a feedback process in which an LLM's previous responses become context for later behavior. Prior work shows substantial multi-turn degradation and that assistant-generated history can affect later behavior. However, it remains unclear how these effects manifest across models, tasks, turns, and inside a model. We study these gaps across six task families and five models. Degradation from fully specified single-turn input (FULL) to progressively revealed multi-turn interaction (SHARDED) is clearly task- and model-dependent, and stronger one-shot performance does not imply greater interaction robustness. We then retrospectively analyze completed SHARDED conversations by replaying the user messages already observed in each trajectory while editing only assistant-generated history. Replacing prior assistant responses with neutral content (termed neutralization) changes downstream min-max normalized performance by +.027 across 2,973 trajectories. On a prespecified length-controlled subset, short and length-matched neutralization yield nearly identical effects (+.069 versus +.068), showing that simple context shortening is insufficient to explain the effect of history editing. Turn Surgery further intervenes on one assistant turn at a time. Among 237 selected degraded trajectories, 63.7% contain at least one beneficial intervention, while most tested positions remain unchanged; for binary tasks, 48.4% admit a fail-to-success reversal. An open-weight case study links behaviorally consequential history changes to measurable downstream state differences, but finds task-dependent rather than universal internal signatures. Overall, assistant-generated history has active but selective effects on multi-turn performance, motivating selective rather than uniform history management.
1 Introduction
Assistant responses become future context, raising whether self-generated history can influence later task performance. The paper examines how degradation and history effects vary across tasks, models, turns, and internal states.
Research gap
- Prior work found substantial SHARDED degradation and possible harmful assistant history, but left variation across tasks, models, turns, and internal states unclear.SHARDED progressively reveals a task across user–assistant turns, unlike presenting the complete instruction at once in FULL.Source: §1 · p. 1
Approach
- The study replays observed user messages while modifying assistant history to compare tasks and models, identify consequential turns, and examine internal-state signatures.Its questions cover degradation, history effects beyond context shortening, one-turn interventions, and internal states.Source: §1 · pp. 1–2
Main contributions and findings
- Across six task families and five models, FULL–SHARDED degradation has a task × model structure, and one-shot rankings do not predict interaction robustness.Length-controlled neutralization effects are +.069 for short and +.068 for length-matched neutralization; 63.7% of selected degraded trajectories contain at least one beneficial surgery.Source: §1 · p. 2
Internal-state findings
- Open-weight analyses associate behavior-changing history interventions with measurable downstream state differences, while internal effects vary across tasks.The paper describes these internal signatures as task-dependent rather than universal.Source: §1 · p. 2
2 Related Work
Prior work evaluates multi-turn degradation, context and memory, and trajectory failures; this paper adds retrospective analysis of assistant turns and their downstream effects.
2.1 Multi-turn LLM Evaluation and Underspecified Interaction
- Sharded simulation compares FULL with progressively revealed SHARDED interaction, while prior work studies multi-turn capability, tool use, refinement, and consistency [37] [32] [13] [1].Related work also examines premature answers and unsupported assumptions under incomplete user intent [20].Source: §2.1 · p. 2
2.2 Context and Memory in Interactive Foundation Models
- Context and memory studies examine retention and organization, while assistant-history work tests omission or transformation; this paper probes individual turns in observed conversations.Whole-history neutralization tests effects beyond surface length, and Turn Surgery yields position-specific sensitivity profiles for future selective management.Source: §2.2 · p. 3
2.3 Behavioral and Counterfactual Analysis of Interaction Trajectories
- Trajectory-diagnosis work includes MAST [3], TRAIL [8], and AgentDebug [38]; Turn Surgery instead intervenes on ordinary assistant turns and replays recorded user messages.It estimates position-specific downstream sensitivity conditional on the observed user-message history.Source: §2.3 · p. 3
3 Method: Evaluation and Intervention Framework
The framework compares fully specified and progressively revealed tasks, then replays observed conversations while intervening on assistant history to measure its effects across turns and inside an open-weight model.
3 Method: Evaluation and Intervention Framework
- Replay analyses condition on the user-message sequence already observed in each completed trajectory, excluding effects mediated through later user adaptation.FULL–SHARDED degradation is the difference between normalized scores under the two conditions; model-level summaries equally weight task means.Source: §3 · p. 3, §3.1 · p. 3, §3.2 · p. 4
3.1 Sharded Multi-Turn Evaluation Foundation
- The benchmark compares complete instructions in one message (FULL) with the same information revealed across turns while retaining earlier assistant responses (SHARDED).It contains 627 instructions across six task families and evaluates five models.Source: §3.1 · p. 3, §3.2 · p. 4
3.2 Model Panel and Statistical Evaluation
- Model-level degradation is the task-macro average of paired FULL–SHARDED score differences, estimated with paired bootstrap resampling.The analysis includes 2,989 instruction–model pairs with valid scores in both conditions; incomplete requests remain missing observations.Source: §3.2 · p. 4
3.3 Neutralizing Self-Generated Assistant History
- Short neutralization replaces every non-final assistant response with “Acknowledged,” then replays observed user messages and scores the regenerated final response.The primary estimate uses paired outcomes and equally weights the 30 model × task cells.Source: §3.3 · p. 4
3.4 Controlling for History Length
- Length-matched neutralization replaces prior responses with neutral text matched to their non-whitespace-token lengths, controlling for simple context shortening.The prespecified comparison uses a common paired subset with valid outcomes for short and length-matched neutralization.Source: §3.4 · p. 4
3.5 Turn Surgery: Turn-Level Counterfactual Replay
- Turn Surgery neutralizes one intermediate assistant turn at a time and replays recorded downstream user messages to measure position-specific effects.It selects up to eight cases per model × task cell from degraded trajectories; the resulting 237 selected trajectories are not the full eligible population.Source: §3.5 · p. 4
- Turn-level effects are summarized by first averaging valid branches within each trajectory, then weighting trajectories equally.Positive, no-change, and negative effects use exact score-difference signs; binary fail-to-success reversals are assessed across 157 binary-task trajectories.Source: §3.5 · pp. 4–5
3.6 Open-Weight Internal Analysis
- The open-weight internal analysis compares downstream state changes between positive and no-change surgeries, then uses state restoration and complementary probes to examine task-dependent signatures.It uses Qwen3-14B and held-out data; state restoration is treated as a bounded causal intervention, not a mediation test.Source: §3.6 · p. 5
4 Results
The results are organized around four questions, moving from the degradation landscape to assistant-history effects, turn-level sensitivity, and internal signatures.
4 Results
- RQ1 characterizes FULL–SHARDED degradation across tasks and models.It establishes the multi-turn robustness landscape.Source: §4 · p. 5
- RQ2 tests whether intervening on self-generated assistant history changes downstream performance and whether length matching rules out simple context shortening.Source: §4 · p. 5
- RQ3 localizes history sensitivity to trajectories and turns, while RQ4 examines internal signatures in an open-weight model.Source: §4 · p. 5
4.1 RQ1: Multi-turn Degradation Forms a Structured Task × Model Landscape
FULL–SHARDED degradation varies across both models and tasks, and one-shot rankings do not establish interaction robustness.
4.1 RQ1: Multi-turn Degradation Forms a Structured Task × Model Landscape
- .137, .164, .193, .211, and .224 are the FULL–SHARDED gaps for Luna, Grok, GPT-4o, DeepSeek, and Qwen, respectively.All five models degrade from fully specified input to progressively sharded interaction; Figure 3A shows the task-macro comparison.Source: §4.1 · p. 5, Figure 3 · p. 6
- Grok leads under FULL but does not have the smallest gap, while Luna does not lead under FULL yet has the smallest gap.Thus, stronger one-shot performance does not necessarily imply greater multi-turn robustness.Source: §4.1 · pp. 5–6
- Actions and Database show relatively large losses, Summary is most stable, and Code and Math vary more by model.Figure 3B presents FULL–SHARDED gaps across tasks and models, with 95% paired-bootstrap confidence intervals.Source: §4.1 · p. 6, Figure 3 · p. 6
4.2 RQ2: Self-Generated Assistant History Shapes Downstream Performance
Retrospective neutralization changes downstream performance, but the effects vary across models, tasks, and trajectories. Matching history length leaves the effect nearly unchanged, so context shortening alone does not explain it.
4.2.1 Large-Scale History Intervention Changes Downstream Out-
- Across 2,973 paired trajectories, short neutralization raises the equal-cell model × task macro score from .550 under SHARDED to .577, an overall effect of +.027.The trajectory-equal effect is +.033; effects vary by model and task, including +.223 for Actions and −.133 for Code under short neutralization.Source: §4.2.1 · p. 6
4.2.2 The History Effect Persists Under Length Control.
- On the common 264-trajectory subset, effects are +.069 for short and +.068 for length-matched neutralization relative to SHARDED, with Figure 4A comparing both conditions.The difference is −.001; the paired comparison shows that restoring surface history length does not remove the effect.Source: §4.2.2 · pp. 6–7, Figure 4 · p. 7
4.2.3 History Intervention Produces Heterogeneous Outcome Transi-
- Under short neutralization, 303 of 595 baseline failures are rescued (50.9%), while 195 of 1,329 baseline successes are harmed (14.7%).The same intervention can improve some trajectories and degrade others, motivating tests of individual assistant turns.Source: §4.2.3 · p. 7
4.3 RQ3: Turn Surgery Reveals Broad but Structured History Sensitivity
Turn Surgery finds history sensitivity across selected degraded trajectories, but effects vary across turns and do not follow a simple recency rule. Beneficial interventions can occur at multiple positions, while many tested positions produce no change.
4.3.1 Average Effect and Trajectory Reach.
- Among 237 selected degraded trajectories, the trajectory-equal mean surgery effect is +.148, and Figure 5 summarizes average effects and intervention reach.The branch-weighted effect is +.110, supporting a positive average under both aggregation schemes.Source: §4.3.1 · p. 7, §4.3 · p. 7, Figure 5 · p. 8
- At least one positive surgery occurs in 151 of 237 trajectories (63.7%); 76 of 157 binary-task trajectories (48.4%) contain a fail-to-success reversal.Among trajectories with a positive surgery, 100 of 151 contain multiple positive positions.Source: §4.3.1 · p. 7, Figure 5 · p. 8
4.3.2 Turn-Level Selectivity and Position.
- Across tested positions, 31.5% have positive effects, 57.6% no change, and 10.8% negative effects; surgery effect is only weakly associated with relative position and nearly unrelated to distance from the final user message.The reported associations are 𝜌= −.139 for relative position and 𝜌= −.005 for distance to final, so proximity alone does not identify influential turns.Source: §4.3.2 · p. 7, Figure 5 · p. 8
4.3.3 Stability of Selected Surgery Effects.
- On the frozen 60-trajectory stability subset, 43 interventions retain the same effect class across all three runs (71.7%), while run-to-run variation remains.Effect-sign agreement across 180 within-intervention run pairs is 81.1%.Source: §4.3.3 · p. 7
4.4 RQ4: Internal Signatures Accompany Behavioral History Sensitivity
Behaviorally consequential history interventions accompany measurable downstream state changes, but internal signatures vary across tasks and probes. Restoration results provide a bounded directional link in Data-to-Text, not evidence of mediation or a task-general mechanism.
4.4 RQ4: Internal Signatures Accompany Behavioral History Sensitivity
- Across tasks and probes, internal signatures vary rather than yielding one consistent predictor of behavioral influence.State restoration and other internal signatures vary across tasks, with no single signature recurring consistently.Source: §4.4 · p. 8, §4.4.3 · p. 8
4.4.1 Positive-Effect Interventions Produce Larger Downstream State
- Positive-effect surgeries show larger downstream residual-state divergence in task-balanced and within-trajectory comparisons, including a task-macro AUC contrast of +.00547 across Actions, Code, and D2T.The pooled AUC contrast is reversed at −.00430 because groups differ in task composition; the within-trajectory contrast over 193 matched pairs is +.00337.Source: §4.4.1 · p. 8
4.4.2 State Restoration Provides a Bounded Causal Follow-Up.
- In D2T, held-out and prospective restoration profiles correlate at 𝜌= .456 across 40 layers, but prospective confidence intervals cross zero and the result does not establish mediation or a task-general mechanism.This is a D2T-specific directional link between downstream residual state and behavior.Source: §4.4.2 · p. 8
4.4.3 No Single Internal Probe Is Consistent Across Tasks.
- Static target-turn attention does not distinguish positive from no-change interventions, and other probe results differ across tasks and evaluation cohorts.Attention redistribution is significant only in Actions, while state-propagation AUC is significantly positive only in Code; prospective D2T probes show a different pattern.Source: §4.4.3 · p. 8
5 Discussion
The discussion distinguishes interaction robustness from one-shot capability and treats assistant-generated history as an active, selective influence on later behavior. It presents Turn Surgery as a diagnostic path toward selective history management while emphasizing the study’s scope limits.
5 Discussion
- FULL rankings do not predict FULL–SHARDED degradation, so stronger one-shot performance does not imply greater interaction robustness.The discussion recommends complementing fully specified benchmarks with controlled multi-turn evaluation.Source: §5 · p. 8
- Neutralization changes downstream behavior even after surface history length is restored, with effects varying across models and tasks.Assistant outputs become future inputs that can support or interfere with later behavior.Source: §5 · p. 8
- Turn Surgery finds positive, unchanged, and negative effects across history positions, motivating selective rather than uniform history management.Uniform removal could discard helpful turns alongside harmful ones.Source: §5 · p. 9
- Internal signatures of behaviorally consequential interventions are task-dependent, not a single universal mechanism.Figure 6 summarizes state divergence, restoration profiles, and held-out task outcomes across Actions, Code, and D2T.Source: §5 · p. 9, Figure 6 · p. 9
5.1 Limitations
- Replay effects are conditional on observed user messages and do not estimate fully adaptive interaction, where users might respond differently.The limitation applies to retrospectively altered assistant histories within observed conversations.Source: §5.1 · p. 9
- Length matching rules out simple surface shortening, while Turn Surgery remains retrospective and does not validate an online history selector.Length-matched neutralization uses a shared whitespace-span estimator and repeated neutral filler; Turn Surgery signals require counterfactual intervention and downstream evaluation.Source: §5.1 · p. 9
6 Conclusion
The conclusion studies self-generated assistant history through cross-task evaluation, replay interventions, Turn Surgery, and open-weight analysis. It finds selective, task-dependent effects and motivates future history-management methods based on turn-level intervention signals.
6 Conclusion
- The study combines cross-task evaluation, whole-history intervention, Turn Surgery, and open-weight internal analysis of self-generated assistant history.Its focus is multi-turn reliability effects.Source: §6 · p. 9
- Multi-turn degradation has task × model structure, and altering assistant history changes outcomes even after controlling for surface history length.Sensitivity is broad across selected degraded trajectories but selective across individual history positions; internal signatures are task-dependent.Source: §6 · p. 9
- Assistant history is part of the computational context that interactive models must manage selectively, motivating methods that use turn-level intervention signals.Those methods could learn which prior responses to retain, downweight, or discard.Source: §6 · p. 10
7 Ethical Considerations
The ethical considerations describe retrospective interventions on benchmark conversations and delimit what their results establish. The analyses do not directly establish deployed-system safety, reliability, or failure rates.
7 Ethical Considerations
- The study uses controlled benchmark evaluation and counterfactual replay, not deployment to real users or human-subject experimentation.It also reports no collection of personal or sensitive user data.Source: §7 · p. 10
- Interventions modify assistant history in observed benchmark trajectories while preserving the user messages already present.The stated purpose is diagnosing downstream history effects.Source: §7 · p. 10
- Benchmark history sensitivity is not direct evidence of deployed-system safety, reliability, or failure rates.The causal claims are conditional on observed user-message histories, and Turn Surgery signals are retrospective diagnostics rather than validated online decisions.Source: §7 · p. 10