PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs Get Lost in Evolving User Intent

Jihoon Tack, Philippe Laban, Jennifer Neville

arXiv:2607.20734v1cs.LG

TL;DR

LLMs are increasingly expected to track user intent as it evolves, but existing evaluations mainly test fully specified, single-turn tasks. This paper converts verifiable benchmarks into dynamic multi-turn conversations and finds substantial degradation when intent changes, including GPT 5.5 falling from 99.0% to 80.5% on math after six transitions.

  • Problem

    Existing evaluations mainly test fully specified, single-turn tasks, leaving limited evidence about whether LLM agents track and act on intent disclosed and revised throughout conversations.

  • Method

    The framework backward-synthesizes multi-turn conversations from verifiable single-turn benchmarks using incremental reveal, revision, and task-switch transitions while preserving the original verifier.

  • Results

    GPT 5.5 drops from 99.0% to 80.5% on math after six intent transitions, with substantial degradation also observed across multiple domains and model families.

  • Takeaways & Limitations

    Dynamic intent tracking remains a fundamental gap for LLM agents that static, single-turn evaluation does not expose.

  • Takeaways & Limitations

    The framework models intent evolution but not finer-grained user variation such as persona, communication style, typos, or grammatical errors.

Abstract

from arXiv · show

As LLMs become more capable, they are increasingly deployed as collaborative agents, taking on user-delegated tasks through iterative interaction. Yet genuine interaction is inherently dynamic: users rarely specify their intent upfront, instead disclosing, revising, and reshaping it as the conversation unfolds. Despite this, LLMs are still predominantly evaluated or trained in single-turn, fully-specified settings, leaving open a fundamental question: how well do LLMs track and act on user intent as it evolves over the course of a conversation? To study this, we introduce a framework that transforms static, single-turn tasks into dynamic multi-turn conversations in which the user's intent evolves across turns--incrementally revealed, revised, and at times redirected mid-conversation--while preserving each task's original evaluation protocol, enabling existing benchmarks to be reused as controlled testbeds without new annotation. Across multiple tasks, we surface a consistent phenomenon: strong static-setting performance does not transfer to the evolving-intent setting, with substantial drops across model families. Our findings point to a fundamental gap: today's LLMs do not yet faithfully track and act on the user's evolving intent, a capability invisible to static evaluation yet critical for future collaborative agents.

1 Introduction

The paper introduces a framework that converts verifiable single-turn benchmarks into dynamic multi-turn conversations with evolving user intent while preserving automatic evaluation. Across domains, models that perform strongly in static settings degrade substantially after intent evolves through conversation.

  • Motivation: LLM agents must track and act on user intent as it is disclosed, revised, and reshaped during long-horizon collaboration.Existing evaluations mostly specify the task fully in a single user turn, leaving this capability insufficiently tested.
  • Limitations: Existing multi-turn benchmarks limit scalable evaluation through LLM-based judging, short horizons, and weak control over user behavior.Their emphasis on incremental disclosure overlooks broader interaction dynamics and does not provide a fully verifiable setting for long interactions.
  • Framework: The proposed framework converts any verifiable single-turn dataset into a multi-turn environment while preserving automatic verification against the original answer.It reuses scalable, verifiable supervision from static benchmarks rather than constructing new environments and annotations from scratch.
  • Framework: Three dynamics model evolving intent: incremental disclosure under under-specification, revision of prior details, and task switching to a related task.Task switching retains part of the previous task context, while the final turn remains anchored to the source intent for direct verification.
  • Results: 99.0% to 80.5%: GPT 5.5’s math performance drops after 6 intent transitions.The framework transforms benchmarks in math, text-to-SQL, search, and coding and evaluates both frontier and open-source models.

2 Related Works

Related work increasingly evaluates agentic LLMs in multi-turn interactions, where user behavior creates a combinatorially larger space of trajectories than single-turn tasks. Recent user-simulation environments model under-specification by making prompts vague or distributing information across turns, but these approaches cover only a single axis of intent dynamics.

  • Multi-turn evaluation: Multi-turn interaction broadens the evaluation surface because user behavior induces a combinatorially large space of trajectories.This contrasts with single-turn evaluation based on a single fully specified query.
  • User simulation environment: Recent benchmarks embed simulated users to study how agents handle more realistic multi-turn interaction.One approach targets under-specification by transforming fully specified problems into vague prompts or sharding information across turns.
  • User simulation environment: These constructions cover only a single axis of intent dynamics and remain bounded by their source tasks.

3 Formulating Evolving User Intent

The paper models user intent as a controllable structured state that evolves across conversational turns. Its transition framework captures argument reveals, argument revisions, and function switches, requiring agents to revise their understanding rather than merely accumulate information.

  • Controllable user intent: User intent is represented as a structured state whose transition dynamics govern its evolution across turns.The formulation enables precise control over what the user reveals and distinguishes evolving intent from fixed under-specified states.
  • Controllable user intent: The intent state comprises a desired function f_t and argument set C_t, while C_rev,t records arguments already revealed to the agent.The task-dependent answer y_t may be a target answer for math or a unit-test suite for coding.
  • Intent state transitions: The framework defines three potentially co-occurring transitions: argument reveal, argument revision, and function switch.These transitions respectively disclose a missing argument, change a revealed argument’s value, or pivot to a different task while carrying over shared arguments.
  • Intent state transitions: Argument reveals expand the agent’s observations, revisions change the underlying intent, and function switches change the task while preserving relevant shared arguments.The agent must gather missing information, update beliefs instead of anchoring to prior values, and detect task shifts while reusing relevant intent.

4 Simulating Evolving User Intent

The framework converts single-turn verifiable tasks into automatically verifiable, long-horizon conversations whose intents evolve through reveals, revisions, and function switches. It anchors each simulation in the source task while retrospectively constructing predecessor intents and rendering consistent multi-turn user behavior.

  • Anchor intent: The framework treats each single-turn example’s function, arguments, and ground-truth answer as the final-turn anchor intent.This reuses existing verifiable datasets while preserving the source task’s evaluation target.
  • Automatic verification: The final action a_T can be scored against the source answer y∗ using the original dataset verifier.Because the simulation terminates at the source intent by construction, the source dataset’s verifier remains applicable.
  • Preceding intents: Preceding intents are synthesized retrospectively through counterfactual arguments for revisions and overlapping predecessor functions for switches.Argument reveals require no synthesis because source arguments can be distributed across turns; predecessor functions can be recursively chained.
  • Situated simulation: Intent transitions are scheduled across T turns under consistency rules that end with the fully revealed source intent and require coherent predecessor and revision ordering.The rules include reveal-before-switch and retrospective placement of predecessor functions and counterfactual arguments.
  • Turn rendering: User turns are rendered from each turn’s intent update rather than the full intent, combining newly revealed or revised arguments with natural-language transition cues.An optional LLM-based renderer rephrases the same update conditioned on the prior agent turn for more reactive settings.

5 Experiments

Across four benchmarks, strong single-turn performance generally fails to transfer when user intent evolves across multi-turn conversations. Performance worsens with more complex transitions, especially function switches, while practical mitigations recover some but not all of the lost accuracy.

  • Main result: Strong single-turn performance usually does not carry over to evolving-intent conversations, with substantial degradation even for models that excel in static settings.The evaluation compares fully specified single-turn interactions with conversations containing reveal, revision, and function-switch transitions across four datasets.
  • Task effects: Search and software-engineering tasks show particularly large degradation, with agents sometimes exhausting their per-turn tool-call budgets and timing out.For SWE-Bench, several agents, including GPT 5.1, easily exhaust the 100-call budget during evolving-intent interactions.
  • Transition effects: Increasing the count of any intent-transition type gradually reduces accuracy, while function switches cause the steepest decline.Function switches require discarding more previously accumulated context and updating the agent’s belief state more substantially than reveals or revisions.
  • Context carryover: Additional reveal or revision transitions after a function switch produce substantially worse performance than evaluation immediately after the switch.The compared cases have similar source accuracy, suggesting that later updates make it harder to retain and use relevant pre-switch context.
  • Mitigation: Oracle recap raises GPT 5.5’s BIRD-SQL accuracy from 65% to 75% under function switch, but remains below the 80% single-turn accuracy.Both evaluated recap designs yield non-trivial gains, while oracle recap provides the larger improvement.
  • Task difficulty: Increasing source-task difficulty hurts multi-turn performance more than single-turn performance on BIRD-SQL with GPT 5.5.The experiment varies difficulty for the same source problems by controlling whether domain-expertise hints are provided.

6 Conclusion

The paper investigates whether LLM agents can faithfully track and act on user intent as it evolves during conversation. It introduces a framework that converts verifiable single-turn benchmarks into multi-turn environments while preserving their original verifiers.

  • Conclusion: The framework evaluates whether LLM agents faithfully track and act on evolving user intent across conversations.It studies intent evolution rather than treating user intent as fixed upfront.
  • Conclusion: It converts any verifiable single-turn benchmark into a multi-turn environment by anchoring the source intent at the final turn.Preceding turns are backward-synthesized from that final-turn intent.
  • Conclusion: The synthesized conversations model three controllable transitions: argument reveal, revision, and function switch.The original dataset verifier is preserved throughout the transformation.

Limitations and Future Directions · A User-Agent Interaction

The framework captures evolving intent but abstracts away important user-behavior variation, multiple intents per turn, and intermediate verification. The user-agent interaction is modeled as a POMDP in which the agent infers latent intent from utterances and conversation history.

  • Limitations and Future Directions: The framework does not model persona, communication style, typos, or grammatical errors, which may make rendered utterances stylistically uniform.A proposed extension is persona conditioning so counterfactual arguments, predecessor functions, and rendered utterances reflect diverse user backgrounds and communication.
  • Limitations and Future Directions: Each user turn is assumed to contain a single intent transition, excluding turns that revise one request while switching tasks or ask multiple related questions.Extending the framework to multiple intents and multiple verifiable targets would broaden the conversational behaviors captured.
  • Limitations and Future Directions: The single-transition assumption keeps the dynamics controllable and the final verifier well defined.
  • Limitations and Future Directions: The verifier is exact only at the final turn because each conversation converges to a source single-turn problem.This is a deliberate scalability and automatic-evaluation tradeoff: scheduled intent transitions still control intermediate trajectories, whose effects appear in the final outcome.
  • Limitations and Future Directions: Future work could design intermediate verifiers that preserve scalability while allowing more flexible user-intent updates.
  • A User-Agent Interaction: The user-agent collaboration loop is formulated as a POMDP with latent user intent, interaction history, rendered utterances, agent observations, actions, and rewards.
  • A User-Agent Interaction: At user turn t, the renderer observes latent intent state I_t and history h_t−1, while the LLM agent observes only o_t = (u_t, h_t−1) and must infer intent.The utterance is conditioned on the updated function and newly revealed or revised conditions.

B Experimental Setup Details · B.1 Model Details

The experimental setup describes the models and datasets used in the paper, evaluating both closed-source and open-source LLMs. The model suite includes multiple proprietary systems and large mixture-of-experts models with specified parameter counts.

  • B Experimental Setup Details: The experimental setup covers the models and datasets used in the paper.
  • B.1 Model Details: The evaluation includes several closed-source LLMs.
  • B.1 Model Details: The closed-source models include GPT 5.1, GPT-5.2, and GPT 5.5.
  • B.1 Model Details: The evaluation also includes Gemini 3.1 Pro and Grok 4.20.
  • B.1 Model Details: GPT 5.4, GPT 5.4 mini, and GPT 5.4 nano are used in Section F for model scaling experiments.
  • B.1 Model Details: Closed-source models lack publicly available full parameter counts.
  • B.1 Model Details: The open-source evaluation includes Kimi K2.5 and Kimi K2.6, both 1T-parameter MoE models with 32B activated parameters.
  • B.1 Model Details: Mistral Large 3 is a 675B-parameter MoE model with 39B activated parameters.

B.2 Dataset Details … D.3 Verifying Predecessor Functions

The paper constructs evolving-intent conversations from single-turn benchmarks by extracting structured intents, scheduling valid transitions, rendering user turns, and verifying that trajectories preserve the original task. The resulting pipeline supports revisions, task switches, memory interventions, and automatic final-answer verification across four benchmark domains.

  • B.2 Dataset Details: Four single-turn benchmark domains—math, text-to-SQL, agentic search, and agentic software engineering—are converted into evolving-intent evaluation tasks.The datasets include GSM8K, BIRD-SQL, BrowseComp+, and SWE-Bench Verified, with final turns anchored to original tasks for verification.
  • C Simulating Multi-Turn Conversations: The simulator extracts an anchor intent, creates predecessor functions and counterfactual arguments, schedules their transitions, and renders each structured update as a natural-language user message.Intent trajectories end in the original benchmark state while earlier turns may reveal incomplete information, revise arguments, or switch tasks.
  • C.1 Turn Scheduling: Scheduling requires T ≥1 + g + p turns and reserves function switches and argument revisions before filling remaining turns with valid argument reveals.Revision values must appear before correction deadlines, and reveals cannot occur under an incompatible active function.
  • C.2 Text Rendering: Two interchangeable renderers produce user utterances: a rule-based renderer uses transition-specific prefixes, while an LLM naturalizer creates shorter conversational forms for BrowseComp+.Naturalized turns are checked against rule-based references for critical tokens and fall back to rule-based text after repeated validation failure.
  • C.3 Memory Mechanism Examples: Memory interventions append either a fixed prompt recap or an oracle recap to non-initial turns, with the latter supplying the active function and currently relevant argument values.Oracle recap removes unrevealed arguments and substitutes the latest values for revised arguments.
  • D Verification of Simulation Components: Verification preserves the final-turn anchor by checking extracted-intent coverage and answer equivalence, while allowing earlier turns to be incomplete, incorrect, or task-switched.Extraction is accepted when a reference solver agrees with the original question or produces the original gold answer.
  • D.2 Verifying Counterfactual Arguments: Counterfactual arguments are accepted only when they replace one localized value while preserving surrounding structure, rejecting additive, deletive, incompatible, or excessively broad edits.The verification also applies light normalization and requires the replacement value to remain within a bounded length ratio.
  • D.3 Verifying Predecessor Functions: Predecessor functions must share context without duplicating the final function, and predecessor-only arguments must preserve answers for the final task and other tasks in multi-predecessor chains.The pipeline rejects predecessors when added arguments change solver outputs, indicating cross-turn information leakage.

E Prompt Templates

The prompt templates specify three pipeline stages—extraction, predecessor-function synthesis, and counterfactual-argument synthesis—using domain-specific prompts with GPT 5.1. They enforce structured decomposition, factual value swaps, and causally connected predecessor goals for dynamic search conversations.

  • E Prompt Templates: The pipeline comprises extraction, predecessor function synthesis, and counterfactual argument synthesis, with domain-specific prompts illustrated through BrowseComp+ search.The prompts for other domains follow the same structure, and GPT 5.1 is used as the underlying LLM.
  • E.1 Extraction Prompts: Extraction prompts decompose complex research queries into a self-contained GOAL and separate identifying CONDITIONS.The requested output is a JSON object containing a goal and a list of condition objects.
  • E.1 Extraction Prompts: The extraction rules prohibit embedding specific constraints, dangling references, or unsupported pronouns in the goal, requiring an abstract description of what to find.The prompt explicitly distinguishes what to find from the identifying criteria.
  • E.1 Extraction Prompts: The example converts the query into the function “What is the name of the ancient village?” plus three landmark-related arguments.The three arguments preserve the landmark date, trail-length range, and aerial-distance condition from the raw question.
  • E.2 Synthesize Prompts for Counterfactual Argument: Counterfactual prompts require replacing exactly one qualifying detail with a nearby, plausible value that creates a genuine factual contradiction.The replacement must be a value swap rather than an addition or elaboration, and it may target dates, quantities, ranges, attributes, categories, or temporal phrases.
  • E.3 Synthesize Prompts for Predecessor Functions: Predecessor prompts generate a short, self-contained earlier question whose answer causally enables or motivates the next goal.The predecessor uses shared conditions plus 1-3 new, noncontradictory conditions, while seeking a genuinely different entity or information type.
  • E.3 Synthesize Prompts for Predecessor Functions: The predecessor output records its goal, sought entity type, shared condition IDs, and new conditions in a structured format.The predecessor goal is capped at 30 words and the entity type at 1-3 words.

F Additional Experiments and Analysis · F.1 Effect of Reasoning

This experiment tests whether stronger per-turn reasoning helps agents adapt to evolving user intent. GPT 5.1’s instant and reasoning modes are compared on GSM8K and BIRD-SQL under conditions designed to isolate inference-time reasoning effects.

  • F.1 Effect of Reasoning: GPT 5.1 is evaluated in instant and reasoning modes to test whether stronger per-turn reasoning mitigates performance degradation.The comparison targets adaptation to evolving intent.
  • F.1 Effect of Reasoning: The experiment uses GSM8K and BIRD-SQL as the evaluation datasets.These datasets provide well-defined final answers that are automatically verifiable.
  • F.1 Effect of Reasoning: GSM8K and BIRD-SQL are selected because their final answers are well defined and automatically verifiable.This makes it possible to assess outcomes without additional manual verification.
  • F.1 Effect of Reasoning: The selected tasks do not require long-horizon tool interaction or open-ended planning.This limits the experiment’s focus to per-turn reasoning rather than extended agent execution.
  • F.1 Effect of Reasoning: The study is designed to isolate whether additional inference-time reasoning helps agents adapt to evolving intent.Task selection removes long-horizon interaction and open-ended planning as primary explanations for performance changes.
  • F.1 Effect of Reasoning: Table 4 reports the comparison between GPT 5.1’s instant and reasoning modes.The supplied passage introduces the table but does not include its complete quantitative results.

F.2 Response Length Analysis

Under evolving intent, open-source models produce longer responses than in single-turn settings on BIRD-SQL, likely because they process accumulated and potentially revised context.

  • Response Length Analysis: On BIRD-SQL, four open-source models produce longer responses under evolving intent than in the single-turn setting.The analysis uses open-source models because hidden reasoning traces make response-length estimation difficult for closed-source models.

F.3 Effect of Model Capacity · F.4 Effect of the Number of Conversation Turns · F.5 RL with Evolving Intent

The appendix shows that evolving intent interacts with model capacity and intent transitions more strongly than with conversation length alone. Preliminary GRPO training on evolving-intent examples improves dynamic performance while preserving single-turn performance.

  • F.3 Effect of Model Capacity: GPT 5.4 nano degrades substantially more under evolving intent than GPT 5.4 and GPT 5.4 mini, despite nearly identical single-turn GSM8K accuracy.The exact model sizes are undisclosed; the variants are compared in reasoning mode.
  • F.3 Effect of Model Capacity: Under evolving intent, GPT 5.4 and GPT 5.4 mini remain comparable, indicating that the capacity-related degradation is concentrated in the nano variant.
  • F.4 Effect of the Number of Conversation Turns: Adding turns through intent-consistent repetition does not reduce GPT-5.1 GSM8K accuracy, whereas intent transitions at the same seven-turn length do.Table 5 compares four- and seven-turn conversations and separates repetition from intent-transition composition.
  • F.5 RL with Evolving Intent: GRPO training Qwen3-4B on evolving-intent GSM8K examples improves evolving-intent performance while preserving single-turn performance.The preliminary training uses fewer than 50 steps and standard outcome reward.
  • F.4 Effect of the Number of Conversation Turns: Turn-matched controls isolate evolving intent from conversation length by increasing turns without changing the underlying intent dynamics.The control repeats a previous conversation turn while holding intent constant.
  • F.5 RL with Evolving Intent: The framework can transform existing single-turn tasks into multi-turn verifiable conversations for training models to track and act on changing user intent.This extends the framework beyond evaluation to constructing training examples.
Loading 2607.20734v1…