Source-linked AI summary
Fewer Clarifications, Better Code: Benchmarking Cross-Session Personalized Ambiguity Adaptation in Coding Assistants
Zijian Xu, Wenshuo Zhang, Zisen Qin, Rui Sheng, Yushi Sun, Huamin Qu, Chuhan Shi
TL;DR
Coding assistants often miss recurring, user-specific ambiguity across sessions, repeatedly asking for clarification or producing unintended implementations. This paper formulates cross-session personalized ambiguity adaptation, introduces the CAPA benchmark, and finds that same-user history improves completion and reduces clarification across nearly all evaluated models.
Problem
Whether resolved same-user session history can resolve recurring personalized ambiguity in newly opened coding sessions remains underexplored.
Method
The paper introduces CAPA, a benchmark covering six ambiguity mechanisms across 600 executable coding sessions with balanced same-user histories and held-out evaluations.
Results
Same-user history improves ES for 11 of 12 models, FT-ES for all 12, and reduces average TTC for every model.
Takeaways & Limitations
Cross-session history can improve coding task completion and reduce clarification by helping assistants apply users’ recurring ambiguity-resolution patterns.
Takeaways & Limitations
The benchmark assumes each user’s ambiguity–resolution assignment remains stable across sessions even though the coding task changes.
Abstract
from arXiv · showhide
AI-assisted coding increasingly translates informal user intent into executable software, yet coding requests often contain ambiguities that recur in user-specific ways across tasks and sessions. Existing disambiguation methods typically address each ambiguous request in isolation within the current coding session, often through eliciting additional clarification. However, whether resolved session history from the same user can serve as memory for resolving recurring personalized ambiguity in a newly opened session remains underexplored. We formulate personalized ambiguity adaptation as a new task: given a user's previously resolved coding sessions and a new ambiguous request, an assistant should identify the recurring ambiguity pattern, produce the intended executable solution, and minimize clarification. To benchmark this task, we introduce CAPA, which characterizes personalized coding ambiguity through six mechanisms and injects these mechanisms into unambiguous executable tasks using a controlled three-stage generation pipeline. CAPA contains 600 coding sessions across 60 balanced user--ambiguity cells, including 300 held-out evaluation sessions. We evaluate 12 recent LLMs under no-history and same-user-history conditions using executable success, first-turn success, and turns-to-completion. Our analyses examine task difficulty, user identity, and memory-based history use, and we further propose same-user history gating as a lightweight inference-time method. CAPA provides a foundation for developing long-term coding assistants that better align generated code with user intent while reducing repeated clarification.
Introduction
The paper frames personalized ambiguity adaptation as using a user’s resolved coding-session history to infer intended implementations in new ambiguous sessions while minimizing clarification. It introduces CAPA to benchmark this task and evaluates LLM performance, task difficulty, user identity, memory use, and same-user history gating.
- Related work: Existing ambiguity methods generally treat uncertainty as local to the current coding session, while personalization and long-term memory research addresses related issues separately.Prior approaches detect underspecification, rewrite requests, or ask clarification questions rather than transferring recurring user-specific interpretations across sessions.
- Motivation and task formulation: Personalized ambiguity adaptation uses resolved sessions to infer a user’s recurring interpretation of ambiguous coding requests and produce correct executable solutions with minimal clarification.Without this capability, assistants may repeatedly ask for clarification or make technically valid but unintended assumptions.
- Benchmark construction: CAPA combines a six-mechanism taxonomy of personalized coding ambiguity with a three-stage pipeline that injects controlled ambiguity into unambiguous HumanEval tasks and validates consistency.The mechanisms represent recurring user-related causes of omitted, obscured, or underspecified implementation information and are grounded in real coding conversations.
- Evaluation and analysis: The study benchmarks 12 recent LLMs using Executable Success (ES), First-Turn Executable Success (FT-ES), and Turns-to-Completion (TTC).Analyses examine task difficulty, correctly matched user identity, and memory-based history management, motivating same-user history gating as a lightweight method.
Related Work
Prior work studies ambiguity detection and resolution through multiple interpretations or clarifying questions, including coding-specific benchmarks. Separate research examines personalized cross-session memory and executable or interactive coding evaluation.
- Ambiguity and clarification: Ambiguity research addresses requests with multiple plausible interpretations or insufficient information through taxonomies, question answering, response generation, and clarification.These approaches target syntax, lexical meaning, and missing conversational context.
- Ambiguity and clarification: Coding-specific systems study ambiguous requirements in function-level generation and pre-generation clarification, but require additional inference, output, or interaction.Orchid and ClarifyCodeBench represent these two directions.
- Personalization and long-term memory: Personalization research preserves cross-session evidence, infers stable user characteristics, and selectively applies them through personalized classification, generation, profiling, and response selection.LaMP and PersonaMem examine evolving personal information, while LoCoMo, LongMemEval, and Momento test multi-session recall, temporal reasoning, knowledge updates, and memory.
- Interactive coding benchmarks: Executable coding benchmarks assess program correctness, spanning function-level generation from explicit specifications and repository-level issue resolution.HumanEval, MBPP, and EvalPlus focus on function-level evaluation, while SWE-bench addresses repository-level tasks.
- Interactive coding benchmarks: Interactive coding benchmarks also evaluate conversational generation in reproducible feedback environments.ConvCodeWorld exemplifies this interactive evaluation setting.
Benchmark Task
CAPA tests whether coding assistants can use resolved same-user sessions to infer recurring ambiguity patterns and solve new ambiguous coding tasks. The benchmark separates session history from the held-out task and evaluates direct resolution, clarification, and executable completion.
- History-conditioned evaluation: The assistant receives resolved same-user history and a new ambiguous request, then applies inferred ambiguity patterns within the held-out session.History is supplied as inference-time context without parameter updates or automatic persistent storage.
- Task formulation: Each coding session includes an initial request, a multi-turn dialogue trace, and hidden executable tests judged for task completion.A user trajectory orders multiple coding sessions from the same user.
- Task formulation: CAPA models users through communication profiles, recurring ambiguity mechanisms, and personalized resolution patterns.These user-specific patterns remain stable while coding tasks change across sessions.
- Interaction and evaluation: At each turn, the assistant either asks for clarification or submits candidate code, with success determined by an external judge executing the submission against hidden tests.The session ends at the first successful submission or the turn limit.
- Interaction and evaluation: CAPA measures Executable Success, First-Turn Executable Success, and Turns-to-Completion on held-out sessions.These metrics capture overall completion, immediate completion, and the average first-success turn, respectively.
Data Generation
CAPA generates personalized multi-session coding data by organizing recurring user-specific ambiguity into six mechanisms and applying a controlled three-stage pipeline. The resulting dataset preserves stable ambiguity patterns across varied executable tasks and dialogue traces while separating history from held-out evaluation sessions.
- Ambiguity taxonomy: Six mechanisms reorganize 11 linguistic ambiguity types around recurring user-related causes of underspecified coding intent.The mechanisms are domain-cognitive polysemy, structural logic misalignment, habitual context omission, system-boundary misconception, conversational context misalignment, and implicit constraint under-specification.
- Three-stage pipeline: Stage One converts HumanEval tasks with executable tests into ambiguous requests using an assigned mechanism, resolution pattern, and user profile.The generator extracts required implementation constraints, retains omitted information for later dialogue and judgment, and preserves the original specification and tests as executable ground truth.
- Three-stage pipeline: Stage Two uses AmbiSimu to generate resolved multi-turn sessions through user-agent interaction, clarification or code generation, execution against hidden tests, and judgment.The user agent separates expression from judgment, while sessions terminate on successful external judgment or a predefined turn limit.
- Three-stage pipeline: Stage Three validates that independently generated sessions for each user share the same profile, ambiguity mechanism, and ambiguity-resolution pattern despite differing tasks and dialogue traces.Accepted sessions are organized into user trajectories after cross-session consistency checking.
- Dataset statistics: 600 coding sessions form 60 balanced profile–ambiguity-pair cells, with 10 sessions per cell and five history sessions plus five held-out evaluation sessions.The dataset includes 10 user profiles, each appearing in six cells, and 15 unordered mechanism pairs, each appearing in four cells.
Experiments
Experiments show that same-user history improves coding-assistant adaptation to recurring ambiguity, especially first-turn success and interaction efficiency, while CAPA remains challenging. Difficulty, user identity, and history-management analyses show that personalized evidence and lightweight gating improve history use beyond generic context benefits.
- Main results: Same-user history improves ES for 11 of 12 models, FT-ES for all 12, and average TTC for every model.Frontier models still span 78.7–90.0% ES and 14.0–60.3% FT-ES, indicating that CAPA remains far from saturated.
- Effect of task difficulty: Complex sessions reduce ES by 20.7–26.4 pp and increase TTC by 1.55–2.12 turns versus simple sessions under same-user history.FT-ES is consistently lowest on complex sessions across the three analyzed models.
- User identity: Shuffled histories improve ES by up to 10.67 pp, while correctly matched histories add 2.0–12.0 pp FT-ES and 0.04–0.19 TTC-turn gains.The correctly matched history effect on ES is small and mixed, showing both generic context and user-specific adaptation benefits.
- History management: Same-user history gating improves FT-ES by 0.66–13.33 pp over raw history while keeping ES within ±1.0 pp across all three models.The parameter-free gate highlights consistent ambiguity–resolution evidence or supplies clarification guidance, and reduces TTC for GPT-5.5 and GLM-5.2.
Conclusion
The paper frames personalized ambiguity adaptation as a task for long-term coding assistants and introduces CAPA to benchmark it. CAPA targets recurring user-specific ambiguity across coding sessions while minimizing clarification.
- Task formulation: Personalized ambiguity adaptation infers a user’s recurring ambiguity pattern from resolved sessions and applies it in a new session.The task is designed for long-term coding assistants and emphasizes minimal clarification.
- Motivation: Recurring user-specific ambiguity can cause repeated clarification or misaligned implementations when coding assistants fail to use prior sessions.
- Benchmark: CAPA is a benchmark featuring six coding-oriented ambiguity mechanisms and 600 executable sessions.
Dataset Balance and Difficulty Analysis
CAPA is balanced across users and ambiguity mechanisms, with held-out sessions matched to resolved-history difficulty and distributed across simple, moderate, and complex cases. A blank-control experiment shows the underlying programming tasks are nearly fully solvable with complete specifications, isolating ambiguity resolution as the principal challenge.
- Dataset balance: CAPA contains 600 sessions organized into 60 balanced user–ambiguity cells spanning ten user profiles and six ambiguity mechanisms.The cells provide balanced user–ambiguity coverage for the benchmark.
- Difficulty distribution: The 300 held-out sessions comprise 97 simple, 89 moderate, and 114 complex sessions.Simple sessions require one or two assistant turns, moderate sessions three or four, and complex sessions at least five.
- Difficulty comparability: Held-out evaluation sessions have comparable no-history baseline difficulty to resolved-history sessions, with no systematic ease or difficulty difference.Mean differences are −2.33 percentage points and +0.080 turns; median cell-level differences are 0.0 percentage points and +0.1 turns.
- Blank-control analysis: Both models solve all 164 HumanEval tasks in the blank-control experiment, reaching 100.00% ES.The complete original specifications and direct debugger feedback make the underlying tasks almost fully solvable without CAPA ambiguity.
- Blank-control analysis: DeepSeek V4 PRO achieves higher FT-ES than ChatGPT-5.5, solving 150 tasks on the first attempt versus 146.DeepSeek V4 PRO also requires slightly fewer turns on average.
Analysis by Personalized Ambiguity Mechanism
The analysis defines six personalized coding-ambiguity mechanisms and examines GPT-5.5 across them under same-user history. Performance remains broadly comparable, although mechanisms differ in executable success, first-turn success, and turns-to-completion.
- Mechanism taxonomy: Table 7 defines each ambiguity mechanism through its source linguistic ambiguity type, information-control operation, and personalized resolution pattern.For each user, a_i specifies how required information becomes ambiguous, while r_i records the personalized interpretation resolving it.
- Evaluation method: GPT-5.5 is evaluated across mechanisms under same-user history using multi-label marginal aggregation because each held-out session contains two mechanisms.Each session contributes to both corresponding categories, with ES, FT-ES, and TTC computed as marginal statistics over sessions containing each mechanism.
- Results: Across mechanisms, ES ranges from 81.0% to 88.0%, FT-ES from 24.0% to 38.0%, and TTC from 2.630 to 3.410 turns.Habitual context omission has the highest ES and lowest TTC; structural logic misalignment has the highest FT-ES; implicit constraint under-specification has the highest TTC.
Prompts for Data Generation
The data-generation prompts construct diverse coding-user personas and encode recurring ambiguity mechanisms through explicit behavioral and task-transformation instructions. Together, they vary users’ priorities, interaction styles, domain assumptions, omitted context, logical boundaries, and system-capability expectations.
- Persona Profiles: Persona prompts define users with distinct priorities, including architectural control, deadline-driven execution, exploratory learning, and security-focused risk avoidance.The profiles range from senior backend architects and deadline-pressured junior developers to beginning students and defensive security auditors.
- Persona Profiles: Additional personas emphasize domain expertise, prompt control, snippet integration, visual design, performance, and legacy-system compatibility.These profiles shape how users evaluate generated code and what kinds of implementation tradeoffs they prioritize.
- Ambiguity Taxonomy Prompts: The taxonomy also includes context-defocus ambiguity as a distinct ambiguity category alongside domain-cognitive, structural-logic, habitual-omission, and system-boundary mechanisms.Its presence establishes that the prompt set covers multiple recurring sources of underspecified coding requests.
- Ambiguity Taxonomy Prompts: Domain-cognitive ambiguity prompts inject field-specific terminology and assumptions, requiring specialists to express mainstream coding artifacts through established domain nouns.Examples include wetlab, clinical, trading, and curriculum-design contexts with terms such as specimen run, cohort key, and settlement line.
- Ambiguity Taxonomy Prompts: Structural-logic ambiguity prompts preserve multiple legal readings by obscuring precedence, numeric cutoffs, branch structure, boundaries, loop exits, and quantifier scope.The instructions explicitly target AND/OR interpretation, scope, and batch-versus-item distinctions.
- Ambiguity Taxonomy Prompts: Habitual-omission ambiguity prompts model users who assume shared project memory while omitting concrete schemas, names, formats, interfaces, and algorithm choices needed for execution.The prompt frames this omission as a curse of knowledge in which the user’s internal context never reaches the request.
- Ambiguity Taxonomy Prompts: System-boundary ambiguity prompts encode mistaken assumptions that the chat assistant can access files, screens, servers, daemons, or logs without human execution or credentials.The objective is instead to produce scripts, commands, or steps for a human to run locally or remotely.