PaperlayerView exact revision on arXiv

Source-linked AI summary

Clarify User Expertise: Towards Proactive Conversational Agents Tailoring Responses to User Proficiency

Zhihong Cao, Chen Huang

arXiv:2608.22266v1cs.AIcs.CL

TL;DR

The paper addresses the difficulty of determining a user’s query-specific expertise, which is needed to tailor responses for comprehension. It introduces PASSING, which uses self-play-induced What-to-Ask and How-to-Ask strategies for targeted probing. PASSING improves expertise estimation accuracy nearly threefold with an average of 1.2 inquiry turns, while the authors identify prompt sensitivity and underexplored self-play configurations as limitations.

  • Problem

    Existing agents struggle to infer query-specific user expertise from queries alone, although expertise varies by query and affects the appropriate response detail and complexity.

  • Method

    PASSING proactively probes users with targeted inquiries guided by LLM-induced What-to-Ask and How-to-Ask strategies derived through offline self-play.

  • Results

    With an average of only 1.2 inquiry turns, PASSING achieves nearly a threefold improvement in user expertise estimation accuracy and is experimentally validated across baselines, backbones, and datasets.

  • Takeaways & Limitations

    The findings support proactive probing as a lightweight mechanism for acquiring missing user information before generating tailored responses.

  • Takeaways & Limitations

    Performance may be influenced by prompt design, and the effects of self-play configurations on induced strategy quality and downstream personalization remain underexplored.

Abstract

from arXiv · show

In the context of information seeking, conversational agents are undergoing an evolution from reactive tools to proactive, personalized assistants. A critical aspect of this evolution is the ability to tailor strategic interactions to a user's unique needs and expectations. Unlike existing studies that focus on proactively clarifying query ambiguities, we center on clarifying the user's expertise in order to tailor responses for better user comprehension. We find that existing agents struggle to determine user expertise from queries alone, a limitation that prevents them from dynamically adapting their responses. To address this gap, we introduce PASSING to empower the agent to proactively clarify a user's expertise through targeted inquiries. This is achieved by our What-to-ask and How-to-ask strategies, induced by LLM self-play. Our extensive experiments also show our superiority. We believe that PASSING represents a crucial step towards creating more human-centric conversational agents.

1 Introduction

The paper argues that proactive agents should clarify query-specific user expertise, because expertise varies by query and affects response comprehension. PASSING addresses this gap through targeted probing and reports substantially more stable expertise estimation.

  • Motivation: User expertise can vary by query, so agents should estimate query-specific proficiency before tailoring response detail and complexity.Otherwise, novices may be overwhelmed by technical details.
  • Motivation: Existing agents struggle to infer expertise from query text alone, producing highly stochastic judgments for the same query.A user may submit jargon-heavy material without understanding it, creating a façade of expertise.
  • PASSING: PASSING is a proactive probing method that estimates query-specific expertise through targeted inquiries before generating a tailored response.It probes the user’s knowledge scope and reasoning processes through iterative questions.
  • PASSING: PASSING uses LLM-induced What-to-Ask and How-to-Ask strategies obtained through offline self-play simulation.The strategies guide informative questions while balancing information gain against conversational overhead.
  • Results: With an average of only 1.2 inquiry turns, PASSING achieves nearly a threefold improvement in user expertise estimation accuracy.Experiments use various baselines, LLM backbones, and datasets.
  • Contribution: The work highlights expertise estimation as an important step toward more human-centric proactive conversational agents.The proposed method aims to improve comprehension while minimizing conversational overhead.

2 Related Work

Related work shows that query-specific expertise cannot be reliably obtained from fixed user profiles or query-only inference. This motivates proactive probing before response generation.

  • User Expertise Estimation: Existing personalization research often assumes user expertise is known through predefined profiles, but expertise can vary substantially across queries.A general profile is insufficient for determining proficiency on a particular query.
  • User Expertise Estimation: Recent LLM-based inference methods frequently miscalibrate user knowledge by underestimating or overestimating expertise, reducing satisfaction.ExpertPrompting is described as a query-specific method, while other baselines infer general expertise from conversation history.
  • Proactive Conversational Agents: Proactive conversational agents take initiative to uncover task-relevant information and steer information-seeking dialogues toward productive outcomes.Prior proactive information-seeking work has primarily focused on improving response accuracy.
  • Evaluation Context: Table 1 reports prediction stability and unknown rates for existing methods estimating query-specific user expertise.The table frames evaluation around whether methods make consistent, informative judgments.

3 Preliminary Experiment

The preliminary experiment tests whether LLM agents can reliably determine query-specific expertise from repeated judgments. Results indicate that existing agents are inconsistent, and apparent stability can reflect aggregation or abstention rather than accurate inference.

  • Experiment Overview: The experiment evaluates whether LLM agents can reliably determine user expertise for a given query.The setup treats repeated prediction stability as the key evaluation focus because labeled query-specific expertise data are scarce.
  • Experiment Overview: A single query may come from users with very different expertise levels, including domain experts who are novices in the queried topic.Quantum mechanics is given as an example of this query-specific variation.
  • Experiment Setup: The evaluation uses MMLU-Pro and ARC-MCAS queries with six output classes: five Dreyfus expertise levels plus Unknown.The classes are Novice, Advanced Beginner, Competent, Proficient, Expert, and Unknown.
  • Experiment Setup: Prediction stability is measured with Fleiss’ Kappa across repeated expertise judgments.The method evaluates inter-run agreement for individual predictions.
  • Results: Existing agents remain unreliable for query-specific expertise estimation, with consistently low Kappa scores across methods.Their judgments fluctuate across runs despite sometimes exceeding fully random baselines.
  • Results: Self-consistency’s high Kappa can result from majority voting, while high ARC-MCAS scores for CoT and ExpertPrompting coincide with very high Unknown rates.The reported Unknown rates are 89.60% for CoT and 94.00% for ExpertPrompting.
  • Results: ExpertPrompting often fails at query-level expertise prediction because jargon-heavy copied queries can create a façade of expertise.This motivates probing users rather than relying solely on query text.

4 PASSING: Clarify User Expertise

PASSING estimates query-specific user expertise through iterative, slot-based probing, then uses that assessment to tailor the response. Its inquiry process combines What-to-Ask and How-to-Ask strategies induced and refined through offline self-play.

  • Multi-turn Probing and Slot Filling: PASSING tracks expertise evidence in a slot-based schema and updates it after each user response during multi-turn probing.The schema is based on the Dreyfus Model, with slots representing key assessment aspects.
  • Query-specific Expertise Assessment: After probing terminates, PASSING produces a query-specific expertise assessment and generates a personalized response.Termination occurs after a predefined probing budget or another stopping condition described by the method.
  • Strategy Induction via Self-play: PASSING induces probing strategies through offline self-play with controllable user simulators and iteratively refines them across simulation sessions.Simulators vary masked knowledge to represent expertise levels, while refusal behaviors encourage strategies that balance information gain and user cooperation.
  • Strategy Induction via Self-play: The self-play pipeline initializes separate What-to-Ask and How-to-Ask strategy sets, runs bounded simulations, and extracts and refines strategies from dialogue experience.Algorithm 1 specifies maximum sessions and turns, separate strategy sets, user simulators, and accumulated experience.
  • Strategy-guided Inquiry Generation: PASSING selects a diagnostic target before choosing how to phrase the next inquiry.What-to-Ask strategies determine which expertise aspect to probe, while How-to-Ask strategies determine the conversational elicitation method.

5 Experiments

PASSING consistently outperforms baselines in query-specific expertise estimation while improving comprehensibility and preserving factual accuracy. Its strategy analyses indicate that both probing content and interaction style matter, and richer strategy sets generally improve performance.

  • Main Results: +288% average improvement in query-level expertise estimation accuracy over the strongest baseline across LLM backbones and evaluation benchmarks.PASSING also reduces the unknown response rate to nearly zero while maintaining relatively high prediction stability.
  • Main Results: +4.1% improvement in answer comprehensibility, while factual accuracy remains comparable to existing baselines.The result indicates that proactive probing improves adaptation to users’ knowledge boundaries without compromising response correctness.
  • Main Results: PASSING maintains relatively high prediction stability, improving it by 42.5% on average over strong baselines.The authors attribute remaining variance to LLM stochasticity in strategy selection and expertise estimation.
  • In-depth Analysis: Ablations show that removing either What-to-Ask or How-to-Ask strategies degrades performance, with How-to-Ask removal consistently reducing user satisfaction.The findings emphasize both selecting probing content and generating effective questioning interactions.
  • Main Results: Human evaluations show PASSING outperforming IDL across both backbones in expertise estimation accuracy and user satisfaction while preserving competitive response quality.Human–human and human–LLM evaluation agreement averages 0.72 and 0.79, respectively.
  • In-depth Analysis: The induced What-to-Ask and How-to-Ask strategies are not semantically separated, with a silhouette score of 0.006.Their organization is therefore functional: What-to-Ask identifies targets, whereas How-to-Ask operationalizes them through conversational tactics.
  • In-depth Analysis: Increasing the number of probing strategies generally improves expertise estimation accuracy and user satisfaction.The evaluation samples k strategies from one category while keeping the other category complete, suggesting richer probing diversity helps capture expertise and interaction preferences.

6 Conclusion

The paper introduces PASSING to infer query-specific user expertise before generating responses through two LLM-induced strategy sets. Its conclusion argues that personalization should actively acquire missing user information through interaction rather than rely solely on passive modeling.

  • 6 Conclusion: PASSING enables LLM agents to infer query-specific user expertise before generating responses through LLM-induced What-to-Ask and How-to-Ask strategies.The method frames proactive probing as a lightweight, scalable mechanism for real-world interactions.
  • 6 Conclusion: The findings suggest that effective personalization should actively acquire missing information through interaction instead of relying solely on limited passive observations.The authors connect this direction to interaction-centric personalization, adaptive information acquisition, and more human-aware LLM agents.

Limitations

PASSING has limitations related to prompt sensitivity, self-play configuration, and semantic overlap between its probing-strategy categories.

  • PASSING’s performance may be influenced by prompt design, reflecting the broader prompt-sensitivity challenge of LLM-based systems.
  • The study does not systematically examine how self-play sessions or maximum conversation turns affect induced-strategy quality.
  • The What-to-Ask and How-to-Ask strategies have semantic overlap, while systematic disentanglement remains future work.

Ethical Considerations

The study supplements large-scale LLM-simulator evaluation with voluntary human evaluations conducted under a standard, non-sensitive question-answering setting.

  • Primary experiments rely on LLM-based user simulators, supplemented by human evaluations with voluntary participants.
  • Participants were informed about the experimental setting and completed standard question-answering interactions without sensitive or psychologically stressful tasks.

LLM Usage

LLMs are used throughout the study as conversational-agent backbones, evaluators, simulators, strategy-induction components, and implementation aids. The experiments combine benchmark datasets, self-play, expertise modeling, and response evaluation.

  • LLM roles: LLMs serve as backbone models for conversational agents and evaluators, and are also used for language polishing.
  • Datasets: The study evaluates query-specific expertise estimation on MMLU-Pro and ARC-MCAS, using dataset splits and samples specified for strategy induction and evaluation.
  • Baselines: Expertise-estimation baselines include direct prompting, Chain-of-Thought, self-consistency, Diverse-Aspect, and dialogue-history-based IDL approaches.
  • Strategy design: What-to-Ask identifies diagnostic expertise evidence, while How-to-Ask phrases probes naturally and with low interaction burden.
  • Strategy induction: PASSING induces probing strategies through self-play with simulated users whose expertise is controlled by masking portions of query-specific knowledge graphs.
  • Evaluation: Fleiss’ Kappa measures prediction stability across random seeds, while GPT-5 evaluates tailored-response factual accuracy across four five-point dimensions.

B.7 Implementation Details of Experiment

The experiment analyzes 21 induced strategies using text embeddings and cosine similarity, reporting within-category, cross-category, diversity, and silhouette statistics. Nearest cross-category pairs support a functional relationship between the two strategy types.

  • The analysis encodes 21 induced strategies with text-embedding-3-small and represents each as a 1536-dimensional embedding.
  • The strategy set contains 13 What-to-Ask and 8 How-to-Ask strategies, with pairwise cosine similarities computed for within- and cross-category comparisons.
  • The reported statistics include intra-class similarity, inter-class similarity, diversity gap, and silhouette score under the two-category labeling.
  • The closest cross-category pairs are S-03/S-17 (0.825), S-01/S-16 (0.807), and S-02/S-19 (0.759).
  • These nearest-neighbor pairs support interpreting How-to-Ask strategies as specifying questions that elicit evidence for What-to-Ask diagnostic targets.

B.8 Human Evaluation Reliability

The human evaluation shows moderate-to-strong agreement among human raters and comparable alignment between human and LLM judgments. Expertise-level predictions are also generally close to ground truth.

  • Setup: Reliability is assessed from both human–human and human–LLM perspectives to evaluate participant consistency and automatic-evaluator alignment.Human–human agreement uses independent ratings, while human–LLM agreement compares evaluator scores with aggregated human scores.
  • Metric and computation: Ordinalweighted Gwet’s AC2 compares observed agreement with chance-expected agreement using ordinal penalties for larger Likert-score differences.The evaluation uses 1–5 Likert-scale scores and applies ordinal weights.
  • Agreement results: Human–LLM AC2 ranges from 0.74 to 0.85 (µ = 0.79), comparable to human–human agreement.The comparison uses aggregated human scores and corresponding LLM-evaluator scores at the dimension level.
  • Expertise-level prediction: 96.97% of predictions fall within one Dreyfus level of ground truth, with pooled MAEs of 0.230 for GPT-4o and 0.224 for DeepSeekV4-Pro.The reported dataset-specific MAEs are 0.343 and 0.147 for GPT-4o, and 0.243 and 0.211 for DeepSeekV4-Pro on MMLU-Pro and ARC-MCAS, respectively.
  • Expertise-level prediction: Confusion matrices organize ground-truth Dreyfus levels by rows and predicted levels by columns from Novice through Expert.The matrices are reported across LLM backbones and datasets.

C Case Study

The case study illustrates PASSING’s adaptive probing process: it combines What-to-Ask and How-to-Ask strategies to diagnose expertise from user reasoning, then produces a tailored response. In a telescope example, two turns populate the schema, classify the user as Advanced Beginner, and support an explanation matched to the user’s misconception.

  • Case-study pipeline: PASSING maps an initial query through probing questions, user responses, schema filling, expertise estimation, and a tailored response.This pipeline is illustrated in the case study’s full process.
  • Initial probing: A telescope case uses S-03 and S-14 to ask about light-gathering power versus magnification, revealing explicit knowledge scope and an overwhelmed perspective.The user initially associates a larger aperture mainly with zooming rather than brightness.
  • Follow-up probing: A follow-up comparison uses S-02 and S-18 to elicit transferable reasoning about collecting rainwater and telescope light.The resulting schema records context-free components, intuitive action, and detached commitment.
  • Termination and estimation: All five schema slots are populated within 2 turns, and PASSING estimates the user as Advanced Beginner.The estimate follows the two probing exchanges in the case study.
  • Tailored response: The tailored answer explains that light-gathering power depends on aperture area, so doubling diameter roughly quadruples collected light and reveals fainter objects.The response directly addresses the user’s confusion between aperture and magnification.
  • Strategy design: The strategy set includes probing applications, conceptual distinctions, boundary cases, causal reasoning, and synthesis to distinguish levels of understanding.These strategies target action, knowledge scope, perspective, and integrated understanding across different expertise levels.
  • Comparison with IDL: IDL can underestimate a Competent computer-science user and overestimate a Competent chemistry user when historical question signals are misleading.The contrasting errors are attributed to reliance on topic difficulty, domain consistency, and interaction format rather than direct reasoning evidence.
  • Comparison with IDL: PASSING probes concept boundaries, mechanisms, uncertainty, and applied judgment to observe reasoning rather than infer expertise from historical topics alone.The case comparison contrasts PASSING’s reasoning-based diagnosis with IDL’s proxy-based inference from question history.
Loading 2608.22266v1…