PaperlayerView exact revision on arXiv

Source-linked AI summary

XYBench: Can LLMs Respond Pragmatically to Queries with Misconceptions?

Akhila Yerukola, Jena D. Hwang, Mingqian Zheng, Jenna Godsey, Hyunwoo Kim, Valentina Pyatkin, Jennifer Hu, Maarten Sap

arXiv:2609.06842v1cs.CLcs.AI

TL;DR

LLMs often receive queries whose stated methods reflect misconceptions about users’ underlying goals, motivating a benchmark for pragmatic redirection. XYBENCH evaluates responses across 8,115 technical and everyday queries, finding that models usually answer literally and remain substantially behind humans at identifying misconceptions and generating pragmatic alternatives.

  • Problem

    LLMs may follow users’ stated methods without recognizing the intended goals and procedural misconceptions underlying those requests.

  • Method

    XYBENCH is a human-validated benchmark of 8,115 queries, evaluated using cooperative-response dimensions for stated solutions, misconception identification, and pragmatic intended solutions.

  • Results

    Models overwhelmingly answer literal requests, rarely identify misconceptions, and lag behind humans at redirecting users toward solutions suited to their underlying goals.

  • Takeaways & Limitations

    Pragmatic response generation remains underdeveloped: explicit intent improves generation, but models still lag behind human responses.

  • Takeaways & Limitations

    The benchmark lacks human judgment studies for interpreting misconception queries and human reference responses for synthetically constructed WikiHow queries.

Abstract

from arXiv · show

When non-expert users ask LLMs for assistance, their queries can often have misconceptions (e.g., "How do I parse XML with regex?"). In such cases, often referred to as the XY-problem, LLMs must identify the misconception ("regex are fragile") and meaningfully direct the user toward a pragmatic solution that will address the root problem implicit in the request ("use an XML parser"). We introduce XYBench, a benchmark of 8,115 such queries, drawn from technical (StackOverflow/StackExchange) and everyday (WikiHow and a manually-curated subset) domains. We design an evaluation paradigm that assesses model responses along three criteria grounded in cooperative response theory: (a) presence and (b) emphasis on pragmatic solutions, and (c) identification of misconceptions. Our experiments show that even the strongest LLMs predominantly answer the literal request (0.75--0.92) and far less often the intended one (0.33--0.71), while substantially lagging behind humans at identifying misconceptions (at most 63% vs. 79--90%). Further, models overwhelmingly prefer pragmatic responses in a multiple choice setting yet consistently fail to generate them. Oracle ablation experiments show that providing explicit user intent at generation time helps; however a large gap remains, suggesting pragmatic redirection is a fundamentally underdeveloped capability in current LLMs.

1 Introduction

XYBench studies whether LLMs can move beyond misconception-laden stated requests to infer users’ intended goals and respond cooperatively. The benchmark and experiments show that models usually answer literally, identify misconceptions poorly, and struggle to generate pragmatic redirections.

  • Humans infer intended goals behind misconception-laden requests and redirect users toward solutions that address those goals.Examples include replacing regex with an XML parser for structured-data extraction and a hammer with a screwdriver for fastening.
  • XYBENCH contains 8,115 human-validated queries spanning technical and everyday domains, each linking a stated request, intended request, and misconception.Its sources include Stack Overflow and Stack Exchange, WikiHow, and manually curated online queries.
  • The benchmark evaluates stated-request solutions, misconception identification, and pragmatic solutions to users’ underlying goals.These dimensions are grounded in cooperative response theory.
  • Models address stated requests at 0.75–0.92 but intended requests at only 0.33–0.71, while misconception identification reaches at most 60–63% versus 79–90% for humans.Models also prefer pragmatic answers in multiple-choice settings yet fail to generate them reliably.

2 Related Work

Prior work addresses ambiguity, false presuppositions, and literal instruction following, but largely emphasizes factual knowledge rather than procedural misconceptions about how to pursue a goal. XYBENCH targets this underexplored pragmatic capability.

  • Existing NLP research studies multiple interpretations, underspecified questions, incremental intent disclosure, and false presuppositions.
  • Much prior work focuses on knowledge-level factual correctness, leaving procedural presumptions about problem decomposition and method selection less directly addressed.
  • A misconception is an internally coherent but incorrect or suboptimal problem framing caused by prematurely adopting a particular solution interpretation.
  • Cooperative systems should identify the misconception, answer truthfully, and address the user’s intended problem rather than merely following the surface query.

3 XYBENCH Design and Construction

XYBENCH combines real, synthetic, and manually curated misconception queries, extracts stated and intended requests with associated response dimensions, and filters and validates the resulting benchmark. Its evaluation distinguishes literal solutions, pragmatic alternatives, and misconception identification.

  • Data sources: XYBENCH combines Stack Overflow and Stack Exchange queries, WikiHow-generated procedural queries, and manually curated troubleshooting scenarios across technical and non-technical domains.
  • Query structure: Each instance pairs a stated request anchored to a suboptimal solution path with an intended request inferred from contextual clues.
  • Response dimensions: The benchmark defines three response dimensions: S-solution, S-I-misconception-identification, and I-solution.
  • Extraction: A two-stage extraction pipeline first identifies stated and intended requests, then maps reference answers to the three response dimensions.For Stack Overflow and Stack Exchange, it uses top-ranked and accepted answers, with a minimally rewritten query to mitigate memorization-based recall.
  • Filtering and validation: The construction process filters unsuitable examples using oracle multiple-choice judgments and validates retained data through human annotation.The sampled extraction pipeline achieved 91–95% accuracy against human judgments and 84.0% full correctness across all fields.
  • Evaluation: Evaluation metrics assess which solutions appear, the response’s overall orientation, misconception identification, and fully pragmatic or cooperative behavior.

4 Results

Across XYBENCH sources, models frequently address stated requests but less often prioritize intended goals, identify misconceptions, or generate pragmatic responses. Providing explicit intent improves pragmatic emphasis, yet models generally remain below human performance.

  • Generative evaluation: Models include S-solutions in 0.57–0.91 of responses but I-solutions in only 0.33–0.71, with the latter trailing humans across most sources.Even when mentioned, I-solutions are rarely made the response focus: models emphasize them only 22–41% on SO and SE, versus 52–60% for humans.
  • Misconception identification: Models lag humans by 25–40 points in misconception identification across SO, SE, and the Manual set.Misconception identification increases the likelihood of including an I-solution, but does not guarantee it.
  • Misconception identification: Even after identifying a misconception, models include the I-solution in only 61–85% of responses on SO, SE, and the Manual set, and as few as 36% on WikiHow.This indicates a remaining difficulty in steering responses toward the user’s intended goal.
  • Multiple-choice evaluation: Models choose pragmatic responses in 71–84% of SO, 61–79% of SE, 64–79% of Manual, and 56–83% of WikiHow multiple-choice trials.The contrast between preference and generation suggests models can recognize the more helpful response but struggle to produce it.
  • Oracle experiment: Giving models S-request and I-request at generation time substantially improves I-solution emphasis, up to twice as often, but most models still trail humans.Only Gemini 3.1 Pro closes the human gap, and only on SO and SE; models continue to lag on misconception identification.

5 Conclusion & Discussion

XYBENCH shows that LLMs usually answer literal requests rather than redirecting users toward intended goals, despite recognizing pragmatic responses when explicitly presented with them. The findings identify intent inference, misconception identification, and integrating both into generation as persistent weaknesses.

  • Discussion: Explicitly providing stated and intended requests can reduce misconception identification for Claude Sonnet 4.6 and GPT-5 on Stack Overflow.The authors suggest memorization may lead models to treat these inputs as a resolved dual-answer task rather than reasoning about divergent goals.
  • Main findings: Models overwhelmingly answer literal requests, identify misconceptions infrequently, and trail human experts in redirecting users toward underlying goals.The evaluation covers literal solutions, misconception identification, and pragmatic alternatives grounded in cooperative response theory.
  • Main findings: Models prefer pragmatic responses in multiple-choice evaluation but consistently fail to generate them in ordinary responses.Providing explicit intent or correction instructions improves pragmatic generation, yet model responses still lag behind humans.
  • Discussion: Intent inference and misconception identification remain primary bottlenecks, while instruction-following objectives encourage literal compliance.The authors argue that models must represent users’ underlying goals separately from literal requests and explicitly model communicative intent.

6 Limitations

The benchmark’s limitations concern incomplete human judgment coverage, synthetic and procedure-focused data sources, English-only evaluation, and possible author bias in manual curation. These boundaries restrict how exhaustively the results represent real-world misconceptions across domains, languages, cultures, and expertise levels.

  • Human judgment data: The benchmark lacks human judgment studies for how people interpret misconception-containing queries and lacks human references for synthetic WikiHow queries.The paper identifies variation across expertise levels and cultural perspectives as future work.
  • Domain coverage: Its sources may overrepresent procedure-focused problems compared with misconceptions arising in daily life and natural conversation.The authors note that these sources still provide naturalistic grounding across technical and everyday domains.
  • Synthetic query generation: WikiHow queries are synthetically generated, which may introduce biases or distributional differences from naturally occurring XY problems.The authors mitigate this by grounding generation in literature on misconceptions and propose mining naturally elicited queries in future work.
  • Language coverage: The benchmark evaluates only English-language queries, limiting generalizability across languages and cultures.The paper calls for cross-lingual and cross-cultural evaluation.
  • Manual curation: The manually curated subset may reflect the authors’ intuitions about XY problems and cover only part of the broader misconception space.The paper suggests automatically identifying misconceptions in the wild as a future direction.

B Benchmark Extraction of Stack Overflow and Stack Exchange

The Stack Overflow and Stack Exchange benchmark pipeline uses oracle information and human validation to filter and assess misconception-containing queries. Validation shows substantial annotator agreement and high agreement with GPT-5 filtering, while follow-up evidence indicates that users sometimes accept expert reframing.

  • Human validation: Cohen’s κ = 0.764 and 93.0% agreement with GPT-5 indicate substantial human agreement with the filtering decisions.Two authors independently annotated 100 balanced retained and excluded instances.
  • Human validation: The MCQ filtering validation uses 100 balanced instances and distinguishes strict agreement from lenient agreement that treats borderline judgments as correct.Strict pipeline accuracy counts agree-only judgments.
  • Pipeline validation: Pipeline validation on Stack Overflow samples reports exact annotator agreement of 0.970–1.000 and Krippendorff’s α of 0.687–1.000 across five dimensions.The pipeline achieves 91.0–95.0% strict accuracy against human judgments across individual dimensions.
  • Follow-up behavior: Among 50 manually evaluated follow-ups, 42% show users accepting the reframe and converging on their real goal, while 26% show partial pushback or negotiation.The source reports that 32% of follow-ups were unrelated to the current query.

C Benchmark Construction of WikiHow

The WikiHow benchmark data were generated and repeatedly filtered to make intended requests inferable while removing obvious clues. Human annotation then assessed query and response-quality dimensions.

  • Data generation: The generation prompt produced an initial query, stated and intended requests and solutions, and misconception identification.
  • Filtering and rewriting: Queries were checked for whether experts could infer the intended request and were rewritten when necessary.The identifiability check could be repeated up to three times.
  • Filtering and rewriting: Queries were rewritten to remove obvious clues and add psychologically grounded explanations motivating the stated request.
  • Human validation: Two authors independently annotated 400 generated instances across request capture, plausibility, and the three response criteria.Each dimension was scored as correct, partial, or incorrect.
  • Evaluation setup: The benchmark’s evaluation distinguishes literal and pragmatic responses while checking for misconceptions in human and model answers.

D Data Distribution of Stack Overflow and Stack Exchange

The paper characterizes misconception types in Stack Overflow and Stack Exchange data and describes automated procedures for constructing and filtering WikiHow examples.

  • Misconception taxonomy: Approach Fixation dominates both Stack Overflow and Stack Exchange, reaching 59.2% in Stack Exchange.Missing Knowledge is second most prevalent, while Tool & Syntax Fixation and False Belief are smaller but meaningful categories.
  • Misconception taxonomy: The taxonomy includes Approach Fixation, Missing Knowledge, False Belief, and Tool & Syntax Fixation.
  • WikiHow construction: The WikiHow pipeline generated queries by selecting critical procedural steps and modeling how novices might pursue fixes that miss the underlying problem.
  • WikiHow construction: Automated filtering tested intended-request identifiability, removed obvious clues, and regenerated queries with psychologically grounded motivation for the stated request.
  • WikiHow construction: A separate prompt filtered queries according to the quality of the gap between stated and intended requests.

E.1 Reasoning for Decomposed Metrics for Generative Evaluation of LLM Responses

Because verbose and hedged model responses complicate direct judging, the paper decomposes evaluation into local measures of recommendation presence, emphasis, and misconception identification.

  • Motivation: LLM responses are difficult to evaluate directly because verbosity, alternate suggestions, and hedging can distract even strong judges.
  • Decomposed metrics: The evaluation separately measures what a response recommends, how prominently it recommends it, and whether it flags the misconception.
  • Decomposed metrics: The decomposed setup uses local classification of short extracted units, where LLM judges agree well with humans.
  • Robustness: Human validation across benchmark construction and judge evaluation indicates that pipeline quality is not contingent on Claude Sonnet 4.5.

E.2 Human Validation Setup for Claude Sonnet 4.5

The paper validates Claude Sonnet 4.5 as an evaluation judge using independently annotated responses and agreement comparisons across evaluation dimensions and datasets.

  • Validation setup: Two authors independently annotated 500 responses from Stack Overflow and WikiHow across the three evaluation dimensions.
  • Human agreement: Human annotators reached perfect agreement for solution presence and emphasis, with strong but slightly lower agreement for misconception identification.Misconception-identification agreement ranged from 0.955–0.991 pairwise and 0.910–0.979 by Krippendorff’s α.
  • Judge agreement: Claude Sonnet 4.5 matched human annotations most closely on presence detection, with lower agreement on emphasis and misconception identification.Presence agreement was at least 0.975 pairwise and α at least 0.915.
  • Judge agreement: Human–model agreement was commensurate with human–human agreement, supporting Claude Sonnet 4.5 as a suitable evaluation proxy.
  • Judge robustness: Additional tests compared GPT-4.1 and GPT-5-mini with Claude Sonnet 4.5 and human annotations on WikiHow and Stack Overflow subsets.
  • Misconception and solution relationship: Models provided I-solutions more often when they identified misconceptions, but misconception identification did not guarantee that an I-solution was mentioned.

F.2 Multiple Choice Evaluation Setup

The evaluation presents models with a query and competing literal versus pragmatic responses, then assesses which response they prefer and how generated answers handle stated and intended requests. Models generally prefer pragmatic options in multiple choice, but added prompting produces only partial improvements in misconception identification and intended-solution behavior.

  • Evaluation setup: Models receive a query and two responses, contrasting an S-solution with a response combining misconception identification and an I-solution.Responses are randomly ordered, and the model selects which is most helpful and relevant.
  • Ablations: Providing explicit S- and I-requests improves performance, particularly on WikiHow and SE, but the gap with human answers remains.The oracle experiment reports roughly twice as many I-solutions on WikiHow and nearly four times as many intended-request responses, while manually curated gains are weaker.
  • Discriminative evaluation: Multiple-choice results show models overwhelmingly choose pragmatic responses on Stack Exchange and Stack Overflow, and largely do so on manually curated and WikiHow queries.The figures report pragmatic-response preference across all four source blocks, with stronger wording for Stack Exchange and Stack Overflow.
  • Ablations: A prompt asking models to correct misconceptions improves misconception identification by 6–26 points on WikiHow and leaves models behind humans by up to 13 points on SO and 22 points on SE.The intervention produces only modest I-solution gains of +2–8 points and minimal changes in pragmatic emphasis.
  • Evaluation dimensions: The benchmark evaluates stated-request solutions, intended-request solutions, misconception identification, and response emphasis using annotation frameworks for these dimensions.The referenced figures cover correctness of extracted requests and response dimensions, solution presence, emphasis orientation, and misconception identification.
Loading 2609.06842v1…