PaperlayerView exact revision on arXiv

Source-linked AI summary

You Know What I Mean: A Benchmark for Agentic Conversational Reference Grounding

Karen Fuchs, Uri Katz, Yoav Goldberg

arXiv:2608.29834v1cs.CLcs.IR

TL;DR

CoRG addresses how agents can resolve indirect conversational references to unique external artifacts when the relevant evidence is distributed across conversation and workspace tools. The paper formalizes the task and introduces REPOREF, a GitHub benchmark requiring multi-step search and verification. Current agents remain unreliable: even the best reported system achieves 67.0% success, while failures commonly occur during exploration before the correct artifact is surfaced.

  • Problem

    Collaborative conversations contain indirect references whose intended external artifacts must be inferred from distributed conversational and workspace evidence.

  • Method

    The paper formalizes CoRG and evaluates tool-using agents on REPOREF, a benchmark of developer-chat references grounded in GitHub artifacts.

  • Results

    67.0% accuracy is achieved by the best performing ReAct model, while many stronger-model failures occur before the gold artifact is surfaced.

  • Takeaways & Limitations

    REPOREF provides a concrete setting for studying how agents search, inspect, and verify artifacts through tools in conversational grounding tasks.

  • Takeaways & Limitations

    The benchmark minimally rewrites conversations that originally contained direct GitHub links, which may differ systematically from naturally indirect references.

Abstract

from arXiv · show

Collaborative conversations frequently contain references whose targets are indirect rather than named: resolving "this looks like the fix discussed yesterday" requires combining conversational context with evidence from the surrounding workspace which is accessible through APIs or user interfaces. We formalize this problem as Conversational Reference Grounding (CoRG): using a given set of tools to resolve a reference in conversation to the unique external item intended by the speaker. CoRG is challenging because it combines lexical, semantic, and temporal cues distributed across the conversation and the external workspace. Agents must translate these heterogeneous signals into effective tool use: formulating strategies, discovering plausible candidates, inspecting their metadata and content, and ruling out close alternatives. We study CoRG through RepoRef, a benchmark of 400 developer-chat segments grounded in GitHub issues, pull requests, and commits across 92 repositories. Unlike single-shot retrieval tasks, RepoRef often requires multi-step tool use. Our results show that CoRG remains challenging for current agents, even the best agent reaches only 67.0% success rate, leaving one third of references unresolved. These findings position CoRG as a concrete benchmark for studying how agents search, inspect, and verify information in realistic multi-tool environments.

1 Introduction

Conversational Reference Grounding (CoRG) asks agents to resolve indirect references in collaborative conversations to unique external items using tools. The paper defines this task and introduces REPOREF, a GitHub benchmark for evaluating search, disambiguation, and tool-mediated grounding.

  • Indirect references require combining distributed conversational context with evidence from external systems that change over time.
  • CoRG resolves an underspecified conversational reference to the unique external item intended by the speaker.
  • CoRG challenges agents to use lexical, semantic, temporal, speaker, and artifact-specific cues to distinguish close candidates.
  • REPOREF contains 400 developer-chat conversation segments grounded in GitHub artifacts across 92 repositories.The benchmark includes issues, pull requests, and commits and requires agents to search repositories and return the exact referenced item.
  • The evaluation studies state-of-the-art LLM agents under a shared tool-use protocol, measuring accuracy, cost, and systematic failure modes.

2 The Conversational Reference Grounding Problem

CoRG formalizes reference grounding as tool-mediated identification of one external resource from a conversation and inaccessible environment. Solving it requires adaptive search, candidate inspection, and efficient sequencing of tool calls.

  • 2.1 Formal Definition: CoRG identifies the unique external resource referred to by an anchor message in a multi-participant conversation.The environment and its resources are accessible only through APIs or other specified tools.
  • 2.2 CoRG as a Search Task: The agent must formulate a search strategy from partial information and perform sequential information-seeking tool calls.
  • 2.2 CoRG as a Search Task: Search begins with distributed lexical, semantic, temporal, speaker, artifact-type, content, and metadata cues rather than a direct question.
  • 2.2 CoRG as a Search Task: Agents discover candidates, inspect evidence, reformulate queries, and balance additional search against efficiency.

3 Benchmarking the CoRG Task

RepoRef benchmarks conversational reference grounding by masking direct GitHub references in real developer chats and requiring tool-based recovery of the unique target. Its construction preserves natural context while filtering for identifiable, unambiguous examples and selecting a controlled, diagnostically diverse evaluation set.

  • Benchmark definition: RepoRef contains 400 reference-centered Gitter conversation segments grounded in GitHub issues, pull requests, and commits.The benchmark uses real developer chat and externally verifiable GitHub artifacts.
  • Benchmark definition: Each instance marks an indirect reference whose hidden target must be located using 22 read-only GitHub search tools.Agents are evaluated on identifying the correct resource from the available external environment.
  • Data construction: The benchmark retains naturally occurring conversations with enough evidence to identify one target and distinguish it from plausible alternatives.Construction starts from direct GitHub links, then verifies identifiability and non-ambiguity after masking.
  • Conversation segmentation: Reference-centered segments span the topic-introducing message through the final technical response, preserving context distributed around the anchor.An LLM-assisted segmenter matched human boundary judgments in 96.3% of 50 evaluated cases.
  • Reference masking: Natural masking minimally rewrites only the reference-bearing message while retaining surrounding context and removing the artifact’s identifier, title, and URL.Replacing natural masks with type-only placeholders produced similar success rates of 77% and 75%, respectively, with p = 0.83.
  • Final selection: Filtering reduced 2.6k candidate conversations to 2.0k, and the final benchmark contains 400 cost-feasible, diagnostically diverse instances.An extended set of approximately 2,000 instances is also released for broader evaluation.

4 Experiments

The experiments evaluate tool-using language-model agents on CoRG under a shared GitHub tool protocol, measuring exact-match resolution alongside resource use and search efficiency. A primary fixed-budget comparison is followed by a budget increase for the strongest model.

  • Experimental setup: The primary experiment evaluates ReAct agents with a fixed tool-call budget of B = 10, plus Claude Code with Opus 4.7.The study also increases the budget for the best-performing model in a subsequent experiment.
  • Models: Seven frontier and midtier models span open-weight and closed-source systems, with provider-default settings for the ReAct evaluations.The evaluated models include Claude Sonnet 4.6, DeepSeek-V4-Pro, Gemini variants, GPT-5-mini, Grok-4.1-Fast-Reasoning, and Llama-3.3-70B.
  • Tools: Agents access 22 read-only GitHub tools covering search and inspection of issues, pull requests, commits, branches, tags, releases, labels, and repository files.A submit_answer action returns the final prediction.
  • Metrics: Evaluation uses exact-match GitHub-URL accuracy and reports average tool calls, token consumption, price per example, and excessive tool calls.Average Excessive Tool Calls compares a correct run with the minimal-tool-call correct run for examples solved by at least two models.

5 Results

CoRG remains difficult at the fixed budget: even the strongest model resolves only part of the benchmark, while greater exploration improves Gemini-3-Flash accuracy at substantial cost. Per-instance outcomes also reveal broad variation in difficulty and tool requirements.

  • Overall performance: 67.0% accuracy is achieved by Gemini-3-Flash on the 400-example benchmark, the best performance among the reported ReAct models.DeepSeek-V4-Pro reaches 60.25%, while lightweight Llama-3.3-70B and Gemini-2.5-Flash-Lite reach 1.5% and 4.0%, respectively.
  • Cost: Success rates are not generally correlated with token costs, with the most expensive models ranking middle-low.The passage names GLM-5.1 and Claude Sonnet 4.6 as examples.
  • Cost: 6.27 average excessive tool calls accompany Gemini-3-Flash’s high performance, indicating substantial extra search among solved examples.The model uses many more tool calls than required.
  • Per-instance difficulty: 18.5% of instances are unsolved by all models, and manual inspection found their conversations and GitHub evidence supported the gold artifact.These universally failed cases were therefore treated as genuine task difficulty rather than invalid examples.
  • Tool budget: Gemini-3-Flash accuracy rises from 23.21% at B = 1 to 73.93% at B = 16 on the 280-example budget-sweep subset.Most gains occur by B = 6, where accuracy reaches 63.93%, but further gains require substantial tool and token cost.

6 Analysis

Analysis shows that CoRG failures usually arise before the correct artifact is surfaced, making exploration strategy central to performance. Diagnostic buckets further show difficulty from lexical mismatch, ambiguity, sparse evidence, and commit-level grounding, while efficiency trades off against recall.

  • Exploration behavior: Among stronger models, 70–92% of failures occur before surfacing, while 87–91% of surfaced gold artifacts are selected correctly.The main bottleneck is discovering the correct artifact rather than making the final selection.
  • Exploration behavior: Gemini-3-Flash reaches 75.0% gold discovery by step 10, whereas Claude Code Opus 4.7 reaches 72.0% of examples.Several other agents plateau after only a few tool calls.
  • Tool-call efficiency: Gemini-3-Flash achieves the highest accuracy with 6.27 excess calls, while Grok-4.1 and Claude Code Opus 4.7 average 0.38 and 0.25 excess calls.The models illustrate different trade-offs between candidate recall and tool-call efficiency.
  • Diagnostic difficulty: All four diagnostic buckets challenge current agents: surface mismatch, competitive alternatives, sparse evidence, and commit-level grounding.Surface-mismatch cases remain below 80% success for every model, and sparse-evidence cases remain far from saturated.
  • Diagnostic difficulty: Competitive alternatives reduce performance, while commit-level references are substantially harder to ground than issues or pull requests.The analysis associates commit difficulty with artifact-specific search and verification demands.
  • Overall interpretation: CoRG becomes especially difficult when agents must search the correct artifact space, compare plausible candidates, and verify the intended referent.The analysis frames these operations as extending beyond familiar semantic-matching challenges.

7 Related Work

Prior work studies conversational retrieval, entity linking, repository-level software tasks, and tool-use agents; REPOREF combines these concerns by grounding indirect developer references in GitHub through tools.

  • Conversational retrieval and entity-linking research addresses context-dependent and underspecified language in dialogue.
  • Software-engineering benchmarks evaluate repository-level retrieval, code completion, issue resolution, bug fixing, and agentic development workflows.
  • Tool-use and workplace-agent benchmarks study reasoning and action over APIs, web environments, and simulated workplaces.
  • REPOREF adds conversational grounding to repository navigation, requiring agents to convert indirect developer clues into tool-mediated search and verification.

8 Conclusion

The paper introduces CoRG as tool-mediated resolution of indirect conversational references and presents REPOREF, a GitHub-grounded developer-chat benchmark. Evaluation finds current agents still lack this capability, with failures often arising during exploration rather than only final selection.

  • CoRG requires resolving an indirect conversational reference to a unique external item through tool-mediated search.
  • REPOREF grounds developer-chat segments in real GitHub issues, pull requests, and commits.
  • Current agents remain weak on this capability, often failing to surface, inspect, or verify the correct artifact.
  • The paper identifies higher-recall exploration, better metadata and temporal-cue use, and lightweight candidate verification as promising improvement directions.

Limitations

The benchmark may not fully represent naturally indirect references because it minimally rewrites conversations that originally contained direct GitHub links. Its scope is also limited to open-source GitHub collaboration and a fixed read-only tool setting.

  • REPOREF replaces direct GitHub identifiers in naturally occurring messages with indirect references, which may differ from naturally indirect conversations.
  • The construction retains cases whose intended artifact remains recoverable from surrounding conversational and GitHub evidence.
  • REPOREF covers open-source GitHub collaboration and issues, pull requests, and commits, potentially excluding other workspaces and target types.
  • Evaluation uses a fixed read-only tool environment and limited budget, so results may vary with different tools, retrieval systems, budgets, or agent configurations.

Ethical considerations

The paper uses publicly accessible Gitter data for non-commercial research without identifying or profiling individuals, and illustrates benchmark difficulties through anonymized examples across several diagnostic cases.

  • The benchmark uses naturally occurring Gitter conversations from open-source software communities.
  • The public messages are licensed under Creative Commons BY-NC-SA and used solely for non-commercial research.
  • The authors do not analyze data to identify, profile, or link users to external personal information.
  • Examples are drawn from hard cases spanning seven diagnostic capability buckets, with speakers anonymized.
  • Cross-speaker evidence integration: Cross-speaker cases distribute relevant clues across multiple turns, with no single message uniquely identifying the target.
  • Surface mismatch: Low lexical overlap makes surface keyword matching insufficient when chat language differs from artifact terminology.
  • Author-crowded disambiguation: Author-crowded cases require combining the target author with temporal cues because several recent artifacts may be plausible.
  • Competitive alternatives: Competitive alternatives remain plausible but differ in details that permit recovery of the correct artifact.

E.1 Metadata Cue Candidate Refutation

Metadata cues can refute many incorrect candidate targets, but they do not certify correctness. Date mismatches are the most common refutation, while stronger models tend to make subtler metadata-inconsistent errors.

  • Verifier design: A metadata-cue verifier checks candidate consistency with conversationally inferred target type, author, and temporal constraints.The verifier is refutational: it flags inconsistencies but does not certify that a candidate is correct.
  • Refutation rates: One third to one half of wrong predictions are refuted by metadata cues alone across models.Refutability decreases with model strength: weaker models violate more explicit expectations, while stronger models make softer-cue errors.
  • Implication: Agents often submit targets that remain inconsistent with conversationally implied constraints.The findings motivate lightweight verification as a future direction.

G Rates of zero-tool-call runs and tool-call errors by model

The evaluation tracks no-tool behavior, tool-use errors, and answer composition to characterize how agents fail during RepoRef. It also separates failures to discover the gold artifact from failures to select it after discovery.

  • Run and failure diagnostics: Zero-tool-call runs are measured as submissions made without issuing any tool calls.Table 9 further distinguishes whether unsuccessful runs fail during discovery or selection.
  • Run and failure diagnostics: Discovery failures occur when the gold artifact never appears in the trajectory, whereas selection failures occur after it has been surfaced.The decomposition reports the probability of selecting the correct artifact conditional on surfacing it.
  • Tool-call errors: Invalid tool use, hallucination, and information gain provide trajectory-level diagnostics of agent behavior.Invalid calls fail schema validation, hallucinations produce non-resolving GitHub URLs, and information gain counts calls surfacing previously unseen artifacts.
  • Answer composition: Answer composition distinguishes correct matches, wrong real artifacts, hallucinated URLs, malformed URLs, and runs with no prediction.These categories are reported as percentages of runs at a tool-call budget of 10.
  • Answer composition: Figure 6 reports per-model answer composition over 400 capability examples at a budget of 10.Values are percentages of runs.
Loading 2608.29834v1…