PaperlayerView exact revision on arXiv

Source-linked AI summary

From Retrieval to Customer Context: Evaluating Frontier-Model Systems for Voice-of-Customer Analysis

Raviraja G, Viraj Bagal, Prabhath Chellingi

arXiv:2610.09375v1cs.LG

TL;DR

VoC answers require more than fluent retrieval: they must combine defensible counts, semantic groupings, comparisons, and evidence. The paper compares three system architectures on a shared Cursor corpus and 30 questions, finding the context-graph system scores highest under an analyst-aligned rubric. The reported advantage is descriptive and bounded by rubric calibration and the single-corpus setting.

  • Problem

    VoC analysis must combine denominators, semantic grouping, comparisons, quotations, and limits on inference, while the paper asks whether context architecture or stronger query-time planning better supports these tasks.

    Source: §1 · pp. 1–2

  • Method

    The study compares three complete systems on a shared frozen Cursor corpus and 30 fixed questions, scoring paired answers equally across answer quality, analytical depth, and evidence quality.

    Source: §1 · p. 2, §4 · p. 4, §5 · p. 6

  • Results

    0.961 overall for the Context Graph Agent versus 0.710 for the Deep Research Agent and 0.651 for Agentic RAG; it leads the Deep Research Agent on 27 of 30 paired questions.

    Source: §8 · p. 8

  • Takeaways & Limitations

    Within this frozen configuration, more query-time agency alone does not reliably reconstruct the value of persistent relationships among customer context.

    Source: §8 · p. 8

    The analyst-aligned rubric was adjusted after inspecting 28 of 30 questions, so scores measure agreement with that rubric rather than an independent test.

    Source: §4 · p. 5, §7 · p. 7

Abstract

from arXiv · show

Organizations increasingly use frontier language models to analyze customer feedback, but answer quality also depends on how that feedback is organized and made available. We define a \emph{customer context graph} as a unified model of customer and business context. Typed relationships connect customer objects (feedback, conversations, users, and accounts), operational objects (tickets, support agents, opportunities, and competitors), and analytical or action objects (taxonomy concepts, evidence, insights, work items, and outcomes). This lets an agent investigate not only what customers say, but why, who is affected, what action followed, who owns it, and whether it was resolved. For this experiment, the graph is populated from public Cursor feedback; the same architecture can support any type of feedback source. We compare Agentic RAG, a Deep Research Agent, and a Customer Context Graph-backed Agent on the same 9,432 public Cursor feedback records using 30 realistic product, incident, comparison, and metadata questions. Without exhaustive ground truth, we jointly score responses on answer quality (coverage and organization), analytical depth (specificity and decomposition), and evidence quality (citation support and traceability), using a comparative rubric calibrated on 28 of the 30 questions. We sample cited records against their claims and weight the three dimensions equally. Under this aligned rubric, the Customer Context Graph-backed Agent scores 0.961 overall, versus 0.710 for the Deep Research Agent and 0.651 for Agentic RAG, and leads the Deep Research Agent on 27 of 30 paired questions (sign-test p < 10^(-5); strictly best on 26 of 30). Its largest advantage is analytical depth (0.967 versus 0.642), reflecting more specific, hierarchically developed findings with quantified themes and traceable evidence...

1 Introduction

The paper asks whether organizing customer context before a question improves voice-of-customer analysis beyond stronger query-time retrieval and planning. It introduces a controlled comparison of three system architectures and scopes its claims to one frozen Cursor corpus and an analyst-aligned rubric.

Motivation

  • VoC answers must combine denominators, semantic grouping, comparisons, representative quotations, and limits on what the data establish; retrieved passages alone cannot establish corpus prevalence.The authors argue that fluent but incorrect counts or customer segments can destabilize downstream product, engineering, and support decisions.Source: §1 · p. 1

Research questions

  • The study asks how context architecture affects answer quality, analytical depth, quantification, evidence quality, and restraint on unsupported groupings, and whether query-time planning can close any gap.These questions are designated RQ1–RQ6.Source: §1 · pp. 1–2

Contribution

  • Persistent customer context, rather than additional query-time agency, is the paper’s primary proposed lever for voice-of-customer analysis on this corpus.Supporting contributions include a 30-query benchmark, a common answer-package contract, and paired evaluation across answer quality, analytical depth, and evidence quality.Source: §1 · p. 2

Scope

  • The comparison uses one product corpus and a three-month frozen window, and its scores reflect an analyst-aligned rubric rather than a component-level or capability-equalized comparison.The study also notes that the same base analyst model is dispatched to structurally different roles across candidates.Source: §1 · p. 2

2 Related Work

Prior work spans customer-review analysis, query-time retrieval agents, graph-structured corpus methods, and grounded-answer evaluation. This paper positions its contribution as a complete-system comparison for linked, corpus-level analytical questions.

Voice-of-customer and app-review analysis

  • VoC separates identifying customer needs, structuring them, and estimating their priority, while app-review research covers classification and multi-stage summarization.The paper distinguishes its focus—linked questions requiring exhaustive counts, repeated concept use, evidence reconciliation, and downstream decisions—from classification and one-shot summarization.Source: §2 · p. 2

RAG, agentic retrieval, and deep research

  • Agentic RAG interleaves reasoning with retrieval and tools, while deep-research workflows add delegated specialist investigations, evidence audits, and revision.The paper’s candidates instantiate this ladder as a bounded single-analyst workflow and a decomposed workflow, respectively.Source: §2 · p. 2

Corpus-level and graph-structured RAG

  • Graph-structured approaches shift work to ingestion, but the Context Graph Agent stores typed feedback entities, explicit taxonomy, and durable record-level predictions for reuse across questions.The paper says its benchmark compares this graph-indexed workflow with agentic and deep-research systems on a shared corpus, questions, and answer contract.Source: §2 · p. 2

Theme reliability and grounded-answer evaluation

  • Existing grounded-answer evaluators assume a known answer or labeled QA data, whereas corpus-level analysis combines counts, drill-down citations, theme quantification, and refusal on unsupported groupings.The study instead pairs candidate answers and audits cited records; public-review prevalence is interpreted as describing the frozen corpus, not the full customer population.Source: §2 · pp. 2–3

3 Compared Systems

The benchmark compares three implementations on the same queries and frozen corpus, while preserving each system’s native preparation and question-execution architecture. Their key distinction is whether analysis relies on transient retrieval, query-time orchestration, or persistent customer context.

Comparison setup

  • All three implementations receive the same query text and evidence from the same frozen corpus, but each candidate name refers only to the evaluated implementation.The experiment records versions, prompts, token use, tool calls, failures, corpus and configuration hashes; failed or incomplete runs remain in the denominator without credit.Source: §3 · p. 3

Agentic RAG

  • Agentic RAG uses one analyst with bounded retrieval, inspection, aggregation, comparison, and cited synthesis, without persisting semantic groups across questions.For q013, it compares preceding and recent 30-day periods using a keyword reliability proxy and labels the trend low-confidence.Source: §3 · p. 3

Deep Research Agent

  • The Deep Research Agent delegates qualitative, quantitative, and incident investigations, then verifies and revises the answer; its query-specific assignments do not become a shared ontology.For q013, it uses matched rolling 14-day windows containing all three sources and localizes a GLM 5.2 tool-call cluster to two dates.Source: §3 · p. 3

Context Graph Agent

  • The Context Graph Agent prepares a persistent graph before questions, then traverses relationships to group, count, compare, and cite feedback.For q013, it uses 86 reliability-adjacent themes but declines a trend claim because the preceding calendar window contains only Trustpilot conversations.Source: §3 · p. 3

4 Dataset and Benchmark

The benchmark uses a frozen public-feedback corpus and 30 fixed questions, with shared input and answer requirements but system-specific preparation. Its design specifies denominator and evidence rules while acknowledging limits in semantic ground truth and rubric calibration.

Corpus

  • The corpus contains 9,432 public Cursor feedback records from a frozen three-month UTC window, with stable identifiers, source, timestamp, text, URL, available ratings, and public metadata.Missing account or commercial attributes remain missing, and reported prevalence describes this corpus rather than the full Cursor user base.Source: §4 · p. 3

Benchmark architecture

  • A neutral parsing layer supplies shared feedback and metadata, while candidates retain native preparation and question execution before outputs enter a common answer package and paired evaluation.The common package preserves the evaluated answers for the shared comparison.Source: Figure 1 · p. 4, §4 · p. 4

Information parity and denominator contract

  • All systems receive the same normalized feedback and source metadata, while only the graph system receives its ingestion-time summaries, taxonomies, clusters, and labels.The graph system groups records into 5,035 conversations; quantified claims must give a denominator and cite the record set, with the graph candidate also stating conversation and constituent-record totals when using conversation grain.Source: §4 · pp. 3–4, Figure 1 · p. 4

Query benchmark

  • The benchmark fixes 30 product-adapted questions across seven analysis families and four depth levels before candidate execution, sending identical question text and as-of dates to every system.Hidden family invariants test exact-value reuse, semantic stability, segment summaries, trend direction, sampling restraint, and evidence-linked recommendations.Source: §4 · p. 4, Table 3 · p. 4

Semantic evaluation limits

  • The benchmark lacks exhaustive semantic labels, so the judge scores only records supplied by each candidate and cannot identify false negatives; semantic recall and F1 are not reported.Each query contract specifies operations, permitted fields, count semantics, citation requirements, scope, gold source, and known limitations.Source: §4 · p. 4

Freeze order and calibration

  • The reported rubric was calibrated after inspection of 27 questions and later narrowed after q029, so 28 of 30 questions informed scoring and the results are exploratory rather than confirmatory.Questions q028 and q030 provide only a small internal stability check; q029 is excluded from that check because its rubric was narrowed after its answers were inspected.Source: §4 · pp. 4–5

5 Automatic Evaluation

The evaluation compares anonymized answers for each open-ended query and separately audits citation support. It scores answer quality, analytical depth, and evidence quality, then equally averages the dimensions and queries.

5 Automatic Evaluation

  • Each query is judged across all three anonymized answers, with resolvable citation groups audited separately.Query-level rationales and citation audits are retained; answer order is deterministically shuffled.Source: §5 · p. 5, Figure 3 · p. 5
  • Answer quality rewards question coverage and organization, while analytical depth rewards a hierarchy from conclusions through themes, drivers, counterevidence, and synthesis.The depth rubric penalizes umbrella labels that merge independently actionable issues.Source: §5 · p. 5
  • Evidence quality combines sampled citation precision with comparative evidence strength using a harmonic mean.Up to 15 frozen records are sampled per eligible citation; answers without citations receive zero, and traceable aggregates without exported members are not member-sampled.Source: §5 · p. 6
  • The overall score equally weights the three dimensions and macro-averages query-level scores across 30 queries.Paired wins, ties, and losses use a prespecified 0.01 absolute-difference tie tolerance.Source: §5 · p. 6

6 Results

Across 30 paired queries, the Context Graph Agent leads on every evaluation dimension and on most paired comparisons. Its evidence advantage reflects aggregate drill-down citations, while the authors identify rubric alignment as the principal limitation.

6 Results

  • All three systems completed all 30 questions, and query-level scores are macro-averaged; semantic recall is not reported without an exhaustive relevant-record set.Category means and per-depth slices are descriptive because families contain only 1–5 queries.Source: §6 · p. 6
  • 27 of 30 overall queries favor the Context Graph Agent over the Deep Research Agent, with one tie and two losses (sign-test p = 1.6 × 10−6).It is strictly best on 26 of 30 individual queries.Source: Paired outcomes · p. 6, Table 5 · p. 7
  • The Context Graph Agent’s evidence-quality advantage mostly comes from aggregate drill-down citations, not higher sampled precision.Of its 913 citations, 127 are member-audited and 786 are traceable aggregates without exported members; the score does not independently validate every underlying record.Source: Citation-audit coverage · p. 6, Findings across RQs and interpretation · p. 7
  • The decisive validation remains unperformed: new questions authored after the rubric is frozen and scored without further rubric changes.The internal check covers only two later questions, which share sessions and families with calibration questions.Source: Citation-audit coverage · p. 6
  • Quantification uses different grains: baselines count exclusive classes across records, while the Context Graph Agent counts conversation-level themes with per-theme drill-down.The strongest in-run epistemic-restraint signal is its refusal of a period-over-period trend claim on q013; a pre-registered refusal battery remains future work.Source: Findings across RQs and interpretation · pp. 6–7
  • 0.961 overall for the Context Graph Agent, 0.710 for the Deep Research Agent, and 0.651 for Agentic RAG; analytical-depth scores are 0.967 versus 0.642 for the largest gap.The Context Graph Agent scores highest on every dimension.Source: §6 · p. 6, Table 4 · p. 7
  • Deep Research Agent wins the two bespoke matched-comparison questions, while Agentic RAG wins the App-Store-praise question.The authors describe query-specific computation as able to beat stored concepts when bespoke comparison matters more than concept reuse.Source: Findings across RQs and interpretation · p. 7
  • The Context Graph Agent’s durable summaries, taxonomy predictions, metadata, and relationships support query-time traversal to bounded themes and aggregate-citation quantification.The paper scopes its comparison to one product corpus and one three-month frozen UTC window.Source: Findings across RQs and interpretation · p. 7

7 Limitations and Threats to Validity

The evaluation is limited by rubric calibration on nearly all questions, partial member-level citation audits, and a single-corpus, single-window scope.

Evaluation validity

  • Because 28 of 30 questions informed rubric calibration, scores measure agreement with that rubric, not independent validation; post-freeze questions are needed.The two withheld questions provide only a weak internal check; semantic recall is also absent because unobserved false negatives are unidentified.Source: §7 · pp. 7–8
  • The reported 0.961/0.710/0.651 ordering is presented as exploratory pending confirmatory evaluation on questions authored after rubric freeze.Sign tests rule out sampling noise, not rubric bias.Source: §7 · pp. 7–8

Internal and aggregate-audit validity

  • Only 127 of 913 Context Graph citations were member-sampled; the remaining 786 support traceability, not independent verification of every associated record.Traceable aggregates retain drill-down metadata but were not member-sampled.Source: §7 · p. 8, Table 6 · p. 8

Proof scope, sampling, and generalization

  • Conclusions cover 30 questions, one product corpus, and one three-month frozen UTC window, not universal architectural superiority.The comparison is complete-system, and holding the analyst model constant does not equalize model capability across candidates.Source: §7 · pp. 7–8
  • Self-selected public reviews estimate prevalence within the frozen corpus, not among all Cursor users, and lack enterprise account, contract, and revenue relationships.Public reviews can overrepresent unusually positive or negative experiences.Source: §7 · p. 8
  • App reviews may be short, noisy, multilingual, sarcastic, off-topic, or fraudulent, and workflow scores may not reflect downstream business outcomes.Other stated threats include theme-rubric construct validity, query-time implementation choices, and temporal drift.Source: §7 · p. 8

8 Conclusion

Across the evaluated Cursor questions, the Context Graph Agent leads under the analyst-aligned rubric, while the authors characterize the findings as system-level and descriptive.

8 Conclusion

  • 0.961 overall for the Context Graph Agent versus 0.710 for the Deep Research Agent and 0.651 for Agentic RAG across 30 questions, leading on 27 paired questions.Its largest advantage is analytical depth; the comparison is system-level and descriptive.Source: §8 · p. 8
  • More query-time agency alone does not reliably reconstruct the value of persistent relationships among feedback, summaries, taxonomy predictions, metadata, and available user information.The conclusion is scoped to the frozen configuration and calibrated rubric.Source: §8 · p. 8
  • Next steps include a hybrid design with bespoke matched-window computation, component-level ingestion ablations, and a held-out confirmatory question set scored against a frozen rubric.The hybrid is motivated by q013.Source: §8 · p. 8

Ethics and Data Statement

The statement describes privacy-conscious handling of public feedback, alongside examples that expose both analytical strengths and a stored-taxonomy limitation.

Ethics and Data Statement

  • Released artifacts minimize personal identifiers, avoid cross-source display-name linkage, respect platform terms, and quote only text needed to audit claims.The study does not infer protected, commercial, or account-level attributes from review text, and treats corpus prevalence as distinct from population prevalence.Source: Ethics and Data Statement · p. 10

A Representative Response Excerpts

  • The Context Graph Agent decomposes recurring conversation-level themes and links quantified claims to evidence, while the Deep Research Agent localizes a time-bounded GLM tool-call cluster.The Context Graph Agent refuses a trend conclusion because the preceding window has only 24 Trustpilot-only conversations against a recent window that is 98.4% Reddit.Source: A Representative Response Excerpts · p. 10
  • On q022, a catch-all “general product delight” taxonomy node leaves the Context Graph Agent with tier-two quality and depth, while Agentic RAG develops five specific praise themes.All three systems agree on the denominator, 63 of 113 App Store reviews rated 4–5; the graph-backed answer has perfect evidence quality.Source: A Representative Response Excerpts · p. 10

B Extended Results

The Context Graph Agent’s advantage over query-time agents is substantial, but results vary by depth and question family. The discussion details the evaluation, stored-context workflow, quantification approach, and limits of the comparison.

B Extended Results

  • The Context Graph Agent leads at every benchmark depth, while Deep Research’s clearest advantage over Agentic RAG is at depths 3 and 4.Family means are descriptive for groups of only 1–5 queries; Deep Research is not uniformly second, as Agentic RAG scores higher on pricing/packaging and source/metadata.Source: B Extended Results · p. 10

C Evaluation at a Glance

  • The evaluation anonymizes and jointly presents complete answers, audits record citations separately, and scores answer quality and analytical depth on [0.5, 1] tiers.Evidence quality combines sampled citation precision and comparative evidence strength harmonically; up to 15 member records are sampled per resolvable citation group, while aggregate citations without exported members are not member-audited.Source: C Evaluation at a Glance · p. 10

D Extended Discussion

  • 0.251 overall advantage over the Deep Research Agent exceeds that agent’s 0.059 advantage over Agentic RAG; query-time agency closes only part of the gap.Because the treatment bundles ingestion, summarization, taxonomy induction, metadata mapping, relationship storage, retrieval, and agent behavior, the comparison does not isolate any single component’s causal effect.Source: D Extended Discussion · p. 10
  • The Context Graph Agent reads stored feedback summaries, taxonomy predictions, metadata, available user information, and relationships, then computes counts and comparisons when asked.This makes customer context durable across questions while preserving links from themes to supporting conversations.Source: D Extended Discussion · p. 10
  • The context-graph answers more often attach counts to bounded themes and develop their principal drivers; citation volume alone is not rewarded.The evaluator rewards numerical claims when aggregate drill-down citations and prose distinguish counts, examples, and limitations.Source: D Extended Discussion · pp. 10–11
  • Query-specific computation can outperform stored concepts for bespoke comparisons; public feedback supports only source, rating, country, version, and time slices where those fields exist.The systems cannot support plan, account, revenue, or renewal comparisons from absent metadata; examples include Deep Research wins on autocomplete overview and matched-window reliability, and Agentic RAG wins on App-Store praise.Source: D Extended Discussion · p. 11

E Extended Evaluation Details

  • Query-level evidence precision pools sampled claim–record pairs, so claims with more sampled pairs contribute more; the method does not macro-average by citation.A candidate can receive Pq = 1.0 from three obviously relevant records without a minimum retrieval count per citation; comparative evidence strength separately rewards denominators, drill-down links, and claim-level architecture.Source: E Extended Evaluation Details · p. 11
  • The 0.01 paired tie tolerance is tighter than the approximately 0.083 smallest nonzero overall-score difference, leaving ties nearly always empty; aliases and answer order are shuffled.Structured outputs must include every candidate and sampled claim–record pair or the evaluation fails.Source: E Extended Evaluation Details · p. 11

F Extended Limitations

The study’s conclusions are bounded by its rubric, complete-system comparison, corpus, and frozen implementations. It also documents limitations in construct validity, deployment outcomes, model roles, and audit artifacts.

F Extended Limitations

  • The q013 table presents bespoke query-time reconstruction outperforming stored-context response; on q022, all systems share the denominator of 63 of 113 high-rated App Store reviews, with the gap attributed to decomposition quality.The q013 case is described as a context-graph loss from correctly refusing an unsupported trend claim; q022 is identified as its clearest unforced loss.Source: Table 8 · p. 12, Table 9 · p. 13
  • The study lacks an exhaustive independently labeled reference set, so it cannot measure semantic recall or verify that rankings find every relevant record; final-answer quality remains subjective.It evaluates usefulness and consistency of the resulting customer context, not whether the stored taxonomy is uniquely optimal, fully MECE, or universally reusable.Source: F Extended Limitations · p. 12
  • Workflow scores assess evidence quality, internal consistency, and reviewer judgments of actionability, but do not show that recommendations improve product or business outcomes.The comparison is scoped to a comparative rubric on 30 questions, complete systems, one product corpus, and one frozen three-month UTC window.Source: F Extended Limitations · p. 12

G Model Disclosure

  • Holding the analyst model constant does not equalize capability: Agentic RAG uses one analyst turn budget, Deep Research splits it among five specialist roles, and the graph agent reads precomputed context.The shared provider-family design neutralizes judge self-preference in the reported ranking, but not preference for answer structures correlated with a candidate’s format.Source: G Model Disclosure · p. 12

H Reproducibility Package

  • The reproducibility package provides corpus manifests and record hashes, query and system artifacts, all 90 answer packages with audit records, and scripts to regenerate metrics, tables, and figures.Raw record text is not redistributed in bulk; the paper says this would conflict with platform terms and its Ethics and Data Statement.Source: H Reproducibility Package · p. 13

I Answer Package Contract

  • Member-level citation audits require stable record identifiers; aggregate citations with valid drill-down metadata remain traceable but are not member-sampled unless frozen members are available.Source: I Answer Package Contract · p. 13

J Benchmark Coverage

The 30-query benchmark spans seven analysis families and mixes overview, comparison, diagnosis, and decision questions, including requests that test unsupported grouping.

J Benchmark Coverage

  • Seven analysis families cover overview, comparison, diagnosis, and decision roles, with unsupported grouping represented when required metadata is absent.Examples ask for plan, account, revenue, or renewal comparisons that the public corpus cannot supply; systems receive no hidden signal that refusal is desired.Source: J Benchmark Coverage · p. 14
Loading 2610.09375v1…