Source-linked AI summary
Staged Linguistic Seeding: Grounded Query Expansion for Verified-Unit QA in AI Contact Centers
Hyeonseop Yoon, Jeong-Eun Park
TL;DR
Voice-hotline contact-center QA faces tight latency and high costs for unsupported or wrong automatic answers. The paper uses verified-unit answering and offline staged linguistic seeding, reporting gains across two domains while keeping generation out of serving. The results also show the remaining scope limits of the offline study.
Problem
Production contact-center QA faces tight voice-hotline latency and asymmetric costs for unsupported or wrong automatic answers.
Source: §1 · p. 1Method
Results
Hybrid R@1 reaches 0.881 on ELEC and 0.930 on AUTO, gains of +0.27/+0.34 over a single representative question, with gains across all five retrievers.
Source: §1 · p. 2Takeaways & Limitations
Verified-unit answering bounds responses to approved content and routes weak-evidence cases, while automatically inducing the seeding recipe remains open.
Source: §10 · p. 8The retrieval study covers 319 verified units and is offline without controlled A/B; the faithfulness judge is LLM-based without human adjudication.
Source: §8 · p. 8
Abstract
from arXiv · showhide
Customer-service QA in an AI contact center (AICC) runs under deployment constraints that benchmark QA misses: tight voice-hotline latency and a high cost for unsupported or wrong automatic answers. We deploy a system that answers only from a closed set of verified QA units: it returns a retrieved unit verbatim, or routes to clarify, abstain, or handoff. The index is enriched offline by staged linguistic seeding (SLS): a human authors a per-unit world-grounded slot recipe, gpt-4.1-mini renders it into variants, and a light human gate filters them. One methodology is reused across both domains, so inference stays a single retrieval pass with no query-time generation. On held-out query variants from two industrial domains, SLS lifts hybrid R@1 to 0.881/0.930 (+0.27/+0.34), with gains across all five retrievers tested. At the same gpt-4.1-mini generation budget, SLS beats doc2query by +0.20/+0.32, while cross-provenance evaluation provides additional evidence of transfer across generated-query distributions. Verified-unit answering also removes free-form generation's unsupported-content surface (7-13% versus approximately 0%). We report this as an application study, including negative results.
1 Introduction
Production contact-center QA must balance hotline latency and the high cost of unsupported answers. The paper pairs verified-unit answering with offline linguistic seeding, reporting retrieval gains and a bounded answer surface.
1 Introduction
- Contact-center benchmarks miss voice-hotline latency and asymmetric error costs, while query-time generation adds roughly a second and is impractical on the live line.The deployment goal is to relieve agents while preserving a safe human escape without escalating every ambiguous case.Source: §1 · p. 1
- SLS expands each verified unit offline from a human-authored, world-grounded slot recipe, with model-rendered variants filtered by a light human gate.The methodology and prompt are reused across both domains, leaving serving as a single retrieval pass with zero added serving cost.Source: §1 · pp. 1–2
- The system returns a retrieved verified answer verbatim or routes the query to clarify, abstain, or handoff, avoiding free-form generation at serve time.Unsupported answers are zero by construction, unlike free-form RAG answers judged unsupported in 7–13% of cases on the same evidence.Source: §1 · p. 1
- Hybrid R@1 reaches 0.881 on ELEC and 0.930 on AUTO, gains of +0.27/+0.34 over a single representative question.At matched gpt-4.1-mini budget, SLS beats doc2query by +0.20/+0.32; gains hold across all five retrievers.Source: §1 · p. 2
- Cross-provenance transfer is asymmetric: SLS is 0.146 below doc2query on doc2query-held-out queries, while doc2query is 0.399 below SLS on SLS-held-out queries.The authors treat this control as suggestive rather than volume-controlled because index sizes differ.Source: §1 · p. 2
- On AUTO, SLS lifts the surface-disjoint half of held-out queries by +0.59, with the gain declining monotonically to +0.01 as phrasing overlap approaches full overlap.49% of held-out AUTO queries share zero content words with the canonical question.Source: §1 · p. 2
- The expansion gain and a closed-set leakage pitfall replicate more modestly on public Quora Question Pairs, while the router's action macro-F1 rises from 0.36 to 0.51 under heuristic-label bounds.Neither a larger Qwen3 nor a Korean-specialized fine-tune beats off-the-shelf BGE-M3.Source: §1 · p. 2
2 Related Work
The paper situates verified-unit answering among FAQ retrieval, grounded retrieval, offline query expansion, retriever comparisons, and routing. It adds unit-ID attribution, leakage-controlled held-out queries, and explicit non-answer actions on two-domain enterprise logs.
2 Related Work
- FAQ retrieval answers from a fixed set of verified question–answer units; this work evaluates recovery of the source unit and restricts answers to approved content.The cited related work also includes generated expansion of FAQ entries and constrained or verified-quote answering.Source: §2 · p. 2
- Retrieval and attribution alone do not prevent unsupported claims, so the paper evaluates unit-ID recovery while restricting the answer surface to verified units.The passage notes documented hallucination even when RAG receives correct context.Source: §2 · p. 2
- Prior offline expansion methods can improve retrieval unevenly; SLS instead uses a reusable, world-grounded slot recipe and a light human check, with no serving-time generation.The related-work discussion places this design within doc2query-style enrichment and a longer query-expansion lineage.Source: §2 · pp. 2–3
- The evaluation compares representative lexical, sparse, and dense retrievers under one verified-unit protocol.The passage names BM25, SPLADE, BGE-M3, and dense Qwen3 as spanning the lexical–semantic spectrum.Source: §2 · p. 3
- Unlike model-routing work, this system routes queries out of automation through clarify, abstain, or handoff actions under asymmetric costs.The related-work discussion connects these actions to selective classification, refusal, escalation, and cost-sensitive decision theory.Source: §2 · p. 3
3 Problem Setup
The system selects among answer, clarify, abstain, and handoff actions, with answers tied to retrieved verified units rather than generated strings. This bounds unsupported content but leaves retrieval errors as the remaining answer risk.
3 Problem Setup
- Each indexed unit stores an identifier, representative question, and verified answer; retrieval targets the unit identifier rather than a generated string.The unit is the conceptual grounding boundary, while the stored record is its implementation object.Source: §3 · p. 3
- Free-form generation can produce unsupported claims even with correct evidence, and its overall unsupported rate includes the rate under wrong evidence.The passage attributes this risk to conditional unsupported rates for correct and incorrect retrieval cases.Source: §3 · p. 3
- Verified-unit answering returns the retrieved unit's human-verified text, which remains within the evidence and cannot introduce new claims.This removes unsupported answers by construction, not wrong answers from retrieval of an incorrect unit.Source: §3 · p. 3
- Verified-unit answering removes the open-ended generation-error term and leaves retrieval error, ρ = 1 − R@1, as the measurable failure term.The router declines to answer on the low-confidence part; a wrong retrieved unit can still produce a wrong answer.Source: §3 · p. 3
4 Data Construction
The study constructs a verified-unit retrieval asset from two anonymized enterprise domains and a separate operational log for routing. Retrieval is tested on held-out protocol variants, not raw logged utterances.
4 Data Construction
- The retrieval asset contains 319 verified units—229 ELEC and 90 AUTO—and 7,947 query variants seeded from representative questions and real-log phrasings.Per-unit variant counts track linguistic variability; the retained authoring recipe is available for AUTO, while ELEC retains deployed variants only because of data-retention constraints.Source: §4 · pp. 3–4
- The retrieval study uses the held-out portion of real-log-seeded query variants as a leakage-free test set in both domains; these are protocol variants, not raw logged utterances.Source: §4 · p. 4
- A separate routing set contains 6,000 clustered human queries with heuristic action labels, created by cleaning, deduplicating, clustering, and assigning answerability or routing labels.Retrieval is evaluated on held-out variants, while routing is evaluated on this labeled log.Source: §4 · p. 4
- Enterprise-log governance limits reporting to aggregate statistics, anonymized examples, and unit IDs; raw user text remains access-controlled.The paper treats privacy as part of the system boundary.Source: §4 · p. 4
5 System
The system builds a verified-unit index offline using staged linguistic seeding, then retrieves from that index online. A cost-sensitive router handles queries that should not receive a direct answer.
Offline query expansion: staged linguistic seeding (SLS)
- SLS stores either each unit’s representative question alone or that question plus offline-generated variants.One retriever per family searches the same verified QA units.Source: §5 · p. 4
- For each unit, a human authors world-grounded slots and a light gate keeps or drops the model-rendered candidate strings.The reusable asset is the recipe, not the 7,947 strings; inference remains a single retrieval pass.Source: §5 · p. 4
Cost-sensitive routing (the policy we test)
- The router selects an expected-cost-minimizing action from retrieval-confidence features, with wrong answers penalized more than clarification.The tested policy uses a class-balanced gradient-boosted classifier and is reported as a bounded secondary study.Source: §5 · p. 4
6 Experiments
Experiments test leakage-controlled retrieval over held-out query variants, comparisons with automatic expansion, transfer across query distributions, latency, faithfulness, and routing. SLS improves retrieval across retriever families, while several evaluations remain bounded by their data or labeling conditions.
Metrics and protocol
- The test holds out 30% of each unit’s query variants, never indexes test phrasing, and measures generalization to unseen phrasings from the same per-unit pool.This is in-distribution evaluation, not transfer to an independent query stream; indexing a unit’s own representative question can inflate R@1 up to about 8× for BM25.Source: §6 · pp. 4–5
Staged linguistic seeding is the dominant lever
- Hybrid R@1 rises from 0.609 to 0.881 on ELEC (+0.272) and from 0.588 to 0.930 on AUTO (+0.342), with gains across all five retrievers.The hybrid is best; every gain remains significant under query-level and per-unit cluster bootstrap analyses.Source: Table 1 · p. 5, §6 · p. 5
The lever is the seeding design, not the volume: a matched-budget control
- At matched generator and per-unit budget, SLS beats the best automatic baseline by +0.20/+0.29 hybrid R@1, at zero query-time generation cost.Doc2query N-matched reaches only 0.685/0.611; the comparison attributes the gap to the seeded recipe rather than model or text volume.Source: Table 2 · p. 6, §6 · p. 6
Cross-provenance control: asymmetric transfer across generated-query distributions
- On AUTO cross-provenance queries, SLS is 0.146 below doc2query on doc2query-held-out queries, while doc2query is 0.399 below SLS on SLS-held-out queries.Index sizes differ (16.4 versus 23.5 variants per unit), so the asymmetric transfer is suggestive, not a causal isolation of provenance or volume.Source: §6 · p. 6
Offline expansion adds no serving cost
- Median retrieval latency is 0.7/13/14 ms for BM25/dense BGE-M3/hybrid, while query-time HyDE/query2doc add a 1.0–1.4 s gpt-4.1-mini call.The reported retrieval measurements use one NVIDIA H100, with an Intel Xeon 8480C for BM25.Source: §6 · p. 6
External validity and embedding choice
- On Quora, expansion gains replicate on lexical/hybrid retrieval but dilute on near-ceiling dense retrieval; with expansion BM25 catches up, while larger or fine-tuned models do not beat official BGE-M3.The embedding comparison finds that expansion matters more than the choice among the tested embeddings.Source: §6 · pp. 5–7
Does verified-unit answering reduce hallucination?
- On correct retrievals, free-form answers contain unsupported content in 7.4%–13.2% of cases, versus verified-unit answering’s unsupported-free-by-construction result.The comparison uses the same top-5 evidence; gold coverage is comparable, and the LLM judge had no human adjudication.Source: Table 3 · p. 7, §6 · p. 7
Routing: a bounded secondary result
- Routing raises action macro-F1 from 0.36 to 0.51 (0.45 with retrieval-only features), but four-way results are bounded by subjective labels (κ≤0.09).The classifier recovers rare abstain/handoff actions; features indicate whether, but not why, to escalate.Source: §6 · p. 7
7 Discussion
The discussion explains SLS’s advantage as coverage of meaning-linked reformulations that surface-form methods may miss. Its analysis finds the largest gains on queries with no content words in common with the canonical question.
Why human-authored expansion outperforms learned query generation
- A query about setting GPS to reach a dealership can seek the showroom address despite sharing no content words with “where is the showroom?”The authors frame this surface-form/meaning gap as an instance of symbol grounding: the connection runs through the referent, not lexical overlap.Source: §7 · p. 7
- An operator factorizes each representative question into orthogonal world-grounded slots, and the language model recombines them within that operator-defined manifold.The method supplies grounding through the recipe rather than asking free-form generation to infer the factorization.Source: §7 · p. 7
This account predicts where the gain should land
- On AUTO, 49% of held-out queries share zero content words with the representative question; SLS raises BM25 R@1 on that subset from 0.28 to 0.87 (+0.59).The gain declines with overlap: +0.34 for partially overlapping queries and +0.01 for fully overlapping queries near ceiling; the authors caution this does not show humans beat an LLM generator.Source: §7 · p. 7
8 Limitations
The authors identify boundaries in evaluation, deployment evidence, and method scalability, with human authorship of the grounding recipe remaining the central open problem.
Evaluation scope
- The retrieval and faithfulness studies are offline and lack a controlled A/B test; latency is measured only in a single-query microbenchmark.The retrieval study covers 319 verified units, including 90 AUTO units.Source: §8 · p. 8
- The faithfulness judge is LLM-based without human adjudication, while the four-way router relies on subjective labels and is presented as forward-looking.Routing features bound per-action scores, so only the binary answer-versus-escalate slice is treated as deployable.Source: §8 · p. 8
Method and scale
- The world-grounded recipe remains human-authored, and automatically inducing the factorization is the central open problem.The authors have not run a gate-only ablation or quantified authoring effort; scaling to much larger FAQ sets remains future work.Source: §8 · p. 8
9 Ethics
The paper treats privacy as part of the system boundary and cautions that verified-unit answering is not a guarantee of a correct response.
9 Ethics
- Enterprise logs are reported only through aggregate statistics, anonymized examples, and unit IDs.Raw user text remains access-controlled.Source: §9 · p. 8
- A wrong retrieved unit can still produce a wrong response, so the method is not a truth guarantee.Deployments should log unit IDs, confidence, and action decisions for audit.Source: §9 · p. 8
- The paper frames privacy as part of the system boundary, not merely an appendix concern.Source: §9 · p. 8
10 Conclusion
The conclusion presents verified-unit answering with staged linguistic seeding as the system’s core, while the leakage audit shows why held-out query phrasing matters. The authors also leave automatic induction of the seeding recipe open.
10 Conclusion
- Hybrid R@1 reaches 0.881/0.930 (+0.27/+0.34) across two domains, with the matched-budget control attributing the main gain to seeding design.Cross-provenance evaluation provides suggestive evidence of asymmetric transfer.Source: §10 · p. 8
- Verified-unit answering removes the unsupported-generation surface (approximately 0% versus 7–13%), but automatic induction of the recipe remains open.Source: §10 · p. 8
A The Question-Identity Leakage Audit
- With question fields indexed, BM25/dense/hybrid R@1 falls from 0.863/0.570/0.873 to 0.103/0.222/0.160 when those fields are removed.The held-out-variant protocol avoids this self-retrieval leakage because test phrasing is never indexed.Source: A The Question-Identity Leakage Audit · p. 11
B Routing: Full Results
Routing results show gains in action classification but bound the four-way result; the accompanying analyses cover public-benchmark transfer, corpus differences, the seeding example, checkpoint performance, and matched generation prompts.
B Routing: Full Results
- Under the illustrative 20× wrong-answer cost and answer-only hit@1≈0.13, the cost-minimizing policy auto-answers under 1% of queries at 0% unsafe-answer rate.Its mean cost is 1.10 versus 16.5 for always-answer.Source: B Routing: Full Results · p. 11
- Action macro-F1 rises from 0.36 to 0.51 with class-balanced gradient boosting, while retrieval-only features reach 0.45 and avoid circularity.The four-way result remains bounded by heuristic, subjective labels and features that offer little signal about why to escalate.Source: Table 5 · p. 11, B Routing: Full Results · p. 11
C External Validity: Quora Question Pairs
- On Quora Question Pairs, expansion helps lexical and hybrid retrieval (BM25 +0.124; hybrid +0.053), while near-ceiling dense retrieval declines −0.026.The leakage pitfall also reappears: an indexed phrasing gives R@1 1.000 versus 0.81–0.94 on held-out paraphrases.Source: C External Validity: Quora Question… · pp. 11–12
D Dataset Statistics
- Production handled 74,298 requests with 89.0% combined containment, a deflection measure rather than a correctness guarantee.AUTO and ELEC account for 24,126 and 50,172 requests, respectively.Source: D Dataset Statistics · p. 12
E An Example Verified Unit
- The AUTO example seeds a representative question with location synonyms, inflectional and honorific endings, and spacing variants before generation and human gating.Accepted variants and held-out, never-indexed test queries illustrate the distinction between index enrichment and evaluation phrasing.Source: E An Example Verified Unit · p. 12
F How the Data Explains the Results
- AUTO’s 23.5 variants per unit and shorter answers accompany higher AUG R@1 (0.930) than ELEC’s 15.1 variants and longer answers (0.881).Representative questions average 19–21 characters; the added word forms particularly benefit lexical BM25 retrieval.Source: F How the Data Explains the Results · p. 12
G Domain-Adapted Checkpoint: Training Details and Korean Competence
- A Korean-specialized, persona-fine-tuned checkpoint remains a competent Korean retriever but does not beat off-the-shelf BGE-M3 with offline SLS on the closed customer-service units.The paper characterizes this as a domain-transfer failure rather than a weak-model artifact.Source: G Domain-Adapted Checkpoint: Training… · pp. 12–13
H Baseline Generation Prompts
- Automatic expansion baselines use the same gpt-4.1-mini, Korean-native prompts, and matched contact-center framing, so the SLS advantage is not attributed to language or domain framing.Source: H Baseline Generation Prompts · p. 13