Source-linked AI summary
Topic Matching in the Wild: Benchmark and Lessons from Real-World ASR Transcripts
Saman Rahbar, Xiliang Zhu, Irvin Cardoza, David Rossouw
TL;DR
The paper addresses binary topic matching for noisy, spontaneous-speech ASR in real-time contact-center agent assist. It builds a human-annotated benchmark comparing regex, zero-shot embeddings, and Gemini LLMs across keyphrase and natural-language topic representations. Lightweight LLMs reading descriptions achieve the strongest reported performance, while the authors delimit claims to English data from one contact-center domain and note model-availability constraints.
Problem
Real-time agent-assist systems must determine whether noisy ASR utterances match predefined topics, but prior resources do not target this production contact-center setting.
Method
The paper constructs a human-annotated benchmark from real call-center transcripts and compares regex, zero-shot embedding, and Gemini LLM matchers using keyphrase lists or natural-language descriptions.
Results
F1 = 0.847 is achieved by Gemini-3-Flash reading a natural-language description, exceeding the regex baseline at 0.721 and the best embedding at 0.708.
Takeaways & Limitations
Flash-tier LLMs are sufficient for this task, with LLMs favoring natural-language descriptions while embeddings favor keyphrase lists.
Takeaways & Limitations
The data are English from one contact-center agent-assist domain, and the best matcher is a proprietary Gemini-3-Flash preview model whose results are a snapshot.
Abstract
from arXiv · showhide
In contact centers, real-time agent-assist tools determine, for each of many predefined topics, whether a live customer utterance is relevant and display a coaching card to the agent when it is. The input is noisy and challenging: ASR(Automatic Speech Recognition) transcripts of spontaneous phone conversations, which can be unclear, repetitive, and mostly lack punctuation. To systematically study this real-world task, we curate a human-annotated topic-utterance judgments dataset sourced from real call-center transcripts. We compare three types of matchers: a regex-based baseline, zero-shot sentence embedding encoders, and Gemini-based LLM matchers. In addition, two types of topic representations are studied in our benchmark:keyphrases and natural language description. Our empirical experiments highlight the superior performance of lightweight LLM matchers over embedding and regex models when equipped with natural language descriptions.
1 Introduction
The paper frames binary topic–utterance matching as a latency-sensitive problem over noisy production ASR, comparing topic representations and matcher families. It reports that lightweight LLMs, especially when reading natural-language descriptions, outperform regex and embedding approaches.
- Motivation: Real-time agent-assist systems match noisy customer ASR utterances against predefined topics and surface coaching cards when topics are detected.The setting requires processing hundreds of topics simultaneously with low latency.
- Benchmark design: The benchmark compares keyphrase-list and natural-language topic representations using regex, zero-shot embedding, and LLM matchers.Raw ASR transcripts are processed directly to avoid additional latency.
- Data setting: 11.4% of transcripts contain disfluent repetitions and 24.2% contain filler markers, quantifying the noise in the production ASR setting.The dataset is drawn from sensitive customer calls, so the paper releases its evaluation protocol and findings rather than the transcripts.
- Results: 12.6 F1 points is the best LLM improvement over the regex baseline, with p < 10^-14, and LLMs also beat every zero-shot encoder.The strongest models are Flash-tier Gemini variants; heavier models do not outperform them.
- Results: F1 = 0.847 is achieved by an LLM reading a natural-language description, while embeddings perform best with keyphrase lists.The optimal topic representation depends on the matcher family.
2 Related Work
Related work covers noisy user-generated text, ASR robustness, zero-shot semantic matching, and intent detection. This paper differs by evaluating binary topic matching on real contact-center ASR and systematically comparing regex, embeddings, and LLMs.
- Noisy text and normalization: Normalization research addresses non-standard vocabulary and syntax, while speech work removes repetitions, false starts, and self-corrections.This paper matches topics directly rather than normalizing first, leaving normalize-then-match comparisons for future work.
- ASR robustness: ASR-robustness studies show that models trained on clean text degrade on transcription noise, but many benchmarks use read or synthetic speech.The paper instead uses spontaneous customer speech transcribed by call-center ASR.
- Semantic matching: Zero-shot prompting and instruction tuning support LLM-based semantic matching, while zero-shot encoders provide a non-generative alternative.The paper compares these approaches with a regex baseline for noisy ASR topic matching.
- Intent detection: Canonical intent datasets generally contain clean crowdsourced text, whereas deployment-oriented work includes call-center transcripts and live support ASR.The paper studies short-utterance topic matching in the latter setting.
3 Task and Data
The benchmark studies binary topic–utterance matching for latency-sensitive agent assist using real, noisy English call-center ASR. It combines two topic representations with multi-annotator gold labels and explicitly characterizes transcript noise.
- Task formulation: The task maps each finalized ASR utterance and topic definition to a binary fire-or-no-fire decision for a coaching card.Because classes are imbalanced, evaluation foregrounds precision, recall, and F1 over accuracy.
- Topic representations: Topics are represented either by filler-tolerant keyphrase lists or by short natural-language descriptions scored YES/NO by a model.The design tests which representation works best with each matcher and whether descriptions can replace curated keyphrases.
- Data source: The benchmark covers real English customer-service calls from 11 companies and 173 topics, with 3,658 keyphrases across the coaching cards.The keyphrase regex runs over the full configured set rather than a truncated sample.
- Noise characterization: Mean utterance length is 15.8 tokens; 11.4% contain immediate repetitions, 24.2% contain fillers, and the ASR has 14.26% WER.These measurements quantify spontaneous-speech disfluency and recognition noise.
- Annotation and gold standard: 2,655 annotated pairs receive decisive positive or negative gold labels, with approximately 24% positive after ambiguous or non-majority cases are excluded.Three annotators independently choose match, no-match, or ambiguous.
- Annotation and gold standard: Fleiss’ κ = 0.660 across three annotation categories indicates substantial agreement, while ambiguity reflects the difficulty of interpreting disfluent utterances.The consensus-based gold standard and authentic call-center ASR define the paper’s evaluation setting.
4 Methods
The methods benchmark regex, zero-shot embedding, and instruction-tuned LLM matchers on a shared human-labeled set, measuring predictive performance alongside latency and API cost. LLMs use deterministic YES/NO decisions, while embeddings use thresholded cosine similarity.
- Matcher families: Three matcher families are evaluated on the same 2,655-row gold set: regex, zero-shot sentence encoders, and instruction-tuned LLMs.Figure 1 links each family to its topic representation and decision procedure.
- Regex baseline: The regex baseline fires when an utterance matches any keyphrase under a word-boundary pattern with filler and formatting tolerance.It uses the full keyphrase configuration across 173 coaching cards.
- Embedding matchers: Zero-shot encoders embed the utterance and topic, then classify their cosine similarity without fine-tuning.The benchmark includes six widely adopted encoders selected partly by MTEB standing.
- LLM matchers: LLM matching is formulated as a prompted YES/NO decision, with four Gemini models spanning speed and capability at temperature 0.Each model emits a hard YES/NO output used directly for evaluation.
- Evaluation: Precision, recall, and F1 are emphasized over accuracy, with bootstrap confidence intervals, McNemar tests, and adjudicated-subset evaluation.Embedding scores are evaluated at a cross-validated threshold, while hard-output methods use native decisions.
- Efficiency measurement: Latency is measured per topic–utterance decision, and LLM marginal API cost is reported per 1,000 decisions.The comparison excludes deployment-specific serving compute and is not a production service-level objective.
5 Results and Discussion
LLM matchers achieve the strongest F1 on noisy ASR topic matching, especially when reading natural-language descriptions. Lightweight Flash-tier models offer the most practical balance of accuracy, latency, and cost.
- Gemini-3-Flash reaches F1 = 0.847, ahead of the regex baseline (0.721) by 12.6 points and the best embedding (0.708).
- LLM matchers outperform regex and zero-shot encoders, with gains attributed largely to recall on paraphrases and disfluent phrasing.
- Gemini-3-Flash performs better with natural-language descriptions than keyphrases, while embeddings perform better with keyphrase lists.
- The two smallest LLMs lead the board, and no larger model beats them.
- Gemini-2.5-Flash-Lite has p50 latency of 0.35 s and costs $0.010 per 1,000 decisions, approximately 16× faster and 500× cheaper than Gemini-2.5-Pro.
6 Conclusion
The paper benchmarks topic matching on noisy call-center ASR using regex, embeddings, and Gemini LLMs across two topic representations. Lightweight LLMs with natural-language descriptions perform best, while representation choice substantially affects matcher performance.
- The benchmark compares regex, zero-shot embeddings, and Gemini LLM matchers on noisy call-center ASR transcripts.
- Gemini-3-Flash reaches an F1 of 0.847, above the regex baseline (0.721) and the best embedding (≈0.71).
- Lightweight Flash-tier matchers lead, while larger models do not show superior performance.
- LLMs are strongest with written descriptions, whereas embeddings work better with keyphrase lists.
- Because the calls contain sensitive customer data, the paper presents the evaluation protocol and findings rather than the data itself.
Limitations
The study’s claims are bounded by its English, single-domain dataset, its reliance on proprietary Gemini models, and restricted access to sensitive transcripts.
- The data cover only English contact-center agent assist from 11 companies and 173 coaching cards, so the study makes no cross-lingual or cross-domain claims.
- Without a controlled clean-versus-noisy comparison, the accuracy results do not isolate noise robustness from general semantic capability.
- The best matcher, Gemini-3-Flash, is a proprietary preview model whose reported numbers are only a snapshot.
- The underlying sensitive transcripts cannot be released, so reproducibility is supported through protocols, prompts, configurations, and aggregate statistics rather than the data itself.
Ethics and Broader Impact
The study uses sensitive customer–agent dialogues under data-processing and consent procedures, reports only aggregated findings, and uses fairly compensated internal annotators.
- Sensitive customer–agent dialogues are processed under the service’s data-processing and consent procedures without attempting customer or agent re-identification.
- The paper reports aggregate results, model performance scores, and evaluation methodology rather than the underlying data.
- Gold labels were obtained from internal specialized annotators who were compensated fairly.
A Evaluation Prompt
The evaluation prompt asks a matcher to judge whether a customer utterance matches a topic description and requires an exact YES or NO response.
- Each example supplies a topic description and a customer utterance through fixed prompt fields.
- The system instruction defines the task as precise topic–utterance matching and requires exactly one output word: YES or NO.
- The final prompt question explicitly asks whether the utterance matches the topic and repeats the YES-or-NO output constraint.
B Annotation Guidelines
Annotation uses independent judgments of semantic fit, allows ambiguity, and constructs gold labels by majority vote while excluding unresolved cases.
- Up to three annotators independently judge whether each customer utterance fits the topic description.
- Annotators choose match, no-match, or ambiguous, with ambiguity reserved for utterances too underspecified to decide.
- The labels target semantic fit rather than surface keyword overlap, using only the utterance itself without surrounding call context.
- The per-item gold label is the majority vote, while ambiguous-majority and disputed rows are excluded from the usable evaluation set.