PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support

Cen Mia Zhao, Tiantian Zhang, Hanchen Su, Yufeng Wayne Zhang, Shaowei Su, Mingzhi Xu, Yu Elaine Liu, Wei Han, Jeremy Werner, Claire Na Cheng, Yashar Mehdad

arXiv:2510.06674v2cs.AI

TL;DR

Customer support systems must adapt to changing interactions, but offline batch annotation does not provide the live feedback loop described here. AITL collects four types of annotations in customer operations and feeds them into model updates. Its US-based pilot reports gains across retrieval, generation, citation correctness, and agent adoption, with update cycles reduced from months to weeks.

  • Problem

    Customer-support systems face evolving products, preferences, policies, and procedures, while static models can struggle to adapt to real-world interactions.

    Source: §1 · p. 1

  • Method

    AITL captures pairwise response preferences, agent adoption and rationales, knowledge relevance, and missing knowledge during live operations, then integrates these signals into model updates.

    Source: §1 · p. 1, §6 · p. 7

  • Results

    The US-based pilot reports gains in retrieval recall@75 and precision@8, generation helpfulness, citation correctness, and agent adoption, while reducing update cadence from months to weeks.

    Source: §6 · p. 7

  • Takeaways & Limitations

    The pilot supports embedding human feedback in customer-support operations to continuously refine retrieval, ranking, and generation.

    Source: §6 · p. 7

    The study focused exclusively on English-language customer support, leaving effectiveness in multilingual or culturally diverse contexts open for future research.

    Source: Limitation · p. 8

Abstract

from arXiv · show

We introduce an Agent-in-the-Loop (AITL) framework that implements a continuous data flywheel for iteratively improving an LLM-based customer support system. Unlike standard offline approaches that rely on batch annotations, AITL integrates four key types of annotations directly into live customer operations: (1) pairwise response preferences, (2) agent adoption and rationales, (3) knowledge relevance checks, and (4) identification of missing knowledge. These feedback signals seamlessly feed back into models' updates, reducing retraining cycles from months to weeks. Our production pilot involving US-based customer support agents demonstrated significant improvements in retrieval accuracy (+11.7% recall@75, +14.8% precision@8), generation quality (+8.4% helpfulness) and agent adoption rates (+4.5%). These results underscore the effectiveness of embedding human feedback loops directly into operational workflows to continuously refine LLM-based customer support system.

1 Introduction

Customer support systems must keep pace with changing products, preferences, and policies, but static models can struggle to adapt. The paper introduces live annotations and a continuous learning pipeline to maintain a human-driven data flywheel for customer support.

1 Introduction

  • Static models paired with retrieval lost more than 20 percentage points of accuracy on news questions within a few years in a daily oracle benchmark, motivating continuous feedback for changing support contexts.The introduction notes that support features, user preferences, policies, and procedures evolve.Source: §1 · p. 1
  • AITL captures response preferences, adoption rationales, knowledge relevance, and missing knowledge during live conversations, then integrates the annotations into training datasets.The pipeline reduces model update cycles from months to weeks.Source: §1 · p. 1
  • A US-based pilot reports improvements in retrieval accuracy, response helpfulness, citation correctness, and agent adoption.The authors recommend delaying preference, adoption, and knowledge-relevance annotations, while annotating missing knowledge immediately when SLAs permit.Source: §1 · p. 1

2 Related Work

Prior work covers human preference optimization and data flywheels, but the cited approaches leave gaps in real-time human feedback and domain-specific retrieval. AITL extends these approaches with live human feedback on preferences and knowledge, alongside an LLM-based data-quality judge.

Human-in-the-Loop and Preference Optimization

Data Flywheel and Continuous Learning Pipelines

  • Arena Learning uses simulated LLM self-play and AI judges to generate offline preference labels, but focuses on open-domain dialogue and lacks domain-specific retrieval and human feedback.The authors say it therefore does not directly address real-world preference drift in dynamic environments.Source: §2 · p. 2
  • AITL collects online human preference feedback as well as feedback on knowledge relevance and missing knowledge.The authors contrast this with Arena Learning's offline preference-labeling approach.Source: §2 · p. 2
  • AITL also uses an LLM-based virtual judge to filter data quality and aligns annotation methods with its training pipeline to achieve update cycles comparable to Arena Learning's.The comparison of feedback mechanisms and training pipelines is summarized in Table 1.Source: §2 · p. 2

3 Method

AITL turns customer-support interactions into feedback for retrieval, ranking, and generation: agents annotate live interactions, reviewers check annotation consistency, and a periodic pipeline retrains, evaluates, and redeploys models.

3 Method

  • The workflow presents two candidate responses for agent review, gathers annotations during customer service, and sends them through human and LLM review before reintegrating feedback into model training.The agent can indicate a preference, adoption decision, critique, knowledge relevance, or missing information.Source: §3 · p. 2

3.1 Unified Knowledge Base

  • The Unified Knowledge Base consolidates customer guides, FAQs, policies, workflows, dynamic context, and historical cases with metadata for real-time annotation and retrieval..Source: §3.1 · p. 2

3.2 Agent Annotation

  • Agents compare candidate responses, record adoption rationales, score the relevance of retrieved knowledge, and flag missing information such as policies or unrecorded best practices.Newly identified gaps and references feed back into training to improve retrieval recall and adapt to changing support knowledge.Source: §3.2 · pp. 2–3

3.3 Review Annotation

  • Human and LLM verifiers check annotation consistency against agent-customer interactions, looking for preference mismatches, incorrect relevance judgments, adoption discrepancies, and omitted knowledge.The paper reports strong correlation between human and LLM verifier evaluations.Source: §3.3 · p. 3

3.4 Continuous Learning Pipeline

  • The periodic learning pipeline filters annotations, retrains retrieval, ranking, and generation models, evaluates them, and deploys retrained models back into the RAG system.Filtering combines rule-based thresholds with an LLM virtual judge; evaluation uses curated datasets, ground-truth metrics, and virtual judges.Source: §3.4 · pp. 3–4

4 Experiments

The experiments compared AITL with offline annotation workflows in a US-based production pilot, evaluating annotation quality, model performance, and update-cycle efficiency. AITL annotations improved retrieval, generation, and human preference outcomes over the reported baselines.

4 Experiments

  • The pilot collected annotations from more than 5,000 cases with 40 US-based agents; offline model updates took three months.Agents annotated about 11 cases daily while maintaining productivity comparable to non-participating agents.Source: §4 · p. 4

4.1 Evaluation Metrics

  • Annotation quality used expert and LLM agreement on annotation-response consistency, while model evaluation covered retrieval, helpfulness, citation overlap, and response correctness.Reviewers were blinded to agent labels; retrieval used Recall@75 and Precision@8.Source: §4.1 · p. 4

4.2 Annotation Quality Comparison

  • Online annotation agreement was higher than offline for preference, adoption, and knowledge relevance; missing-knowledge annotation had no offline comparison.The authors attribute the missing-knowledge comparison gap to limits of the offline tool in annotating all potential missing knowledge.Source: §4.2 · pp. 4–5, Table 2 · p. 5

4.3 Impact on Model Performance

  • Recall@75 rose from 0.634 to 0.708 (+11.7%), and Precision@8 from 0.357 to 0.410 (+14.8%), both exceeding offline annotations.The gains over offline were 3.8% for Recall@75 and 4.1% for Precision@8.Source: §4.3 · p. 5
  • Helpfulness rose from 0.658 to 0.713 (+8.4%), citation accuracy from 0.097 to 0.134 (+38.1%), and response correctness from 0.851 to 0.882 (+3.6%).Each AITL fine-tuning result exceeded its stated offline value.Source: §4.3 · p. 5
  • 60.12% of fine-tuned responses were preferred over baseline (33.32%), and overall agent adoption increased by 4.5%.Figure 7 presents the human preference comparison.Source: §4.3 · p. 5, Figure 7 · p. 5

5 Learnings

The learning studies identify when annotation should be immediate, how LLM filtering affects data quality, and how preference data and training choices affect model outcomes. Results also report transfer across model architectures and scales.

5.1 Annotation Timing Ablation Study

  • Immediate annotation raised Missing-Knowledge agreement from 63.9% to 76.5% (+12 pp, p < 0.05); differences for Steps 1–3 were negligible.The roughly 2,000-case comparison motivates immediate Step 4 annotation when SLA allows brief delays, while delaying other steps.Source: §5.1 · p. 5

5.2 Annotation Quality Ablation Study with LLM-based filter

  • LLM filtering improved Recall@75 by +3.8 pp and Citation by +1.9 pp, while Precision@8, Helpfulness, and Response Correctness remained statistically unchanged.LLM and human evaluations correlated at r > 0.90; the authors position filtering as validation, not generation, for nuanced missing-knowledge labels.Source: §5.2 · p. 6

5.3 Pairwise Preference Data Effectiveness.

  • Adding agent adoption as a filter restored citation correctness to 11.4% while retaining a 3.5% helpfulness gain from plus-level preference data.Source: §5.3 · p. 7

5.4 Continuous Training Strategies.

  • Mixing historical and new annotations increased Precision@8 by 8% on historical data and 4% on recent data versus training only on new data.The paper reports similar benefits for offline annotation approaches.Source: §5.4 · p. 7

5.5 Cross-Model Generalization Study

  • AITL gains were consistent on smaller and medium models; at 70B, helpfulness effects were mixed while citation was improved or stable across variants.The results use the same data split and evaluator as the main study.Source: §5.5 · p. 7

6 Conclusion and Future Work

The conclusion presents AITL as a near-real-time data flywheel that turns customer-support operations into supervision for retrieval, ranking, and generation. The pilot reports gains across model and agent outcomes while reducing update cadence from months to weeks.

6 Conclusion and Future Work

  • AITL captures pairwise preferences, agent adoption and rationales, knowledge relevance, and missing knowledge during support operations.The resulting supervision targets retrieval, ranking, and generation.Source: §6 · p. 7
  • The US-based pilot reported gains in retrieval, generation helpfulness, citation correctness, and agent adoption, while shrinking update cadence from months to weeks.Source: §6 · p. 7
  • Future work proposes lighter optional feedback with active sampling, product-embedded evaluation, and greater dataset-curation automation while preserving human oversight.The proposed evaluation includes productivity measures and study of cognitive load, trust calibration, and skill formation.Source: §6 · p. 7

Limitation

The paper identifies annotation fatigue, English-only support, and short study duration as limitations. Its appendix describes annotation errors and an LLM verifier prompt for checking whether annotated knowledge addresses a customer issue.

Limitation

  • Prolonged real-time annotation may increase agent workload and cause annotation fatigue; rotating duties, managing workloads, breaks, training, and incentives are proposed mitigations.The paper presents these as risks and possible mitigation strategies, not measured outcomes.Source: Limitation · p. 7
  • The study covers only English-language support, leaving AITL’s effectiveness in multilingual or culturally diverse contexts open for future study.Source: Limitation · p. 8
  • The study’s short duration limits understanding of how annotation practices evolve over time and scale to larger agent groups.Source: Limitation · p. 8

A Annotation Error Types and LLM Verifier Prompt

B Helpfulness Metric Definition

The section defines point-wise helpfulness as an estimate of how likely internal domain experts are to prefer a response. It combines a human-preference-trained reference evaluator with GPT-4 prompt evaluations.

B Helpfulness Metric Definition

  • Point-wise helpfulness estimates the likelihood that internal domain experts would prefer a particular response.Source: B Helpfulness Metric Definition · p. 10
  • The Mistral-7B reference evaluator, fine-tuned on human preference data with a pairwise loss, agrees with expert ground-truth annotations at approximately 80%.Source: B Helpfulness Metric Definition · p. 10
  • Seven GPT-4 prompts yield binary helpfulness indicators summed to a 0–7 score and averaged; this approach needs no inference-time ground-truth labels and agrees with expert labels at approximately 80%.Source: B Helpfulness Metric Definition · p. 10

C Retrieval and Ranking Metrics

The section motivates Recall@75 and Precision@8 as retrieval and ranking metrics based on observed retrieval behavior and production context limits. It also describes the baseline RAG generation, retrieval, and ranking models.

C Retrieval and Ranking Metrics

  • Recall@75 and Precision@8 were selected as primary metrics; recall reached approximately 70%–80% at topN=75 before gains plateaued, while topN=8 balanced ranking performance and context limits.More than eight retrieved snippets reduced helpfulness through information dilution.Source: C Retrieval and Ranking Metrics · p. 10

D Prior AITL Baseline Details

  • The baseline generator is an 8×7B Mistral MoE model, first supervised-fine-tuned on offline annotations and then aligned with agent-approved preferences using ORPO.Source: D Prior AITL Baseline Details · p. 10
  • The baseline retrieval model fine-tunes Zeta-Alpha-E5-Mistral 7b embeddings and retrieves top-N article chunks by cosine similarity; an in-house FLAN-T5 model then ranks them by relevance score.Source: D Prior AITL Baseline Details · p. 10

E AITL Training Details

AITL training uses agent annotations to fine-tune generation, retrieval, and ranking components. In live-support RAG, AITL-trained retrieval and ranking models outperform models trained on offline datasets, with results also identifying preferred training-data mixtures.

E.1 Generation Model

  • Generation fine-tuning starts from the 8×7B Mistral MoE baseline and uses ORPO with equal parts newly annotated and prior training examples, followed by QLoRA and merging LoRA weights back into the model.Training used three epochs and a batch size of 64.Source: §E.1 · pp. 10–11
  • ORPO fine-tuning often yields higher coherence and factual correctness than SFT alone; slight divergences between offline and AITL online preference data motivate continuous updates.Source: §E.1 · p. 11

E.2 Retrieval Model

  • Retrieval fine-tuning uses MultipleNegativesRankingLoss with relevant or helpful chunks and agent-generated content as positives, and labeled negatives plus same-batch random chunks as negatives.Source: §E.2 · p. 11
  • Retrieval accuracy benefited from larger training datasets and showed resilience to noise, with optimal performance from combined high- and low-confidence data.Source: §E.2 · p. 11
  • AITL-trained retrieval models substantially outperform baseline and offline-data models in live support RAG, while AITL-trained ranking models outperform offline-data and historical-only models.Source: §E.2 · p. 11, §E.3 · p. 12

E.3 Ranking Model

  • The ranking model is trained with agent-labeled relevant/helpful and not-relevant/not-helpful chunks; its relevance score is the probability of the output being “relevant.”Source: §E.3 · p. 11
  • The ranking model was trained for three epochs on one A10 GPU, using a learning rate of 1e −5, batch size 16, and gradient accumulation of 64 steps.Source: §E.3 · p. 11
  • Ranking performance was optimized on high-confidence annotations mixed at a 1:2 ratio of historical to newly generated AITL data.Source: §E.3 · p. 12

F Training Experiment Efficiency

GLOW combines reusable offline experiment components with infrastructure-aware and templated workflow design; the comparison section reports a hybrid metric and uses median annotation time amid right-skewed times.

F Training Experiment Efficiency

  • GLOW provides reusable, parameterized fine-tuning, batch-scoring, and evaluation components, with infra-aware LLM development and templated end-to-end workflows as its two main focuses.Its unified API is described as improving production readiness and reducing prototype-to-production cycles.Source: F Training Experiment Efficiency · p. 12
  • GLOW pairs each LLM task with an on-demand Ray cluster configured to avoid insufficient GPU memory or low utilization and support stable, reproducible runs.The configuration addresses sensitivity to infrastructure and task hyperparameters, including compute strategy, dataset and batch size, context length, model capacity, and architecture.Source: F Training Experiment Efficiency · p. 12

G Immediate and Delayed Comparison

  • Human and LLM scores are averaged into a hybrid metric; their strong correlation (r > 0.90) is reported to support human-level reliability at lower scaling cost.Subsequent analyses, including the immediate-versus-deferred comparison, use this metric.Source: G Immediate and Delayed Comparison · p. 12
  • Table 9 annotation times are right-skewed, with medians of 1–2 min, so the authors use the median as the best indicator of agent effort and the trimmed mean as a reference.The table reports annotation-time statistics per step in minutes.Source: G Immediate and Delayed Comparison · p. 12, Table 9 · p. 12
Loading 2510.06674v2…