Source-linked AI summary
From Gradient-Boosted Trees to Deep Recommenders: Practical Lessons from Migrating a Production Customer Support Recommender
Sonia Sharma, Jeyendran Balakrishnan, Shreya Rajpal, Swapnil Parekh, Nagaraj Janardhana, Andrew Mattarella-Micke
TL;DR
Fast-changing bundled catalogs expose the fixed-label and limited-text assumptions of the production tree-based recommender, while live deployment requires preserving recommendation quality. The paper studies a pairwise-binary deep recommender with structured training, transcript attention, and architecture and inference choices. Across conversation stages, deep models match or surpass CatBoost, with transcript conditioning identified as the central advantage and later-stage gains exceeding call-start performance.
Problem
Dynamic bundles and live transcript signals challenge a multiclass tree model built for a fixed, slower-moving product catalog, in a system where recommendation regressions affect downstream pitch augmentation.
Method
The paper migrates the production system to pairwise-binary deep recommendation using negative sampling, noise injection, transcript attention pooling, architecture comparisons, and inference-time optimizations.
Results
Deep models beat CatBoost at ER and GZ, while CatBoost remains stronger at CL; attention pooling and target attention are the interventions reported to beat CatBoost outright.
Takeaways & Limitations
The case study supports transcript conditioning as the central justification for migration, with other techniques needed to reach that regime without regressing call-start quality.
Takeaways & Limitations
The study has not audited fairness across customer segments or applied explicit bias correction, so historical outcome bias remains an open limitation.
Abstract
from arXiv · showhide
Product catalogs in fast-moving service businesses are shifting from static, independently priced SKUs toward dynamically bundled, discount-coupled offerings--a shift that strains the tree-based classifiers traditionally preferred for sparse and highly imbalanced data. These classifiers assume a fixed, slowly changing label space and struggle to incorporate multimodal signals such as tabular data and transcripts. We present the migration of a live, production conversational recommendation system from a gradient-boosted multiclass model to a pairwise-binary deep recommender. Because this system is critical to ecosystem growth initiatives and downstream features like dynamic pitching--surfacing the most relevant pitch text to a support agent in real time during a live customer conversation--maintaining live recommendation quality was a non-negotiable constraint. We detail the techniques that made this migration successful--reformulating recommendation as pairwise binary prediction to learn jointly from user and item features, and enhancing learned representations via negative sampling and noise injection. To efficiently incorporate long, live conversation context, we apply attention pooling over transcript chunks and benchmark it against TF-IDF and sentence-embedding baselines. Finally, we explore multiple architectures (including two-tower models, DeepFM, and their variants) and loss functions such as contrastive loss. Evaluating against a CatBoost baseline across all conversational stages, we demonstrate that our approach achieves parity at conversation beginning and outperforms at later conversational stages.
1 Introduction
The paper motivates migrating from a per-product multiclass tree model because changing bundle catalogs, live transcript signals, and architectural constraints exceed its design. It presents a production-focused set of techniques for making that migration without regressing recommendation quality.
- Dynamic bundles make the recommendation problem compositional, because bundle identity depends on constituent SKUs and discounts rather than an atomic product.
- A multiclass-over-products model assumes a fixed label set, whereas pairwise-binary prediction treats new bundles or SKUs as item-side feature vectors.
- Deep recommenders can condition on live transcript representations through attention pooling, while the legacy model relies on aggregated tabular features.
- The migration addresses negative sampling, contrastive learning, transcript pooling, architecture choice, and inference-time optimization in a live customer-support system.
- Deep recommenders introduce a larger tuning surface, sparser supervision, and higher per-request inference cost than boosted-tree models.
- Recommendation quality is constrained by downstream pitch augmentation and ecosystem growth initiatives, so experiments are evaluated against the deployed CatBoost baseline at deployment-relevant conversation points.
2 Problem Setup and Evaluation Methodology
The system scores incrementally chunked customer-support conversations and evaluates models at both pair and contact levels. Temporal and contact-level partitioning, shared evaluation, and fixed-population metrics make comparisons with CatBoost directly comparable.
- Conversations are chunked into exactly 20 utterances, with one scoring pass per chunk; Case-Load and Early-Recommendation correspond to chunks 0 and 1.
- Contacts from 2025-01 through 2026-03 are split 75/25 by contact ID, while test uses an out-of-time holdout from 2026-04 through 2026-06.
- Evaluation uses pair-level log-loss, ROC-AUC, and PR-AUC, plus contact-level decisions at defined points in the call.
- All three contact-level metrics are micro-F1 over the same fixed positive-contact population of 16,709 contacts used for CatBoost evaluation.
- The legacy CatBoost baseline has CL 0.4985, ER 0.5310, and GZ 0.5676.
3 Baseline Setup: Negative Sampling and the CatBoost/DeepFM Baseline
The baseline setup standardizes pairwise-binary training and negative definitions across CatBoost and DeepFM. Severe pair-level imbalance makes negative construction central, while the compared text configuration uses shared TF-IDF features with SVD compression.
- Negative selection is the dominant modeling lever because severe pair-level imbalance and invisible non-converted candidates can distort the top product choice.
- 3.1 A taxonomy of negatives: Negatives are classified as explicit or implicit, then as hard or soft according to the strength of the rejection or non-conversion signal.
- 3.1 A taxonomy of negatives: Pure-negative contacts are distinct from per-row negative categories because they contain no conversion anywhere in the contact.
- Contact-level sampling is deliberately balanced, but chunk-level training becomes approximately 4.1% positive because negative contacts generate more chunks and implicit negatives are added to positive chunks.
- At matched K=2, independently sampling implicit-hard negatives per chunk beats sampling one negative set per contact on CL 0.437 vs. 0.407 and ER 0.530 vs. 0.525.
- Both CatBoost and DeepFM use pairwise-binary rows pairing a contact-chunk with a candidate product and labeling each pair match or no-match.
4 Architecture Comparison: DeepFM vs. Two-Tower
The architecture comparison weighs retrieval efficiency against interaction modeling at the current catalog scale. DeepFM is stronger at call start, while two-tower performs better after transcript context accumulates, leading the paper to adopt DeepFM as its base.
- Two-tower models separate user/context and item encoders for retrieval-style scoring, while DeepFM jointly models interactions through factorization-machine, DNN, and wide components.
- Two-tower retrieval serving is advantageous for very large catalogs, but the current nine-product catalog and planned low-hundreds scale remain within DeepFM’s exhaustive-scoring range.
- DeepFM wins at and near call start on CL, whereas two-tower is stronger at later stages ER and GZ.
- DeepFM is adopted as the primary architecture because early recommendation quality matters most under the paper’s non-regression constraint.
5 Negative Sampling: A Thorough Investigation
This section compares structured and random implicit-negative sampling across negative counts, showing that better pairwise ranking does not consistently improve contact-level recommendation metrics. The study therefore adopts K=2–4 as a practical range.
- Experimental setup: The experiments compare a structured rule-based sampler with random sampling at K=2 and K=4.Structured sampling selects implicit negatives through interpretable customer and product patterns rather than uniformly at random.
- Structured versus random sampling: Structured sampling at K=4 raises test PR-AUC from 0.5244 to 0.6293 but lowers CL from 0.4857 to 0.4495.The result illustrates that sharper pairwise discrimination can diverge from the final top-product decision.
- Structured versus random sampling: At K=2, structured sampling improves validation PR-AUC but produces only modest ER/GZ movement and no CL or test-PR-AUC gain.
- Number of implicit negatives: The K sweep finds that the validation–test PR-AUC gap shrinks from 0.19 at K=0 to near zero by K=6–7.The sweep uses a 50% training subsample while holding architecture and training objective fixed.
- Number of implicit negatives: CL, ER, and GZ vary non-monotonically with K and do not track the shrinking validation–test PR-AUC gap.Because contact-level metrics do not improve reliably with larger K, the paper uses K in the range 2–4 thereafter.
6 Contrastive and Binary Cross Entropy Loss Training
This section combines binary cross-entropy with a user–item contrastive objective and controls minibatch composition to address sparse positives and competing negatives. Batch composition independently improves contact-level metrics, while combining it with contrastive learning and informative negatives gives the strongest reported behavior.
- Joint loss design: BCE models the overall conversion distribution, while contrastive learning sharpens discrimination between positive products and competing negatives.The combined objective is L = LBCE + LCL.
- Contrastive objective: The contrastive objective represents each positive product with K implicit negatives and compares contact–product compatibility using cosine similarity.The contact representation masks product-specific inputs, while the product representation comes from product-text embeddings projected into the same latent space.
- Contrastive objective: Pure-negative contacts contribute to BCE but not contrastive loss because they contain no positive anchor.The contrastive temperature is fixed at τ = 1.0 and the contrastive-loss weight is fixed at 1.0.
- Batch composition: A 50:50 mixture of positive contrastive groups and pure-negative units changes sampling-unit composition, not the number of positive and negative BCE labels contributed by each group.At K=2, a positive group contributes one positive and two negative BCE samples plus one contrastive group, whereas a pure-negative unit contributes only to BCE.
- Batch composition: With BCE only, controlling minibatch composition improves ER from 0.5021 to 0.5441 and GZ from 0.4906 to 0.5286.Adding contrastive learning on top of 50:50 batching at K=4 reaches CL 0.4981, ER 0.5510, and GZ 0.5354.
7 Double Descent: Characterizing the Capacity/Regularization Regime
The section examines whether DeepFM’s early overfitting reflects a double-descent regime and finds that smaller, better-regularized configurations generally outperform the 147.8M-parameter baseline. Noise injection improves performance at fixed capacity, while substantially longer training does not recover the observed quality loss.
- Motivation: Validation PR-AUC peaks early at epoch 5 (0.752) before decaying, motivating a test of whether DeepFM’s apparent overfitting is only interpolation onset.The baseline has 147.8M parameters and monotone training-loss decline.
- Experimental design: Four condition arms are evaluated across a nine-point DNN capacity ladder from 648K to tens of millions of parameters, with DeepFM architecture and data held fixed.The grid isolates capacity, regularization, and noise-related mechanisms.
- Results: Nearly every one of 36 configurations beats the DeepFM baseline on test PR-AUC and CL F1 while using far fewer parameters.The result indicates that capacity alone does not explain the baseline’s ranking quality.
- Results: At fixed 17.2M-parameter capacity, noise injection raises CL from 0.4772 to 0.5003 compared with low regularization.This is the only configuration across the full grid that beats CatBoost’s CL.
- Limitations: No configuration clears CatBoost on all three contact metrics simultaneously, so the section reports the migration’s remaining performance boundary explicitly.This limitation is stated alongside the double-descent results rather than rounded away.
- Results: The 200-epoch check shows one early validation PR-AUC peak at epoch 16 (0.7428), followed by decay to 0.2902 at epoch 199, with no dip–rise–dip recovery.At this capacity point, longer training is not a recovery lever; the observed recovery comes from the capacity and regularization search.
8 Conditioning on Transcript Text: Attention Mechanisms Across Architectures
Attention pooling lets deep recommenders condition on relevant parts of a live transcript, with candidate-conditioned two-tower attention and shared-query DeepFM attention improving later-stage recommendations while preserving early-stage quality constraints.
- Motivation: Attention pooling replaces uniform transcript pooling with learned per-chunk relevance weights over per-utterance embeddings.The approach was first explored in two-tower models and then transferred to DeepFM.
- Attention mechanisms: The two-tower model uses candidate-conditioned attention, whereas DeepFM uses one shared query and feeds the pooled vector to its DNN tower.Candidate-conditioned attention allows different products to attend differently to the same conversation.
- Two-tower ablations: +3.3pp Top-1: target attention lifted two-tower Top-1 from 0.5822 to 0.6152 while leaving ER intact.The gain was replicated in two of three independent runs and was attributed to avoiding pooled summaries that blur multiple customer needs.
- Two-tower ablations: Cold-loss levers restored two-tower CL to near-base parity, 0.4546 vs. 0.4575, while retaining attention gains.The best-balance TT-full configuration achieved the campaign’s best ER and GZ.
- Architecture comparison: TT-full beat CatBoost by 2.8pp on ER and 5.2pp on GZ but trailed by 4.4pp on CL, where transcript signal is unavailable.The comparison used the fixed-population CL/ER/GZ evaluator.
- DeepFM results: DeepFM attention pooling beat CatBoost on ER, 0.5630 vs. 0.5310, and GZ, 0.5943 vs. 0.5676, while coming within 1.2pp on CL.This was the paper’s strongest reported result and the first configuration to beat CatBoost outright on ER and GZ.
9 Inference-Time Challenges and Optimizations
Deep recommenders introduce serving costs centered on text featurization and require batching for scalable throughput. Batching lowers cost per request but can increase latency when traffic is insufficient to fill batches, while quantization offers only modest gains with metric trade-offs.
- Serving cost: Text featurization dominates both systems’ end-to-end inference cost, while DeepFM’s model scoring is not the main bottleneck.CatBoost’s model call is 2.4ms, but TF-IDF makes its end-to-end cost 641ms; DeepFM uses transformer embeddings for text.
- Serving cost: DeepFM is faster than CatBoost for a single unbatched request, at 124ms/113ms versus 641ms, but this reverses under production-scale concurrent serving.The single-request comparison is not representative of many-request throughput.
- Batching: Batching is the key throughput lever: CatBoost reaches 9,954 req/s at batch 64, while neural models benefit only when serving code explicitly coalesces requests.Naive per-request threading can reduce throughput beyond four concurrent requests.
- Batching: Batching increases per-request latency because requests wait for a batch or timeout before scoring.The request-coalescing server held requests up to 8ms or until 64 arrived.
- Traffic dependence: At moderate arrival rates, coalescing produced 288–377ms typical latency versus 124ms without batching because batches did not fill quickly.Batching therefore depends on traffic volume, not throughput capacity alone.
- Quantization: INT8 dynamic quantization shrank the model file by 75% and raised batch-64 throughput by 11%, but reduced Early-Reco F1 and Test PR-AUC by about 2 points.Because quality effects were metric-dependent, the paper does not adopt quantization by default.
- Production implications: DeepFM is usable in a live system only with batching, while a smaller or distilled text encoder remains an untested optimization.Text embedding accounts for nearly half of DeepFM’s CPU cost; peak VRAM is 1.4GB at batch 64.
10 Discussion and Conclusion
The migration’s central finding is that transcript conditioning, rather than a single deep architecture, drives later-stage gains over CatBoost. The paper presents this as a production case study whose validated techniques support flexible item representations but do not establish universal superiority or directly measure zero-shot item scoring.
- Central finding: Deep models beat CatBoost only at ER and GZ, where live transcript exists; CatBoost remains stronger at CL before transcript availability.Negative sampling and noise injection narrow the early-stage gap.
- Migration lessons: Attention pooling was the intervention that beat CatBoost outright, making transcript conditioning the paper’s primary justification for migration.This result complements two-tower target attention across a different base architecture.
- Migration lessons: Negative sampling, contrastive learning, and regularization progressively improved the migration, with noise injection moving CL from 0.4772 to 0.5003 at fixed 17.2M-parameter capacity.The contrastive objective remained additive to group batching across contact-level metrics.
- What this unlocks: Pairwise-binary scoring can represent new SKUs and bundles as item-side features instead of fixed output classes, but that capability was not measured directly.The paper identifies direct evaluation of unseen-item scoring as a next step.
- Scope: The work is a production migration case study rather than a benchmark result, and it does not claim a universal winner.Its techniques were evaluated under a live, non-regression-constrained system.
Ethical Considerations
The paper addresses privacy, fairness, misuse, and automation boundaries in a live customer-support recommender. Transcripts are redacted, human agents remain responsible for recommendations, and fairness auditing and bias correction remain open limitations.
- Privacy: PII and other sensitive information are redacted before transcript featurization during both training and inference.
- Fairness: Historical agent and contact outcomes may encode and amplify product-pitching bias across customer segments.The paper reports no fairness audit and no explicit bias-correction step.
- Fairness: Retraining on recent data may track changes in agent behavior but does not remove bias already encoded in historical outcomes.
- Safety and misuse: Real-time discount and bundle recommendations could steer vulnerable customers toward unneeded purchases.The paper identifies this as a plausible misuse path.
- Human oversight: A human agent decides whether and how to raise a recommendation, so the system augments rather than replaces agent judgment.
- Future direction: The paper suggests optimizing for long-term customer lifetime value rather than short-horizon conversion to better align incentives with sustained customer benefit.This is presented as future work, not an evaluated intervention.