PaperlayerView exact revision on arXiv

Source-linked AI summary

MERIT: Mitigating Exposure Bias in Generative XMC for User-Interest Propensity Modeling

Abhinav Mahajan, Arindam Sarkar, Prakash Mandayam Comar

arXiv:2608.28931v1cs.IRcs.LG

TL;DR

Large-scale user-interest propensity modeling suffers from exposure bias, as early autoregressive errors steer later predictions toward correlated labels. MERIT trains self-correction with mixed gold and hard-negative prefixes while retaining teacher-forcing efficiency, improving retrieval metrics and production conversion.

  • Problem

    Exposure bias in generative extreme multi-label classification can cause co-occurrence cascades that reduce recall of unrelated user interests at large label scale.

  • Method

    MERIT uses a permutation-invariant multi-target loss over shuffled gold and mined-negative prefixes, training recovery from incorrect labels in a single teacher-forced pass.

  • Results

    MERIT improves recall by at least 11.9%, average Hit@k by at least 6.1%, and production conversion by +0.26%.

  • Takeaways & Limitations

    The resulting propensity-aligned representations support a lightweight scorer for retrieval and offline customer segmentation at production scale.

  • Takeaways & Limitations

    MERIT intentionally trades some precision for recall, so its noisier generated sets may require downstream filtering in precision-sensitive applications.

Abstract

from arXiv · show

Matching users to interest categories at scale is central to personalized shopping, but the task is challenging in large e-commerce platforms, where label spaces continually evolve and user-interest signals are sparse and long-tailed. Autoregressive language models are appealing because their world knowledge and semantic priors over descriptors generalize across extreme label spaces and accommodate multiple valid label assignments. Yet under teacher-forced fine-tuning, inference-time predictions become part of the conditioning context: early errors steer later outputs toward co-occurring labels, over-generating near-correlates and missing unrelated true interests. We present MERIT, a framework for user-interest propensity modeling that mitigates this exposure bias through a self-correction objective. A permutation-invariant multi-target loss over shuffled mixtures of gold and mined hard-negative labels exposes the generator to erroneous prefixes while preserving the efficiency of teacher-forced training. This training objective concentrates supervision at classification positions, yielding propensity-aligned hidden states powering a lightweight scorer for bidirectional retrieval (interests for users and users for interests). On a proprietary e-commerce dataset with 250k+ interest categories, MERIT improves global recall by at least 11.9% and average Hit@k by 6.1%. In production A/B tests, it achieves +0.26% gain in user conversion.

1 Introduction

The paper frames user-interest propensity modeling as extreme multi-label retrieval over dynamic, long-tailed label spaces. It identifies exposure bias as a recall failure: erroneous early labels amplify co-occurrence and suppress unrelated interests.

  • Motivation: Dynamic products and campaigns make candidate-level propensity learning brittle, motivating stable semantic interest descriptors as the prediction space.Descriptors generalize to newly launched candidates through their associated tags.
  • Problem: The task predicts sparse descriptor sets from a large vocabulary while supporting both interest-for-user and user-for-interest retrieval.Its extreme multi-label setting includes long-tailed labels, sparse positives, and ambiguous noninteractions.
  • Generative modeling: Generative language models fit this setting because textual descriptors, semantic priors, and structured multi-label outputs support generalization across extreme label spaces.Descriptor co-occurrence provides useful structure but must not override unrelated interests supported by user history.
  • Exposure bias: Teacher forcing creates an exposure mismatch: inference conditions on model outputs, allowing early semantic commitments to trigger co-occurrence cascades that reduce recall.The resulting outputs overproduce near-correlates while missing unrelated true interests.
  • Contribution: MERIT addresses this failure with a permutation-invariant multi-target loss over mixed gold and mined prefixes, improving recall by at least 11.9%.The objective trains recovery from erroneous prefixes within a standard teacher-forcing pass without per-step generation.

2 Background & Related Work

Related work spans user-interest models, extreme multi-label retrieval, generative XMC, and exposure-bias mitigation. MERIT combines these lines by applying self-correction to multi-label generation while retaining interpretable text descriptors and exploiting generator errors as negatives.

  • User-interest modeling and XMC: Unlike candidate-scoring user-interest models, XMC predicts descriptor sets over extreme vocabularies to provide a retrieval interface.The interface supports retrieval in both user-to-interest and interest-to-user directions.
  • User-interest modeling and XMC: Dual encoders generalize to new labels but depend on informative hard negatives, which are costly to mine at extreme scale.MERIT uses the generator’s confident errors as hard negatives without separate expensive mining.
  • Generative XMC: Generative XMC preserves textual interpretability, whereas Semantic IDs improve latency at the cost of descriptor interpretability; prompted LLM recommendation lacks calibrated extreme-scale propensities.MERIT therefore retains text-level descriptors while using the LLM within a propensity-modeling pipeline.
  • Exposure bias: Prior exposure-bias methods mainly target fluent-generation quality, while MERIT studies early-label errors as co-occurrence cascades that reduce multi-label coverage.Its self-correction approach targets recall in multi-label generation rather than sequence fluency alone.

3 Methodology

MERIT combines an exposure-mitigated generator with a lightweight propensity scorer. It trains recovery from erroneous prefixes in one teacher-forced pass, then reuses classification-position states and generator errors for bidirectional retrieval.

  • Task formulation: The task maps a user’s year-long interaction history to a sparse tag set from an extremely large interest-category space, excluding the final seven days before purchase.The exclusion window prevents trivial prediction from immediate pre-purchase signals and matches precomputed production scoring.
  • Pipeline: The pipeline tokenizes and adaptively compresses user history, generates interest categories, and extracts early category hidden states for propensity scoring.Adaptive patching preserves recent interactions at full resolution while compressing older windows.
  • Propensity model: Because generation alone lacks scores comparable across users, MERIT trains a lightweight propensity model on exposure-mitigated generator states and hard negatives.This enables ranking interests for users and users for interests.
  • Hard-negative mining: Offline temperature sampling mines plausible incorrect labels, which are filtered by similarity before augmenting the gold set as hard negatives.The offline pass is cheaper than sampling at every training step, and filtering reduces near-synonym false negatives.
  • Exposure-mitigated teacher forcing: A shuffled mixture of gold labels and hard negatives is scored in one forward pass, while a prefix-dependent multi-target loss exposes erroneous prefixes without rewarding negatives.The valid target set pools first tokens of unfinished gold labels, making the objective permutation-invariant; negative prefixes are completed coherently.
  • Representation and scoring: Supervision concentrates at classification positions, producing prefix-independent propensity-aligned states and informative negatives for the downstream scorer.The first-token states of generated categories feed a smaller transformer and cached MLP scoring pipeline.

4 Experiments

Experiments evaluate MERIT on a 291k-category proprietary e-commerce dataset across generative classification, propensity ranking, and production campaigns. MERIT improves recall and Hit@k, with a deliberate precision trade-off, and yields positive conversion lift online.

  • Setup: The evaluation uses 500k training users, 100k disjoint test users, and 291k interest categories, covering generative classification, propensity ranking, and A/B testing.The setup uses a frozen 3.8B phi-3.5-mini generator and a 230M propensity encoder over five generated categories.
  • Generative classification: MERIT improves Global recall by +14.1 over Trivial and by at least +11.9 over the strongest baseline.The reported comparison is an absolute improvement in generative classification recall.
  • Precision-recall trade-off: The method accepts lower precision in some verticals because recall is the operative upstream retrieval metric and missed interests cannot be recovered downstream.GROOV outperforms MERIT on Apparel F1/precision despite MERIT’s 3.5× higher recall.
  • Ablation: MERIT’s recall gain is not explained by output diversity alone: it exceeds four independently sampled Standard Teacher Forcing outputs by +6.75 recall and +1.76 precision.The ablation attributes the advantage to recovery from erroneous prefixes.
  • Propensity ranking: MERIT improves average Hit@k by +6.1 over DEXML-DS with a smaller 230M encoder, while producing scores comparable across users for bidirectional retrieval.The authors associate the gain with less-collapsed representations and mined generator negatives.
  • Production A/B test: Across three one-week campaigns, conversion lift was +0.23% in Personal Care, +0.19% in Toys, and +0.37% in Sports, averaging +0.26%.All campaigns met the internal shipping threshold of P(lift > 0) ≥0.90.

5 Conclusion

MERIT is an efficient exposure-mitigated framework for generative extreme multi-label classification in user-interest retrieval. It uses single-pass self-correction to improve robustness and supports downstream retrieval and production applications.

  • MERIT teaches generators to recover from erroneous prefixes without the per-step cost of reinforcement learning or scheduled sampling.
  • The framework yields propensity-aligned embeddings that power a lightweight scorer for offline customer segmentation.
  • MERIT outperforms strong generative and dual-encoder baselines and delivers lift in production settings.

6 Limitations

MERIT’s reported benefits are bounded by recall-oriented noise, proprietary single-dataset evaluation, next-purchase targets, and offline deployment. These constraints limit precision-sensitive use, public benchmark comparison, propensity stability, and real-time scoring.

  • MERIT trades precision for recall by design, so its noisier generated sets may require downstream filtering in precision-sensitive applications.
  • The evaluation uses one proprietary e-commerce dataset and reports relative rather than absolute metrics, limiting direct comparison with public XMC benchmarks.
  • Because gold labels represent only a user’s immediate next purchase, mined negatives can occasionally be unobserved true interests.
  • Training and evaluation over multiple future purchases may yield more stable propensities, but this extension remains future work.
  • Deployment is two-stage and offline, supporting segmentation and merchandising while leaving real-time on-demand scoring for future analysis because of its cost implications.

Ethical Considerations

MERIT is trained and used within boundaries intended to reduce exposure of sensitive information and constrain the effect of noisy predictions.

  • MERIT uses anonymized interaction logs and derives interest descriptors from product metadata rather than personal or demographic attributes.
  • The system does not model or predict sensitive attributes such as health, financial, or demographic status as interest categories.
  • Predicted propensities pass through business-rule and eligibility filtering before influencing customer-facing surfaces.

A Detailed Walkthrough of Exposure Mitigation Training

MERIT trains a generator to recover from incorrect prefixes while preserving valid multi-label orderings and coherent hard-negative completions. Supervision concentrates at classification positions, encouraging prefix-independent interest predictions.

  • Single-pass training: MERIT processes a shuffled target sequence in one forward pass, computing position-wise losses without autoregressively generating tokens during training.The method remains close to teacher forcing while modifying which target-token probabilities are penalized.
  • Classification positions: At classification positions, the loss rewards first tokens of all uncompleted golden labels rather than the incorrect token appearing in the shuffled sequence.This trains predictions from user context even when the consumed prefix contains an error.
  • Hard-negative completion: When an incorrect prefix matches a hard negative, MERIT enforces completion of that label instead of allowing the model to switch mid-label.This preserves label coherence and prevents malformed outputs after an error.
  • Permutation invariance: Permutation-invariant loss pools probability over multiple valid continuations, so arbitrary label ordering is not penalized.For example, a prefix such as “Sports” can be completed as either “Sports Enthusiast” or “Sports Gear” when both remain valid.
  • Exposure mitigation: Classification positions after correct and incorrect labels teach independent decisions, making the resulting hidden states propensity-aligned and largely prefix-independent.Completion losses decline as vocabulary is learned, while classification losses remain challenging across 250k+ candidates; this concentrates learning signal where exposure bias must be suppressed.

B Threshold Sensitivity and Human Agreement

The paper tests whether MERIT’s recall advantage is robust to soft-matching thresholds and validates the τ=0.7 boundary with targeted human judgments. It also finds that the propensity-model setting kg is not fragile across tested values.

  • Threshold sensitivity: MERIT’s recall advantage holds across τ=0.70, 0.80, 0.90, and normalized Exact Match, while method rankings remain unchanged.Some baselines exceed MERIT in precision at stricter thresholds, but recall is the operating metric because downstream filtering is cheap and missed candidates are irrecoverable.
  • Human agreement: 96.2% of pairs accepted at τ=0.7 were judged at least loosely equivalent, and 65.4% were judged strictly equivalent by human annotators.The 230 boundary pairs were sampled around the decision threshold; disagreements concentrated on near-synonymy.
  • Sensitivity to kg: Every tested kg setting beat both DEXML baselines in 89–94% of 100 vertical–cutoff cells, indicating that the propensity-model choice is not fragile.The paper uses kg=5; kg=3 achieved +15.0 mean Hit@k improvement over DEXML-STk, while kg=5 achieved +13.2.

E Full Per-Vertical Propensity-Ranking Results

Across 20 product verticals and Hit@k thresholds, MERIT consistently improves propensity ranking over DEXML baselines, while adaptive history compression adds a smaller recall benefit.

  • Per-vertical results: +9.0 Hit@5k points over DEXML-STk and +4.3 over DEXML-DS averaged across 20 verticals.The comparison covers all 20 verticals and reports both baselines relative to DEXML-STk.
  • Online experiments: All three campaigns clear the prespecified P(lift > 0) ≥0.90 shipping threshold, although only Campaign 3 has a credible interval entirely above zero.Two campaigns have 95% credible intervals overlapping zero, so the online evidence is strongest for Campaign 3.
  • Sequence compression: Adaptive patching the full year of history into an 8k context budget adds +0.5 global recall over using only the 4k most-recent interactions.The ablation indicates that additional compressed history helps without degrading the retained recent signal.

H Implementation Details and Hyperparameters

MERIT combines hierarchical generation, frozen-generator hidden states, hard-negative mining, and lightweight propensity scoring, with specified compression, optimization, and evaluation procedures.

  • Generator: The generator uses phi-3.5-mini (3.8B) and hierarchically predicts 30k semantic clusters before 291k leaf categories.The hierarchy manages the extreme label space during generation.
  • Propensity model: The 230M propensity model uses a 4-layer transformer encoder, a 3-layer MLP, and first-token hidden states extracted from the frozen generator.Category representations use pretrained Cohere embeddings, and the MLP outputs propensity scores from concatenated user-category representations.
  • Optimization and evaluation: Training uses AdamW at 2×10^-5 with batch size 32 and fp16 on 8×A100 GPUs; generator and propensity training take approximately 48h and 3h.The reported evaluation includes soft-matching generative metrics and Hit@k propensity ranking improvements over specified baselines.
  • Sequence compression: Adaptive patching compresses older interaction windows with W=10 and S=10, fitting substantially more history within a fixed context budget.The sequence-compression ablation reports results as absolute improvement over the trivial baseline.
  • Hard negatives: Hard-negative mining draws four independent temperature samples per example and trains the propensity model with mined, popular, and random negatives.Recall plateaus beyond four samples, and negatives are mined from the trained mitigated generator.
Loading 2608.28931v1…