PaperlayerView exact revision on arXiv

Source-linked AI summary

RexBERT: Context Specialized Bidirectional Encoders for E-commerce

Rahul Bajaj, Anuj Garg

arXiv:2602.04605v1cs.CLcs.AI

TL;DR

Generic encoders provide efficient language representations but have limited coverage of specialized e-commerce semantics. RexBERT addresses this gap with the 350 billion token Ecom-niverse corpus and a three-phase curriculum combining general pre-training, context extension, and annealed domain specialization. Across masked-token recovery and ESCI semantic similarity, it outperforms general-purpose encoders despite using 2–3× fewer parameters while remaining competitive on GLUE.

  • Problem

    Generic encoders trained on broad corpora often provide limited coverage of fine-grained e-commerce distinctions and attributes needed for search, recommendations, extraction, and compliance routing.

  • Method

    RexBERT combines the Ecom-niverse corpus with a three-phase curriculum of general pre-training, context extension, and annealed domain specialization using Guided MLM.

  • Results

    Across masked-token recovery and ESCI semantic similarity, RexBERT outperforms general-purpose encoders despite using 2–3× fewer parameters and remains competitive on GLUE.

  • Takeaways & Limitations

    RexBERT provides a template for building efficient, high-quality domain-specialized encoders from well-curated open data and gradual training curricula.

  • Takeaways & Limitations

    The marginal contribution of Guided MLM relative to corpus composition and annealing remains to be quantified in future controlled studies.

Abstract

from arXiv · show

Encoder-only transformers remain indispensable in retrieval, classification, and ranking systems where latency, stability, and cost are paramount. Most general purpose encoders, however, are trained on generic corpora with limited coverage of specialized domains. We introduce RexBERT, a family of BERT-style encoders designed specifically for e-commerce semantics. We make three contributions. First, we release Ecom-niverse, a 350 billion token corpus curated from diverse retail and shopping sources. We describe a modular pipeline that isolates and extracts e-commerce content from FineFineWeb and other open web resources, and characterize the resulting domain distribution. Second, we present a reproducible pretraining recipe building on ModernBERT's architectural advances. The recipe consists of three phases: general pre-training, context extension, and annealed domain specialization. Third, we train RexBERT models ranging from 17M to 400M parameters and evaluate them on token classification, semantic similarity, and general natural language understanding tasks using e-commerce datasets. Despite having 2-3x fewer parameters, RexBERT outperforms larger general-purpose encoders and matches or surpasses modern long-context models on domain-specific benchmarks. Our results demonstrate that high quality in-domain data combined with a principled training approach provides a stronger foundation for e-commerce applications than indiscriminate scaling alone.

1 Introduction

RexBERT addresses gaps in generic encoder representations for e-commerce by combining a specialized corpus with a three-stage training curriculum. The resulting models outperform general-purpose encoders on e-commerce tasks despite using fewer parameters.

  • Motivation: Generic encoders often miss fine-grained distinctions among complementary, substitute, and irrelevant products and detailed product attributes.These distinctions matter for search, recommendations, attribute extraction, and compliance routing.
  • Contributions: RexBERT curates Ecom-niverse, a 350 billion token collection of retail-relevant text distilled from FineFineWeb.The corpus spans categories including fashion, beauty, automotive, and entertainment.
  • Training approach: The training recipe combines general pre-training, context extension to 8,192 tokens, and annealed specialization toward Ecom-niverse with Guided MLM.It uses a diverse mixture of open web, books, code, and technical documents before shifting toward commerce data.
  • Evaluation: RexBERT models span 17M to 400M parameters and are evaluated on GLUE and Amazon ESCI-derived e-commerce benchmarks.Evaluation covers token classification, semantic similarity, and general language understanding.
  • Results: Despite 2-3x fewer parameters, RexBERT achieves higher token classification accuracy and Spearman correlation than contemporary models in similar parameter ranges.The introduction reports consistent gains over general-purpose models on the targeted tasks.

2 Related Work

Prior work modernizes encoder efficiency and adapts encoders to specialized domains, including scientific, biomedical, financial, and e-commerce settings. RexBERT builds on this trajectory while emphasizing that smaller models can excel when paired with targeted pre-training.

  • Encoder architectures: ModernBERT modernizes encoders with RoPE, GeGLU, alternating global and local attention, unpadding, and flash attention.Related work presents these changes as architectural and efficiency advances for encoder models.
  • RexBERT’s positioning: RexBERT advances this literature by showing that much smaller models can excel on specialized domains when paired with targeted pre-training.This positions domain-focused training as a complement to architectural and scaling improvements.
  • Domain adaptation: Targeted pre-training has improved specialized encoders such as BioBERT, ClinicalBERT, BioClinical ModernBERT, and SciBERT on domain-specific tasks.These models use biomedical or scientific corpora and, in SciBERT’s case, a domain-tailored vocabulary.
  • E-commerce encoders: E-commerce encoders such as E-BERT add phrase-level and product-level pre-training tasks to capture fine-grained product associations.Reported applications include review question answering, aspect extraction, sentiment classification, and product classification.
  • Encoder–decoder comparisons: The Ettin suite compares paired encoder-only and decoder-only models under matched recipes, finding encoders stronger for classification and retrieval while decoders excel at generation.The comparison addresses differences in data, parameter counts, and objectives across architectures.

3 Corpus Curation

Ecom-niverse is built by selecting commerce-relevant FineFineWeb domains and applying staged filtering, labeling, auditing, distillation, and thresholding. The resulting corpus contains more than 350 billion tokens, while RexBERT’s curriculum proceeds from mixed data to long-context training and domain annealing.

  • Source selection: FineFineWeb provides the source corpus, with selected domains grouped into those requiring filtering and those with near-complete commerce overlap.FineFineWeb is a 4.4 trillion token CommonCrawl-derived corpus organized into approximately fifty categories.
  • Output corpus: More than 350 billion tokens of e-commerce content remain after language detection, boilerplate and advertisement removal, deduplication, and safety filtering.The corpus is described as one of the largest domain-specific corpora available.
  • Curation pipeline: The curation pipeline performs domain selection, sampling, LLM labeling, QA auditing, fastText distillation, and thresholded filtering at scale.The staged design concentrates labeling and filtering effort on potentially commerce-relevant distributions.
  • Training curriculum: RexBERT first trains on 1.7 trillion mixed-data tokens, extends context to 8,192 tokens for 250 billion tokens, then anneals onto 350 billion Ecom-niverse tokens.The curriculum gradually moves from general to domain-specific training distributions.
  • Quality control: QA auditing uses Llama3-70B to estimate labeling errors and refine prompts or guidelines until label consistency stabilizes.Validated per-domain fastText classifiers then approximate the LLM decision boundary for high-throughput corpus scoring.

4 Training Methodology

RexBERT uses a three-phase masked-language-modeling curriculum that moves from broad linguistic coverage through long-context training to annealed e-commerce specialization. The recipe adds span-aware and guided masking to emphasize multi-token units, entities, and attributes.

  • Curriculum: RexBERT trains with a three-phase curriculum: general pre-training, context extension, and annealed domain specialization.The procedure adapts ModernBERT to e-commerce text properties such as high entity density, attribute–value structure, and specialized terminology.
  • Masking: Dynamic span-aware masking targets multi-token lexical units while reducing information leakage from partially observed subword fragments.
  • Masking: Guided MLM preferentially masks domain-relevant entities and attributes during Phase 3, interleaving guided examples with standard random span masking.
  • Phase 1: general pre-training: Phase 1 uses a 1.7 trillion token mixture of web text, books, code, technical papers, and multilingual content at 512-token sequence lengths.Evaluation on general MLM perplexity and natural language understanding probes is used to maintain broad linguistic coverage.
  • Phase 2: context extension: Phase 2 extends sequences to 8,192 tokens and trains for 250 billion tokens, using NTK-aware RoPE scaling and alternating global/local attention.The phase targets long product pages, FAQs, and concatenated attribute blocks.
  • Phase 3: annealing: Phase 3 specializes on Ecom-niverse for approximately 350 billion tokens while reducing masking to 10%-15% and gradually upsampling e-commerce data.The stated goal is to preserve general knowledge during specialization.

5 Model Architecture

RexBERT is a BERT-style encoder that incorporates ModernBERT-inspired architectural changes for efficient training and long-context processing. Its design replaces several original BERT choices with normalization, positional, activation, and attention modifications.

  • Core changes: RexBERT removes bias terms and uses pre-layer normalization to improve training stability.
  • Core changes: Rotary positional embeddings replace absolute positional embeddings, supporting long-context extrapolation and efficient implementation.
  • Core changes: GeGLU activations replace GELU and are described as providing better optimization.
  • Core changes: Attention layers alternate between global and local attention to combine full-context and localized processing.
  • Configurations: Table 2 summarizes the configurations of the RexBERT models.

6 Evaluation

RexBERT is evaluated on e-commerce token classification and semantic similarity tasks derived from Amazon ESCI, alongside GLUE tests of general language understanding. Across these evaluations, it shows strong performance against general-purpose encoders, including at smaller parameter scales.

  • Benchmark: Amazon ESCI contains approximately 130,000 queries and 2.6 million labeled query-product pairs with Exact, Substitute, Complement, and Irrelevant relevance levels.The experiments use the English subset and public train/test splits.
  • Benchmark: ESCI tests nuanced product semantics by requiring models to distinguish substitutes from complements and other relevance relationships.
  • Benchmark: GLUE evaluation covers sentiment, inference, paraphrase detection, and semantic similarity using task-specific fine-tuning protocols and metrics.
  • Token Classification: RexBERT-base improves product-title top-1 accuracy from 60.5%→69.2% at 128 tokens and from 65.5%→72.6% at 512 tokens over ModernBERT-base.
  • Semantic Similarity: RexBERT consistently yields higher Spearman correlation than comparable-scale general-purpose encoders on ESCI semantic similarity.The reported comparison includes ModernBERT, Ettin, and EmbeddingGemma models.
  • Natural Language Understanding: RexBERT-large leads on GLUE paraphrase detection, multi-genre inference, and semantic similarity, while ModernBERT-large leads on sentiment classification and question-answering inference.
  • Natural Language Understanding: RexBERT-mini outperforms DistilBERT across all five GLUE tasks, while RexBERT-micro consistently outperforms ettin-17m.

7 Discussion

RexBERT’s gains are strongest on e-commerce-shaped evaluations, where specialized data and curriculum appear more important than raw model scaling. The discussion connects these gains to domain-specific language, practical deployment needs, and an unresolved question about Guided MLM’s contribution.

  • Performance and mechanism: On e-commerce-shaped evaluations, data and curriculum dominate raw scaling, with gains linked to long-tail entities and attribute-value constructions.The reported pattern spans masked-token recovery and ESCI semantic similarity at comparable or larger parameter counts.
  • Performance and mechanism: E-commerce language is entity-dense, compositional, and semi-structured, motivating corpus coverage of domain terms and structure.The discussion identifies these properties as settings where specialization gains are most pronounced.
  • Performance and mechanism: Guided MLM allocates additional learning signal to rare, high-value spans while retaining standard random masking.The mechanism targets salient tokens that matter for retrieval and ranking.
  • Limitations: The work does not isolate Guided MLM through an ablation, leaving its marginal contribution relative to corpus composition and annealing for future study.This limits attribution of the overall improvements to any single training component.
  • Practical implications: Stronger token recovery and higher Spearman correlation support attribute normalization, retrieval, and candidate generation, while 8k-token input reduces heuristic truncation.The deployment discussion connects token recovery to catalog quality and graded relevance to embedding geometry.

8 Conclusion

RexBERT combines open-data e-commerce specialization with a three-phase curriculum built on modern encoder design choices. It outperforms general-purpose encoders on domain-focused evaluations with fewer parameters while remaining competitive on GLUE, supporting efficient domain-specific representation learning.

  • Contributions: RexBERT is an e-commerce-specialized encoder family trained entirely on open data, centered on the 350B-token Ecom-niverse corpus.The corpus is produced through domain-conditioned filtering and validation.
  • Contributions: The training curriculum combines general pre-training, long-context extension, and annealed domain specialization augmented with Guided MLM.The approach builds on modern encoder design choices including RoPE, GeGLU, and alternating local/global attention.
  • Results: Across masked-token recovery and ESCI semantic similarity, RexBERT outperforms general-purpose encoders despite using 2–3× fewer parameters and remains competitive on GLUE.The reported result indicates target-domain gains without sacrificing broad language understanding under gradual specialization.
  • Implications: The released models and methodology provide a template for constructing efficient, high-quality encoders for other specialized domains from well-curated open data.The proposed transfer is framed as replacing Ecom-niverse with a corpus from another vertical while applying the same recipe.
Loading 2602.04605v1…