PaperlayerView exact revision on arXiv

Source-linked AI summary

ZooWork-ShopRanker: An Open, Preference-Aligned E-Commerce Reranker

Siqiao Xue, Shuxuan Liu, Ning Hu

arXiv:2609.31002v1cs.CL

TL;DR

E-commerce rerankers must capture shopping preferences and constraints that topical relevance alone does not represent, yet real search traffic lacks clean pairwise labels. ZooWork-ShopRanker trains open rerankers using judge-labeled preferences and evaluates them on SHOPRANK-BENCH. Its 8B and 4B models significantly outperform the strongest open baseline, and its 0.6B model beats its size peer.

  • Problem

    E-commerce ranking must distinguish shopper preference and product constraints beyond topical relevance, but real search traffic provides no clean pairwise labels.

    Source: §1 · p. 1

  • Method

    ZooWork-ShopRanker aligns open e-commerce rerankers to shopping preferences labeled by reasoning-capable LLM judges.

    Source: §1 · p. 3

  • Results

    ZooWork-ShopRanker-8B and -4B significantly outperform the strongest open reranker baseline, while -0.6B beats its size peer in both formats and gains extend to common MTEB tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    SHOPRANK-BENCH provides a private-traffic preference evaluation in two product-text formats, alongside attribute-hierarchy and budget diagnostics.

    Source: §1 · p. 2

    The natural-language format varies attribute inclusion, so format comparisons combine rendering sensitivity with possible attribute omission, which the authors do not audit against preference labels.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Open rerankers trained for general web retrieval transfer imperfectly to e-commerce, where ranking decisions depend not only on topical relevance but also on user preferences, product constraints, and comparative product fit. These preference signals are difficult to supervise at scale: real search traffic provides authentic queries and candidates but no clean pairwise labels. We present ZooWork-ShopRanker, a family of e-commerce rerankers (0.6B, 4B, and 8B) aligned to judge-labeled shopping preference. Training pairs are labeled by a panel of reasoning large language models (LLMs) from different families acting as a preference oracle, with position-debiased judgments and agreement tiers, and the rerankers are trained on these labels. The aligned 8B flagship then serves as a distillation teacher for the efficient 4B and 0.6B models, which are fit to its scores and sharpened on judged pairs. To measure progress, we introduce ShopRank-Bench, a contamination-limited benchmark of ~10,000 private-traffic preference pairs in both text formats, tiered by how many judge families committed to each label. ZooWork-ShopRanker-8B and -4B significantly outperform the strongest open reranker baseline, every model significantly beats its own un-aligned base, and ZooWork-ShopRanker-0.6B beats its size peer; the gains hold in both formats and extend to common MTEB benchmarks. We release the models and the dual-format ShopRank-Bench to facilitate further research.

1 Introduction

E-commerce rerankers must select products that fit a shopper’s preferences and explicit constraints, not just topical relevance. ZooWork-ShopRanker aligns open rerankers to judge-labeled preferences and is evaluated on a private-traffic benchmark.

1 Introduction

  • On four gold-tier pairs decided by one explicit constraint, all four open baselines select the violating product, while the 8B and 4B models honor the constraint on all four.The 0.6B model honors the constraint on two of the four pairs.Source: Table 1 · p. 2
  • Open baselines score 14–20 points lower on queries stating a constraint than on those that do not, while ZooWork-ShopRanker is flat across these subsets.These are different pair sets, so the comparison is an association, not a measured effect of adding a constraint.Source: §1 · p. 2, Figure 1 · p. 2
  • Real search traffic supplies authentic queries and candidates but no clean pairwise labels, so the authors use reasoning-capable LLM judges as a preference oracle.The judgments are cross-checked across model families, scored in both presentation orders, and tiered by agreement strength.Source: §1 · p. 2
  • SHOPRANK-BENCH uses private search traffic and is released in structured and natural-language formats, with attribute-hierarchy and budget diagnostics.The paper reports preference-track results with intervals and paired significance tests.Source: §1 · p. 2
  • ZooWork-ShopRanker-8B and -4B significantly outperform the strongest open reranker baseline, while -0.6B beats its size peer in both formats and gains extend to common MTEB tasks.Alignment is applied to 0.6B, 4B, and 8B models.Source: §1 · p. 3

2 Related Work

Related work spans open reranker architectures, scoring methods that avoid text generation, preference optimization with LLM judges, and domain-specific benchmarks.

Open rerankers

Scoring without generating

  • Rerankers can read a score from fixed-vocabulary logits in one forward pass rather than generate prose; setwise prompting and instruction distillation explore efficiency–effectiveness tradeoffs (Zhuang et al., 2024).The paper adopts this scoring approach, rather than proposing it, to evaluate a 27B model on 10,511 pairs affordably.Source: §2 · p. 3

Preference optimization and LLM judges

  • DPO converts pairwise preferences into a stable classification objective (Rafailov et al., 2023), while the paper uses multiple reasoning-capable LLM families and both candidate orders to label preferences.A separate evaluation family reduces circularity between training labels and evaluation.Source: §2 · p. 3

Domain-specific benchmarks

  • Domain-specific benchmarks for fashion retrieval and code search share SHOPRANK-BENCH’s motivation: exposing limitations that general-purpose benchmarks cannot (Gensmo.ai et al., 2026; Xue & Xu, 2026; Xue et al., 2026a).Source: §2 · p. 3

3 The SHOPRANK-BENCH Benchmark

SHOPRANK-BENCH evaluates judged shopping preferences and rule-following diagnostics using real search traffic and products surfaced by a production retrieval stack. Its preference pairs are judged across model families, while the diagnostic tracks use controlled queries and labels.

3 The SHOPRANK-BENCH Benchmark

  • SHOPRANK-BENCH contains ∼10,000 hard preference pairs from private search traffic, and its suite separates the two-format preference track from attribute-hierarchy and explicit-budget diagnostics.Preference pairings and labels have not been public, although queries and product attributes may appear elsewhere on the web.Source: §3 · p. 3, Table 2 · p. 4

3.1 Shared substrate: traffic, retrieval, and product text

  • The benchmark uses natural, sometimes messy queries across e-commerce categories and compares adjacent production-ranked candidates, which are treated as near-equivalent and difficult to distinguish.Products come from production retrieval and share a canonical text rendering pipeline across tracks.Source: §3.1 · p. 4, §3.2 · p. 5

3.2 Preference track: construction, judging, and release

  • Three judge families assess each preference pair in both candidate orders; 10,511 of 23,000 judged pairs survive when committed judges agree and none contradicts them.Abstentions are permitted, and a judge commits only when both presentation orders yield the same choice.Source: §3.2 · p. 5
  • The bronze tier is 40% of the benchmark, and every system scores 20–26 points lower on it than on gold; bronze labels are largely Gemma-decided.Per-judge verdicts are released, and a separate analysis checks performance on pairs decided by a training-disjoint family.Source: §3.2 · p. 5
  • The natural-language product text is rendered from the same attributes as the structured format, but varying attribute inclusion means format comparisons also reflect possible attribute omission.The authors do not audit whether every rendering preserves the attributes relevant to its preference label.Source: §3.2 · p. 5

3.3 Diagnostic tracks: construction and rationale

  • The diagnostic tracks use intent-conditioned query variants and rule-based labels rather than verbatim traffic queries and judge labels.Source: §3.3 · p. 5
  • The AHP track isolates one hierarchy decision per pair using two differing attributes, but its labels encode a hand-designed heuristic rather than observed preference.The heuristic is intent-conditioned and the track is diagnostic because it anti-correlates with judged preference.Source: §3.3 · p. 6
  • The budget track tests explicit price-cap compliance with a threshold slice and a control slice where the pricier product wins on a secondary attribute.A price-monotone shortcut cannot win both slices.Source: §3.3 · p. 6

4 Training ZOOWORK-SHOPRANKER

Section 4 distinguishes the directly aligned 8B flagship from the 0.6B and 4B students distilled from it and sharpened on judged pairs. It details the training data, LoRA setup, preference objective, and clean-label ceiling for on-policy mining.

4 Training ZOOWORK-SHOPRANKER

  • The released 8B is aligned directly on judge-labeled pairs, while the 0.6B and 4B students fit its soft scores before judged-pair sharpening.An earlier on-policy refinement round and a superseded 4B alignment-from-base pass are not part of the released small-model recipe.Source: §4 · pp. 6–7

4.1 Training setup

  • LoRA adapters remain the only trainable weights, and a controlled comparison reports matching or slightly better performance than full fine-tuning at roughly 1% of trainable parameters.The rerankers score products through the yes/no token-logit difference in a single forward pass.Source: §4.1 · p. 7
  • The training corpus uses a traffic slice disjoint from the benchmark, with about 4.4k judged preference pairs, retrieval replay, and 134,601 teacher-scored examples for the students.Training labels require agreement from Qwen3.5-122B and Gemma-4-31B in both presentation orders; DeepSeek-V4-Pro is withheld from training.Source: §4.1 · p. 7
  • Development-set accuracy tracks progress across training stages but is not directly comparable to the three-judge ShopRank-Bench results.The development slices are easy, hard-structured, and hard-NL; the hard pairs are adjacent in production rank.Source: §4.1 · pp. 7–8

4.2 Training components

  • The judge-labeled pairs use a pairwise logistic objective on preferred and rejected products, with general-retrieval replay mixed into training.The paper identifies this as the classical Bradley–Terry objective, equivalently RankNet with a temperature, and claims no novelty in the loss.Source: §4.2 · p. 8
  • On-policy mining gains only modestly in its first round, then stalls as the yield of judge-agreed gold pairs falls from 26% to 6% on maximally confusable pairs.After alignment, the smallest-margin pairs often compare two good or two bad products that judges tie, making the frontier judge-ambiguous rather than merely hard to mine.Source: §4.2 · p. 8
  • Distillation beats direct alignment from the same base at both student sizes and also outperforms a costlier panel teacher built from DeepSeek and Gemma grades.At feasible scale, the passage reports that label volume matters more than the nominal strength of the teacher.Source: §4.2 · p. 8

5 Experiments

The experiments evaluate preference alignment on SHOPRANK-BENCH and diagnostic constraint-following tracks, finding gains over open baselines and model bases alongside format-specific efficiency and budget behavior.

5.1 Experimental Setup

  • The evaluation covers 10,511 preference pairs in two formats and three agreement tiers, plus AHP and explicit-budget diagnostic tracks.Comparisons include open rerankers and Qwen3-Reranker bases; intervals are query-clustered and paired comparisons use McNemar tests.Source: §5.1 · pp. 8–9

5.2 Results and Analysis

  • ZOOWORK-SHOPRANKER outperforms every open baseline on SHOPRANK-BENCH in both formats; the 8B leads, and 4B exceeds Jina-m0.On the natural-language gold tier, 0.6B also beats Jina-m0, 91.2 vs. 88.8.Source: Figure 2 · p. 9, §5.2 · p. 9
  • On structured text, distilled 0.6B is statistically indistinguishable from base 4B (−0.9 points, 95% CI [−1.9, +0.2]) at roughly 7× less size.It sustains 2.8× the 4B throughput on 40% of the memory; the base 4B retains a significant edge on natural-language text.Source: §5.2 · pp. 9–10
  • Across three MTEB tasks, 4B and 8B lead the average, while 0.6B leads its base and BGE size peers, with some SciDocsRR slippage.These evaluation-only results have no confidence intervals or significance tests.Source: §5.2 · p. 10, Table 3 · p. 11
  • On AHP, preference alignment reaches just above chance at 8B without AHP supervision; attribute-score linear reweighting reaches 69.5% held-out.A learned attribute model with linear pooling scores 38.8% on judge-labeled pairs, so the passage bounds this decomposition rather than all attribute models.Source: §5.2 · p. 10
  • A 0.6B specialist trained on the automatic budget rule is strongest on both threshold and control slices and reaches 94.7% overall.Transfer to other checkable constraints was not tested, and the specialist was not evaluated on the general benchmarks.Source: §5.2 · p. 11

6 Conclusion

ZooWork-ShopRanker is an open family of preference-aligned e-commerce rerankers, with the flagship distilled into smaller models. The paper reports benchmark gains, describes a judge-labeling protocol, and qualifies the scope of its benchmark and label audit.

6 Conclusion

  • On ShopRank-Bench, the 8B and 4B significantly outperform the strongest open baseline, while the 0.6B beats its size peer.The smaller models are distilled from the aligned flagship.Source: §6 · p. 11

A.1 Category coverage of the preference set

  • Apparel accounts for about two-thirds of the 10,511 pairs; transfer to a catalogue weighted differently is untested.Beauty, electronics, home, baby and kids, and health make up roughly a quarter, with the rest in an uncategorised long tail.Source: §A.1 · p. 14

A.2 Judge protocol and agreement

  • Pairs are judged in both presentation orders; training retains about 4.4k pairs with Qwen3.5-122B and Gemma-family agreement in both orders.The benchmark adds DeepSeek-V4-Pro as a training-disjoint third family and tiers decisive labels by how many families committed.Source: §A.2 · p. 14

A.3 Held-out judge audit

  • A fourth-family audit agreed with 43 of 45 released labels, but its small sample is a consistency check, not an accuracy estimate or human validation.Both disagreements were in bronze; the authors report wide intervals for the 15-pairs-per-tier sample.Source: §A.3 · p. 14

A.4 Does the margin depend on the training judges?

The authors test whether gains depend on training-judge decisions by splitting non-gold pairs according to whether DeepSeek voted. The advantage persists on the DeepSeek-voted subset, but the analysis bounds rather than excludes a training-family effect.

A.4 Does the margin depend on the training judges?

  • The split contains 5,100 pairs where DeepSeek committed a verdict and 3,568 where it abstained and training families alone supplied the label.The two training families also produced labels; the DeepSeek-voted subset is therefore not training-disjoint.Source: §A.4 · p. 14
  • On the DeepSeek-voted subset, the advantage over Jina-m0 persists but shrinks to +2.9 for 8B and +0.9 for 4B.The authors interpret this as bounding a training-family effect, not excluding one.Source: §A.4 · p. 14, Table 4 · p. 15
  • The subsets are not cleanly disjoint, differ in tier composition, and lack an interaction test; human agreement remains the main open item.All systems’ absolute accuracies fall on the training-panel-only subset, which the authors attribute to label strength rather than circularity.Source: §A.4 · p. 15

A.5 Contamination audit

The appendix audits benchmark contamination and construction, release details, model-size diagnostics, and constraint-specific results. It reports strong budget-constraint performance while noting limits in transfer scope and the behavior of the general models.

A.5 Contamination audit

  • The benchmark uses private traffic from non-overlapping windows, with zero shared query strings or labeled triples between training and benchmark.The audit also finds no repeated released preference triple, reversed labeled pair, or product paired against itself; private labels do not establish that queries or public catalogue attributes were unseen in pretraining.Source: §A.5 · p. 15
  • Simple heuristics do not match learned-model results: choosing the cheaper product wins 58.7% on unequal-price pairs, while the weakest learned model scores 71.6%.Choosing the longer text wins 53.3%; the cheaper-product comparison uses a subset, whereas BM25’s 56.2% is on the full pair set.Source: §A.5 · p. 15

A.6 Data release and privacy

  • Released preference records omit user, account, session, location, timestamp, and behavioral fields; the queries were screened for direct identifiers.The release contains product searches and product text rendered from public catalogue attributes.Source: §A.6 · p. 16

A.7 Natural-language rendering styles

  • The five natural-language renderings vary sentence order, attribute selection, and phrasing while retaining the same preference label.A local LLM produces each rendering from structured fields only, without adding scraped descriptions.Source: §A.7 · p. 16

A.8 Product text schema

  • The structured product view uses a canonical pipe-delimited representation with 16 fields.Table 5 identifies the product-text schema fields.Source: §A.8 · p. 16, Table 5 · p. 16

A.9 Track record examples

  • An AHP diagnostic example changes exactly two hierarchy attributes: the preferred product satisfies higher-priority price but violates lower-priority color.The other catalogue fields are identical, and the hierarchy label selects the price-compliant product.Source: §A.9 · p. 17

B.1 Training hyperparameters

  • Training uses rank-16 LoRA adapters with base weights frozen; judge-labeled pairs are sampled at 0.60 in alignment and 0.80 in mixed-format and sharpening stages.The remaining sampling probability is assigned to the general-relevance replay anchor.Source: §B.1 · pp. 17–18

B.3 Constraint-subset breakdown

  • On 338 gold pairs with a price ceiling, ZooWork-ShopRanker-4B scores 97.9, while BM25 scores 47.9.The table also shows cross-encoders at 55.6–70.4 on these pairs, versus 89.6–96.1 when queries state no constraint.Source: §B.3 · p. 18

B.4 Pipeline ablation

The ablations distinguish the released distillation route from development-only pipeline results, find no benchmark benefit for mixed-format training, and compare serving costs. The paper reports that alignment adds no inference cost but retains a quality-throughput trade-off against an encoder reranker.

B.4 Pipeline ablation

  • On ShopRank-Bench, the released distilled 4B significantly outperforms the final alignment-from-base 4B in both formats: 81.2 vs. 80.4 structured and 79.5 vs. 78.3 prose.The authors conclude that distillation, not only the 0.6B development evidence, justifies the 4B route.Source: §B.4 · p. 19
  • On development hard pairs, the full pipeline lifts every model above the next-larger base, with the distilled 0.6B reaching base-8B level (78.3 vs. 78.0).This is a development-set result; on ShopRank-Bench, the narrower claim is that 0.6B is statistically indistinguishable from base-4B on structured text only. A panel-grade teacher yields 70.7 versus 78.3 for the reranker teacher on the hard-structured development slice.Source: §B.4 · p. 19, Table 7 · p. 19

B.5 Mixed-format training

  • On ShopRank-Bench, mixed-format 8B is indistinguishable from structured-only 8B in both formats, and both have the same 1.6-point format gap.The controlled comparison does not establish a robustness benefit for mixed-format training or test whether alignment increases format sensitivity.Source: §B.5 · p. 20

B.6 Serving cost

  • Alignment is free at inference, while BGE-v2-m3 is 4.6× faster at batch 32 and uses one-third the memory despite trailing by 3.3 structured benchmark points.The 0.6B matches base-4B on structured pairs at 2.8× throughput; the 8B gains 6.8 accuracy points over 0.6B at 4.2× the serving cost.Source: §B.6 · p. 20

B.7 Zero-shot LLM reference

On structured SHOPRANK-BENCH pairs, both reasoning LLMs beat every reranker, though their shared model class leaves a circularity caveat. LoRA matches or edges full fine-tuning at 0.6B, and user-context tests show explicit query intent overrides a conflicting profile.

B.7 Zero-shot LLM reference

  • Both zero-shot LLMs clearly beat every reranker on structured SHOPRANK-BENCH, and GLM-5.3 and Qwen3.5-27B do not differ significantly (p = 0.46).The models count a pair correct only when both presentation orders select the preferred product; Table 9 reports their reference results.Source: §B.7 · p. 20, Table 9 · p. 21
  • A 27B LLM decides a pair in 224 ms versus 2 × 25.8 ms for two ZOOWORK-SHOPRANKER-8B scores, but its lead may partly reflect shared LLM judgment.The 224 ms cost comes from a separate 2,000-pair run, not the full-benchmark accuracy evaluation; rerankers must score many candidates per query.Source: §B.7 · p. 21

B.8 Parameter efficiency

  • LoRA matches or edges full fine-tuning at 0.6B while updating approximately 1% of parameters.The adapter is a few megabytes rather than a multi-gigabyte checkpoint; the controlled comparison varies only update rank.Source: §B.8 · p. 21, Table 10 · p. 22

B.9 User-context conditioning

  • Across 240 synthetic user-context cases, explicit query intent overrides a conflicting user profile, while all models achieve 100% accuracy.The comparison of decision margins between base-4B and ZOOWORK-SHOPRANKER-8B changes both scale and training and does not isolate fine-tuning.Source: §B.9 · p. 22

B.10 Attribute hierarchy: full analysis

Hand-designed attribute priorities do not match judge-labeled preference, even when conditioned on query intent. Preference alignment improves the AHP diagnostic only partly, with the 8B model barely above chance.

B.10 Attribute hierarchy: full analysis

  • Zero-shot, every open reranker and un-fine-tuned Qwen base falls well below the 50% chance baseline on AHP, with price the hardest attribute and scale unhelpful.Each pair tests whether the model chooses the product satisfying the higher-priority of two differing attributes.Source: §B.10 · p. 22
  • 38.8% on the judge-labeled development slice (n = 5,076) for hand-designed attribute scores trails 82.5% for the relevance-trained 0.6B, and best linear pooling reaches only 69.5%.The attribute hierarchy was already intent-conditioned, so the failure is not simply that it imposes one global priority order.Source: §B.10 · p. 22
  • With no AHP supervision, ZOOWORK-SHOPRANKER-8B rises from far below chance to just above it, while the unaligned bases stay flat across sizes.The best model only barely clears a coin flip, leaving attribute hierarchy an open problem.Source: §B.10 · p. 22

B.11 Explicit budget: full analysis

General preference alignment shifts models toward cheaper products but does not install budget-threshold behavior; targeted programmatic budget supervision performs best on that diagnostic. The AHP breakdown shows alignment redistributes accuracy across attributes, while clustered significance checks leave the reported comparisons unchanged.

B.11 Explicit budget: full analysis

  • The budget track separates threshold cases, where the under-budget product wins, from controls, where both are under budget and a secondary attribute favors the pricier product.Always choosing cheaper scores 100% on threshold but 0% on control; always choosing pricier does the reverse.Source: §B.11 · pp. 22–23
  • 94.7% overall for the 0.6B budget specialist exceeds 88.6% for ZOOWORK-SHOPRANKER-4B and 85.0% for ZOOWORK-SHOPRANKER-8B.The specialist is trained on automatically labeled budget pairs with general-relevance replay; it is not evaluated on the general benchmarks of Table 3.Source: §B.11 · p. 23

B.12 Zero-shot AHP breakdown by hierarchy level

  • ZOOWORK-SHOPRANKER-4B reaches 0.973 on AHP price but 0.015 on style at 0.6B and 0.030 at 4B, showing alignment redistributes rather than uniformly lifts attribute accuracy.The table caption notes that only the 8B model clears the 0.50 overall chance line.Source: §B.12 · p. 23, Table 11 · p. 23

B.13 Full SHOPRANK-BENCH results

  • Query-clustered bootstrap tests with Holm–Bonferroni correction leave all eight comparisons' conclusions unchanged; the 0.6B versus base-4B non-significance is specific to structured text.The table captions report structured and natural-language SHOPRANK-BENCH results by judge-agreement tier; every reranker loses 20–26 points from gold to bronze.Source: §B.13 · p. 24, Table 12 · p. 24, Table 13 · p. 24

B.14 Further limitations

The benchmark's bronze tier has asymmetric judge-family participation, and the paper identifies additional limits in label supply, rendering, public-benchmark contamination, and query and category coverage.

B.14 Further limitations

  • The judge panel is not uniformly active, leaving the single-judge bronze tier largely Gemma-decided, as described in Section 3.2.Source: §B.14 · p. 24
  • On hard adjacent pairs, judges return a unanimous tie or outright conflict 54.3% of the time, capping clean on-policy labels.The breakdown is 53.7% unanimous ties and 0.6% conflicts.Source: §B.14 · p. 25
  • The natural-language benchmark uses LLM-rendered text, public-benchmark comparisons may reflect pretraining contamination, and queries are English- and US-centric.The category mix also leans toward apparel.Source: §B.14 · p. 25

Related papers

Loading 2609.31002v1…