Source-linked AI summary
Retrieve, Match, Escalate: Accurate and Scalable Product Linking with VLM-Distilled Cross-Encoders and Agentic VLMs
Jian Wang, Steven Xu, Sanjyot Thete, Maryam Barouti, Tom Tang, Elaine Wu, Charu Sareen, Kyle MacDonald
TL;DR
Marketplace product linking must resolve billions of noisy records against tens of millions of canonical products without applying expensive reasoning uniformly. The paper presents a confidence-routed retrieve-then-match cascade with a distilled cross-encoder for routine cases and an agentic multimodal VLM for ambiguous cases. Its production design raises end-to-end link coverage while reducing operator workload and wrong links, and uses a self-hosted reasoning tier with a paper-supported failure-mode boundary.
Problem
Billions of noisy merchant records must be matched to tens of millions of canonical products, while a uniform model is either too costly for easy pairs or too weak for hard cases.
Method
The pipeline retrieves candidates, uses a dual-VLM-consensus-distilled text cross-encoder for confidence routing, and escalates ambiguous pairs to a tool-using multimodal VLM agent.
Results
The production cascade raises end-to-end link coverage while reducing operator workload and wrong links, with its open-weight self-hosted agent reaching precision parity with a closed frontier model at lower cost without fine-tuning.
Takeaways & Limitations
Confidence-routed escalation makes expensive multimodal reasoning practical by reserving it for the ambiguous tail while the cheap stage resolves routine cases.
Takeaways & Limitations
Retrieval recall remains an upper bound, and heterogeneous, multilingual, frequently missing merchant fields constrain candidate generation.
Abstract
from arXiv · showhide
Product linking, the entity-resolution task of mapping merchant product records to canonical catalog products, consolidates fragmented listings so downstream search, recommendation, and advertising see one clean entry per product. At marketplace scale, billions of noisy, multi-category records must be resolved against tens of millions of canonical products, where scoring every candidate with a single model is either too weak for the hard cases or too costly for the easy ones. We present a production retrieve-then-match cascade that spends computation in proportion to difficulty: retrieval surfaces plausible matches, a lightweight text cross-encoder auto-resolves the high-confidence majority, and an agentic multimodal vision-language model settles the ambiguous remainder by inspecting product images and issuing web searches for evidence that is in neither record. The cross-encoder is distilled from millions of dual-VLM-consensus labels, retiring human annotation from the training set, and is calibrated to auto-accept links at a 98% precision bar validated against a smaller operator-certified audit. The agent is a self-hosted open-weight model that reaches a closed frontier VLM's precision at a four-point recall cost (88% versus 92%) for roughly one-seventh the per-pair cost, with no fine-tuning. Per-pair cost spans nearly five orders of magnitude from the cheap cross-encoder to the frontier VLM, so escalating only the hard tail to the agent raises end-to-end link coverage from the cheap stage's 68% to 77%.
1 Introduction
Product linking maps fragmented merchant listings to canonical products, but marketplace-scale noise and heterogeneous evidence make uniform matching either costly or inadequate. The paper proposes a confidence-routed cascade combining a distilled cross-encoder with agentic multimodal reasoning for ambiguous pairs.
- Product linking maps merchant product records to canonical catalog products so metadata and behavioral signals can be pooled for search, recommendation, and advertising.
- Billions of noisy, multilingual, multi-category records must be matched against tens of millions of canonical products while preserving distinctions such as size and flavor.
- A uniform model creates a cost-capability trade-off because easy pairs waste computation while a cheap model may inadequately resolve ambiguous pairs.
- The proposed cascade scores retrieved candidate pairs with a cross-encoder, directly accepts or rejects high-confidence cases, and escalates uncertain pairs to a tool-using multimodal VLM agent.
- The cross-encoder is trained on dual-VLM-consensus labels, while the self-hosted agent reaches precision parity with a closed frontier model at lower cost without fine-tuning.
2 Related Work
Related work frames product linking within entity resolution and adaptive, cost-tiered inference. This paper extends those ideas to multimodal product evidence and web-retrieved information absent from catalog records.
- Product matching and entity resolution: Entity resolution classically uses a blocking-then-matching pipeline, while learned matchers such as Ditto cast pairwise matching as Transformer sequence-pair classification.
- Adaptive routing and cost-tiered inference: Adaptive routing allocates computation by difficulty, from classical cascades and reject options to systems that escalate hard inputs from cheap models to expensive reasoning.
3 System Architecture
The system links merchant records to deduplicated canonical products through a retrieve-then-match funnel. It routes computation by difficulty, using retrieval, a cheap text cross-encoder, and an agentic multimodal VLM.
- 3.1 Overview: Each merchant record is linked to the canonical product, if any, that denotes the same consumer product, with successor targets possible after catalog merges.
- 3.1 Overview: Retrieval returns the top K=20 candidates per record, the text-only cross-encoder scores each pair and routes by confidence, and the VLM adjudicates unresolved pairs.
3.2 Retrieval
Retrieval narrows the global catalog before matching by merging text, image, and barcode channels into a small candidate set. Its recall limits the full funnel, and heterogeneous or missing merchant fields remain an active constraint.
- A correct match absent from retrieval cannot be recovered downstream, so candidate generation upper-bounds recall for the entire funnel.
- Text ANN, image ANN, and exact or partial barcode retrieval are merged into a single list of K=20 canonical candidates per merchant record.
- Figure 1 routes high-confidence pairs to acceptance, low-confidence pairs to rejection, and only the ambiguous medium band to the agentic multimodal VLM.
- Retrieval improvements under evaluation include upstream attribute filtering, larger K, stronger multimodal embeddings, lexical retrieval, dedicated barcode retrieval, and broader image coverage.
- Retrieval must handle heterogeneous, multilingual, and frequently missing fields, including free-text sizes, unnormalized barcodes, absent images, and names across scripts.
3.3 Cross-Encoder Matcher
The cross-encoder is a calibrated, text-only matcher that auto-resolves high-signal candidate pairs and routes ambiguous pairs onward. It is trained primarily from dual-VLM-consensus labels rather than human annotations.
- Matcher design: The text-only cross-encoder scores each retrieved candidate pair and partitions matches into HIGH auto-accept, LOW auto-reject, and MEDIUM escalation bands.Its design prioritizes calibrated confidence over peak accuracy because many pairs have strong signals such as exact barcodes or near-identical names.
- Architecture: The matcher uses a ModernBERT-base cross-encoder with a sigmoid classification head and no image input.Both products are serialized into one sequence, enabling joint cross-attention over their fields.
- Labeling and training data: 5.3M training pairs were created primarily from dual-VLM consensus, retaining the 87% of pairs on which two VLMs agreed.The remaining 13% were dropped as label-ambiguous.
- Labeling and training data: The VLM-labeling process replaces 40k human labels with 5.3M higher-consistency labels, a 130× scale-up.Labels also include medium-confidence production predictions, while a zero-leak 6k-row development set is held out.
3.4 VLM Agent
The VLM agent handles the ambiguous tail that text-only matching cannot reliably resolve. It combines multimodal product inspection with optional web search, using self-hosted open-weight inference and constrained tool use.
- Motivation: The agent targets MEDIUM-band pairs whose missing, noisy, or conflicting textual evidence cannot distinguish true matches from near-duplicates.Examples include missing barcodes, placeholder images, and sizes buried in free text.
- Agent operation: Stage 3 receives both product records and their images, then may autonomously search the open web for evidence absent from the records.The agent decides whether to gather more evidence at each reasoning step before emitting a structured JSON verdict.
- Production implementation: The production system uses a self-hosted Qwen 3.6 35B-A3B open-weight VLM with an internally maintained MCP web-search tool.The model has 35B total and 3B active parameters, is FP8-quantized, and is served on H200-class GPUs.
- Tool-use control: A four-round tool cap reaches the empirical F1 and precision peak while avoiding later searches that repeat already-seen information.Rounds five through twenty add latency and cost without accuracy gains.
3.5 Routing
Routing uses calibrated cross-encoder scores to resolve easy pairs cheaply and escalate only uncertain pairs to expensive Stage 3. This cascade makes escalation rate the central cost-control lever.
- Threshold routing: Scores at or above the HIGH threshold are auto-accepted, scores at or below LOW are auto-rejected, and the residual MEDIUM band is escalated.The HIGH threshold is the lowest score implying at least 98% precision; LOW is set to retain at least 98% of positives above it.
- Expected cost: Expected per-pair cost equals cheap-stage cost plus the MEDIUM escalation fraction multiplied by expensive-stage cost.The expensive term is paid only for escalated pairs.
- Cost tiers: Nearly five orders of magnitude separate the cross-encoder and closed frontier VLM across the cost ladder.The open-weight agent call costs roughly 7,000× a cross-encoder call, while the closed VLM is 7× costlier still at equal precision.
- Cost tiers: The cascade routes the easy majority away from Stage 3 because its per-call cost is far higher than the cross-encoder’s.Consequently, narrowing the MEDIUM band is the primary aggregate cost lever.
4 Evaluation
The evaluation measures retrieval, matcher behavior, and agentic escalation across independent and production-grounded audits. Results show strong retrieval and matcher performance, a rare barcode-collision failure mode, and improved coverage from escalating ambiguous records to a cheaper open-weight agent.
- 4.1 Stage 1: Retrieval Recall: 94.27%→94.05% primary recall followed adding UPC retrieval, because correlated barcode channels displaced stronger candidates from the fixed pool.The category prefilter was roughly neutral on primary recall but introduced regression risk on already-linked pairs.
- 4.1 Stage 1: Retrieval Recall: 0.40% deduplication-slice recall from image-only retrieval produced zero unique fused hits, while text reached 87–92% and GTIN reached 73%.Qualitative review attributed some remaining misses to duplicate canonical products and mislabeled ground truth rather than weak retrieval alone.
- 4.2.1 Model Discovery via Autoresearch.: +0.8pp F1 came from feeding raw barcode digits instead of a binary equality flag, but stress auditing exposed high-confidence errors on reused or corrupted barcodes.The mitigation combines barcode dropout with a name-similarity guardrail; its fix remains a candidate pending fresh non-adversarial validation.
- 4.2.3 Results: 77.05% R@P98 was achieved on a zero-leak held-out set, while an independent 24k-pair operator-certified audit found 43.7% HIGH-band auto-accept coverage at the 98% precision bar.BF16 and FP32 inference showed 99.95% band agreement over 32.9M rows.
- 4.3.1 Accuracy Versus Trained Human Operators.: 13.7 pp accuracy, 18.5 pp recall, and 4.7 pp precision gains over trained operators were reported on hardest medium-confidence tasks, reaching 99.15% precision at 95.47% recall.The comparison used independent labeling with adjudication of disagreements by a separate reviewer panel.
- 4.3.2 End-to-End Production Coverage.: 9.0 percentage points raised record-level production coverage, from 68.1% for the cheap stage to 77.1% after escalating residual cases to the agent.At equal 98.0% precision, the open-weight agent costs 7× less per pair than the closed stack while reaching 88% versus 92% recall.
5 Discussion
The cascade’s accuracy and economics depend on label quality, difficulty-based routing, and a self-hosted agent that trades some recall for lower cost. Its design addresses noisy catalog inputs while concentrating multimodal reasoning on ambiguous pairs.
- Labeling and distillation: Dual-VLM consensus labels trade modest coverage for higher consistency and expose disagreement as a signal of pair difficulty.The distilled matcher inherits supervision from expensive multimodal adjudications, while disagreement helps identify harder pairs.
- Labeling and distillation: Data volume moved the operating point more than model size or dedicated hyperparameter sweeps.The authors therefore argue that label-generation pipelines deserve more investment than model search for production entity resolution.
- Operational trade-offs: The cascade routes multimodal reasoning only to ambiguous MEDIUM pairs because retrieval expands each record into K further candidate pairs at marketplace scale.This difficulty-based allocation avoids applying heavyweight reasoning uniformly across the expanded candidate set.
- Catalog quality: Raw barcode digits let the matcher exploit partial and noisy identifiers, while the multimodal agent remains usable with missing or placeholder images.These choices respond directly to missing, invalid, and unreliable merchant-supplied fields.
6 Conclusion
The paper presents a production retrieve-then-match system that uses a distilled cross-encoder for high-signal cases and escalates ambiguous pairs to a multimodal VLM agent with web search. This design raises coverage while reducing operator workload and wrong links, and its routing and distillation components are portable across domains.
- 6 Conclusion: The system auto-resolves high-signal pairs with a distilled text-only cross-encoder and escalates ambiguous pairs to an agentic multimodal VLM.The agent can search for missing information beyond the catalog records.
- 6 Conclusion: The cascade raises end-to-end link coverage while reducing operator workload and wrong links.Confidence routing concentrates expensive reasoning where difficulty is highest.
- 6 Conclusion: The escalation policy, VLM-as-labeler distillation loop, and self-hosted reasoning tier are domain-independent, while noisy catalogs and unreliable fields remain domain-specific.Each stage can be swapped as models improve.
Declaration on Generative AI
The authors used generative AI tools during preparation for drafting, content enhancement, paraphrasing, and grammar and spelling checks, then reviewed and edited the content.
- Declaration on Generative AI: Generative AI tools were used for drafting, content enhancement, paraphrase and rewording, and grammar and spelling checks.The authors state that they reviewed and edited the resulting content and retain full responsibility for the publication.