Source-linked AI summary
Adapting Vision-Language Models for E-commerce Understanding at Scale
Matteo Nulli, Vladimir Orshulevich, Tala Bazazo, Christian Herold, Michael Kozielski, Marcin Mazur, Szymon Tuzel, Cees G. M. Snoek, Seyyed Hadi Hashemi, Omar Javed, Yannick Versley, Shahram Khadivi
TL;DR
E-commerce product understanding needs multimodal models that handle fine-grained attributes, multiple images, and noisy listings, yet no established adaptation strategy is documented. The paper develops a backbone-agnostic adaptation recipe and evaluation suite, finding substantial in-domain gains while preserving broad capabilities. A production-style case study further reports strong quality gains and multiple-times-faster inference than general-purpose VLMs.
Problem
Existing general-purpose VLMs lack a documented strategy for e-commerce’s attribute-centric, multi-image, and noisy data, while multimodal evaluation remains insufficiently tailored to retail.
Method
The paper builds verified e-commerce instruction data, adapts open-weight VLMs through extensive backbone ablations, and evaluates them with four production-oriented multimodal benchmark suites.
Results
Targeted adaptation delivers substantial in-domain gains while preserving broad capabilities and improving on out-of-distribution e-commerce data.
Takeaways & Limitations
A reproducible, backbone-agnostic strategy can adapt open-weight VLMs to e-commerce while retaining strong general capabilities.
Takeaways & Limitations
Evaluation is not exhaustive: the approximately 1k-example Dynamic Attribute Extraction set and selected category coverage constrain evidence for long-tail, rare, regional, and atypical cases.
Abstract
from arXiv · showhide
E-commerce product understanding demands by nature, strong multimodal comprehension from text, images, and structured attributes. General-purpose Vision-Language Models (VLMs) enable generalizable multimodal latent modelling, yet there is no documented, well-known strategy for adapting them to the attribute-centric, multi-image, and noisy nature of e-commerce data, without sacrificing general performance. In this work, we show through a large-scale experimental study, how targeted adaptation of general VLMs can substantially improve e-commerce performance while preserving broad multimodal capabilities. Furthermore, we propose a novel extensive evaluation suite covering deep product understanding, strict instruction following, and dynamic attribute extraction.
1 Introduction
E-commerce understanding requires multimodal reasoning over attribute-centric, multi-image, and noisy seller content, but adapting general VLMs to this setting remains insufficiently documented. The paper studies targeted adaptation and proposes benchmarks designed to improve e-commerce performance while retaining general capabilities.
- E-commerce product understanding combines listing text, item images, packaging, and general visuals.
- General-purpose VLMs perform strongly across broad multimodal applications, yet lack a reproducible recipe for e-commerce-specific reasoning and noisy content.
- The study tests whether customized language models are necessary by running ablations across multiple visual and text decoder backbones.
- The authors introduce multimodal instruction data intended to strengthen e-commerce abilities without hindering general performance.
- Targeted VLM adaptation considerably improves online-shopping performance without loss of capabilities on other domains.
- The work contributes production-oriented vision-e-commerce benchmarks and evaluates adaptation across data mixtures, model sizes, and architectures.
2 Related Work
Prior e-commerce evaluations primarily emphasize text, while general VLM suites target broad visual-language abilities rather than fine-grained retail attributes and tool use. This motivates multimodal benchmarks tailored to production e-commerce scenarios.
- Existing e-commerce suites measure shop-assistant abilities and product-related tasks mainly through textual signals.
- General-purpose VLM evaluations cover broad tasks such as visual question answering and object recognition, not retail-specific attributes and tool use.
3 Methodology
The methodology combines production-oriented multimodal benchmarks, verified e-commerce instruction data, staged VLM training, and specialized multi-image fine-tuning. It evaluates attribute prediction, fashion understanding, dynamic extraction, and compliance-focused item intelligence.
- Our E-commerce Benchmarks: Four evaluation suites cover Aspect Prediction, Deep Fashion Understanding, Dynamic Attribute Extraction, and Multi-image Item Intelligence.
- Our E-commerce Benchmarks: Aspect Prediction includes 2600 general e-commerce questions and two Fashion subsets with 1600 examples each, evaluated through string matching.
- Our E-commerce Benchmarks: Deep Fashion Understanding contains 3000 samples spanning shirts and tops, handbags, and sneakers, with category-specific attribute classes.
- Our E-commerce Benchmarks: Dynamic Attribute Extraction uses 1,000 human-verified synthetic examples to test schema-free enumeration and structuring of visually grounded attributes.
- Our E-commerce Benchmarks: Multi-image Item Intelligence compiles compliance attributes from multiple product images into structured JSON for verification and recall matching.
- Data Curation: The visual verification pipeline collects nearly 15 million listings, captions images with InternVL-2.5-26B, and uses Mistral-Small-3-24B to verify textual-image correspondence.
- VLM Adaptation Training Stages: Training follows Vision-Language Alignment, Mid-Stage Training, and Visual Instruction Tuning, including an approximately 4M internal e-commerce instruction set.
- Multi-Image Item Intelligence: Multi-image fine-tuning uses 100,000 items with a median of 5 images, while targeted crops and GPT-4.1 re-annotation improve label quality.
4 Experiments
The experiments compare adapted and external VLMs across e-commerce, public multimodal, multi-image, and item-intelligence settings. They show that targeted adaptation, decoder choices, fine-tuning, and targeted cropping materially affect quality, efficiency, and generalization.
- 4.2 Comparison against existing VLMs: Internal adaptation outperforms external VLMs on most e-commerce-specific benchmarks, despite newer external models leading on many general-domain benchmarks.The comparison motivates building customized VLMs for relevant e-commerce tasks.
- 4.3 Importance of Vision Encoder: Vision-encoder comparisons are inconclusive across low- to mid-resolution e-commerce and general-domain benchmarks, with task characteristics influencing which architecture helps.Qwen2.5ViT’s native high-resolution capability may benefit small-detail attribute prediction, while SigLIP2 performs better in some reasoning-oriented settings.
- 4.4 Importance of Text-Decoder: E-commerce knowledge improves e-commerce adaptability, while stronger general-domain text-decoder capabilities also improve general benchmarks and some e-commerce tasks, especially Aspect Prediction.The study also reports a consistent advantage for larger decoders until task-dependent size thresholds are reached.
- 4.5 Public E-commerce Benchmarking: +11% and +3%/+11% gains are reported for internal adapted models over open-source counterparts on public e-commerce benchmarks.The results indicate that e-commerce adaptation generalizes beyond the internal evaluation sets.
- 4.5 Public E-commerce Benchmarking: Single-image training generalizes to multi-image e-commerce evaluation, while decoder size and prior e-commerce training substantially affect Multi-Image E-comMMMU performance.Using e-commerce-pretrained LLMs produces respective increases of 5% and 12% in the cited comparisons.
- 4.6 Item Intelligence: Access to all listing images improves Item Intelligence predictions, while targeted bounding-box cropping improves labels and outperforms both no cropping and Gemma3’s Pan & Scan approach.The efficient Gemma3-4B model is worse zero-shot, but targeted cropping improves performance across model sizes.
- 4.6 Item Intelligence: Fine-tuning improves Item Intelligence quality and enables the smaller Gemma3-4B to achieve strong performance with approximately 3.8x faster inference than fine-tuned Gemma3-27B replacement.The cited comparison also reports improved F1 Score for the fine-tuned Gemma3-4B model.
5 Conclusion
The paper presents a reproducible, backbone-agnostic recipe for adapting open-weight VLMs to e-commerce’s attribute-centric, multi-image, and noisy data. Across benchmarks and a production-style Item Intelligence case study, targeted adaptation improves in-domain and out-of-distribution e-commerce performance while preserving broad capabilities and accelerating inference.
- 5 Conclusion: The recipe targets attribute-centric, multi-image, and noisy e-commerce characteristics through reproducible adaptation of open-weight VLMs.The benchmark suite covers Aspect Prediction, Deep Fashion Understanding, Dynamic Attribute Extraction, and multi-image Item Intelligence.
- 5 Conclusion: Targeted cropping, improved labels, and fine-tuning yield strong quality gains and multiple-times-faster inference than general-purpose VLMs in production-style Item Intelligence.The conclusion presents this case study as evidence of practical benefits from task-specific adaptation.
6 Limitations
The study identifies limitations involving language scope, platform dependence, LLM-mediated supervision, evaluation coverage, and long image sequences. These constraints bound how broadly the reported improvements should be interpreted.
- All adaptation, supervision, and evaluation were conducted in English, leaving cross-lingual transfer to locale-specific product conventions uncharacterized.The uncharacterized conventions include multilingual OCR, regional sizing, and currency or decimal formats.
- The instruction corpus and benchmarks predominantly come from one marketplace, making portability to platforms with different schemas or listing norms uncertain.Potential coupling includes platform taxonomy, seller conventions, imaging styles, and metadata density.
- LLM-based supervision and evaluation introduce annotator, style, and measurement biases, while evaluator–model overlap can distort judged gains.The passage specifically notes possible inflation or deflation from inductive-bias alignment in LLM-as-judge settings.
- The evaluation under-constrains long-tail categories, rare attributes, regional variants, composited images, and atypical listing styles.The Dynamic Attribute Extraction set contains approximately 1,000 examples and emphasizes selected fashion and high-volume verticals.
- More than 10 images can cause out-of-memory issues and long inference times, creating a production-use boundary for multi-image benchmarks.The authors describe such sequences as rare and suggest larger-context models or token-efficient strategies as future remedies.
A.1 Related Work (Continued)
Related work combines general multimodal model development with domain adaptation and increasingly rigorous benchmarking. The paper positions its contribution within the need for evaluation tailored to e-commerce multimodal performance.
- Multimodal systems have advanced by combining CLIP vision encoders with large language models for cross-modal understanding.
- General-domain pretrained language models often struggle with domain-specific tasks, motivating domain-specific pretraining or targeted adaptation.
- Pretraining a domain-specific language model from scratch offers extensive customization but is costly, slow, and data-intensive.
- Continuous in-domain pretraining and fine-tuning existing models can improve domain-specific performance, with less overall customizability.
- VLM benchmarking spans image reasoning, knowledge acquisition, perception, and vision-centric analysis, motivating domain-specific evaluation frameworks.
A.2 General Domain Multimodal Benchmarks
The paper evaluates e-commerce models against eComMMMU and general multimodal benchmarks. This pairing monitors online-shopping performance while checking broader multimodal capabilities.
- eComMMMU contains over 35k multi-image samples spanning more than eight tasks for existing e-commerce evaluation.
- The evaluation also uses eight general multimodal understanding benchmarks to monitor general-domain performance.These benchmarks include MMBench, which covers object detection, text recognition, and action recognition among other abilities.
- Figure 4 presents representative benchmark examples from Aspect Prediction and Deep Fashion Understanding to illustrate those tasks.
A.4 Our E-commerce Benchmarks
The e-commerce benchmark suite covers aspect prediction, multi-image item intelligence, deep fashion understanding, and dynamic attribute extraction. These tasks test fixed and open-ended product understanding across varied categories and attributes.
- Aspect Prediction: Aspect Prediction evaluates 2,600 general questions plus two fashion subsets of 1,600 examples with and without item-title and category context.All three parts are evaluated through string matching.
- Multi-Image Item Intelligence: Multi-Image Item Intelligence extracts and normalizes visible product information into consistent JSON for verification and recall matching.Its schema includes product identifiers, attributes, origin, and regulatory safety information.
- Deep Fashion Understanding: Deep Fashion Understanding contains 3,000 samples across shirts, women’s tops, handbags, and sneakers, targeting product-specific attribute categories.Examples include sleeve length, neckline, pattern, color, and brand labels.
- Dynamic Attribute Extraction: Dynamic Attribute Extraction uses 1,000 human-verified synthetic examples to enumerate visually grounded attributes without a predefined schema.Each instance requires selecting salient properties, naming attributes, and serializing values as key–value pairs while avoiding unsupported fields.
A.5 Item Intelligence Fine-tuning
The section addresses the computational cost of multi-image inference and lists components of the LLaVA-OneVision fine-tuning data configuration. Inference cost grows with the number of images because Gemma-3 assigns 256 visual tokens to each image.
- Inference efficiency: 256 visual tokens per image make inference cost scale linearly with the number of images, including small images.Using original images together with all derived crops is computationally expensive.
- Inference efficiency: 12 median and 43 maximum images per item characterize the training dataset's multi-image input scale.These counts motivate reducing the number of images processed during inference.
- Fine-tuning data: The configuration includes LLaVA-ReCap-558K and LLaVA-ReCap-118K among the listed fine-tuning data sources.The passages identify these sources through their JSON paths, without describing their contents.
- Fine-tuning data: The listed visual instruction-tuning configuration includes datasets for chart, diagram, infographic, OCR, mathematics, geometry, and visual question answering.Sampling strategies vary across datasets, including all-data selection and first-percentage subsets.