Source-linked AI summary
HMGCLIP: Heterogeneous Multi-Granularity Contrastive Learning for E-commerce Representation Learning
Qiuyu Zhu, Yi Gao, Zhichao Wan, Mingyang Ma
TL;DR
Fine-grained e-commerce tasks require representations that preserve attribute distinctions beyond global product similarity. HMGCLIP constructs a heterogeneous hypergraph, mines structure-aware hard negatives, and aligns semantics at relation and hyperedge levels. Experiments on the introduced dataset and MAVE show state-of-the-art performance across fine-grained and coarse-grained tasks.
Problem
Global embeddings from general multimodal models can miss fine-grained attribute distinctions needed to discriminate visually similar e-commerce products.
Method
HMGCLIP constructs a heterogeneous hypergraph, mines multi-granularity structure-aware hard negatives, and aligns relation-level and hyperedge-level semantics in a unified embedding space.
Results
HMGCLIP achieves state-of-the-art performance across fine-grained and coarse-grained tasks on the introduced dataset and the public MAVE benchmark.
Takeaways & Limitations
The framework provides a unified multimodal embedding approach that generalizes across downstream tasks without task-specific encoder fine-tuning.
Abstract
from arXiv · showhide
Although recent Multimodal Large Language Models (MLLMs) have advanced general product understanding, they implicitly encode product information into global embeddings, thereby limiting their ability to capture fine-grained attributes. This limitation hinders performance in tasks requiring precise attribute discrimination, such as distinguishing subtle material differences among visually similar products. To address this challenge, we propose HMGCLIP, a unified multimodal embedding framework. By constructing a heterogeneous hypergraph, we leverage hypergraph topology to mine structure-aware hard negatives and align multi-granular semantics at both relation and hyperedge levels. This design enables a dual-granularity inference mechanism that dynamically fuses attribute evidence for both fine-grained and coarse-grained downstream tasks. Furthermore, we release a comprehensive fine-grained e-commerce dataset to facilitate future benchmarking. Extensive experiments on this new dataset and the public MAVE benchmark show that HMGCLIP outperforms strong multimodal encoders, MLLMs, and e-commerce baselines, validating the superiority of HMGCLIP.
1 Introduction
HMGCLIP addresses fine-grained e-commerce representation challenges by using heterogeneous structure and multi-granularity contrastive learning. It introduces a unified framework and dataset, with experiments showing strong performance across attribute prediction and product classification.
- General vision-language models struggle to distinguish fine-grained e-commerce attributes such as mesh versus leather or air-cushion versus flat-rubber soles.
- Standard contrastive learning often samples negatives uniformly or randomly, missing semantically confusable hard negatives needed for attribute-level decision boundaries.
- Global visual similarity can rank semantically incorrect products highly when decisive evidence such as material texture and outsole appearance is overlooked.
- HMGCLIP models e-commerce data as a heterogeneous hypergraph and mines hard negatives across coarse category-level and fine attribute-level granularities.
- Its unified multimodal embedding framework supports fine-grained and coarse-grained tasks without task-specific encoder fine-tuning.
- Extensive experiments on the introduced dataset and public MAVE benchmark show strong and often state-of-the-art performance across attribute prediction and product classification.
2 Related Work
Prior work provides transferable vision-language representations, hard-negative strategies, and structured e-commerce modeling, but rarely unifies these elements in one embedding framework. HMGCLIP targets this integration gap.
- Vision-language pre-training has produced transferable representations for retrieval and recognition, while e-commerce adaptations use domain-specific contrastive pretraining and multimodal fusion.
- Recent MLLM-based e-commerce models improve generative product understanding but are often optimized for task-specific prediction rather than reusable embedding spaces.
- Contrastive learning methods improve representation quality through label-aware positives, prototypes, graph augmentations, and hard-negative mining.
- E-commerce negatives are constrained by category hierarchies, attribute relations, and visual similarity, while structured product relations support retrieval, recommendation, and attribute extraction.
- Existing methods rarely combine relation-level hard negatives, hyperedge-level semantic alignment, and multimodal product representations within one embedding framework.
3 Preliminaries
E-commerce products are compositional entities represented through images, text, structured attributes, and categories at different semantic granularities. The paper learns unified embeddings evaluated through embedding-based attribute and category matching.
- Products combine images, titles and descriptions, structured attribute–value pairs, and categories as heterogeneous multimodal inputs.
- These modalities encode semantics ranging from fine-grained physical properties to coarse-grained product roles in platform taxonomies.
- The goal is to learn a unified multimodal embedding function that captures fine-grained semantics while preserving discriminative boundaries.
- Representations are evaluated on attribute prediction for fine-grained discrimination and product classification for semantic clustering quality.
- Classification is formulated as embedding-based matching, selecting the candidate aspect or category with the highest semantic similarity to the query product.
4 Methodology
HMGCLIP comprises three stages: heterogeneous hypergraph semantic construction, multi-granularity contrastive representation learning, and dual-granularity inference. Together, they organize structured semantics, align representations, and support attribute prediction with evidence fusion.
- Heterogeneous Hypergraph Semantic Construction: Heterogeneous hypergraph semantic construction organizes products, aspects, and categories into a structured graph with pairwise relations and higher-order hyperedges.
- Multi-Granularity Contrastive Representation Learning: Multi-granularity contrastive representation learning aligns entities at node, relation, and hyperedge levels in a unified semantic space.
- Dual-Granularity Inference: Dual-granularity inference performs attribute prediction and evidence-fused downstream inference.
4.1 Heterogeneous Hypergraph Semantic Construction
HMGCLIP represents e-commerce structure with a heterogeneous hypergraph connecting products, aspects, and categories through pairwise relations and higher-order hyperedges.
- The heterogeneous graph uses product, aspect, and category nodes connected by product–aspect and product–category edges.
- Products sharing an aspect can form coherent clusters across different categories, which pairwise relations alone cannot capture.
- Aspect-level hyperedges group each aspect with all associated products to capture fine-grained semantic consistency.
- Category-level hyperedges group each category with its member products to preserve coarse-grained semantic coherence.
- Combining pairwise edges with aspect- and category-level hyperedges provides the structural foundation for subsequent contrastive learning.
4.2 Multi-Granularity Contrastive Learning
HMGCLIP learns fine-grained representations by combining relation-level and hyperedge-level contrastive views with structure-aware sampling of semantically confusable negatives.
- Multi-granularity contrastive learning aligns relation-level and hyperedge-level views in a unified embedding space.The approach uses e-commerce-specific structural priors while fine-tuning a pretrained multimodal encoder.
- Structure-aware sampling constructs informative positive and negative pairs for pairwise relations and higher-order semantic groups.
- Figure 2 places hypergraph construction, multi-granularity contrastive learning, and dual-granularity inference in one HMGCLIP pipeline.
- Hard negatives share an aspect key and category context with the anchor product but are not ground-truth aspects.This produces semantically confusable product–aspect negatives that sharpen distinctions among directly related aspects.
- Hyperedge alignment treats co-occurring products within an anchor’s hyperedge as positives and products outside it as negatives.The objective increases anchor–positive similarity and decreases anchor–negative similarity to form coherent semantic clusters.
4.3 Joint Relation-Level and Hyperedge-Level Contrastive Optimization
HMGCLIP jointly optimizes relation-level and hyperedge-level contrastive supervision using an InfoNCE-based objective for pairwise and group semantics.
- InfoNCE provides the basic optimization framework for the unified contrastive objective.
- For each anchor sample, the loss distinguishes positive samples from negative samples.
- The loss is formulated over a batch of N samples with τ as the temperature parameter.
- The objective applies the loss to both relation-level and hyperedge-level views.Relation-level loss uses product–aspect pairs with hard negatives, whereas hyperedge-level loss uses product groups within semantic hyperedges.
- λ balances the contributions of the relation-level and hyperedge-level supervision signals.
4.4 Dual-Granularity Inference
HMGCLIP uses two inference paths that adapt the learned embedding space to fine-grained aspect retrieval and coarse-grained category retrieval. The coarse-grained path fuses retrieved aspect evidence with the product representation through semantic-anchor and residual Transformer-based mechanisms.
- Path I: Fine-Grained Retrieval: Fine-grained inference retrieves the best-matching aspect by maximizing cosine similarity between the query embedding and aspect candidates.This non-parametric search uses the multimodal embedding directly and requires no additional classification head.
- Path II: Evidence-Fused Coarse-Grained Retrieval: Coarse-grained inference enriches the product representation with semantic evidence to reduce ambiguity among semantically similar categories.The enhanced representation is subsequently matched against taxonomy-defined category candidates.
- Path II: Evidence-Fused Coarse-Grained Retrieval: Semantic anchor fusion mean-pools the top-K retrieved aspect embeddings and linearly interpolates their aggregate with the product’s intrinsic multimodal embedding.The interpolation balances intrinsic features and aggregated semantic evidence before category retrieval.
- Path II: Evidence-Fused Coarse-Grained Retrieval: Residual Transformer fusion combines the product token with aspect-evidence tokens, adds type embeddings, and models their interactions with a lightweight Transformer encoder.This interaction-aware fusion goes beyond mean pooling or scalar weighting.
- Path II: Evidence-Fused Coarse-Grained Retrieval: The final residual representation preserves the original product embedding as a semantic anchor while applying a controlled aspect-aware correction.L2 normalization keeps the fused representation compatible with the learned embedding space for taxonomy matching.
5 Experiments and Analysis
HMGCLIP achieves strong performance on fine-grained aspect prediction and coarse-grained product classification across the internal dataset and MAVE. Qualitative results and evidence-fusion analyses show improved discrimination through multi-granular semantic alignment and carefully weighted aspect evidence.
- Aspect Prediction: 75.38% Hit@1/MRR@1 on the internal dataset surpasses Qwen3-VL-Emb by 21.28% for aspect prediction.HMGCLIP also reaches 97.61% Hit@5 and generalizes to MAVE with 24.61% Hit@1/MRR@1 and 36.69% MRR@5.
- Aspect Prediction: HMGCLIP’s graph-derived hard negatives expose semantically confusable attribute values and sharpen product–aspect decision boundaries.This supports fine-grained semantic capture beyond global visual similarity.
- Product Classification: 84.23% Hit@1 on the internal dataset exceeds Qwen3-VL-Emb’s 48.30% for product classification.On MAVE, HMGCLIP achieves 96.70% Hit@1 and 99.57% Hit@5, outperforming FashionCLIP’s 75.66% Hit@1.
- Evidence Fusion: Uniform mean pooling slightly improves Hit@1 but degrades Hit@3 and Hit@5, indicating that indiscriminate aspect aggregation can disturb product-embedding neighborhoods.Preserving the product representation as an anchor and using instance-adaptive weighting improves evidence fusion because products rely on different evidence subsets.
- Qualitative Analysis: Qualitative comparisons show correct retrieval of ambiguous material and structural attributes where Qwen3-VL-Emb fails.Examples include plastic versus brass, polyester versus vinyl, and button versus side slit.
- Qualitative Analysis: Residual transformer fusion injects retrieved aspect evidence into classification, improving disambiguation among products with overlapping contextual features.HMGCLIP correctly predicts specific sub-categories such as Electrical Circuitry & Parts and Power Tools Parts.
6 Conclusion
HMGCLIP is a unified multimodal embedding framework that uses heterogeneous hypergraph structure and multi-granularity contrastive learning for e-commerce representation learning. It generalizes across fine-grained and coarse-grained tasks without task-specific fine-tuning and achieves state-of-the-art performance on both evaluated datasets.
- Conclusion: HMGCLIP aligns relation-level and hyperedge-level semantics within a unified multimodal embedding space.The framework uses a heterogeneous hypergraph as a structural prior for multi-granularity representation learning.
- Conclusion: HMGCLIP’s dual-granularity inference generalizes across fine-grained and coarse-grained tasks without task-specific fine-tuning.The paper also releases a comprehensive multimodal e-commerce dataset.
- Conclusion: Experiments on both datasets show state-of-the-art performance in aspect prediction and product classification.The conclusion characterizes the method as effective and versatile across both tasks.