Source-linked AI summary
FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrieval
Dehong Gao, Linbo Jin, Ben Chen, Minghui Qiu, Peng Li, Yi Wei, Yi Hu, Hao Wang
TL;DR
FashionBERT addresses the mismatch between fine-grained fashion descriptions and insufficiently detailed RoI image representations. It uses image patches as BERT image tokens with adaptive multitask loss, and outperforms prior approaches on Fashion-Gen while being applied in e-commerce retrieval. The paper also identifies remaining room for finer-grained modeling and faster inference.
Problem
Fashion text–image matching requires fine-grained visual information, but RoI representations capture insufficiently detailed object-level information for fashion descriptions.
Method
FashionBERT feeds image patches and text tokens into a pretrained BERT backbone and adaptively weights Masked Language Modeling, Masked Patch Modeling, and Text&Image Alignment.
Results
FashionBERT significantly outperforms SOTA and pioneer approaches on Fashion-Gen across Text&Image alignment classification and cross-modal retrieval.
Takeaways & Limitations
The patch method and adaptive algorithm are task-agnostic, and FashionBERT has been applied to a concrete e-commerce cross-modal retrieval application.
Takeaways & Limitations
The authors report remaining room for improvement in fashion-domain matching and seek finer-grained tasks, model understanding, and faster online inference.
Abstract
from arXiv · showhide
In this paper, we address the text and image matching in cross-modal retrieval of the fashion industry. Different from the matching in the general domain, the fashion matching is required to pay much more attention to the fine-grained information in the fashion images and texts. Pioneer approaches detect the region of interests (i.e., RoIs) from images and use the RoI embeddings as image representations. In general, RoIs tend to represent the "object-level" information in the fashion images, while fashion texts are prone to describe more detailed information, e.g. styles, attributes. RoIs are thus not fine-grained enough for fashion text and image matching. To this end, we propose FashionBERT, which leverages patches as image features. With the pre-trained BERT model as the backbone network, FashionBERT learns high level representations of texts and images. Meanwhile, we propose an adaptive loss to trade off multitask learning in the FashionBERT modeling. Two tasks (i.e., text and image matching and cross-modal retrieval) are incorporated to evaluate FashionBERT. On the public dataset, experiments demonstrate FashionBERT achieves significant improvements in performances than the baseline and state-of-the-art approaches. In practice, FashionBERT is applied in a concrete cross-modal retrieval application. We provide the detailed matching performance and inference efficiency analysis.
1 Introduction
Fashion text–image matching is difficult because fashion descriptions are fine-grained while detected RoIs provide sparse, object-level image information. FashionBERT addresses this mismatch with image patches, BERT-based multimodal modeling, and adaptive multitask loss, achieving strong benchmark performance and practical deployment.
- Motivation: Cross-modal retrieval learns matching between one modality used as a query and relevant data in another modality.The paper places text–image matching within broader multimedia retrieval research.
- Motivation: Fashion images yield fewer detected RoIs than general-domain images, averaging 6.4 for Fashion-Gen versus 19.8 for MSCOCO.The paper attributes this difference to the characteristics of fashion images and connects it to the difficulty of matching fine-grained descriptions.
- Motivation: Object-level RoIs are not fine-grained enough to match fashion texts that describe detailed styles and attributes.This limitation motivates replacing RoI-based image tokens with a finer-grained representation.
- FashionBERT: FashionBERT splits each image into ordered, non-repeated patches and treats them as image tokens for interaction with text tokens in BERT.Patches retain more raw pixel information than object-level RoIs and are suitable as sequence inputs.
- FashionBERT: Adaptive loss balances three multitask objectives—Masked Language Modeling, Masked Patch Modeling, and Text&Image Alignment—by estimating task weights per batch.The adaptive algorithm treats loss-weight determination as an optimization problem.
- Evaluation and application: FashionBERT is evaluated on Text&Image alignment classification and bidirectional cross-modal retrieval, outperforming SOTA and pioneer approaches on Fashion-Gen.The model was also applied to an e-commerce cross-modal retrieval system, with matching performance and inference efficiency analyzed.
2. Methodology
FashionBERT extends pre-trained BERT to jointly model text tokens and image patches, using three training tasks combined through an adaptive loss. Its architecture lets text and patches interact in Transformer layers while patch masking supplies fine-grained image modeling.
- Image representation: Each fashion image is represented as an ordered sequence of non-repeated patches rather than object-level RoIs.The patch representation preserves raw pixel information and is intended to contain more detailed information than RoIs.
- Cross-modal architecture: FashionBERT uses pre-trained BERT as a cross-modal backbone in which text tokens and image patches interact through multiple self-attention layers.The model outputs final representations for each token or patch.
- Training tasks: FashionBERT jointly trains masked language modeling, masked patch modeling, and text-image alignment as a multitask objective.MLM masks whole words, MPM masks patches and predicts their feature distributions, and TIA classifies whether text and image pairs match.
- Training tasks: MLM predicts masked text tokens from surrounding tokens, while MPM predicts distributions over masked-out patch features.The MLM procedure uses Whole Word Masking with a 15% masking probability; MPM masks patches with a 10% probability.
- Training tasks: TIA feeds the [CLS] representation into a binary classifier that distinguishes matched product text-image pairs from randomly mismatched pairs.The task optimizes binary cross-entropy loss.
- Adaptive loss: An adaptive loss assigns weights to the three task losses by minimizing total weighted loss while treating task learning fairly.The weights are formulated as an optimization problem and solved using KKT conditions; training aggregates the weighted losses and updates parameters with ADAM.
3. Experiments
FashionBERT is evaluated on Fashion-Gen through text-image matching and cross-modal retrieval, with ablations covering adaptive loss, patch feature extraction, and BERT depth. The paper also examines deployment in industrial retrieval, including the trade-off between matching performance and inference speed.
- Experimental Settings: Fashion-Gen provides 67,666 products and 293,008 image-description pairs, with professional stylist descriptions and multiple photographic angles.
- Experimental Settings: Cross-modal retrieval uses 1,000 text and 1,000 image queries, each ranked against one ground-truth match and 100 randomly sampled negatives.Performance is measured with Rank@1, Rank@5, and Rank@10 for text-to-image and image-to-text retrieval.
- Experimental Settings: FashionBERT uses 64 image patches, 2048-dimensional patch features, and a maximum combined sequence length of 512 tokens.ResNeXt101 extracts the patch features, while text occupies the remaining sequence capacity.
- Evaluation of the SOTA and Pioneer Approaches: The experiments compare FashionBERT with VSE, VSE++, PFAN, ViLBERT, and VLBERT baselines or pre-trained approaches.The comparisons test performance against both pioneer matching systems and state-of-the-art pre-trained cross-modal models.
- Ablation Studies: Adaptive loss improves FashionBERT performance, while its learned weights shift attention toward harder MPM and MLM tasks after text-image alignment becomes strong.The ablation compares adaptive loss with uniform weights and reports the task-weight behavior during training.
- Ablation Studies: ResNeXt-101 patch features outperform InceptionV3 features, and deeper BERT encoders improve metrics except Rank@1.The model-size study notes that Rank@1 is more sensitive and that BERT-Large was not tested because of limited resources.
- Industry Applications: Industrial retrieval vectorizes queries and products for nearest-neighbor search, but deployment requires reducing FashionBERT to two layers to balance latency and matching performance.The VSL strategy accelerates inference with little effect on performance, while online inference remains challenging for BERT-like models.
4. Related Work
Prior work applies pre-training and shared representations to text–image matching, but BERT-based approaches commonly use RoIs that are not fine-grained enough for fashion descriptions.
- Pre-training has been adopted across computer vision and natural language processing, supporting downstream multimodal modeling.
- Text–image matching has progressed from shared-space methods such as CCA and VSE to BERT-based approaches using interacting visual and textual inputs.
- BERT-based matching approaches typically detect image RoIs and treat them as image tokens.
- Fashion images often yield relatively few RoIs, making object-level representations insufficiently fine-grained for fashion text and image matching.
5. Conclusions
FashionBERT addresses fashion-domain matching with image patches, BERT-based multimodal encoding, and adaptive multitask loss. The authors report advantages over object-level RoIs and identify further improvement directions.
- FashionBERT splits images into patches and feeds them with text tokens into a BERT backbone for fashion-domain matching.
- An adaptive loss automatically determines task weights and shifts attention between tasks during training.
- The patch method shows advantages over object-level RoIs for matching fashion texts and images.
- Future work targets finer-grained alignment, visualization of matching behavior, and model reduction or distillation for faster inference.
APPENDIX
The appendix outlines a proof for adaptive-loss weighting by applying Lagrange multipliers, deriving intermediate relations, and verifying non-negativity.
- The adaptive-loss weighting proof begins by expressing the learning weights and omitting the non-negative constraint before introducing Lagrange multipliers.
- The derivation obtains a solution through successive intermediate equations referenced as Equations (3.1) and (3.2).
- Substituting the result from Equation (4) into Equation (5) yields the stated solution.
- The resulting value is asserted to be non-negative and to satisfy the non-negative constraint in Equation (1).