PaperlayerView exact revision on arXiv

Source-linked AI summary

Kaleido-BERT: Vision-Language Pre-training on Fashion Domain

Mingchen Zhuge, Dehong Gao, Deng-Ping Fan, Linbo Jin, Ben Chen, Haoming Zhou, Minghui Qiu, Ling Shao

arXiv:2103.16110v3cs.CV

TL;DR

Fashion tasks need fine-grained image-text representations, beyond the coarse matching emphasized by general VL models. Kaleido-BERT combines multi-grained patches with alignment-guided pre-training, and the authors report it outperforms fixed-patch or RoI-based VL models in fashion.

  • Problem

    General VL models focus on coarse matching, while fashion tasks require fine-grained representations for attributes such as sleeve type and fabric.

    Source: §1 · p. 2

  • Method

    Kaleido-BERT generates multi-grained patches, pre-aligns patches with text tokens, and uses alignment-guided masking alongside five patch-level pre-training tasks.

    Source: §1 · p. 2

  • Results

    Kaleido-BERT outperforms fixed-patch or RoI-based VL models in the fashion domain.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present Kaleido-BERT as a scalable framework for fine-grained fashion representation learning that bridges image-text semantic gaps.

    Source: §1 · p. 2

Abstract

from arXiv · show

We present a new vision-language (VL) pre-training model dubbed Kaleido-BERT, which introduces a novel kaleido strategy for fashion cross-modality representations from transformers. In contrast to random masking strategy of recent VL models, we design alignment guided masking to jointly focus more on image-text semantic relations. To this end, we carry out five novel tasks, i.e., rotation, jigsaw, camouflage, grey-to-color, and blank-to-color for self-supervised VL pre-training at patches of different scale. Kaleido-BERT is conceptually simple and easy to extend to the existing BERT framework, it attains new state-of-the-art results by large margins on four downstream tasks, including text retrieval (R@1: 4.03% absolute improvement), image retrieval (R@1: 7.13% abs imv.), category recognition (ACC: 3.28% abs imv.), and fashion captioning (Bleu4: 1.2 abs imv.). We validate the efficiency of Kaleido-BERT on a wide range of e-commerical websites, demonstrating its broader potential in real-world applications.

1. Introduction

Fashion tasks require fine-grained image-text representations, whereas general-domain VL models focus on coarse matching. Kaleido-BERT combines multi-grained patches, pre-alignment, and alignment-guided masking for fashion pre-training.

1. Introduction

  • General VL models are suboptimal for fashion tasks that need fine-grained attributes such as sleeve type and fabric rather than coarse what-and-where matching.Attribute-aware search and catalog tasks require fine-grained features or similarities.Source: §1 · p. 2
  • Kaleido-BERT uses multi-grained image patches and attention-based pre-alignments between patches and text tokens to guide masking and learn cross-modal semantics.The framework is designed to alleviate coarse representations and bridge semantic gaps between modalities.Source: §1 · p. 2
  • Five tasks—rotation, jigsaw, camouflage, grey-to-color, and blank-to-color—support fine-grained cross-modal learning, outperforming fixed-patch or RoI-based VL models in fashion.The kaleido patch generator produces multi-grained features.Source: §1 · p. 2
  • Alignment-guided masking explicitly targets semantic connections between vision and language, and experiments show the importance of both pre-alignment and alignment masking.Source: §1 · p. 2

2. Related Work

Related work spans transformer-based vision-language pre-training and fashion-focused representation learning. Kaleido-BERT is positioned as a patch-variant approach with alignment-guided masking for fine-grained fashion tasks.

2. Related Work

  • Transformer-based vision-language research covers many modeling approaches, with the section highlighting cross-modal literature and summarizing representative methods in Table 1.Source: §2 · p. 2

2.1. Vision-Language Pre-training

  • Video-text pre-training includes pioneering CBT and VideoBERT, while ActBERT and HERO emphasize downstream applications and UniVL addresses understanding and generation.Source: §2.1 · pp. 2–3
  • Image-text models differ in stream architecture, while VirTex learns visual representations from dense captions and Pixel-BERT uses pixel-level rather than region-based features.Kaleido-BERT is described as masking at the embedding level, unlike task-level LXMERT and UNITER or input-level OSCAR.Source: §2.1 · p. 3

2.2. Fashion-Based Task

  • Prior fashion VL work includes FashionBERT's fixed-patch masking and MAAF's image-level attention, which the authors argue restrict fine-grained representation learning.Source: §2.2 · p. 3
  • Kaleido-BERT is presented as the first fashion-domain study to demonstrate alignment-guided masking focused jointly on image-text coherence.Source: §2.2 · p. 3

3. Proposed Kaleido-BERT

Kaleido-BERT combines multi-grained image patches, token–patch pre-alignment, and alignment-guided masking in a transformer for fine-grained fashion vision-language representations. It trains this model with standard cross-modal objectives and five patch-level tasks spanning spatial structure, classification, and image generation.

3.1. Model Overview

  • Kaleido-BERT processes text and generated image patches through five stages, adding semantic pre-alignment, alignment-guided masking, cross-modal interaction, and five kaleido tasks alongside language modeling and image-text matching.The model adopts a standard NLP transformer to learn fine-grained fashion vision-language features.Source: §3 · p. 3, §3.1 · p. 4

3.2. Kaleido Patch Generator

  • KPG uses a saliency detection network to obtain a foreground mask and lock the domain object, then splits the image at scales 1×1 through 5×5, yielding 55 patches per input image.More detailed divisions, such as 6×6 or N×N, can be considered according to the difficulty of the specific task.Source: §3.2 · p. 4

3.3. Attention-based Alignment Generator

  • AAG infers coarse token–patch alignments from generated text-token attention maps, token co-occurrence, and overlap between image regions and kaleido patches.It uses an image-captioning text generator to produce the attention heat map.Source: §3.3 · p. 4

3.4. Alignment Guided Masking

  • AGM prioritizes masking pre-aligned token–patch pairs and masks either member, then independently masks unaligned candidates if more are needed.It is applied at kaleido levels 3–5, not levels 1–2, because masking larger patches increases modeling difficulty.Source: §3.4 · pp. 4–5

3.5. Cross-Modality Transformer

  • Text tokens and image patches are projected into a shared embedding space, with image embeddings combining segment, image-feature, and position projections before layer normalization.Text embeddings combine token, segment, and position embeddings, followed by layer normalization.Source: §3.5 · p. 5

3.6. Pre-training Kaleido-BERT

  • Pre-training combines AMLM, ITM, and AKPM, with AMLM predicting masked words from surrounding text and image patches and ITM matching paired product text and images.ITM uses same-product examples as positive pairs and different-product examples as negative pairs.Source: §3.6 · p. 5
  • AKPM assigns rotation recognition, jigsaw permutation classification, and camouflage detection to spatial-context and classification objectives across kaleido patches.Rotation recognition predicts one of four rotations for a 1×1 patch; jigsaw solving classifies the permutations of unordered 2×2 patches, and camouflage prediction identifies a replaced patch among 3×3 patches.Source: §3.6 · pp. 5–6
  • AKPM also reconstructs greyed or blanked patches toward color targets, and its losses join AMLM and ITM in the total objective.The paper reports that validation losses decay smoothly, indicating the pre-training tasks are learned with Kaleido-BERT.Source: §3.6 · p. 6

4. Experiments

Kaleido-BERT is evaluated on four fashion vision-language tasks using Fashion-Gen, with comparisons against competing models and ablations of its key components. Results emphasize broad downstream gains and the contributions of multi-grained patches, alignment-guided masking, and staged pre-training tasks.

4.1. Pre-training Settings

  • The model is pre-trained on Fashion-Gen’s 260,480 training image-text pairs and evaluated on four downstream tasks, using the same settings as Top-1 FashionBERT.Fashion-Gen contains 67,666 products; each has one to six images, and the stated test split contains 35,528 pairs.Source: §4.1 · p. 6, §4 · p. 6, §4.2 · p. 7

4.2. Downstream Tasks

  • Retrieval uses same-subcategory examples, category recognition classifies product labels, and fashion captioning evaluates cross-modality generation.For image-text retrieval, unmatched pairs from the same subcategory serve as negatives; text-image retrieval ranks images against titles using 100 unrelated captions from other products in the same subcategory.Source: §4.2 · p. 7

4.3. Competing Models

  • Kaleido-BERT improves nearly all downstream evaluations; FashionBERT also outperforms ViLBERT and VLBERT, while Kaleido-BERT adds alignment masking and kaleido pre-training tasks.The comparisons cover retrieval, category recognition, and fashion captioning; the supplied table captions identify the retrieval and category/captioning result tables but provide no cell values.Source: §4.3 · p. 7, Table 2 · p. 7, Table 3 · p. 7

4.4. Ablation Study

  • Scheme-2 improves over fixed-scale Scheme-1 by +9.7%, +0.5%, and +4.0% on the reported metrics; Scheme-3 improves by 16.2%, +1.5%, and +9.7%.Scheme-1 scores are 276.97 Sum R, 313.43 Sum CLS, and 76.6 Sum CAP; the passage does not map each relative gain to a specific metric.Source: §4.4 · p. 8
  • Against random masking, AGM obtains +6.7%, +1.2%, and 6.9% improvements, and its masking is more semantically related.The passage attributes this comparison to the alignment-guided masking strategy.Source: §4.4 · p. 8
  • Adding AKPM sub-tasks sequentially improves performance; B+V yields +0.9% Sum CLS but -2.7% SumR, while replacing V with I improves all three reported downstream tasks.Replacing V with I gives +0.5%, +0.4%, and +1.4%, respectively; the authors argue V contributes when comprehensive image embeddings have been learned from I–IV.Source: §4.4 · p. 8

5. Conclusion

Kaleido-BERT is a fashion vision-language pre-training architecture comprising a kaleido patches generator, attention-based alignment generator, and alignment-guided masking. The authors report improved efficiency and new state-of-the-art performance on several downstream tasks.

5. Conclusion

  • Kaleido-BERT combines a kaleido patches generator, attention-based alignment generator, and alignment-guided masking strategy for fashion tasks.The components cooperate to learn intra-modal and inter-modal image-text feature embeddings.Source: §5 · p. 8
  • The components are described as easy to implement and as cooperating to learn image-text feature embeddings.The conclusion specifies both intra-modal and inter-modal embeddings.Source: §5 · p. 8
  • The authors report new state-of-the-art performance and improved accuracy on Image-Text Retrieval, Category Recognition, and Fashion Captioning.The conclusion also describes the model as more efficient than existing models.Source: §5 · p. 8

Related papers

Loading 2103.16110v3…