Source-linked AI summary
Producing augmentation-invariant embeddings from real-life imagery
Sergio Manuel Papadakis, Sanjay Addicam
TL;DR
The challenge asks systems to match query images to a large reference set despite class imbalance and varied augmentations. The authors train CNN embeddings with ArcFace, combine models, and normalize query embeddings. Their final ensemble achieved µAP = 0.59 and second place in Phase 2.
Problem
The challenge requires matching query images to a much larger reference set despite varied augmentations, while forbidding reference-set training and requiring pair scores independent of other images.
Source: §1 · p. 1Method
The authors trained CNN backbones with ArcFace, ensembled their embeddings through PCA, and normalized query embeddings using the competition training images.
Source: §2.2 · p. 2, §2.6 · p. 5, §3 · p. 8Results
The final ensemble reached µAP = 0.59 and won 2nd place in Phase 2; Method 2 improved µAP from 0.53 to 0.59.
Source: §3 · p. 8, §2.7.2 · p. 8Takeaways & Limitations
With the progressive training scheme, a model with a high number of classes could converge in about an hour.
Source: §2.5.1 · p. 4Training-image query normalization assumes the closest training embeddings are distributed similarly to the closest reference embeddings.
Source: §2.7.2 · p. 7
Abstract
from arXiv · showhide
This article presents an efficient way to produce feature-rich, high-dimensionality embedding spaces from real-life images. The features produced are designed to be independent from augmentations used in real-life cases which appear on social media. Our approach uses convolutional neural networks (CNN) to produce an embedding space. An ArcFace head was used to train the model by employing automatically produced augmentations. Additionally, we present a way to make an ensemble out of different embeddings containing the same semantic information, a way to normalize the resulting embedding using an external dataset, and a novel way to perform quick training of these models with a high number of classes in the ArcFace head. Using this approach we achieved the 2nd place in the 2021 Facebook AI Image Similarity Challenge: Descriptor Track.
1 Introduction
The Facebook Image Similarity Challenge tests whether query images derive from a large reference set, despite severe class imbalance and varied augmentations. This paper summarizes its second-place solution.
1 Introduction
- The benchmark asks systems to identify which of 100,000 query images were built from 1 million reference images, including augmented versions.The training set also contains 1 million images; reference images are prohibited for training.Source: §1 · p. 1
- Challenge constraints include varied query augmentations and pairwise similarity scores that must be independent of other reference and query images.Source: §1 · p. 1
- The description track requires embeddings of at most 256 dimensions and evaluates micro Average Precision from negative squared Euclidean distances.Source: §1 · p. 1
- The paper summarizes the authors’ second-place solution and says its code was made publicly available.Source: §1 · p. 1
2.1 Global approach
The approach addresses severe class imbalance by training on synthetic query images and using ArcFace rather than triplet loss. It combines varied models into a normalized ensemble.
2.1 Global approach
- The authors identify the imbalance between 50,000 query images and 1 million reference images as the challenge’s main problem.Source: §2.1 · p. 1
- They used Augly and newly created augmentations to build synthetic query images from training samples.Source: §2.1 · p. 2
- They chose an ArcFace head after triplet-loss training made finding sufficiently good negative samples computationally difficult and performed poorly.Source: §2.1 · p. 2
- They trained models with different image sizes, embedding dimensions, and ArcFace class counts, then ensembled them and normalized the final space with the competition training dataset.Source: §2.1 · p. 2
- The 25,000 ground-truth samples supplied by the competition were used to validate models during training.Source: §2.1 · p. 2
2.2 Modeling
The modeling pipeline maps augmented training images to L2-normalized embeddings. An ArcFace head trains the models, with augmented images represented as classes.
2.2 Modeling
- Augmented images pass through a CNN backbone, trainable-p GeM pooling, bias-free linear projection, and final L2 normalization to form embeddings.The authors report that including a bias term caused optimization problems.Source: §2.2 · p. 2
- Each ArcFace class corresponds to an augmented training image; models vary the class count, with arc margin 0.4 and softmax scale 40.Source: §2.2 · p. 2
- Table 1 lists all models participating in the final ensemble.Source: Table 1 · p. 2
2.3 External Datasets
The authors added ImageNet and Deepfake Detection data as external training sources, but only ImageNet contributed to the final models.
2.3 External Datasets
- ImageNet supplied 1.43 million additional samples for use with the ArcFace head.Source: §2.3 · p. 2
- Deepfake Detection supplied 476,000 face-containing frames to address face-embedding difficulties, but redundant viewpoints made training harder and did not improve validation µAP, so the data was excluded from final models.Source: §2.3 · p. 2
2.4 Augmentations
The approach uses Augly’s available augmentations and adds three channel transformations observed in query images to create synthetic queries.
2.4 Augmentations
- All augmentations available in Augly were used to build synthetic query images from training samples.The paper describes augmentation as crucial to improving model performance.Source: §2.4 · pp. 2–3
- Figure 2 illustrates the three newly created channel augmentations.Source: Figure 2 · p. 3
- Three additional transformations—channel inversion, channel swapping, and channel shifting—were implemented after inspection of query images.These transformations were not available in Augly.Source: §2.4 · p. 3, Figure 2 · p. 3
2.5 Training method
Training uses augmented images and staged schedules, while Drip Training gradually expands the ArcFace class count and rebuilds its centroids to support larger-class training.
2.5 Training method
- Training uses Adam because tested SGD was slower, and increases augmentation intensity and input resolution as training progresses.The paper also used ReduceLROnPlateau and LinearWarmupCosineAnnealingLR learning-rate schedulers.Source: §2.5 · p. 3
2.5.1 Drip Training
- Drip Training gradually increases model complexity to address convergence problems encountered with more than one million ArcFace classes, without forgetting learned embeddings.The procedure iteratively increases the number of classes in the ArcFace head.Source: §2.5.1 · p. 3, §2.5 · p. 3
- Training starts with 40k target classes, which can converge in about an hour; after loss falls below 1.0, the ArcFace output-class count is increased to roughly twice its prior size.Source: §2.5.1 · pp. 3–4
- To preserve learned embeddings after expansion, the backbone averages one unaugmented and four augmented embeddings per class to rebuild the ArcFace centroid matrix W.Training then resumes from the updated W, and expansion and retraining repeat until the desired class count is reached.Source: §2.5.1 · p. 4
2.6 Ensembling
The ensemble method projects different models’ embeddings into a common space, selects 256 dimensions, and applies eigenvalue-based scaling followed by L2 normalization.
2.6 Ensembling
- Embeddings from different models are concatenated and projected with PCA into a common vector space, using unaugmented training samples to fit PCA.The Faiss library was used for PCA.Source: §2.6 · p. 5
- The ensemble output retains the first 256 most important dimensions of the projected space.Source: §2.6 · p. 5
- Scaling the output embedding by the square root of eigenvalues and then applying L2 normalization was the best-scoring tested normalization.Source: §2.6 · p. 5
2.7 Query embedding normalization
Query embeddings can be rescaled to normalize similarity comparisons when the competition requires submitted embeddings rather than normalized scores. The section presents two approaches, with the method using nearby training embeddings performing best in Phase 2.
2.7 Query embedding normalization
- The normalization addresses the need to compare query-reference similarity scores while submitting embeddings, by using an external dataset such as the training set.The competition requires embeddings, so the score-normalization expression cannot be used directly for the description track.Source: §2.7 · pp. 5–6
2.7.1 Method 1: Escaping from the sphere
- Method 1 rescales a query embedding along its existing direction using its three closest training embeddings, moving it outside the unit sphere and away from reference embeddings.The procedure preserves the query vector’s direction.Source: §2.7.1 · p. 6
2.7.2 Method 2: Using training images
- Method 2 improved Phase 2 µAP from 0.53 to 0.59, making it the most effective of the two query-normalization methods.It moves each query embedding away from its 100 closest training embeddings, assuming their distribution resembles that of the closest reference embeddings.Source: §2.7.2 · pp. 7–8
3 Conclusion
The solution combines CNN backbones trained with ArcFace with embedding ensembling, query normalization, and progressive training. The ensembled system scored µAP = 0.59, placing 2nd in Phase 2.
3 Conclusion
- µAP = 0.59 earned 2nd place in Phase 2 after ensembling the models.Source: §3 · p. 8
- The solution used different CNN backbones trained with an ArcFace head to produce image embeddings.Source: §3 · p. 8
- The authors presented an ensemble, two query-embedding normalization methods, and progressive training for rapid model training.Source: §3 · p. 8