Source-linked AI summary
Quality Aware Network for Set to Set Recognition
Yu Liu, Junjie Yan, Wanli Ouyang
TL;DR
Set-to-set recognition benefits from complementary images, but poor-quality samples can damage pooled representations. QAN jointly learns image features and sample quality without explicit quality labels, then uses quality-weighted aggregation; experiments report improvements on face verification and person re-identification benchmarks.
Problem
Set-to-set recognition must exploit complementary images while handling samples whose poor quality can hurt the learned metric.
Method
QAN uses separate feature and quality branches, aggregates their outputs into a set representation, and trains both branches end-to-end without explicit quality supervision.
Results
QAN improves recognition on person re-identification and face verification benchmarks, including 14.6% and 9.0% top-1 gains and 15.6% and 29.32% miss-ratio reductions.
Takeaways & Limitations
The learned quality scores are reported to benefit set representation and perform better than human-labeled quality scores for recognition.
Takeaways & Limitations
The image-level quality model may overlook discriminative regions within poor-quality images, motivating a future fine-grained regional version.
Abstract
from arXiv · showhide
This paper targets on the problem of set to set recognition, which learns the metric between two image sets. Images in each set belong to the same identity. Since images in a set can be complementary, they hopefully lead to higher accuracy in practical applications. However, the quality of each sample cannot be guaranteed, and samples with poor quality will hurt the metric. In this paper, the quality aware network (QAN) is proposed to confront this problem, where the quality of each sample can be automatically learned although such information is not explicitly provided in the training stage. The network has two branches, where the first branch extracts appearance feature embedding for each sample and the other branch predicts quality score for each sample. Features and quality scores of all samples in a set are then aggregated to generate the final feature embedding. We show that the two branches can be trained in an end-to-end manner given only the set-level identity annotation. Analysis on gradient spread of this mechanism indicates that the quality learned by the network is beneficial to set-to-set recognition and simplifies the distribution that the network needs to fit. Experiments on both face verification and person re-identification show advantages of the proposed QAN. The source code and network structure can be downloaded at https://github.com/sciencefans/Quality-Aware-Network.
1. Introduction
Set-to-set recognition can exploit complementary images, but poor-quality samples may mislead pooled representations. QAN learns sample quality jointly with image features to improve recognition across face verification and person re-identification.
- Motivation: Appearance variations from pose or illumination can still cause failures despite strong training data and convolutional methods.
- Motivation: Set-to-set recognition measures distances between image sets, while complementary samples can provide more information than a single image.The tasks are used in applications including financial identity authentication and video surveillance.
- Motivation: Equal-weight max or average pooling lets noisy images caused by shake or blur mislead the final set representation.
- Quality aware network: QAN uses a feature-generation branch and a quality-generation branch, aggregating image features according to predicted quality scores.
- Quality aware network: The branches are trained end-to-end without explicit quality annotations, using set-level optimization with joint triplet and softmax losses.The set-pooling gradient supports automatic quality learning during back propagation.
- Results: 14.6% and 9.0% top-1 matching-rate improvements occur on iLIDS-VID and PRID2011, while miss ratios fall 15.6% and 29.32% on YouTube Face and IJB-A.The paper reports new state-of-the-art performance on four person re-identification and face verification benchmarks.
2. Related work
Prior set-to-set recognition methods either model sets as geometric spaces or aggregate samples into a single representation. QAN follows the aggregation approach while jointly learning image features and quality-based aggregation.
- Geometric representations: One line of prior work represents image sets as convex hulls, affine hulls, or subspaces and formulates recognition as metric learning.These representations place samples in Hilbert or Grassmann spaces.
- Feature aggregation: Another line reduces set-to-set recognition to point-to-point recognition by aggregating all images into one hyperspace representation.
- Feature aggregation: Aggregation methods include Bag of Features, VLAD, temporal max or average pooling, second-order statistics, pose-quality bins, and attention mechanisms.These methods use different statistics, partitions, or learned attention to summarize image sets.
- QAN’s distinction: QAN belongs to the aggregation approach and jointly learns feature representation and aggregation rather than relying on fixed features or images.
- QAN’s distinction: Unlike human-defined visual-entropy scores, QAN automatically learns quality scores and jointly trains quality and feature generation units.The paper attributes significant improvement to jointly optimizing aggregation parameters and the image feature generator.
3. Quality aware network (QAN)
QAN maps variable-length image sets to fixed-dimensional representations by weighting each image’s feature embedding with an automatically learned quality score. Its two branches are trained jointly so useful samples receive greater influence while poor or hard samples contribute less.
- Image set embedding: QAN maps an image set to a fixed-dimensional representation by aggregating image representations with an optimized function.This makes sets containing different numbers of images comparable.
- QAN architecture: The network splits into feature and quality branches: one generates image representations, while the other predicts scores used to weight and aggregate them.The weighted features are summed and passed through a fully connected layer to produce the final set representation.
- Training QAN: QAN trains end to end with image-level Softmax loss and set-level triplet loss applied to anchor, positive, and negative set representations.The triplet loss minimizes intra-class variation while improving discrimination between set representations.
- Mechanism for learning quality score: Quality scores emerge through back propagation: samples aligned with the set gradient gain weight, whereas samples in opposing directions receive lower scores.The mechanism interprets images as useful or hard samples based on their contribution to the set-level objective.
- Mechanism for learning quality score: Quality scores also scale feature gradients, weakening the harmful effect of poor or noisy samples and focusing feature learning on better samples.This reduces the influence of hard images during back propagation and benefits set-to-set recognition.
- Details of quality generation part: The quality generation unit converts middle representations into normalized image weights, using convolution, pooling, a fully connected layer, sigmoid activation, and group L1 normalization.The unit is designed to produce one normalized score per image in a set.
4. Experiments
The experiments examine what QAN learns, how feature-level choices affect its quality prediction, and its performance on person re-identification and face verification benchmarks.
- The experiments analyze QAN’s learned quality score, sensitivity to feature level, benchmark performance, and relationship to human quality labels.
4.1. What is learned in QAN?
QAN assigns lower quality to visually degraded or ambiguous images, and its predictions correlate strongly with human judgments while sometimes outperforming human scoring for recognition.
- Images with deformity, superposition, blur, or extreme lighting tend to receive lower QAN quality scores than normal images.
- Images scoring above 0.70 are generally easy for humans to recognize, whereas lower-scoring images are harder, often containing multiple bodies.
- QAN quality scores are strongly correlated with human-defined quality across ten human-score partitions.
4.2. Person re-identification
QAN improves person re-identification over pooling baselines and generalizes across datasets, with larger gains on the noisier iLIDS-VID benchmark.
- 4.2.1 Evaluation on common setting: On PRID2011, QAN increased top-1 matching rate by 11.1% versus CNN+AvePool and 29.4% versus CNN+Min(cos).
- 4.2.1 Evaluation on common setting: On iLIDS-VID, QAN increased top-1 matching rate by 12.21% versus CNN+AvePool and 37.9% versus CNN+Min(cos).
- 4.2.1 Evaluation on common setting: QAN’s gains were larger on noisy iLIDS-VID, where Min(cos) is more sensitive to noisy samples than average pooling.
- 4.2.1 Evaluation on common setting: QAN exceeded the previous best CNN+RNN top-1 matching rate by 20.3% on PRID2011 and 10% on iLIDS-VID.
- 4.2.2 Dataset cross evaluation: In cross-dataset evaluation without fine-tuning, QAN improved top-1 matching by 15.6% and 8.2% over the baselines.
4.3. Unconstrained face verification
QAN improves unconstrained face verification across YouTube Face and IJB-A, particularly under low false-positive-rate evaluation.
- 4.3.1 Results on YouTube Face and IJB-A benchmark: On YouTube Face, QAN reduced the error ratio by 15.6% relative to its baselines.
- 4.3.1 Results on YouTube Face and IJB-A benchmark: At 0.001 FPR, QAN surpassed NAN by 8% and DeepFace by 80% under ROC evaluation.
- 4.3.1 Results on YouTube Face and IJB-A benchmark: On IJB-A, QAN outperformed NAN by 10.81% at 0.001 FPR, 4.5% at 0.01 FPR, and 2.12% at FPR=0.1.
- 4.3.1 Results on YouTube Face and IJB-A benchmark: Compared with average pooling on IJB-A, QAN reduced false negative rate by 29.32%, 6.45%, and 7.91% at the three reported FPRs.
4.4. Quality by QAN VS. quality by human
Replacing QAN’s learned quality scores with volunteer ratings improves over two baselines but performs worse than the original QAN.
- Volunteer-rated quality scores outperform two baselines but remain inferior to the original QAN result on YouTube Face.The comparison indicates that learned scores are similar to human judgments but more suitable for recognition.
4.5. Diagnosis experiments
Diagnosis experiments vary where the quality-generation branch connects within the network and compare jointly trained versus fixed feature generation. Performance peaks at Pool3, while joint training performs better than fixing feature generation.
- Middle representation: QAN performance improves initially and reaches its highest accuracy when the quality-generation part is connected at Pool3.The experiment compares configurations ranging from the image input through successive pooling layers and a fully connected layer.
- Training strategy: Jointly training the feature-generation and quality-generation parts performs better than fixing the feature-generation part.The result supports end-to-end optimization of both branches.
5. Conclusion and future work
The paper concludes that QAN learns sample quality without supervision and uses it to aggregate discriminative images for set representation. It identifies fine-grained region-level quality modeling as future work because poor-quality images may still contain discriminative regions.
- Conclusion: QAN learns sample quality without supervised signals and aggregates the most discriminative samples into a set representation.The conclusion states that the predicted quality benefits set representation and is better than human labeling.
- Future work: Future work will model quality at the region level because a poor-quality image may still contain discriminative regions.The proposed direction extends attention from high-quality images to high-quality regions within images.