Source-linked AI summary
Neural Aggregation Network for Video Face Recognition
Jiaolong Yang, Peiran Ren, Dongqing Zhang, Dong Chen, Fang Wen, Hongdong Li, Gang Hua
TL;DR
Video face recognition requires compact representations that integrate variable-quality frames while avoiding costly frame-to-frame matching. NAN uses a CNN embedding module and adaptive attention aggregation trained with standard supervision, and it consistently outperforms baseline strategies across three challenging datasets.
Problem
Video face recognition needs compact, fixed-size representations that integrate informative frames while handling variable video lengths and noisy face images.
Method
NAN combines deep CNN frame embeddings with an adaptive aggregation module whose coefficients are learned through standard supervised face-recognition training.
Results
NAN showed consistent margins over baseline strategies and competing methods for verification and identification on YouTube Face, IJB-A, and Celebrity-1000.
Takeaways & Limitations
The network produces compact, order-invariant representations and can also aggregate multiple data sources into a fixed-size subject-level feature.
Takeaways & Limitations
The aggregation is designed for orderless image sets and does not use temporal information from sequential inputs.
Abstract
from arXiv · showhide
This paper presents a Neural Aggregation Network (NAN) for video face recognition. The network takes a face video or face image set of a person with a variable number of face images as its input, and produces a compact, fixed-dimension feature representation for recognition. The whole network is composed of two modules. The feature embedding module is a deep Convolutional Neural Network (CNN) which maps each face image to a feature vector. The aggregation module consists of two attention blocks which adaptively aggregate the feature vectors to form a single feature inside the convex hull spanned by them. Due to the attention mechanism, the aggregation is invariant to the image order. Our NAN is trained with a standard classification or verification loss without any extra supervision signal, and we found that it automatically learns to advocate high-quality face images while repelling low-quality ones such as blurred, occluded and improperly exposed faces. The experiments on IJB-A, YouTube Face, Celebrity-1000 video face recognition benchmarks show that it consistently outperforms naive aggregation methods and achieves the state-of-the-art accuracy.
1. Introduction
Video face recognition needs compact, order-invariant representations that integrate useful information across variable-quality frames without costly frame-to-frame matching. NAN addresses this with supervised, adaptive feature aggregation and performs consistently better than baseline and competing methods across three challenging datasets.
- Video face recognition must integrate information across varying poses and illumination while discarding noisy frame content.
- Frame-pair matching preserves information but incurs O(n^2) complexity per match when videos average n frames.
- A compact, fixed-size video representation enables direct constant-time similarity computation regardless of video length.
- NAN adaptively learns weights to linearly combine frame features using standard supervised face-recognition training without extra supervision.
- Its feature embedding CNN extracts frame-level vectors, while an aggregation module adaptively fuses them into a video-level representation.
- The attention mechanism supports arbitrary input sizes, order-invariant representations, and selective emphasis or suppression of individual inputs.
- NAN was evaluated on video face verification and identification, showing consistent margins over baselines and competing methods on YouTube Face, IJB-A, and Celebrity-1000.
2. Neural Aggregation Network
NAN maps variable-length face-image inputs into compact representations by embedding each image and adaptively aggregating the resulting features with attention. Its design supports order-invariant aggregation and learns image weighting through standard recognition training.
- Overview: NAN takes a face-image set or video sequence and outputs a single fixed-size feature representation.The aggregated vector has the same size as an individual CNN-extracted face feature.
- Feature embedding module: The feature embedding module uses a deep CNN to convert each face image into a normalized feature vector.The paper adopts GoogLeNet with batch normalization, producing 128-dimensional image features.
- Aggregation module: The aggregation module computes adaptive linear weights for all input features instead of using fixed or heuristic weighting.Uniform weights reduce the method to naive averaging, which the paper describes as usually non-optimal.
- Aggregation module: Attention-based aggregation handles varying numbers of images and remains unchanged when the input order is permuted.The result does not depend on the number of input features or their ordering.
- Attention blocks: The full NAN uses two cascaded attention blocks, with the second block's kernel computed from the first aggregated feature for content-aware weighting.The two-block design is trained end-to-end within the aggregation module and is used for the remaining experiments.
- Training: Training supports verification with contrastive loss or identification with a classification loss, using standard recognition objectives.The CNN and aggregation modules may be trained jointly or separately; this work chooses separate training.
3. Experiments
Experiments evaluate NAN against pooling and pairwise-comparison baselines on IJB-A, YouTube Face, and Celebrity-1000. NAN produces compact 128-d representations with O(1) similarity computation and consistently improves recognition performance, especially under large face variation.
- Evaluation setup: The evaluation covers IJB-A, YouTube Face, and Celebrity-1000 using verification and identification benchmarks.IJB-A reports TAR/FAR, TPIR/FPIR, and Rank-N accuracies; ROC, CMC, and DET curves are also shown.
- Baselines and efficiency: NAN and feature-wise max or average pooling produce 128-d video representations and compute similarity in O(1) time.Pairwise frame-comparison baselines require O(n) storage and generally O(n^2) similarity computation.
- IJB-A: On IJB-A without media aggregation, NAN outperforms all baselines, with TAR 0.860 at FAR 0.001 and 0.933 at FAR 0.01.These results reduce the best-baseline errors by about 39% and 23%, respectively.
- IJB-A: With media aggregation on IJB-A, NAN reports 0.881 versus 0.836 TAR at FAR 0.01, 0.817 versus 0.774 TPIR at FPIR 0.01, and 0.958 versus 0.928 Rank-1 accuracy.The comparison is against the state-of-the-art method cited as [10].
- YouTube Face: On YouTube Face, NAN achieves 95.72% mean accuracy and reduces FaceNet’s error by 12.3%.The reported VGG-Face result is higher at 97.3% with metric learning, but 91.5% without it.
- Celebrity-1000: On Celebrity-1000, NAN consistently outperforms baselines for both VideoAggr and SubjectAggr, while average pooling worsens at the subject level.NAN benefits from SubjectAggr and remains better than or on par with VideoAggr, indicating robustness to large data variations.
4. Conclusions
NAN fuses video frames with content-adaptive weights into a compact, frame-order-invariant representation. It uses modest computation and memory while producing quality face representations after training.
- NAN fuses all input frames using content-adaptive weights into a compact face representation.The weights depend on the input content rather than being fixed.
- The resulting representation is invariant to the input frame order.
- NAN uses small computation and memory footprints while generating quality face representations after training.