Source-linked AI summary
Linkage Based Face Clustering via Graph Convolution Network
Zhongdao Wang, Liang Zheng, Yali Li, Shengjin Wang
TL;DR
Face clustering must handle complex face-representation distributions without relying on restrictive assumptions or a known number of identities. The paper learns identity links from local neighbor context using Instance Pivot Subgraphs and a GCN, reporting robust, scalable performance comparable to state-of-the-art methods.
Problem
Face clustering must group faces by identity despite complex representation distributions, while avoiding assumptions about data distribution or the number of identities.
Method
The method formulates clustering as link prediction and uses context-encoding Instance Pivot Subgraphs with a GCN to learn linkage likelihoods between pivots and neighbors.
Results
The method is reported as more robust than conventional approaches, favorably comparable to state-of-the-art face clustering methods, and scalable to large datasets.
Takeaways & Limitations
The approach can cluster without the number of clusters as input, handle noises and outliers, and extend to multi-view face clustering.
Abstract
from arXiv · showhide
In this paper, we present an accurate and scalable approach to the face clustering task. We aim at grouping a set of faces by their potential identities. We formulate this task as a link prediction problem: a link exists between two faces if they are of the same identity. The key idea is that we find the local context in the feature space around an instance (face) contains rich information about the linkage relationship between this instance and its neighbors. By constructing sub-graphs around each instance as input data, which depict the local context, we utilize the graph convolution network (GCN) to perform reasoning and infer the likelihood of linkage between pairs in the sub-graphs. Experiments show that our method is more robust to the complex distribution of faces than conventional methods, yielding favorably comparable results to state-of-the-art methods on standard face clustering benchmarks, and is scalable to large datasets. Furthermore, we show that the proposed method does not need the number of clusters as prior, is aware of noises and outliers, and can be extended to a multi-view version for more accurate clustering accuracy.
1. Introduction
The paper treats face clustering as predicting identity links between faces, using local context and a GCN to learn linkage likelihoods without distributional or cluster-count assumptions.
- Motivation: Face clustering groups faces by underlying identity without assuming the distribution of face representations or the number of identities.The task supports photo organization, large-scale retrieval, and dataset cleaning or labeling.
- Motivation: Conventional clustering methods struggle with complex face-representation distributions because they impose restrictive data assumptions.Linkage-based methods avoid these assumptions by predicting whether nodes or clusters share an identity.
- Approach: The proposed method learns linkage likelihood from node context with a parametric model instead of relying on heuristic metrics.This context-based formulation is illustrated as the paper’s alternative to distance thresholding and heuristic linkage estimation.
- Approach: Clustering is formulated as link prediction, with local Instance Pivot Subgraphs representing pivot-neighbor context for GCN reasoning.The GCN predicts linkage likelihoods, after which linked nodes are transitively merged into clusters.
- Results: The method is reported as accurate, scalable, noise-aware, independent of the input cluster count, and extensible to multi-view data.The paper contrasts its learned linkage likelihood with heuristic linkage methods such as ARO.
2. Related Work
Prior work addresses complex face-clustering distributions with hierarchical, linkage-based, and graph-based methods, while this paper combines efficient local-neighbor reasoning with a spatial GCN.
- Face Clustering: Face clusters vary in size, shape, and density, limiting classic methods that assume convex, balanced, or similarly dense distributions.Agglomerative hierarchical approaches are described as more robust to these complex distributions.
- Face Clustering: Hierarchical methods improve robustness to complex distributions, but their computational complexity can limit large-scale clustering.Prior work also addresses noise, outliers, and density imbalance through specialized affinity or distance designs.
- Linkage-Based Clustering: ARO improves efficiency by linking each node to k nearest neighbors, with complexity O(kn) before nearest-neighbor acceleration.Overall complexity is reported as O(n log n) for constant k or O(n^2) when k increases with n.
- Link Prediction: The paper relates face clustering to link prediction, where graph methods estimate links using local neighborhoods or learned local subgraph structure.This connection motivates reducing clustering to link prediction and learning from local graphs.
- Graph Convolutional Networks: Graph convolutional networks operate on graph-structured data through spectral or spatial formulations.The paper uses a spatial-based GCN for inductive link-prediction classification.
- Motivation: An upper-bound experiment on IJB-B-512 reports reasonably high F-measure and NMI for same-identity links among k nearest neighbors.This supports using kNN linkage as an efficient basis for clustering.
3. Proposed Approach
The method formulates face clustering as link prediction and uses local Instance Pivot Subgraphs with graph convolutions to infer pivot-neighbor linkages before merging linked instances into clusters.
- Face clustering assigns pseudo labels by predicting linkages between instance pairs, with linked pairs forming clusters.
- IPS construction: The method focuses linkage prediction on each instance’s nearest neighbors and constructs a local Instance Pivot Subgraph containing neighbors up to h hops.The subgraph represents the local context around a pivot.
- IPS construction: IPS node features are normalized by subtracting the pivot feature, producing residual vectors that encode pivot-neighbor relationships.For each node q, the normalized feature is x_q − x_p.
- IPS construction: Edges connect IPS nodes to their top u nearest neighbors when those neighbors also belong to the IPS, preserving local topology.The resulting adjacency matrix and feature matrix represent the subgraph input.
- Graph convolutions: A graph convolution layer aggregates neighbor information, concatenates it with each node’s features, and transforms the result with learned weights and a nonlinear activation.The aggregation may use mean, weighted, or attention-based strategies.
- Graph convolutions: The four-layer GCN predicts linkage likelihoods for IPS nodes, while gradients are backpropagated only for 1-hop neighbors to accelerate training.The predicted weighted edges are then used to transitively merge linked instances into clusters.
4. Experiment
Experiments evaluate the method with NMI and BCubed F-measure on IJB-B, testing hyperparameters, aggregators, baselines, representation effects, singleton filtering, scalability, and multi-view extension. Results support strong performance across comparisons, efficient large-scale processing, and gains from multi-view data.
- Evaluation setup: The evaluation uses normalized mutual information and BCubed F-measure, with separate training and testing datasets and IJB-B clustering subtasks.ArcFace provides face representations; CASIA is used for GCN training, while IJB-B is used for testing.
- Parameter analysis: Using h = 2, the testing analysis varies k1, k2, and u; k1 = 80, k2 = 5, u = 5 provide a reported efficiency–performance trade-off.The study reports increasing F-measure with larger k1 and k2, while limiting these values for efficiency.
- Ablation comparisons: Attention aggregation is marginally better than mean and weighted aggregation, but the improvement is not significant given its computation cost.The method therefore uses mean aggregation in subsequent experiments.
- Baseline comparisons: The proposed method consistently outperforms other state-of-the-art methods on three subtasks in both F-measure and NMI, while outperforming AHC without threshold sensitivity.The paper cautions that PAHC and ConPaC use different face representations, limiting direct comparison.
- Representation effects: With the same representation, the method outperforms state-of-the-art methods and benefits from stronger representations such as ArcFace.These comparisons are reported under both ArcFace and ResNet-50 representations.
- Scalability and multi-view extension: The local-graph design gives overall O(n log n) complexity with approximate nearest-neighbor search, and the method gains performance from multi-view face and audio features.The paper reports that heuristic methods can fail on concatenated face+audio features, whereas the proposed parametric model leverages their complementary information.
5. Conclusion
The paper proposes a linkage-based face-clustering method that uses instance pivot subgraphs and a graph convolution network to reason about linkage likelihood. Experiments indicate robustness to complex face distributions, favorable comparison with state-of-the-art methods, scalability, and effectiveness for visual-audio clustering.
- The method constructs instance pivot subgraphs to represent each node’s local context.
- A graph convolution network reasons over each subgraph to estimate linkage likelihood between a node and its neighbors.
- Experiments indicate greater robustness to complex face distributions than conventional methods.
- The method achieves favorably comparable results to state-of-the-art methods on standard face-clustering benchmarks and scales to large datasets.
- The approach is also demonstrated for visual-audio face clustering.