Source-linked AI summary
Multi-Level Factorisation Net for Person Re-Identification
Xiaobin Chang, Timothy M. Hospedales, Tao Xiang
TL;DR
Person Re-ID needs representations that retain discriminative, view-invariant appearance factors across low and high semantic levels without costly manual attribute annotation. MLFN learns and dynamically selects latent factor modules across stacked blocks, then fuses their compact Factor Signature with conventional deep features. It achieves state-of-the-art results on three Re-ID datasets and compelling results on CIFAR-100, although reported results remain below some larger-network state-of-the-art models in an ablation study.
Problem
Existing Re-ID models typically use holistic final-layer representations or require costly, error-prone manual annotations to model multi-level discriminative appearance factors.
Method
MLFN stacks blocks of factor modules and factor selection modules that dynamically identify latent factors at different semantic levels and fuse their outputs into a compact Factor Signature with deep features.
Results
MLFN achieves state-of-the-art performance on three person Re-ID benchmarks and compelling results on the general object categorisation CIFAR-100 dataset.
Takeaways & Limitations
The architecture provides an annotation-free way to learn multi-level latent factors and supports both person Re-ID and a general object categorisation task.
Takeaways & Limitations
Some ablation results remain below state-of-the-art results from much larger networks because those models and batch sizes exceeded the available GPU resources.
Abstract
from arXiv · showhide
Key to effective person re-identification (Re-ID) is modelling discriminative and view-invariant factors of person appearance at both high and low semantic levels. Recently developed deep Re-ID models either learn a holistic single semantic level feature representation and/or require laborious human annotation of these factors as attributes. We propose Multi-Level Factorisation Net (MLFN), a novel network architecture that factorises the visual appearance of a person into latent discriminative factors at multiple semantic levels without manual annotation. MLFN is composed of multiple stacked blocks. Each block contains multiple factor modules to model latent factors at a specific level, and factor selection modules that dynamically select the factor modules to interpret the content of each input image. The outputs of the factor selection modules also provide a compact latent factor descriptor that is complementary to the conventional deeply learned features. MLFN achieves state-of-the-art results on three Re-ID datasets, as well as compelling results on the general object categorisation CIFAR-100 dataset.
1. Introduction
Person Re-ID must handle drastic cross-camera appearance changes while preserving discriminative factors across multiple semantic levels. MLFN addresses this by dynamically factorising appearance and fusing compact multi-level factor signatures with deep features, achieving state-of-the-art results on three Re-ID benchmarks and showing effectiveness on CIFAR-100.
- Motivation: Cross-camera changes in illumination, background, view-angle, and pose make person Re-ID difficult, despite view-invariant identity-discriminative factors existing in appearance.These factors range from low-level colour and texture to high-level clothing type and gender.
- Motivation: Standard deep Re-ID models typically use only top-layer features, limiting their ability to preserve discriminative factors across multiple semantic levels.Higher network layers generally capture more abstract concepts, while lower layers represent lower-level visual information.
- MLFN Architecture: MLFN stacks blocks containing factor modules and factor selection modules to learn and dynamically activate latent identity-discriminative, view-invariant factors at different semantic levels.Bottom blocks can represent low-level attributes such as clothing colour, while top blocks can represent higher-level attributes such as object carrying and gender.
- MLFN Architecture: MLFN concatenates factor-selection outputs into a compact Factor Signature and fuses it with the final-layer deep feature for training and recognition.This provides multi-level fusion without exploding dimensionality and supplies deep supervision through skip connections.
- Results: MLFN achieves state-of-the-art performance on Market-1501, CUHK03, and DukeMTMC-reID, and is also effective on CIFAR-100 object categorisation.The reported results indicate applicability beyond person Re-ID.
2. Related Work
Prior Re-ID approaches either rely on final-layer representations, manually annotated attributes, or handcrafted multi-level fusion mechanisms. MLFN instead extends modular architectures with image-dependent module selection to learn and encode latent factors across semantic levels.
- Deep Neural Networks for Person Re-ID: Most deep Re-ID systems extract final-layer features, although network representations become more abstract toward the top and cannot capture all semantic levels there.This creates a mismatch between final-layer feature extraction and the need for multi-level discriminative appearance information.
- Attribute-Based Methods: Attribute-based methods require manually defining diverse attributes and collecting large-scale image-attribute annotations.Such annotations are costly and error-prone, and exhaustive factor dictionaries are difficult to define.
- Multi-Level Fusion: Multi-level fusion methods complement final-layer features with representations from other layers but may require body-part detection, attention mechanisms, or handcrafted layer selection.These additional design choices increase effort and can lead to suboptimal solutions.
- Related CNN Architectures: MLFN extends the ResNeXt split-transform-merge pattern by dynamically selecting which subset of duplicate modules activates for each input image.This enables modules to specialise in different latent appearance factors while selection outputs encode factor information.
- Mixture-of-Expert Models: Unlike conventional mixture-of-experts models that identify a single expert, MLFN allows multiple factor modules to be active and uses their selections as a compact latent semantic feature.The architecture is designed to model multiple factors within each semantic level.
3. Methodology
MLFN factorises each input into latent factors across multiple semantic levels, dynamically selects relevant factors, and combines their compact signatures with final-layer features. Its architecture uses stacked blocks, factor modules, factor selection modules, shortcut connections, and deep supervision to learn discriminative factors efficiently.
- MLFN architecture: MLFN stacks blocks across semantic levels, with each block containing identically structured factor modules that model latent factors and a factor selection module that selects them dynamically.The factor selection module produces a sigmoid-activated vector so multiple factors can be selected simultaneously for an input.
- MLFN architecture: Each block combines the selected factor-module outputs with its input through a shortcut connection to produce the block output.The selected outputs are modulated through a mode-4 tensor-matrix product before being added to the block input.
- Factor Signature: The Factor Signature concatenates factor-selection vectors from all blocks, preserving multi-level information in a dimension independent of the deep feature-map dimensions.Its dimensionality depends on the number of blocks and factor modules, rather than the height, width, or channel dimensions of the block outputs.
- Fusion: MLFN averages projected final-block features and the Factor Signature after mapping both to a common dimension d.The projection function T is implemented as a fully connected layer.
- Dynamic factor selection: Each factor selection value indicates a factor module’s contribution: values near 1 apply the input fully to that module, whereas values near 0 stop its parameter update.This dynamically gates both factor processing and learning for each input image.
- Training and supervision: Fusing the Factor Signature with the final representation supplies direct supervision to factor-selection modules through skip connections without the parameter cost of conventional deep supervision.This addresses weak indirect supervision for selection modules in lower blocks.
4. Experiments
MLFN is evaluated across three person Re-ID benchmarks, CIFAR-100, component ablations, factor-signature matching, and latent-factor analyses. It consistently performs strongly, with results supporting multi-level fusion, dynamic factor selection, and compact factor signatures.
- Person Re-ID Results: 82.8% rank-1 accuracy is achieved on CUHK03 Setting 1 with detected person bounding boxes.On the harder CUHK03 Setting 2, MLFN’s rank-1 performance is more than 11% higher than DPFL using either labelled or detected person images.
- Person Re-ID Results: MLFN is 1.8% higher than DPFL on rank-1 accuracy and 2.2% higher on mAP for DukeMTMC-reID.The authors relate this performance to fusing features from multiple network levels, which helps cope with large scale changes.
- Object Categorisation Results: MLFN improves over ResNeXt on CIFAR-100, indicating that dynamic factor-module selection and the factor-signature feature benefit general object categorisation.It also outperforms DualNet despite DualNet doubling model size through an ensemble-like pair of ResNet branches.
- Analysis on Latent Factors: Ablations show that each added component improves performance across all three person Re-ID datasets.The comparisons specifically support complementary information from the factor signature and benefits from dynamic module selection.
5. Conclusion
MLFN learns discriminative, view-invariant latent factors at multiple semantic levels and dynamically identifies those relevant to each input image. Fusing factor selections with conventional deep features yields a powerful person representation, with state-of-the-art Re-ID results and promising broader categorisation performance.
- MLFN discovers and dynamically identifies discriminative latent appearance factors in input images for person Re-ID.
- Factors computed at different network levels correspond to latent attributes with different semantic levels.
- Fusing factor selections with conventional deep features produces a powerful view-invariant person representation.
- MLFN achieves state-of-the-art results on three large Re-ID datasets and promising results on general object categorisation.
A.1. MLFN Architecture Parameter Selection
The architecture uses 16 blocks and selects a 1024-dimensional final feature without tuning to match ResNeXt-50's overall size. Performance remains consistently good when the feature dimension exceeds 512.
- 16 blocks are used in MLFN, following the ResNeXt-50 architecture.
- The final feature dimension d is set to 1024 for comparison with widely used Person Re-ID feature dimensions.
- MLFN and ResNeXt have similar runtime, at 0.81s/batch and 0.78s/batch respectively.
- Re-ID performance is consistently good when the final feature dimension d > 512.
- Each MLFN block includes a Factor Selection Module, while FSM networks use three-layer MLPs with global average pooling, batch normalisation, and ReLU.
- Table 8 reports FSM architecture details across the MLFN blocks, with block 16 denoting the last block.
A.2. Examples of FS Predicted Attributes
The factor signature can predict human-annotated attributes, including in images with occlusion, blur, or rear viewpoints. Errors occur for attributes that are visually hidden or ambiguous even to humans.
- The factor signature alone already outperforms the supervised APR attribute-prediction model on DukeMTMC-reID.The reported comparison is 82.30% for MLFN versus 80.12% for APR.
- Human annotators provide 35 binary attributes at the identity level, so images of the same person share identical attribute vectors.
- Some attributes are subjective, including age categories without known ground truth or clear definitions.
- The factor-signature feature correctly predicts all attributes in examples involving face occlusion and a blurred rear-view image.
- In a frontal example, the factor-signature prediction misses a backpack and confuses blue and black lower-body clothing.