Source-linked AI summary
Person Re-Identification via Recurrent Feature Aggregation
Yichao Yan, Bingbing Ni, Zhichao Song, Chao Ma, Yan Yan, Xiaokang Yang
TL;DR
The paper addresses multi-shot person re-identification by seeking discriminative representations from tracked human sequences rather than relying only on single images or complex sequence matching. It proposes LSTM-based recurrent feature aggregation to fuse temporally ordered frame-wise features, and reports favorable performance against state-of-the-art methods on two benchmarks. The method also remains effective with simple hand-crafted texture and color features, although failures occur under color inconsistency, occlusion, and highly similar appearance conditions.
Problem
Multi-shot person re-identification requires combining frame-wise appearance and temporal dynamics, while prior single-shot, graph-matching, and pooling approaches have limited sequence modeling or computational robustness.
Method
RFA-Net uses an LSTM to progressively fuse temporally ordered frame-wise features, propagating informative features and forgetting noisy ones before identity classification.
Results
The method performs favorably against state-of-the-art person re-identification methods on two public benchmark datasets, including more than 20% higher rank-1 matching rate than DVR.
Takeaways & Limitations
Even simple texture and color features can produce discriminative and noise-robust sequence-level representations when augmented with temporal cues.
Takeaways & Limitations
Reported failures include cases involving color inconsistency and occlusion, similar clothing colors, and similar poses; training also has only two sequences per person with variable sequence lengths.
Abstract
from arXiv · showhide
We address the person re-identification problem by effectively exploiting a globally discriminative feature representation from a sequence of tracked human regions/patches. This is in contrast to previous person re-id works, which rely on either single frame based person to person patch matching, or graph based sequence to sequence matching. We show that a progressive/sequential fusion framework based on long short term memory (LSTM) network aggregates the frame-wise human region representation at each time stamp and yields a sequence level human feature representation. Since LSTM nodes can remember and propagate previously accumulated good features and forget newly input inferior ones, even with simple hand-crafted features, the proposed recurrent feature aggregation network (RFA-Net) is effective in generating highly discriminative sequence level human representations. Extensive experimental results on two person re-identification benchmarks demonstrate that the proposed method performs favorably against state-of-the-art person re-identification methods.
1 Introduction
Multi-shot person re-identification must combine appearance and temporal information across tracked human sequences, but existing approaches have limitations in representation and aggregation. RFA-Net uses LSTM-based recurrent feature aggregation to produce discriminative sequence-level representations and performs favorably on benchmarks.
- Existing multi-shot methods use key-frame or fragment selection, feature fusion or encoding, and spatio-temporal appearance models.
- Multi-shot person re-identification seeks to aggregate frame-wise appearance and temporal dynamics into a discriminative sequence-level representation.
- RFA-Net feeds temporally ordered frame-wise features into an LSTM network to build a sequence-level representation.
- LSTM propagation accumulates discriminative information, filters non-informative features caused by occlusion or detection failures, and supports variable-length sequences.
- Experiments on two public benchmark datasets show favorable effectiveness and efficiency against state-of-the-art methods, while augmenting hand-crafted features with temporal cues.
2 Related work
Related work addresses person re-identification through feature representation and distance metric learning, with approaches spanning hand-crafted descriptors, sequence representations, metric learning, and deep networks. The paper positions LSTM-based modeling as a way to incorporate space-time information that image-pair methods do not consider.
- Person re-identification research commonly addresses feature representation and distance metric learning.
- Hand-crafted texture, color, and gradient features are widely used, but may not sufficiently distinguish people with similar appearances.
- Multiple-image and video methods accumulate features or construct space-time representations, but simple extensions of 2D features may remain insufficiently powerful.
- Metric-learning and ranking methods include Mahalanobis distance, boosting, learning-to-rank, and relative distance comparison formulations.
- Deep image-pair methods jointly learn representations and metrics but do not consider video space-time information and can overfit small databases.
3 Sequential Feature Aggregation for Person Re-Identification
RFA-Net addresses multi-shot person re-identification by sequentially aggregating frame-wise appearance features with an LSTM. The resulting sequence representation retains useful information, suppresses noise, and is matched using cosine distance or RankSVM.
- Multi-shot person re-identification requires aggregating frame-wise appearance and temporal information into a discriminative sequence representation.
- RFA-Net uses an LSTM to propagate information across temporally ordered frame features for progressive sequence-level fusion.
- Color and texture descriptors from overlapping patches form a 58950-dimensional frame input, while each LSTM time stamp outputs 512 dimensions.
- The LSTM memory cell weights previous memory and current input through forget and input gates, enabling useful features to accumulate while noisy features are discarded.
- The fused hidden state is trained through an N-way softmax identity-classification objective using stochastic-gradient backpropagation.
- Variable-length sequences are handled during training by randomly sampling fixed-length subsequences, using L = 10 in the experiments.
- Testing concatenates ten 512-dimensional LSTM outputs into a 5120-dimensional sequence representation for person matching.
- Similarity is measured with cosine distance or RankSVM, with distances ranked to compute top-k matching rates.
4 Experiments
Experiments evaluate RFA-Net on two person re-identification benchmarks, testing fusion depth, robustness to noisy frames, subsequence averaging, and comparisons with existing methods. Results show that deeper recurrent aggregation and multiple subsequences improve matching, while the method remains effective under substantial noise and outperforms several baselines.
- Experimental settings: RFA-Net was evaluated on iLIDS-VID and PRID 2011 using train/test splits and average CMC curves over 10 trials.The first camera supplies probes and the other camera supplies gallery sequences.
- Effectiveness of recurrent aggregation: Matching rates consistently increased with fusion depth, with the final sequence representation outperforming the first-node frame representation by about 10% on both datasets.Rank-1 matching increased faster than the other reported ranks as network depth grew.
- Effectiveness of recurrent aggregation: Using Color&LBP features, RFA-Net exceeded the baseline and DVR by more than 60% in rank-1 matching rate, even without metric learning.The comparison included average pooling with RankSVM, DTW sequence matching, and DVR; RFA-Net used cosine distance or RankSVM-based distance.
- Robustness: With up to 50% of frames replaced by noise images, RFA-Net achieved 29.8% and 44.7% rank-1 matching rates on the two datasets.Rank-10 and rank-20 rates decreased only slightly relative to noise-free sequences.
- Feature averaging: Averaging several randomly selected subsequence features improved performance by making representations more robust to pose and illumination changes; 10 and 15 subsequences performed similarly.The authors report that 10 subsequences were already sufficient to represent human dynamics well.
- Comparison to the state of the art: RFA-Net outperformed DVR by more than 20% in rank-1 matching rate and also surpassed DVDL, which does not encode temporal information into its dictionaries.The authors attribute the comparison advantage to using several subsequences and LSTM-based temporal encoding.
5 Conclusions
The paper proposes recurrent feature aggregation for learning discriminative sequence-level person representations from simple frame-wise features. Experiments show robustness to noise and performance better than or comparable to state-of-the-art methods.
- RFA-Net learns discriminative sequence-level representations from simple frame-wise features for person re-identification.
- The learned features remain robust to noise, including when texture and color features are used as inputs.
- The proposed method achieves performance better than or comparable to state-of-the-art person re-identification methods.